How to enable TCP keep-alive communications?

[CastagnaIT]

Hi there,
i am working on a multiplatform project and i am trying to do a migration from Requests module to httpx module,
because i will need to use HTTP/2 in the near future but i do not know how to do with the TCP keep-alive communications.

The Requests module uses urllib3 which by default have disabled the TCP keep-alive communications,
then in order to enable the TCP keep-alive communications previously i had done this:
https://github.com/CastagnaIT/plugin.video.netflix/blob/master/resources/lib/services/tcp_keep_alive.py#L56
as you can see in the enable_tcp_keep_alive method i set socket options in order to enable it,
and for Windows case only i use ioctl on the session adapter object to be able to enable it

but now how can i enable TCP keep-alive communications on httpx module?
can someone guide me?

[tomchristie]

We don't have any public API that'd allow you to do that, no.

You'd probably need to monkey-patch httpcore._backends.sync.SyncBackend...

https://github.com/encode/httpcore/blob/3f33e159f8802a191d09a186ae816aa7a8e37062/httpcore/_backends/sync.py#L137

[CastagnaIT]

i have done some test on Windows by add this on suggested line:

diff --git a/packages/httpcore/_backends/sync.py b/packages/httpcore/_backends/sync.py
index ee8f94b7..dc48031a 100644
--- a/packages/httpcore/_backends/sync.py
+++ b/packages/httpcore/_backends/sync.py
@@ -118,6 +118,11 @@ class SyncSemaphore:
         self._semaphore.release()


+TCP_KEEP_IDLE = 45
+TCP_KEEPALIVE_INTERVAL = 10
+TCP_KEEP_CNT = 6
+
+
 class SyncBackend:
     def open_tcp_stream(
         self,
@@ -137,6 +142,10 @@ class SyncBackend:
             sock = socket.create_connection(
                 address, connect_timeout, source_address=source_address  # type: ignore
             )
+            print('open_tcp_stream: SET KA')
+            sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
+            sock.ioctl(socket.SIO_KEEPALIVE_VALS, (1, TCP_KEEP_IDLE * 1000, TCP_KEEPALIVE_INTERVAL * 1000))
+
             if ssl_context is not None:
                 sock = ssl_context.wrap_socket(
                     sock, server_hostname=hostname.decode("ascii")

and seem works as expected at least on Windows OS (i will try on others OS next week)

I've also noticed something else that doesn't depend on this,
making multiple HTTP requests with the same endpoint may result in the same IP not being re-used
which i have never noticed happening with HTTP1.1 of "Requests" module,
maybe this is normal with HTTP/2 but i wanted to point it out

[CastagnaIT]

Ok i have tested also on Linux Mint and to me seem works correctly
also compared to the website i see same IP's and ports used in TCP KA

is it possible to implement somehow an interface to set options and ioctl in the socket to be able to manage TCP KA?

[CastagnaIT]

i have to open an Issue request?

[CastagnaIT]

i have encountered a error that happen every time and only after doing multiple requests:
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.COMPRESSION_ERROR, last_stream_id:2147483647, additional_data:485041434b202d20696c6c6567616c20696e6465>
https://paste.kodi.tv/iyubevuxey

i not sure what i can do

EDIT: not happen always but sometimes, now i have re-tried multiple times but not happen anymore, i have no idea...

[tomchristie]

i have to open an Issue request?

I'd prefer not since I don't think we've adequately figured out exactly what it is we'd want to do here.
At the moment we're probably just going to treat that as out of scope.
You'll need to monkey patch the backend if you do want to support that use-case.