v0.12.3 – TCP keepalive on the inference port (router 0.6.3)

Router 0.6.3, agent unchanged (0.16.0). TCP keepalive on the inference port.

• What went wrong: a non-streaming /api/chat request ran 6.6 minutes. During that time not a single byte flowed between router and client, the connection state on the way to the client expired silently, and the finished answer never arrived. The client waited until its own timeout; the router counted the request as ok.
• Now: the inference port sets TCP keepalive (first probe after 60 s of silence, then every 15 s, dead after 4 unanswered probes). The probes keep connection state fresh along the way, and a connection that is dead anyway now shows up as an error instead of a silent loss.
• New selftest selftest_keepalive.py in CI checks that accepted connections inherit the options from the listening socket.
• Clients that can stream should still do so for long requests: bytes then flow while the model thinks.

Deutsch

Router 0.6.3. TCP-Keepalive auf dem Inferenz-Port (erste Probe nach 60 s, dann alle 15 s, nach 4 ohne Antwort tot). Eine Anfrage ohne Stream lief 6,6 min ohne ein Byte, der Verbindungszustand auf dem Weg verfiel still, die fertige Antwort kam nie an, und der Router zählte sie als ok. Die Proben halten den Zustand frisch und lassen tote Verbindungen als Fehler auffallen.