clients disconnected with the new keepalive algorithm #3286

Activity

1. github-actions

   added

   Status: AvailableNo one has claimed responsibility for resolving this issue.No one has claimed responsibility for resolving this issue.

   on Jun 10, 2025
2. jesferman commented on Jun 10, 2025

   

   jesferman

   on Jun 10, 2025

   Last edited by jesferman

   ContributorAuthor

   More actions

   I don't have time right now to debug the algorithm but in case it helps to the original developer here is the ChatGPT analysis of the possible cause:

   Here’s what’s happening under the hood:

   1. Ring-buffer size and indexing

   When you set

   max_keepalive 60

   the broker allocates a ring of size

   $$\text{keepalive\_list\_max} = \frac{60 \times 3}{2} + 1 = 91$$

   slots.
   Each client with keepalive = 60 s is bound into slot

   $$(\text{last\_msg\_in} + 1.5 \times 60)\bmod 91 = (\text{last\_msg\_in} + 90)\bmod 91$$

   which is exactly where the broker will look when it believes the client’s 1.5× keepalive window has expired.

   2. The timing mismatch

   • At broker startup (or config reload), last_keepalive_check is initialized to the current time, say T₀.
   • When a new client connects at time T₁ > T₀, its last_msg_in is (correctly) set to T₁, and it’s hashed into slot
     $(T₁ + 90)\bmod 91$.
   • But the very next time keepalive__check() runs (which iterates every second over all seconds from last_keepalive_check up to now), it will scan the interval [T₀ … T₁) — which, by modular arithmetic, can include exactly the same slot index you just added the client to. As soon as the loop variable i hits a value such that

   $$i \bmod 91 = (T₁ + 90)\bmod 91$$

   the broker thinks “Aha! this client’s slot has fired,” and immediately disconnects it with a KEEPALIVE timeout.

   3. Why you didn’t see it before

   • With the default max_keepalive of 65535, your ring is enormous (98 303 slots), so $(T_1 + 1.5×)\bmod\mathrm{ring_size}$ never collides with any of the few seconds that get scanned on connection.
   • As soon as you shrink the ring down to something on the order of your keepalive interval (like 91), that collision window appears, and clients get nuked almost immediately or after a “random” short time.

   4. In summary

   The back-ported O(1) implementation never updates last_keepalive_check when you add a new client, so your newly-connected client can end up in the “to be expired” slot right away if the small ring size wraps around into the current scan window. That is why setting max_keepalive to a low value suddenly causes perfectly well-behaved clients to get dropped with “exceeded timeout” messages.

   ---

   Workarounds / fixes:

   • Patch the new code so that:

     1. last_keepalive_check is bumped to db.now_s whenever you keepalive__add() a client, or
     2. the scanning loop skips the very first second after an add, or
     3. the ring buffer is always at least keepalive*3/2 + keepalive_scan_window in size (i.e. leave extra slack).
3. ralight

   added a commit that references this issue on Jul 11, 2025

   Fix clients being incorrectly expired.

   ...

   This issue will close once commit da34d3b is merged into the 'master' branch.

   da34d3b
4. ralight

   closed this as completedin da34d3bon Jul 11, 2025
5. github-actions

   locked as resolved and limited conversation to collaborators on Oct 12, 2025