RDY handling on new connections #38
Conversation
|
RFR @mreiferson |
|
I'm not sure what you were experiencing but I dont think this is necessary. The first (existing) clause already handles all non-backoff cases (meaning every new connection while not in backoff will attempt to send I'm assuming then that the worker in question was already in backoff... in which case it is correct for subsequent new connections to stay at What am I missing? |
|
Also, if a connection is going to get an initial non-0 RDY count, we had previously decided to always use 1 to simplify. I don't see any reason to change this. |
|
Hmmm you are right. It might be the case that at the time of subsequent connect finishing the first connection had acquired the full max_in_flight. |
|
If there is indeed some edge case here one idea is for initial RDY 0'd connections to enter a poll loop and try to set RDY 1 until it either is non-0 or successful.... (every 5 secs or something?) |
|
(ie it sounds possible for an atypical redistribute case where other conns "stole" all the max in flight to never get a RDY 1) |
|
see mreiferson/pynsq@4c99a96 for one possible solution |
|
pushed another commit addressing what we discussed (somewhat unrelated to this specific issue but we can just update subject/description) @jehiah |
|
I like the last commit. It's a good clean way to spread RDY to other connections, and should sove this situation. |
|
@mreiferson I'm good on this. any changes you want before I squash? |
|
im good |
* occasionally choose a random connection for RDY when num conns > max_in_flight * loop on a new connection when we're at RDY 0 and we cant send RDY non-0
RDY handling on new connections
When a new connection is added (beyond the first one) it never get's an initial RDY count (even though max_in_flight might be high enough). As a result subsequent connections may stay at RDY 0 (until backoff and recover happens at which time expected max_in_flight is distributed across connections correctly).