Due to emergent behavior between three different systems:
- Re-establishing the stream on error.
- Pausing the stream when over load.
- Factoring request queue size in load.
The subscriber can deadlock itself if it tries to re-establish a stream while the stream is paused due to request pressure, for example:
===== Subscriber Monitor =====
Messages processed: 1659
Rate: 0.82 Messages/second
Run time: 0:33:35.001386
Load: 1.67
p99 ack: 299 seconds
Leased Messages: 0
Executor queue size: 0
Callback queue size: 0
Request queue size: 167
The load is 1.67 because there are 167 requests which means the stream is paused. Because the stream is paused when the consumer tries to restart the stream the RPC is never started because to start the RPC the iterator must at least be started and the request iterator is blocked on the load going down. It won't go down because it can't send requests until the RPC is started.
Due to emergent behavior between three different systems:
The subscriber can deadlock itself if it tries to re-establish a stream while the stream is paused due to request pressure, for example:
The load is
1.67because there are167requests which means the stream is paused. Because the stream is paused when the consumer tries to restart the stream the RPC is never started because to start the RPC the iterator must at least be started and the request iterator is blocked on the load going down. It won't go down because it can't send requests until the RPC is started.