Flush messages to disk in batches. #1388
Conversation
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228
src/rabbit_variable_queue.erl
Outdated
| case get(waiting_bump) of | ||
| true -> ok; | ||
| _ -> self() ! bump_reduce_memory_use, | ||
| put(waiting_bump, waiting) |
There was a problem hiding this comment.
Instead of waiting, I think the value here should be true otherwise the skip-case won't be hit on future calls that are waiting for the message. Also see #1393
There was a problem hiding this comment.
Right. Missed that.
If messages should be embedded to a queue index, there will be no credit flow limit, so message batches can be too big and block the queue process. Limiting the batch size allows consumer to make progress while publishers are blocked by the paging-out process. [#151614048]
gerhard
force-pushed
the
rabbitmq-server-batch-betas
branch
from
786df6a
to
c574fc5
Compare
|
Here are some (fairly basic) benchmark results in a constraint environment, both with and without consumer rate limiting. When consumer throughput is really constrained, we see that publishers are throttled more aggressively but consumers are never completely blocked — stdev for consumer latency is significantly lower on this branch. On other workloads the difference is fairly small but surprisingly this branch demonstrates a slight higher overall throughput (which I wasn't expecting) and lower consumer latency (which was easier to foresee). |
|
Great improvement! Page out behaviour before this change: Page out behaviour after this change: This is what used to happen when filling & draining a queue (no paging to disk): This is what happens now when filling & draining a queue (no paging to disk): |
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
Alternative for #1388 that does not use process dictionary. Requires rabbitmq/rabbitmq-common#228 Fix waiting_bump values
If messages should be embedded to a queue index, there will
be no credit flow limit, so message batches can be too big
and block the queue process.
Limiting the batch size allows consumer to make progress while
publishers are blocked by the paging-out process.
[#151614048]