-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
QQ: Revise checkpointing logic to take more frequent checkpoints for large message workloads #13587
Conversation
bb1988e
to
e0fe933
Compare
e0fe933
to
d2a294e
Compare
Testing: Try various workloads with both small and large messages and monitor the disk to ensure that there is no undue segment buildup that is retained when workload is paused. Also ensure segments are cleared up reasonably after queue is emptied / purged. For large message workloads this PR works best with a lower |
To take more frequent checkpoints for large message workload Lower the min_checkpoint_interval substantially to allow quorum queues better control over when checkpoints are taken. Track bytes enqueued in the aux state and suggest a checkpoint after every 64MB enqueued (this value is scaled according to backlog just like the indexes condition). This should help with more timely checkpointing when very large messages is used. Try evaluating byte size independently of time window also increase max size
8e4a3aa
to
6695282
Compare
QQ: Revise checkpointing logic to take more frequent checkpoints for large message workloads (backport #13587)
Lower the
min_checkpoint_interval
substantially to allow quorum queues better control over when checkpoints are taken.Tracking message bytes written to the log and use this to request a checkpoint every 64MB if no other checkpoint condition was met. This ensures that queues where the messages are very large (1MB+) are checkpointed based on their data ingress rather than indexes.