Skip to content

Bound per-cycle binlog purge to avoid LOCK_index write stalls#239

Merged
alexole merged 4 commits into
masterfrom
cap-per-cycle-binlog-purge
Jul 6, 2026
Merged

Bound per-cycle binlog purge to avoid LOCK_index write stalls#239
alexole merged 4 commits into
masterfrom
cap-per-cycle-binlog-purge

Conversation

@vladislav-manafov-aiven

@vladislav-manafov-aiven vladislav-manafov-aiven commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Reducing binlog_retention_period on a write-heavy service made the next 60s purge tick collect the entire newly-eligible backlog and issue one PURGE BINARY LOGS over all of it. MySQL runs that under LOCK_index, which every committing transaction also needs, so a large backlog stalls all writes for the full purge duration (hours / >1 TB).

Cap each purge cycle by file count and total bytes so a large backlog drains as many small purges across successive ticks instead of one long stall. The cap trims the tail of the already-computed contiguous oldest-first prefix, so all existing safety semantics (stream safety, min_binlog_age_before_purge, replication state, no-GTID handling) are preserved. At least one file is always purged to guarantee progress.

Also emit myhoard.binlog.purged_per_cycle.{files,bytes} so the cap can be verified.

Reducing binlog_retention_period on a write-heavy service made the next
60s purge tick collect the entire newly-eligible backlog and issue one
PURGE BINARY LOGS over all of it. MySQL runs that under LOCK_index, which
every committing transaction also needs, so a large backlog stalls all
writes for the full purge duration (hours / >1 TB).

Cap each purge cycle by file count and total bytes so a large backlog
drains as many small purges across successive ticks instead of one long
stall. The cap trims the tail of the already-computed contiguous
oldest-first prefix, so all existing safety semantics (stream safety,
min_binlog_age_before_purge, replication state, no-GTID handling) are
preserved. At least one file is always purged to guarantee progress.

Also emit myhoard.binlog.purged_per_cycle.{files,bytes} so the cap can be
verified.
Comment thread myhoard/controller.py
Comment thread myhoard/controller.py
Comment thread myhoard/controller.py
@alexole
alexole merged commit 9216b99 into master Jul 6, 2026
5 checks passed
@alexole
alexole deleted the cap-per-cycle-binlog-purge branch July 6, 2026 14:24
@alexole

alexole commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Please create a follow up to reduce interval when throttled to 1 second or even less, otherwise the rate of binlogs generation might be higher then removing and there is a risk of filling up the disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants