fix(phlower): reclaim freed pages so DROP actually shrinks the file#27
Merged
Conversation
…file prod-us PVC filled at 49 GB on 2026-05-04 04:47 UTC during the hourly purge. Emergency purge dropped invocations_legacy (26 GB), but the file didn't shrink — SQLite's DROP TABLE marks pages as free in the file's internal freelist without returning them to the OS, and the DROP transaction's WAL temporarily ballooned to fill what little headroom was left. Result: disk full, INSERTs failing with sqlite3.OperationalError, ~14h of records silently dropped through the buffer-cap safety net. Three changes: - PRAGMA auto_vacuum=INCREMENTAL set in _connect() before any tables are created. Tracks freed pages in a separate list so they can be reclaimed by PRAGMA incremental_vacuum. Must be set on a fresh DB — on existing DBs it requires a full VACUUM to apply, which is exactly why this PR pairs with a PVC drop / fresh-start deploy. - _reclaim_free_pages() called after DROP TABLE in purge_old_partitions(). Uses PRAGMA incremental_vacuum to truncate the freed pages off the file. The PRAGMA emits one result row per freed page so the cursor MUST be drained with fetchall() — without that, only one page gets reclaimed and the file barely shrinks. - SQLITE_DISK_USAGE_PCT_CAP default lowered from 85 → 75, giving more headroom for WAL spikes during a DROP. With incremental_vacuum keeping the file lean this should rarely fire, but the cushion matters when it does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
prod-us PVC filled at 49 GB on May 4 ~04:47 UTC during the hourly purge. Emergency purge dropped
invocations_legacy(26 GB), but the file didn't shrink — SQLite'sDROP TABLEmarks pages as free in an internal freelist without returning them to the OS, and the DROP transaction's WAL temporarily ballooned to fill what little headroom was left. Result: disk full,INSERTs failing withsqlite3.OperationalError, ~14h of records silently dropped through the buffer-cap safety net.What
PRAGMA auto_vacuum=INCREMENTALset in_connect()before any tables are created. Tracks freed pages in a separate list so they can be reclaimed viaPRAGMA incremental_vacuum. SQLite requires this PRAGMA on a fresh DB — on existing DBs it's a no-op without a full VACUUM, which is exactly why this PR pairs with a PVC drop and fresh-start deploy._reclaim_free_pages()called after DROP TABLE inpurge_old_partitions. UsesPRAGMA incremental_vacuumto truncate freed pages off the file. The PRAGMA emits one result row per freed page so the cursor MUST be drained withfetchall()— without that, only one page gets reclaimed.SQLITE_DISK_USAGE_PCT_CAPdefault lowered from 85 → 75, giving headroom for WAL spikes during DROP. With incremental_vacuum keeping the file lean this should rarely fire.Verified locally
20 MB of records → DROP TABLE → file truncates to 50 KB. Without
fetchall()the same test only reclaimed 1 page out of 5188.Deploy plan
auto_vacuum=INCREMENTALat first connectTest plan