Skip to content

v1.0.1 — bound the archive's write-ahead log

Latest

Choose a tag to compare

@AndrewCTF AndrewCTF released this 16 Jul 13:06
175297b

Patch release. If you are running v1.0.0, upgrade — it ships an archive that
grows without bound.

The fix

v1.0.0's replay bar asked the backend for archive coverage every 5 seconds, and
each request scanned the whole archive. That scan takes longer than the gap
between polls, so a read transaction was always open — and SQLite can only
checkpoint its write-ahead log past the oldest live reader. The recorder never
stops writing, so the log grew forever.

It compounds, which is why it runs away rather than settling: every page read has
to search the log for that page's newest version, so a bigger log makes the scan
slower, which holds the reader open longer, which grows the log faster.

On the development box that produced a 49.6 GB write-ahead log against 15.2 GB
of actual data
— 71 GB of disk for 15 GB of archive. A checkpoint reclaimed
48.6 GB of it, so 98% was redundant page versions that could never be written
back. Left recording on a smaller disk, it fills it.

Coverage is now cached behind a lock, so the scan runs at most once every two
minutes and never twice at once, and the log is bounded by journal_size_limit
so it is truncated back whatever the read pattern does.

Replaying the exact workload that caused it, against a real 16 GB archive:

v1.0.0 v1.0.1
write-ahead log grew to 49.6 GB 0 MB, flat
memory +4-5 GB/min, peaked at 83 GB flat, ~660 MB
coverage request 73 s, every call 14.8 s once, then cached

Upgrading

Nothing to migrate. Pull and restart; the oversized log from a v1.0.0 run is
checkpointed back into the database on the next clean start. If you want the
disk back immediately, stop the API and run:

sqlite3 data/history.db "PRAGMA wal_checkpoint(TRUNCATE);"

That took 37.7 s against a 49.6 GB log here. It is lossless — the log's contents
are written into the database, which is what a clean shutdown does anyway.

Everything else

Unchanged from v1.0.0. 1,721 backend tests passing. AGPL-3.0.