Ladybug version
v0.18.3
What operating system are you using?
Windows 11
What happened?
Authorship disclosure: this report was written by Claude, Anthropic's AI, working as the developer on the project that found these behaviors. The test campaigns and evidence are the joint work of the AI and the project's owner, who reviewed this text and filed it.
Summary
If the WAL contains any single record larger than roughly 2 KB and the process is hard-terminated (crash, kill, power loss) before the next checkpoint, the WAL is unreplayable on next open: recovery fails and the engine refuses to open the database file entirely — including all data folded into the main file long before the death. A CHECKPOINT fully disarms the condition; the exposure window is from each large write until the next fold. The behavior is inherited, not fork-introduced: we reproduce it deterministically on every version we have tested, from 0.15.3 through 0.18.3 (the current release, retested on its publication day) — including the 0.18.x stored WAL record lengths, which we hoped would resist this failure class but which did not change the outcome.
We consider two distinct defects here: (1) recovery converts a torn journal tail — the normal case a journal exists to survive — into total loss, where the correct worst outcome is "discard the unreplayable tail, open with everything up to the last checkpoint"; and (2) on one field corpse, replay did not even fail gracefully but hit an internal assertion (UNREACHABLE_CODE in wal_record.cpp) during startup recovery. Recovery code asserting on corrupt input is a bug in its own right, independent of (1).
This issue is the severity multiplier for the fold-race crash reported separately [cross-reference once filed]: that crash strikes mid-fold, which is exactly when large records are guaranteed to be sitting unfolded in the WAL, so a race death usually costs the whole file rather than a tail.
Environment
- ladybug 0.18.3, ladybug 0.18.1, and real_ladybug 0.15.3 (Python wheels); Python 3.13.14; reproduced on both Windows 11 and Linux
- Large records arise naturally in our workload: FLOAT vector properties of 512–1280 dimensions written per item, and a 60-column node table whose DDL is itself a trigger record
Reproduction (deterministic, single process, no concurrency required)
- Open a database; write a handful of records including at least one above ~2 KB (a 512-d FLOAT vector property suffices; so does creating a wide table).
- Hard-exit the process without closing (abort / kill; on Windows,
TerminateProcess; a crash serves equally).
- Reopen: recovery fails and the database refuses to open. On 0.15.x the error is "Corrupted wal file. Read out invalid WAL record type."; on recent versions, a checksum verification failure.
Most recent confirmation (July 21, 0.18.3): one 512-dimension embedding in the WAL, hard kill, reopen fails with "Checksum verification failed, the WAL file is corrupted"; the paired controls both recover (60 small records + hard kill → opens with all 60; embedding + CHECKPOINT + hard kill → opens clean).
Repeat step 1 with only small records (<~2 KB) and the same hard exit: recovery succeeds and the database opens with all committed data. The record size is the discriminating variable; we have run this pair many times on both engines with fully consistent results.
Confirmed incidentally at scale: our concurrency test campaign's watchdog hard-kills hung trials, and every such kill with large records in the journal produced an unopenable database — 7 corpses across one overnight run alone, on both 0.15.3 and 0.18.1 legs, plus 4 more on 0.18.3 in a July 21 session (each produced by the separately-reported checkpoint-race crash landing mid-fold with large records in the journal).
Field evidence
- Preserved corpse pair from production use (main file 157.86 MB + WAL 13.79 MB): the WAL's final entry timestamps to the second an overnight batch completed; the next startup's replay failed. This corpse is the one that asserts
UNREACHABLE_CODE in wal_record.cpp during recovery.
- A second production corpse (from the fold-race crash reported separately): main file 27.6 MB, a 0-byte
.shadow, and an orphaned 16 MB .wal.checkpoint from the interrupted fold.
- Reproduction scripts for both platforms are available, along with corpse files produced by the synthetic reproduction (no real data). The production corpses are preserved but contain private data and stay local; we can run instrumented builds or diagnostics against them on our side and share the output.
Expected behavior
A journal's purpose is surviving dirty deaths; a torn or unreplayable tail after a hard termination is its design-basis event, not an edge case. Expected: replay everything that verifies, stop at the first record that does not, open the database in the state of the last checkpoint plus whatever tail was salvageable, and report what was discarded. Losing writes made after the last fold would be understood; refusing the file — and in one case asserting — is the defect.
Notes that may help
- The threshold is consistent around ~2 KB per record in our measurements, suggesting a page- or buffer-boundary interaction in how large records are journaled or replayed rather than a general corruption.
- 0.18.0's length-prefixed WAL records looked structurally relevant to this failure class, and we specifically retested after upgrading: the reproduction still holds on 0.18.1 and 0.18.3, so the length prefix alone does not close it.
- Happy to run instrumented builds or candidate patches against the minimal reproduction on either platform, and against the preserved production corpses locally, reporting results back.
Are there known steps to reproduce?
No response
Ladybug version
v0.18.3
What operating system are you using?
Windows 11
What happened?
Authorship disclosure: this report was written by Claude, Anthropic's AI, working as the developer on the project that found these behaviors. The test campaigns and evidence are the joint work of the AI and the project's owner, who reviewed this text and filed it.
Summary
If the WAL contains any single record larger than roughly 2 KB and the process is hard-terminated (crash, kill, power loss) before the next checkpoint, the WAL is unreplayable on next open: recovery fails and the engine refuses to open the database file entirely — including all data folded into the main file long before the death. A
CHECKPOINTfully disarms the condition; the exposure window is from each large write until the next fold. The behavior is inherited, not fork-introduced: we reproduce it deterministically on every version we have tested, from 0.15.3 through 0.18.3 (the current release, retested on its publication day) — including the 0.18.x stored WAL record lengths, which we hoped would resist this failure class but which did not change the outcome.We consider two distinct defects here: (1) recovery converts a torn journal tail — the normal case a journal exists to survive — into total loss, where the correct worst outcome is "discard the unreplayable tail, open with everything up to the last checkpoint"; and (2) on one field corpse, replay did not even fail gracefully but hit an internal assertion (
UNREACHABLE_CODEinwal_record.cpp) during startup recovery. Recovery code asserting on corrupt input is a bug in its own right, independent of (1).This issue is the severity multiplier for the fold-race crash reported separately [cross-reference once filed]: that crash strikes mid-fold, which is exactly when large records are guaranteed to be sitting unfolded in the WAL, so a race death usually costs the whole file rather than a tail.
Environment
Reproduction (deterministic, single process, no concurrency required)
TerminateProcess; a crash serves equally).Most recent confirmation (July 21, 0.18.3): one 512-dimension embedding in the WAL, hard kill, reopen fails with "Checksum verification failed, the WAL file is corrupted"; the paired controls both recover (60 small records + hard kill → opens with all 60; embedding + CHECKPOINT + hard kill → opens clean).
Repeat step 1 with only small records (<~2 KB) and the same hard exit: recovery succeeds and the database opens with all committed data. The record size is the discriminating variable; we have run this pair many times on both engines with fully consistent results.
Confirmed incidentally at scale: our concurrency test campaign's watchdog hard-kills hung trials, and every such kill with large records in the journal produced an unopenable database — 7 corpses across one overnight run alone, on both 0.15.3 and 0.18.1 legs, plus 4 more on 0.18.3 in a July 21 session (each produced by the separately-reported checkpoint-race crash landing mid-fold with large records in the journal).
Field evidence
UNREACHABLE_CODEinwal_record.cppduring recovery..shadow, and an orphaned 16 MB.wal.checkpointfrom the interrupted fold.Expected behavior
A journal's purpose is surviving dirty deaths; a torn or unreplayable tail after a hard termination is its design-basis event, not an edge case. Expected: replay everything that verifies, stop at the first record that does not, open the database in the state of the last checkpoint plus whatever tail was salvageable, and report what was discarded. Losing writes made after the last fold would be understood; refusing the file — and in one case asserting — is the defect.
Notes that may help
Are there known steps to reproduce?
No response