fix(journal): repair a torn WAL tail instead of panicking the shard - #3976
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3976 +/- ##
============================================
- Coverage 85.00% 84.91% -0.10%
+ Complexity 1402 1401 -1
============================================
Files 1225 1225
Lines 180283 180306 +23
Branches 146587 146803 +216
============================================
- Hits 153248 153105 -143
- Misses 22993 23146 +153
- Partials 4042 4055 +13
🚀 New features to boost your workflow:
|
spetz
previously approved these changes
Sep 1, 2026
mmodzelewski
previously approved these changes
Sep 1, 2026
hubcio
requested changes
Sep 1, 2026
spetz
previously approved these changes
Sep 1, 2026
hubcio
requested changes
Sep 1, 2026
hubcio
left a comment
Contributor
There was a problem hiding this comment.
the body says this fixes debian 12 / al2023 (6.1) and ubuntu 24.04 (6.8). it doesn't on those kernels: recovery runs before the client listeners bind, and plain tcp/ws/http bind through IORING_OP_BIND/LISTEN (6.11) with the same no-pool panic, so the crash just moves from recover to listener startup. only tls/wss/quic setups with http off gain today. rewrite the body before squash, and drop the sync_all over sync_data line - fdatasync already flushes a size change.
spetz
previously approved these changes
Sep 1, 2026
spetz
force-pushed
the
torn_wal_on_kernel_version
branch
from
September 1, 2026 07:06
3019859 to
e9fa9ac
Compare
spetz
approved these changes
Sep 1, 2026
hubcio
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FileStorage::truncateis the boot-time repair for a torn metadata WAL tail.It used compio's
set_len, which submitsIORING_OP_FTRUNCATE. That opcodelanded in kernel 6.9. Below it the driver probes the opcode as unsupported and
falls back to
push_blocking, but shard proactors are built withthread_pool_limit(0), so the fallback panics withthe thread pool is needed but no worker thread is running. The panic fires inside dispatch, outsidecatch_unwind_io.Net effect: a crash that tore the last WAL append made the next boot kill the
shard instead of repairing it, and the repair is idempotent, so the node stayed
down across restarts.
Two conditions have to coincide, so this is not a routine path:
kernel < 6.9 IORING_OP_FTRUNCATE absent, compio falls back
AND
torn WAL tail crash mid-append, so boot calls truncate_or_fail
The affected range is 5.19 through 6.8, not everything below 6.9. Ring setup
already requires IORING_SETUP_COOP_TASKRUN and IORING_SETUP_TASKRUN_FLAG,
which need 5.19, so RHEL 9 (5.14) and stock Ubuntu 22.04 (5.15) never start the
server at all and were never exposed. What this actually broke is Debian 12 and
AL2023 (6.1), and Ubuntu 24.04 and 22.04-HWE (6.8). macOS aarch64 is exempt
because create_shard_executor keeps a blocking pool there by design.
The fix
Truncate synchronously through std::fs on the stored path, which needs
neither the opcode nor the blocking pool. This mirrors what segment recovery
already does in truncate_to. The sync_all moves inside truncate, so the
repair is durable on its own and the caller no longer pairs it with a separate
fsync. sync_all rather than sync_data because the file length is metadata,
and without it a power cut right after the repair re-presents the torn tail.
truncate is no longer async. An async fn that never awaits trips
clippy::unused_async, and the journal crate denies clippy::pedantic.
Blocking the shard thread costs nothing here: the sole caller is boot-time
repair, before the shard serves traffic.