Skip to content

fix(journal): repair a torn WAL tail instead of panicking the shard - #3976

Merged
spetz merged 9 commits into
masterfrom
torn_wal_on_kernel_version
Sep 1, 2026
Merged

fix(journal): repair a torn WAL tail instead of panicking the shard#3976
spetz merged 9 commits into
masterfrom
torn_wal_on_kernel_version

Conversation

@numinnex

@numinnex numinnex commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

FileStorage::truncate is the boot-time repair for a torn metadata WAL tail.
It used compio's set_len, which submits IORING_OP_FTRUNCATE. That opcode
landed in kernel 6.9. Below it the driver probes the opcode as unsupported and
falls back to push_blocking, but shard proactors are built with
thread_pool_limit(0), so the fallback panics with the thread pool is needed but no worker thread is running. The panic fires inside dispatch, outside
catch_unwind_io.

Net effect: a crash that tore the last WAL append made the next boot kill the
shard instead of repairing it, and the repair is idempotent, so the node stayed
down across restarts.

Two conditions have to coincide, so this is not a routine path:

kernel < 6.9 IORING_OP_FTRUNCATE absent, compio falls back
AND
torn WAL tail crash mid-append, so boot calls truncate_or_fail

The affected range is 5.19 through 6.8, not everything below 6.9. Ring setup
already requires IORING_SETUP_COOP_TASKRUN and IORING_SETUP_TASKRUN_FLAG,
which need 5.19, so RHEL 9 (5.14) and stock Ubuntu 22.04 (5.15) never start the
server at all and were never exposed. What this actually broke is Debian 12 and
AL2023 (6.1), and Ubuntu 24.04 and 22.04-HWE (6.8). macOS aarch64 is exempt
because create_shard_executor keeps a blocking pool there by design.

The fix

Truncate synchronously through std::fs on the stored path, which needs
neither the opcode nor the blocking pool. This mirrors what segment recovery
already does in truncate_to. The sync_all moves inside truncate, so the
repair is durable on its own and the caller no longer pairs it with a separate
fsync. sync_all rather than sync_data because the file length is metadata,
and without it a power cut right after the repair re-presents the torn tail.

truncate is no longer async. An async fn that never awaits trips
clippy::unused_async, and the journal crate denies clippy::pedantic.
Blocking the shard thread costs nothing here: the sole caller is boot-time
repair, before the shard serves traffic.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 27, 2026
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.82192% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.91%. Comparing base (328b289) to head (e9fa9ac).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/server/src/bootstrap.rs 53.33% 5 Missing and 2 partials ⚠️
core/journal/src/file_storage.rs 93.75% 0 Missing and 2 partials ⚠️
core/message_bus/src/replica/io.rs 0.00% 0 Missing and 2 partials ⚠️
core/journal/src/prepare_journal.rs 50.00% 0 Missing and 1 partial ⚠️
core/message_bus/src/client_listener/mod.rs 92.85% 0 Missing and 1 partial ⚠️
core/server/src/http.rs 75.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3976      +/-   ##
============================================
- Coverage     85.00%   84.91%   -0.10%     
+ Complexity     1402     1401       -1     
============================================
  Files          1225     1225              
  Lines        180283   180306      +23     
  Branches     146587   146803     +216     
============================================
- Hits         153248   153105     -143     
- Misses        22993    23146     +153     
- Partials       4042     4055      +13     
Components Coverage Δ
Rust Core 85.93% <80.82%> (+0.02%) ⬆️
Java SDK 67.28% <ø> (-0.02%) ⬇️
C# SDK 75.48% <ø> (+0.10%) ⬆️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 94.50% <ø> (-1.74%) ⬇️
Go SDK 69.35% <ø> (+0.03%) ⬆️
Files with missing lines Coverage Δ
core/message_bus/src/client_listener/tcp.rs 83.33% <100.00%> (ø)
core/message_bus/src/client_listener/ws.rs 84.21% <100.00%> (ø)
core/journal/src/prepare_journal.rs 92.30% <50.00%> (+0.06%) ⬆️
core/message_bus/src/client_listener/mod.rs 85.00% <92.85%> (+11.92%) ⬆️
core/server/src/http.rs 92.96% <75.00%> (ø)
core/journal/src/file_storage.rs 75.72% <93.75%> (+8.16%) ⬆️
core/message_bus/src/replica/io.rs 82.17% <0.00%> (ø)
core/server/src/bootstrap.rs 80.54% <53.33%> (+0.04%) ⬆️

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

spetz
spetz previously approved these changes Sep 1, 2026
mmodzelewski
mmodzelewski previously approved these changes Sep 1, 2026
Comment thread helm/charts/iggy/README.md.gotmpl Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 1, 2026
@spetz
spetz dismissed stale reviews from mmodzelewski and themself via c86dcc0 September 1, 2026 05:43
spetz
spetz previously approved these changes Sep 1, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the body says this fixes debian 12 / al2023 (6.1) and ubuntu 24.04 (6.8). it doesn't on those kernels: recovery runs before the client listeners bind, and plain tcp/ws/http bind through IORING_OP_BIND/LISTEN (6.11) with the same no-pool panic, so the crash just moves from recover to listener startup. only tls/wss/quic setups with http off gain today. rewrite the body before squash, and drop the sync_all over sync_data line - fdatasync already flushes a size change.

Comment thread core/message_bus/src/client_listener/mod.rs Outdated
Comment thread core/server_common/src/diagnostics.rs Outdated
Comment thread core/journal/src/file_storage.rs
Comment thread core/journal/src/file_storage.rs Outdated
Comment thread core/journal/src/file_storage.rs Outdated
Comment thread core/journal/src/file_storage.rs Outdated
Comment thread core/journal/src/file_storage.rs Outdated
spetz
spetz previously approved these changes Sep 1, 2026
@spetz
spetz force-pushed the torn_wal_on_kernel_version branch from 3019859 to e9fa9ac Compare September 1, 2026 07:06
@spetz
spetz merged commit 8c4986a into master Sep 1, 2026
98 checks passed
@spetz
spetz deleted the torn_wal_on_kernel_version branch September 1, 2026 07:25
@github-actions github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants