Skip to content

🐛 fix(storage): issue no receipt when a directory flush fails - #1999

Merged
gaborbernat merged 1 commit into
mainfrom
fix/no-receipt-after-sync-failure-1391
Aug 31, 2026
Merged

🐛 fix(storage): issue no receipt when a directory flush fails#1999
gaborbernat merged 1 commit into
mainfrom
fix/no-receipt-after-sync-failure-1391

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

A local-filesystem commit issued a DurabilityCapabilities::FILESYSTEM receipt whether or not the directory entry naming the blob ever reached disk. sync_parent swallowed the directory open behind if let Ok(...) and its sync_all behind let _ =, so publish returned success after the kernel refused the flush and commit_staged handed back a receipt. Replication and acknowledgement then weighed that receipt as proof of an entry a crash can still lose. #516 requires the parent-directory sync before any receipt, and the POSIX fsync contract says why.

The helper returns its failure now and every publication path propagates it, so a refused flush ends as an I/O error with no receipt. 🔒 Two cases needed more than plumbing. A fresh store has no digest fan-out, and create_dir_all leaves each level it creates as an unflushed entry in its own parent, so the store flushes those new levels from the leaf toward the first that already existed. A commit that finds matching bytes already resident flushes too: those bytes can come from a writer whose own sync failed, so their presence proves nothing about the entry naming them. That second rule is what lets a retry after a successful rename finish the durability step without touching the resident blob, the resumption path #1501 builds on.

flowchart LR
    Stage[Stage synced] --> Rename[Rename into digest path]
    Rename --> Flush{Directory flush}
    Flush -- ok --> Receipt[Placement receipt]
    Flush -- refused --> Error[I/O error, no receipt]
    classDef accent fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a;
    classDef warn fill:#ffe3a3,stroke:#d29200,color:#3a2c00;
    class Stage,Rename,Receipt accent;
    class Flush,Error warn;
Loading

Unix flushes the directory with fsync. Windows keeps the behavior it had, since it exposes no directory flush and NTFS orders the rename in the metadata log its recovery pass replays, which is what crates/peryx/src/operator/restore.rs already relies on. The issue also asked for a documented Win32 write-through path; I left that out rather than guess, because FlushFileBuffers rejects a handle opened with backup semantics.

Two behavior changes to watch. A store whose directories the process cannot open for reading now fails its commits instead of returning a receipt for a flush that never happened. A deduplicating write pays one extra directory fsync.

Closes #1391

The filesystem durability boundary discarded every parent-directory
failure: sync_parent ignored both the directory open and its sync_all,
so commit_staged went on to hand out a DurabilityCapabilities::FILESYSTEM
receipt for a rename that may never have reached disk. Replication and
acknowledgement code then weighed that receipt as proof of a directory
entry a crash could still lose.

The helper now returns the failure and every publication path propagates
it, so a refused flush produces an I/O error and no receipt. A fresh
digest fan-out flushes each directory it creates from the leaf toward the
first pre-existing level, since create_dir_all leaves each new level as an
unflushed entry in its own parent. A commit that finds matching bytes
already resident still flushes: those bytes may come from a writer whose
own sync failed, so their presence says nothing about the durability of
the entry naming them. That also lets a retry after a successful rename
finish the durability step without touching the resident blob.

Unix uses directory fsync. Windows exposes no directory flush, so it keeps
relying on the NTFS metadata log that orders the rename, matching the
operator restore path.
@gaborbernat gaborbernat added the bug Something isn't working label Aug 31, 2026
@codspeed-hq

codspeed-hq Bot commented Aug 31, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 29 untouched benchmarks
⏩ 133 skipped benchmarks1


Comparing fix/no-receipt-after-sync-failure-1391 (913ba53) with main (74eb4f9)

Open in CodSpeed

Footnotes

  1. 133 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat
gaborbernat merged commit 2d20723 into main Aug 31, 2026
27 checks passed
@gaborbernat
gaborbernat deleted the fix/no-receipt-after-sync-failure-1391 branch August 31, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Do not issue filesystem receipts after directory sync failure

1 participant