Skip to content

Issue 4855: fail closed on entry log flush failure - #4863

Open
yangxianjungree wants to merge 9 commits into
apache:masterfrom
yangxianjungree:issue-4855-master
Open

Issue 4855: fail closed on entry log flush failure#4863
yangxianjungree wants to merge 9 commits into
apache:masterfrom
yangxianjungree:issue-4855-master

Conversation

@yangxianjungree

@yangxianjungree yangxianjungree commented Aug 5, 2026

Copy link
Copy Markdown

Descriptions of the changes in this PR:

This PR makes entry-log write/flush failures fail closed instead of allowing a bookie to continue after the entry-log write position becomes uncertain. This is the master-first fix for #4855. A branch-4.16 backport is tracked separately in #4860.

Motivation

A partial entry-log write/flush/force failure can leave the physical log file behind the logical BufferedChannel position. If the bookie continues accepting writes after that point, ledger storage may persist locations that point past the bytes actually written to the entry log, making entries unreadable after restart or recovery.

Changes

  • Introduce EntryLogWriteException as the fatal boundary for entry-log-level write failures.
  • Poison BufferedChannel after entry-log write, flush, or force-write failure so later writes cannot continue on an uncertain channel.
  • Harden entry-log header and ledgers-map writes with full-write checks and fatal failure propagation.
  • Propagate entry-log write failures through DefaultEntryLogger, EntryLogManager, SyncThread, BookieImpl, DbLedgerStorage, and SingleDirectoryDbLedgerStorage.
  • Make startup requestFlush() fail on fatal entry-log flush failure instead of racing with asynchronous shutdown.
  • Fail closed for InterleavedLedgerStorage and SortedLedgerStorage entry-log write failures while keeping NoWritableLedgerDirException behavior unchanged.
  • Harden the entryLogPerLedgerEnabled eviction path so appendLedgersMap() failure triggers fatal shutdown and the failed channel is not treated as a normal rotated log.
  • Clean up newly allocated log channels if allocation fails before publication.
  • Make entry-log shutdown cleanup idempotent to avoid noisy repeated-shutdown failures after fatal entry-log errors.
  • Add unit coverage plus a real BookKeeper client e2e that verifies client-visible write failure and bookie shutdown after an entry-log flush failure.

Scope notes:

  • The DirectEntryLogger / direct I/O path is not changed in this PR.
  • This PR does not implement a two-phase header fsync / log-id publication protocol for new entry-log allocation. Allocation failure cleanup is included; stronger allocation lifecycle handling can be addressed separately.

Fixes #4855

Tests:

  • Red reproduction on apache/master with the BufferedChannelTest regression patch only: 5 failures.
  • mvn -pl bookkeeper-server -am -Dnative.io.pure.rust=true -Dtest=BufferedChannelTest -Dsurefire.failIfNoSpecifiedTests=false test
  • mvn -pl bookkeeper-server -am -Dnative.io.pure.rust=true -DtestRetryCount=0 -Dtest=BufferedChannelTest,SyncThreadTest,BookieImplTest,DefaultEntryLogTest,DbLedgerStorageTest,DbLedgerStorageWriteCacheTest,DbLedgerStorageEntryLogFlushFailureE2ETest -Dsurefire.failIfNoSpecifiedTests=false test
  • The targeted server suite above passed 3 times in a row; logs were checked for dump files, leak reports, unexpected executor errors, and refcount errors.

@yangxianjungree yangxianjungree changed the title fix: fail closed on entry log flush failure Issue 4855: fail closed on entry log flush failure Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DefaultEntryLogger can publish stale entry locations after BufferedChannel partial flush failure

1 participant