Skip to content

Issue 4855: fail closed on entry log flush failure - #4860

Open
yangxianjungree wants to merge 9 commits into
apache:branch-4.16from
yangxianjungree:issue-4855-branch-4.16
Open

Issue 4855: fail closed on entry log flush failure#4860
yangxianjungree wants to merge 9 commits into
apache:branch-4.16from
yangxianjungree:issue-4855-branch-4.16

Conversation

@yangxianjungree

@yangxianjungree yangxianjungree commented Aug 4, 2026

Copy link
Copy Markdown

Descriptions of the changes in this PR:

This PR makes entry-log write/flush failures fail closed instead of allowing a bookie to continue after the entry-log write position becomes uncertain. The issue was reproduced on the 4.16.7 line with DefaultEntryLogger + BufferedChannel + DbLedgerStorage, and this PR targets branch-4.16. This is the backport of #4863.

Motivation

A partial entry-log write/flush/force failure can leave the physical log file behind the logical BufferedChannel position. If the bookie continues accepting writes after that point, ledger storage may persist locations that point past the bytes actually written to the entry log, making entries unreadable after restart or recovery.

Changes

  • Introduce EntryLogWriteException as the fatal boundary for entry-log-level write failures.
  • Poison BufferedChannel after entry-log write, flush, or force-write failure so later writes cannot continue on an uncertain channel.
  • Harden entry-log header and ledgers-map writes with full-write checks and fatal failure propagation.
  • Propagate entry-log write failures through DefaultEntryLogger, EntryLogManager, SyncThread, BookieImpl, DbLedgerStorage, and SingleDirectoryDbLedgerStorage.
  • Make startup requestFlush() fail on fatal entry-log flush failure instead of racing with asynchronous shutdown.
  • Fail closed for InterleavedLedgerStorage and SortedLedgerStorage entry-log write failures while keeping NoWritableLedgerDirException behavior unchanged.
  • Harden the entryLogPerLedgerEnabled eviction path so appendLedgersMap() failure triggers fatal shutdown and the failed channel is not treated as a normal rotated log.
  • Clean up newly allocated log channels if allocation fails before publication.
  • Add unit coverage plus a real BookKeeper client e2e that verifies client-visible write failure and bookie shutdown after an entry-log flush failure.

Scope notes:

  • The DirectEntryLogger / direct I/O path is not changed in this PR.
  • This PR does not implement a two-phase header fsync / log-id publication protocol for new entry-log allocation. Allocation failure cleanup is included; stronger allocation lifecycle handling can be addressed separately.

Master Issue: #4855

Tests:

  • mvn -pl bookkeeper-server -am -Dtest=DbLedgerStorageEntryLogFlushFailureE2ETest -Dsurefire.failIfNoSpecifiedTests=false test
  • mvn -pl bookkeeper-server -am -Dtest=BufferedChannelTest,SyncThreadTest,BookieImplTest,DefaultEntryLogTest,DbLedgerStorageWriteCacheTest,DbLedgerStorageEntryLogFlushFailureE2ETest -Dsurefire.failIfNoSpecifiedTests=false test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant