Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read

[kaminski-dev]

In case others have seen it: I ran into transaction pending-ack replay loop. On one broker, a pulsar-transaction-executor-N-1 thread was stuck forever inside MLPendingAckStore$PendingAckReplay.run(), sitting on Thread.sleep(1) (MLPendingAckStore.java:431), and it never recovered on its own.

Versions: I confirmed this on 4.0.10. The same replay code is identical on branch-4.1 (4.1.x) and on branch-4.2 (4.2.x): the Thread.sleep(1) replay loop, the single-thread executors that are shared by hash, and the consumer that waits on pendingAckHandleFuture(). All line numbers below are from the 4.0.10 tag (pulsar-broker and managed-ledger).

My observation: These replay executors are single-threaded, and many subscriptions share the same thread by hashing. When transactions are enabled, every persistent subscription builds a PendingAckHandle — even consumers that never use transactions — and PersistentSubscription.addConsumerInternal waits for pendingAckHandleFuture() before the consumer can be added. So, if one replay is stuck, every other subscription on that same thread cannot create consumers. It can even turn into a full broker restart if the health-check subscription happens to land on the stuck thread.

How it gets stuck: the replay loop (MLPendingAckStore.java:406-439) keeps running until a read either succeeds or fails. A read failure sets isReadable = false (:470), and fillQueue() (:474-483) only starts a new read when outstandingReadsRequests == 0. So, if a read is sent but its callback never comes back:

• outstandingReadsRequests stays at 1, so no new read is ever started;
• the entry queue stays empty, so entryQueue.poll() returns null;
• isReadable stays true, so the loop never exits.

The result is an endless 1ms sleep loop. Normally a watchdog would rescue this: ManagedLedgerImpl.checkReadTimeout() (ManagedLedgerImpl.java:4531) calls lastReadCallback.readFailed(TimeoutException) after a timeout. But that watchdog is turned off when managedLedgerReadEntryTimeoutSeconds=0, so nothing ever completes the read.

Possible trigger: the read gets lost because the ledger it is reading is deleted underneath it — and this does not need a rare bookie garbage-collection race. New ledgers are created with timestamp=0 (ManagedLedgerImpl.java:618 on init, :1640 on rollover; the real timestamp is only written when the ledger is closed, :1846). And hasLedgerRetentionExpired(retentionTimeMs, 0) (:2770-2772) is always true for these system ledgers, which run with retention 0. So, an empty or just-rolled-over pending-ack ledger becomes eligible for deletion right away in internalTrimLedgers (:2967). In other words, the normal create-and-delete of empty ledgers is enough to race the replay read and make its callback disappear.

Related issues I found:

• The transaction thread continues consume CPU without any load #14852 (closed) fixed only the CursorAlreadyClosedException case — where the read failure callback does fire — not the "callback never comes back" case.
• [Bug] managedLedgerMaxReadsInFlightSizeInMB limit doesn't work properly if managedLedgerReadEntryTimeoutSeconds isn't set #23506 confirms that without a read timeout, managed-ledger reads can hang forever.
• [fix][txn] Fix deadlock when loading transaction buffer snapshot #24401 fixed the transaction buffer replay by giving it its own executor (already in 4.0.10), but the pending-ack replay (MLPendingAckStore.PendingAckReplay) still has no such isolation.

A couple of directions that might help: giving PendingAckReplay its own time limit, independent of managedLedgerReadEntryTimeoutSeconds, so it can never loop forever on Thread.sleep(1) (:431); and moving pending-ack replay onto its own executor, the same way #24401 did for the transaction buffer, so one stuck replay can't block unrelated subscriptions.

I have a small, repeatable local reproduction (one replay thread, small ledgers, read timeout set to 0) that leaves the executor stuck while the broker otherwise keeps running. Happy to share the reproduction and thread dumps.

Has anyone seen this behavior? Does this analysis seem plausible? Could this explain the broker pod restarts, and is there a recommended configuration to avoid it besides setting managedLedgerReadEntryTimeoutSeconds to a non-zero value?

[lhotari]

Thanks for the unusually careful write-up — the stack frame, the line numbers and the co-hashing
analysis are all accurate, and this is a real bug. I dug into it and I think the conclusion is:
you found a genuine defect, the impact is wider than you describe, but the mechanism is different
from the one you propose — and, importantly, the workaround you're asking about does not work.

The short answer to your question

managedLedgerReadEntryTimeoutSeconds will not get you out of this, so please don't rely on it.

checkReadTimeout() delivers createManagedLedgerException(BKException.Code.TimeoutException).
TimeoutException is not in isBkErrorNotRecoverable(...) (that switch only covers
NoSuchLedgerExists* and NoSuchEntry), so it returns a plain ManagedLedgerException
(ManagedLedgerImpl.java createManagedLedgerException(int)). That reaches
MLPendingAckStore.FillEntryQueueCallback.readEntriesFailed, and — see below — a plain
ManagedLedgerException matches none of the conditions that stop the replay. It just decrements
outstandingReadsRequests and lets the loop issue the read again.

It's still reasonable defence-in-depth, but framing it as the mitigation would be misleading.

The thing I think you'll find most interesting

TopicTransactionBuffer has the same recovery loop with the same FillEntryQueueCallback class
name — and it terminates, because its fillQueue() has an else branch that MLPendingAckStore
lacks:

// TopicTransactionBuffer.FillEntryQueueCallback.fillQueue()
if (cursor.hasMoreEntries()) {
    outstandingReadsRequests.incrementAndGet();
    cursor.asyncReadEntries(...);
} else {
    if (entryQueue.size() == 0) {
        isReadable = false;      // <-- loop exits
    }
}

// MLPendingAckStore.FillEntryQueueCallback.fillQueue()  -- v4.0.10 :474-483
if (cursor.hasMoreEntries()) {
    outstandingReadsRequests.incrementAndGet();
    readAsync(NUMBER_OF_PER_READ_ENTRY, this);
}
// no else branch
return isReadable;

That escape was added by #13739 (7dee63ed707, merged 2022-01-18). Modification #3 of its commit
message reads:

if cursor.hasMoreEntries() == false && entryQueue.size() == 0, return false and stop recovering.

• If the cursor is cleared when transaction is recovering, there will no entries can be read, but
  currentLoadPosition < lastConfirmedEntry.

So the project diagnosed this exact stall in the sibling class four years ago, fixed it there, and
never ported it to MLPendingAckStore. That's both the strongest evidence your report is real and
the natural shape of the fix.

Why I think the mechanism is different

The loop guard and the read gate measure different things:

• lastConfirmedEntry is a construction-time snapshot of the ML's LAC (it's final), and
  currentLoadPosition is seeded from the cursor's mark-delete position;
• but fillQueue()'s gate is cursor.hasMoreEntries(), driven by the cursor's read position.

So "there is still work to do" and "there is still something to read" can disagree permanently, with
outstandingReadsRequests == 0 — no lost callback needed anywhere. That state issues no read and
logs nothing, which matches a silent hang better than a dropped callback does.

Two smaller corrections, both in your favour in the sense that the bug is worse than stated:

• "A read failure sets isReadable = false (:470)" — v4.0.10 :470 is the field declaration. The
  assignment is :506, and it's conditional. Parenthesised, the guard is
  (autoSkipNonRecoverableData && NonRecoverableLedgerException) || ManagedLedgerFencedException || CursorAlreadyClosedException. With the default autoSkipNonRecoverableData=false, a definitive
  LedgerNotExistException leaves isReadable == true — so the loop re-issues the same doomed read
  ~1000×/s forever, one ERROR line per iteration. Your deleted-ledger scenario lands here.
• The trim race as described doesn't hold: internalTrimLedgers only iterates
  ledgers.headMap(slowestReaderLedgerId, false), bounded by the slowest durable cursor's
  mark-delete position, and empty ledgers are removed inline in ledgerClosed() before the trimmer
  ever sees them. Also worth knowing: BookKeeper's own readEntryTimeout is on by default (30s, from
  bookkeeperClientTimeoutInSeconds), so a silent bookie-side loss is hard to achieve regardless of
  the ML-level setting.

Everything else you wrote holds: the co-hashed single-thread executors, addConsumerInternal waiting
on pendingAckHandleFuture() with no timeout, and a PendingAckHandle for every persistent
subscription. The shape is identical on 4.0/4.1/4.2/master and back through 3.x to 2.10.0.

What would settle it

The thread dump can't distinguish the cases — they all park on the same Thread.sleep(1) frame. If
you still have the broker (or can reproduce):

1. From a heap dump: outstandingReadsRequests, isReadable, and the cursor's readPosition
   vs currentLoadPosition vs lastConfirmedEntry. 1 means a genuinely lost callback; 0 means
   one of the two states above.
2. Were "MLPendingAckStore of topic [...] stat reply fail!" ERROR lines repeating? A high repeat
   count points at the failing-read case; total silence points at the no-more-entries case.
3. Had that broker restarted shortly before the hang?
4. Did message TTL, skip, clear-backlog, or a topic unload coincident with a commit/abort touch
   that subscription?
5. Is transactionPendingAckBatchedWriteEnabled on?
6. Did it recur after unloading the topic?

And yes please — your reproduction would be very welcome. My analysis is code-level only; I
haven't run a repro, and yours would pin down which of the states you actually hit.

Next steps

This deserves a tracking issue rather than a Q&A thread — happy to open one, or feel free to file it
and I'll link my notes. My current thinking on the fix is: port the #13739 else branch, and pair it
with routing aborted replays to replayFailed rather than replayComplete (today every
isReadable=false exit falls through to replayComplete(), which would mark the handle Ready with
partial pending-ack state — so widening the exit conditions without that change trades a hang for
silent incorrect transaction state). Re-checking cursor.isClosed() inside the loop and restoring
the interrupt are cheap additions that make unload a reliable remedy.

---

_{Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). The code paths, line
numbers and commit references above were reviewed by me against the source before posting; the
analysis is code-level only — I have not run a reproduction.}

[lhotari]

Filed this as #26368 and opened #26369 with a fix.

What the PR does

• Ports the [Transaction] Fix recover of TransactionBuffer #13739 else branch into MLPendingAckStore.FillEntryQueueCallback.fillQueue(): when the
  cursor has nothing left to read and the queue is drained, the replay ends instead of sleeping forever.
  That is the state where the loop guard and the read gate disagree permanently, and it is the one I
  could reproduce deterministically.
• Re-checks cursor.isClosed() inside the loop, so closing the subscription ends a replay whose read
  completion never arrives.
• Restores the interrupt flag and ends the replay via replayFailed() instead of continuing, so the
  task is cancellable and a stuck replay no longer survives ExecutorProvider.shutdownNow().
• Releases entries left in the queue on every abnormal exit — a read can still land while the replay is
  ending, and those buffers were never released.
• PendingAckHandleImpl.exceptionHandleFuture() no longer retries once the handle is closing. The retry
  path resets the state to None before scheduling init(), which defeats the checkIfClose() guard
  and reopens the pending-ack store of a subscription that is going away. That was reachable before, but
  the new in-loop close check would have made it happen on every unload during a replay.

Three tests, each verified to fail when its corresponding change is reverted.

What it does not fix

Being explicit, because it matters for whether this actually covers your incident:

• If what you hit was the "read was issued and its callback never came back" variant — your original
  model, outstandingReadsRequests == 1 — the loop still waits, because there is genuinely nothing to
  decide on. What changes is that it is no longer unrecoverable: unloading the topic now ends it, and
  it no longer outlives broker shutdown. Preventing that state belongs in managed-ledger, not here.
• A terminal read failure such as LedgerNotExistException under the default
  autoSkipNonRecoverableData=false still leaves isReadable == true and re-issues the same doomed
  read. Fixing that is entangled with something I did not want to change quietly: every
  isReadable = false exit currently falls through to replayComplete(), and
  TransactionTest.testEndTPRecoveringWhenManagerLedgerDisReadable explicitly asserts that a fenced
  managed ledger or a closed cursor leaves the handle Ready. I tried the obvious fix, that test caught
  it, and I backed it out — it is a deliberate old decision and reversing it deserves its own PR and its
  own discussion.
• TopicTransactionBuffer has the mirror-image gap: it has the else escape (that is where [Transaction] Fix recover of TransactionBuffer #13739
  landed) but no cursor.isClosed() check at all and a bare //no-op interrupt handler. Its abort path
  closes the topic, so it needs its own change and test.

All three are written up in #26368.

Still useful from you

The fix addresses the state I could prove. It would be good to know it is the state you actually hit,
and your reproduction is the fastest way to settle that — please attach it to #26368 if you can share
it. Failing that, the heap-dump values of outstandingReadsRequests / isReadable /
readPosition / currentLoadPosition / lastConfirmedEntry from the stuck broker would tell us
directly, as would whether "...stat reply fail" ERROR lines were repeating and whether the broker had
restarted shortly before.

Review on the PR is welcome, particularly on the Ready-versus-failed question above.

---

_{Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). The code paths, line numbers
and commit references were reviewed by me against the source before posting.}

[kaminski-dev]

Thanks, @lhotari for the quick response and even the PR on that!

We agree that managedLedgerReadEntryTimeoutSeconds is not a fix for this replay loop. A managed-ledger timeout becomes a plain ManagedLedgerException; MLPendingAckStore.readEntriesFailed does not set isReadable=false for it, so replay retries rather than terminating.

Our proposed empty-ledger trim race was also incorrect. As you pointed out, internalTrimLedgers is bounded by the slowest durable cursor's mark-delete position.

We have now compared two environments and appear to have observed both replay failure modes.

SIT: silent replay stall
In the original SIT incident window, "MLPendingAckStore ... stat reply fail!" appeared only twice, both as CursorAlreadyClosedException on broker-2. There were no LedgerNotExist, NoSuchLedger, or NonRecoverable errors from the transaction executor, and no error storm on the repeatedly affected broker-1.
The stuck executor was silent. This strongly favours the outstandingReadsRequests == 0 / cursor.hasMoreEntries() == false state addressed by #26369, rather than our original lost-callback hypothesis or a repeatedly failing ledger read. Without the heap dump, however, we cannot prove the value of outstandingReadsRequests. As of now we did not have a heap dump which we can provide.

Answers for the SIT incidents:

1. We do not have the heap values for outstandingReadsRequests, isReadable, or the cursor positions.
2. No repeating "stat reply fail!" errors were present around the silent stalls.
3. Several affected brokers had restarted shortly beforehand. In four of six retained shutdown dumps, the broker had started approximately 5–18 minutes earlier.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation. Topic unload messages occurred during broker shutdown and therefore appear to be consequences rather than proven triggers.
5. We could not confirm transactionPendingAckBatchedWriteEnabled from the runtime logs. There is no override, so it is the v4.0.10 default of false.
6. Broker shutdown unloaded topics, but the replay executor still failed to terminate. Because the shutdown dump does not identify the owning topic, we cannot prove whether unloading that specific topic had been attempted earlier.

UAT: repeating read-failure loop
On UAT, which uses the same broker configuration, we observed the other case.
On 2026-07-26, "MLPendingAckStore ... stat reply fail!" repeated at very high frequency on all four brokers. In a 15-minute window, the approximate counts were:

• broker-0: 47,616
• broker-1: 1,450
• broker-2: 59,358
• broker-3: 31,149
  Every sampled error was:
  ManagedLedgerException:
  Bookie handle is not available error code: -8
  Affected pending-ack stores included:
  < internal-tenant >/< internal-namespace >/< internal-topic >-v1
  < sub-name >
  and:
  < internal-tenant >/< internal-namespace >/< internal-topic >-v1
  < sub-name >

This is clearly the repeatedly failing-read case. It is not evidence of a lost callback because the callbacks were arriving with failures and replay was immediately issuing another read.

Answers for the UAT incident:

1. Again, we do not have the heap values.
2. Yes, "stat reply fail!" repeated thousands of times with plain ManagedLedgerException and BookKeeper error -8.
3. Yes. The cluster was in a broker restart loop. For example, broker-0 started at 06:28:37Z, and failures were repeating by 06:28:49Z.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation for the affected subscriptions. There was extensive shutdown/unload/reload activity, together with BookKeeper unavailability.
5. The deployment configuration matches SIT, but the batched pending-ack value was not independently visible in the logs.
6. The affected topics were unloaded and subsequently recovered. Both subscriptions later logged pending ack state reply success!, and no further "stat reply fail!" errors appeared after approximately 06:30Z.

A separate silent PendingAckReplay shutdown stall occurred again in UAT on 2026-08-05 without a corresponding "stat reply fail!" storm. That reinforces the conclusion that we have encountered both states at different times: the silent no-more-entries stall and the repeatedly failing-read loop.

In the attachments you will find the test which I run to "prove" my thesis and the output of the test itself.
I will try to get the heap dump, but if anything else is needed please let me know as I am happy to help here in the investigations.

test_output.log
DeletedPendingAckLedgerTest.java

[lhotari]

This is excellent — thank you. The two environments plus the reproduction settle a lot.

Your reproduction is the other state, and #26369 does not fix it

I traced it against the PR head. Deleting the ledger gives
BKNoSuchLedgerExistsOnMetadataServerException, which isBkErrorNotRecoverable classifies as
non-recoverable, so createManagedLedgerException returns LedgerNotExistException. With
autoSkipNonRecoverableData=false, OpReadEntry.internalReadEntriesFailed skips its
advance-past-the-bad-ledger branch (that branch is gated on auto-skip) and fails the read without
moving the read position. In readEntriesFailed that exception matches none of the three cases that
clear isReadable, so the outstanding counter drops back to zero and fillQueue() issues the identical
read again.

Both of #26369's new exits are unreachable there: the completion branch requires
hasMoreEntries() == false, and it stays true forever because the read position never advances; the
in-loop close check requires a closed cursor, and the cursor is open. So your reproduction would still
block the lane with the PR applied. I did not add it to #26369 — a test that passes identically before
and after would be verifying nothing.

I filed it instead: #26374, with your reproduction and the UAT numbers as the evidence. Your test's
mock seams are the right skeleton for that fix's tests, so it was far from wasted — it is what pinned
the classification.

Why that half was never fixed

Worth knowing, because it explains the shape: #12700's stated motivation was exactly your scenario —
"if any ledger was deleted from bookkeeper, or ManagerLedger was fenced, MLPendingAckStore will not
stop recovering and continue to report the exception". It added the guard and the test that pins
it. But that test sets autoSkipNonRecoverableData(true), so the deleted-ledger case was only ever
fixed for the non-default configuration. The default was left spinning, and that is what you hit.

That also means the fix does not need a design discussion, which I had assumed it might: the pinned test
covers only auto-skip-enabled, fenced, and closed-cursor. The two exception classes that actually cause
your storms are pinned by nothing, so they can be handled without touching the existing contract. Your
-8 case (BookieHandleNotAvailable) is retryable and should back off — which also releases the replay
thread between attempts, curing the starvation; the deleted-ledger case is not retryable and should fail
the handle so it is visible.

On the SIT stalls

Your reading matches mine. Silent, no error storm, brokers restarted 5–18 minutes earlier — that is the
no-more-entries state #26369 fixes, and the recent restarts fit its most likely cause. It is not proof
without the heap values, but I would not hold up the fix for them. If you do get a heap dump, the four
positions are still the decisive measurement.

One detail worth flagging from your point 6: "broker shutdown unloaded topics, but the replay executor
still failed to terminate". That is consistent with the old code on both counts — the interrupt was
swallowed, so shutdownNow() could not stop it, and cursor.isClosed() was only checked once before
the loop, so unload could not either. Both are fixed in #26369, so that specific symptom should go away
regardless of which state caused the stall.

Also in #26369 since you last looked

A reviewer spotted that a read still in flight when the replay ends could deliver entries afterwards
that nothing ever released. Fixed, with the caveat that the obvious fix is racy — the queue is
single-consumer, so a late callback must release its own entries rather than drain — plus a test.

Nothing further needed from you unless the heap dump turns up. Thanks for the rigour on this one; the
two-environment comparison is what made the classification unambiguous.

---

_{Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). Code paths, line numbers and
commit references were reviewed by me against the source before posting.}