Observed failure
master push run 30078532797, matrix cell build-maven (ubuntu-latest, 21) — 8 of 9 cells green, single failure in persistit/core:
[ERROR] com.persistit.TransactionTest2.transactionsConcurrentWithPersistitClose -- Time elapsed: 1.194 s <<< FAILURE!
java.lang.AssertionError: All threads should have exited correctly expected:<0> but was:<1>
The worker thread's stack trace printed alongside the assertion:
java.lang.IllegalStateException: JOURNAL_FLUSHER is not running
at com.persistit.JournalManager.waitForDurability(JournalManager.java:1860)
at com.persistit.Transaction.commit(Transaction.java:897)
at com.persistit.TransactionTest2.transfer(TransactionTest2.java:380)
at com.persistit.TransactionTest2.runIt(TransactionTest2.java:313)
at com.persistit.TransactionTest2$3.run(TransactionTest2.java:272)
Analysis
This is a residual failure mode of the flake previously stabilized by #255, with a different mechanism that the #255 fix (deadline-bounded join before the assert) cannot help with:
- The test's main thread calls
_persistit.close(); JournalManager.close() nulls _flusher (JournalManager.java:1327).
- A worker thread is concurrently inside
Transaction.commit() and reaches JournalManager.waitForDurability(), which finds _flusher == null and throws a raw IllegalStateException("JOURNAL_FLUSHER is not running") (JournalManager.java:1860).
IllegalStateException is not a PersistitException, so in TransactionTest2.runIt it bypasses the expected-shutdown catch clauses (PersistitClosedException, PersistitInterruptedException, interrupted-I/O PersistitIOException) and lands in the generic catch (Throwable) branch, which records a failure and does not decrement _strandedThreads.
- The assertion then fails with
expected:<0> but was:<1> regardless of how long the join waits — the worker has already exited, but the counter can no longer reach zero.
Root cause
A transaction that races with Persistit.close() surfaces the shutdown as a raw IllegalStateException instead of the documented PersistitClosedException. Callers (including this test) that treat PersistitClosedException as the expected outcome of a concurrent close cannot distinguish this from a genuine internal error.
Candidate fixes
- Product side (preferred):
JournalManager.waitForDurability() should throw PersistitClosedException when the flusher has been stopped by close(), so a concurrent commit() unwinds through the same path as every other closed-state operation.
- Test side (fallback): treat
IllegalStateException("JOURNAL_FLUSHER is not running") during shutdown as an expected exit in TransactionTest2.runIt.
Observed failure
masterpush run 30078532797, matrix cellbuild-maven (ubuntu-latest, 21)— 8 of 9 cells green, single failure inpersistit/core:The worker thread's stack trace printed alongside the assertion:
Analysis
This is a residual failure mode of the flake previously stabilized by #255, with a different mechanism that the #255 fix (deadline-bounded
joinbefore the assert) cannot help with:_persistit.close();JournalManager.close()nulls_flusher(JournalManager.java:1327).Transaction.commit()and reachesJournalManager.waitForDurability(), which finds_flusher == nulland throws a rawIllegalStateException("JOURNAL_FLUSHER is not running")(JournalManager.java:1860).IllegalStateExceptionis not aPersistitException, so inTransactionTest2.runItit bypasses the expected-shutdown catch clauses (PersistitClosedException,PersistitInterruptedException, interrupted-I/OPersistitIOException) and lands in the genericcatch (Throwable)branch, which records a failure and does not decrement_strandedThreads.expected:<0> but was:<1>regardless of how long the join waits — the worker has already exited, but the counter can no longer reach zero.Root cause
A transaction that races with
Persistit.close()surfaces the shutdown as a rawIllegalStateExceptioninstead of the documentedPersistitClosedException. Callers (including this test) that treatPersistitClosedExceptionas the expected outcome of a concurrent close cannot distinguish this from a genuine internal error.Candidate fixes
JournalManager.waitForDurability()should throwPersistitClosedExceptionwhen the flusher has been stopped byclose(), so a concurrentcommit()unwinds through the same path as every other closed-state operation.IllegalStateException("JOURNAL_FLUSHER is not running")during shutdown as an expected exit inTransactionTest2.runIt.