Skip to content

Close the consumer when removeMessageConsumer removes it - #1847

Merged
jcschaff merged 3 commits into
masterfrom
fix/jms-consumer-close
Aug 7, 2026
Merged

Close the consumer when removeMessageConsumer removes it#1847
jcschaff merged 3 commits into
masterfrom
fix/jms-consumer-close

Conversation

@jcschaff

@jcschaff jcschaff commented Aug 6, 2026

Copy link
Copy Markdown
Member

Last of the JMS resource problems turned up while working through the export-progress
incident (#1837, #1838, #1839, #1842, #1844, #1845).

The leak

removeMessageConsumer only called stop(), which asks the polling thread to finish its
current loop:

if (context.getVCConsumer() == vcMessagingConsumer){
    try { context.stop(); } finally { consumerContexts.remove(context); }
    return;
}

The MessageConsumer, its session and its connection were left open. The broker therefore
went on dispatching to a consumer nobody was reading — and with a prefetch limit those
messages sit in its client-side buffer, undelivered and not redelivered until the
connection dies
. Every VCell consumer is registered with an explicit prefetch limit
(MessageConstants.PREFETCH_LIMIT_*), so any removed consumer strands at least one message.

The fix

ConsumerContextJms.stopAndClose() stops the thread, waits for it to leave its receive()
— bounded by two polling intervals, the same budget VCMessagingServiceJms.close() already
allows — and then closes the consumer, session and connection. removeMessageConsumer uses
it.

Finding the context first and mutating the list afterwards also removes a
remove-while-iterating that only worked because of the immediate return.

Verification

removingAConsumerDoesNotStrandMessages registers a consumer, removes it, sends three
messages, then registers a replacement and requires all three to arrive. Control, with both
production files reverted to master and the test kept:

all 3 messages should reach the replacement consumer; a removed consumer must not
still be holding one ==> expected: <true> but was: <false>

Exactly one message stranded, as predicted by the prefetch limit of 1. vcell-server Fast
group: 61 passed.

This one was not hypothetical

It cost a full debugging cycle on #1845. A responder torn down by an earlier test in the
same class was still holding a prefetched message, so a later test failed in a way that
looked exactly like the concurrency bug that PR was fixing — four concurrent callers passed,
six did not. Running the same test against origin/master first is what ruled out the
change under test and pointed here instead.

In production the same shape means a request silently stranded whenever a consumer is
removed and re-registered: no error, no redelivery, just one message that never gets a
reply until the connection is torn down.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SY1XHgTZXZPqUECo2VBAWg

jcschaff and others added 3 commits August 6, 2026 15:19
removeMessageConsumer only called stop(), which asks the polling thread to
finish its current loop. The MessageConsumer, its session and its connection
were left open, so the broker went on dispatching to a consumer nobody was
reading: with prefetchLimit=1 one message sat in it and was not redelivered
until the connection died.

ConsumerContextJms.stopAndClose() now stops the thread, waits for it to leave
its receive() (bounded by two polling intervals, the same budget close() uses),
and then releases the JMS resources.

This is not hypothetical -- it cost a full debugging cycle on #1845, where a
responder torn down by one test still held a prefetched message and made a
later test look like a concurrency bug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SY1XHgTZXZPqUECo2VBAWg
The push that opened #1847 landed during a GitHub Actions major outage, so no
workflow run was ever created for it -- the PR sat with zero checks and nothing
retroactively schedules one. ci.yml fires on push, not on pull_request, so an
empty commit is what re-triggers it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SY1XHgTZXZPqUECo2VBAWg
stopAndClose() closes the MessageConsumer while the polling thread may still be
parked in receive(). Two problems made that catastrophic rather than merely
untidy:

  - bProcessing was not volatile, so the polling thread could miss stop()
    entirely and keep looping;
  - receive() on a closed consumer throws immediately, and the loop logged it
    via onException() and went straight round again.

The result was a tight spin logging "The Consumer is closed" as fast as the CPU
allows: 2,125,490 such lines in one local run of MessageProducerSessionJmsTest.
In CI it starved the shard badly enough that rpcRoundTripReturnsTheAnswer timed
out after 30s, which is how it surfaced.

bProcessing and thread are now volatile, and a consumer closed while polling is
treated as shutdown -- logged at debug and the loop exits -- rather than as an
error to report and retry. Same run now produces 0 such lines.

This could not happen before the previous commit, because removeMessageConsumer
never closed the consumer at all; it just leaked it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SY1XHgTZXZPqUECo2VBAWg
@jcschaff
jcschaff merged commit b464e8a into master Aug 7, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant