Skip to content

StartCommand: keep waiting past the default start timeout unless a start exception was recorded - #2554

Open
jbonofre wants to merge 1 commit into
apache:mainfrom
jbonofre:fix/startcommand-slave-startasync-timeout
Open

StartCommand: keep waiting past the default start timeout unless a start exception was recorded#2554
jbonofre wants to merge 1 commit into
apache:mainfrom
jbonofre:fix/startcommand-slave-startasync-timeout

Conversation

@jbonofre

@jbonofre jbonofre commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

  • Fixes StartCommand exits with no logged error after 10-minute default timeout when a slave broker legitimately waits for the lock under startAsync=true #2552: a broker started via ./activemq start with startAsync=true, running as a slave, disappeared after roughly 10 minutes with nothing logged.
  • BrokerService.DEFAULT_START_TIMEOUT is 10 minutes, and StartCommand called broker.waitUntilStarted() with that timeout, throwing (and exiting the process) whenever it returned false. Before startAsync existed this was harmless, since start() itself blocked until the broker became master. With startAsync=true, start() returns immediately, and a slave can legitimately wait on the lock far longer than 10 minutes.
  • StartCommand now loops on waitUntilStarted(), only throwing/exiting once broker.getStartException() is actually non-null. This can't spin indefinitely: BrokerService.stop() always records a start exception (BrokerStoppedException) before it ever marks the broker as stopped, so a real stop or startup failure still surfaces immediately.

Test plan

  • mvn -pl activemq-console -am compile succeeds
  • Manual verification: start a slave broker with startAsync=true and confirm the process stays alive past 10 minutes while waiting on the lock, and exits promptly with a logged error on an actual startup failure

Fixes #2552

… exception was actually recorded

BrokerService.DEFAULT_START_TIMEOUT is 10 minutes, and StartCommand called
broker.waitUntilStarted() with that timeout, throwing (and exiting the
process) whenever it returned false. Before startAsync existed this was
never an issue, since start() itself blocked until the broker became
master, so waitUntilStarted() returned almost immediately either way.

With startAsync=true, start() returns right away, and a slave broker can
legitimately wait on the lock far longer than 10 minutes - that's expected,
not a failure. When the timeout hit, waitUntilStarted() returned false with
no start exception set, and StartCommand treated that as a startup failure
and exited the process with nothing logged.

StartCommand now keeps calling waitUntilStarted() as long as
broker.getStartException() is null, and only throws/exits once an actual
start exception has been recorded. BrokerService.stop() always records a
start exception before it ever marks the broker stopped, so this cannot
spin: a real stop or start failure still surfaces immediately.

Fixes apache#2552

if (!broker.waitUntilStarted()) {
throw new Exception(broker.getStartException());
while (!broker.waitUntilStarted()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just like my comment on the other PR, we should probably do an broker.isSlave() check here. The exception should be set, but in case something gets broken in the future this would add an extra check just to ensure we always fail for a primary/active broker just like today if it returns false.

Maybe:

while (!broker.waitUntilStarted()) {
    if (broker.getStartException() != null || !broker.isSlave()) {
        throw new Exception(broker.getStartException());  
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StartCommand exits with no logged error after 10-minute default timeout when a slave broker legitimately waits for the lock under startAsync=true

2 participants