Jetty should start only on active broker (not on passive)

[jbonofre]

I observed a race condition where the Jetty server can start on passive broker (slave), whereas it should not be the case.

The problem is that Jetty server is not a broker service, so Jetty doesn't consider the broker type (active or passive).

I investigated and I found the sequence explaining this behavior:

1. BrokerService.@PostConstruct -> autostart() -> start() is called
2. With startAsync=true, startPersistenceAdapter() and startBroker() both launch background threads and return immediately
3. At this point slave=true - the broker is still waiting to acquire the file/JDBC lock
4. Spring continues initializing the remaining beans from jetty.xml
5. The invokeStart bean - a MethodInvokingFactoryBean that calls Server.start() - fires unconditionally during Spring context initialization
6. Jetty is now running on the passive/slave

The passive/slave stays in this state (Jetty up, broker passive) until the master fails. Then the background thread acquires the lock, calls startAllConnectors(), and slave becomes false. But Jetty was already running 😄

NB: this case only happens with startAsync=true in a master/slave configuration - a combination that's commonly recommended to avoid blocking the Spring context during lock contention.

NB: this case doesn't happen with startAsync=false: BrokerService.@PostConstruct blocks until the broker fully starts (lock acquired -> startAllConnectors() -> startedLatch.countDown()). Spring only continues to the jetty.xml beans after that. By then the broker is already master/active, so invokeStart fires the right time.

[mattrpav]

I'm not sure this is a bug. What is the problem with Jetty starting with a failover/slave broker?

1. The transport connectors and persistence are offline, so no messages can move and no clients connect.
2. API calls will fail as well.

[jbonofre]

@mattrpav for me it's a bug for two reasons:

1. It's a different behavior from the "happy path". Users might be surprised because 99% of the time Jetty won't be started, but sometime the Jetty is there. If the users scan the ports, etc, it can introduce "false positive" or assuming the broker is there.
2. The WebConsole fails if Jetty is running on a slave.

I think this "race condition" should be avoided and fixed. A follow up improvement could be done on WebConsole.

[mattrpav]

I don't think there is a solution in the current setup due to Spring and async combo. Async startup is needed, and the "don't" start flag (<broker .. start="false" ..>) is needed for the RuntimeConfig plugin to work at all. The broker bean will always be present before online and regardless if slave or not slave, so the depends-on only has minor impact here.

[jbonofre]

My idea is to "wrap" Jetty bean to "link" with broker.

@mattrpav do you mind if I try something (and you review) ?