JmsHealthIndicator can hang in case failover with infinite max reconnects is used with ActiveMQ

[filiphr]

First of all, I have to say that it would be wrong to use infinite max reconnects with ActiveMQ. However, that is the default starting from ActiveMQ 5.6 (see failover transport-reference).

In case the default is used then the actuator endpoint /health will hang forever. Do you think that something like this should be addressed in the JmsHealthIndicator itself? I think that it would be good if the endpoint doesn't hang forever.

[snicoll]

@filiphr I am not sure we can do anything about it really. If returning a valid javax.jms.Connection never returns when the remote broker is down, this is beyond what the JMS API allows us to invoke.

[filiphr]

@snicoll sorry I was not clearer. The connection.start() is the place where it blocks. The connection is correctly created. javax.jms.Connection has setExceptionListener that catches exceptions (and I am 90% sure that it catches those as well). I think that Spring Boot can do something with them (not sure whether it is OK or not though). My idea was something like:

protected void doHealthCheck(Health.Builder builder) throws Exception {
    try (Connection connection = this.connectionFactory.createConnection()) {
        ExceptionListener original = connection.getExceptionListener();
        ExceptionListener bootListener = new ExceptionListener() {
            @Override
            public void onException(JMSException exception) {
                if (original != null) {
                    original.onException(exception);
                }
                // spring boot can do something here 
                // (some counts, or have some timer how long it should try, or something similar)
                // and close the connection
            }
        }
        connection.start();
	builder.up().withDetail("provider",
		     connection.getMetaData().getJMSProviderName());
	}
}

How exactly Spring Boot will decide to close the connection I am not sure yet. Maybe this is out of scope for Spring Boot. I just wanted to check if there might be something that you can do. I was really confused why the /health endpoint was being blocked.

[snicoll]

@filiphr I am not keen to use a feature that has this kind of warning:

This method must not be used in a Java EE web or EJB application. Doing so may cause a {@code JMSException} to be thrown though this is not guaranteed.

Also the Javadoc of ExceptionListener states that

A JMS provider should attempt to resolve connection problems itself before it notifies the client of them.

I'll try to look at other options. If you have a working patch, please submit a PR.

[filiphr]

@snicoll I didn't notice that, and you are right it is not good to use something like that.
The thing is that the JMS provider is trying to resolve the connection problem (infinitely trying to reconnect 😄).

I had another idea and that is to run it wrapped into some separate Thread that we would block on. If it doesn't return in some time we can write "down" or "unstable" (I don't know what is there in the Actuators) and just write that.

I can prepare an patch and PR for this in the next few days.

[filiphr]

I've created a PR as promise. It is WIP and I am open to discussions about it