JmsTemplate/CachingConnectionFactory doesn't reset sessions if the underlying connection was physically closed [SPR-17499]

[spring-projects-issues]

Kevin Möchel opened SPR-17499 and commented

If one uses the JmsTemplate to produce messages together with the CachingConnectionFactory it's possible to break the cached Sessions in a way that the application doesn't recover from this by itself, leaving no other options then to restart the application.
We use Oracle UCP and Oracle AQ for messaging. I think the problem cam occur with every JMS implementation though.

This is the exception we observe:

org.springframework.jms.IllegalStateException: JMS-131: Session is closed; nested exception is javax.jms.IllegalStateException: JMS-131: Session is closed
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:580)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:668)
at com.example.jms.SessionClosedTest.testJms131(SessionClosedTest.java:171)
...
Caused by: javax.jms.IllegalStateException: JMS-131: Session is closed
at oracle.jms.AQjmsError.throwIllegalStateEx(AQjmsError.java:471)
at oracle.jms.AQjmsSession.checkSessionStarted(AQjmsSession.java:4450)
at oracle.jms.AQjmsSession.getDBConnection(AQjmsSession.java:4392)
at oracle.jms.AQjmsSession.getAQOwner(AQjmsSession.java:6447)
at oracle.jms.AQjmsSession.createQueue(AQjmsSession.java:1387)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:386)
at com.sun.proxy.$Proxy73.createQueue(Unknown Source)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:84)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:58)
at org.springframework.jms.support.destination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:98)
at org.springframework.jms.core.JmsTemplate.access$200(JmsTemplate.java:90)
at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:583)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
... 33 more

How to reproduce the error

Application Configuration

Configuration of the connection pool:

<bean id="appDataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory" factory-method="getPoolDataSource"
    lazy-init="true">
    <property name="connectionFactoryClassName" value="oracle.jdbc.pool.OracleDataSource"/>
    <property name="user" value="${database.username}"/>
    <property name="password" value="${database.password}"/>
    <property name="URL" value="${database.url}"/>
    <property name="initialPoolSize" value="5"/>
    <property name="minPoolSize" value="2"/>
    <property name="maxPoolSize" value="20"/>
    <property name="connectionWaitTimeout" value="10"/>
    <property name="inactiveConnectionTimeout" value="180"/>
    <property name="timeToLiveConnectionTimeout" value="0"/>
    <property name="abandonedConnectionTimeout" value="3"/>
    <property name="maxConnectionReuseTime" value="3"/>
    <property name="maxConnectionReuseCount" value="0"/>
    <property name="validateConnectionOnBorrow" value="true"/>
    <property name="maxStatements" value="80"/>
    <property name="timeoutCheckInterval" value="3"/>
    <property name="connectionProperties">
        <props merge="default">
            <prop key="oracle.net.disableOob">true</prop>
            <prop key="oracle.net.CONNECT_TIMEOUT">10000</prop>
            <prop key="oracle.jdbc.ReadTimeout">12000</prop>
        </props>
    </property>
</bean>

For arguments sake I set the abandonedConnectionTimeout, maxConnectionReuseTime and timeoutCheckInterval to 3, in order to force the ConnectionPool to close all connections that haven't been used for a little while.

XML-Configuration of the CachingConnectionFactory and the JmsTemplate:

<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="sessionCacheSize" value="4"/>
    <property name="targetConnectionFactory">
        <bean class="com.example.jms.oracleaq.OracleAqConnectionFactoryBean">
            <property name="datasource" ref="appDataSource"/>
        </bean>
    </property>
</bean>

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="sessionAcknowledgeMode" value="0"/>
    <property name="sessionTransacted" value="true"/>
    <property name="deliveryPersistent" value="true"/>
    <property name="explicitQosEnabled" value="true"/>
    <property name="defaultDestinationName" value="SAMPLE_QUEUE"/>
</bean>

Implementation of OracleAqConnectionFactoryBean:

public class OracleAqConnectionFactoryBean implements FactoryBean<QueueConnectionFactory> {
    ...
    public QueueConnectionFactory getObject() throws Exception {
        return oracle.jms.AQjmsFactory.getQueueConnectionFactory(datasource);
    }
    ...
}

.h3 Test

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(value = { "/resources/spring/jpa.xml" })
public class SessionClosedTest {
    @Autowired
    private JmsTemplate jmsTemplate;
    
    @Test
    public void testRF242() throws Exception {
        MyBusinessObject someObj = new MyBusinessObject();
        this.jmsTemplate.convertAndSend(someObj);
        // Funny note: If you type Thread.sleep(15000); in the line below and remove this comment, Jira is actually not able to create the issue and the "Create Issue"-dialog completly breaks. On our own Jira we don't have this problem.
        Thread.sleep(15000);
        this.jmsTemplate.convertAndSend(someObj);
    }
}

The first call of convertAndSend works as expected. However the second call will fail with the exception at the beginning.

It's possible to get the test above to run successfully if you call CachingConnectionFactory#resetConnection after the sleep.

Spring-JMS will call CachingConnectionFactory#resetConnection on its own given any JmsException, but only for Message-Consumer. I couldn't find similar behaviour for Message-Producer.

Is this something that should be adressed by Spring? The easiest fix is probably to reset the connections in the JmsTemplate if a JmsException occurs.

---

Affects: 4.3.19

[spring-projects-issues]

Juergen Hoeller commented

This is a general issue with JMS resources: We can't easily inspect whether they're actually still open. In the case of a listener, we simply abort the receive attempt, propagate the exception (leading to a connection reset) and wait for the message to arrive again. In the case of a producer, we'd have to implicitly retry the send which is not trivial to do at this level (we'd have to react to specific state exceptions at specific times in order to not retry unnecessarily).

If there was an isOpen() method on Session or Connection, we could simply check it before we return a cached Session within CachingConnectionFactory itself. I guess we could call Connection.getClientID() or Connection.getMetaData() and assume it would throw an IllegalStateException if the connection has been closed in the meantime? Could you check whether that's the case with Oracle AQ for either of those two methods?

[spring-projects-issues]

Kevin Möchel commented

Sorry it took me so long to get back to this issue.

Unfortunately neither Connection.getClientID() nor Connection.getMetaData() have the desired behaviour. We tried a few other things to find out whether the connection is still alive or not but nothing gave the right answer.
Internally we bypass this error now by calling CachingConnectionFactory.resetConnection given any org.springframework.jms.IllegalStateException and then try to send the JMS message again. This is not optimal because we might call CachingConnectionFactory.resetConnection when it's not necessary (as you already mentioned), however the Exception is so rare that it doesn't matter for us.

[ranganath2334]

Hi Juergen Hoeller,
We are also facing the same issue with the combination JmsTemplate/CachingConnectionFactory.
When the exception "org.springframework.jms.IllegalStateException: JMSCC0020 This session is closed" occured, we have called explicitly CachingConnectionFactory#resetConnection in the catch block and retried sending the message. But still the same exception occurred again

The issue of "org.springframework.jms.IllegalStateException: JMSCC0020 This session is closed." is occurring only intermittently

Spring JMS version is 5.2.7
IBM MQ 7.5.0.8
Spring-boot is 2.2.8.

there was an exception when it tried to close the connection: so, the connection was not closed and in the retry of sending message closed session was used and failed again.
stack_trace: com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment$1: MQJMS2000: failed to close MQ queue.
at com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment.newException(ConfigEnvironment.java:379)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.closeQ(MQMessageConsumer.java:1135)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.close(MQMessageConsumer.java:1069)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.close(JmsMessageConsumerImpl.java:266)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionImpl.java:390)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:305)
at com.ibm.mq.jms.MQConnection.close(MQConnection.java:98)
at org.springframework.jms.connection.SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:501)
at org.springframework.jms.connection.SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:389)
at org.springframework.jms.connection.CachingConnectionFactory.resetConnection(CachingConnectionFactory.java:205)
at org.springframework.jms.connection.SingleConnectionFactory.onException(SingleConnectionFactory.java:367)
at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:721)
at com.ibm.msg.client.jms.internal.JmsProviderExceptionListener.run(JmsProviderExceptionListener.java:427)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:231)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1241)
Caused by: com.ibm.mq.MQException: Message catalog not found, completion=2, reason=2009
at com.ibm.msg.client.wmq.v6.base.internal.MQManagedObject.close(MQManagedObject.java:207)
at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.close(MQQueue.java:360)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.closeQ(MQMessageConsumer.java:1127)
... 16 common frames omitted

Could you please help on this

Thanks