Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JmsTemplate/CachingConnectionFactory doesn't reset sessions if the underlying connection was physically closed [SPR-17499] #22031

Open
spring-projects-issues opened this issue Nov 15, 2018 · 3 comments
Labels
in: messaging type: enhancement

Comments

@spring-projects-issues
Copy link
Collaborator

@spring-projects-issues spring-projects-issues commented Nov 15, 2018

Kevin Möchel opened SPR-17499 and commented

If one uses the JmsTemplate to produce messages together with the CachingConnectionFactory it's possible to break the cached Sessions in a way that the application doesn't recover from this by itself, leaving no other options then to restart the application.
We use Oracle UCP and Oracle AQ for messaging. I think the problem cam occur with every JMS implementation though.

This is the exception we observe:

org.springframework.jms.IllegalStateException: JMS-131: Session is closed; nested exception is javax.jms.IllegalStateException: JMS-131: Session is closed
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:580)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:668)
at com.example.jms.SessionClosedTest.testJms131(SessionClosedTest.java:171)
...
Caused by: javax.jms.IllegalStateException: JMS-131: Session is closed
at oracle.jms.AQjmsError.throwIllegalStateEx(AQjmsError.java:471)
at oracle.jms.AQjmsSession.checkSessionStarted(AQjmsSession.java:4450)
at oracle.jms.AQjmsSession.getDBConnection(AQjmsSession.java:4392)
at oracle.jms.AQjmsSession.getAQOwner(AQjmsSession.java:6447)
at oracle.jms.AQjmsSession.createQueue(AQjmsSession.java:1387)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:386)
at com.sun.proxy.$Proxy73.createQueue(Unknown Source)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:84)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:58)
at org.springframework.jms.support.destination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:98)
at org.springframework.jms.core.JmsTemplate.access$200(JmsTemplate.java:90)
at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:583)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
... 33 more

How to reproduce the error

Application Configuration

Configuration of the connection pool:

<bean id="appDataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory" factory-method="getPoolDataSource"
    lazy-init="true">
    <property name="connectionFactoryClassName" value="oracle.jdbc.pool.OracleDataSource"/>
    <property name="user" value="${database.username}"/>
    <property name="password" value="${database.password}"/>
    <property name="URL" value="${database.url}"/>
    <property name="initialPoolSize" value="5"/>
    <property name="minPoolSize" value="2"/>
    <property name="maxPoolSize" value="20"/>
    <property name="connectionWaitTimeout" value="10"/>
    <property name="inactiveConnectionTimeout" value="180"/>
    <property name="timeToLiveConnectionTimeout" value="0"/>
    <property name="abandonedConnectionTimeout" value="3"/>
    <property name="maxConnectionReuseTime" value="3"/>
    <property name="maxConnectionReuseCount" value="0"/>
    <property name="validateConnectionOnBorrow" value="true"/>
    <property name="maxStatements" value="80"/>
    <property name="timeoutCheckInterval" value="3"/>
    <property name="connectionProperties">
        <props merge="default">
            <prop key="oracle.net.disableOob">true</prop>
            <prop key="oracle.net.CONNECT_TIMEOUT">10000</prop>
            <prop key="oracle.jdbc.ReadTimeout">12000</prop>
        </props>
    </property>
</bean>

For arguments sake I set the abandonedConnectionTimeout, maxConnectionReuseTime and timeoutCheckInterval to 3, in order to force the ConnectionPool to close all connections that haven't been used for a little while.

XML-Configuration of the CachingConnectionFactory and the JmsTemplate:

<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="sessionCacheSize" value="4"/>
    <property name="targetConnectionFactory">
        <bean class="com.example.jms.oracleaq.OracleAqConnectionFactoryBean">
            <property name="datasource" ref="appDataSource"/>
        </bean>
    </property>
</bean>

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="sessionAcknowledgeMode" value="0"/>
    <property name="sessionTransacted" value="true"/>
    <property name="deliveryPersistent" value="true"/>
    <property name="explicitQosEnabled" value="true"/>
    <property name="defaultDestinationName" value="SAMPLE_QUEUE"/>
</bean>

Implementation of OracleAqConnectionFactoryBean:

public class OracleAqConnectionFactoryBean implements FactoryBean<QueueConnectionFactory> {
    ...
    public QueueConnectionFactory getObject() throws Exception {
        return oracle.jms.AQjmsFactory.getQueueConnectionFactory(datasource);
    }
    ...
}

.h3 Test

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(value = { "/resources/spring/jpa.xml" })
public class SessionClosedTest {
    @Autowired
    private JmsTemplate jmsTemplate;
    
    @Test
    public void testRF242() throws Exception {
        MyBusinessObject someObj = new MyBusinessObject();
        this.jmsTemplate.convertAndSend(someObj);
        // Funny note: If you type Thread.sleep(15000); in the line below and remove this comment, Jira is actually not able to create the issue and the "Create Issue"-dialog completly breaks. On our own Jira we don't have this problem.
        Thread.sleep(15000);
        this.jmsTemplate.convertAndSend(someObj);
    }
}

The first call of convertAndSend works as expected. However the second call will fail with the exception at the beginning.

It's possible to get the test above to run successfully if you call CachingConnectionFactory#resetConnection after the sleep.

Spring-JMS will call CachingConnectionFactory#resetConnection on its own given any JmsException, but only for Message-Consumer. I couldn't find similar behaviour for Message-Producer.

Is this something that should be adressed by Spring? The easiest fix is probably to reset the connections in the JmsTemplate if a JmsException occurs.


Affects: 4.3.19

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Nov 20, 2018

Juergen Hoeller commented

This is a general issue with JMS resources: We can't easily inspect whether they're actually still open. In the case of a listener, we simply abort the receive attempt, propagate the exception (leading to a connection reset) and wait for the message to arrive again. In the case of a producer, we'd have to implicitly retry the send which is not trivial to do at this level (we'd have to react to specific state exceptions at specific times in order to not retry unnecessarily).

If there was an isOpen() method on Session or Connection, we could simply check it before we return a cached Session within CachingConnectionFactory itself. I guess we could call Connection.getClientID() or Connection.getMetaData() and assume it would throw an IllegalStateException if the connection has been closed in the meantime? Could you check whether that's the case with Oracle AQ for either of those two methods?

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Jan 7, 2019

Kevin Möchel commented

Sorry it took me so long to get back to this issue.

Unfortunately neither Connection.getClientID() nor Connection.getMetaData() have the desired behaviour. We tried a few other things to find out whether the connection is still alive or not but nothing gave the right answer.
Internally we bypass this error now by calling CachingConnectionFactory.resetConnection given any org.springframework.jms.IllegalStateException and then try to send the JMS message again. This is not optimal because we might call CachingConnectionFactory.resetConnection when it's not necessary (as you already mentioned), however the Exception is so rare that it doesn't matter for us.

@spring-projects-issues spring-projects-issues added in: messaging type: enhancement labels Jan 11, 2019
@ranganath2334
Copy link

@ranganath2334 ranganath2334 commented Dec 7, 2020

Hi Juergen Hoeller,
We are also facing the same issue with the combination JmsTemplate/CachingConnectionFactory.
When the exception "org.springframework.jms.IllegalStateException: JMSCC0020 This session is closed" occured, we have called explicitly CachingConnectionFactory#resetConnection in the catch block and retried sending the message. But still the same exception occurred again

The issue of "org.springframework.jms.IllegalStateException: JMSCC0020 This session is closed." is occurring only intermittently

Spring JMS version is 5.2.7
IBM MQ 7.5.0.8
Spring-boot is 2.2.8.

there was an exception when it tried to close the connection: so, the connection was not closed and in the retry of sending message closed session was used and failed again.
stack_trace: com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment$1: MQJMS2000: failed to close MQ queue.
at com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment.newException(ConfigEnvironment.java:379)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.closeQ(MQMessageConsumer.java:1135)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.close(MQMessageConsumer.java:1069)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.close(JmsMessageConsumerImpl.java:266)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionImpl.java:390)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:305)
at com.ibm.mq.jms.MQConnection.close(MQConnection.java:98)
at org.springframework.jms.connection.SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:501)
at org.springframework.jms.connection.SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:389)
at org.springframework.jms.connection.CachingConnectionFactory.resetConnection(CachingConnectionFactory.java:205)
at org.springframework.jms.connection.SingleConnectionFactory.onException(SingleConnectionFactory.java:367)
at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:721)
at com.ibm.msg.client.jms.internal.JmsProviderExceptionListener.run(JmsProviderExceptionListener.java:427)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:231)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1241)
Caused by: com.ibm.mq.MQException: Message catalog not found, completion=2, reason=2009
at com.ibm.msg.client.wmq.v6.base.internal.MQManagedObject.close(MQManagedObject.java:207)
at com.ibm.msg.client.wmq.v6.base.internal.MQQueue.close(MQQueue.java:360)
at com.ibm.msg.client.wmq.v6.jms.internal.MQMessageConsumer.closeQ(MQMessageConsumer.java:1127)
... 16 common frames omitted

Could you please help on this

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: messaging type: enhancement
Projects
None yet
Development

No branches or pull requests

2 participants