Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JmsTemplate/CachingConnectionFactory doesn't reset sessions if the underlying connection was physically closed [SPR-17499] #22031

Open
spring-issuemaster opened this issue Nov 15, 2018 · 2 comments

Comments

@spring-issuemaster
Copy link
Collaborator

@spring-issuemaster spring-issuemaster commented Nov 15, 2018

Kevin Möchel opened SPR-17499 and commented

If one uses the JmsTemplate to produce messages together with the CachingConnectionFactory it's possible to break the cached Sessions in a way that the application doesn't recover from this by itself, leaving no other options then to restart the application.
We use Oracle UCP and Oracle AQ for messaging. I think the problem cam occur with every JMS implementation though.

This is the exception we observe:

org.springframework.jms.IllegalStateException: JMS-131: Session is closed; nested exception is javax.jms.IllegalStateException: JMS-131: Session is closed
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:580)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:668)
at com.example.jms.SessionClosedTest.testJms131(SessionClosedTest.java:171)
...
Caused by: javax.jms.IllegalStateException: JMS-131: Session is closed
at oracle.jms.AQjmsError.throwIllegalStateEx(AQjmsError.java:471)
at oracle.jms.AQjmsSession.checkSessionStarted(AQjmsSession.java:4450)
at oracle.jms.AQjmsSession.getDBConnection(AQjmsSession.java:4392)
at oracle.jms.AQjmsSession.getAQOwner(AQjmsSession.java:6447)
at oracle.jms.AQjmsSession.createQueue(AQjmsSession.java:1387)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:386)
at com.sun.proxy.$Proxy73.createQueue(Unknown Source)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:84)
at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:58)
at org.springframework.jms.support.destination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:98)
at org.springframework.jms.core.JmsTemplate.access$200(JmsTemplate.java:90)
at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:583)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
... 33 more

How to reproduce the error

Application Configuration

Configuration of the connection pool:

<bean id="appDataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory" factory-method="getPoolDataSource"
    lazy-init="true">
    <property name="connectionFactoryClassName" value="oracle.jdbc.pool.OracleDataSource"/>
    <property name="user" value="${database.username}"/>
    <property name="password" value="${database.password}"/>
    <property name="URL" value="${database.url}"/>
    <property name="initialPoolSize" value="5"/>
    <property name="minPoolSize" value="2"/>
    <property name="maxPoolSize" value="20"/>
    <property name="connectionWaitTimeout" value="10"/>
    <property name="inactiveConnectionTimeout" value="180"/>
    <property name="timeToLiveConnectionTimeout" value="0"/>
    <property name="abandonedConnectionTimeout" value="3"/>
    <property name="maxConnectionReuseTime" value="3"/>
    <property name="maxConnectionReuseCount" value="0"/>
    <property name="validateConnectionOnBorrow" value="true"/>
    <property name="maxStatements" value="80"/>
    <property name="timeoutCheckInterval" value="3"/>
    <property name="connectionProperties">
        <props merge="default">
            <prop key="oracle.net.disableOob">true</prop>
            <prop key="oracle.net.CONNECT_TIMEOUT">10000</prop>
            <prop key="oracle.jdbc.ReadTimeout">12000</prop>
        </props>
    </property>
</bean>

For arguments sake I set the abandonedConnectionTimeout, maxConnectionReuseTime and timeoutCheckInterval to 3, in order to force the ConnectionPool to close all connections that haven't been used for a little while.

XML-Configuration of the CachingConnectionFactory and the JmsTemplate:

<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="sessionCacheSize" value="4"/>
    <property name="targetConnectionFactory">
        <bean class="com.example.jms.oracleaq.OracleAqConnectionFactoryBean">
            <property name="datasource" ref="appDataSource"/>
        </bean>
    </property>
</bean>

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="sessionAcknowledgeMode" value="0"/>
    <property name="sessionTransacted" value="true"/>
    <property name="deliveryPersistent" value="true"/>
    <property name="explicitQosEnabled" value="true"/>
    <property name="defaultDestinationName" value="SAMPLE_QUEUE"/>
</bean>

Implementation of OracleAqConnectionFactoryBean:

public class OracleAqConnectionFactoryBean implements FactoryBean<QueueConnectionFactory> {
    ...
    public QueueConnectionFactory getObject() throws Exception {
        return oracle.jms.AQjmsFactory.getQueueConnectionFactory(datasource);
    }
    ...
}

.h3 Test

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(value = { "/resources/spring/jpa.xml" })
public class SessionClosedTest {
    @Autowired
    private JmsTemplate jmsTemplate;
    
    @Test
    public void testRF242() throws Exception {
        MyBusinessObject someObj = new MyBusinessObject();
        this.jmsTemplate.convertAndSend(someObj);
        // Funny note: If you type Thread.sleep(15000); in the line below and remove this comment, Jira is actually not able to create the issue and the "Create Issue"-dialog completly breaks. On our own Jira we don't have this problem.
        Thread.sleep(15000);
        this.jmsTemplate.convertAndSend(someObj);
    }
}

The first call of convertAndSend works as expected. However the second call will fail with the exception at the beginning.

It's possible to get the test above to run successfully if you call CachingConnectionFactory#resetConnection after the sleep.

Spring-JMS will call CachingConnectionFactory#resetConnection on its own given any JmsException, but only for Message-Consumer. I couldn't find similar behaviour for Message-Producer.

Is this something that should be adressed by Spring? The easiest fix is probably to reset the connections in the JmsTemplate if a JmsException occurs.


Affects: 4.3.19

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Nov 20, 2018

Juergen Hoeller commented

This is a general issue with JMS resources: We can't easily inspect whether they're actually still open. In the case of a listener, we simply abort the receive attempt, propagate the exception (leading to a connection reset) and wait for the message to arrive again. In the case of a producer, we'd have to implicitly retry the send which is not trivial to do at this level (we'd have to react to specific state exceptions at specific times in order to not retry unnecessarily).

If there was an isOpen() method on Session or Connection, we could simply check it before we return a cached Session within CachingConnectionFactory itself. I guess we could call Connection.getClientID() or Connection.getMetaData() and assume it would throw an IllegalStateException if the connection has been closed in the meantime? Could you check whether that's the case with Oracle AQ for either of those two methods?

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Jan 7, 2019

Kevin Möchel commented

Sorry it took me so long to get back to this issue.

Unfortunately neither Connection.getClientID() nor Connection.getMetaData() have the desired behaviour. We tried a few other things to find out whether the connection is still alive or not but nothing gave the right answer.
Internally we bypass this error now by calling CachingConnectionFactory.resetConnection given any org.springframework.jms.IllegalStateException and then try to send the JMS message again. This is not optimal because we might call CachingConnectionFactory.resetConnection when it's not necessary (as you already mentioned), however the Exception is so rare that it doesn't matter for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.