Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix connection manager deadlock for purgePolicy=FailingConnectionOnly #14657

Closed
gjwatts opened this issue Oct 22, 2020 · 0 comments · Fixed by #14668
Closed

Fix connection manager deadlock for purgePolicy=FailingConnectionOnly #14657

gjwatts opened this issue Oct 22, 2020 · 0 comments · Fixed by #14668
Assignees
Labels
release bug This bug is present in a released version of Open Liberty release:200012 team:Zombie Apocalypse

Comments

@gjwatts
Copy link
Member

gjwatts commented Oct 22, 2020

Describe the bug
Need to rework the com.ibm.ejs.j2c.PoolManager.validateConnections(ManagedConnectionFactory, boolean) method to prevent connection validation from occurring on a different thread which can lead to a deadlock when purgePolicy=FailingConnectionOnly.

javacore
---
MTHREADINFO      "Default Executor-thread-36,serviceName=executor" J9VMThread:0x0000000004633600, omrthread_t:0x00007FD95863A210, java/lang/Thread:0x000000004F28D520, state:B, prio=5
.....
3XMTHREADBLOCK     Blocked on: com/ibm/db2/jcc/am/dt@0x00000000D00BFC78 Owned by: "Default Executor-thread-14,serviceName=executor" (J9VMThread:0x00000000023D7600, java/lang/Thread:0x0000000040FE5148)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/db2/jcc/am/dt.close(dt.java:71)
4XESTACKTRACE                at com/ibm/ws/rsadapter/impl/WSRdbManagedConnectionImpl.destroy(WSRdbManagedConnectionImpl.java:2592)
4XESTACKTRACE                at com/ibm/ejs/j2c/MCWrapper.destroy(MCWrapper.java:1751)
4XESTACKTRACE                at com/ibm/ejs/j2c/FreePool.cleanupAndDestroyMCWrapper(FreePool.java:473)
4XESTACKTRACE                at com/ibm/ejs/j2c/FreePool.returnToFreePool(FreePool.java:273(Compiled Code))
4XESTACKTRACE                at com/ibm/ejs/j2c/PoolManager.release(PoolManager.java:951(Compiled Code))
4XESTACKTRACE                at com/ibm/db2/jcc/am/SqlSyntaxErrorException.getMessage(SqlSyntaxErrorException.java:54)
4XESTACKTRACE                at com/ibm/ws/rsadapter/AdapterUtil.translateSQLException(AdapterUtil.java:774)
4XESTACKTRACE                at com/ibm/ws/rsadapter/impl/WSRdbManagedConnectionImpl.validate(WSRdbManagedConnectionImpl.java:4286)
4XESTACKTRACE                at com/ibm/ws/rsadapter/impl/WSManagedConnectionFactoryImpl.getInvalidConnections(WSManagedConnectionFactoryImpl.java:1066)
4XESTACKTRACE                at com/ibm/ejs/j2c/PoolManager.validateConnections(PoolManager.java:473)
---
3XMTHREADINFO      "Default Executor-thread-14,serviceName=executor" J9VMThread:0x00000000023D7600, omrthread_t:0x00007FD9280C97A0, java/lang/Thread:0x0000000040FE5148, state:B, prio=5
.....
3XMTHREADBLOCK     Blocked on: com/ibm/db2/jcc/t4/T4XAConnection@0x00000000D00BF5A0 Owned by: "Default Executor-thread-36,serviceName=executor" (J9VMThread:0x0000000004633600, java/lang/Thread:0x000000004F28D520)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/db2/jcc/am/Connection.clearWarnings(Connection.java:1871(Compiled Code))
5XESTACKTRACE                   (entered lock: com/ibm/db2/jcc/t4/T4XAConnection@0x00000000D00BF5A0, entry count: 1)
4XESTACKTRACE                at com/ibm/db2/jcc/am/dt.clearWarnings(dt.java:204(Compiled Code))
5XESTACKTRACE                   (entered lock: com/ibm/db2/jcc/am/dt@0x00000000D00BFC78, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.initialize(WSJdbcConnection.java:242(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:154(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:112(Compiled Code))

Steps to Reproduce
Only happens for purgePolicy=FailingConnectionOnly on fast systems

Expected behavior
Prevent validation from occurring on a connection that may be used on another thread, potentially causing deadlock.

Diagnostic information:
WAS Liberty core 19.0.0.12
JAVA SDK 8.0.6
@rhel 7.4

@gjwatts gjwatts added bug This bug is not present in a released version of Open Liberty team:Zombie Apocalypse labels Oct 22, 2020
@gjwatts gjwatts self-assigned this Oct 22, 2020
@gjwatts gjwatts added release bug This bug is present in a released version of Open Liberty and removed bug This bug is not present in a released version of Open Liberty labels Oct 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release bug This bug is present in a released version of Open Liberty release:200012 team:Zombie Apocalypse
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants