Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock in in OracleConnectionDispenser.ProcessInstanceUp() #401

Closed
rauhs opened this issue Aug 2, 2024 · 4 comments
Closed

Deadlock in in OracleConnectionDispenser.ProcessInstanceUp() #401

rauhs opened this issue Aug 2, 2024 · 4 comments

Comments

@rauhs
Copy link

rauhs commented Aug 2, 2024

Hi,

we're at oracle.manageddataaccess 21.14.0 and just experienced a deadlock during parallel OracleConnection.Open(). The stacks of three threads:

win32u.dll!NtUserMsgWaitForMultipleObjectsEx() + 20 bytes	Unknown	No symbols loaded.
user32.dll!MsgWaitForMultipleObjectsEx() + 158 bytes	Unknown	No symbols loaded.
combase.dll!CoGetMarshalSizeMax() + 8409 bytes	Unknown	No symbols loaded.
combase.dll!WindowsCompareStringOrdinal() + 1765 bytes	Unknown	No symbols loaded.
combase.dll!CoWaitForMultipleHandles() + 118 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]


ntdll.dll!NtWaitForMultipleObjects() + 20 bytes	Unknown	No symbols loaded.
KERNELBASE.dll!WaitForMultipleObjectsEx() + 240 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]


ntdll.dll!NtWaitForMultipleObjects() + 20 bytes	Unknown	No symbols loaded.
KERNELBASE.dll!WaitForMultipleObjectsEx() + 240 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]

I also have a process dump (zipped around 300MB) at the deadlock if you want me to send it (on private channels).

Not reproducible.
@alexkeh
Copy link
Member

alexkeh commented Aug 2, 2024

Was this behavior observed after an ODP.NET upgrade, meaning the same code ran without a deadlock with the earlier ODP.NET version? If so, what was the version? Is this ODP.NET Core or managed ODP.NET?

Since this problem isn't reproducible on another system, I recommend opening up a service request with Oracle Support. Typically, we'll need to collect further diagnostics to determine the root cause. And the only way that would be possible is to run those diagnostics in the environment with the problem.

@rauhs
Copy link
Author

rauhs commented Aug 5, 2024

We've been on 21.14.0 since May 1st and had no issues since then.

This was only a one time thing. We have never seen it afterwards so I cannot offer more information than the memory dump I created when the deadlock was happening.

@alexkeh
Copy link
Member

alexkeh commented Aug 10, 2024

I still recommend the Oracle Support SR route if you want to get the dump analyzed. The Oracle Support team has the most experience with working with memory dumps.

@alexkeh
Copy link
Member

alexkeh commented Aug 14, 2024

Closing. The next step is to open an SR for Oracle Support to analyze the dump and collect more diagnostics.

@alexkeh alexkeh closed this as completed Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexkeh @rauhs