oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser on heap dump

[dhuygu]

I have an application deployed to weblogic 12c R2(12.2.1). jdk is 1.8. I've placed ojdbc6(11.2.0.4) and than also ojdbc7(12.1.0.2). OutOfMemoryError error is thrown after a few redeployments. jconsole shows that "oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" is not killing until the server is restarted. I have created a new class that extends ClassLoaderLeakPreventor to solve my memory leak problem but still having leak problem. What can be the problem?

[mjiderhamn]

Hi @dhuygu. The oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser should already be adressed as per issue #23. If you are still having issues using the latest version of this library, could you please follow the instructions in this blog post and either provide screenshots or make the dump available?

[dhuygu]

​Hi Mattias,
​

I am using the latest version of this library. If I dont create a new
class that extends ClassLoaderLeakPreventor an exception is thrown because
of log4j2 and lmax disruptor library.
So I extended and added below code not to get exception:

if (LogManager.getContext() != null) { try { AsyncLoggerContext logContext
= (AsyncLoggerContext) LogManager.getContext(); logContext.stop(); } catch
(Exception e) { }

exception:
​
Feb 13, 2016 5:00:28 PM com.lmax.disruptor.FatalExceptionHandler
handleEventException

SEVERE: Exception processing: 762 org.apache.logging.log4j.core.async.
RingBufferLogEvent@dd477f3 java.lang.InterruptedException at java.util.
concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.
concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:2048) at com.lmax.disruptor.
BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) at com.lmax.
disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:
55) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:
123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)​

<13-Feb-2016 17:00:28 o'clock EET>
<com.lmax.disruptor.FatalExceptionHandler> <Exception
processing: 762
org.apache.logging.log4j.core.async.RingBufferLogEvent@dd477f3 java.lang.
InterruptedException at java.util.concurrent.locks.
AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(
AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.
AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:2048) at com.lmax.disruptor.
BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) at com.lmax.
disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java
:55) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.
java:123) Truncated. see log file for complete stacktrace>

With the overrided version no exception but still having memory leak.
Before deployment no "oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser"
thread in threads tab. After undeployment still this thread is alive:

​[image: Inline image 1]​

​On heap dump I found extra leaks also:

[image: Inline image 2]​

[image: Inline image 3]

[image: Inline image 4]

All these screenshots are from the same heap dump. I am unable to
understand what the exact problem is. jconsole and dumps show different
things. Spring version is: 4.2.4.RELEASE. log4j2 version is 2.3. If ı use
log4j 2.5 Log4jThread classes stay alive after undeployment. Could you
please help me? Thanks

​

[mjiderhamn]

I have created separate issues for log4j2 and LMAX Disruptor. It would be helpful if you could supply your log4j configuration and some info on how you use LMAX Disruptor in the respective issues.

As for this issue, unfortunately your heap dump images did not make it into the GitHub issue. Could you give it another try to add them to this issue?

[dhuygu]

Hi Mattias,
My log4j2 version is 2.3 (I want to upgrade 2.5 but there are some Log4JThread leaks in this version )
I am using log4j2 asynchronus loggers. Asynchronous Loggers internally use the lmax disruptor.
I added log4j2.component.properties file to say log4j2 to use AsyncLoggerContextSelector:
Log4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector.
log4j2.xml file all my loggers are AsyncLogger and appenders are RollingRandomAccessFile. If you need additional data please ask.

My memory leak problem:
Before deployment no "oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser"
thread in threads tab. After undeployment or few deployment/undeployment still this thread is alive: (I hope images are ok)

In ClassLoaderLeakPreventor class deregisterJdbcDrivers method not deregister Oracle driver.
In order to try I overrided this method and forced to deregister not ignoring this classLoaders driver or not by using directly "DriverManager.deregisterDriver(driver);" I have tried with different ojdbc driver versions:
ojdbc6 driver 11.2.0.4
ojdbc6 12.1.0.2
ojdbc7 12.1.0.2

But BlockReleaser is still exist.

Thanks

[dhuygu]

Also I want to provide some extra info:
If I dont use ClassLoaderLeakPreventor there are two "oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" references:

If U add ClassLoaderLeakPreventor as listener:

When I debug deregisterJdbcDrivers method isLoadedInWebApplication(driver) part returns false. The BlockReleaser that is not kiled is loaded with PolicyClassLoader.
if(isLoadedInWebApplication(driver)) // Should be true for all returned by DriverManager.getDrivers()
driversToDeregister.add(driver);

[mjiderhamn]

Regarding log4j2, have you tried registering a ShutdownCallbackRegistry as per http://stackoverflow.com/a/28835409 ?

If I use the shutdown mechanism in a test case, I cannot reproduce the problem.

[mjiderhamn]

Regarding BlockReleaser, are you seeing createAccessControlContext not granted and AccessControlContext could not be created via reflection in your logs...?

In your snapshot, can you list outgoing references from the AccessControlContext, so I can see its DomainCombiner and ProtectionDomains?

[nikolaikopernik]

same problem, guys
after loading ojdbc7.jar in a web-based app this thread runs forever and blocks the jar file. So I cannot redeploy my app

"oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" #31 daemon prio=4 os_prio=-1 tid=0x000000002415a800 nid=0x24a8 in Object.wait() [0x000000002886e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
    - locked <0x000000076bd52728> (a oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)

   Locked ownable synchronizers:
    - None

[mjiderhamn]

@nikolaikopernik, maybe you could answer my questions from February 17?

[nikolaikopernik]

After undeploy web app I still have BlockReleaser:

@mjiderhamn here is my AccessControlContext, as you asked

[serba]

Note: I do not use classloader-leak-prevention yet, still investigating if using it in my app would solve my problems.

I'm having similar problem with ojdbc7 driver's oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser holding a reference to a custom class loader and causing permgen/metaspace memory leak. It's weird that I can't forcibly stop that thread by running Thread.stop or even Thread.stop0 methods - no exceptions/throwables, but thread is not getting stopped as a result (observed with YourKit profiler) and thus no classes getting gc-ed. I'm wondering if you've seen something similar or have any ideas why that can happen.

[gkorland]

We have seen the same issue with ojdbc10 https://stackoverflow.com/questions/62657400/oracle-leaks-threads-on-when-closing-hibernate-sessions-with-oracle-jdbc-driver

[quaff]

@gkorland Replied in SO thread.

[gkorland]

@quaff Thanks! but I was looking for a more generic solution, the users might use another DB or another driver. Is that the only way to force Oracle driver to close these threads?

[quaff]

@gkorland Oracle close other resources except oracle.jdbc.driver.HAManager.timer while deregister driver, it's a bug, but I don't know where to file an issue.
You could get the timer instance by reflection then call cancel method to close it.

		try {
			if (ClassUtils.isPresent("oracle.jdbc.OracleDriver", cl)) {
				if (Thread.getAllStackTraces().keySet().stream().anyMatch(t -> t.getName().equals("InterruptTimer"))) {
					// oracle driver started
					try {
						String className = "oracle.jdbc.driver.NoSupportHAManager";
						Object noSupportHAManager = Class.forName(className).getMethod("getInstance").invoke(null);
						if (noSupportHAManager != null) {
							Field f = Class.forName("oracle.jdbc.driver.HAManager").getDeclaredField("timer");
							f.setAccessible(true);
							Timer timer = (Timer) f.get(noSupportHAManager);
							if (timer != null)
								timer.cancel();
						}
					} catch (NoClassDefFoundError err) {
						// java.lang.NoClassDefFoundError: Loracle/simplefan/FanManager;
					}
				}
			}
		} catch (Throwable e) {
			e.printStackTrace();
		}