New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blocked thread on java.lang.ClassLoader.findBootstrapClass #1155

Closed
ocafebabe opened this Issue May 16, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@ocafebabe

ocafebabe commented May 16, 2018

Environment

HikariCP version: 2.4.1
JDK version     : 1.8.0_131
Database        : Oracle
Driver version  : 12.1.0.2.0

Hi,

I have been experiencing weird random freezes for some time in my application and after analyzing the thread dump I narrowed it down to a class loading call made by Hikari during its initialization but I don't understand what could be causing such an issue:

Thread 87375: (state = BLOCKED)
 - java.lang.ClassLoader.findBootstrapClass(java.lang.String) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ClassLoader.findBootstrapClassOrNull(java.lang.String) @bci=12, line=1015 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=48, line=413 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=38, line=411 (Compiled frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=81, line=335 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=357 (Compiled frame)
 - com.zaxxer.hikari.util.Sequence$Factory.create() @bci=7, line=53 (Interpreted frame)
 - com.zaxxer.hikari.util.QueuedSequenceSynchronizer.<init>() @bci=18, line=60 (Interpreted frame)
 - com.zaxxer.hikari.util.ConcurrentBag.<init>(com.zaxxer.hikari.util.ConcurrentBag$IBagStateListener) @bci=33, line=95 (Interpreted frame)
 - com.zaxxer.hikari.pool.HikariPool.<init>(com.zaxxer.hikari.HikariConfig) @bci=86, line=116 (Interpreted frame)
 - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=100, line=96 (Interpreted frame)

Based on the thread dump and the code here: https://github.com/brettwooldridge/HikariCP/blob/HikariCP-2.4.1/src/main/java/com/zaxxer/hikari/util/Sequence.java, line 53 seems to be the culprit, but again I don't understand why it would randomely block at this stage!

At first, I thought I had the same issue as #418 (freezes on a call to getConnection() and same Hikari version) but I'm not 100% sure as our stacktraces differ a bit...

I know that my Hikari version is old but it's not an easy task for me to deploy a new version and prior of doing that I would like to get a confirmation that it's a known issue...

Thanks

@brettwooldridge

This comment has been minimized.

Owner

brettwooldridge commented May 19, 2018

@ocafebabe FIrst of all, I of course would suggest upgrading HikariCP to the latest version. However, this issue appears to be outside of HikariCP.

You do not show a full stacktrace, but I suspect there is some kind of recursion going on where HikariCP is being initialized by a classloader due to a static initializer in your code. The JDK classloaders themselves have locks, but should be reentrant. But clearly there is some kind of race condition or inverted lock acquisition going on in order to encounter this kind of blocking.

It may be hard to track down, so if HikariCP is being initialized in a static initializer of a class, the simplest solution may be to move that to a lazy singleton initialization pattern instead. Otherwise, you could try upgrading the JDK. It is possible that if it was a JDK classloader issue, it may have been addressed -- but knowing that with certainty would require reading all of the release notes (bug manifest) of the versions since 1.8.0_131.

@ocafebabe

This comment has been minimized.

ocafebabe commented May 19, 2018

Hi @brettwooldridge,

Thanks a lot for your reply!

Here's the full stacktrace in case it might help diagnose the issue (as you can see, Hikari is being initialized by Spring and then one connection is opened by Hibernate) :

Thread 87375: (state = BLOCKED)
 - java.lang.ClassLoader.findBootstrapClass(java.lang.String) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ClassLoader.findBootstrapClassOrNull(java.lang.String) @bci=12, line=1015 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=48, line=413 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=38, line=411 (Compiled frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=81, line=335 (Compiled frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=357 (Compiled frame)
 - com.zaxxer.hikari.util.Sequence$Factory.create() @bci=7, line=53 (Interpreted frame)
 - com.zaxxer.hikari.util.QueuedSequenceSynchronizer.<init>() @bci=18, line=60 (Interpreted frame)
 - com.zaxxer.hikari.util.ConcurrentBag.<init>(com.zaxxer.hikari.util.ConcurrentBag$IBagStateListener) @bci=33, line=95 (Interpreted frame)
 - com.zaxxer.hikari.pool.HikariPool.<init>(com.zaxxer.hikari.HikariConfig) @bci=86, line=116 (Interpreted frame)
 - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=100, line=96 (Interpreted frame)
 - org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection() @bci=48, line=139 (Interpreted frame)
 - org.hibernate.engine.jdbc.internal.JdbcServicesImpl$ConnectionProviderJdbcConnectionAccess.obtainConnection() @bci=4, line=279 (Interpreted frame)
 - org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(java.util.Map) @bci=89, line=124 (Interpreted frame)
 - org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(org.hibernate.service.spi.ServiceBinding) @bci=24, line=111 (Interpreted frame)
 - org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(org.hibernate.service.spi.ServiceBinding) @bci=55, line=234 (Interpreted frame)
 - org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(java.lang.Class) @bci=30, line=206 (Compiled frame)
 - org.hibernate.cfg.Configuration.buildTypeRegistrations(org.hibernate.service.ServiceRegistry) @bci=13, line=1885 (Interpreted frame)
 - org.hibernate.cfg.Configuration.buildSessionFactory(org.hibernate.service.ServiceRegistry) @bci=17, line=1843 (Interpreted frame)
 - org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform() @bci=30, line=850 (Interpreted frame)
 - org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform() @bci=1, line=843 (Interpreted frame)
 - org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl$Work) @bci=27, line=397 (Interpreted frame)
 - org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build() @bci=35, line=842 (Interpreted frame)
 - org.hibernate.jpa.HibernatePersistenceProvider.createContainerEntityManagerFactory(javax.persistence.spi.PersistenceUnitInfo, java.util.Map) @bci=20, line=152 (Interpreted frame)
 - org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory() @bci=235, line=336 (Interpreted frame)
 - org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet() @bci=217, line=318 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(java.lang.String, java.lang.Object, org.springframework.beans.factory.support.RootBeanDefinition) @bci=109, line=1612 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(java.lang.String, java.lang.Object, org.springframework.beans.factory.support.RootBeanDefinition) @bci=61, line=1549 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition, java.lang.Object[]) @bci=235, line=539 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition, java.lang.Object[]) @bci=124, line=475 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject() @bci=16, line=304 (Interpreted frame)
 - org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(java.lang.String, org.springframework.beans.factory.ObjectFactory) @bci=129, line=228 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(java.lang.String, java.lang.Class, java.lang.Object[], boolean) @bci=368, line=300 (Interpreted frame)
 - org.springframework.beans.factory.support.AbstractBeanFactory.getBean(java.lang.String) @bci=5, line=195 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.getBean(java.lang.String) @bci=9, line=973 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(org.springframework.beans.factory.config.ConfigurableListableBeanFactory) @bci=79, line=750 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.refresh() @bci=54, line=482 (Interpreted frame)
 - org.springframework.context.support.FileSystemXmlApplicationContext.<init>(java.lang.String[], boolean, org.springframework.context.ApplicationContext) @bci=15, line=140 (Interpreted frame)
 - org.springframework.context.support.FileSystemXmlApplicationContext.<init>(java.lang.String) @bci=11, line=84 (Interpreted frame)
 - com.expretio.appia.job.SpawnJobResourceProvider.readObject(java.io.ObjectInputStream) @bci=20, line=67 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame)
 - java.io.ObjectStreamClass.invokeReadObject(java.lang.Object, java.io.ObjectInputStream) @bci=24, line=1058 (Interpreted frame)
 - java.io.ObjectInputStream.readSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=119, line=2136 (Interpreted frame)
 - java.io.ObjectInputStream.readOrdinaryObject(boolean) @bci=181, line=2027 (Interpreted frame)
 - java.io.ObjectInputStream.readObject0(boolean) @bci=401, line=1535 (Interpreted frame)
 - java.io.ObjectInputStream.defaultReadFields(java.lang.Object, java.io.ObjectStreamClass) @bci=150, line=2245 (Interpreted frame)
 - java.io.ObjectInputStream.readSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=298, line=2169 (Interpreted frame)
 - java.io.ObjectInputStream.readOrdinaryObject(boolean) @bci=181, line=2027 (Interpreted frame)
 - java.io.ObjectInputStream.readObject0(boolean) @bci=401, line=1535 (Interpreted frame)
 - java.io.ObjectInputStream.readObject() @bci=19, line=422 (Interpreted frame)
 - org.apache.commons.lang3.SerializationUtils.deserialize(java.io.InputStream) @bci=26, line=224 (Interpreted frame)
 - com.expretio.jobmanager.executor.SpawnJobExecutor$SpawnJob.main(java.lang.String[]) @bci=12, line=294 (Interpreted frame)

I'll definitely try to upgrade Hikari to the latest version and see if it changes anything!

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment