Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deadlock on jvm start #827

Closed
bradrust opened this issue Aug 6, 2020 · 3 comments
Closed

deadlock on jvm start #827

bradrust opened this issue Aug 6, 2020 · 3 comments
Assignees
@dpsoft
Labels
Bug

Comments

@bradrust
Copy link

bradrust commented Aug 6, 2020

Occasionally my instances fail to start. They hang right after the kanela banner. I am including a jstack output which identifies a deadlock.

kanela-agent: 1.0.6
kamon depedencies/bundles: 2.1.4

stdout

`OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

| |/ / | | \ \ \
| ' / __ _ _ __ | | __ _ \ \ \
| < / | '_ \ / _ \ |/ _ | ) ) ) )
| . \ (| | | | | __/ | (| | / / / /
||__,|| ||___||_,| ///_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.6)`

jstack output

`2020-08-06 18:24:15
Full thread dump OpenJDK 64-Bit Server VM (14.0.2+12 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f29e4002010, length=12, elements={
0x00007f2a3002e800, 0x00007f2a30ac7800, 0x00007f2a30ac9800, 0x00007f2a30acf800,
0x00007f2a30ad1800, 0x00007f2a30ad3800, 0x00007f2a30ad5800, 0x00007f2a30ad7800,
0x00007f2a30b11000, 0x00007f2a3178a800, 0x00007f2a31b37800, 0x00007f29e4001000
}

"main" #1 prio=5 os_prio=0 cpu=1637.37ms elapsed=44.96s tid=0x00007f2a3002e800 nid=0x45 waiting on condition [0x00007f2a39920000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@14.0.2/Native Method)
- parking to wait for <0x00000007db14ad28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@14.0.2/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:714)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:937)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@14.0.2/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@14.0.2/ReentrantLock.java:322)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.offer(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1100)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1127)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(java.base@14.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(java.base@14.0.2/ScheduledThreadPoolExecutor.java:342)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(java.base@14.0.2/ScheduledThreadPoolExecutor.java:562)
at java.util.concurrent.Executors$DelegatedScheduledExecutorService.schedule(java.base@14.0.2/Executors.java:779)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.scheduleEntry(ExpiringMap.java:1361)
- locked <0x00000007d68a98c0> (a kanela.agent.libs.net.jodah.expiringmap.ExpiringMap$ExpiringEntry)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.resetEntry(ExpiringMap.java:1304)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.get(ExpiringMap.java:747)
at kanela.agent.cache.PoolStrategyCache.locate(PoolStrategyCache.java:56)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$PoolStrategy$WithTypePoolCache.typePool(AgentBuilder.java:2764)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10261)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10722)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10660)
at java.security.AccessController.executePrivileged(java.base@14.0.2/AccessController.java:753)
at java.security.AccessController.doPrivileged(java.base@14.0.2/AccessController.java:391)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10219)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at sun.instrument.TransformerManager.transform(java.instrument@14.0.2/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(java.instrument@14.0.2/InstrumentationImpl.java:563)
at java.lang.ClassLoader.defineClass1(java.base@14.0.2/Native Method)
at java.lang.ClassLoader.defineClass(java.base@14.0.2/ClassLoader.java:1017)
at java.security.SecureClassLoader.defineClass(java.base@14.0.2/SecureClassLoader.java:151)
at jdk.internal.loader.BuiltinClassLoader.defineClass(java.base@14.0.2/BuiltinClassLoader.java:821)
at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@14.0.2/BuiltinClassLoader.java:719)
at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@14.0.2/BuiltinClassLoader.java:642)
- locked <0x00000007d68d6f18> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@14.0.2/BuiltinClassLoader.java:600)
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@14.0.2/ClassLoaders.java:178)
at java.lang.ClassLoader.loadClass(java.base@14.0.2/ClassLoader.java:522)
at kamon.instrumentation.akka.remote.MessageBufferInstrumentation.(MessageBufferInstrumentation.scala:15)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@14.0.2/Native Method)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@14.0.2/NativeConstructorAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@14.0.2/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@14.0.2/Constructor.java:500)
at java.lang.reflect.ReflectAccess.newInstance(java.base@14.0.2/ReflectAccess.java:124)
at jdk.internal.reflect.ReflectionFactory.newInstance(java.base@14.0.2/ReflectionFactory.java:346)
at java.lang.Class.newInstance(java.base@14.0.2/Class.java:604)
at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
at kanela.agent.InstrumentationLoader$$Lambda$149/0x0000000800bf7040.apply(Unknown Source)
at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
at kanela.agent.InstrumentationLoader$$Lambda$148/0x0000000800bf6c40.apply(Unknown Source)
at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:898)
at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
at kanela.agent.InstrumentationLoader$$Lambda$146/0x0000000800bf6440.apply(Unknown Source)
at kanela.agent.libs.io.vavr.collection.List.map(List.java:1038)
at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
at kanela.agent.Kanela$$Lambda$42/0x0000000800b81440.accept(Unknown Source)
at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
at kanela.agent.Kanela$$Lambda$14/0x0000000800b6fc40.run(Unknown Source)
at kanela.agent.util.Execution.timed(Execution.java:32)
at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
at kanela.agent.Kanela.start(Kanela.java:67)
at kanela.agent.Kanela.premain(Kanela.java:47)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@14.0.2/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@14.0.2/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@14.0.2/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@14.0.2/Method.java:564)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(java.instrument@14.0.2/InstrumentationImpl.java:513)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(java.instrument@14.0.2/InstrumentationImpl.java:525)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.25ms elapsed=44.91s tid=0x00007f2a30ac7800 nid=0x4c waiting on condition [0x00007f2a103ae000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@14.0.2/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@14.0.2/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@14.0.2/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.34ms elapsed=44.91s tid=0x00007f2a30ac9800 nid=0x4d in Object.wait() [0x00007f2a1036d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@14.0.2/Native Method)
- waiting on <0x00000007dc3027f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@14.0.2/ReferenceQueue.java:155)
- locked <0x00000007dc3027f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@14.0.2/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@14.0.2/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.46ms elapsed=44.91s tid=0x00007f2a30acf800 nid=0x4e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.26ms elapsed=44.91s tid=0x00007f2a30ad1800 nid=0x4f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=1336.62ms elapsed=44.91s tid=0x00007f2a30ad3800 nid=0x50 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task

"C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=786.11ms elapsed=44.91s tid=0x00007f2a30ad5800 nid=0x51 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task

"Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=13.25ms elapsed=44.91s tid=0x00007f2a30ad7800 nid=0x52 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=0.33ms elapsed=44.87s tid=0x00007f2a30b11000 nid=0x53 in Object.wait() [0x00007f2a01ebc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@14.0.2/Native Method)
- waiting on <0x00000007dc35d9a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@14.0.2/ReferenceQueue.java:155)
- locked <0x00000007dc35d9a0> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@14.0.2/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@14.0.2/Thread.java:832)
at jdk.internal.misc.InnocuousThread.run(java.base@14.0.2/InnocuousThread.java:134)

"tinylog-WritingThread" #12 prio=3 os_prio=0 cpu=188.64ms elapsed=43.93s tid=0x00007f2a3178a800 nid=0x56 sleeping [0x00007f2a016fe000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@14.0.2/Native Method)
at kanela.agent.libs.org.pmw.tinylog.WritingThread.run(Unknown Source)

"ExpiringMap-Expirer" #13 daemon prio=5 os_prio=0 cpu=2.41ms elapsed=43.25s tid=0x00007f2a31b37800 nid=0x57 waiting on condition [0x00007f2a0173d000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@14.0.2/Native Method)
- parking to wait for <0x00000007d7c6aed8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@14.0.2/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:714)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@14.0.2/AbstractQueuedSynchronizer.java:1026)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@14.0.2/ReentrantReadWriteLock.java:738)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.getEntry(ExpiringMap.java:1243)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.get(ExpiringMap.java:742)
at kanela.agent.cache.PoolStrategyCache.locate(PoolStrategyCache.java:56)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$PoolStrategy$WithTypePoolCache.typePool(AgentBuilder.java:2764)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10261)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10722)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10660)
at java.security.AccessController.executePrivileged(java.base@14.0.2/AccessController.java:753)
at java.security.AccessController.doPrivileged(java.base@14.0.2/AccessController.java:391)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10219)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at sun.instrument.TransformerManager.transform(java.instrument@14.0.2/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(java.instrument@14.0.2/InstrumentationImpl.java:563)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@14.0.2/AbstractQueuedSynchronizer.java:1650)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@14.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@14.0.2/ThreadPoolExecutor.java:1056)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@14.0.2/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@14.0.2/ThreadPoolExecutor.java:630)
at java.lang.Thread.run(java.base@14.0.2/Thread.java:832)

"Attach Listener" #14 daemon prio=9 os_prio=0 cpu=0.39ms elapsed=0.10s tid=0x00007f29e4001000 nid=0x74 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=6.60ms elapsed=44.91s tid=0x00007f2a30ac4800 nid=0x4b runnable

"GC Thread#0" os_prio=0 cpu=3.67ms elapsed=44.96s tid=0x00007f2a30088800 nid=0x46 runnable

"G1 Main Marker" os_prio=0 cpu=0.09ms elapsed=44.96s tid=0x00007f2a300a5800 nid=0x47 runnable

"G1 Conc#0" os_prio=0 cpu=0.05ms elapsed=44.96s tid=0x00007f2a300a7000 nid=0x48 runnable

"G1 Refine#0" os_prio=0 cpu=0.12ms elapsed=44.92s tid=0x00007f2a30a3e800 nid=0x49 runnable

"G1 Young RemSet Sampling" os_prio=0 cpu=9.19ms elapsed=44.92s tid=0x00007f2a30a40000 nid=0x4a runnable
JNI global refs: 9, weak refs: 0

Found one Java-level deadlock:

"main":
waiting for ownable synchronizer 0x00000007db14ad28, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "ExpiringMap-Expirer"

"ExpiringMap-Expirer":
waiting for ownable synchronizer 0x00000007d7c6aed8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "main"

Java stack information for the threads listed above:

"main":
at jdk.internal.misc.Unsafe.park(java.base@14.0.2/Native Method)
- parking to wait for <0x00000007db14ad28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@14.0.2/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:714)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:937)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@14.0.2/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@14.0.2/ReentrantLock.java:322)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.offer(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1100)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1127)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(java.base@14.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(java.base@14.0.2/ScheduledThreadPoolExecutor.java:342)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(java.base@14.0.2/ScheduledThreadPoolExecutor.java:562)
at java.util.concurrent.Executors$DelegatedScheduledExecutorService.schedule(java.base@14.0.2/Executors.java:779)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.scheduleEntry(ExpiringMap.java:1361)
- locked <0x00000007d68a98c0> (a kanela.agent.libs.net.jodah.expiringmap.ExpiringMap$ExpiringEntry)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.resetEntry(ExpiringMap.java:1304)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.get(ExpiringMap.java:747)
at kanela.agent.cache.PoolStrategyCache.locate(PoolStrategyCache.java:56)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$PoolStrategy$WithTypePoolCache.typePool(AgentBuilder.java:2764)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10261)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10722)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10660)
at java.security.AccessController.executePrivileged(java.base@14.0.2/AccessController.java:753)
at java.security.AccessController.doPrivileged(java.base@14.0.2/AccessController.java:391)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10219)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at sun.instrument.TransformerManager.transform(java.instrument@14.0.2/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(java.instrument@14.0.2/InstrumentationImpl.java:563)
at java.lang.ClassLoader.defineClass1(java.base@14.0.2/Native Method)
at java.lang.ClassLoader.defineClass(java.base@14.0.2/ClassLoader.java:1017)
at java.security.SecureClassLoader.defineClass(java.base@14.0.2/SecureClassLoader.java:151)
at jdk.internal.loader.BuiltinClassLoader.defineClass(java.base@14.0.2/BuiltinClassLoader.java:821)
at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@14.0.2/BuiltinClassLoader.java:719)
at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@14.0.2/BuiltinClassLoader.java:642)
- locked <0x00000007d68d6f18> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@14.0.2/BuiltinClassLoader.java:600)
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@14.0.2/ClassLoaders.java:178)
at java.lang.ClassLoader.loadClass(java.base@14.0.2/ClassLoader.java:522)
at kamon.instrumentation.akka.remote.MessageBufferInstrumentation.(MessageBufferInstrumentation.scala:15)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@14.0.2/Native Method)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@14.0.2/NativeConstructorAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@14.0.2/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@14.0.2/Constructor.java:500)
at java.lang.reflect.ReflectAccess.newInstance(java.base@14.0.2/ReflectAccess.java:124)
at jdk.internal.reflect.ReflectionFactory.newInstance(java.base@14.0.2/ReflectionFactory.java:346)
at java.lang.Class.newInstance(java.base@14.0.2/Class.java:604)
at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
at kanela.agent.InstrumentationLoader$$Lambda$149/0x0000000800bf7040.apply(Unknown Source)
at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
at kanela.agent.InstrumentationLoader$$Lambda$148/0x0000000800bf6c40.apply(Unknown Source)
at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:898)
at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
at kanela.agent.InstrumentationLoader$$Lambda$146/0x0000000800bf6440.apply(Unknown Source)
at kanela.agent.libs.io.vavr.collection.List.map(List.java:1038)
at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
at kanela.agent.Kanela$$Lambda$42/0x0000000800b81440.accept(Unknown Source)
at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
at kanela.agent.Kanela$$Lambda$14/0x0000000800b6fc40.run(Unknown Source)
at kanela.agent.util.Execution.timed(Execution.java:32)
at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
at kanela.agent.Kanela.start(Kanela.java:67)
at kanela.agent.Kanela.premain(Kanela.java:47)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@14.0.2/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@14.0.2/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@14.0.2/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@14.0.2/Method.java:564)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(java.instrument@14.0.2/InstrumentationImpl.java:513)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(java.instrument@14.0.2/InstrumentationImpl.java:525)
"ExpiringMap-Expirer":
at jdk.internal.misc.Unsafe.park(java.base@14.0.2/Native Method)
- parking to wait for <0x00000007d7c6aed8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@14.0.2/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@14.0.2/AbstractQueuedSynchronizer.java:714)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@14.0.2/AbstractQueuedSynchronizer.java:1026)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@14.0.2/ReentrantReadWriteLock.java:738)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.getEntry(ExpiringMap.java:1243)
at kanela.agent.libs.net.jodah.expiringmap.ExpiringMap.get(ExpiringMap.java:742)
at kanela.agent.cache.PoolStrategyCache.locate(PoolStrategyCache.java:56)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$PoolStrategy$WithTypePoolCache.typePool(AgentBuilder.java:2764)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10261)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10722)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10660)
at java.security.AccessController.executePrivileged(java.base@14.0.2/AccessController.java:753)
at java.security.AccessController.doPrivileged(java.base@14.0.2/AccessController.java:391)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10219)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at sun.instrument.TransformerManager.transform(java.instrument@14.0.2/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(java.instrument@14.0.2/InstrumentationImpl.java:563)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@14.0.2/AbstractQueuedSynchronizer.java:1650)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@14.0.2/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@14.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@14.0.2/ThreadPoolExecutor.java:1056)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@14.0.2/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@14.0.2/ThreadPoolExecutor.java:630)
at java.lang.Thread.run(java.base@14.0.2/Thread.java:832)

Found 1 deadlock.`
@dpsoft dpsoft self-assigned this Aug 7, 2020
@dpsoft dpsoft added the Bug label Aug 10, 2020
@bradrust
Copy link
Author

bradrust commented Sep 3, 2020

Would this be served better to be moved to the kanela github issues?

I am facing this issue quite frequently with production images/pods

@dpsoft
Copy link
Contributor

dpsoft commented Sep 3, 2020

@bradrust yes please! I'll take this issue ASAP

@bradrust bradrust mentioned this issue Sep 3, 2020
Closed
@SimunKaracic SimunKaracic mentioned this issue Sep 25, 2020
Merged
@SimunKaracic
Copy link
Contributor

Version 2.1.7 contains the kanela-agent with the fix for this bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dpsoft dpsoft

Labels
Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bradrust @dpsoft @SimunKaracic