Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock between DefaultListableBeanFactory and DefaultSingletonBeanRegistry, perhaps due to lazily-instantiated aspect [SPR-7718] #12374

Closed
spring-issuemaster opened this issue Nov 3, 2010 · 8 comments
Assignees
Milestone

Comments

@spring-issuemaster
Copy link
Collaborator

@spring-issuemaster spring-issuemaster commented Nov 3, 2010

John Lee opened SPR-7718 and commented

Deploying EAR on Weblogic 10.0. Root XmlWebApplicationContext ends up w/ about 300 bean definitions from 3 WARs, which takes long enough to instantiate to see this behavior. Each WAR publishes web services specified in Spring contexts as Apache CXF endpoints. Service implementations are annotated so that an aspect can monitor traffic to all services. The traffic is reported to a monitoring process (deployed separately, but also a CXF webservice) at set intervals by Quartz jobs. The Quartz jobs are not wired via Spring integration; the Spring context just declares an org.quartz.Scheduler which gets injected into "updater" classes that add associated jobs right after instantiation (@PostConstruct).

This architecture has worked well for over a year, but we recently added a JAX-WS Handler to the traffic reporting, and this seemed to affect the timing of the context initialization enough that this deadlock started appearing (and it happens everytime without fail, usually for the same bean names as viewed in debugger, although the same EAR deployed on JBoss 4.2 works fine; different classloader architecture).

What it looks like is that one of the Quartz jobs triggers while Spring is in the process of pre-instantiating singletons, and an aspect on the traffic reporting client (picked up via <component-scan/> and annotated w/ AspectJ @Aspect and Spring @Component, pointcuts defined AspectJ-style) required for the Quartz job is lazily instantiated for some reason (a guess, from the LazySingletonAspectInstanceFactoryDecorator.getAspectInstance call in the stack trace), leading to the deadlock. Tweaking the triggers so that the start date is some time after instantiation, e.g. 30 sec., is my current workaround, but I thought that if this is a bug in Spring (didn't see any other reports of this scenario in the forums or in general web search), it's worth fixing.

The last thing written to the log is:

[many other services being bound to endpoints...]

16:09:24,496 INFO ReflectionServiceFactoryBean:437 - Creating Service {http://services.mycompany.com/diagnostics}Diagnostics from class com.mycompany.services.Diagnostics
16:09:26,275 INFO QuartzScheduler:461 - Scheduler QuartzScheduler_$_NON_CLUSTERED started.
16:09:26,275 INFO PeriodicalUpdater:73 - ConfigurationManagerUpdater scheduled with interval 5000 ms.
16:09:29,268 INFO ServerImpl:91 - Setting the server's publish address to be /Diagnostics

Here's the thread dump:

Found one Java-level deadlock:


"QuartzScheduler_Worker-2":
waiting to lock monitor 0x001b702c (object 0x0a7e7e80, a java.util.concurrent.
ConcurrentHashMap),
which is held by "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Defa
ult (self-tuning)'"
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'"
:
waiting to lock monitor 0x001b6d4c (object 0x0a7e7be0, a java.util.concurrent.
ConcurrentHashMap),
which is held by "QuartzScheduler_Worker-2"

Java stack information for the threads listed above:


"QuartzScheduler_Worker-2":
at org.springframework.beans.factory.support.DefaultListableBeanFactory.
getBeanDefinitionNames(DefaultListableBeanFactory.java:192)

  • waiting to lock <0x0a7e7e80> (a java.util.concurrent.ConcurrentHashMap
    )
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.
    getBeanNamesForType(DefaultListableBeanFactory.java:209)
    at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIn
    cludingAncestors(BeanFactoryUtils.java:187)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.
    findAutowireCandidates(DefaultListableBeanFactory.java:652)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.
    resolveDependency(DefaultListableBeanFactory.java:610)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanP
    ostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.j
    ava:412)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject
    Fields(InjectionMetadata.java:105)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanP
    ostProcessor.postProcessAfterInstantiation(AutowiredAnnotationBeanPostProcessor.
    java:240)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBean
    Factory.populateBean(AbstractAutowireCapableBeanFactory.java:959)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBean
    Factory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBean
    Factory$1.run(AbstractAutowireCapableBeanFactory.java:409)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBean
    Factory.createBean(AbstractAutowireCapableBeanFactory.java:380)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getOb
    ject(AbstractBeanFactory.java:264)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
    y.getSingleton(DefaultSingletonBeanRegistry.java:222)

  • locked <0x0a7e7be0> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBe
    an(AbstractBeanFactory.java:261)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean
    (AbstractBeanFactory.java:185)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean
    (AbstractBeanFactory.java:164)
    at org.springframework.aop.aspectj.annotation.BeanFactoryAspectInstanceF
    actory.getAspectInstance(BeanFactoryAspectInstanceFactory.java:76)
    at org.springframework.aop.aspectj.annotation.LazySingletonAspectInstanc
    eFactoryDecorator.getAspectInstance(LazySingletonAspectInstanceFactoryDecorator.
    java:47)

  • locked <0x0ab1eb00> (a org.springframework.aop.aspectj.annotation.Lazy
    SingletonAspectInstanceFactoryDecorator)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMet
    hodWithGivenArgs(AbstractAspectJAdvice.java:627)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMet
    hod(AbstractAspectJAdvice.java:609)
    at org.springframework.aop.aspectj.AspectJMethodBeforeAdvice.before(Aspe
    ctJMethodBeforeAdvice.java:39)
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceIntercept
    or.invoke(MethodBeforeAdviceInterceptor.java:49)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
    ReflectiveMethodInvocation.java:160)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.p
    roceed(MethodInvocationProceedingJoinPoint.java:77)
    at com.mycompany.services.commons.aspects.ClientProxyAspect.fillRequ
    estInfo(ClientProxyAspect.java:68)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMet
    hodWithGivenArgs(AbstractAspectJAdvice.java:627)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMet
    hod(AbstractAspectJAdvice.java:616)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAro
    undAdvice.java:64)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
    ReflectiveMethodInvocation.java:160)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invok
    e(ExposeInvocationInterceptor.java:89)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
    ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterc
    eptor.intercept(Cglib2AopProxy.java:635)
    at com.mycompany.services.bindings.trafficreporter.client.ws.ConfigurationW
    SClient$$EnhancerByCGLIB$$b4564aa2.getConfiguration(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at com.mycompany.services.trafficreporter.client.Clie
    ntProxy.executeMethod(ClientProxy.java:159)
    at com.mycompany.services.trafficreporter.client.Clie
    ntProxy.getConfiguration(ClientProxy.java:95)
    at com.mycompany.services.trafficreporter.manager.updater.Configurat
    ionManagerJob.updateAction(ConfigurationManagerJob.java:63)
    at com.mycompany.services.trafficreporter.manager.updater.UpdateJob.
    execute(UpdateJob.java:83)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
    ava:534)
    "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'"
    :
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
    y.getSingleton(DefaultSingletonBeanRegistry.java:180)

  • waiting to lock <0x0a7e7be0> (a java.util.concurrent.ConcurrentHashMap
    )
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
    y.getSingleton(DefaultSingletonBeanRegistry.java:166)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBe
    an(AbstractBeanFactory.java:206)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean
    (AbstractBeanFactory.java:185)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean
    (AbstractBeanFactory.java:164)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.
    preInstantiateSingletons(DefaultListableBeanFactory.java:429)

  • locked <0x0a7e7e80> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.context.support.AbstractApplicationContext.finish
    BeanFactoryInitialization(AbstractApplicationContext.java:728)
    at org.springframework.context.support.AbstractApplicationContext.refres
    h(AbstractApplicationContext.java:380)

  • locked <0x0a7e7610> (a java.lang.Object)
    at org.springframework.web.context.ContextLoader.createWebApplicationCon
    text(ContextLoader.java:255)
    at org.springframework.web.context.ContextLoader.initWebApplicationConte
    xt(ContextLoader.java:199)
    at org.springframework.web.context.ContextLoaderListener.contextInitiali
    zed(ContextLoaderListener.java:45)
    at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run
    (EventsManager.java:458)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(Authenticate
    dSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(Unknown Source)
    at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(Eve
    ntsManager.java:168)
    at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAp
    pServletContext.java:1721)

  • locked <0x0a5f7d50> (a weblogic.servlet.internal.WebAppServletContext)

    at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletCon

text.java:2890)
at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.jav
a:948)
at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:353)
at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleSta
teDriver.java:204)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineD
river.java:26)
at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStat
eDriver.java:60)
at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedMod
uleDriver.java:200)
at weblogic.application.internal.flow.ModuleListenerInvoker.start(Module
ListenerInvoker.java:117)
at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleSta
teDriver.java:204)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineD
river.java:26)
at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStat
eDriver.java:60)
at weblogic.application.internal.flow.StartModulesFlow.activate(StartMod
ulesFlow.java:26)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.ja
va:635)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineD
river.java:26)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.
java:212)
at weblogic.application.internal.DeploymentStateChecker.activate(Deploym
entStateChecker.java:154)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(Ap
pContainerInvoker.java:80)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicD
eployment.java:182)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromSer
verLifecycle(BasicDeployment.java:359)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(De
ploymentAdapter.java:51)
at weblogic.management.deploy.internal.DeploymentAdapter.activate(Deploy
mentAdapter.java:196)
at weblogic.management.deploy.internal.AppTransition$2.transitionApp(App
Transition.java:30)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionA
pps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.activate(Co
nfiguredDeployments.java:169)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(Conf
iguredDeployments.java:123)
at weblogic.management.deploy.internal.DeploymentServerService.resume(De
ploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(Dep
loymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

Found 1 deadlock.


Affects: 2.5.6

Issue Links:

  • #13117 Performance bottleneck and potential thread deadlock in DefaultSingletonBeanRegistry
  • #11684 Support initial delay attribute for @Scheduled and task:scheduled

11 votes, 14 watchers

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Feb 10, 2011

Ari Meyer commented

Hi,

We've now also encountered this, though on WLS 10.3.1. Can anyone from Spring address this issue?

Thanks,
Ari

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Feb 10, 2011

Ari Meyer commented

BTW, we're using Spring 2.5.6 SEC02.
Ari

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented May 23, 2011

Al Belsky commented

Hello Everyone;

I ran into a very similar deadlock (Spring 2.5.5 and 2.5.6) in a Web application with a DefaultMessageListenerContainer. If there were any messages on the JMS queue at start-up, the thread from the DefaultMessageListenerContainer instance would deadlock with the web context initialization thread. The stack trace looks very close to the one reported in this issue:

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinitionNames(DefaultListableBeanFactory.java:192)
- waiting to lock <0x28283a38> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:209)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:187)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:652)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:610)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:431)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:409)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:537)
at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:180)
at org.springframework.beans.factory.annotation.InjectionMetadata.injectFields(InjectionMetadata.java:105)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessAfterInstantiation(CommonAnnotationBeanPostProcessor.java:289)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:959)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
- locked <0x282837c0> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.aop.aspectj.annotation.BeanFactoryAspectInstanceFactory.getAspectInstance(BeanFactoryAspectInstanceFactory.java:76)
at org.springframework.aop.aspectj.annotation.LazySingletonAspectInstanceFactoryDecorator.getAspectInstance(LazySingletonAspectInstanceFactoryDecorator.java:47)
- locked <0x284ba198> (a org.springframework.aop.aspectj.annotation.LazySingletonAspectInstanceFactoryDecorator)

I'm happy to report that upgrading to Spring 3.0.5-RELEASE resolved the deadlock issue in my case.

Since these issues look related to me, I'd recommend to try to upgrade to the latest Spring 3.0.x distribution and see if this resolves the deadlock issue for you as well.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Oct 17, 2011

Paul Khodchenkov commented

Having the same issue when using spring task schedulers. Here is the dead locks thread dumps:

Found one Java-level deadlock:


"taskScheduler-2":
waiting to lock monitor 0x00007f7f1541e0f0 (object 0x00007f7f1cfefe78, a java.util.concurrent.ConcurrentHashMap),
which is held by "Thread-4"
"Thread-4":
waiting to lock monitor 0x0000000040a4d310 (object 0x00007f7f1cfef928, a java.util.concurrent.ConcurrentHashMap),
which is held by "taskScheduler-2"

Java stack information for the threads listed above:


"taskScheduler-2":
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinitionNames(DefaultListableBeanFactory.java:288)

  • waiting to lock <0x00007f7f1cfefe78> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:305)
    at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:185)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:833)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:790)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:707)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:433)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:411)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:543)
    at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:147)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:84)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:298)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1074)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
  • locked <0x00007f7f1cfef928> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
    at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:33)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:182)
    at $Proxy71.getUserLiveTimerMapKeySet(Unknown Source)
    at com.lifereader.bean.task.LiveUsersTask.run(LiveUsersTask.java:28)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
    at $Proxy77.run(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
    at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    "Thread-4":
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)
  • waiting to lock <0x00007f7f1cfef928> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:881)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:566)
  • locked <0x00007f7f1cfefe78> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
  • locked <0x00007f7f1cfe7cc0> (a java.lang.Object)
    at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:282)
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:204)
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4723)
    at org.apache.catalina.core.StandardContext$1.call(StandardContext.java:5226)
    at org.apache.catalina.core.StandardContext$1.call(StandardContext.java:5221)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Oct 17, 2011

Paul Khodchenkov commented

P.S Using Spring 3.0.6.RELEASE

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Jan 5, 2012

Nordin Haouari commented

We had a similar situation and were able to work around it. In our case the deadlock was caused by the initialization of session scoped beans.

We have lazy-init set to false (default), so our singleton beans are eagerly initialized on startup. Session scoped beans however are still lazily initialized. This caused our deadlock, as two requests (threads) were requesting a session bean that was not yet initialized. The way we worked around this issue was to use a scoped proxy, which basically creates a singleton bean proxying the session scoped bean. This singleton bean however will be eagerly initialized on startup, thus avoiding the deadlock situation.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Feb 29, 2012

John Cotter commented

With Spring 3.0.5, I am experiencing the same thing in an OSGi Virgo environment using Spring OSGi project - similar to https://jira.springsource.org/browse/OSGI-816

The common theme is that a second thread is accessing DefaultListableBeanFactory while context creation is still not fully complete. In my case it is due to the OSGi ConfigAdmin creating a manged service via managed service factory.

There is a race condition leading to deadlock with usage of DefaultListableBeanFactory and DefaultSingletonBeanRegistry by two threads. The two locks involved are beanDefinitionMap (ConcurrentHashMap in DefaultListableBeanFactory) and singletonObjects (ConcurrentHashMap in DefaultSingletonBeanRegistry).

This issue should be set higher than 'minor'.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Nov 25, 2012

Juergen Hoeller commented

This has finally been addressed now for Spring Framework 3.2 RC2, along with the changes in #13117, and will also get backported to 3.1.4. We minimize the scopes of the affected locks now, avoiding potential interleaving wherever possible - in particular during the preInstantiateSingletons phase.

Juergen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.