Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@Async with cglib based proxy causes memory leak in heap [SPR-11275] #15899

Closed
spring-projects-issues opened this issue Jan 2, 2014 · 2 comments
Closed

Comments

@spring-projects-issues
Copy link
Collaborator

@spring-projects-issues spring-projects-issues commented Jan 2, 2014

Lukasz Rozek opened SPR-11275 and commented

running mvn clean package && mvn exec:exec
against repro project that can be found here
https://github.com/lrozek/spring-leak
or https://github.com/lrozek/spring-leak/archive/master.zip
or git clone https://github.com/lrozek/spring-leak.git
causes java.lang.OutOfMemoryError: GC overhead limit exceeded

Here is sample of stack trace:

2014-01-02 11:02:49,886 [main] INFO  pl.lrozek.spring.leak.main.LeakMain - 1350 calleeThreadName is SimpleAsyncTaskExecutor-1, callerThreadName is main, areThreadsTheSame: false
[Full GC [PSYoungGen: 39936K->39413K(41984K)] [ParOldGen: 86640K->86551K(87040K)] 126576K->125965K(129024K) [PSPermGen: 57938K->57855K(58368K)], 0.4292090 secs] [Times: user=2.89 sys=0.01, real=0.43 secs] 
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid23319.hprof ...
Heap dump file created [215260624 bytes in 1.822 secs]
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.ClassLoader.getBootstrapResources(ClassLoader.java:1317)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1183)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1181)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(PathMatchingResourcePatternResolver.java:304)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:273)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(PathMatchingResourcePatternResolver.java:339)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:269)
	at org.springframework.context.support.AbstractApplicationContext.getResources(AbstractApplicationContext.java:1170)
	at org.springframework.context.support.GenericApplicationContext.getResources(GenericApplicationContext.java:223)
	at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:268)
	at org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan(ClassPathBeanDefinitionScanner.java:242)
	at org.springframework.context.annotation.ComponentScanAnnotationParser.parse(ComponentScanAnnotationParser.java:134)
	at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:236)
	at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:205)
	at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:182)
	at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:152)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:299)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:243)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:254)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:94)
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:609)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464)
	at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
	at pl.lrozek.spring.leak.main.LeakMain.main(LeakMain.java:18)

Here is another stacktrace:

2014-01-02 11:27:58,748 [main] INFO  pl.lrozek.spring.leak.main.LeakMain - 1376 calleeThreadName is SimpleAsyncTaskExecutor-1, callerThreadName is main, areThreadsTheSame: false
[Full GC [PSYoungGen: 39936K->39337K(41984K)] [ParOldGen: 86819K->86819K(87040K)] 126755K->126157K(129024K) [PSPermGen: 55355K->55355K(55808K)], 0.1434080 secs] [Times: user=0.86 sys=0.01, real=0.14 secs] 
[Full GC [PSYoungGen: 39936K->39391K(41984K)] [ParOldGen: 86819K->86819K(87040K)] 126755K->126210K(129024K) [PSPermGen: 55355K->55355K(55808K)], 0.1388310 secs] [Times: user=0.88 sys=0.01, real=0.13 secs] 
[Full GC [PSYoungGen: 39936K->39369K(41984K)] [ParOldGen: 86819K->86763K(87040K)] 126755K->126132K(129024K) [PSPermGen: 55355K->55323K(55808K)], 0.4117840 secs] [Times: user=2.87 sys=0.01, real=0.41 secs] 
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid26941.hprof ...
Heap dump file created [215055088 bytes in 1.795 secs]
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.reflect.Method.copy(Method.java:151)
	at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
	at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
	at java.lang.Class.copyMethods(Class.java:2852)
	at java.lang.Class.getDeclaredMethods(Class.java:1855)
	at org.springframework.util.ReflectionUtils.findMethod(ReflectionUtils.java:154)
	at org.springframework.util.ClassUtils.getMostSpecificMethod(ClassUtils.java:758)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.findResourceMetadata(CommonAnnotationBeanPostProcessor.java:351)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(CommonAnnotationBeanPostProcessor.java:283)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyMergedBeanDefinitionPostProcessors(AbstractAutowireCapableBeanFactory.java:908)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:354)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1094)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:989)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:220)
	at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:618)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
	at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
	at pl.lrozek.spring.leak.main.LeakMain.main(LeakMain.java:18)

Stack traces differ from run to run depending where OOME was thrown. Sometimes exception isn't thrown but than in the console there are only logs of garbage collector activity

java:

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

maven:

Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 17:22:22+0200)
Maven home: /opt/mvn/maven
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: /opt/jvm/jdk1.7.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.0.0-32-generic", arch: "amd64", family: "unix"

Here is the screenshot of the heap when OOME is about to be thrown
!heap leak.png!


Affects: 3.2.6, 4.0 GA

Reference URL: https://github.com/lrozek/spring-leak

Attachments:

Issue Links:

  • #17158 Inconsistent hashCode/equals implementation in AbstractPointcutAdvisor results in CGLIB-related memory leak ("is duplicated by")
  • #16398 ClassLoader leak with @Async, CGLIB and DefaultListableBeanFactory
  • #15411 Method injection causes memory leak
  • #15900 @Async with cglib based proxy causes memory leak in permgen

Backported to: 3.2.7

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Jan 2, 2014

Lukasz Rozek commented

The source of the leak is
org.springframework.cglib.proxy.Enhancer -> SOURCE -> cache

cache isn't cleaned after closing application context

!heap leak-mat.png!

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Jan 4, 2014

Juergen Hoeller commented

After some digging, it turns out that the root cause is a missing equals/hashCode implementation in AnnotationMatchingPointcut. That generic pointcut class is rarely used within the framework, since we're using specific pointcut implementations in most cases.

However, Async processing does rely on AnnotationMatchingPointcut. Its lack of a proper 'equals' implementation prevented reuse of any affected proxy class in CGLIB, leading to recreation of identical proxy classes for every new context. Fixed for 4.0.1 and 3.2.7 now.

Juergen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants