Memory leak in tycho 3.0.0 (results in OOM in JDT model tests) #879
Milestone
Comments
Any chance to get a dump how many memory is used there (maybe 1,99GB?) or run with e.g. 2,5GB to get an idea of the magnitude of memory used more? |
|
I believe (there are no logs preserved) the Jenkins reported 1 GB being set as total heap, however we need definitely 2G for some tests. 4G used now in https://github.com/eclipse-jdt/eclipse.jdt.core/blob/master/Jenkinsfile is an oversight from multiple attempts to understand the root cause . I've created eclipse-jdt/eclipse.jdt.core#9 to restore heap to 2GB - which was otriginally. If that fails, we have also other issues... |
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
|
tycho 3.0.0: failure See https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-11/1/console The log file says The system properties and environment were: |
|
tycho 2.7.0 (OK) The system properties and environment are below I don't see relevant differences. |
|
So whatever tycho 3.0.0 is doing, it causes significant additional heap usage. OK, tried - didn't work of course, because tycho forks jvm and ignores any maven options for the fork. Why not. But nevertheless, I've added the memory output and it shows that with tycho 3.0 we are eating all the heap: Will try again with 2.7.0 and see how much memory we eat there. |
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every suite - increased max heap for model tests to 2GB - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every suite - increased max heap for model tests to 2GB - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
|
With configuration/argline change we finally able to see custom JVM heap of 2 GB for tycho: https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-11/4/console I will add few gc() calls to better see the actual memory being used. |
|
Starting with https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-11/5/console There must be a memory leak with tycho 3.0.0 now, the used memory goes over 1.5 GB at the end of model tests (all tests were fine with 1 GB before!) |
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every suite - increased max heap for model tests to 2GB - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every suite - increased max heap for model tests to 2GB - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
|
With tycho 2.7.0 we see memory use below 0.4 GB: https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-11/9/consoleFull Same testsuite with tycho 3.0 reports > 1.5 GB... |
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every suite - increased max heap for model tests to 2GB - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
iloveeclipse
changed the title
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every model test suite - set explicit max heap for model tests to 1 GB (to identify memory leaks, they need max 0.5 GB) - print system properties / environment once for started JVM See eclipse-jdt#2 troubles and eclipse-tycho/tycho#879
iloveeclipse
added a commit
to eclipse-jdt/eclipse.jdt.core
that referenced
this issue
Apr 11, 2022
- log memory total/used/free on starting every model test suite - set explicit max heap for model tests to 1 GB (to identify memory leaks, they need max 0.5 GB) - print system properties / environment once for started JVM See #2 troubles and eclipse-tycho/tycho#879
|
Thanks for all the hints and testing, I'm still have not a clear idea why this happens because actually it should not matter what is run inside the test... Could you try with Tycho 2.7.1 (we have upgraded the Eclipse version there) if that works it might be the Surefire Upgrade (@mickaelistria any idea? Do we changed any defaults there regarding fork/test execution). I'm not very familiar with JDT code base, can these test (or at least one of it) run locally without much effort? I can then try to run a profiling session (or at least generate a heap-dump on OOM). |
|
The tests can be executed locally, sure, they are headless so you could even work on same display if they are running. I will try later 2.7.1. If I would guess I would say the leak depends on test count, not on the test content. There are lot of tests, and they are pretty fast, typically doing some compilation checks etc. |
iloveeclipse
added a commit
to iloveeclipse/eclipse.jdt.core
that referenced
this issue
Apr 12, 2022
|
See https://github.com/eclipse-jdt/eclipse.jdt.core/pull/13/files and https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-13/1/console The 2.7.1 version seem to be OK, tests are green & memory use also as expected. |
|
This OOM may be related to surefire 3.0.0.M6 usage. I'm investigating it with high priority (not sure when I'll have a patch or conclusion though) |
|
The issue can easily be reproduced with the org.eclipse.jdt.core.tests.model. Thanks @iloveeclipse for adding tweaks in order to see memory consumption. I added some extra code to get memory dump and can see while analyzing this memory dump (with Eclipse MAT which is still doing a really good job at it), that the TestSetRunListener is indeed always growing. Going deeper we can get this output (image) |
|
This seems caused by https://issues.apache.org/jira/browse/SUREFIRE-1845 , I opened https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-2072 as a followup. I'm looking at whether we can find some way to not capture stdout |
mickaelistria
added a commit
to mickaelistria/tycho
that referenced
this issue
Apr 14, 2022
|
Let's try reverting to surefire-3.0.0-M5 #890 |
mickaelistria
added a commit
to mickaelistria/tycho
that referenced
this issue
Apr 14, 2022
mickaelistria
added a commit
to mickaelistria/tycho
that referenced
this issue
Apr 14, 2022
mickaelistria
added a commit
that referenced
this issue
Apr 15, 2022
|
This should now be fixed. |
This was referenced Sep 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See various attempts to get JDT core tests running in
eclipse-jdt/eclipse.jdt.core#2
The net result is that tests pass without OOM with tycho 2.7.0, and fail with tycho 3.0.0.
The tests used to setup 2 GB heap for JVM, but may be that setting (or something else?) is broken now with tycho.
Related Jenkinsfile: https://github.com/eclipse-jdt/eclipse.jdt.core/blob/master/Jenkinsfile
The text was updated successfully, but these errors were encountered: