-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
org.eclipse.mat.parser.index.IndexWriter$Identifier.add(IndexWriter.java 91) run out of memory #24
Comments
By Andrew Johnson on Nov 14, 2015 12:24 Are there any messages in the error log? |
By G Xu on Nov 14, 2015 15:40 no additional information. We know that we run out index range a few times. The work around was to take a dump early to avoid the issue. However, we still have an issue to address here. |
By Andrew Johnson on Jun 06, 2020 10:03 I've written a work around for this problem in bug 563960, but that is just for HPROF. |
By G Xu on Jun 08, 2020 09:16 it is worth to proceed. MAT is still a tool we use on daily basis. |
By Andrew Johnson on Jun 10, 2020 15:43 I've made the changes for DTFJ including adding some help in bug 563960. |
By G Xu on Jun 11, 2020 00:18 thanks and will do! |
By Tomas Hamal on Nov 02, 2020 05:09 Tried to read 149GB hprof file without sucess with snapshot version: java.lang.OutOfMemoryError: Requested length of new long[2,147,483,640] exceeds limit of 2,147,483,639 |
By Andrew Johnson on Nov 02, 2020 05:38 Thank you for trying the snapshot version. There is still a limit of approximately 2^31 objects in the snapshot, but there is a new facility to discard some objects Please could you test that to see if the discard option is helpful and whether the help makes sense. [The help should be updated to have the full text of that error message (it is missing java.lang. before OutOfMemoryError) to aid searches. Also, perhaps the exception should suggest trying Window > Preferences > Memory Analyzer > Enable discard ] |
By G Xu on Nov 02, 2020 09:19 perhaps the default is on, and when this type of things happens, a warning message is generated with suggestions if available? |
By Andrew Johnson on Nov 07, 2020 03:21 The current discard settings can't always be on, because they discard a percentage of objects of specified types, which would change the results for every dump. It might be possible to discard all objects after a certain number, but that is more likely to change the object graph than discarding objects with no outbound references (other than the class), such as char[], and objects only referencing those types of object, like String. Someone could write a query generating a list of candidate classes for discard, based on those characteristics, ready for a re-parse, but does this problem occur often enough to make that worthwhile? |
Nov 07, 2020 03:25 New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/171933 |
Nov 07, 2020 08:51 Gerrit change https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/171933 was merged to [master]. |
By Tomas Hamal on Nov 12, 2020 09:57 I need to say, the I found nothing in the documentation (Memory Analyzer Configuration) about discard option. So Thank You very much for pointing it out. |
By Andrew Johnson on Nov 12, 2020 10:59 I glad it helped. The information is not yet in the online documentation at help.eclipse.org |
By Jason Koch on Apr 30, 2021 13:49 I am coming across more heaps that fit this case. Our interim solution is likely to pre-parse the heap and count how many objects, and then implement discard if it is over the limit. Is there a longer term plan to get MAT onto 64-bit identifiers? I think this is feasible, and would actually simplify some of the codebase, at the obvious cost of increasing heap footprint. OTOH, I think this only affects parsing stage, since most of the "use" of indexes at read time doesn't retain significantly more footprint and due to SoftRefs is very manageable heap usage. I haven't looked at how much work this would be. I suspect the main technical challenge would be (1) backwards compatibility for already-indexed heaps. I think discard & reindex is likely appropriate solution here, (2) increased memory footprint by using long[] during parsing. I think view/read phase unlikely impacted significantly. Thoughts? |
By Jason Koch on May 05, 2021 17:30 For what it's worth, with the discard solution, I see some issues, I expect we need to handle this in a couple of places. For ex, I cannot open the System Properties box as some strings have been removed. This is not unexpected, but might require some explanation for the user. |
By Andrew Johnson on May 06, 2021 05:00 The discard objects feature was a simple work-around without a major rewrite and I'm pleased it worked at all. Some minor enhancements ideas:
Moving to 64-bit is going to be hard. One idea to assess the scope and possibly ease the migration would be to have an annotation @objectid which we could use to mark up the source. A global replace '@objectid int' with @objectid long' won't work, but might show where the next problems are. A Java compiler annotation process for @objectid might help the migration. |
By Jason Koch on May 07, 2021 11:32 Yes, there is the dirct int->long swap, and on top of this, there is also likely to be the absence of primitive arrays of appropriate length which will drive use of some custom array layer. |
By Andrew Johnson on May 17, 2021 09:10 For comment 16, yes there are problems with discarding objects. I'm considering some improvements which could be made without really changing the API. System properties - yes the query could return null for those invalid key or value fields, either always, or when some objects have been discarded in the snapshot. I think there is scope to have the inspector view display unindexed objects. Some of the MAT API uses int object IDs, some does not. In the Inspector view unindexed fields appear as: Type|Name|Value
|
By G Xu on May 17, 2021 09:14 nice investigation. As I see more and more objects not having valid objectId, this can be useful. Also, I am wondering if there is some underline reason that we still see this more often in last couple of years? |
By Jason Koch on May 17, 2021 15:27 Hardware is getting cheaper and bigger, and users are using larger application instances: |
By Andrew Johnson on May 24, 2021 02:28 Created attachment 286443 Unindexed object screenshot.png |
By Andrew Johnson on May 24, 2021 02:44 I've made the changes in discussed in comment 19 in bug 573598. Now, after discarding objects, they remain accessible via ObjectReference but not via object ID. It won't solve a problem with objects having an outbound field pointing to an object which is not in the heap at all. If this is useful we can look at fixing some of the other queries, so please give it a try. This is not as good as a proper >32 bit index version. This change did show some of the difficulties which will arise. |
By Axel Uhl on Jun 01, 2022 14:52 I can only agree that larger applications need a remedy to this sort of problem. We just produced two heap dumps of an application that ended up in unpleasant and unexpected Full GCs, one ~270GB, the other 350GB in size, and there is no way we can get them analyzed with Memory Analyzer. Any progress here? |
By Andrew Johnson on Jun 06, 2022 09:39 (In reply to Axel Uhl from comment #24)
Not much progress - but some ideas in bug 572512. Breaking the 2G object limit The 270GB and 350GB heaps are huge, and might have between 3500 million and 5000 million objects, which would exceed the MAT limit. Did the object discard option https://help.eclipse.org/latest/topic/org.eclipse.mat.ui.help/tasks/configure_mat.html#task_configure_mat__discard help at all? |
By Tomas Hamal on Jun 06, 2022 09:44 For us the discard option worked quite well and we was able to diagnose over 100GB heap dumps. Sure, it has the consequences that you can discard also useful information as we needed to discard about 50% of all data, just basic discard options was not enough. However we were able to locate the memory leaks. |
By Andrew Johnson on Jul 09, 2022 10:42 Bug 580107 fixes the problem in comment 16 of system properties not displaying values when strings are discarded. |
| --- | --- |
| Bugzilla Link | 473113 |
| Status | NEW |
| Importance | P3 normal |
| Reported | Jul 20, 2015 16:40 EDT |
| Modified | Jul 09, 2022 10:42 EDT |
| Version | 1.5 |
| Depends on | 563960, 573598 |
| See also | Gerrit change 171933, Git commit 81b6c67d |
| Reporter | G Xu |
Description
When processing a customer java system dump, we run into the
following error.
java.lang.OutOfMemoryError: Requested length of new long[2,147,483,640]
exceeds limit of 2,147,483,639
at
org.eclipse.mat.parser.index.IndexWriter$Identifier.add(IndexWriter.java
:91)
at
org.eclipse.mat.dtfj.DTFJIndexBuilder.rememberObject(DTFJIndexBuilder.ja
va:3179)
at
org.eclipse.mat.dtfj.DTFJIndexBuilder.fill(DTFJIndexBuilder.java:1162)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactor
yImpl.java:237)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(Snapsho
tFactoryImpl.java:141)
at
org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.ja
va:145)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.parse(ParseSnapshotApp.ja
va:134)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.start(ParseSnapshotApp.ja
va:106)
at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.j
ava:196)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplicat
ion(EclipseAppLauncher.java:110)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(Eclip
seAppLauncher.java:79)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:
369)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:
179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:620)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
at org.apache.tools.ant.taskdefs.Exit.execute(Exit.java:164)
at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
06)
at org.apache.tools.ant.Task.perform(Task.java:348)
at
org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:68)
at net.sf.antcontrib.logic.IfTask.execute(IfTask.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
06)
at org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:154)
at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
06)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:435)
at org.apache.tools.ant.Target.performTasks(Target.java:456)
at
org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
at
org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecut
or.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
at org.apache.tools.ant.Main.runBuild(Main.java:851)
at org.apache.tools.ant.Main.startAnt(Main.java:235)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
*Behaviour differences: na
*Impact on product plans: Can't proceed and we don't know we run out of
memory.
The text was updated successfully, but these errors were encountered: