New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live-error checker is inefficient with memory. #1

Closed
aengelke opened this Issue Jun 29, 2013 · 10 comments

Comments

Projects
None yet
4 participants
@aengelke

See processing/processing#1861, processing/processing#1764.

OS: Windows 7 32bit, 3GB RAM

  1. Open a blank PDE and switch to the experimental mode.
  2. Look at the task manager: Processing --> 90MB of the memory
  3. Open a sketch: Processing --> 120MB of the memory
  4. Close the blank sketch: Processing --> still 120MB of the memory
  5. Redo Steps 3 and 4 about 10 times: At about 200MB of memory, PDE won't open another sketch. OutOfMemoryError: PermGen space

After restarting my system and on a fresh restart, I could open 11 sketches and closing them again (so I only have one sketch open!). After trying opening another sktech, Processing says in the output window:

Part 1:
image

Part 2:
image

And it repeats printing the last lines. (Exception: One time it says Thread-11 insteaf of the AWT-EventQueue) The cause I am making screenshots is that Processing isn't able to copy the text, because there is not enough memory. I also can't save, edit or copy the sketch. I can't open menus. After about 100 lines of OutOfMemoryErrors, Processing doesn't react anymore.

Processing has to be closed via task manager, because it doesn't react.

@ghost ghost assigned Manindra29 Sep 18, 2013

@Manindra29 Manindra29 referenced this issue Oct 9, 2013

Closed

PermGen Space #26

@StanLepunK

This comment has been minimized.

Show comment
Hide comment
@StanLepunK

StanLepunK Oct 9, 2013

But it's a real problem, isn't it ? For example for me I must work generally with minimuym three sketches open in same time...And I like PDE-X. There is a project to resolve this problem of PermGem Space or it's not possible ?

But it's a real problem, isn't it ? For example for me I must work generally with minimuym three sketches open in same time...And I like PDE-X. There is a project to resolve this problem of PermGem Space or it's not possible ?

@benfry

This comment has been minimized.

Show comment
Hide comment
@benfry

benfry Oct 9, 2013

Member

@Manindra29 It's possible that you're dealing with a new ClassLoader for each X Mode window, causing it to reload all those Eclipse JAR files (which can't be fast either). There might be a way around this... whether we need to do a design change in the PDE so that it's only creating single Mode instances, or that you might be able to do something in the X Mode code for how things are pulled from the CL.

Member

benfry commented Oct 9, 2013

@Manindra29 It's possible that you're dealing with a new ClassLoader for each X Mode window, causing it to reload all those Eclipse JAR files (which can't be fast either). There might be a way around this... whether we need to do a design change in the PDE so that it's only creating single Mode instances, or that you might be able to do something in the X Mode code for how things are pulled from the CL.

@Manindra29

This comment has been minimized.

Show comment
Hide comment
@Manindra29

Manindra29 Oct 9, 2013

Member

For each instance of the Editor, a new ClassLoader is needed because:

Every sketch uses its own set of P5 libraries. Sketches are checked for errors and other stuff by a CompilationChecker instance which uses Eclipse JDT libraries. The last time I'd tried, I was not able to manually specify the classpath of these P5 libraries along with the sketch path - a limitation of JDT since it's not running in a full fledged OSGi environment. There are API methods for it, but I couldn't get them to work..

The workaround I devised was to have a separate CompilationChecker instance for every open sketch(window) and add the library paths of the particular sketch(including 'code' folder imports) to the ClassLoader loading the CompilationChecker instance. So for different sketches, each ClassLoader would load the only the appropriate libraries. But I suspect that when a window is closed, the resources aren't getting freed up for some reason, even after I've manually tried to free those classes. On closing a window, RAM usage doesn't go down with PDE X Mode.

The other alternative could be to add paths of all installed libraries to the classpath of a master ClassLoader and use a single CompilationChecker instance. But I think this could affect the efficiency of code completion and other features. Plus, how would this solve the sketch specific 'code' folder imports?

I'm open to ideas.

Member

Manindra29 commented Oct 9, 2013

For each instance of the Editor, a new ClassLoader is needed because:

Every sketch uses its own set of P5 libraries. Sketches are checked for errors and other stuff by a CompilationChecker instance which uses Eclipse JDT libraries. The last time I'd tried, I was not able to manually specify the classpath of these P5 libraries along with the sketch path - a limitation of JDT since it's not running in a full fledged OSGi environment. There are API methods for it, but I couldn't get them to work..

The workaround I devised was to have a separate CompilationChecker instance for every open sketch(window) and add the library paths of the particular sketch(including 'code' folder imports) to the ClassLoader loading the CompilationChecker instance. So for different sketches, each ClassLoader would load the only the appropriate libraries. But I suspect that when a window is closed, the resources aren't getting freed up for some reason, even after I've manually tried to free those classes. On closing a window, RAM usage doesn't go down with PDE X Mode.

The other alternative could be to add paths of all installed libraries to the classpath of a master ClassLoader and use a single CompilationChecker instance. But I think this could affect the efficiency of code completion and other features. Plus, how would this solve the sketch specific 'code' folder imports?

I'm open to ideas.

@benfry

This comment has been minimized.

Show comment
Hide comment
@benfry

benfry Oct 9, 2013

Member

Ah, I see... The libraries are not necessarily just the Eclipse RCP/OSGI stuff, but all the libraries that are part of the sketch so that autocomplete, etc work.

The Eclipse stuff may not be unloading since things are being added to a faux OSGI registry or something like that. Or there's just a simpler memory leak somewhere.

Guess you'd have to futz around with VisualVM or something else to profile the code and figure out what's hanging around.

Member

benfry commented Oct 9, 2013

Ah, I see... The libraries are not necessarily just the Eclipse RCP/OSGI stuff, but all the libraries that are part of the sketch so that autocomplete, etc work.

The Eclipse stuff may not be unloading since things are being added to a faux OSGI registry or something like that. Or there's just a simpler memory leak somewhere.

Guess you'd have to futz around with VisualVM or something else to profile the code and figure out what's hanging around.

@Manindra29

This comment has been minimized.

Show comment
Hide comment
@Manindra29

Manindra29 Oct 11, 2013

Member

Guess you'd have to futz around with VisualVM

I did just that. From the heap dump, it seems just the CompilationChecker isn't getting unloaded after closing its parent Editor window. I've yet to locate the source of the bug though..

Member

Manindra29 commented Oct 11, 2013

Guess you'd have to futz around with VisualVM

I did just that. From the heap dump, it seems just the CompilationChecker isn't getting unloaded after closing its parent Editor window. I've yet to locate the source of the bug though..

@Manindra29

This comment has been minimized.

Show comment
Hide comment
@Manindra29

Manindra29 Oct 11, 2013

Member

@benfry I think I've found a possible solution. The Profiler was showing that heap space wasn't getting reclaimed after a Editor window is closed. I manually set the CompilationChecker's classloader as null and added a System.gc() at the end of exiting a Editor instance. Now heap space gets reclaimed correctly on closing of Editor window. Wasn't expecting garbage collection needed to be done manually..

Previously, PDE X froze after opening about 7-8 sketches(irrespective of whether they stay open or closed), and one had to force quit. By making the above change, it's now possible to keep opening multiple sketches as long as one stays below the limit of 7-8 open sketches at a time. Will test it out a bit more..

The max window limitation can be overcome by increasing the default 64 MB heap space allotted to Processing by adding the following JVM options to the PDE launch command.

-Xmx1024m -XX:MaxPermSize=256m

Member

Manindra29 commented Oct 11, 2013

@benfry I think I've found a possible solution. The Profiler was showing that heap space wasn't getting reclaimed after a Editor window is closed. I manually set the CompilationChecker's classloader as null and added a System.gc() at the end of exiting a Editor instance. Now heap space gets reclaimed correctly on closing of Editor window. Wasn't expecting garbage collection needed to be done manually..

Previously, PDE X froze after opening about 7-8 sketches(irrespective of whether they stay open or closed), and one had to force quit. By making the above change, it's now possible to keep opening multiple sketches as long as one stays below the limit of 7-8 open sketches at a time. Will test it out a bit more..

The max window limitation can be overcome by increasing the default 64 MB heap space allotted to Processing by adding the following JVM options to the PDE launch command.

-Xmx1024m -XX:MaxPermSize=256m

@benfry

This comment has been minimized.

Show comment
Hide comment
@benfry

benfry Oct 20, 2013

Member

I'm nervous about setting the minimum memory to 1 GB across the board—even if it's not used, it can cause launching to fail on machines that have less memory (netbooks, etc).

I think 7-8 sketches open at once should be fine. Are you doing anything to catch the OutOfMemoryError if that's exceeded? Letting people know what's going on would go a long way here (and would give us the opportunity to tell them how to manually increase the memory setting if they want to have a large number of sketches open).

Member

benfry commented Oct 20, 2013

I'm nervous about setting the minimum memory to 1 GB across the board—even if it's not used, it can cause launching to fail on machines that have less memory (netbooks, etc).

I think 7-8 sketches open at once should be fine. Are you doing anything to catch the OutOfMemoryError if that's exceeded? Letting people know what's going on would go a long way here (and would give us the opportunity to tell them how to manually increase the memory setting if they want to have a large number of sketches open).

@StanLepunK

This comment has been minimized.

Show comment
Hide comment
@StanLepunK

StanLepunK Oct 20, 2013

For information, In my case, I use 4 sketches in same time in my work session, I open few other sketches and close after I take what I need in this one. Processing freeze after few opening sketches may be trhee or four. For me the problem is not open 7-8 sketches in same time but the PermGem problem come after the 7-8th opening.
My memory alowed to Processing are 10G.

For information, In my case, I use 4 sketches in same time in my work session, I open few other sketches and close after I take what I need in this one. Processing freeze after few opening sketches may be trhee or four. For me the problem is not open 7-8 sketches in same time but the PermGem problem come after the 7-8th opening.
My memory alowed to Processing are 10G.

@Manindra29

This comment has been minimized.

Show comment
Hide comment
@Manindra29

Manindra29 Oct 21, 2013

Member

I'm nervous about setting the minimum memory to 1 GB across the board—even if it's not used, it can cause launching to fail on machines that have less memory (netbooks, etc).

That's definitely not something we would want. I'm catching the Exception generated eventually and pausing the error checking, but it still freezes the PDE. I'll look into catching the Error at the source, directly.

would give us the opportunity to tell them how to manually increase the memory setting if they want to have a large number of sketches open

Agreed. People often get confused that the memory setting in the preference is for the entire PDE.

@StanLepunK The memory you've allocated in Preferences is the one allocated to Processing sketches, not the editor itself.

Edit - Updated the bug title.

Member

Manindra29 commented Oct 21, 2013

I'm nervous about setting the minimum memory to 1 GB across the board—even if it's not used, it can cause launching to fail on machines that have less memory (netbooks, etc).

That's definitely not something we would want. I'm catching the Exception generated eventually and pausing the error checking, but it still freezes the PDE. I'll look into catching the Error at the source, directly.

would give us the opportunity to tell them how to manually increase the memory setting if they want to have a large number of sketches open

Agreed. People often get confused that the memory setting in the preference is for the entire PDE.

@StanLepunK The memory you've allocated in Preferences is the one allocated to Processing sketches, not the editor itself.

Edit - Updated the bug title.

@Manindra29 Manindra29 added the bug label May 11, 2014

@Manindra29

This comment has been minimized.

Show comment
Hide comment
Member

Manindra29 commented Jul 11, 2014

@Manindra29 Manindra29 closed this Jul 11, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment