Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Possible) Memory Leak #33

Closed
jrezin opened this issue Apr 4, 2020 · 35 comments
Closed

(Possible) Memory Leak #33

jrezin opened this issue Apr 4, 2020 · 35 comments
Labels
bug Something isn't working

Comments

@jrezin
Copy link

jrezin commented Apr 4, 2020

I'm not sure how to properly debug and provide proper data about this bug, but basically, enabling this plugin causes a huge memory leak and disabling it fixes it.
I've initially posted it, before narrowing the problem TabOrder, in this issue OctoPrint/OctoPrint#3405

octoprint_memoryleak20200331 png

@jneilliii
Copy link
Owner

You might want to check this again while not on the control tab...that memory consumption is because of the webcam I think.

@jrezin
Copy link
Author

jrezin commented Apr 4, 2020

Yes, I did that. I happens while outside the webcam too, just starting octoprint and leaving it be, causes the memory consumption to grow and keep growing you terminate the process or crashing.

I've so, 6 GiBs of extra memory... against 150-250 MiB (even with the webcam in the control tab) of normal consumption (with TabOrder disabled).

@jrezin
Copy link
Author

jrezin commented Apr 4, 2020

This is Chrome's Task Manager.
You can summon it with Shift+Esc in the chrome window, or by 3dots > More tools > Task manager.

As the memory leak itself, I don't know what to say. I'm consistently getting it every time I enable the plugin.
If there's a procedure to better track the problem, I'd like to know how.

@jneilliii
Copy link
Owner

Yeah, I see it growing in the regular task manager. Not sure what has introduced this or if it's been there from the beginning. I'll try to figure this out.

@jneilliii
Copy link
Owner

jneilliii commented Apr 4, 2020

Ok, I think I've isolated the issue down to the latest release that incorporated an external library (bootstrap-tabdrop) for the ability to push items into the drop-down list, effectively "hiding" the tab from default view. This basically overwrites the bundled version of OctoPrint. Now, figuring out why it is happening will be a challenge,

@jneilliii
Copy link
Owner

I think I got it. Went back the default bootstrap-tabdrop module included with OctoPrint and just patched it with the settings needed to push the hidden tabs into the drop-down. The caveat is we lose the label on the dropdown for the active tab, but I think killing a memory leak is more important. If you don't mind testing to confirm this does fix the issue that would be greatly appreciated. Use the URL below in Plugin Manager to install the updated version.

https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7.zip

@jrezin
Copy link
Author

jrezin commented Apr 5, 2020

After finishing a print, I've installed 0.5.7, enabled it and left the UI open and running overnight.
Unfortunately this morning OctoPrint is using 3.6GiBs of memory.

octoprint_memoryleak20200405

@jneilliii
Copy link
Owner

Did you force a browser refresh to overwrite the local cache?

@jneilliii
Copy link
Owner

Never mind. It's still a problem. I didn't leave it long enough before to know for sure. I'm curious if this issue has been there longer than my tabdrop modifications. Going to install an old version and see if it's the same.

@jneilliii
Copy link
Owner

So I'm starting to think my previous memory growth was coincidental. I've been running two tabs overnight, one with version 0.5.7 (highlighted) and one with version 0.5.5 and it's showing "normal" memory consumption.

image

@jrezin
Copy link
Author

jrezin commented Apr 6, 2020

I'm testing version 0.5.4, and after few hours, it looks fine.
Octoprint using about ~140MiB of memory, steady.

@jneilliii
Copy link
Owner

I haven't shut down my browser tab running version 0.5.7 since last night and it is steady at ~200MB now, so I think whatever issue was there before is either specific to a certain condition/steps or not related to the plugin at all.

@jrezin
Copy link
Author

jrezin commented Apr 6, 2020

I didn't have problems with 0.5.4 after several hours.
upgraded to 0.5.5. and after about an hour, memory usage have past 800 MiBs already.

@jneilliii
Copy link
Owner

Yeah, this is the strange part because both my 0.5.5 version and 0.5.7 version are still in the 200s.

image

Curious if it's a conflict with another plugin maybe or like I mentioned before some other random occurrence/conflict like mentioned in the other post in regards to update notifications maybe.

@jneilliii
Copy link
Owner

jneilliii commented Apr 7, 2020

I did just notice something @jrezin that is different to what you have versus what I have on my 0.5.7 version instance. You appear to be using "icon only" mode by not showing any labels, but on mine my labels are visible. At this point I'm just stabbing in the dark to isolate this issue and determine if it's specific to TabOrder or not.

Edit: Doesn't appear to be icon only options. I changed my 0.5.7 instance to just icons and I'm still floating in the 200s.

@jneilliii
Copy link
Owner

12 hours later and my instance is still hovering in the 200s...

@jrezin
Copy link
Author

jrezin commented Apr 7, 2020

well, my setup current other enabled plugins:

  • Cancel Objects (0.4.1)
  • Dragon Order (0.1.4)
  • Fullscreen Plugin (0.0.4)
  • Gcodebar Plugin (0.1.5)
  • GcodeEditor (0.2.8)
  • LayerDisplay (0.4.1)
  • Navbar Temperature Plugin (0.13)
  • OctoKlipper (0.2.5-JEL-0.1)
  • Python 3 Check (0.1.2)
  • Stateful Sidebar (0.1.2)
  • Themeify (1.2.0)
  • Wemo Switch (0.1.5)

@jneilliii
Copy link
Owner

@MFHFozzy, do you have any of these same plugin's installed?

@MFHFozzy
Copy link

MFHFozzy commented Apr 9, 2020

#1 has GecodeEditor.
#2 has Cancel objects. (#1 might have had this before a couple days ago...)
Both have Gcodebar Plugin, LayerDisplay, Themeify.

@jneilliii
Copy link
Owner

Common Plugins So Far and some assumptions since I don't use all these:

  • Cancel Objects, memory usage high during printing because of gcode monitoring? I don't have this one.
  • Gcodebar, monitors keyup/keydown events on the textinput, highly unlikely conflict. I don't have this one.
  • Themeify, potential conflict due to icon manipulation option on tabs? I have this but only for debugging other user issues and typically have it disabled.
  • LayerDisplay, memory usage higher during printing? I don't have this one but do have DisplayLayerProgress plugin version 1.19.0, which I believe is similar. @jrezin you may want to upgrade if your on OctoPrint 1.4.0 because the older version seems to have a deprecated callback or something.
  • Gcodeeditor, unlikely unless you use it to load a gcode file. I don't have this one.
  • TabOrder, manipulation of tabs on window resize event potential problem by monkey patching the bootstrap-tabdrop module or related to the fontawesome filepicker?

Once you see the higher memory usage does refreshing the page drop the memory back down?

@jrezin
Copy link
Author

jrezin commented Apr 9, 2020

"OctoPrint 1.4.0 running on OctoPi 0.17.0".

LayerDisplay - https://plugins.octoprint.org/plugins/layerdisplay/

0.5.7 compared to 0.5.4, its 100MiB of extra memory usage from the start.
0.5.7 does leak memory much less/slower than 0.5.6.
Refreshing does drop the memory usage back down (after a minute or two), but not all the leak (after refreshing, still a LOT more memory than Ending Tab's Process). With time, I'll leak a lot again.

@jneilliii
Copy link
Owner

Thanks @jrezin, the memory jump from 0.5.4 to 0.5.7 is more than likely due to the inclusion of the fontawesome icon picker. Let me pull together an update to 0.5.7 without that feature and let's see if the issue is still there or not and that will help in identifying if that is the true cause or not. Updates to come for additional testing.

@MFHFozzy
Copy link

I reinstalled tab order 0.5.6 yesterday (from repo) on #1 of my octopies, but so far today no leaking.
On the other, i just re-installed bed leveler 0.1.13 (from repo) and restarted it to see what happens.

@jrezin
Copy link
Author

jrezin commented Apr 10, 2020

a few hours in with https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7_iconpicker_removed.zip and it looks like 0.5.4 levels of memory usage, does not appear to be leaking for now.

@jneilliii
Copy link
Owner

So I think I've been able to get the full functionality from before without the memory leak. So far my testing of https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7_pureComputed.zip which converts the computed observables to pureComputed observables and removes the fa-lg class from being automatically added is doing well. If you do try it out, make sure to force a browser refresh.

@jrezin
Copy link
Author

jrezin commented Apr 14, 2020

installed 0.5.7_pureComputed.zip last night, was using 0.5.7_iconpicker_removed.zip before.
Unfortunately the memory usage this morning is 3.9 GiB.
Changing tabs didn't make a difference.
Refreshing OctoPrint did drop the memory usage, but to around 950 MiBs at first, and 650 MiB after 1 ou 2 minutes, which is still much more than normal.

@jneilliii
Copy link
Owner

That's so strange. I left mine running overnight and although it did grow, it didn't get anywhere near that much. It did double though. @MFHFozzy is your #1 printer still not showing signs of memory leak with the official 0.5.6 release?

@MFHFozzy
Copy link

Im trying to nail things down better, as nothing is making sense.
Using/Monitoring Waterfox about:memory, i am so far unable to reproduce the leak...
but the initial report was using Opera, and looking at its task manager.

In frustration, i opened octopi the other night in chrome, and i also did a print last night (small, 1 hour). This morning was at 822MB. (yay, at least i got a reproduction)
Ive switched back to water fox, and am doing the same print right now.
If no leak again on waterfox, ill re-test with/without plugins on chrome, unless anyone has a better idea?

Logging so far:
Memory Tracking Octopi

2020-04-05 1:50pm, removed tab order, bed visualizer from both
96.69 MB (05.52%) ++ top(http://192.168.0.92/#control, id=1593) Connected
107.12 MB (06.11%) ++ top(http://192.168.0.93/#temp, id=1599) Not connected

2020-04-07 4:13pm
102.25 MB (05.90%) ++ top(http://192.168.0.92/#temp, id=1593)
44.24 MB (02.55%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-08 2:51pm
91.15 MB (04.99%) ++ top(http://192.168.0.92/#temp, id=1593)
40.11 MB (02.20%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-08 2:53pm reisntalled tab order 0.5.6 on .92 from repo, restarted
104.39 MB (05.68%) ++ top(http://192.168.0.92/#temp, id=1593)
39.70 MB (02.16%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-09 5:49pm
94.27 MB (05.67%) ++ top(http://192.168.0.92/#temp, id=1593)
40.81 MB (02.46%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-09 5:49pm reisntalled bed visualizer 0.1.13 on .93 from repo, restarted

2020-04-10 1:57pm
84.70 MB (04.91%) ++ top(http://192.168.0.92/#temp, id=1593)
87.51 MB (05.07%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:37am
78.58 MB (04.23%) ++ top(http://192.168.0.92/#temp, id=1593)
57.06 MB (03.07%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:07pm
80.88 MB (04.25%) ++ top(http://192.168.0.92/#temp, id=1593)
57.16 MB (03.01%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:15pm reinstalled tab order 0.5.6 on .93, bed visualizer 0.1.13 on .92
257.40 MB (10.75%) ++ top(http://192.168.0.92/#temp, id=1593
127.34 MB (05.32%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 3:19pm
106.92 MB (04.86%) ++ top(http://192.168.0.92/#temp, id=1593)
72.83 MB (03.31%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 3:19pm
76.53 MB (04.05%) ++ top(http://192.168.0.92/#temp, id=1593)
70.18 MB (03.72%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-13 1:10pm
93.93 MB (03.97%) ++ top(http://192.168.0.92/#temp, id=1593)
60.43 MB (02.55%) ++ top(http://192.168.0.93/#temp, id=1599)

Chrome: 2020-04-13 to 2020-04-14 10:07am
.92 822,480K (did a print alst night)
.93 101,396k (no print, still not connected)
Switched back to Waterfox (10:10am)
142.08 MB (06.04%) ++ top(http://192.168.0.92/#temp, id=1593)
did same print

@jneilliii
Copy link
Owner

Great stats @MFHFozzy, but for the most part it looks like the leak is not present until that last day in chrome, which is what I've been testing with and haven't been able to get reliable reproduction of the issue. That did make me think though, what extensions do you guys have enabled/installed in chrome? I noticed during my memory heap snapshots in developer tools that there was memory being consumed by installed extensions, not just the page content.

This may be something else that you guys could do to help diagnose. On the memory tab of developer tools take a heap snapshot, wait a couple of hours then press the little delete button followed by another snapshot, then switch the view mode to Statistics (or compare) and let's see where the consumption is happening.

image

The files end up being pretty big, but if you can upload/share them somewhere it might also help. Like I said I haven't been able to reliably reproduce. There was another user on the community forum that was discussing memory leaks with AutomaticShutdown plugin. Also, have either of you tried with just TabOrder enabled to see if the issue persists?

@MFHFozzy
Copy link

I have switched back to chrome, and will take some snapshots as described.

@MFHFozzy
Copy link

I should mention that i has also saved a bunch of the measure reports from Waterfox as well. I dont know how to use them though :) I can post them if that will help.

@MFHFozzy
Copy link

My printers are currently non-stop printing face shield parts for the covid-19 efforts in LA, so 1 change is that i cant let them just "sit" for a day anymore.
That being said, here is some additional findings:

I switched back to chrome @ 12:01am, took snapshots and recorded task manager tab memory values, then started another 3 hour print:

2020-04-17 12:01am Started 3 hour print
Task manager:
.92 139 MB
.93 112 MB
Heap Snapshots:
.92 19 MB
.93 21.7 MB

~3 hours later, after the print was done:
Task manager:
.92 1.2 GB
.93 348 MB
Heap Snapshots:
.92 20.6 MB
.93 23.4 MB

So Chrome task manager sees a huge memory tab use by Octoprint, but the heap doesnt?

Ive attached all the log files i have so far. Both heap files and the waterfox mem dumps
I did a quick look over to make sure theres no personal or identiy data in them..but if you see/know of something that should be removed, please let me know.

92-Heap-20200417T120509.zip
92-Heap-20200417T154020.zip
93-Heap-20200417T153957.zip
93-Heap-20200417T120611.zip
2020-04-05x1-50pm-memory-report.json.gz
2020-04-11x01-38am-memory-report.json.gz
2020-04-11x03-02pm-memory-report.json.gz
2020-04-15x09-55pm-memory-report.json.gz
2020-04-17x11-58am-memory-report.json.gz
2020-04-07x12-56pm-memory-report.json.gz
2020-04-08x02-49pm-memory-report.json.gz
2020-04-10x01-57pm-memory-report.json.gz

@jneilliii
Copy link
Owner

Thanks @MFHFozzy, not sure about what's captured in those heap dumps but from what I've seen so far I don't think there's any personal information in them.

I was seeing the same discrepancy between chrome task manager window and the heap snapshots, not sure what that means. I'm new to memory leak debugging.

@stale
Copy link

stale bot commented May 1, 2020

This issue has been automatically marked as stale because it has not had activity in 14 days. It will be closed if no further activity occurs in 7 days.

@stale stale bot added the stale label May 1, 2020
@jneilliii jneilliii added the bug Something isn't working label May 2, 2020
@stale stale bot removed the stale label May 2, 2020
@jneilliii
Copy link
Owner

Releasing version 0.5.8 that should resolve the memory leak issue and retain the icon picker by binding it only once to a popup editor form instead of every row.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants