Octoprint web interface extremely slow after upgrade to 1.2.3

[foulowl]

1. I have octoprint open in a single tab in chromium 43.0.2357.65 on Debian 8. This system is running no other user programs. (It's a dedicated computer for working with Octoprint)
2. I expect the octoprint web interface to respond promptly, and not peg cpu usage.
3. The system grinds to a halt and becomes unusable after awhile. Sometimes it's after an hour, usually around 15 minutes or so. Chromium window does not get redrawn. top shows chromium using most of the CPU %. Same thing happens with iceweasel 31.8.0. This issue did not occur in 1.2.2. I can ssh to my octoprint server when this issue is occuring, and the load is only around 0.15, so I know it's not an issue with server load. My dedicated Octoprint computer is two cores only. I have another system with 8 cores that can handle the Octoprint web interface better, but if I leave the web interface open in a tab for more than an hour, the system grinds to a halt. Closing the Octoprint tab immediately restores normal performance on the system. Sometimes it takes less time for the browser to hang the system.
4. 1.2.3
5. Prusa i3 and Marlin release branch, but N/A in this case.
6. Debian Linux 8 and Iceweasel 31.8.0 and Chromium 43.0.2357.65.
7. https://gist.github.com/foulowl/58bf137c2751ce3c76c0 (NA in this case)
8. serial.log is empty.
9. NA.
10. NA.

Issue occurs whether I have
[["set", "devel.webassets.bundle", false]]
or
[["set", "devel.webassets.bundle", true]]
although system is slower when set to false.

Issue occurs whether gcode is loaded into viewer or not, but it is slower when gcode is loaded in the viewer.

[GitIssueBot]

Hi @foulowl,

It looks like there is some information missing from your ticket that will be needed in order to process it properly. Please take a look at the Contribution Guidelines and the page How to file a bug report on the project wiki, which will tell you exactly what your ticket has to contain in order to be processable.

If you did not intend to report a bug, please take special note of the title format to use as described in the Contribution Guidelines.

I'm closing this one now as it is currently incomplete. Please feel free to comment here to request a reopen of this issue once you can provide all required information.

Best regards,
~ Your friendly GitIssueBot

PS: I'm just an automated script, not a human being.

[foosel]

Your bug report should always follow this template - copy and paste it completely into your ticket(!):

... Reopened.

[foosel]

octoprint.log is truncated. I need the full log - it contains information like what plugins are installed which are essential here, which is why I state in the bug template to never ever truncate it.

If there are no Javascript errors or anything, please state so in 9., do NOT just put "NA", that's for when it's a backend only issue, which this clearly isn't. Output of Chromium's Task Manager over time might be interesting here too (Shift + ESC I think)

Shot into the dark without much else to go on right now: Make sure "Autoscroll" is enabled on the Terminal tab (if it isn't, the communication log history will be stored on your client, which will make it consume more and more memory, and I'm not sure how both browsers do handle too much memory consumption). Another idea: Is the webcam tab focused when it happens? Do you frequently switch back and forth between it when it happens? You could also try disabling it by mis-configuring the webcam stream url for a test (note down the correct value).

[foulowl]

Sorry, looks like logrotate must be set up as a part of OctoPi and I didn't notice. Here is the reconstructed log file:
https://gist.github.com/foulowl/20f5cedf0534fa225236

Unfortunately when this issue occurs, the browser becomes unresponsive and I have to end up killing the browser process, so I haven't been able to gather any javascript info yet.

Ah interesting, I usually do turn autoscroll off. Let me try leaving it on and see if that helps.

Sometimes the webcam tab is focused. I'll have to test that and see if it has an effect.

Let me also make a note of memory usage as well.

Thank you!

[foosel]

If you turn auto scroll off, that sounds a LOT like the culprit, since that's coupled to the terminal cutoff at 300 lines.

I suggest enabling the task manager right along side the tab, before it happens. So you'll hopefully spot a memory usage increase or odd cpu behaviour before it becomes a crash-causing problem.

I also searched around a bit and it looks like that some people are responding memory-leak issues with mjpg streams on both firefox/iceweasel and chrome/chromium, although the tickets are quite inconclusive as to which builds, which versions etc. Still, might also be the webcam causing this.

[foulowl]

Here are the results of my testing:
I boot up my debian system, load octoprint in a single tab in chromium. I make sure autoscroll is on, set the tab to "Temperature" and let idle for 3 hours.

Upon return, chromium is unresponsive. It takes 30-60 seconds to change tabs. System load goes up to 2.0 or so. Top shows chromium using 150% cpu. Memory usage for the system goes up to 550 mb, which seems high but not horrible. top reports chromium is using 42% memory.

When I restart chromium, symptoms return immediately.

Load on server is around 0.5.

Thank you!!

[foosel]

When I restart chromium, symptoms return immediately.

That makes me suspicious. If you kill Chromium, restart it and the problem immediately reappears, it can't be a memory leak or some rouge processing in the web application but sounds rather like a general problem with that specific system.

Have you tested with the single tab you leave open being something like say, the Google homepage?

[foulowl]

Yes, I have left google up for long periods of time without issue. Tried doing that as a second tab as well, with octoprint in the first tab. The problem goes away when the octoprint tab is closed, and returns when the octoprint tab is reopened.

I would like to note as well that this issue does not occur in 1.2.2, but does in 1.2.3.

Thanks!!

[foosel]

Just to get the facts straight... The issue manifests on the two core system reliably, so far so clear. Does it also manifest on the eight core system at all (the "coming to a screeching halt" part)? The initial post can be read both ways, which is why I'm asking.

I have never seen this behaviour so far and also cannot reproduce it.

[foosel]

One idea... Very much a shot in the dark, but just judging from the fact that you say it happens with 1.2.3 but not 1.2.2, one thing that changed between those two versions was that the sorting of the filelist by size was fixed - do you happen to have it set to sort by size?

Could you also do me the favor and test out a fresh chromium browser profile?

[foosel]

For reference: the 1.2.3 change log and the related commits. You wouldn't happen to be familiar and comfortable with git bisect, would you? ;)

[foosel]

@foulowl ping ;)

[foulowl]

Ah sorry, I was out of town!

The issue manifests on the 8 core system as well, but in a milder form. My two browsers, both chromium and iceweasel, will slow down to the point of being nearly unusable. ie, it will take 10-15 seconds for a button click to have an effect, etc.

[foulowl]

Issue persists in 1.2.4 as well.

[foosel]

Can you try out my suggestions from earlier?

[foulowl]

Ah, sorry I missed those two recent suggestions.

Right now I am sorting by uploaded date (decending).

I launched chromium with --temp-profile and loaded up two tabs. One with google and one with Octoprint. After using Octoprint for a few minutes, top shows chromium using 80-140% cpu. When I close the Octoprint tab, that cpu usage immediately drops down to 10%.

I think maybe this system is on the lower end in terms of specs, so perhaps that is why this issue hasn't manifested yet.

[foulowl]

I'll let it idle and see if the system comes to a halt.

[foulowl]

Running into this issue on my 8 core system now. Octoprint web interface takes 30-60 seconds to load, webcam does not load at all, button clicks take 15-30 seconds to respond.

Load on server is around 0.3.

[foosel]

I still cannot reproduce this. Having several instances running in browser tabs (Chrome, Firefox, doesn't matter which) for hours on end, with webcam and without, printing and not printing, nothing.

I'm running out of ideas. A git bisect might help but since I can't reproduce it at all that would have to be done by you. Do you think you could do that? If so I could prepare a guide for you to follow to find which change between 1.2.2 and 1.2.3 causes it. But that's about the only idea I have left right now.

[foulowl]

I know almost nothing about benchmarking frontend slowdowns, so I guess it's time for me to learn something in that area!

I wonder if there is any way I can run some sort of profiling tools to see where the slowdown is occurring.

Why don't we keep this open for now, and I'll look into the profiling stuff on my own?

[foulowl]

A bit more info if this helps. I can reproduce on my phone. Galaxy S5, running either firefox or the stock browser. If I leave Octoprint up for too long, I get the message "a script has stopped responding..." etc. and it grinds to a halt.

This has also started happening on chromium on my 8 core machine. I don't want to waste your time of course, so perhaps it would make sense to call this issue "general performance improvements" and mark it a feature request since I think the most I have been able to pin down is "octoprint seems to be heavy on resource usage on the browser side".

I'm going to keep trying to gather more data. If I can reproduce on a Raspberry pi or vm, it might be easier for you to reproduce, since those machines are more standardized.

Thanks!!

[foosel]

OctoPrint being a bit heavy on the client resource consumption is actually intended - it is written to minimize calls to the server and take off as much work of that as it can in order to not overwhelm a 1st generation Pi that is currently trying to push gcode as fast to a busy printer as possible.

However, what you are seeing is certainly not what I'd consider normal behaviour. I have still not being able to reproduce it though. At all. Desktop browsers, phone, tablet, all absolutely fine.

You said it started with 1.2.3 for you. So the only way to figure out what caused this and at least getting an idea of the change that introduced that for you would be I'd you did a git bisect between 1.2.2 and 1.2.3, checking out various intermittent versions and seeing if you can reproduce the behaviour on each, telling git about it and letting it do its magic. I would love to do it myself, but it wouldn't make much sense considering I cannot reproduce it at all.

Maybe @Salandora or @markwal still have an idea, but I'm drawing a blank. Can't be a browser extension since you can reproduce with a fresh profile and on your phone, can't be a plugin since the only stuff you have installed is bundled or something I have too without reproduction, can't be the webcam since it also manifests when that one isn't focused, can't be the terminal since you enabled autoscroll.

One thing that just came to mind while writing this summary.... Just to make sure, if you downgrade to 1.2.2, does the issue go away? You said it manifested with the update to 1.2.3 but have you since rolled back to verify 1.2.2 is still good for you?

[Salandora]

I can only think of some caching problem right now.
I try to reproduce the problem later but I believe this will be a problem.

My first guesses would be Temperature graph, and Terminal window / Viewmodels.

I would need a few more Informations:

1. What hardware do you use for Octoprint
2. Is it connected by wireless or by cable.
3. If wireless wich wlan stick/controller so you use?

When the system hungs:
4. Can you check if data goes through your network
Maybe some sort of datalogger :-) that logs what goes in and out of the network card/stick :-)
5. Can you connect from another PC/Phone, is it responding there normally?
6. Can you ping your server?

[foosel]

Doesn't sound like anything caused by the server side of things for me (including network connectivity) since their client app completely crashes on its own.

[Salandora]

yup I thought of something like: slow connection -> blocking request -> hung

@foulowl
Also 1 more thing: Chrome has a Profiles tab in his developer tools, maybe you could run this and gather some more infos with it.

[foosel]

But then I'm all out of ideas myself so why not try that angle :)

[nophead]

I have known the gcode preview to bring Firefox to is knees, so I normally
run with it off. Could that be the issue here?

On 13 September 2015 at 18:21, Gina Häußge notifications@github.com wrote:

But then I'm all out of ideas myself so why not try that angle :)

—
Reply to this email directly or view it on GitHub
#982 (comment).

[foulowl]

Via iftop, looks like I'm seeing transmit rate of 4.7 Mb/s at idle. This seems pretty high to me. My guess is perhaps my web cam defaulted to the highest resolution, or it's not compressing before sending.

Is there a way to configure the bitrate (not of the timelapse, which I have disabled) of the stream itself?

Thanks!

[foulowl]

Tweaking the web cam resolution does not help. As an update, I can sometimes no longer view the octoprint web interface on my netbook at all (it times out trying to load), I can load it on my phone but the browser (both standard browser and firefox on android) stops responding after the web interface loads. The web interface takes ~30 seconds to load and render on my phone.

I can only get octoprint to load usably on my desktop with chromium or iceweasel. Even so it is somewhat unresponsive. It takes 5-10 seconds to switch tabs, respond to button presses, etc. If I keep the tab open for too long, the load spikes to above 100 and my system locks up completely. I can usually ssh into my desktop and do a "killall -9 iceweasel" (although that takes 5-10 minutes) and once done the system immediately returns to normal. Keep in mind the only tab I have open is octoprint.

There are no load or network connectivity issues with my pi at any time.

Also I wonder if I'm seeing the same thing as:
#1065

Even though you are unable to reproduce on your end, have you done any profiling to see what % of cycles are spent executing what code? Also could I potentially see an improvement in rendering time if I disable unneeded assets? I did try disabling the temperature graph, and have seen a noticeable improvement in performance. Things are still slow, but that has definitely helped. Thus, I suspect it's some kind of rendering or javascript issue that could be helped by profiling.

Thanks!!

[foosel]

In that case also try to disable the gcode viewer. The temperature graph does paint on a canvas, same as the viewer. So if there is some issue with canvas related stuff, that might give a hint. Also please make sure to test with the most recent version which contains a potential improvement for the ticket you referenced. That however didn't sound like the same issue you are encountering since it was only reproducable while printing and switching away from the browser or browser tab with OctoPrint. Still might be worth a shot.

I'll try to look into profiling the front-end code when I'm back at work in January.

[foosel]

Also please try the things listed above: downgrading, bisecting etc.

[foulowl]

Update: things seem to be running faster with the temp graph disabled. Let me try disabling the gcode viewer as well and see if that gives me a performance increase. Let me try upgrading to 1.2.8 as well.

I will also try downgrading and bisecting if I can. I'm going to get a second printer going here soon as I do use my current printer day to day and very strongly prefer to not have it go offline for very long.

Thank you! Sorry my symptoms here have been so vague. Again, Octoprint is an amazing tool and I greatly appreciate all the work you've put into it.

[foosel]

@foulowl does this still manifest with recent versions? Still no reproduction on my end and no other users reporting this.

[gmarsh]

Confirming it still exists here. Firefox gets nonresponsive on me if I've got Octoprint running in the background for an extended period of time. It chugs more and more when I'm using other tabs, until either firefox throws a "this webpage is slowing down your browser" error (I can't remember the exact text) or I get fed up with it and close the Octoprint tab.

Doesn't seem to matter if I've got the g-code viewer open, the temperature graph or the terminal.

I'll do more testing with this or haul some log files out if it helps. I'm using an older machine (Lenovo T410 laptop) and maybe it can't keep up with some client side scripting that Octoprint is making it do shrug

Edit: Octoprint 1.3.2 running on a classic Raspberry Pi model B.

[foosel]

@gmarsh can you try running in safe mode, see if that fixes things? I still cannot reproduce this at all. Multiple instances, running flawlessly for days in various browser tabs.

[RodneyMcKay715]

I can also confirm this with version 1.3.2. Firefox , Internet Explorer, Opera, Edge, Safari get slower and slower until the page takes around 30 seconds to load. Reverting to 1.2.2 fixed all issues i had. I would suggest you get back to that version. The new version does not seem to work very well. I was running it on a Raspberry Pi 3 and the Pi never used more than 5% cpu which slowed everything down. Energy Saving was off. WLAN or Ethernet = no difference at all. With 1.3.2 i had the problem that opening the webcam tab 5 times (terminal,webcam,terminal.....) causes the pi to reboot or lock up entirely.
Since i reverted to the old version every issue is gone.

[JohnOCFII]

No obvious issues for me running OctoPrint 1.3.2 on a Pi 3 and watching 4-6 hour prints with both Safari 10.0.3 on MacOS 10.12.3 or Mobile Safari on iOS 10.2.1.

I usually spend 90% of the time in the Control tab watching the camera, and 10% of the time in the GCode viewer tab.

[maukcc]

We shipped a machine with Octoprint to a customer. It worked perfect in our office, and it has these symptoms in their office.
The only thing I can see that has changed, is the IP address (by dhcp) and the fact that their office uses multiple subnets (255.255.0.0).
I have no idea if this has anything to do with it, but just wanted to mention it, as I am also unclear what is happening.

[foosel]

@maukcc any reverse proxies in the mix? Because that definitely sounds like either that (potentially nuking websockets so that it has to fall back to polling which is slower) or general network throughput issues (can't be fast if there's huge latency between server and client).

[foosel]

Closing this. If there are any issues like this still happening with the current code base, feel free to open a new ticket. It doesn't make sense to keep a ticket open that looks at issues with a version that's over 2.5 years old at this point.