New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Old OctoPrint ".egg" folders in easy-install.pth cause outdated JS files to get loaded in the frontend #2875

Open
ctgreybeard opened this Issue Nov 6, 2018 · 28 comments

Comments

7 participants
@ctgreybeard

ctgreybeard commented Nov 6, 2018

What were you doing?

  1. Upgrade to 1.3.10rc1
  2. Attempt to connect to printer

What did you expect to happen?

Printer connects

What happened instead?

The Connect/Disconnect button says "Disconnect" but the printer is not connected. The port, profile and baud selections are disabled.

Did the same happen when running OctoPrint in safe mode?

Yes

Version of OctoPrint

1.3.10rc1

Operating System running OctoPrint

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian

Printer model & used firmware incl. version

Original Prusa i3 MK2 Kit

Browser and version of browser, operating system running browser

Firefox 64.0b6, Chrome 70.0.3538.77

Link to octoprint.log

https://pastebin.com/L0ssvBQx

Link to contents of terminal tab or serial.log

terminal tab is blank and disabled

Link to contents of Javascript console in the browser

https://www.dropbox.com/s/ypduzze6kimuuga/Screenshot%202018-11-06%2010.13.29.png?dl=0

Screenshot(s)/video(s) showing the problem:

https://www.dropbox.com/s/62oj3l53wl8y2qp/Screenshot%202018-11-06%2008.36.05.png?dl=0

I have read the FAQ.

Yes

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

From your log, it doesn't even look like the connection request is making it through. Please try if clearing your browser's cache does anything. I still cannot reproduce that here, happily connecting and reconnecting to a printer. Do other things such as opening the settings, changing and saving them work as expected? Does stuff only "lock up" when you click the Connect button or is something broken before? Can you attempt to connect and then share a screenshot of the contents of your Network tab (in the debug console where the JS console is also located):

image

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard your Pi is still reporting undervoltage:

2018-11-06 10:02:15,293 - octoprint.plugins.pi_support - WARNING - This Raspberry Pi is reporting problems that might lead to bad performance or errors caused by overheating or insufficient power.
!!! UNDERVOLTAGE REPORTED !!! Make sure that the power supply and power cable are capable of supplying enough voltage and current to your Pi.

Based on my past experience with brownout issues on those things (which can be summed up as: any weird behaviour is possibly), I wouldn't rule out that this is the reason here unless that's taken care of first and the issue definitely persists.

@ctgreybeard

This comment has been minimized.

ctgreybeard commented Nov 6, 2018

I connected directly to Octoprint (not through haproxy) and the problem is the same. In that attached screenshot I clicked the "Disconnect" button. This is a capture of the Network debug tab.

https://www.dropbox.com/s/5w2ufgli5w17xrs/Screenshot%202018-11-06%2010.34.35.png?dl=0

@ctgreybeard

This comment has been minimized.

ctgreybeard commented Nov 6, 2018

According to the vcgencmd the Pi WAS throttled (this is not usual for my system) but is not currently throttled. (vcgencmd get_throttled returns throttled=0x50000). If it were currently throttled it would be throttled=0x50005

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

Based on that it's sending the request but the push updates of the state change don't make it through. Next thing to try:

  1. Open OctoPrint in your browser
  2. Open the debug console, go to the network console
  3. Do NOT close the console. Reload OctoPrint in the browser.
  4. The network tab should have a ton of stuff in it now. Click the connect button. Make a screenshot of everything and also the JS console.

My current assumption is that for some reason the websocket connection isn't receiving data. It should be visible on the tab as something under /sockjs.

Is there the possibility for me to take a look at that instance myself? E.g. through some SSH tunnel or something?

@trendelkamp

This comment has been minimized.

trendelkamp commented Nov 6, 2018

Since i can not go back to stable too (check my issue), i would highly doubt that this problem is related to a power supply.
But i replaced the power supply. So the undervoltage is gone.
Problem still exits.

If you want i can let go on my system. Teamviewer or so
1
2

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

Teamviewer or the like would be nice, I have to go AFK by 17:30 though. Best hit me up at gina -at- octoprint.org to exchange some contact details.

@ctgreybeard

This comment has been minimized.

@trendelkamp

This comment has been minimized.

trendelkamp commented Nov 6, 2018

Teamviewer or the like would be nice, I have to go AFK by 17:30 though. Best hit me up at gina -at- octoprint.org to exchange some contact details.

Send you a mail with contact details.

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard do you have access control enabled or disabled? @trendelkamp has it disabled

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

So, the debug session with @trendelkamp revealed that there is a still a bug in the new Force Login plugin that doesn't handle disabled ACLs correctly. Disabling the plugin and restarting OctoPrint made it work again for @trendelkamp.

@ctgreybeard Based on your screenshot up there, it looks like you do have ACLs enabled though. So your issue is probably a different one, but potentially caused by a similar core problem. I'll ping you regarding access to the instance so I can poke around a bit. Might not make it today though, got a hard cut at 17:30ish.

@ctgreybeard

This comment has been minimized.

ctgreybeard commented Nov 6, 2018

OK. I'm not sure what ACL's you refer to but take your time; I have nothing pressing ...

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard in your case for some reason the web interface is loading old javascript which lacks some stuff that's now needed. I tried a hard reload but that didn't bring me new JS either. Is there any kind of proxy in between that could be the reason here?

For comparison, this is how it looks in your instance:

image

And this is how it should look:

image

All the new logic added in 1.3.10 is missing (which also explains why I didn't get a warning about logging in from a remote IP).

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard After trying a different URL to access the JS in question and still getting the wrong one I've come to the conclusion that what's on disk is wrong. The update log doesn't indicate any reason for that though.

Could you fetch /home/pi/.virtualenvs/octoprint/lib/python2.7/site-packages/octoprint/static/js/app/viewmodels/loginstate.js from disk and send it over to my mail address?

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard Thanks, I got your mail - now it's getting really weird, because the js file you sent me is the correct one. So the data on disk appears to be correct, but what the webserver delivers isn't.

I did a quick backup and took a look into the generated webassets. And those are incorrect. So please delete the generated webassets and their cache:

rm -r /home/pi/.octoprint/generated/webassets
rm -r /home/pi/.octoprint/generated/.webassets-cache

Then restart OctoPrint. That should force things to get regenerated - they should actually get regenerated on every server start, but maybe something's off there with permissions or such...

Would be interesting to see what happens then.

@ctgreybeard

This comment has been minimized.

ctgreybeard commented Nov 6, 2018

No joy.

I did start up the virtualenv and printed the sys.path just to see what it said. I posted the result here: https://pastebin.com/CqJ2iG3z

One thing that popped out at me was the occurrence of '/home/pi/.virtualenvs/octoprint/lib/python2.7/site-packages/OctoPrint-1.3.6rc2-py2.7.egg' in the list ... seemed out of place.

@OutsourcedGuru

This comment has been minimized.

OutsourcedGuru commented Nov 6, 2018

Same here. After adding 1.3.10rc1 the Connection side panel thinks that it's connected (and is expanded) but nothing else seems to work.

Linux charming-pascal 4.14.62-v7+ #1134 SMP Tue Aug 14 17:10:10 BST 2018 armv7l GNU/Linux
Robo C2
OctoPrint 1.3.10rc1, OctoPi 0.15.0
Marlin 1.1.7-C2 (Github)
Nothing in the Terminal tab
Safe Mode (same symptoms)
No access control
Earlier undervolt condition after first use. I added a dedicated Raspi adapter and that error has gone away.

screen shot 2018-11-06 at 12 46 13 pm

octoprint-5.log.txt

Interestingly, the Conky desktop on the TFT seems to be somewhat happy. It's managed to query the temperature of the hotend, for example.

screen shot 2018-11-06 at 1 18 30 pm

@foosel

This comment has been minimized.

Owner

foosel commented Nov 6, 2018

@ctgreybeard if that didn't help either, I'm a bit at a loss right now. That syspath entry looks fishy though. Any chance I could get SSH access to that machine? Or alternatively a copy of the image to poke around in?

@OutsourcedGuru do you have access controls enabled or not? If not that's the issue that @trendelkamp ran into, fix for that is already pushed and will be released with rc2, temporary workaround is to disable the force login plug-in. If you do have access controls enabled - would it be possible for me to get access to this instance in some way so I can try some debug steps?

@OutsourcedGuru

This comment has been minimized.

OutsourcedGuru commented Nov 6, 2018

I got it. Turn off the new Force Login plugin if you don't have Access Control turned on. I'm now back in business with all my third-party plugins again.

Oh... :reading what you just read: ...yeah, I just figured that out.

@cmock

This comment has been minimized.

cmock commented Nov 6, 2018

I guess I'm the next one with a problem that fits the general description.

I'm running klipper and get the following in serial.log:

2018-11-06 23:22:51,426 - Connecting to: /tmp/printer
2018-11-06 23:22:51,448 - Changing monitoring state from "Offline" to "Opening serial port"
2018-11-06 23:22:51,459 - Connected to: Serial<id=0x6a1dad90, open=True>(port='/tmp/printer', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=20.0, xonxoff=False, rtscts=False, dsrdtr=False), starting monitor
2018-11-06 23:22:51,460 - Changing monitoring state from "Opening serial port" to "Connecting"
2018-11-06 23:23:31,515 - There was a timeout while trying to connect to the printer
2018-11-06 23:23:31,527 - Changing monitoring state from "Connecting" to "Offline"
2018-11-06 23:23:31,534 - Connection closed, closing down monitor

I tried clearing the cache and disabling the forcelogin plugin, starting in safe mode to no avail. The only change I made was upgrading to 1.3.10rc1 from 1.3.9 (after a successful print so it definitely worked before).

I verified that klipper is indeed working:

$ ls -l /tmp/printer 
lrwxrwxrwx 1 pi pi 10 Nov  6 23:07 /tmp/printer -> /dev/pts/1
$ socat /tmp/printer -
m115
ok FIRMWARE_VERSION:v0.6.0-529-gaa693fb FIRMWARE_NAME:Klipper

Linux octopi 4.14.71-v7+ #1145 SMP Fri Sep 21 15:38:35 BST 2018 armv7l GNU/Linux
Raspberry Pi 3
OctoPrint 1.3.10rc1, OctoPi 0.15.0
Klipper v0.6.0-529-gaa693fb

@bonkas

This comment has been minimized.

bonkas commented Nov 7, 2018

Just leaving a message to confirm Disabling Force Login also resolved this issue for me also.

@foosel

This comment has been minimized.

Owner

foosel commented Nov 7, 2018

@cmock that looks rather like the same issue as reported in #2872 - fix is prepped and ready to roll out with 1.3.10rc2.

@bonkas now the important question: did you have access control enabled or not?

@foosel foosel changed the title from Unable to connect to printer after upgrade to 1.3.10rc1 to [1.3.10rc1] Connect not possible with enabled access control Nov 7, 2018

@foosel

This comment has been minimized.

Owner

foosel commented Nov 7, 2018

@ctgreybeard thanks for the SSH access. I took a look and experimented a bit, and it indeed looks that the 1.3.6rc2 install on the sys.path was to blame. I edited easy-install.pth and removed that line (made a backup though, .bck suffix), restarted OctoPrint and now the static files are as they should be.

The question is how that old version made it onto the sys.path... it could be a left over from back when OctoPrint was still updating itself via python setup.py install instead of pip, but I've so far never seen that scenario.

In any case, whatever it is, it's not an issue with 1.3.10rc1. It's possibly a general issue though with the update script and the way it currently works, so I'll have to investigate that a bit further. Not sure if I'll dare to make any changes to that that I might uncover still part of 1.3.10 though - it would go against the code freeze for 1.3.10 that started with the release of the RC.

@ctgreybeard

This comment has been minimized.

ctgreybeard commented Nov 7, 2018

@foosel

This comment has been minimized.

Owner

foosel commented Nov 7, 2018

They aren't necessary, and a simple rm should suffice. They should also still be left overs from the old setup.py install updates. pip cleans up after itself, uninstalls old versions and such. setup.py install didn't/doesn't do that.

@bonkas

This comment has been minimized.

bonkas commented Nov 13, 2018

@foosel Access Control is disabled in my case. Apologies for the late reply.

@foosel

This comment has been minimized.

Owner

foosel commented Nov 13, 2018

Considering this to be more of an environmental issue, not 1.3.10 specific.

@foosel foosel changed the title from [1.3.10rc1] Connect not possible with enabled access control to Connect not possible with enabled access control Nov 13, 2018

@foosel foosel changed the title from Connect not possible with enabled access control to Old OctoPrint ".egg" folders in easy-install.pth cause outdated JS files to get loaded in the frontend Dec 5, 2018

@foosel

This comment has been minimized.

Owner

foosel commented Dec 5, 2018

FWIW, I've now added this to the FAQ.

Still thinking about some way to resolve this automatically. Just a bit worried that might break things in some very rare corner cases...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment