-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OctoPrint reports "Exception while pushing current data: [...] dictionary changed size during iteration" #1951
Comments
To be honest, I don't think the observed exception is the reason for the printer behaviour you observed. What the behaviour you observed sounds like is more a very specific kind of garbled communication I've seen on my own printers from time to time. A move command gets send, the firmware receives it in a garbled state (can be due to noise on the serial line, or due to brown out issues or something like that). The checksum happens to match regardless (it's only an XOR, nothing fancy). The garbled line gets parsed and the parsing result triggers a move to X/Y max/min. If you don't have (soft) endstops active in that case, stuff goes south fast. The reason that I'm suspecting something like that rather than the exception you observed being the cause here is that there is simply no possible way at all for that exception to trigger random print head movements. The particular thread its triggered in doesn't even have access to the printer communication layer (and apart from that, OctoPrint never sends out G0/G1 on its own or touches G0/G1 commands and swaps coordinates in them or anything like that unless you have third party plugins in place that do that). With that being said though, the exception looks like there's still a possible race condition in the code that needs to be addressed to prevent update issues on the frontend. |
Hmm, that actually makes some sense. We have been having brown outs in the
area recently. Really do, need to put the setup on a proper ups insted of
just surge protection. Mater of fact, after I successfully deduced the
layer the print failed at and resumed. it I had a full power failure 30
minuets latter just 5.minuets etl before it finished.
Is there any additional info I can provide for locating the front-end race condition?
|
If possible, the file with which you ran into this might help. I spent a couple of hours yesterday looking at things but I have currently no idea how it was possible for the data in question to change during iteration, considering its created specifically for triggering updates as far as I could tell from the code. So being able to recreate an environment as close to original runtime as possible would maybe help, and that includes the printed file. Oh, and information on whether you had temperature offsets configured or not :) |
sadly it seems I've lost that gcode file. I though I saved a copy first but It looks like I overwrote it when I made the modifications to resume the print. I can attempt to reslice with the same settings but as it was done in cure with two separate stl's I can't guaranty the positioning on the bead and thus the gcode would be the same. and I don't' believe we've configured and temperature offsets for the printer. I take it this is not a state that can't be avoided with a shallow/deep copy or thread mutexes? -not that either of those are easy or trivial to implement, but, if you can ensure it's not a case that can come up, you potentially never need to find that actual cause / it will become painfully obvious how it can up during implementation of the "fix". |
Thing is that the data that was in the process of being deep copied when it fell on its face in your stack trace is - as far as I could tell in my lengthy analysis yesterday - the only copy of said data, freshly instantiated (from immutable scalars) just for this status update, never shared with any other bits of the code. I could do copies earlier in the data collection, but as long as I don't understand how that data could even be modified in the first place (considering that there's only one single reference to it and it's freshly created too) that would feel more like slapping bandaids on a possibly broken leg. |
hmm,... that is weird. perhaps I should take a look at the code myself. I'm reminded of a bug I encountered about 3 years ago with similar properties in Python 2.6. I was working on embedding Python in c at the time and it was the result of a misunderstanding I had about the c python api. the nature of the source might be similar. I'll report back here in about 2 hours If I find anything. EDIT: ... And so, I think it's time to blame angry ghost pixies in the hardware flipping bits that should not be flipped. |
I was just about to ask if you'd found anything and then I saw your edit. Sorry for the lack of responding to that so far, simply didn't see it until now :) At least I'm now not the only one who's stumped about this ;) I just took another long hard look at the huge diagram I painted on my whiteboard the other day, and I still have not even an inkling of an idea how you ran into this exception. And I hate to blame bit flipping pixies. Then again - that's about the only possibility I see myself. I'll mark this as unreproduced I guess, in the "hopes" that maybe someone else will run into it and be able to provide some more data to narrow down things. But something tells me that this might never happen. Problems like this are incredibly frustrating IMHO. |
2017-09-21 09:27:53,067 - octoprint.printer.standard - ERROR - Exception while pushing current data |
@jmdearras what did you do when that happened? Just that traceback alone is the equivalent of "it's broken, fix it" and sadly not very helpful 😔 |
running g-code I had run before and since. I was not looking when it failed. Raspberry Zero W and Creality CR-10 |
@jmdearras can you provide that file? |
Hi, Log: |
This does sound like something different. And from your log, OctoPrint doesn't actually crash. The connection to your printer might crash though. Could you provide the full log file please, not just the above excerpt? And also the files with which that happened? Also this:
looks like your machine lost its connection to the internet. Not sure if related though. |
A follow up on the error @jmdearras ran into. I also got the dictionary file error. 2017-12-28 09:32:11,371 - octoprint.printer.standard - ERROR - Exception while pushing current data Here is the gcode I was running. This was a multi-day print, and was about 97% done. Let me know if there is anything else I can provide to help. |
Ok, so, EVERYONE RUNNING INTO THIS, provide:
This ticket is still open because it has been impossible to reproduce or even get a remote idea of what might be happening here from just looking at the code. What I'm absolutely sure though is that I won't get any chance to figure out what is happening if people don't provide logs and context - so if you want to see this fixed (I know that I certainly do) please provide all of the above. Thanks :) |
which logs?
…On Thu, Jan 11, 2018 at 1:23 PM, Gina Häußge ***@***.***> wrote:
Ok, so, *EVERYONE RUNNING INTO THIS*, provide:
- octoprint.log
- the file you were printing when it happened (if you were printing)
- as much information as you can on what was going on when it happened
- print ongoing or not, web interface open or not, anything you can think
of with regards to context really
This ticket is still open because it has been impossible to reproduce or
even get a remote idea of what might be happening here from just looking at
the code. What I'm absolutely sure though is that I won't get any chance to
figure out what is happening if people don't provide logs and context - so
if you want to see this fixed (I know that I certainly do) please provide
*all* of the above. Thanks :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1951 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAksiReuTyaq7N-X-p6NianVlOnwRFkfks5tJlGSgaJpZM4NvKku>
.
|
As listed up there (first item, but don't miss the others either please): |
In my case it never happened again... fingers crossed. |
I had the same issue multiple times. First time short after Octoprint 1.3.6 Update. Then I reinstalled octoprint from scratch (1.3.4 and updated directly) and only installed absolute base plugins I Need to have. Still same issue. My Anet A8 just resets when it happens and stops print where it is. full actual log: First I had Marlin 1.1.4 installed with some changes in Speed and accelleration but nothing in communication Settings. Recently I updated to 1.1.8 Marlin and today it happened again during an 15 hour print. Then the web Interface is not accessible and I just got an error page of the web Server where to find the log and how to restart octoprint through Shell. SSH is still accessible and also no Connection error logged in my router at this time. when i restart octoprint over SSH, everything works fine again like normal restart. the log stops today at 8:32, exactly 2 minutes after I left the house and after about 8 hours of print time. This is the gcode file I was printing. Generally if i restart the same file, it works fine so it cannot be the gcode itself or at least not alone: Just tried to manually simulate this Situation but Now the Printer continues. As the case opener, I cannot print multiple days in safe mode do check if it happens again because this is very rarely. I had a 20 hours print recently that worked fine. This issue is completely unpredictable. Last crash was nearly 2 Weeks ago. As Addition to my Hardware Setup: Unfortunately it EVERYTIME only happens when I'm at work or I sleep at night :( |
@b14ckyy Thank you for the detailed data! Maybe that will help unearth something. I sadly still haven't been able to see it even once myself. Long stretch but: Is anyone else running into this running Printoid Premium? |
I just noticed this same error and i'm using Printoid Premium. I'll keep an eye on the logs to see if it happens again. |
Since the last time, i reported this, it happened 2 days ago again. A very long period unfortunately. |
Just to mention I receive the same problem since about a Month. Happened about 4 times now while using Printoid Premium. Nothing obvious in the Logs except the Snipped that was already provided here. I have the feeling it happens mostly with recently uploaded gcode where no estimated printtime is calculated yet. Don´t know if that helps but if Printoid could be the cause I will try to print my next files directly from Octoprint Frontend without using Printoid. Btw: I´m running Octoprint Version 1.3.6 and Printoid Version 9.11 which was just updated about an hour ago and still have the same issue. |
Last time it happened without any API application running. I used Simplify 3D to slice and copied the file via Browser front end. I did not start Octoprint or Cura. After about 10min of print, the printer suddenly stopped and has done a reset. Then Octoprint was in freeze state again, like the other times. I don't think it is an API issue. |
Well, that at least sounds like it rules out Printoid as the common denominator then. Which is bad because then I'm back at the beginning in terms of trying to make heads and tails of this ;)
Does a printer reset always precede this for you? Or are you saying this is another symptom? Because this issue here usually should not make the printer reset. |
I'm not sure. When this issue happened in the past, I never was onsite. This was the first time I saw that. But the Anet A8 board generally resets if USB serial connection is lost (unplug cable), even when printing from SD Card. After the printer rebooted, i had the same state as mentioned above in my long log, tzhat octoprint is not running, when opening the website. I sent you a screenshot a few days ago on Twitter. |
It's a bit embarrassing that I only notice that now, but @b14ckyy, I don't see the error this ticket is about in your log from January 18th, and the screenshot also rather looks like the whole server crashing, which does not match the error so far described in this ticket. Your's however is another incident of a crashing server with the OctoPrint Anywhere plugin logging itself senseless beforehand (for the other see #2424), and to be honest I'm starting to suspect that something might be up with that plugin. Could you please try the suggestions outlined in #2424 and report back there on the results?
Please don't do that. Always gather information related to a ticket in the ticket, otherwise it becomes completely impossible for me to keep track on things (and also to stay sane). Out of curiousity, I simulated this error now by adding an exception at the point this one arises (still due to completely mysterious reasons). What I'm seeing is an ongoing continuing print but an unresponsive web interface. No crashing server, no failed print. So - this error here should never make the whole server crash/become unreachable, if at all it should only make the web interface stop updating. If full blown server crash is what you are seeing, you are seeing another problem. |
I can only talk for myself but when this particular error happen to me, the printer stops printing but apparently doesnt do a reset because heatbed and nozzle stay on temperatur. It just appears that it does not receive any gcode anymore from the server. |
Again, that sounds like a different issue. We are looking at something that causes this log entry:
If you can't even reach your Pi via SSH or even Ping anymore, it's something a couple layers below OctoPrint that's causing issues here. Could be an insufficient power supply, could be that the machine is simply overwhelmed (the Pi Zero W is not recommended due to an issue with its built in Wifi causing significant CPU load). |
Is not a different Issue because I get exactly the same error in the log. Thats how I just found this ticket here.. Maybe the actual problem leads into a heavy overload of the Raspberry so it goes down. Don´t know. Here a Snipped from two days ago: EDIT: Copy Paste didnt really work so attached the full log |
It is technically impossible for this log entry to cause your Pi to become unresponsive to a ping. It might be that the fact that you are seeing this log entry is a symptom of the reason that makes your Pi become unresponsive, but it cannot possibly be the cause. The very fact that it is captured and logged the way it is means it gets handled gracefully. |
Maybe you are right, I´m not a developer but and Admin and I can only tell you what I observed. Maybe I have also more then one problem and they just show similar symthoms. But for sure I have the same Log entry which causes the printer to stop printing. |
I have used a Pi Zero WiFi for half a year without an issue.
Jim
…On Tue, Mar 6, 2018 at 9:58 AM, Gina Häußge ***@***.***> wrote:
Again, that sounds like a different issue. We are looking at something
that causes this log entry:
2017-12-13 22:34:42,385 - octoprint.printer.standard - ERROR - Exception while pushing current data
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/OctoPrint-1.3.6-py2.7.egg/octoprint/printer/standard.py", line 157, in _sendCurrentDataCallbacks
try: callback.on_printer_send_current_data(copy.deepcopy(data))
File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/lib/python2.7/copy.py", line 256, in _deepcopy_dict
for key, value in x.iteritems():
RuntimeError: dictionary changed size during iteration
If you can't even reach your Pi via SSH or even Ping anymore, it's
something a couple layers below OctoPrint that's causing issues here. Could
be an insufficient power supply, could be that the machine is simply
overwhelmed (the Pi Zero W is *not* recommended due to an issue with its
built in Wifi causing significant CPU load).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1951 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAksicu8odpA0lcNzTCpKUAnx3sHvWuQks5tbqQFgaJpZM4NvKku>
.
|
@jmdearras if it works for you, great. It certainly wasn't reliable at all in my own tests and hence I do not recommend it. |
Attempt to narrow down on #1951. To be on the safe side, the use of frozendict here can be disabled by settings devel.useFrozenDictForPrinterState to False in the config.
It was more reliable than the Pi III talking to a CR-10. I have a
collection of half-prints from that combination, It could not maintain a
USB connection, tried several Pis and cables.
…On Wed, Mar 7, 2018 at 2:50 AM, Gina Häußge ***@***.***> wrote:
@jmdearras <https://github.com/jmdearras> if it works for you, great. It
certainly wasn't reliable at all in my own tests and hence I do not
recommend it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1951 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAksicnnchrCd4fCTP2KNc3hWsR3NqaNks5tb5FJgaJpZM4NvKku>
.
|
Possible break-through I think! All of you who have encountered this in the past - could you please check if you also have the "Detailed Progress" plugin installed? That is modifying the progress data (which it actually shouldn't) and that would explain this error. |
I know for sure I have it installed. |
me too it’s installed.
|
The plot thickens... Chances are good this will be fixed in 1.3.7 |
Great to here :) Thank you for your investigation |
Yes, I have it installed.
…On Thu, Mar 22, 2018 at 8:24 AM, iRazoR ***@***.***> wrote:
Great to here :) Thank you for your investigation
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1951 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAksiRsdHq1tUBFSuBfwYX3LLdo4BYCoks5tg5gPgaJpZM4NvKku>
.
|
Closing as 1.3.7 was just released and by now I'm very sure that what 1.3.7rc1 unearthed was indeed the cause. |
What were you doing?
Printing a custom raspi and IR blaster case slices in Cura 2.5.0 and uploaded to octoprint. on an Anet A8
about 6.5 hours into a 7 hour print the extruder head on the xasis just started moving all the way to the right binding on the right side z carriage and losing its place while the rest of the print continued. carriage stayed on the right and never moved back continuing to try to move to the right. This continued for approximately 1 minute before it got turned off. this information is what could be gleaned from the last few frames of the time-lapse and the amount of filament extruded below the print head (about a meter of extruded filament from the 0.4 mm nozzle).
about a hour latter I had time to start it back up and diagnose so you;'ll see that timestamp gap in the log file.
What did you expect to happen?
the print to execute gcode correctly
What happened instead?
apparently, there was an error sending the gcode to the printer according to the log.
Did the same happen when running OctoPrint in safe mode?
Unfortunately, I have neither the time nor material to test this error in safe mode.
The nature of the problem makes it unlikely to be repeatable.
in retrospect I'v had a similar issue in the pased on octoprint 1.3.2 but at the time I assumed it was a problem with the zprobe ceatching on the print or somthing (a possibility that has since been removed as an option.)
Branch & Commit or Version of OctoPrint
OctoPrint 1.3.4 (master branch)
Operating System running OctoPrint
OctoPi
$ uname -a
Linux octopi 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux
on a raspi 3
Printer model & used firmware incl. version
ANet A8
Marlin SkyNet3D-V2.3.2 (Github)
Browser and Version of Browser, Operating System running Browser
Google Chrome 58.0.3029.110 (Official Build) (64-bit)
Link to octoprint.log
gist contains multiple files:
the full log, the log truncated to the last two boots, and the relevant error section
a sequence of screen grabs from the time-lapse (10 seconds apart)
https://gist.github.com/Ryex/d7130aeb9a2a5154339a12afc95fa680
Link to contents of terminal tab or serial.log
sadly I do not have a serial.log nor can I get the terminal tab as the printer and pi were on the same power source and the power was pulled to prevent further damage to the stepper motors. I regret this in retrospect as it will make debugging (and finding the line of gcode where the print failed) extremely hard.
Link to contents of Javascript console in the browser
N/A
Screenshot(s)/video(s) showing the problem:
Included in log gist
I have read the FAQ.
The text was updated successfully, but these errors were encountered: