[Request] Automatic reconnect and resume print

[llama]

Like it or not, the biggest problem users have with Octoprint is random disconnects. I would not be surprised if more than 50% of users experience this issue at some point. Even with the official Raspberry Pi power supply, random factors such as fluctuations in wall voltage can cause a disconnect.

There should be an option in "Serial Connection" to automatically reconnect to the printer and resume the print. This would involve:

On serial disconnect:
- Reconnect
- Home
- Set temperatures to values at disconnect
- Set position to value at disconnect
- Resume print file gcode

I'm not sure how tricky it will be to store temperature and position values. There may be other printer state that I'm forgetting (e.g. feed rate), but it probably doesn't matter.

The new FAQ entry on disconnect issues - https://github.com/foosel/OctoPrint/wiki/FAQ#octoprint-randomly-loses-connection-to-the-printer-with-a-serialexception

Disconnect Issues:
#1603
https://github.com/foosel/OctoPrint/issues?utf8=%E2%9C%93&q=is%3Aissue%20device%20reports%20readiness%20to%20read%20but%20returned%20no%20data

[Erutan409]

@llama, as someone who suffers from the same problem, I'm confused to why @foosel should have to compensate for a host inadequacy? If she or anyone else was to add in support for this, it merely band aides a pretty significant issue on the raspberry pi's power supply and sets a precedent for supporting more workarounds for other one-off's.

It should be a troubleshooting opportunity for the user. Not a headache for someone else to solve when the usual cause for this has been discovered and reported on how to correct...

[llama]

@Erutan409 of course you have this problem. Everybody has this problem.

Yes this is a band-aid. No, she doesn't need to. But I think this feature would improve users' experience with Octoprint more than any other single feature.

In addition, I would suggest the very easy change of linking to the new FAQ entry directly from the error (Unexpected error while reading serial port, please consult octoprint.log for details). Also, I believe that a fantastic power supply is insufficient in any location with less than perfect line voltage quality. Probably also need to solder a big electrolytic capacitor on the 5v rail or something.

[Erutan409]

So, to be clear - I'm not looking to get into an argument about this. But, I find it difficult to get on-board with blanket statements such as "everyone having the problem." That seemingly doesn't seem to be the case. If someone can find a wall wart that provides sufficient voltage regulation and link it in the wiki, I think that'd be a far better solution than to alter a stack to accommodate an issue that's isolated to one hardware platform.

If anything, the raspberry pi community should be tackling this issue. Not individual software package developers.

[llama]

Fair enough. Sorry for the provocative blanket statement--I don't have much data here except that I've seen many Octoprint instances running into this problem. As far as the system design principles go, I don't think there's anything wrong with making the software resilient to common failures in other parts of the stack.

I agree that a good power supply recommendation would be helpful, since even the official Raspberry Pi power supply seems to have this problem. I probably should have started this as a brainstorming thread instead about how we could help mitigate this issue (even if that means better helping users help themselves).

[Erutan409]

Fair enough.

I mentioned in another issue that I'm working on a potential solution with instructions that might solve the issue. Although, it may require some moderate soldering; possibly.

[foosel]

So here's why this isn't built-in: It covers up serious issues. Like, really serious issues that WILL nuke your print jobs and WILL make users unhappy. If the connection to the printer isn't reliable, OctoPrint can't work reliable either. Users need to solve these kinds of issues instead of just going "meh" and restarting their failed print jobs. They are bound to fail again unless the underlying issue is tackled, be it an unsufficient power supply, a broken USB cable or anything else commonly causing these kinds of problems.

I fear that if I add something like an automatic reconnect, I'll drown in tickets like "OctoPrint didn't finish my print", simply because the disconnect outside of OctoPrint's control was never observed, and that will frustrate users just as much if not more ("well, it should not simply reconnect then but instead show me the error, shouldn't it?"). I really feel uncomfortable about this being an OctoPrint core feature. It would probably be easy to build it as a plugin though, simply detecting the error state (potentially parsing for serial errors), and then doing all that you proposed above.

[nophead]

Also most control boards reset when the USB connection is made. That means any moves in the planner buffer will be lost, so even if OctoPrint reconnected, homed, restored the heaters and last position it sent you would still have some of your print missing. Also a lot of machines home towards the bed, which would be really bad part way through a print.

I have four machine running from RPIs for many years and I never get a USB disconnect. They all power the RPI from the same PSU as the machine, eliminating ground loops. I either feed the power in via the GPIO connector or just use a cut down USB cable a few inches long because most micro USB cables have too much resistance.

I don't use fancy USB cables with ferrite, I just use short ones because the RPI is right next to the controller.

[foosel]

Apparently my reading competency suffered from the long weekend and I overlooked the bit about "auto resuming the print job". As @nophead already pointed out, you can completely forget about that.

Even if the printer doesn't reset on reconnect, there will be no way for OctoPrint to reliably reconstruct the position it was at - that working even remotely depends on OctoPrint knowing beforehand that a pause (which this effectively is) will happen and querying a bunch of information from the printer to be able to reconstruct state after it ran through all planned moves, and even that is incomplete thanks to printer firmware simply not reporting back all information that would actually be needed for a full reconstruction of state (no data on any but the currently selected extruder, no info which extruder is selected, no info on the feedrate, the flow and feedrate multipliers, whether relative or absolute mode is active, fan speeds, etc etc). Add to that the fact that in case of a "sudden interruption" OctoPrint knows how far it got in the file it's streaming, but it doesn't know at all which of the lines it sent to the printer were actually already executed by it (thanks to how motion planning works the firmware will acknowledge lines sometimes WAY before them actually getting turned into motion), so it can't really resume from the exact point and would have to basically guess.

This is also the reason that even though OctoPrint does record the print head position on pause after waiting for all moves to finish and does make this information available in pause and resume GCODE scripts, it doesn't ship with anything resembling an intelligent pause or resume script at the moment. It simply would only work in a small handful of well defined scenarios (e.g. no printing from SD, not printing with multiple extruders, not printing with active flow or feedrate modifiers and also without any manipulation of the fan) and hence produce more errors for users than not.

tldr: forget about resuming from unplanned interruptions in print jobs, even resuming from planned pauses is already next to impossible thanks to firmware not producing sufficient information on its internal state and on top of that the majority of controllers resetting on reconnect.

[lalalandrus]

I would like this functionality for a completely different reason. While continuing prints may not be achievable, most Arduino based printers will reset upon reconnection of serial due to the FTDI chip pulling the reset line.

The scenario which happened to me this weekend almost resulted in a fire whereby the Arduino crashed with the heater mosfet turned on. The watchdog failed to reset the arduino and octopi detected the timeout as expected. If auto-reconnect was available, then octopi would have tried to reconnect, reset the Arduino and fire averted as upon reset the mosfet will be turned off again.

With this "feature" a printer can be doubly protected by both the raspberry pi and the Arduino watchdog.

While this can be achieved through a plugin, it seems to me that the best place to have this is the serial connection as a checkbox.

[Erutan409]

Interesting... Honest question:

Is there any reason why this shouldn't be more directed at the maintainer(s) of the software running on the Arduino, itself? Rather than try to compensate for that dangerous scenario by depending on OctoPrint to save you from potential disaster?

Would this not be a good start to implement the built-in functionality of the Mega's to deal with these situations?

[alex-gibson]

I'd like to preface by saying that I agree that putting even a great workaround in place to fix as serious a problem as a serial disconnect is a dangerous thing to implement as default behaviour.

That said, I would love to have Octoprint, or a plugin, enable this as an opt-in behaviour.

It's 100% possible in some circumstances to successfully resume even a total failure of all electronics downstream of the OctoPrint server, if the machine remains mechanically stable. Let me describe a scenario from today:

[First up - I was not using Octoprint at all (I normally use OctoPi). The internal Pi3-Arduino Mega based setup is normally so stable it probably wouldn't have had the problem, but never mind...]

I had a very large 3D print running, which had hours of printing invested in it already, just to get to the top stages of the raft.

I was serving this from Simplify3D over a USB cable and an extension lead, of unknown quality. [Yep, I deserved everything I got, nobody needs to make this point.] I just walked near the cable, didn't even touch it, but heard the Windows PC 'bing-bong', and had time to build up a feeling of dread before the print paused. So I know the root cause was a serial disconnect.

I copied the Simplify3D machine control serial terminal log into Notepad++. I went down to the previous G1 command above the last acknowledgment, and copied this to a scratchpad - this gave X, Y and extruder co-ordinates.

I then opened the GCODE file, and searched for the copied G1 command, found it, noted the line number. I then ensured the cursor was on this and searched for the previous instance of the letter Z - a G1 command. I copied just the Z***.*** value, then I went back to the end of the G1 command I looked up, pasted the Z***.*** to the end of that line, changed its G1 to a G92, deleted all previous lines and saved it as a new gcode file.

I unplugged and re-plugged the USB cables, having found a known good longer cable this time, then back in Simplify3D, I simply set the extruder temperature, waited for it to get up to temp and then loaded and started the GCODE. To my delight, the print resumed perfectly, with only a blotch of molten plastic where the nozzle stopped as evidence.

This worked because:
a) the cause was a serial disconnect - so the Arduino just ran out of instructions. It wasn't interrupted halfway through.
b) Simplify3D had been monitoring the commands sent to the Arduino and acknowledgements received.
c) The printer remained mechanically stable - especially the moving Z platform did not drop.

A plugin that would enable Octoprint to help out automating those fiddly tasks would be very, very helpful. I'd suggest the approach could be 'we've noticed a serial disconnect - would you like to try to recover?'

[foosel]

@alex-gibson the problem is that most controllers reset on serial (re)connect and that means they forget the head position. You can just tell the head where its at, but that doesn't necessarily need to be true (because what OctoPrint already sent to the printer isn't guaranteed to have been executed before the disconnect). X and Y you can home and then move to the last position they should have been at. Can be slightly off though.

The good news is that something like that could already be built via the plugin interface. OctoPrint will create a print recovery file on print failure (also on serial disconnect) and persist the byte position in the file it was at when stuff went south. A plugin could detect an error disconnect, read that data, basically do what you did, and then attempt to recover. Will that work? Sometimes yes, sometimes now, and it strongly depends on circumstances and might also do horrible things.

Last year I wrote a similar plugin for a client, but there they could pretty much guarantee that there would never be a disconnect from the printer, so stuff like "wait for current moves to finish, query position and move head to a secure position" was very much possible. And from that data a resume file was then generated that could be used to simply restart things again. But that depends on a VERY specific setup with some firmware additions as well (a lot of the data sent to the printer boils down to a write-only interface - no way for example to get information about the currently selected extruder for multi extruder setups, the current feedrate or the E coordinate of any but the currently selected extruder).

[jedwoffinden]

I would like to know if anyone experiencing these intermittent drops in connection, have found a solution. I'm 24 hours into a 48 hour print, and it stopped printing. Sigh. I've been using Octoprint for roughly 2 weeks, and it's the second time this has happened. I'm using a good, short USB cable and my Pi has no power issues, and has been used for countless hours in my other projects without any problems.

[Erutan409]

@jedwoffinden I'd be willing to bet money that it actually is power issue related.

How are you powering the pi?

[jedwoffinden]

I'm using this: https://www.amazon.com/gp/product/B01FM0XOH8/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

[Erutan409]

@jedwoffinden Any flutters in voltage seems to affect those wall warts. If you can intercept the voltage and put a capacitor in between the source and the load, you may be able to account for said voltage irregularity.

[jedwoffinden]

Interesting. I do have the PS connected to the same surge protector that my laptop, monitor, and 3d printer are connected to. I wonder if moving the Pi to a separate outlet altogether might fix the problem. Alternatively, in the link you provided it sounded like the issue Bryan was having was due to his usb cable. I guess I have a couple of things to test. Thank you for your help.

[Erutan409]

You're welcome.

If it's the same circuit, it doesn't matter. Your computer, monitor, etc., are better equipped to filter the voltage fluctuation because of the built in capacitors to maintain its consistent DC voltage. Unfortunately, those wall warts don't have the same robustness. You could even make the argument that the RPi, itself, is to blame for not doing a better job at filtering the source. But, they were designed to be lean. Just a little too lean for certain applications, unfortunately.

If you're not powering a camera from the RPi, you may even try to running it off from the USB port on your computer to see if it maintains the print, long-term. It's one way to test.

Also, another way I'd suggest testing (and how I determined the root cause of my problem) is by doing the following (preferably while idle - you don't need to be running a print for this):

1. SSH into your RPi
2. Run the following command: sudo watch -n 0.5 'cat /var/log/syslog | tail -n 15'
3. Flip a a light switch on and off (on the same circuit)
4. Watch for these in real-time: {TIMESTAMP} octopi kernel: [ {SOME_NUMBER}] usb 1-1.3: USB disconnect, device number {DIGIT}

If you see the latter, your problem is confirmed. And along with that, you should see subsequent messages alluding to reconnecting.

[dustin]

While resuming a print would be a nice magical feature, I'd be very happy if octoprint could just reconnect and run a bit of gcode script. e.g., on connection failure, raise Z a bit, ensure I'm at a good temperature, extrude a bit, retract slightly and cool down.

My printer's been sitting with the hotend in plastic for about three hours. The print is already damaged at this point, but a bit of safety code here would prevent anything else from getting damaged (printer, house, etc...)

[Whippertoast]

So this is a similar issue, but related to power failures and not unintentional disconnects.
I plan on having my setup work like this eventually (currently all I have is a stock printer, no octoprint or anything like that yet)

120VAC input --> Cheap UPS (~250 watt) --> ATX power supply --> Raspberry Pi powered by 5v standby rail so that it is always on --> printer powered by 12v rail on ATX power supply

So from what I'm reading on here, it would be very doable to input some kind of power failure detection on the raspi, have it send a pause command (or even a custom pause macro) and ensure that all the power guzzling devices are immediately turned off, print head is moved to a save place, etc. After this the raspberry pi could turn off everything but the 5v stand by rail using the PS_on pin.

Now the raspberry pi is only consuming it's standby power, and the printer mainboard is being powered through the USB. If we wanted to really conserve power I'm sure there is a way to disable the LCD screen while the print is paused. Now an APC 250 watt UPS is estimated to have a runtime of ~3 hours at a load of 10 watts. The raspberry pi3 at idle consumes roughly 1.4w, so even with powering a printer board it will surely be under 10 watts, which will give it more than enough backup time to outlast most power outages.

When power is restored, octoprint sends a resume command, gets everything up to temp, and continues.

Am I missing anything big here? I'm thinking that I shouldn't run into any of the issues you guys mentioned regarding disconnects, because octoprint and the printer board never power down or disconnect using this method.

EDIT: Probably the trickiest part of this would be detecting a power failure. Most of the cheaper UPS systems don't include any kind of USB connection to provide notifications or other data. But I think this could be easily overcome with a little bit of soldering. All of them have an audible and visual (LED) warning when the power fails, so it should be a simple matter of wiring the LED pins on the UPS to a Raspberry pi pin.

[Erutan409]

@Whippertoast Problem with depending on a UPS to filter your voltage is that you'll kill the battery within a week with the sensitivity setting set to its max value to kick in and even out the fluctuation. But, if you run your RP via a voltage filter (they sell them on Amazon) AND connect everything to a UPS, you could probably find some open source project out there that will allow you to capture power outage events to the RP via a USB connection and kick off all of the subsequent steps you want to take in said event.

[dragonnn]

Eee the reset isn't triggered by RTS/DTR? Just do not set RTS/DTR - pleas lock here https://arduino.stackexchange.com/questions/439/why-does-starting-the-serial-monitor-restart-the-sketch
so we can easily reconnect without resetting the Arduino. This would be really useful when something happens like this:
[57109.293089] usb usb2-port1: disabled by hub (EMI?), re-enabling...
[57109.318370] usb 2-1: USB disconnect, device number 3
With happens for me today during 12h print grrrrr

[adrianmihalko]

Happened me same as @dragonnn . Any solution to this?

[SystemDisc]

I'm with @dustin on this one.

@foosel Would you consider adding a "Reconnect and run gcode" on error/failure? If resuming the print is too difficult or impossible, I'd really like to be able to have OctoPrint reconnect and run a few commands for safety (like what @dustin listed as an example).

I too have had OctoPrint disconnect and end up leaving the hotend and bed heated, with the hotend sitting directly on plastic, resulting in some cooked plastic, and a bit of smoke when I manually sent commands to raise the head and extrude a bit before cooling off.

Edit: Actually, I think my issue is actually related to #2647 and/or #2424, so I'm not sure OctoPrint could even detect this and run custom gcode. I'll add an M85 command to my custom starting gcode to avoid the heaters being left on for so long (maybe that would help you, too @dustin). Plus, I'll use screen like you described here, to see if I can help you resolve any potential bugs.

[foosel]

@foosel Would you consider adding a "Reconnect and run gcode" on error/failure?

In principle I'd be happy to accept such a pull request. What I'm a bit worried about is the sense of false security that might add for users - even if such a feature were available, it wouldn't be guaranteed that any such gcode would actually run since it's not guaranteed that OctoPrint would even be able to reconnect reliably enough for that, depending on the type of error. Additionally it would have to be discussed how such a feature would behave in case of prints from the printer's SD card (since a reconnect might reset the controller and hence abort such a print if it happily continued to run even though the serial interface died)

[FrostyWolf]

I dunno enough about how Octoprint works to know if this is possible, but if the gcode is on the SD card in the printer (not locally), is it possible to start the print with octoprint in a way that just triggers the printer to do it "normally", so that if octoprint crashes, turns off, etc, it doesn't affect the print cause the printer is handling it?

[Ashram56]

Would it be possible for Octoprint to display the last layer change it processed ?

Getting the exact XY coordinates and where the head is in the print is probably problematic, but at least displaying the layer number (and associated Z-height) would allow to perform some manual Gcode surgery to recover.

It's probably somewhere in the log though, just saying displaying it as part of the progress bar would be nice.

And yes, I've had a lot of disconnect issues myself :-)

[foosel]

https://plugins.octoprint.org/plugins/displayz/

[ignis32]

Hi,

Probably I am a little bit late with my story, but:

I've run into same disconnect issue with both Raspberry Pi 3 and old ubuntu asus laptop, each time ending up with Anycubic i3 Mega freezing with heatbed and nozzle being heated for several hours until I notice the problem (my prints tend to be 5-12 hours long avg, and I can't guard it full-time). Tried different usb cables, different Pi power supplies, Pi power cable - that's all the same.

While I am sad to lost the print, my biggest concern is that heating does not turn off after failure.

However, looks llike I've found some kind of workaround at least for the heating.
I've added the following code to GCODE Scripts ->After connection to printer is established:

M104 S0; Set Hot-end to 0C (off)
M140 S0; Set bed to 0C (off)

This makes Octoprint to disable heaters after connection.

Octorprint does not reconnect by its own, only during server startup - but seems like PortLister plugin makes the autoreconnect after failure like I'm expecting it to happen:

https://plugins.octoprint.org/plugins/portlister/

With these two things working together it should disable heat after such a failure. At the moment, i've tested it only under artifical conditions - by setting temperatures manually at the printer's interface, and reconnecting usb cable, but probably it should work for the original issue as well.

Disclaimer- As is, please do not consider this workaround as a panacea or reliable safety mechanism, who knows a real behaviour.

To conclude - probably, it is something that we could expect to be basic behaviour for Octoprint

P.S. There is another hypothetical workaround solution - send via sd card, evading usb cable issues.
It should be more or less convinient, comparing to loosing hours of printing and tons of filament.
However, Cura's octoprint plugin has only "print" option, and no "send to remote SD card" option, while as far as I understand octoprint itself is capable of that.

[Ejay-3D]

Hi Guys,
Sorry if I wake up this issue lately but I have some idea on how we can maybe achieve this .
I have no real python dev skill so i'm not sure it would be achievable but let's expose my idea and see.

Gcode files are just line to line text files .
When we print we read those lines . one is done we read the next one .

My though is could it be possible to generate a new gcode file on the fly at each command passed or even at each layer ( because depending on how big is the file it could be quite I/O expensive )

In other word , each time a command is passed we generate a "resume gcode file" .
On top of this file we would have always the same header eg:
Increaze Z Go to the right temp etc . Etc then resume print ( with user intervention or not )
following that all the command left to do in the gcode file .

The only issue I see here is the file creation speed . Indeed some command are passed each second sometime less . So again that would be expansive to create a file at each command .
But at each layer or timely based ( 30 seconds ( could be set by user )) seems to be reasonable .

I hope I made myself clear enought to get my points.

What do you think about this ? Is that achievable ?

Thanks for your inputs !

[lalalandrus]

Wouldnt this burn through sd cards rather quick. Especially when the deletion is by sectors and each line is only a fraction of a sector.

[Ejay-3D]

Indeed I think the idea is not so bad but the I/O work it require is too important . I have to figure out how to workaournd this .

[darrylhadfield]

Sorry for zombie-ing the thread, however, I just encountered this issue thanks to a Duke Energy power outage.

I'm still relatively new to 3d printing, and newer to octoprint, but it occurs to me a simpler 'fix' may be viable: it will create more I/O but why not drop a layer reference into the log on start of each layer, or, on completion of each layer? better yet, start a new log file, specific to the given print, that is deleted based on user input in system configuration. SD cards are cheap; I've got a 64gb card in my Octoprint Pi, and I doubt I'm going to consume that anytime soon...

[foosel]

Sending a new layer command to the printer happens at a different point in time than the printer executing said command, meaning that if I write "Layer 0.4mm" into the log and you get a power outage right after, the printer might still be lower, leading to issues.

And if you get a power outage the printer fully loses its position due to the stepper lock going away and hence you get a shift anyhow (you will not be able to reconstruct the exact position even with homing individual axes), plus use the leveling matrix and everything else.

See also this segment in OctoPrint on Air 23.

[radovanobal]

It would be great if the octoprint would at least reconnect and turn off heating. Now it potentially just sits there oozing plastic and just building up a mess and a stink.

[eduncan911]

I know this is a bit off-topic, but talking about power failures...

I've always wanted a way for Octo/Klipper to be notified when I have a power failure. With the printer on a quality UPS, I'd want it to go into "park mode" where it would:

• finish the layer it is on
• store it's last layer, to resume later
• z-hip 5 or 10mm
• home x y
• if z-stops are at top, home z.
• power down hotend, keep power to bed for as long as possible until UPS bat runs out

The trick is to ensure Z doesn't move while powered off. Depending on the printer's design, this could be easy (Delta end-stops at the top, or other end-stops located at the top).

Then when power is resumed, go back to printing.

---

PS: I just had a 71 hour print fail with an Octoprint disconnect at 41 hours. sad panda

[Araforn]

I remember reading this issue some time ago but never experinced an issue until tonight on my ender 3 v2. Melting away for a couple of hours before I saw the issue.

Anyway. I have 2 octoprint instances and since I connect them to home assistant I now have an automation that checks if the state of the connections go from "Printing" to "Unknown". If they do, then I cut power to the printers using a connected sonoff basic.

My issue is a bad cable so I know now that this automation Works very well (plenty of tests since it was dropping every 5 minutes).

Ps. I'd never be worried about failed prints. Just concerned about the printers staying hot for that long. This mitigates it so I'm happy as a work around.

[spacecabbie]

I don't understand the issue if the firmware can do it why not octoprint ?

[foosel]

Because the firmware knows stuff that it doesn't tell OctoPrint and that OctoPrint would need to do that.

[spacecabbie]

Because the firmware knows stuff that it doesn't tell OctoPrint and that OctoPrint would need to do that.

righto might it be an idea to make a custom version of marlin its open source aftherall I am not a advanced coder but just wondering if it would be possible that way?

[foosel]

Possible? To a certain extent. Just like firmware-built-in power loss recovery, there'll be always a margin of error where the power loss happens at just the wrong moment so the recovery fails. But a custom Marlin build (yet another firmware variant with its own quirks) would require to have a REALLY wide adaptation and good enough documentation to hope for mainline integration to justify development against it.

[cniweb]

The idea would actually be that Octoprint takes this "pause", ideally when a layer change is due.
Then the last position is remembered, saved and moved to the home position. Then switch off the printer or close the connection.
As soon as the connection is restored (printer is switched on again) then heats up, cleans the nozzle if necessary and continues printing at the last (saved) position in the model.

[jneilliii]

I thinks this request is covered by the plugin: https://plugins.octoprint.org/plugins/powerfailure/