Power-off Recovery Implementation

[CorvetteCole]

Hi, I am working on an implementation of power-off recovery and wanted to everyone to be aware of it/review my implementation design.

This implementation would depend on a power panic board, much like the Original Prusa i3 Mk3. It would add a new config option for specifying the power panic pin.

Additionally, this assumes the power supply would provide around a second of operation after wall power is lost. Alternatively, the Raspberry Pi and MCU could be powered off of a 5v battery or some form of capacitor setup to substitute.

Here is how it would work:

• power panic signal is received from the power panic board
• the MCU would decelerate the printer to a stop and record the remaining steps in the current move
• the MCU would send an emergency signal to the Raspberry Pi containing the current x, y, and z axis position, and move queue
• the Raspberry Pi would save this move queue, z axis info, set temperatures of the bed + extruders, and the filename of the current print to a file named resume.klipper or something
• the Raspberry Pi then executes a shutdown command to safely turn off
• upon boot Klipper checks to see if a resume file is present. If so, it heats the bed and extruder(s) up to the specified temperatures
• Klipper sets the current z axis position to the one recorded in the resume file and then homes the x and y axis of the printer
• Next Klipper loads the file using the Virtual SDCard feature. It looks for the first G1 command containing the current Z axis position as recorded in the resume file.
• Klipper calculates the move queue as usual, but does not send it. Instead it goes through the file looking for a move queue that matches the one sent over by the MCU previously.
• Klipper then moves the extruder to the recorded x and y position from the resume file
• Klipper then sends over the move queue from the resume file and continues normally to the next set of moves
• Klipper then deletes the resume file and continues on with the print as normal.

If you have any concerns or advice regarding this implementation please let me know. I have looked through the code a lot and read the documentation and will be fully implementing this and sending a pull request soon.

Thanks!
Cole

klippy.log

[Hywelmartin]

I think that some sort of UPS is necessary for the RPi at least.
Lifting the Z-axis is IMO almost a must.
Would it be possible to use the pause/resume function for this.
homeing?

[CorvetteCole]

well I suppose we could lift the Z-axis by a little bit. That would be a good idea to prevent it melting in to the part. I need to look more in to the pause/resume function to determine that. I want to avoid constantly writing what is happening to a file like other solutions (Marlin).

[sniperlucian]

I like the idea very much, since I already had to deal with manual recovery - but not due to an power out. At least here in Germany power out is very rare and usually you don't need an UPS.

Therefore I see two disadvantages:

• you need additional hardware
• you can not recover from any other error than lost power, e.g. hang / crash from RPI or CPU board

[KevinOConnor]

Thanks.

At a high-level my feedback is that this is useful, but will be difficult to accomplish. The main issue I foresee is that g-code is not designed to be interrupted. (One may be able to get some basic test cases working, but g-code has so many bizarre corner cases that I fear making it work in the general case will become overwhelming.)

Cheers,
-Kevin

[CorvetteCole]

We intend on using Klipper in a commercial printer launching later this year so power-off recovery is definitely a big deal and hopefully worth the hardships. I am gonna start working on this and then once I've got a version that works hopefully I can merge it back in assuming not too much has changed.

[KevinOConnor]

Interesting; thanks. I look forward to seeing your results.

Some thoughts:

• I would avoid trying to extend the micro-controller code for this. The micro-controller doesn't have any knowledge of the control algorithms for the steppers, so trying to decelerate in the micro-controller code will be painful. Instead, I'd look to limit the amount of movements queued on the micro-controller. For example, change the host code to never buffer more than 750ms of moves on the micro-controller. On a powerdown signal, then the host can modify its movement plan, generate the necessary deceleration movements, and the micro-controller can just run out its movement queue.
• Since the steppers may move when the power is disabled, it may be difficult to return to the precise location on power resume. Similarly, due to heating / cooling of the part, there's a good chance the final part may be faulty regardless. That may limit the general utility.
• I'd be very leery of any code that automatically started a print on power up. I'd want to manually acknowledge the resume.
• The resume would only work if the print was started via the virtual_sdcard interface. It'd be difficult to resume if OctoPrint was sending the print to Klipper.

Cheers,
-Kevin

[CorvetteCole]

Very good points and I really appreciate the knowledge. I do agree that modifying microcontroller code should be avoided. I will need to dig through octoprint to see if there is some way that Klipper can find the print and give the user an option to resume on boot. Ideally, I'd like octoprint to take care of it but it seems having Klipper handle this may be needed. Maybe I can find a way to trigger octoprint to "resume" a print from a smaller gcode file that contains the rest of the print (as long as I can set the z axis in klipper).

I suspect most Z steppers don't move without power, at least not the ones on any of my boards. My boards shorts the steppers to create resistance when there is no power and also the lead screw prevents much movement regardless. X and Y wouldn't be a problem because the printer could easily rehome.

With the short buffer we could theoretically just skip unexecuted moves. It would probably result in maybe some tiny errors in the print but would be drastically better than having the print fail altogether (and easier to implement than keeping track of what moves have actually happened).

[CorvetteCole]

this power-off recovery would be an advanced config option obviously so I can state the caveats clearly when adding a template so users know not to expect too much. Once I have this implemented that is

[juliandroid]

What about the other way around.

1. The planner on the RPi keeps track where the print currently is (x,y,z) - let's say every one second.
2. Power loss happening, which means the printer keeps going for second (maybe more - on CR-10s Pro is around 1s after power loss), which means the printer could potentially keeps printing for 1 second and stop.
   2.1. Optionally, if power detection loss is possible, move the nozzle on Z up, (2-5mm, depending on the printer and realiable/repeatable in case of power loss - should be configured per printer). This would be helpful to remove the plastic from the nozzle and clean the surface before recovery. Of course having power loss detection will mean stopping the heatbed and hotend immediatelly to have time to move safely the Z axe couple of mm.
   EDIT: Correction, under full load: heatbed+nozzle, moving Z-axe and powering off this effectively kills the printer almost instantly (less than 100ms in my opinion, maybe 50ms), so I can safely assume that the nozzle will not move more than 1mm anyways. The only problem I see now is how to ensure the print will survive? I guess you need to heat the nozzle to last printing temperature and after that to move the z-axe up, otherwise you might remove the object from the bed.
3. Later, printer is turned on and the X,Y,Z position is unknown. Calibrate X,Y only, by remembering the number of steps before hitting X/Y limit swtich and thus you'll know where the X,Y was left before power down. Z axe is moved 5mm up in order to perform safely X/Y homing (possible issue if the Z is already close to the max and there is no Z end switch) .
4. Go back to the planner and compare the last stored position (last second) with the X/Y position and guess how much steps in X/Y (even Z) was performed to continue where printer actually stopped.
   Go back the Z with 5mm (done because of the X/Y homing) and (optional: also compensate for the 2-5mm if the printer supports power loss detection and moved the Z with 2-5mm already).

Based on actuall recovered position you may know not only how much X/Y moved, but also wether the printer went to the next Z layer, because if it is already on X/Y location, which according to the planner, should be on the next Z layer, then you know the printer moved one layer up too.

For very small perimeter prints the nozzle might have done more than one full perimeter, so that is also possible!
Edit: this is quite, quite, unlikely due to heavy power load (nozzle, motors, heatbed, fans, LCD, etc).

Prints with heated bed, can release the print when got cold.

[juliandroid]

Advanced option (can be implemented later, depending on real-life expereince): you could even allow X/Y/Z tweaking by the interface. So, if the printer thinks that it has to continue from x1, y1, z1 position - the user could say z1-1 layer, so the planner goes back one layer down on the same x1, y1 position in order to compensate for the mistake. I.e. user has to be able to move the X/Y/Z (after X/Y homing). It should be possible to move X/Y too, again to compensate.

So, how you recover Z? Based on current knowledge where the X,Y left and the last save on the RPi (last second), plus possible compensation by user (+/- number of layers). Is it possible?

[juliandroid]

Advanced option 2 (implementation could be done even further in time): RPi, potentially could detect upon reboot that the print was interrupted due to power loss and if the total power outage (including reboot) was less than N seconds to continue automatically the print (i.e. bed is still warm, nozzle is cold, but probably could be recovered in a minute or two). This could be useful option in case you had short outage like, couple of seconds and not recovering the print might be dangerous due to object self-release on cold heatbed.

[CorvetteCole]

@KevinOConnor looking for some advice. I've got a working implementation of power-off recovery. After a failure the printer will put in the Octoprint terminal upon reconnection something along the lines of "In-progress print saved, type RESUME_PRINT" to continue. Then it heats up and does the thing. I was wondering if you had any experience working with Octoprint as I'd like to make it give a nice pop-up when a print resume is available. Is this something you've any experience in or should I go delve in to the Octoprint codebase?

Thanks

[KevinOConnor]

@CorvetteCole - I do not have any experience with OctoPrint development. It is possible that @mmone may be able to help.

-Kevin

[CorvetteCole]

@mmone I would appreciate any tips! Thanks

[KevinOConnor]

Any further updates on this issue?

-Kevin

[CorvetteCole]

@KevinOConnor thanks for the reminder. I've been working on trying to integrate it with Octoprint and not just in the Klipper terminal. I'm not sure when that will happen, so I think I will clean up what I have now and submit a pull request for that. It is useful as-is just not user friendly. Perhaps a plugin for Octoprint similar to OctoKlipper would allow me to create a user interface based on the terminal command. Please keep this open for a little longer, I'm quite busy with starting a 3D printing company and I need it open to remind me!

[KevinOConnor]

Any further updates on this? I don't think we need to leave this open on the Klipper issue tracker.

-Kevin

[CorvetteCole]

I agree. I will continue working on this and merge it when I am happy with it.

…

On Sat, Oct 19, 2019, 6:10 PM KevinOConnor ***@***.***> wrote: Any further updates on this? I don't think we need to leave this open on the Klipper issue tracker. -Kevin — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1645>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AESS5DJDRQQ4Q7QKYKUG2H3QPOAW5ANCNFSM4HNRWFVQ> .

[KevinOConnor]

Okay, let us know when you are ready.

-Kevin

[terrorhai]

Hey, any updates on this?

[bjones14]

@CorvetteCole do you have a repository you can share where you are working on this? I would also like this feature and I can do development as well.

[CorvetteCole]

I was developing it privately @bjones14 but I can find and release the code in a repo as-is if you give me a little bit

[Apalasys]

Hi @CorvetteCole. Have you had a change to release the code into a repo yet by any chance? We have power cuts twice a day in South Africa, so I really need to get this working, or revert to Marlin firmware which has this functionality already. Thanks.

#MustBeNiceLivingInFirstWorldCountry

[Bob90]

@CorvetteCole did this become a thing? :)

[CorvetteCole]

@Bob90 wow, this is a trip. So sorry everybody for completely forgetting this. I've gotta find the old repo, I've been wanting to redo this code though

[Bob90]

@corvettecope no problems. So many great ideas, but I appreciate that implementation takes time. Was reading this and just wondered. Doesn't hurt to ask either :)

[ice2009]

Hello @CorvetteCole any update please? :)

[CorvetteCole]

Unfortunately @ice2009 I went to test the old code and I couldn't get it working properly. I think it might be worth rewriting an implementation of this

[dawidmpunkt]

@CorvetteCole: Could I take a look at your code? I can not find it at your github page.
Thanks in advance!

[atharvbadkas]

Hello,
I have recently migrated from Marlin to Klipper
Thank you to all the contributers for such amazing 3D Printer Firmware.
@KevinOConnor
My question-

Can we use this mini UPS module with klipper to make print recovery possible?
https://github.com/bigtreetech/BIGTREETECH-MINI-UPS-V2.0

Using one of Rpi's GPIO pin to detect the power outage status(HIGH if power outage detected) from UPS Module to pause and store the info....It does provide enough power to raise the Z axis and store the data.