Endless Resends after End of Print (perhaps related to a misunderstood command?) #470
Comments
|
It appears, and this may be incidental, but appears the status changing to: To manually recreate this behavior follow the steps demarcated with >>> |
cakeller98
changed the title
|
OK Caught it before the repeating endless resend had filled the Octoprint terminal buffer: |
|
could you provide an example gcode file? I'd like to take a look at your ending gcode. Also a full serial.log of a print's end might shed some light onto the issue, so we see what's causing the loop to start. |
|
I will do as you ask... after the next print, and I will log serial for that. In the mean time... and the error from the end of serial.log (Of course, I'll upload a file and give you a link shortly too :) ) and directly from the terminal window: |
|
all the log files, a redacted g-code file (I can't upload all of it, as it's a proprietary thing, but has only the meat taken out, start and end are complete) - hope this helps. https://github.com/cakeller98/Cakeller98Postings/tree/master/OctoPrintPosts/RepeatingResend |
|
Is it possible to enable the serial logging from g-code? serial logging makes the print stutter, but I don't ever seem to be able to log the actual triggering event. closest I got was above where I missed the actual command by like 5 seconds! (and it fell out the back end of the log) For diagnostics this would be pretty sweet if you could enable serial logging with a message from within the gcode file! :D e.g. ;OCTO_DEBUG(Serial_Logging, "Checking for X after printing completed, during cooldown" serial loggin begins with a note like... Serial Logging Enabled by G-Code command: "Checking for X after printing completed, during cooldown" just a thought - maybe it's already possible :D |
|
Since I've been paying attention to this - it happens every single print near as I can tell. After some time, eventually at some point this file - I can send you... there was no serial logging in this log file - but... I can't really print with serial logging for expected reasons - stuttering.. |
|
In the log with the long list of resends, there is a long sequence of M105 commands in the beginning, with NO "ok"s in between. That means that maybe one command is taking very long to be processed by the machine. That is number 111890. And number 111897 seems to have caused a buffer overrun. It is most likely incomplete and that's the reason why the firmware requests a resend. But this one has already slipped out of Octoprint's memory. However, at least part of the fix to get RepetierFirmware out of that state is to issue SOME command with that number. RepetierFirmware will not move on unless that happens. Pressing Disconnect/Connect should restart the RepetierFirmware and lead out of the condition, too. Well, at least it has done it reliably on my system in the past, but it has stopped resetting the printer reliably somewhen recently with latest Octoprint (devel branch) updates on Pi. The resets now only work sometimes. |
|
I have now noted that this occurs after every print. might I suggest that the buffer ought to PAUSE in case the oldest complete EDIT: Perhaps Repetier firmware (all firmware?) needs an alternate acceptable response to a request for resend. e.g. if the host responds with "GFU" it would tell the firmware it's going to have to deal with never knowing what that command was, and... let's move on from here, because arguing about a missing command over and over again is well... isn't it that - "insanity is doing the same thing over and over and expecting different results" ? so.... somehow there needs to be able to graceful exit to this insanity! |
|
This is pretty much what I'm working on for the communication layer |
|
Cool... It's an inconvenience more than anything :). And at lesst now I On Monday, May 26, 2014, Gina Häußge notifications@github.com wrote:
--- Chris |
|
@cakeller98 // *****************************************
void GCode::requestResend()
{
static uint8_t countResend = 0;
static uint32_t resendLineNumber = 0;
countResend++;
if ((countResend > 6) && (resendLineNumber == lastLineNumber))
{
commandsReceivingWritePosition=0;
Com::println();
Com::printFLN(PSTR("Warning: Line numbers not in sync! Resync'd!"));
Com::printFLN(Com::tOk);
countResend = 0;
waitingForResend = -1;
lastLineNumber = actLineNumber;
return;
}
if (resendLineNumber != lastLineNumber)
{
countResend = 1;
resendLineNumber = lastLineNumber;
}
HAL::serialFlush();
commandsReceivingWritePosition=0;
if(sendAsBinary)
waitingForResend = 30;
else
waitingForResend = 14;
Com::println();
Com::printFLN(Com::tResend,lastLineNumber+1);
Com::printFLN(Com::tOk);
}
// *****************************************Works for me (see below), but please check if it suits your issue, too. 2014-05-27 20:15:22,001 - SERIAL - DEBUG - Send: N1 M105_38 |
|
While what you've done will end the loop. It doesn't protect instances where we need 7 or 8 retries, and don't want it to just give up after only 6 tries. The way I see the issue is this: Repetier receives data that it believes to be corrupt. Maybe it's corrupt, maybe it isn't. Whatever the case, in the condition we're discussing, the host has no way to tell repetier "hey, I can't give you what you want, so please stop asking". However Repetier should not assume this. Repetier, in fact, should simply have a mechanism to simply take the next command. It's similar to what you've proposed except formatted so the host gets to decide when enough is enough instead of Repetier "guessing" why the resend is happening. Maybe I'm missing something - so... take what I've said here with a grain of salt. |
|
Some weeks ago I have helped fixing a weird Refresh bug between Octoprint and Repetier. The issue was that when a Resend situation occurred, the protocol was going two commands ahead and then one back (type "Error:expected line 70 got 69"), making the printer stutter. The core issue there was, too, that initially a buffer overrun in the printer had happened. What we did was that we changed the code in a way that commands that had smaller numbers were not rejected any longer, but just skipped. But you are right, the protocol is rather strict and simple: It expects that all commands are sent in a row without a gap. This is for an easy reason: A perfect 3D-print requires that all commands are performed, in the correct order. So it's up to the sender to make sure that all the commands are sent in a way that they can be reliably processed. IMHO that means that the sender Before a command can be discarded from command history the sender must make sure it has got the confirmation from the printer that the command was successfully processed. That way we can't get into the situation any more that the command requested for resend has already slipped out of the command history. Regarding your question if 6 retries in my code proposal were enough: Make it a dozen or even several dozens, anyway. In my experience I have seen that normally the Refresh is resolved with the first repetition. The 6 retries was to cover the rare occasions where the communication is very bad (but then the printer wouldn't work satisfactorily anyway.) Gina has told us that she's about to rewrite that part of the octoprint code. Let's wait what she will come up with. I'm happy to do beta-testing. |
|
You exactly made my point actually. 6, or 12 or whatever... basically if there is an error in the sending - there must be a way for the printer firmware to allow the host to notify the printer-firmware that it will be impossible to send a command the firmware will like. Everything else I agree with - it is just that if the firmware has no escape route, then it either ends up in an endless loop as it does, or has to stupidly guess that it's ok to continue. I suggest that the 6 retries is fine, however it should not just accept the next command, it should send a challenge request. IOW - ok I reached my 6 retry limit, I'm not going to try and process this command, but I will notify the sender what line I was trying for, and that it did not go. If the sender (Octoprint) decides to resend the oldest command it has, or whatever it has, that's fine. It's up to octoprint to figure it out. Hope that makes sense? - Basically fine, do what you suggested, but give notification that Octoprint can do something smart with. :D Cool - I'm totally fine waiting to see what Gina comes up with. :) |
Not quite sure if I understand your point. If the Firmware has sent a "Resend" request (this can be due to a checksum error, buffer overrun, incomplete command within a second or another format error) it will by default accept ANY command from the sender, provided that new command has the same expected line number. It could be a totally different command, the firmware will not care. Maybe that's the reason why the M110 command is not widely implemented. The sender is simply required to move on with the expected (same) line number and the next command it has decided to continue with.
Again not sure: The command that was not understood was the number contained in the Resend request. IMHO it's clear what caused the hiccup.
Octoprint should be able to handle a Resend request on the very first occasion, which is not always the case with the current implementation. The situations are (according to what I know) when the requested command is no longer in the Octoprint history or if the communcation is out of sync from the beginning (both already discussed above). Just my two cents worth. ;-) |
|
Thanks @scp38 Thanks for clearing that up... I apologize, I didn't realize repetier would take any correctly formatted line of the same number it expected in the request. No need to do anything in the firmware then, it's already doing enough. The host has to simply be smart enough to know, if it's getting a resend request and doesn't have that line... then warn the user, and issue ANY command at the correct line number and move on. I thought repetier was somehow looking for the same line number with the same checksum. But that makes absolutely no sense now that you've explained it. So, the only thing that repetier host can count on is that the line number incremented from the previous line. If something goes wrong on a specific line, the firmware doesn't know what went wrong - only that what it got wasn't right. So the host only needs to send an intelligible command for that same line number. Therefore, repetier firmware doesn't need anything to be fixed. |
|
i would love to see a repeated error stop repeating. iow - if you see the same error x times, shutdown or pause. is that possible to do with events? e.g. if status is operational, and we get a resend request, increment count. when count hits 5, disconnect and reconnect to printer. ? is that something that can be done with events? |
|
Please test if this still occurs with current |
foosel
added
needs information
|
Sorry, but I was impatient and patched my Repetier firmware in May last year. I haven't seen the issue since then, so I can't say if the issue is resolved with your change mentioned above. What should this change do? Avoid flooding the firmware buffer with M105 temperature requests while waiting for the completion of a command that takes very long? |
|
Among other things, yes. |
|
Wrong button. |
|
Closing since assuming this is now fixed by the extensive changes in the comm layer (no feedback, no choice) |
|
Sadly, I'm still getting this in 1.2.4. I first noticed it on an older version (whatever one that was bundled with my octopi image), then upgraded and it still reproduces. Oddly, I've never had a problem printing gcode produced by slic3r on octoprint (dozens of prints, no communication errors), only since I switched to cura for prime towers. Possibly also of note, the printer noticeably slows down and even stutters on curves when printing via octoprint, something that was much less pronounced before I switched from slic3r. It looks like the raspberry pi's cpu is pegged (version 1, b+). I've tried to print the attached file 3 times and it always fails in about the same spot. It prints fine from the printer's sd card (Rostock max v2). https://dl.dropboxusercontent.com/u/80114996/20140205_Marvin_KeyChain_Dual_Color_2_1_2.gcode |
|
Your serial log is truncated, the error might have started before the first line you provided or not, it's not possible to say from the excerpt. The starting point is of utmost importance however. From what information is available, something fishy is definitely going on that looks like some off-by-one resend handling that started somewhere way up: What firmware that is? The ordering of the temperature output plus the line numbers on the Another thing, regarding your problems with "stuttering", you didn't say if you explicitly enabled the serial logging for hunting down this issue or if you always have it enabled. If the latter, disable it (once we have tracked this issue down). Writing stuff to the file system is somewhat slow on the Pi, and doing that for every single line sent to and received from the firmware makes everything stutter, hence the big "warning" next to the serial log checkbox in the settings. Considering that the whole communication layer has changed tremendously since that ticket here was created and hence any information in this ticket does not apply to the current code base any more, I'd also ask you to please open a new bug ticket for this. Please make sure to include all information asked of you. |
|
Prints from Cura over USB stutter because it outputs very short segments if On 24 August 2015 at 08:35, Gina Häußge notifications@github.com wrote:
|
|
Not all prints sliced by Cura necessarily stutter over USB however, and worse than the small segments rendered by Cura is leaving the serial log enabled on a Pi1 while printing something semi complex. Hence the above hint ;) |
|
Hmm, I might have to reduce the stl's poly count then. But to answer your questions foosel, yes it is repetier (seemecnc's customized version 0.91, further customized by me for cyclops support). And yes, I do get the stuttering without serial logging a bit, but it is far far far worse with logging. Without logging, it just slows down a bit and sounds different. I'll open a new ticket when I have a full serial.log to post. Thanks! |
|
You might also want to try current devel, I added a new setting today that makes OctoPrint ignore repeated resend requests for the same line. I also could reproduce the exact behavior you got by faking repetiers resend behavior, so chances are high that problem is solved now too. I'll also add it to the upcoming 1.2.5 release. |
|
So repetier is the ie6 of the 3d printing world? :P https://dl.dropboxusercontent.com/u/80114996/serial_devel.log |
commit: e7088ef
on Windows 8.1 Tablet
Something is happening at the end of my prints that is causing the endless resend loop, until the connect button is pressed 2x.
The first press returns this error:
just messing arond at the terminal I found that any malformed command results in Octoprint thinking it's not connected
The Disconnect turns to a Connect Button - though it has to be pressed 2x to actually reconnect as the first press will fail with the following error:
After the second press of the connect button:
...etc and back to normal operation
I don't know what's happening at the end of a print that's causing this, or how long AFTER the print has finished this is occurring, however you can initiate this exact error behavior by submitting any malformed command into the terminal (e.g. simply send a single character like 'a' or a space.)
My repetier firmware commit (before customization): 2b1c85411069154b71d7aa580b4185837516d126
My octoprint is at commit: e7088ef
The text was updated successfully, but these errors were encountered: