Random motor stall in (0.4.6 <= version <= v0.4.7-rc10)

[RJSavell]

This issue seems this started with 0.4.6 or 0.4.7 but it may have been there all along. This is the 3rd time the Buildbotics controller suddenly and apparently for no reason changed the tool path to a new position and ruined my work piece. The first 2 times I thought it was maybe something I was doing but this time I am sure it was not. Where this happens is not consistent, each time it has happened it seems to just change direction and continue in a different direction, in this case it went from cutting a circle to the start of a figure eight and then continued with the circle but in the wrong place. It then stays aligned with the new position continuing to ruin the work piece.

Hopefully you can find and correct this quickly as I am not able to use the controller with this bug

Buildbotics bug.zip

!

I have attached the debug files, a picture of what happened and a screen shot showing the path viewer still thinking it is in the correct position.

Please let me know there is any more information you need from me.

Thanks,

Randy

[jcoffland]

Thanks for the report I'll look into it. I'm sorry your workpiece was ruined but please be aware that there is some risk involved in choosing to use a beta firmware.

[RJSavell]

Good point about being beta, should I go back to 4.6?

[jcoffland]

I went through the log file and it did contain one weird thing related to reporting modbus communication warnings but I don't think this would have caused the problem above. I extracted the position data from the log file and plotted it. The controller reported the correct positions in the log file. The Buildbotics controller keeps track of how many pulses it sends to the motor drivers so unless the motors slip, it's internal position should correspond. It seems impossible for it to have driven the motors so far off without this being visible in the log file.

This leads me to conclude that the most likely explanation is that there was either an electrical or mechanical failure that caused the motors to stall and lose position. It's odd that both X and Y lost position. However, if the tool head stalled as it was trying to come back around the circle for about 1/4 of the circle in just the right spot it would lead to exactly the failure depicted above.

I'm not saying this absolutely was not the Buildbotics controller's fault but I suggest you also look carefully into other possibilities such as loose wiring, things that could physically block the movement of the tool head or mechanical problems. I'll also keep looking for a better explanation.

Here's the plot I got:

Note that the log file only contains periodic position updates, thus the jagged lines, but it does follow the correct path without drifting out of position.

[RJSavell]

Joe, Thanks for the looking in to this! I went back to 4.6 and I have tried to reproduce the issue but after 15 runs it was still tracking correctly. I was going to try to reproduce it with your latest build 0.4.7 rc8 this morning, however, the motors are not moving at all and you can turn them freely as if there is no power being applied to the motors. I attached the debug file in case that is helpful. I can try again with 0.4.7 rc7 if you think that would be helpful. I am pretty sure nothing was blocked the path. One thing I noticed is the angle between the center of the original circle and the new center of the new circle are close to a 45 degree angle so it seems like whatever happened, happened to both the X and Y motors the same time and for about the same distance. if the motors skipped I would not think it would go as much outside of the circle. From where is seems to have happened it looks like something occurred between a transition from one arc to the next arc or the motors may have moved further then they were told to move. Would changing the setting for the microsteps make this more or less likely to happen, it's currently set to64? If there is anything I can do to help please let me know. Thanks, Randy

…

________________________________ From: Joseph Coffland <notifications@github.com> Sent: Friday, April 26, 2019 3:11 PM To: buildbotics/bbctrl-firmware Cc: RJSavell; Author Subject: Re: [buildbotics/bbctrl-firmware] CRITICAL path error (#215) I went through the log file and it did contain one weird thing related to reporting modbus communication warnings but I don't think this would have caused the problem above. I extracted the position data from the log file and plotted it. The controller reported the correct positions in the log file. The Buildbotics controller keeps track of how many pulses it sends to the motor drivers so unless the motors slip, it's internal position should correspond. It seems impossible for it to have driven the motors so far off without this being visible in the log file. This leads me to conclude that the most likely explanation is that there was either an electrical or mechanical failure that caused the motors to stall and lose position. It's odd that both X and Y lost position. However, if the tool head stalled as it was trying to come back around the circle for about 1/4 of the circle in just the right spot it would lead to exactly the failure depicted above. I'm not saying this absolutely was not the Buildbotics controller's fault but I suggest you also look carefully into other possibilities such as loose wiring, things that could physically block the movement of the tool head or mechanical problems. I'll also keep looking for a better explanation. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#215 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALHJGXKFYQCIH6GWGO5VHATPSNVX5ANCNFSM4HIHT5MQ>.

[RJSavell]

I moved on to a different project and reverted back to 0.4.6. basically his the path is just a oval. On the first run it had a similar problem so I started running air cuts and I was able to reproduce the issue with each run. I was finally able to witness it happening. It seems to happen at the end of an arc. What happened is the x and y motors seemed to abruptly turn slightly back and forth without really moving making a loud grinding sound and then they would start from that position and keep going. Of course at that point the system coordinates do not match the actual position and it just start cutting from there. I checked the guides, the UV joints (the flexible joint where the motor connects to the drive screw) I tried running the axis from one extreme to the other without experencing any mechanical issues. I tried changing the microsteps per step from 64 to 32, idle current and the drive current but I was still able to reproduce the problem. I tried wiggling the wiring while it was running but that did seem to be causing the problem. As a last ditch test I went back to 0.4.5. I have really not used 0.4.6 or 0.4.7 until just recently but had used 0.4.5 quite a bit without any path issues before. After 4 runs with 0.4.5. I was not able to reproduce the issues so I went back to 0.4.6 and during the first run was able to reproduce the problem. The debug file is attached. I will continue working with 0.4.5 for now. Please let me know if there is any other inforamtion that would be helpful.

[jcoffland]

Thank you for the detailed analysis. That is really helpful. There were some changes to arc handling between version 0.4.5 and 0.4.6 so you're probably on to something here.

This is a bit of a stab in the dark but can you tell me if 0.4.7-rc9 works any better?

[jcoffland]

I'm having trouble reproducing this. Could you provide your simplified oval path that exhibits this problem? Note, I didn't get the debug attachment you mentioned in your last message.

[RJSavell]

I will give it a try with RC9 and let you know but it may be a couple of days before I can test with it.

Here is the file for the oval (it is actually for making a planter :-)
Planter.txt

Thanks!

Randy

[jcoffland]

Ok, I was able to reproduce this bug but only a couple of times and I'm still unsure what is causing it. It would be good to narrow it down further. Since you are able to easily reproduce the bug a can you try it with your spindle disabled and/or camera unplugged? These may or may not affect it.

[jcoffland]

I think I've fixed this. Please try 0.4.7-rc10.

The problem was caused by the planner queue bottoming out. I'm not quite sure what was causing it but after reducing the internal communication traffic I can no longer reproduce the problem.

Most likely the RPi was either not sending data fast enough or not reading the responses from the AVR (step generating processor) fast enough. When the queue runs dry the AVR enters the READY state because it has no more commands. This causes the ETA, Remaining and Progress fields on the Web interface to clear. When it switches from RUNNING to READY and back to RUNNING quickly, those fields flash and the motors stall.

[RJSavell]

Joe,

I saw there was a rc11 so I loaded that instead of rc10. I ran it several times and preformed perfectly - Thank you!!!!

Randy

[jcoffland]

Gald to hear it.