iMXRT1062 Master Branch 20210219 pauses when used ganged axis

[phil-barrett]

Latest Master branch build. T41U5XBB_map.h with
#define Y_GANGED 1
Ethernet enabled in my_machine.h. Everything else is default.

When running any GCode program, all motion will stop for about 2 seconds at frequent (3-5 seconds but random) intervals. I spot checked 6 programs and they all exhibited the problem.

If you set Y_GANGED to 0 the problem does not show up.

ioSender Beta 8.

[VER:1.1f.20210219:]
[OPT:VNMSL,35,1024,3,0]
[NEWOPT:ENUMS,RT+,ES,TC,ETH]
[FIRMWARE:grblHAL]
[NVS STORAGE:*FLASH]
[DRIVER:iMXRT1062]
[DRIVER VERSION:210214]
[DRIVER OPTIONS:USB.2]
[BOARD:T41U5XBB]
[IP:192.168.86.187]
[GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0.]

[edit] another observation - it looks like ioSender is not responsive during the pauses. Maybe it is just an update issue? Not sure why it would only happen with a ganged axis.

[terjeio]

Works for me, is this line the same in your local copy?

grblHAL/drivers/IMXRT1062/grblHAL_Teensy4/src/grbl/settings.c

Line 68 in b2f15b8

PROGMEM const settings_t defaults = {

I ask since this issue is similar to the linker problem I struggled with earlier. I have flagged ethernet settings for flash storage in a new commit to master to reduce the risk further. BTW I have v1.53 of the Teensy loader - is this the same as the library version?

[phil-barrett]

Yes, settings.c is the same. I will double check that I properly installed.

1.53 is the current released teensyduino level, so yes. It is what I am using.

[frdfsnlght]

I'm finally getting my machine running and I'm experiencing the same problem.

[VER:1.1f.20210313:]
[OPT:VNMHTSL2,699,1024,3,0]
[NEWOPT:ENUMS,RT+,HOME,ES,TC,ETH]
[FIRMWARE:grblHAL]
[NVS STORAGE:*FRAM]
[DRIVER:iMXRT1062]
[DRIVER VERSION:210313]
[DRIVER OPTIONS:USB.2]
[BOARD:T41U5XBB]
[IP:0.0.0.0]
[PLUGIN:ENCODER v0.01]

I'm also using ioSender Beta 8. I also have ganged Y motors. My settings.c is also as shown.

I'm just jogging the machine around looking at verbose console messages and it seems to pause for 4 seconds, every 4 seconds. Even with no motion.

I was having problems with Telnet before; the controller would lock up sporadically. I hadn't opened an issue for that and I put it aside for now and switched to USB. But maybe the pausing was causing problems with the networking too. If I can get the pausing fixed, I'll try the networking again.

[frdfsnlght]

More info, maybe it will help?

Now that I have my limit sensors calibrated, I'm trying to run a homing cycle and I get this:

<Idle|MPos:-30.310,862.845,7.065|Bf:699,1023|FS:0,0>
<Idle|MPos:-30.310,862.845,7.065|Bf:699,1023|FS:0,0>
<Idle|MPos:-30.310,862.845,7.065|Bf:699,1023|FS:0,0>
$H
<Home|MPos:-30.310,862.845,7.065|Bf:699,1023|FS:0,0>
ALARM:8
ok
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z|A:>
GrblHAL 1.1f ['$' or '$HELP' for help]
[MSG:'$H'|'$X' to unlock]
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z|Ov:100,100,100>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z|WCO:10.000,-39.155,0.000|Ov:100,100,100>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>
Alarm:8|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z|WCO:10.000,-39.155,0.000|WCS:G54|Ov:100,100,100|A:|Sc:|H:0,0|T:0|TLR:0|Enc:1|FW:grblHAL
[VER:1.1f.20210313:]
[OPT:VNMHTSL2,699,1024,3,0]
[NEWOPT:ENUMS,RT+,HOME,ES,TC,ETH]
[FIRMWARE:grblHAL]
[NVS STORAGE:*FRAM]
[DRIVER:iMXRT1062]
[DRIVER VERSION:210313]
[DRIVER OPTIONS:USB.2]
[BOARD:T41U5XBB]
[IP:0.0.0.0]
[PLUGIN:ENCODER v0.01]
ok
$0=10.0
$1=25
$2=7
$3=3
$4=7
$5=0
...
ok
[GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0.]
ok
[G54:10.000,-39.155,0.000]
[G55:0.000,0.000,0.000]
[G56:0.000,0.000,0.000]
[G57:0.000,0.000,0.000]
[G58:0.000,0.000,0.000]
[G59:0.000,0.000,0.000]
[G59.1:0.000,0.000,0.000]
[G59.2:0.000,0.000,0.000]
[G59.3:0.000,0.000,0.000]
[G28:0.000,0.000,0.000]
[G30:0.000,0.000,0.000]
[G92:0.000,0.000,0.000]
[HOME:0.000,0.000,0.000:0]
[TLO:0.000,0.000,0.000]
[PRB:0.000,0.000,0.000:0]
ok
[GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0.]
ok
Alarm:8|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z|WCO:10.000,-39.155,0.000|WCS:G54|Ov:100,100,100|A:|Sc:|H:0,0|T:0|TLR:0|Enc:1|FW:grblHAL
<Alarm|MPos:-30.310,862.845,-1.000|Bf:699,1023|FS:0,0|Pn:Z>

I click the "Home| button, it moves a little, then hangs, then resets. All the above output occurs without any user input other than clicking "Home".

[terjeio]

Can you try with this binary?

http://www.io-engineering.com/downloads/grblHAL_Teensy4_4AxisDoorNetworkingGangedY.zip

There is also a new sender beta available:

http://www.io-engineering.com/downloads/GCode Sender Edge.8.10.zip

[frdfsnlght]

I've got the test code running, but it's not configured for my machine so I can't fully test it. I've tweaked the pin assignments for my second Y motor, and strictly speaking I don't have Y_GANGED=1, but I have Y_AUTO_SQUARE=1.

On the plus side, it appears the pausing is gone.

[dresco]

For what it's worth. I've just built and uploaded the latest code to my ganged axis router, and have not been able to replicate the issue in my environment (am using CNCjs as a sender).

Only source changes were enabling ganged & auto-squared Y axis, and planner buffer increase to 512. Happy to test anything else here if useful..?

> $I
[VER:1.1f.20210313:]
[OPT:VNMSL2,511,1024,3,0]
[NEWOPT:ENUMS,RT+,HOME,ES,TC]
[FIRMWARE:grblHAL]
[NVS STORAGE:*FLASH]
[DRIVER:iMXRT1062]
[DRIVER VERSION:210313]
[DRIVER OPTIONS:USB.2]
[BOARD:T41U5XBB]

[terjeio]

I have been pondering over why this issue present isteslf for some users only and I have one idea that could be tried. The background is that when I added a the new enumerations for settings etc. the teensy driver started to act odd for some reason I do not understand. I solved it by adding the PROGMEM attribute to some of the new arrays of static constants. This attribute is AFAIKT specific to the Arduino framework and is used to place data in read-only memory (flash). Since I had to add this to the core I had to make that portable, and I did it in a similar way as I did for the interrupt handlers for the ESP32 driver. Code here:

grblHAL/drivers/IMXRT1062/grblHAL_Teensy4/src/grbl/grbl.h

Lines 52 to 58 in 4b95f3f

	#ifdef ARDUINO
	#include <Arduino.h>
	#endif
	#ifndef PROGMEM
	#define PROGMEM
	#endif

I check for the ARDUINO symbol beeing present and if so I include the arduino.h header file. This should define the PROGMEM symbol to whatever is needed to ensure it is read from flash (the Teensy4 copies everything, including constants, to RAM per default). If not compiling for Arduino I define the PROGMEM symbol as blank.

Could it be that users experiencing this issue somehow are missing the PROGMEM symbol and the code above sets it to blank? This can easily be checked by changing the blank symbol to generate an error, e.g by changing it to #define PROGMEM xxx, if compilation fails then something is definitely wrong in the framework.

and strictly speaking I don't have Y_GANGED=1, but I have Y_AUTO_SQUARE=1.

When auto squaring is enabled basically the only change (while running a gcode program) is that there are a couple of lines of extra code executed in the step pulse output function. If this is causing the issue the there must be code elsewhere (in the USB driver?) that gets tripped up.

@frdfsnlght Can you post your settings here so I can run a test with those? You may also try a auto_square build I just uploaded:

http://www.io-engineering.com/downloads/grblHAL_Teensy4_4AxisNetworkingSquaredY.zip

Only source changes were enabling ganged & auto-squared Y axis, and planner buffer increase to 512. Happy to test anything else here if useful..?

Thanks, this is the same result I get when testing myself - and it confirms that only some users experience this issue.

[frdfsnlght]

Things have come up and I don't have any time for the next few days, maybe until the weekend, to try anything out. I'll let you know what I find out.

[frdfsnlght]

I've had a few minutes to play around this morning. And I think I found the problem. Here's what I tried:

1. Reverted to my original FW build and used the beta 8.10 ioSender, still pauses
2. Changed BLOCK_BUFFER_SIZE from my original 700 to 512, rebuilt, beta 8.10 ioSender, still pauses
3. I changed the line in grbl.h to "#define PROGMEM xxx" and it compiled fine
4. Set Y_AUTO_SQUARE to 0, still pauses
5. Set Y_AUTO_SQUARE back to 1, BLOCK_BUFFER_SIZE still at 512, DISABLED QEI, and it works!

At least for me, disabling QEI, which I don't need right now so no loss (but I would like it to work someday when hook up to the lathe) fixes the problem. No pausing in the console and smooth jogging.

So, what next?

[phil-barrett]

Hmmm, I was pretty sure I didn't have QEI enabled for the original report. However, I just tested it QEI commented out in my_machine.h and no changes to the source code. No pauses.

I still think there is a ghost bug lurking around, though.

[terjeio]

So, what next?

I am not able to replicate this bug so need more details. Exactly which changes to the source code, settings and a test file that shows the symptoms will help a lot.

However, I just tested it QEI commented out in my_machine.h and no changes to the source code. No pauses.

When commented in there are pauses?

Since I had to add the PROGMEM attribute to some constant data to get rid of weird behaviour then perhaps adding a few more will help? These are obvious candidates:

grblHAL/drivers/IMXRT1062/grblHAL_Teensy4/src/encoder/encoder.c

Line 524 in 4b95f3f

static const setting_group_detail_t encoder_groups [] = {

grblHAL/drivers/IMXRT1062/grblHAL_Teensy4/src/encoder/encoder.c

Line 533 in 4b95f3f

static const setting_detail_t encoder_settings[] = {

Add PROGMEM in front of these definitions and try again.

If this solves the issue then it is the same gremlin I fought with earlier that is showing up again, perhaps due to a bad linker script...?

I still think there is a ghost bug lurking around, though.

It is, but I am not sure it is in my code.

[phil-barrett]

When commented in there are pauses?

No, there are not. So, I can not reproduce the problem.

With
#define BLOCK_BUFFER_SIZE 1000
X and Y speed set to 2000 mm/min, X & Y acceleration set to 100 mm/S^2, it also does not pause.

[frdfsnlght]

customizations.zip

I've attached a zip file with my source changes and settings. Please note:

1. The settings are from when I was using the first test build you provided in this issue, but the firmware I'm using is newer.
2. The way my firmware is configured in the attached files works without pauses. If you only change the QEI define in my_machine.h to enable QEI, it pauses on my machine.

[terjeio]

@frdfsnlght
The settings file is for a four axis machine with door input enabled, this does not match config.h witch is for 3 axes, no door.

Anyway, I have tested with your settings and configuration as supplied with encoder enabled - and I do not see any delays.
When a delay happens is the state still RUN or does it change to IDLE?
Do you have Agressive buffering enabled in Settings: App?

[frdfsnlght]

I know about the settings mismatch. Like I said, the settings came from your test build. But I've applied them to the current build I'm running (i.e., I haven't changed anything and I presume the fourth axis stuff is just ignored in my 3 axis build).

It happens in idle. I've never even seen a "Run" state. I haven't run any gcode yet. It pauses while doing nothing as well as when manually jogging. I don't have to do anything to see the pausing, just load the bad firmware, switch to the console tab, and click the checkbox to show verbose output. The status messages pause about every 4 seconds.

I don't know about the "Aggressive buffering" settings; I've never even looked at that tab so I presume everything there is the default.

I have some time today to play with this a little if you have any suggestions.

[terjeio]

@frdfsnlght Ok, the OP was about motion stopping when it should not. Your issue is that the real time reports does not arrive as they should.

I have some time today to play with this a little if you have any suggestions.

If there is a lot of noise on the encoder inputs then this could explain why enabling it is causing problems. Shorting the pins to ground is a good idea? When touching the pins on my board (with no encoder connected) I managed to hang/crash the sender... I had to restart the sender (but not the controller) to make it work again. This could be due to the controller beeing swamped by interrupt requests from the encoder inputs and not beeing able to handle anything else in a timely manner.

[frdfsnlght]

I have the same problem as the OP. The pauses in the real time reports directly correlate to motion stopping. The reports are just a simple way to see the pauses without having to move anything.

I hadn't considered noise on the encoder inputs. Since they aren't connected to anything yet, they just float, which could indeed cause "false" interrupts. I'll work on a way to ground the inputs when not connected to anything and give that a try.

[frdfsnlght]

I just built a "grounding connector" and plugged it in. It grounds all the ST inputs. Still pauses, no change.

And for reference, here's a settings file exported from my current setup, if that matters.

settings.txt

I played with the "Aggressive buffering" setting but it didn't make a difference.

[terjeio]

Still working fine for me with the new settings file, next step is then to disable QEI interrupts here:

grblHAL/drivers/IMXRT1062/grblHAL_Teensy4/src/driver.c

Lines 1569 to 1594 in 4b95f3f

	#if QEI_ENABLE
	case Input_QEI_A:
	if(qei_enable)
	signal->irq_mode = IRQ_Mode_Change;
	break;
	case Input_QEI_B:
	if(qei_enable)
	signal->irq_mode = IRQ_Mode_Change;
	break;
	#if QEI_INDEX_ENABLED
	case Input_QEI_Index:
	if(qei_enable)
	signal->irq_mode = IRQ_Mode_None;
	break;
	#endif
	#if QEI_SELECT_ENABLED
	case Input_QEI_Select:
	signal->debounce = hal.driver_cap.software_debounce;
	if(qei_enable)
	signal->irq_mode = IRQ_Mode_Falling;
	break;
	#endif
	#endif

Set to signal->irq_mode = IRQ_Mode_None; for all three thar are enabled above (do not comment out).

If the issue then goes away it could be a hardware problem? A bad connection?

"Aggressive buffering" is only used while streaming gcode so not relevant for this.

[frdfsnlght]

Disabling those interrupts fixes it.

Checking for a bad connection is problematic since I'd have to disconnect everything and remove the board to check the solder connections on those pins. I checked them when I made them (I'm not new to soldering) and they looked solid to me. I'm also confident in the wire harness I made that plugs into the soldered pin header, but with a little work I could remove and test that.

[frdfsnlght]

The wiring harness tests good. I even tried wiggling the harness while plugged into the pin header and it doesn't make a difference. The pausing is very consistent.

Any other ideas I can try?

[terjeio]

Disabling those interrupts fixes it.

Good to hear. FYI I once had a to fault find a PCB with a short under the solder mask. So a bad connection can be a short as well.
If you have a scope you may check if there is a signal on one of the pins that should not be there.

[terjeio]

Any other ideas I can try?

If the Teensy issocketed and you have another available you may try to swap them.

[frdfsnlght]

Easier said than done. Way easier. The machine is in my shop and my test equipment is in my office. I don't want to remove the BOB from the machine given the difficulty reinstalling it. I only plan on removing that board to replace it with the pro version when Phil's done with it (and probably soldering the teensy in).

The Teensy is socketed, but I don't have a second one to play with. Those sockets and headers are another possible area for a spurious signal. When I soldered all those things, I closely inspected them and am fairly confident the joints are good, but I also recognize there's a chance something is still wrong (my ego isn't that large).

I'm slowly coming around to the idea that I won't be able to fix this now. I don't need the encoder plugin right now so I could disable it. I plan on hooking it up to my lathe when I get around to converting it to CNC someday, but I don't have a timeline for that and it's more likely I'd have Phil's pro board installed by then anyway. Once the V2 board is replaced, I can play with it in my office where all the test equipment is and try to reproduce and fix the problem there.

I guess what I'm saying is that I don't think the problem is the firmware now, and I'll probably move on.

[phil-barrett]

I am not fully convinced that noise on the QEI pins is the real culprit. They are LP filtered with a 15.9kHz cutoff - that would be some pretty low frequency noise. Also, while the pauses are not strongly periodic, they are not completely random either. Quasi-periodic? Repeating approximately every several seconds.

Perhaps the socketed pins are bouncing but that seems a bit unlikely. One board that I have seen the pauses on is soldered in place.

If I get a case where the pauses happen, I will try a build with QEI interrupts off. This is an insidious problem.

By the way, when I saw the pauses, run state did not change. It appeared that the processor was not taking input - clicking on Stop or Feed Hold did nothing.

[terjeio]

Well, it could still be a firmware bug since code can trigger interrupts. And there are a number of other possible reasons too, faulty chip(s), faulty Teensy board, bad connections, corrupt Arduino installation...

If I had acces to a misbehaving board I would reenable interrupts for each of the pins separately to see if only one is causing the issue. Also commenting out the encoder_start() and encoder_init() with interrupts enabled to see if that changes anything.
And test with other communication channels such as Arduino USB, UART or Ethernet.

But since I cannot replicate this there is not much I can do. The only thing I can think of is that if I get the (bad?) binary I can run a test with that. If working then a corrupt Arduino installation can be ruled out.

[frdfsnlght]

Well, disaster has struck. I won't be doing any more testing for a while. I shorted a connection and blew 2 traces on my custom PCBs. In order to repair the board I have to completely remove it and the V2 BOB. I'm not going to do that only to put the V2 board back in; too much work (I really hate those non-pluggable terminals). I'll have to wait until Phil's pro board is ready.

[terjeio]

This has been resolved in version 1.54 of Teensyduino, the issue was not due to bad grblHAL driver code.