"Long closed loop" from linear encoder into grbl stepper for dynamic backlash compensation

[wolfgangr]

Consider such a venerable near 4thousand pound pile of well crafted cast iron:

Can we enhance it to mill e.g. sth like a pattern of M3-threads in a 1m-plus workpiece?

Of course, we might easily put tooth belt pulleys in place of of the lead screw handles and drive them with some nema23 or larger stepper.
However, we find an Y-axis with some tenth of tactible backlash, an X-Axis equipped with a geared AC-Motor, moving some hundert punds of inertial table mass, and a Z-Axis lifting and lowering same table, yielding in a quite asymmetric load.

So we may end up in very conservative speed an acceleration settings if we go the purely deterministic way grbl is working at the moment.

There are closed loop steppers and servo motors with rotary encoders and grbl-like step/dir inputs. Both may help to shift the safety margins a little bit. We may generate skd of "motor fault" or "motor warning" signal, so at least we could stop a job and avoid further damage, if we pressed too hard.

Threre is skd of a deterministic backlash compensation already provided in grblHAL core (no clue how mature it is).
Any way, it requires precise precalculation of backlash and does not handle cases where the table is pressed by tool force to one or the other end of the backlash interval - or maybe even sticks somewhere inbetween.

We might consider to replace the old trapezoid lead screws by contemporary no-lash SFU-type ball screws. Quite expensive, if we come to the 30mm-plus size. And not easily fit mechanically into the existing cast blocks.

There is another option:
There are linear glass scale encoders. Readily available to the DIY-market. They cost ~70 €/$ each for 5 µm resolution and are available in packages with DRO display units, too. They deliver 90°-phase-shifted-AB-Signals at ttl-level - similiar as rotary encoders do. Easy to mount, cheaper than strong ball screws. But best of all: They are not subject to forces or backslash of the kinematic slide chain. So, they tell us where our table really is (well, let's ignore tilt....)

Now, we could feed the linear encoding signal back to a closed loop stepper or servo motor.
grbl - still unchanged - delivers step/dir signals and independent servo-controllers for each axis perform the fine tuning.
This way, at least we might eliminate backlash - in the end of a movement.
However, during correction of backlash by an independent servo driver, we may be a bit off the planned path, since other axes are not aware of the correction taking place in the lagging axis.
This may be no problem if we just pocket out some large square areas. Maybe, the corners show tiny artefacts.
However, our 3 mm thread presumably might not succed this way.

For this end, compensation has to be kept in sync for all axes. If Y travels through backlash, X and Z have to wait and only then coordinated move may start again.
In terms of spacetime coordinates, we may allow deviation in speed/time, but X-Y-Z lateral space has to be kept in tight control to each other.

Anticipating my findings in trying to understand grbl, I think this ends up in heavy modifications of the stepper code.

In my current vision of the implementation, the planner might still generate the same deterministic spacetime-path.
But when it comes to send signals out to real world, it is the stepper which has to check whether the path is still on track by processing the linear encoder's position signals. And if deviation like backlash, mechanical or inertial lagging or even stalling of a motor occurs in one axis, the whole movement has to slow down. This is similiar to feed rate correction, which is injected between planner and stepper in the current code already. So only the lagging axis is moved at higher speed until on path again, and only then the other non-lagging axes may catch up as well.

I have no clue if somebody in the FOSS world has done such a thing already. Maybe there is a fork of grbl lingering around, being capable of that adaptive operation? What about LinuxCNC, Mach, Estlcam and bretheren?

Anyway, I'd bet my a(...rbitrary part of body to be filled in) that heavy commercial machines work skd of such way.

Are we, as a community, prepared to pick up this challenge?

[wolfgangr]

Let's continue with the findings of my web search.

I've startet this topic in another issue:
#287 (comment)
There are already some findings regarding other pages and valuable input from Terje there, which I'll not repeat here at this point.

I'd just recall the fact that in grblHAL, there is already a spindle-sync feature for lathing threads in place. Albeit under different circumstances, it already implements movements waiting and synced for some external signal, thus allowing deviation from the pre-planned spacetime path beyond a mere change of speed.

And there is Terje's reference to his maslow code, which - I have to admit - I have no clue yet how it fits into the whole framework.

I found one thread in the "legacy" grbl area on processing linear encoder signals:
grbl/grbl#1223 (comment)
grbl/grbl#1223 (comment)
They cover some interesting thoughts there, but I don't see that they ended up in any concrete implementation.

There is a very basic implementation of a selfmade servo:
https://www.instructables.com/CNC-Servo-StepperGRBL-Capable/
However, I don't yet see the pathway from step/dir signals through sth like PID to sth else than steps (eg. PWM-controlled H-Bridges or modbus driven VFD drivers) as critical for success for our challenge.

[wolfgangr]

So the next challenge is to understand the inner structure of grbl(HAL):

• how data is flowing
• how the things are glued together

As Terje pointed "a book had to be written". Agree on that.

Until this is done, some obvious pointers.
https://github.com/grblHAL/core/wiki
https://github.com/grblHAL/core/wiki/Hardware-Abstraction-Layer-(HAL)----developer-reference
http://svn.io-engineering.com/grblHAL/html/

However, in the end, I found more questions risen anew than old ones solved.
It's FOSS, in the end:
If RTFM does not work because of lack of manuals, RTFS is the last and only chance.
But without any previous orientation, the source is not easy to understand.
So I left the realm of grblHAL and found the basis structure still quite identic to other grbl-"dialects".

This is were I started to understand:
https://smoothieware.github.io/smoothieware-website-v1//howitworks
And here where I proceeded:
https://awesome.tech/grbl-demystified/

After understanding the Architecture sketched and explained there, I managed

• to identify the relevant pieces of code in the source
• to roughly estimate the flow of data
• to develop a first idea of how the job might possibly be done
• and draft the text in the previous posts,
  ... which just a week ago I had no clue to understand...

[wolfgangr]

Guided by this plan, we finally find the place where the magic is going on:
https://github.com/grblHAL/core/blob/master/stepper.c
There is a lot of informative text inside the source, albeit I doubt that still anything there is up to date.

The workhorse is
ISR_CODE void ISR_FUNC(stepper_driver_interrupt_handler)(void)

This interrupt pops pre-computed segments,
defined as constant velocity over n number of steps, from the step segment buffer and then
executes them by pulsing the stepper pins appropriately via the Bresenham algorithm.

The ISR is called at 30 kHz rate - as the source comment says.
However, https://www.grbl.org/benefits-of-grblhal :

8-bit GRBL maxes out at a step rate of around 30 kHz while grblHAL ports can run at significantly higher speeds. 250 kHz or higher is possible,

back to stepper.c:

... segment buffer ... is filled by the main program
when steps are "checked-out" from the first block in the planner buffer.
.... conservatively holds roughly up to 40-50 msec of steps.

Neither buffer update timing nor stepper rate is readily available from the source. I suspect it's depending on different driver configuration - still to be figured out.

Digging deeper into the code, we learn that segments may contain things like BACKLASH_COMPENSATION, i.e. producing output without updating the internal position counters.

There is a planner_recalculate() where we can interfere into movement if 'stuff' is happening.
We can execute_hold (like when a ' feed hold' button is pressed by the user) performing Forced Deceleration

This all looks like relevant action that may be performed upon absolute position information.
So for a first iteration, let's plan to feed our action right into this segment buffer.

However, the segment buffer is not the place to compare planned with actual position, since it is some unknown time ahead of pulse output.

When I got it right, the line (number 426 in current version) where a pulse is generated reads
register axes_signals_t step_outbits = (axes_signals_t){0};

The update of internal axis counters follows this line.
So the comparison to actual position read from linear encoders should be quite close to this point.
Let's think positive and expect little (or at least constant) latency in the driver-stepper-mechanic chain.

... which leads us to the question how we get this position into our controller - under severe timing contsraints.
The encoders supply TTL level quadrature signals at 5µm intervals.
At remarkable e.g. 10000 mm/min jog speed, this multiplies to 33 kHz, so we are in the same order of magnitude as stepper pulse output - fine.

Contemporary µControllers can immediately process quadrature encodings. However, asking the datasheet of STM32F4 (my favourate at the moment), there are only 4 out of 11 timers capable of this, 2 of them already configured for other purposes.
At first glance, only these are able to count up and down, which the others are not. Let's hope we can reconfigure the tick generators to use one of the "lesser" timers and free the quadrature-capable timers.
STM32H7 even has 22 timers, but still only 4 labeled as quadrature and up-down capable.

Another issue is that most timers only are capable of 16 bit, which equals 0.32 meters of 5-µm intervals.
So for travel paths larger, we'll have to handle carry over in some way.

[wolfgangr]

My early considerations included driving some PID-controlled DC-Servo or even VFD-PID-driven AC-Motors.

I encountered a venerable paper with some corner values for our task at hand:
http://www-personal.umich.edu/~ykoren/uploads/Reference-Word_Crcular_Interpolators_for_CNC_Systems.pdf
They distinguish between "Reference-Pulse" and "Reference-Word" type systems.
I'd suppose to remap the stepping output of stepper.c to the "Reference-Pulse" technique, while the segment buffer maps to the "Reference-Word" technique.

In the case of servo drives, we might feed these "Reference-Word" to the PID controlling the servos. However, said paper mentions a typical corner frequency of PID controllers not far above 100 Hz. Compared to steppers running at dozens or even hundreds of kHz, we decide not pursue the servo path at the moment, as I don't see any chance to come anyway close to stepper precision in the case of normal, deterministic precision.

Their are NEMA41-steppers available even in DIY sources, delivering 30 Nm torque and > 700 W of power - albeit matching drivers are close to 200 €/$ each.
So let's continue thinking this way.

[wolfgangr]

Based on the prospective insight so far, let's try to structure a bit:

Architecure and data path:

• Encoder TTL signal right fed into dedicated input lines of grblHAL-controller
• if any DRO, it might simply consume split signals
• try to free timers so that quadrature data is updated by DMA without consuming interrupt time or CPU time for polling
• minimal CPU load data capture at each pulse genereation (i.e. 30 kHz or much faster), e.g by writing both steppers and linear input coordinates for asychronous evaluation to some buffer

Evaluation

• outside the stepper routine
• at e.g. 100 Hz just to say a number, but still to be verified
• average, maximum, trend in deviation of position
• lag behind in time (do we really want / need this)?
• stalled/lost steps?
• motor warning, overcurrent, overvoltage if supplied by the stepper driver
• can we calculate skd of "weightet punishment function" to discern further action?

Events to be recognized

• in backlash
• elastic deformation of the machine due to e.g. working force
• lost cycles
• homing errors / missing home
• premature warning, either from driver circuitry or maybe from pattern of deviation

Actions possible

• load current absolute position as new home
• decrease feed rate
• add backlash compensation segment to buffer
• emergency stop - restart possible after homing to absolute encoder position
• soft stop (i.e. deceleration without loosing steps): just continue working

Potential for adaptive auto learning

• backlash prediction
• adopt acceleration close to, but in safe distance to error conditions
• may be asymmetric (e.g. for lifting heavy table, or in relation to tool force)

[wolfgangr]

referring to @terjeio
#287 (comment)

As far as I understand, I'd expect some main loop running where at short intervals skd of "planner" is

Yes, this
....protocol.c#L227-L369
which calls protocol_execute_realtime() at a high frequency. Outputting stepper signals is done in an interrupt handler.

Peeking down the rabbit hole a little bit ('till Ln 600 where the rt_exec line up), it smells like RT-style cooperative multitasking.

So while there is considerable chance that code plugged in here is called several times per second, there is no reliable deterministic way to ensure it is called at all, not even to say at a precise interval, correct?

Not the perfect place I'd like a superviser code to live in?
So we might add another timed interrupt level? But this may not only consume rare timers, but also add potential of race conditions and unpredictable behaviour, correct?

Do we have enough CPU time in spare to risk this?
And if so, wouldn't then it even be better to attach position check to stepper.c nevertheless?
Tiny incremental calculations, as long as there is not much deviation of position?

Or do we already hit the amount of complexity to be handled by a single controller?
Might it be an approach to offload supervision of position to a dedicated gadget?
Reading both stepper and linear encoder signals and generating some warning pulse to one of the interrupt controlled inputs at grblHAL?

[terjeio]

I have not gone through all your comments yet, summer is here and I have plenty of other stuff to do. Nevertheless a few comments:

it smells like RT-style cooperative multitasking

True. However, time critical events are taken care of by interrupts so as close to realtime as possible. And all drivers supports a interrupt driven 1 ms systick which may be used to hook time critical code into. Some drivers also supports a microsecond resolution "timer" which can be used to time events hooked into the main loop which by the way runs pretty frequently most of the time.

no reliable deterministic way to ensure it is called at all, not even to say at a precise interval

How precise is a precise interval? Less than 10 microsecond jitter? Sub microsecond?

Not the perfect place I'd like a superviser code to live in?

This depends on the requirements, a FPU equipped MCU running at a reasonable high clock rate may work perfectly well. Tasks can be attached directly into the main loop or it can be registered with the core task manager which mainly uses the 1ms systick timer interval to check for and run tasks.

So we might add another timed interrupt level? But this may not only consume rare timers, but also add potential of race conditions and unpredictable behaviour, correct?

That depends on how the code is written. I like to add an immediate task on some interrupts and let the foreground process execute the code. This will add a non-deterministic delay which I have not bothered to measure - a few microseconds worst case for the most capable MCUs?

Do we have enough CPU time in spare to risk this?

Depends on the MCU, anyway unpredicable behaviour should not be coded for...

And if so, wouldn't then it even be better to attach position check to stepper.c nevertheless?

I am not sure, driver.c might be a better place? Or a plugin? IMO the most critical part of the solution will be how to handle encoder input - it should be restricted to MCUs having a sufficient number of quadrature encoder peripherals? Doing the decoding in code might be a non starter due to high frequency inputs?

Or do we already hit the amount of complexity to be handled by a single controller?
Might it be an approach to offload supervision of position to a dedicated gadget?

Fast MCUs like the iMXRT1062 or STM32H7xx series might handle this without breaking a sweat. And complexity will only increase if offloading...
The complex bit of this is, IMO, how to handle synchronized backlash among multiple axes - the needed motion should happen without the grblHAL core knowing anything about it?

[wolfgangr]

referring to @terjeio
#287 (comment)

Maslow kinematics combined with a supporting driver is the closest attempt from my side.
https://github.com/grblHAL/core/blob/master/kinematics/maslow.c
It uses PID control for XY and positioning.

hm ... I try to find the PID implementation in that code, but I only can see parameters hold, and static coordinate transfer.

Is KINEMATIC really a good place for dynamic correction?
When I search the core for KINEMATIC_API, I think it is not wise to change coordinate mapping during processing, right?
I think the stepper might get confused when it finds a different mapping of coordinate than the planner used some fractions of a second earlier?

Here is the driver I tinkered with some years back:
driver.zip

ah, OK, I see some PID there for positions.
Did you collect any test data with this driver?
Could you estimate whether it is possible to come close to stepper precision by PID controlled servo control?

[wolfgangr]

I have not gone through all your comments yet,

Never mind.
Many thanks for every minute of your rare time you spend to bridge the huge knowledge gap :-)

all drivers supports a interrupt driven 1 ms systick

Ah, I was looking in vain for this figure for this systick frequency.
That might be right what I was looking for.

I think 1 ms should be enough for most emergency like conditions - at least much faster than any human operator might react.
Regarding the dynamic kind of backlash compensation, I'll do some estimation based on my M3-threading challenge...

main loop which by the way runs pretty frequently most of the time.

Impressive figure, indeed, but only "most of the time".
I read: enough time to perform computationally intensive tasks, as long as the are not time critical.

Essence so far:

• encoder input shall be evaluated by dedicated quadrature capable counters
• at every step generated by stepper.c internal and external positions are saved to a buffer
• every 1 ms systick conditions are evaluated, correction calculated and emergency action, if required, are triggered
• in the main loop, non-time-critical tasks like adaptive update of parameters or restart after emergency stop might be performed

[wolfgangr]

Checking for feasibility of timing:

Start at our "M3-thread-mill-challenge".
E.g. here (https://github.com/FreeCAD/FreeCAD/blob/main/src/Mod/CAM/Data/Threads/metric-internal-6H.csv)
we find appropriate data, including

• tolerance class 6H, which we can trace to -0 / + 6 µ for 3mm holes
• at least 0.1 mm = 100 µ allowed span for the key diameters (minor/pitch/major)
  experience shows that a tenth (~ 10µ) of this span slightly changes tactible thread behaviur
  taking half of that as the radius we are at 5 µ - nearly the same value - plausible.

Travel 6µ in one systick of 1 ms yields a feed rate of 360 mm/min - quite a high value for such tiny threads.
At this small circles, acceleration is the limiting factor.
Standard setting of acceleration in grbl is 10 mm/s².

A common M3 thread mill bit has a diameter of 2.35mm.
To mill a outer circle of 3mm + some "crest", the mill bit spirals at about 0.7 mm, i.e. r=0.35 mm
Some basic physics - e.g. https://en.wikipedia.org/wiki/Circular_motion#Acceleration yields
v = sqrt(a * r) = 0.0019 m/s = 112 mm/min. Sounds reasonable for such a tiny tool.

6µ / 0,0019m/s = 3.2 ms - more than 3 systicks while traveling the allowed tolerance.

Travelling the whole cycle of r=0.35 mm at 0.0019 m/s takes 1.16 seconds:
More than thousand systicks to get skd of self learning parameter "accomodated" to the tooling situation.

Regarding backlash of a single axis, deviation builds up at the turning point.

• Tolerance error is measured radially
• actual feed proceeds tangetially (in case of one axis resting in backlash)
• planned feed runs on the circumference
• the distance from tangent to circumference: error = r * (1 - cos (eps))
• given 0.35 mm toolpath radius and 6 µ allowed distance, eps = 10.6 deg
• so the tool has moved travel = sin(eps) * r = 65 µ in tangetial direction until radial backlash errors mounts up to the allowed limit of 6µ
• ... which takes 0.034 seconds
• i.e. we have 34 systicks to get the backlash correction taking up

Essence: sounds feasible until here

[wolfgangr]

Next feasibility check: peripherals for quadrature capture
in particular referring to my favourate controller STM32F411

As Terje pointed out

IMO the most critical part of the solution will be how to handle encoder input - it should be restricted to MCUs having a sufficient number of quadrature encoder peripherals?

In the Datasheet for STM32F411xC STM32F411xE we read

3.20.2 General-purpose timers (TIMx)
4 full-featured general-purpose timers: TIM2, TIM5, .... TIM3, and TIM4.
....
TIM2, TIM3, TIM4, TIM5 all have independent DMA request generation.
They are capable of handling quadrature (incremental) encoder signals

In an accompanying "Table4" we read that only TIM1, TIM2, TIM3, TIM4, TIM5 are capable of up/down counting, which is a reasonable preassumption for capturing quadrature encoding.

The role of TIM1 is a bit unclear:

3.20.1 Advanced-control timers (TIM1)
The advanced-control timer (TIM1) can be seen as three-phase PWM generators multiplexed on four independent channels.
... It can also be considered as a complete general-purpose timer.
.... If configured as a standard 16-bit timers, it has the same features as the general-purpose TIMx timers.

So while not explicitly listed as quadrature capable, from this text it might be understood that TIM1 is quadrature capable as well.

While the much more advanced STM32H7 counts a total of 22 timers (compared to 11 on the F411), its datasheet reads as there were only the same 4+maybe TIM1 up-down-timers capable of quadrature capture.
I haven't yet checked for other architectures.

In https://github.com/grblHAL/STM32F4xx/blob/master/Inc/driver.h we find:

// Define timer allocations.
#define STEPPER_TIMER_N             5
#define PULSE_TIMER_N               4
   ...
#if !defined(PULSE2_TIMER_N) && STEP_INJECT_ENABLE
#define PULSE2_TIMER_N              3
  ...

So we might reluctantly hope that Timers 1,2,3 might be available for quadrature capture -
but only if

• TIM1 is really capable of quadrature capture
• there are no other places where timers / counters are allocated.
• we dont need Timer 3 for STEP_INJECT_ENABLE

Might STEP_INJECT_ENABLE smooth spindle movement and thus ease feedback control?
If so, where it possible to reallocate it to one of the "minor" i.e. not up-down-capable timers?

And what about stepper and pulse timers?
Could they be re-allocated?
With my limited understanding, I don't see why they might require up-down functionality.
But maybe there are other features not provided by the lesser timers.
Or may be that the other timers might require some special coding?

I see, @terjeio , again, may be the first one to help out here?
sorry for that ....

[wolfgangr]

Have desperately been looking for timer interfacing in the encoder code,
... until I found ... this:
https://github.com/grblHAL/STM32F4xx/blob/286ad6825182cd738e8e9005e7369f91d38bafe0/Src/driver.c#L2131C1-L2132C2

It looks like a polling code, crafted in C for efficiency, reading two GPIO lines A,B, testing for change and updating an counter if appropriate.
It might even be called in a timed loop fast enough not to loose any ticks, but some lines above I see something that looks like interrupt configuration. This way, it is only called if there is really a signal change at the port.

I know that GPIO interrupts in STM32 are a rare ressource, too.
May be we can share Interrupts across multiple GPIO?
At least if the belong to the same group, so no surplus code is called?

This way, we would not require any valuable timer ressources.

[wolfgangr]

... watching some rabbits dancing in https://github.com/grblHAL/STM32F4xx/blob/master/Src/driver.c
... chasing some of them downstairs their holes ...

Looks like stepper timing comes down to what I'd guess as "normal" timer
https://github.com/grblHAL/STM32F4xx/blob/master/Drivers/STM32F4xx_HAL_Driver/Inc/stm32f4xx_hal_tim.h

while quadrature stuff I'd expect to end out in the "extended" timer HAL here
https://github.com/grblHAL/STM32F4xx/blob/master/Drivers/STM32F4xx_HAL_Driver/Inc/stm32f4xx_hal_tim_ex.h

... and I see that systick µs and ms are drawn from skd of core hardware and obviously do not consume any extra timer:
https://github.com/grblHAL/STM32F4xx/blob/master/Drivers/CMSIS/Include/core_cm3.h

searching the web for CMSIS, I learn that while in said core_cm3.h there is not even an FPU, in cm4 (aka Cortex4) there is both an FPU and DSP with SIMD facilities available.
May that come handy when we are caclulating statistic, trends, PID, feedback of positions?
Hope this is available in the C surface....

Back to the timers:
might try to shift STEPPER_TIMER, PULSE2_TIMER PULSE2_TIMER out of TIM1... TIM5 ?
Time to arrange skd. of development and debug setup?

---

reached the end of driver.c
... still not found any setup for a hardware quadrature counter ...
even SPINDLE_ENCODER is counting one direction only

[wolfgangr]

spent quite some hours now, comparing the driver.c of all STM32 variants and for the 'celebrated flagship' Teensy iMXRT1062.
I don't find any implementation of reading quadrature by dedicated peripherals.
And even if encoder plugin provides for up to 5 encoders, I found no driver supplying more than one.
So there is still work to do...

All grblHAL drivers I checked use the same scheme of interrupts triggered by rising and falling edge of two input lines per encoder.
The STM32F4 appears to be the most complex (so I assume the most elaborated?) variant, so let's point to it:
https://github.com/grblHAL/STM32F4xx/blob/master/Src/driver.c#L2131

• read the two lines
• combine to a 16 bit status value
• or it with a 16 bit status check mask
• perform up/down count depending on status
• trigger subsequent action

There is actually a counter based encoder outside grbHAL, but within our "ecosystem":
https://github.com/terjeio/GRBL_MPG_DRO_BoosterPack/blob/master/Firmware/TM4C123/driver.c#L324
We see, there are as well at least 3 prepation instructions in the IRQ-routine to prepare for trigger of subsequent action.

A 5µ encoder moving at (not really realistic) 10m/min generates pulses at 33 kHz, which for a 40+-MHz CPU should not pose too much stress. So for the moment, I'm inclined to implement encoder input by irq triggeres on lines and keep the timer configuration untouched.
In software, we may easily extend the counters to 32 bit to cover any realistic travel without further need of carry over handling, as it were required for 16-bit-counters.

"Further action" includes at least pushing systick time and counter value to some buffer (i.e. spacetime coordinates) for further evaluation.
Glancing over STM32 datasheets, it might even be possible to implement that with coupled timers and DMA stuff, without even bothering the CPU with any interrupt. But not easy to be implemented for the casual programmer....

We might instead try to put all 6 encoder lines into the realm of one single EXTI15_10_IRQHandler, thus turning the burden of this lumping into a virtue: one interrupt routine rules 3 encoders in one stroke.

I've ordered the encoders and steppers for my mill and plan at least to implement data capture.
Let's see if it works, and then what we can do with that data.

[terjeio]

I know that GPIO interrupts in STM32 are a rare ressource, too.
May be we can share Interrupts across multiple GPIO?

Not sure what you mean by this. If sharing an interrupt line for a given pin number across GPIO ports then that is not possible.

At least if the belong to the same group, so no surplus code is called?

Only the STM32F1xx driver (for the 128K flash variants) is nearing the code limit, IRQ code should be fast so inlining is better if for high frequency handlers.

This way, we would not require any valuable timer ressources.

IRQ capable GPIO pins are more valuable? I prefer a 32 bit timer for the stepper interrupt, but a 16 bit one can be used by changing the prescaler to get the required range. So remapping is possible.

there is both an FPU and DSP with SIMD facilities available.
May that come handy when we are caclulating statistic, trends, PID, feedback of positions?
Hope this is available in the C surface....

It is - via assembler.

And even if encoder plugin provides for up to 5 encoders, I found no driver supplying more than one.

The encoder interface is aimed at hand turned encoders - for menu selection and/or jogging. So not suited for positioning?

So for the moment, I'm inclined to implement encoder input by irq triggeres on lines and keep the timer configuration untouched.

I do not know, my hunch is that it is better to read the position at regular intervals, at least PID control requires that (fixed sampling rate) and let the hardware take care of keeping track of the encoders/position. Handling of underflow/overflow would have to be by IRQ though.

Might STEP_INJECT_ENABLE smooth spindle movement and thus ease feedback control?

This is aimed at plasma tool height control (THC) and is for moving an axis (Z) without notifying the core. The axis moved by this should not be moved by the core at the same time. This functionality could be replaced with your dynamic backlash compensation?

I'd just recall the fact that in grblHAL, there is already a spindle-sync feature for lathing threads in place. Albeit under different circumstances, it already implements movements waiting and synced for some external signal, thus allowing deviation from the pre-planned spacetime path beyond a mere change of speed.

This keeps the position in sync with the spindle by changing the time between stepper pulses, not by adding/removing any. Backlash compensation requires additional pulses. And the current spindle sync code is based on having a constant (programmed) feed rate over the synced move. I am not sure it is suitable for rigid tapping, certainly not so if the spindle is moved by hand since the code does not detect direction. FYI I have looked into some electronic lead screw (ELS) projects for ideas for how to implement rigid tapping but have not started coding for it yet. I fear it will be quite a bit of work and will lead to a lot of broken taps...

Projects exists that adds servo control to regular steppers, perhaps some ideas can be extracted from those?

[wolfgangr]

Encouraging to watch my thinking converging towards your knowledge ;-)

... sharing an interrupt line for a given pin number across GPIO ports then that is not possible.

Still decaes old remains of 6502 architecture in my mind.... but ... :
https://github.com/grblHAL/STM32F4xx/blob/master/Src/driver.c#L3331

void EXTI15_10_IRQHandler(void)

... although not configurable, there is a lumping of 6 pins to one interrupt as it seems.
So we might try to put all 3 encoders there to catch all in one handler.

---

Just let's explore the alternative of using timers for encoder reading.
I think the reason I found no code for using timers to capture quadrature encoding is that there is none reqired at all.
Last night I crafted a quick hack with a bluepill along this tutorial:
https://deepbluembedded.com/stm32-timer-encoder-mode-stm32-rotary-encoder-interfacing/
After correctly clicking the hardware part in the MX UI of STM32CubeIDE, the magic boils down to a mere reading of a register:

      uint32_t  Encoder =    TIM2->CNT ;
      printf("enc = %d, push = %s \r\n",   Encoder, pressed);

Nice exercise in STM32CubeIDE, at least.
As I see that most of the code in the Drivers section in grblHAL for STM32** resembles that code, my question is how I could merge my hardware desing changes crafeted in STM32CubeIDE / MX into grblHAL.
I don't see the notoric *.ioc in the grblHAL files.
Looks like the platformIO stuff plays in somwhere... somehow ... there ....
Any pointers for understanding would be great.

---

Next step is to acquire timestamps.
Both stepper pulses and encoder ticks come in the range of dozens of kHz, so 1ms systick is far to coarse.
So my idea was to interpolate the µs by reading of the systick timer.

I found
https://github.com/grblHAL/STM32F4xx/blob/master/Src/driver.c#L3331C6-L3331C27
static uint64_t getElapsedMicros (void)
and after some hours of digging through a pile of STM and CMSIS files I'm quite sure that it quite closely resembles that idea.

However, it looks a bit complicated for beeing called in interrupt handlers every couple of µs.
Above all, it's the do{}while loop that bothers me...

Where is the complexity coming from?
Do we have to handle the case that people set systick to some larger value than 1ms?
I see we have to provide for continuity around systick interrupt occuring.
And maybe some of the apparent complexity comes from intended parametisation for different controller types.

Can we short this down?
Or may we even have a dediacated counter for generating µs timestamps?

[wolfgangr]

I do not know, my hunch is that it is better to read the position at regular intervals, at least PID control requires that

hm...
But at which rate? Speed is varying at a large scale.
We might collect stepper internal / encoder external / stepping pulse time data sets, but they are not at regular intervals, too.
Or does the span of a segment suffice for a PID to stabilize? I don't think so.

A web search for "asynchronous PID" leads me astray to AI-based PID tuning ...
well ... really astray? Or ist that what me might end up in the end ??? ....

For the moment, my idea is to extract "features" such as slope, curves, kinks from the collected data, but haven't yet decided how.
For processing past data skd of least square fitting might be approriate.
I think for ongoing flow of data, there are algorithms around in the wizardry of control techniques that perform continous processing.
May be we could even offload data processing to the PC?
I think of skd if dryrun backlash estimation cycle as part of the homing routine for example.
But can we store / deliver data at a sufficient rate at all?

[terjeio]

Still decaes old remains of 6502 architecture in my mind....

The 6502 is what I started MCU programming on - I have written a fair bit of assembly code for it. Acorn BBC computers, RiscOS, and now ARM...

Or may we even have a dediacated counter for generating µs timestamps?

I am planning to add an API for at least one, similar to the stepper interrupt, that can be claimed by plugin code. At least I need one for high speed step injection when polling is not adequate.

We might collect stepper internal / encoder external / stepping pulse time data sets, but they are not at regular intervals, too.

Steps are delivered in "segments" lasting ~5 ms - during which step rate and direction is not changing. I rely on this in the spindle sync implementation to provide a constant PID sample interval.

May be we could even offload data processing to the PC?

IMO do not even consider that for grblHAL, Klipper or LinuxCNC are better options?
Or offload step generation to a FPGA or a second (on chip) MCU?

But can we store / deliver data at a sufficient rate at all?

For that the code should be as lean and fast as possible. Use KISS. The 'proof is in the pudding'?

[wolfgangr]

Or may we even have a dediacated counter for generating µs timestamps?

I am planning to add an API for at least one,

Sounds promising. You do not need one of the rare up-down-capable "General Purpose" timer need for that, I hope?
There are only 4 of them, or 5 including TIM1, in STM32F411.
On the other hand, I managed to understand getElapsedMicros at least to the point that I may trust to use it for timestamps.

Use KISS.

... and may be bin the idea of timestamping any step outputs and encoders for complicated statistics altogether?
Try a simple PID?

Steps are delivered in "segments" lasting ~5 ms

Hm, I think that is to crude for PID-timing.

PID control requires that (fixed sampling rate)

Is it written in stone that PID has to run on time base?
Could it not be a spatial interval as well?
I just consider updating PID at step rate. As far as I understand till now, this is the spatial interval of the fastest moving axis, right?
Of course there might be a challenge to keep PID state consistent over segments, when direction changes.

The encoder interface is aimed at hand turned encoders -

Ah, see that in the code.
So to say, the encoder plugin is not hardware agnostic as I had expected it?
Or the QEI-driver is not application agnostic?
Common linear encoders deliver a A-B-pattern and a regular index signal (not sure whether I need that at all).
So they might be implemented as subset of MPG-encoders .... as I thought to keep the HAL-idea tidy ....
But of course a "quick'n dirty"-implementation might even be more KISS...

So remapping is possible.

What's "the grblHAL way" intended to do this?
When I use timers for encoders (in terms of KISS), I need 3 timers out of TIM1 ... TIM5 (quadrature capable).
And as far as I've seen in my test on bluepill, the input lines for quadrature capture are a fixed per-timer assignment.
So I might have to move quite some other GPIO assignments out of the way.
Can I rely on moving timers and GPIO lines around just by assigning them in the map file?
That's what I expected from the idea of grblHAL, at least....
But the longer I grep through code, the less sure I am about that.

[terjeio]

You do not need one of the rare up-down-capable "General Purpose" timer need for that, I hope?

I need a timer for it - but it does not need to be a up-down capable timer.

Steps are delivered in "segments" lasting ~5 ms

Hm, I think that is to crude for PID-timing.

It works for spindle synced motion.

Is it written in stone that PID has to run on time base?

I am no expert on PID control so I do not know.

I just consider updating PID at step rate. As far as I understand till now, this is the spatial interval of the fastest moving axis, right?

Yes - but may be divided by the AMASS level.

So to say, the encoder plugin is not hardware agnostic as I had expected it?
Or the QEI-driver is not application agnostic?

It is, but not ideally suited for this application? Since it uses callbacks for encoder events?
The spindle sync encoder data is read when needed, not when position changes.

You will have to block new steps coming from the core until the backlash compensation has catched up? And after backlash compensation has completed compensate for linearity deviations in the screws? So two different algoritms are needed?

and a regular index signal (not sure whether I need that at all).

It can be used for homing if available and from a linear scale?

Can I rely on moving timers and GPIO lines around just by assigning them in the map file?

I have not coded the drivers to support all possible variants of pin allocations - that is a lot of work that I am prepared to do just for the sake of it. So if some combinations are not supported they have to be added as needed.
Moving GPIOs is generally better supported that moving timers, but each MCUs usually has their own way of implementing stuff like which pins are interrupt capable etc. E.g. the STM32 family puts limits on how many and which pins can be assigned to interrupt lines, and some needs to be interrupt capable (e.g. limit, control and possibly aux inputs). So pin assignments are sometimes a tricky cabal to solve.

BTW the new dual core Raspberry Pi RP2350 is an interesting MCU, well suited for grblHAL now that is has gotten a FPU and more GPIOs in the 80 pin package. Perhaps not well suited for this application though...

[wolfgangr]

I really admire your energy to face the challenge of all those different platforms.
I think a dual core Raspi MCU would be far beyond my capabilities to cope with.
What I had considered to implement this project is LinuxCNC on a recent RasPI (is it 4 core I think?).
But haven't even scratched the surface of LinuxCNC yet.

So let's continue on Blackpill.
How does PID fit into the picture?
My mind is swirling between the source code and the data sheets.
In particular around stepper.c, stepper2.c; and all the HAL stuff and system libs required to understand all this.

I like the approach of stepper2.c
I'm not sure whether I'd keep the Bresenham-approach. It's initial advantage - integral numerical precision in the long range - is of no use any more if in the end we adopt to position feed back.
But of course, it is already there, proven to work and well integrated into the planner.

However, it add's loads of complexity (in terms of code lines, learning time and processing time).
It adds differential discretisation noise by forcing steps of independent axes into a integer raster. OK, AMASS alleviates this, at the cost of even more complexity.

I think the stepper2-approach to genereate ticks on independent timers per axis is much more smooth. Although I'll not go for precise evaluation, I'd expect it injects less noise and allows smooth small amlitude feed back.
The price is the requirement of a distinct timer per axis.

Following the datasheets of STM32F411, TIM9, 10, 11 should fit the purpose.
TIM3, 4, 5 can capture the encoder signal then, requiring no code or CPU load and nevertheless delivering precise position without discretion errors in the time domain.
TIM1 may be used to generate PWM-signal for Spindle-RPM. It shares PIN assignment with USART anyway, so I can seperate the issue of modbus VFD control.
TIM2 is left for internal use. Looks like there are no PINs left for outer world connection anyway.
With this configuration, 3 Axes are pushing the Blackpill timer inventory to the edge - is that OK?

Back to the question where to interface to the existent code.
As far as I can see, stepper2 does not even interface to the planner buffer.
So has it even be used at all for multi-axis synced moves? Neither the code nor the accompanying web page give any sign of that.

And how does PID fit to Bresenham? Does it fit at all?
A quick internet research on "PID multidimensional" does not yield any relevant results.
PID is single dimension by concept.
It's Bresenham wich keeps multiple axes in sync.
But this one is deterministic and has no idea of any error signals to feed back to wherever.

Let's step back and reconsider what we want to achieve - in falling priority:

• movement keep on track
• movement keep on going
• movement ending at the planned segment end point
• speeds at end point in the planned relation to each other (i.e. direction of movement)
• acceleration within configured limits
• speed as planned
• acceleration as planned
• total time as planned

If we look at priorities in that list, we see that it is not time, but position that counts.
So time may only be of secondary importance - in terms of progress along a planned line.
... keep on thinking ...

[wolfgangr]

Two night's of sleep later, I 'd rather dismiss the idea of individual timers per axis.
Does not feel right, does not feel KISS, does not feel predictable.

What if instead we have a single high tick rate - pulse generator and PID update, synced for all axes?
I'd like the idea of some binary even fraction of 1 MHz.
No DIY-budget stepper runs anywhere close to 3000 rpm = 50 rps = 10 kHz main step rate at the common 1.8 deg step angle.
So we have a factor of 100 to 1 MHz which we might divide to microstepping and our timer ratio.

E.g. 8 x 8 = 64 easily fits: step rate = 1 MHz / 8 = 125 kHz, microstepping = 8.
I.e. ticking a timer by prescaling sysclock to 1Mhz, reloading at 8µs and generating a compare-event halfway at 4µs.
At one of those events, any 8 µs load a output-step-pattern for all axes, maybe even by DMA, maybe even from a fifo buffer.
At the other, 4µs later, clear all outputs again, similiar than it is done at the moment with two timed interrupts.
And somewhere inbetween, do the PID stuff in a very time efficient way: integer arithmetics, no divisions - is this possible?
How can we estimate the time of code execution?

A DM556 accepts up to 200 kHz with am minimum step time of 2,5 µs - so on the output side, we have a comfortable safety margin.
It's rather cheap, and I'd expect any of its more expensive bretheren to match at least this rate.
With a microstepping of 8 we might already smooth operation, since any step produces only small changes in coil current. We might extend the effect with a microstepping of 16 at max 2300 rpm or to 32 at 1170 rpm.
Alternatively, we may leave some gap by selecting lower microstepping than the speed allows. Then, even at max speed, several clock steps are off between adjacent pulses, resulting in smaller integer ratio for consecutive possible clockrates. This behaves comparably to AMASS, but synced to a fixed pattern in time now instead to fixed stepping of the main axis in space there.

If wee feed distance between planned and measured postion to PID, in the Differentail realm, ds/dt aka v=speed is what dominates operation. So, as long as we get parameters right, we expect PID genereate pulses proportional to the demand of proceeding move requested, right als Bresenham does, as long as we are close to the planned path.
So the cutoff frequency between D and P realm might be configured somewhere below this 125 kHz max step rate.

In the Proportional realm below, speed ist adjusted to match planned speed - which may vary over a segment.
Integral realm in the long run works out accrued differences. Thus it helps to stay on track and come out at planned segment end points.
So a first guess for the P-I transition frequency might be somewhere in the range of segment length.

The syncing of axes might require skd of common slow down (or even speed up?) proportional to all axes, as soon as one axis accrues drift. This way, other axes "wait" until the drifting one comes up. May this be another PID loop? maybe a bit slower than pulse generators?

The dynamic range of axis movement might raise some extra challenge. From e.g. 1mm /min to 1000 mm/min and even more we easily may have to span 3 orders of magnitude and more. Maybe PID tuning had to be adapted?

And when ordering steppers for my mill to go moving, I encountered another issue related to dynamic range:
Acceleration is limited to a fixed rate in grbl (and in grblHAL as well, if I haven't missed something).
However, if we look at a typical speed-torque curve of a stepper, we see a small plateau of max torque at very low speed which is falling in an hyperbolic fashion (i.e. ~ constant power) beyond that point. Still not sure whether this is due to coil inductance limiting current at higher pulse frequency, or the electromagnetic force generated by a turning motor, or both, or maybe even both perspectives referring to the same mechanism.
Anyway, a fixed acceleration limit misses chances for fast moves and corrections at small speed (such as cutting M3 thread with a backlashing drive chain....) and overestimates machine capability at high speed (eg. producing audible loss of pulses at my ttc450 at a mere 800 mm/min Y-jog).
May I just expect that getting this right might tremendously enhance performance of a stepped drive system? Let's see....

Grey is all theory. But clouded the forces of reality are as well.
Inbetween, some simulation may help to gain insight.
LTspice? Nice fit to theory, but hard to translate to software and mechanics.
Some matlab, Perl, Python or even spreadsheet calculations with instructive plots? might be the easy way ...

But isn't there a simulation driver in grblHAL, too?
Maybe this is the hook to a simulation tool closest to the "real thing"?
And might help us push development ahead before we move to the real target, where access to debugging information is restricted?

[terjeio]

What if instead we have a single high tick rate - pulse generator and PID update, synced for all axes?

For some drivers I believe you can add an additional compare/interrupt to the existing step timer, which typically runs at 20+ MHz.

No DIY-budget stepper runs anywhere close to 3000 rpm = 50 rps = 10 kHz main step rate at the common 1.8 deg step angle.

grblHAL is not only aimed at budget DIY machines, my aim is to make it a viable alternative for the top end of the hobby market, and even for semi-professional and professional users. This includes those who runs high RPM servos. Step rates up to around 300KHz is currently possible, I do not want to degrade that.

And somewhere inbetween, do the PID stuff in a very time efficient way: integer arithmetics, no divisions - is this possible?

The current float PID algoritm can be complemented with a fixed point version. The divisions can be replaced with multiplications if the sample rate is constant.

The syncing of axes might require skd of common slow down (or even speed up?) proportional to all axes, as soon as one axis accrues drift. This way, other axes "wait" until the drifting one comes up. May this be another PID loop? maybe a bit slower than pulse generators?

This is possibly where the real problem lies - and is why replacing the screws with high precision ballscrews is the easier option?

But isn't there a simulation driver in grblHAL, too?

Kind of, runs on Linux and Windows. You will have to add simulated encoder input to it to make it usable? Not easy?

And might help us push development ahead before we move to the real target, where access to debugging information is restricted?

Perhaps not too restriced? The spindle sync code logs data to RAM that I can fetch and display, ioSender can render it. For backlash compensation similar logging could be used since not too much data is generated - after all backlash events should not take too long.