IRQ triggering stops after a while

[mhaberler]

not sure where to record this - maybe should move to mk/mk; more of an observation at this point and not yet critical

given the following config:

newthread servo nowait posix   # <---- NB non-RT thread
loadrt hostmot2 debug_idrom=1 debug_module_descriptors=1 debug_modules=1
loadrt hm2_soc_ol config="firmware=socfpga/dtbo/DE0_Nano_SoC_DB25.7I76_7I76_7I76_7I76.dtbo num_encoders=4 num_pwmgens=4 num_stepgens=3" timer1=211812352

addf waitirq.0                 servo

addf hm2_de0n.0.read           servo
addf hm2_de0n.0.read_gpio      servo
addf hm2_de0n.0.write          servo
addf hm2_de0n.0.write_gpio     servo
start

the IRQ count freezes at some point, visible by:

halcmd: show pin hm2.0.
Component Pins:
  Comp   Inst Type  Dir         Value  Name                             Epsilon         Flags
    75        u32   OUT    0x00022F17  hm2.0.irq-count                          0     <--- stops increasing
    75        u32   OUT    0x00000000  hm2.0.irq-missed                         0
    75        u32   OUT    0x00000000  hm2.0.read-errors                        0
    75        u32   OUT    0x00000000  hm2.0.write-errors                       0

If I drop the 'posix' flag making it an RT thread, this currently does not happen (letting it run overnight now. update: 2days later no IRQ lost if using RT thread)

gut feeling: there is a race condition in acknowledging interrupts - if posix scheduling happens too late for the current cycle and the pending IRQ is not acked before a second IRQ is posted, things get stuck after 50-100.000 interrupts

if this is the case - I'd need gpio pins hooked to an LSA to nail that condition - a possible fix would be decouple IRQ acknowledgement and continuing the waitirq function: ack the IRQ right away in the driver, and keep userland out of the IRQ ack business altogether

this might mean a custom driver (based on uio_pdrv_genirq) but it would at the same time get rid of the second system call (write) in waitirq(), something I would like to do anyway, and given we have machinekit-dkms in place it might not be an undue burden

I'd be grateful for a hint how to take some GPIO pins away from the stock GPIO driver (or hm2 for that matter, not sure how that could be done) so they could be used as scoping pins from HAL (i.e. directly manipulated in hal, not through some driver thread function)

[dkhughes]

Hmm...I've been testing the waitIRQ version heavily the past week and I haven't seen this behavior yet. Do you just wait an extended period of time to make it happen? 100,000 interrupts is approximately 100 seconds? Or do you use a rate other than 1kHz? When you say stuck, do you just miss an interrupt or two or does mk lock up and need to be killed? The zynq version has been running for hours continuously without any symptoms of hard hanging except on shutdown. Occasionally, 1 in 10 or 15, on shutdown the rtapi will begin spamming timeout and no connection errors and require a manual intervention. That's the only error I've run into with wait irq.

What should I do to try and duplicate?

As for GPIO, I will try to post zturn today or tomorrow, and I can help get a custom config that will expose some timing gpio pins for you. I've been doing u-boot debugging to get NFS booting on the microzed, but I think I can find a little time to finish a basic zturn design.

[mhaberler]

well with posix it happens within minutes, yes; I have it running now for 20hrs with an RT thread and it did not happen

stuck means: the waitirq function just sits in the read().

The shutdown hang is something else I still need to address - seems the device read is not interruptible.

'timeout' typically means either rtapi_app went away/crashed - the command thread is separate so should run whatever is happening in HAL threads

duplicate: adapt the config= line in above config fragment and run it, then observe the hm2.0.irq-count pin. If it stops increasing, you see the same symptom. I saw no suspicious linuxcnc.log or dmesg entries.

[cdsteinkuehler]

On 6/10/2016 7:24 AM, Michael Haberler wrote:

well with posix it happens within minutes, yes; I have it running now for 20hrs
with an RT thread and it did not happen

stuck means: the waitirq function just sits in the read().

Where's the interrupt code? I'm either looking in the wrong file(s) or
the wrong branch(es).

Usually, a problem like this means an interrupt was incorrectly cleared
(without being properly seen by software), but I'm not sure if the issue
can happen that way since there's a single IRQ from the hm2 hardware.
The process I'm talking about is like so:

• Interrupt A occurs
• Software IRQ process is triggered and sees IRQ A happened
• Interrupt B occurs
• Software clears the IRQ A, which ALSO clears IRQ B!
• ...software waits forever for IRQ B

I'm not sure if something like this can happen in the IRQ handling for a
UIO device, but IIRC the ARM core only really has one actual hardware
interrupt (two if you count the FIRQ), so it might be possible even
though the hm2 only generates one interrupt.

Similar deadlocks are also possible with improper handling of interrupt
enable/mask bits.

Charles Steinkuehler
charles@steinkuehler.net

[mhaberler]

IRQ code is here: https://github.com/machinekit/machinekit/blob/master/src/hal/drivers/mesa-hostmot2/hm2_soc_ol.c#L615-L643

that above scenario could well be it!

[dkhughes]

I was thinking about this while driving and I guess an easy way out would be to have a timeout condition on the blocking read. If the timeout occurs, then we check to see if the flags need to be reset, then make the choice to block until the next interrupt by restarting the read, or we finish the function normally and let other hal calls get their share of the CPU.

From what I've read about kernel latency to interrupts, the double interrupt described above is a real possibility, and the only fix for that is a preemption kernel. If posix is a requirement though, the timeout on read will at least keep the bits flowing even if they're serviced irregularly.

The other option is simply slowing down the interrupt rate since the hardware cannot support the current rate with the latency included.

[mhaberler]

this might touch on the same issue

is there any point in experimenting with level- vs edge-triggered IRQ's?

[mhaberler]

semi-related: the shutdown hangs if employing a read() on a device file (or some other fd for that matter; I am experiencing the same with eventfd(2)) - the scenario is:

• halcmd shutdown causes the thread to be stopped through hal_exit_threads() ->...->rtapi_task_delete_hook() which does a pthread_join()
• the pthread_join() waits on read() to terminate
• all this is happening with rtapi_app holding the rtapi_mutex
• rtapi_app is deadlocked in the halcmd request handler

there are two ways out of this:

1. send a signal to the RT thread if it does not exit right away, which should terminate the read()
2. do not use read() but rather poll() and do the poll on 2 file descriptors - the one we are interested in, and an eventfd which is just used for shutdown signaling - the shutdown sequence would write() to the eventfd before doing the pthread_join, having the poll() return

I'll explore in turn and see how I fare - I think 1. being less intrusive on API use

I already verified that closing the device file does not cancel the read :-/

update: in fact a pthread_cancel() might do as read() is on the list of cancellation points

[mhaberler]

well luckily it seems the pthread_cancel() does the job of terminating the read() (src/rtapi/rt-preempt.c):

@dkhughes - mind trying this patch and see if this gets your shutdown hang sorted?

@@ -197,14 +197,21 @@ void _rtapi_task_delete_hook(task_data *task, int task_id) {
     void *returncode;

     /* Signal thread termination and wait for the thread to exit. */
     if (!extra_task_data[task_id].deleted) {
    extra_task_data[task_id].deleted = 1;
+
+   err = pthread_cancel(extra_task_data[task_id].thread);
+   if (err)
+       rtapi_print_msg(RTAPI_MSG_ERR,
+               "pthread_cancel() on RT thread '%s': %d %s\n",
+               task->name, err, strerror(err));
    err = pthread_join(extra_task_data[task_id].thread, &returncode);
    if (err)
-       rtapi_print_msg
-       (RTAPI_MSG_ERR, "pthread_join() on realtime thread failed\n");
+       rtapi_print_msg(RTAPI_MSG_ERR,
+               "pthread_join() on RT thread '%s': %d %s\n",
+               task->name, err, strerror(err));
     }
     /* Free the thread stack. */
     free(extra_task_data[task_id].stackaddr);
     extra_task_data[task_id].stackaddr = NULL;
 }

the more I think of it - this patch is seriously needed: ANY thread (rt or posix) doing not just HAL but any form of blocking system calls will be subjected to this shutdown hang otherwise

[mhaberler]

on the socfpga, the above patch reliable removes the hang on exiting 'halrun -I irqtest.hal'

on an amd64, this triggers an obscure pthread_cancel() bug causing a segv in the terminating thread
google for '_Unwind_ForcedUnwind crash' or 'pthread_cancel segv' - tons of hits, no clear resolution

[mhaberler]

I think I'm on the trail to this one - very subtle

on 'unload ', the thread functs and pins of this comp are unlinked from the threads, then the comp is unloaded - the theory being thereafter comp code and data cannot be referenced anymore

in the case of a comp doing a blocking call, e.g. read(), the thread is blocked within this read even after the functs and pins are unlinked and the comp unloaded

a later delthread (implicit in shutdown) cancels the system call originating in a - by now unloaded - comp (really a shared library), meaning the code and data segment of this comp are invalid, causing the crash on return from the system call

I guess the resolution is - extend the rather obscure 'halcmd unload all' to delete all threads before any comp is unloaded - in the legacy code, threads were exported by the motion and threads components only and an unload of those implicitly deleted the threads

to round out my monologue ;) yes, the above worked and a patch is coming which covers the 'unload all' and 'halcmd shutdown' cases

it does NOT cover the case where a comp using blocking system calls is loaded, a thread has been started calling this funct, and the user removes the comp with 'unload comp' - this still results in an rtapi crash

the HAL data structures just do not support expressing this kind of referential integrity relation easily

I do see an alternative, more proper fix through shared library reference counting: if a shared libary (component) already was loaded with dlopen(), another dlopen() just increases the reference count on the shared library handle; dlclose() decrements the refcount and unmaps the shared library when the refcount drops to zero. I wonder if this is worth the trouble for now

[dkhughes]

@mhaberler Impressive detective work. So, to make sure I understand what is happening - the thread is stuck in the blocking function call, and the component is deleted out from underneath it?

[mhaberler]

yes, exactly - a scenario which cannot happen with the current nonblocking thread functs
PR coming soonish

[mhaberler]

@dkhughes - machinekit/machinekit#962 should make the exit hang go away

one can still crash rtapi by an explicit unloadrt hm2_soc_ol while running but maybe we'll find a fix for that downstream - for now 'a restriction' as it's easily fixed by preceding the unloadrt with 'delthread all' (that is pretty much what the patch does for 'unload all')

update: merged

[dkhughes]

@mhaberler I've used the patches for a day or so now and the exit hangs have disappeared, great work! I have been looking for side effects to the change but my tests haven't shown anything yet.

[mhaberler]

@dkhughes great to hear, thanks!

re the actual topic of this issue.. next stop is rebuilding uio_pdrv_genirq from source assuming the solution will be at that level

afterall Linus might have had a case ;)