defer signal handling to a singleton thread

[wjwwood]

Fixes #604

[nuclearsandwich]

@ros-pull-request-builder retest this please

[nuclearsandwich]

The Cpr CI failures look like they might actually useful. The errors relate to contents of the PR and dependency packages are all available.

[nuclearsandwich]

Optimistically started CI builds

• Linux
• Linux-aarch64
• macOS
• Windows

[mjcarroll]

Looks like MacOS may have failed due to network issues, then failed again on test_get_node_names: https://ci.ros2.org/job/ci_osx/4837/testReport/

[hidmic]

Another network failure. I don't see any errors though.

[nuclearsandwich]

curl'd the build log locally and grep'd through it and it does appear that every test which ran passed so I'm suggesting we treat that mac build as passing.

[dirk-thomas]

it does appear that every test which ran passed

The summary line explicitly says: 24522 tests, 0 errors, 0 failures, 50 skipped 👍

[Karsten1987]

If this patch makes CI happy, then this looks good to me. I believe I don't have too much context to see to review as deeply as I wanted to, but I had a few minor comments generally to it.

[Karsten1987]

while (is_installed()) ?

[wjwwood]

I don't think this would work because the location of the if (!is_installed()) { is important, in that it comes after the loop over the context's. So I'd pass on changing this logic unless @hidmic thinks it's a good idea to do so.

[hidmic]

As @wjwwood, location of the statement is important. Changing it may cause the process to lock forever if signal handlers are uninstalled as a result of context shutdown. Keeping both may prevent a signal to be handled if it comes while the handler is about to be uninstalled.

[Karsten1987]

that's fine for me. I was just wondering if we could somehow change the while(true), but I guess in this case it's all right.

[sloretz]

I think @hidmic mentioned this before, calling notify_one() from a signal handler is undefined behavior. Is there anything else that can be used to wake the thread?

[wjwwood]

I asked him about this, but I don't remember his answer. Do you know of an alternative? Only thing I can think of is a busy wait in the signal handler loop and an atomic, but that doesn't sound very nice.

[sloretz]

Searching only turns up things that are platform specific.

sem_post() for POSIX platforms http://pubs.opengroup.org/onlinepubs/009695399/functions/sem_post.html

I'm not sure about windows. I can't tell from the documentation if windows semaphores are signal safe, https://docs.microsoft.com/en-us/windows/desktop/sync/using-semaphore-objects

[wjwwood]

Looks like notfiy_one() in libcxx uses pthread:

https://github.com/llvm-mirror/libcxx/blob/35a0c2ce5e18ce57d8f867e1f57deffc8428d76b/include/__threading_support#L274-L278

Which I gather isn't safe enough? Is that your conclusion as well @sloretz?

On Windows it uses WakeConditionVariable():

https://github.com/llvm-mirror/libcxx/blob/b52e084ca81da9f2d03ab21a8625aeada7e64a67/src/support/win32/thread_win32.cpp#L95-L99

We're not using libcxx on Windows, but I imagine the std::condition_variable is also implemented with that function in MSVC.

---

What do you (or others) think we should do?

[hidmic]

@sloretz Yeah, that's the thing with signal handlers 😅

AFAIK there's no completely safe cross-platform synchronization mechanism that can be used from a signal handler (if only Windows was POSIX compliant...). Busy waits, with or without a timeout, are a portable option. Not a very neat one though. We can also do some more research on how to do this right for each platform.

I'd expect the current implementation to work in almost all cases though. Unless a signal comes right when the thread is about to do something with the condition variable. and that only takes place during a negligible fraction of the process lifetime.

[sloretz]

Which I gather isn't safe enough? Is that your conclusion as well @sloretz?

It uses pthread_cond_signal, which this page says

It is not safe to use the pthread_cond_signal() function in a signal handler that is invoked 
asynchronously. Even if it were safe, there would still be a race between the test of the Boolean 
pthread_cond_wait() that could not be efficiently eliminated. 

[wjwwood]

Ok, so I was wrong, this is not safe for pthread, see:

It is not safe to use the pthread_cond_signal() function in a signal handler that is invoked asynchronously. Even if it were safe, there would still be a race between the test of the Boolean pthread_cond_wait() that could not be efficiently eliminated.

Mutexes and condition variables are thus not suitable for releasing a waiting thread by signaling from code running in a signal handler.
-- https://linux.die.net/man/3/pthread_cond_signal

[wjwwood]

I addressed some of the feedback, still need to address the locking of the install/uninstall functions.

Also the safety of notifying from a signal.

new changes

[wjwwood]

Ok, please re-review this. I'm working on a solution for the unsafe notification issue, but I think we should merge it as-is under the idea that's it is an incremental improvement (we were doing much more than using cond var's in the signal handler before).

Can someone do CI and stuff for this while I work on the improved signal notification solution?

We can do that in a follow up pr, since it should be API and ABI neutral.

[sloretz]

CI (testing packages above rclcpp)

• Linux
• Linux-aarch64
• macOS
• Windows

[Karsten1987]

lgtm with green CI. Just to make sure, the flaky test is addressed with the CI configuration?

[sloretz]

I think this is a step in the right direction, though it still doesn't solve ros2/build_farmer#152. I can reproduce the same timed-out test with this PR using

colcon test --retest-until-fail 100 --event-handlers console_direct+ --packages-select test_communication --ctest-args -R test_action_client_server__Fibonacci__rclcpp__rmw_fastrtps_cpp\$

Edit: oops ros2/build_farmer#152 (comment)

[hidmic]

It used to happen roughly once every 10 times or so for me, so I agree here.

[wjwwood]

So is this good to merge then @sloretz? @hidmic have you had a chance to try and reproduce it too with this branch? @sloretz and I were not able to do so, despite the fact that it appeared this way at first (it was failing, not hanging, for a different reason).