Release Python GIL (Global interpreter lock) in Subscription::take_message

[achim-k]

Feature request

Feature description

I am currently looking into performance bottlenecks of rosbridge_server which uses rclpy to dynamically subscribe to ROS topics (typically with raw=True) that are requested by rosbridge clients. The set of subscribed topics can be quite diverse: High frequency topics with rather small message sizes (e.g. tf) and low frequency topics with large message size (e.g. camera images, point clouds).

After doing some profiling, I noticed that a major part of the process' CPU time is spent on Subscription::take_message. I attempted to parallelize this by using a MultiThreadedExecutor and one callback_group per subscriber, but I soon realized that Python's GIL, effectively prevents the C++ code from being executed in parallel.

My question / feature request is the following: Could we release the GIL from extension code in order to improve concurrency when having multiple subscribers? Would there be potential disadvantages in doing so?

Implementation considerations

In achim-k@826ac88 I gave releasing the GIL a try by using Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, which resulted in a significant performance increase of rosbridge_server. I initially tested it without the mutex, but it seemed to be a bad idea since most rcl_* functions are marked as not thread-safe.