Skip to content

feat: release GIL for most "run_block_on" calls#388

Merged
digikata merged 1 commit intofluvio-community:mainfrom
dariogoetz:release-gil-on-block
Mar 11, 2024
Merged

feat: release GIL for most "run_block_on" calls#388
digikata merged 1 commit intofluvio-community:mainfrom
dariogoetz:release-gil-on-block

Conversation

@dariogoetz
Copy link
Contributor

This PR wraps most run_block_on calls in a py.allow_threads function, releasing the Python GIL for the duration of the run_block_on call. This is especially important for PartitionConsumerStream.next which might block indefinitely.

The issue arose with me, when running a PartitionConsumer in a separate thread and using the stream method. The underlying call to PartitionConsumerStream.next was waiting for incoming messages while holding the Python GIL, effectively prohibiting the main thread to do any work.

I think that releasing the GIL is mostly important for the PartitionConsumerStream.next method, because it may block indefinitely (if no messages arrive), but I wrapped all other run_block_on calls as well, while I was at it.

I left out the instances of run_block_on in the context of Cloud and CloudAuth as I have not worked with those, yet.

@digikata
Copy link
Contributor

Thanks for the PR @dariogoetz, we appreciate you taking the time to submit it. We'll queue this up to review and test it right away!

@digikata
Copy link
Contributor

digikata commented Mar 11, 2024

Looks good to me, added an issue to expand further to other areas
#390

Thanks again @dariogoetz

@digikata digikata merged commit b0a8652 into fluvio-community:main Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants