Release the Python interpreter lock while waiting for data #39
Conversation
As long as we keep the global interpreter lock (GIL) active, no other Python thread can make progress. This lead to starvation when there are multiple Python operators on the same runtime node. This PR fixes this by holding the GIL only as long as needed, i.e. when calling code of the operator. Most importantly, we don't hold the GIL anymore when the operator is idle and waiting for new inputs.
|
Note that Python operators still share the same global interpreter lock, they just try to hold that lock as short as possible now. Ideally, we would use a separate Python runtime for each operator, but unfortunately this isn't supported in pyo3 yet (see PyO3/pyo3#879 and PyO3/pyo3#576). |
phil-opp
changed the title
|
So this fix the issue, I was previously facing, but, after testing on |
|
Ok, thanks for trying! How about we merge this PR for now (since it at least improves things) and look into ways to make this faster later? Right now, I can only think of one solution/workaround for the GIL problem: spawn multiple processes. We could achieve that by changing our dataflow YAML to run each Python operator on its own runtime node. |
As long as we keep the global interpreter lock (GIL) active, no other Python thread can make progress. This lead to starvation when there are multiple Python operators on the same runtime node. This PR fixes this by holding the GIL only as long as needed, i.e. when calling code of the operator. Most importantly, we don't hold the GIL anymore when the operator is idle and waiting for new inputs.
Fixes #38