[question] Using PythonNet within TPL dataflow executing threads / .net multithreading pools #2170
Unanswered
vadimkantorov
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
There exists a dotnet TPL library which allows to configure and execute data pipelines and data flows:
For migration to Python I am interested in making PythonNet / CPython instance available to data flow threads. In pipeline components I'd like to execute pure Python functions not modifying the global state if that matters
I wonder if it's possible to somehow use cPython
multiprocessing
module and assign cPython threads to TPL threads, so that later on I could within a TPL thread to wake my cPython thread and ask it to do some work?What is the correct way of proceeding taking into account GIL / multithreading model of PythonNet?
Do I need to somehow initialize a separate PythonNet instance in every thread from TPL threadpool?
Also, a general question on how should I initialize PythonNet, as it seems that PythonEngine.Initialize() call is static/global (https://pythonnet.github.io/pythonnet/dotnet.html) and there's no explicit object that holds the interpreter state (?). Is this global state thread-safe as long as I create a Py.Scope per thread? Can I ensure that every thread has its own interpreter state / its own GIL?
According to https://stackoverflow.com/questions/47978171/how-to-do-multiprocessing-using-python-for-net-on-windows, it seems that having an independent GIL/Python instance per .NET worker thread is not possible. Is it true?
Maybe this question is another instance of Multiple instances of PythonEngine in same app #1110
So I found that https://peps.python.org/pep-0684/ on per-interpreter GIL was recently accepted for Python 3.12 except https://docs.python.org/3.12/whatsnew/3.12.html does not mention PIP 684 it all. I hope that sub-interpreter work advances soon and then this could become possible to support sub-interpreter per .net worker thread in PythonNet. There are also questions on how numerical libraries like NumPy/PyTorch would survive the sub-interpreters https://discuss.python.org/t/pep-684-a-per-interpreter-gil/19583/2
Beta Was this translation helpful? Give feedback.
All reactions