Multiple instances of PythonEngine in same app

[josipjuric]

Is it possible to have multiple PythonEngine instances running at the same time? I want to add scripting functionality to my app where users would write scripts, which would run concurrently independent of each other?

Josip

[filmor]

This is not possible, mostly due to limitations in Python itself. Theoretically you should be able to instantiate nearly independent interpreters in the same program using Py_NewInterpreter, but there are quite some caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats

Feel free to look into this and extending PythonEngine (or rather, a new type) using this API.

[josipjuric]

Will take a look, thanks on response.

Edit:
As a quick solution I decided to overcome this by creating small wrapper .net app which will host PythonEngine. Main app will start that wrapper as regular process and they will communicate using some kind of IPC, probably named pipes.

[novecento]

@joco92 it sounds an interesting solution, may you please explain any details about it?

[josipjuric]

@joco92 it sounds an interesting solution, may you please explain any details about it?

@novecento sorry on late response.

Basically what I will do is create simple .net cli app which will host PythonEngine and on startup run script that is passed to it as CLI parameter.

It will communicate with main application using gRPC over TCP or NamedPipes.

This way main application address space will be completely isolated from PythonEngines which means that PythonEngine can't crew anything inside main app. If you take a look on your browser, you will see that for each tab it uses separate process, so this is same approach.

Also, by using gRPC you have its proto file as contract which describes which parts of main app are exposed to outside

[koubaa]

I think it makes more sense to implement this using subinterpreters. They will be part of CPython's stdlib in 3.10 but it is already part of the C API which means we can attempt it now.

[Jeff17Robbins]

@koubaa unfortunately, subinterpreters still share a single GIL, although work is underway to try to remove that restriction. See issue40512 for more details. Due to sharing a single GIL, you won't see good utilization of cores if you go this route, today.

The multiprocess route works, but there's overhead in the messaging interface. The approach @joco92 outlines works, and I wanted to point out that asyncio has efficient support for both TCP and named pipes in its event loop. It might be faster in some cases to use pickle as your marshaller (rather than protobuf), but you'd have to measure a specific use-case to see what was best for your application.

Another route is to do a custom build of C Python to create a second copy of the C Python DLL (under a different name) that shares nothing with the normal copy. The challenge here is you need to rebuild any C extensions you want in that second instance of Python. Again, depending on your application, this might be work out, and give you completely independent Python threads (that share nothing.) While somewhat ugly, it works, and gives you the efficiency of running threads in a single process rather than sharing memory or processes across processes.

[koubaa]

@Jeff17Robbins Another copy of the C Python DLL won't work on linux due to platform differences there. In general, symbols would be taken from the first loaded library that provides them (CPython.so or CPython_alt.so). Libraries will be loaded in the order they are listed in .dynamic section of executable - which I think would be managed by mono.

This means that anything in the CPython public C API must also be renamed in the CPython dll with separate pinvoke wrappers in pythonnet.

FWIW - I've been trying to contribute to the issue you mention in my spare time.