Ability to send a message from Python back to Elixir

[davydog187]

We have a few interesting use cases for Pythonx at TV Labs, and one of the features that would make it particularly useful, is if long-running Python programs could be executed, with the ability to send messages back to the owning process (or any process).

There is precedent for a feature like this, since NIFs themselves can send message through the enif_send interface.

@cocoa-xu is on vacation this week or I would have asked her, but I imagine this is really only possible if we are able to pull the BEAM header files into Python, or find a way to expose a function in Python that can lift the message up to the BEAM layer.

Thoughts?

[josevalim]

Python can easily call C and this library ships with C code, so one option would be to define a C function alongside our C code and then call it from Python. However, in order to do that, you would probably need to store the environment somewhere (if we don't already do that).

[josevalim]

And yes, we would need to expose this function somewhere. Perhaps this lib could ship with a .py file in priv, I am not sure of the best approach.

[davydog187]

Right, after looking at the implementation for a moment, there is a python prelude already in the NIF where the function could be exposed

https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L388

Following the example of pythonx_handle_io_write, we could expose a function that is quite similar that does the enif_send but off of user input https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L1399

I think I could throw a prototype together here

[jonatanklosko]

Yeah, I think this could be done very similar to IO write.

Perhaps this lib could ship with a .py file in priv, I am not sure of the best approach.

We could patch the default globals to include an extra function that we would define, or rather a pythonx python module with such function, imported by default. We would do this as part of initialization, similarly to how we define custom stdout/stdin objects. The only issue is that this makes the Python code specific to Pythonx, but maybe it's fine for specific use cases.

Note that with IO you can already send messages wherever you want by specifying custom :stdout_device, although it only works with text. I assume you want to send a Python object as the message. One thing to note though is that once you get that Python object, any operations on that object will requires the GIL. If the long-running Python code doesn't hold GIL for much, then it should be fine, but if it does, it can result in bottlenecks.

[davydog187]

Note that with IO you can already send messages wherever you want by specifying custom :stdout_device, although it only works with text.

That wasn't clear to me but is obvious in retrospect

I assume you want to send a Python object as the message

Correct, although I think there are two modes that I might care about

1. References to the Python objects that I can manipulate directly
2. Copies of the Python objects encoded to Elixir terms that I never will manipulate

My motivations here are more for eventing that I think is better suited to 2

[josevalim]

The issue with 1 is precisely that you would trigger the gil so you would have to be very careful about it. Perhaps one option is to allow only binaries to be sent...

[jonatanklosko]

For 2 we also need GIL, we don't really want to decode within the NIF itself, because that has limitations. In other words instead of 2. we would actually do 1. and you call Pythonx.decode/1, so it's the same.

[josevalim]

What about 3, only send binaries which we already encode as binaries before sending them?

[davydog187]

@josevalim isn't that equivalent to setting the :stdout_device, or do you mean something else?

[josevalim]

Yes, it would be equivalent.

[josevalim]

Well, one slight difference, you would be to send it to different PIDs if we allowed PIDs to be encoded to Python.

[davydog187]

Right. I may try to find some time to prototype something here, will report back

[josevalim]

But to be clear, if IO works for you, then go for it. That’s how you would communicate through a port anyway.

[lalabuy948]

Hi everyone, we potentially have a nice usecase for this library at CyanView, I'm working on PoC at the moment.
@davydog187 did you have a chance to make prototype for exchanging events between python and elixir?

[josevalim]

Do you have an example of what you are trying to do? It is worth saying that you can emulate this by calling the Python code, having it stop once you have the intermedia result, yielding back to Elixir, and then you call it back passing the previous state/objects. This is probably even preferable to letting Python run in a dirty scheduler for really long periods of time.