Sub-Interpreter Support: Development Discussion

[Aequitosh]

All relevant discussion regarding the design and development of sub-interpreter support should preferably go on here (until we find a better place, but I think this will do).

I'll post my thoughts and findings here from time to time, and if you're reading this, you're more than welcome to pitch in!

Edit: The tracking issue can be found here: PyO3#3451

[letalboy]

Hello @Aequitosh, I accepted to enter the fork of PyO3, and I'm doing some planing here of what files to change first, as we talk in the main thread on David account it will be nice to add the lacking ffi support, I'm already doing part of the C API extension in the rust module to the impl that we are proposing on the enhancement of adding sub compilers, I think until the end of the week I have something promising to we test, also feel free to share your ideas and opinions about the topic too, let's keep talking!

[Aequitosh]

Sounds good! I'll start drafting up some stuff as well in the coming days. Keep me updated!

[letalboy]

Nice! Any news I'll let you know ;)

[letalboy]

I will add the things that we discuss with David on the other issue, what you think is a good practice, do directly on main since it's a fork or create a branch that we can cut when we finish?

[letalboy]

I'm still exploring the crate and the Python C-API (there is so much more than I anticipated)

Here, too, is too many things on this, if you keep going deep you find that a thing that at first look simple have a lot related to kkk

For example, what happens if the caller does end up switching sub-interpreters between PyGILState_Ensure() and PyGILState_Release() calls? Even if we, let's say, implemented an alternative to PyCell, this would still cause unsound behavior inside PyO3, because we're still relying on the GIL.

Yeah, this makes sense, since we have the limitations impose by GIL, looking from this perspective, it is much more complicate to go around it than it is to develop some alternative way to do what we are planing to do..

A thing that I was thinking here that can help us, is that if we find a lot of limitations to run multiple Python threads we can manage to create multiple instances that theoretically each one will have their independent link like when we are running various python scripts in multiple terminals, this will theoretically give an independent controller to each session like if we are running each one in parallel in the level of the system, and then manage to sync this in the rust side, removing the complexity of working with almost non-predictable states that emerges from this link between the sub compilers, like if we create a sub-controler that each track a independent sub compiler and them control this predictable sub-controlers with some sort of head controller, like a pool or something like this, idk just insights...

but can we know for sure that we're tracking it correctly?

I agree, since it isn't directly sync and that we have to keep checking we can't guarantee 100% safety in this, unfortunate the only way to guarantee that we have it working is running one in each process and only to tasks that doesn't take to much time, since it can also drop at some cases and unexpectedly kill the process, at least in some tests that I have done.

I don't have much knowledge about the idea of no GIL, what is the planing? Can you give me some links besides the topic PyO3#2885 to I study about this? And by the way tks for the ref to subcompilers code, I was looking for this other side of the API

[Aequitosh]

What you mentioned is similar to the idea I have currently in my head: I want to find a mechanism with which we can tell the Python interpreter to synchronize with PyO3 instead, but I don't know if that's actually possible yet.

In other words, I want to find a way for the Python interpreter to acquire a lock on PyO3's side, instead of PyO3 acquiring a lock (the GIL) on Python's side, if that makes sense.

This probably won't work for data that's stored on Python's heap (so, most things, unfortunately) - in order to acquire ownership of data on Python's heap, we must acquire the GIL. One idea that I have is to add some "layer of indirection" - let me explain below:

Acquiring Memory Ownership

Example situation:

There is some data (to be precise, a Python object) on the Python interpreter's heap.

There are also two "actors" that want to safely access that data:

1. The Python interpreter
2. A Python module implemented via PyO3

Current Method

1. Acquire GIL
2. Do something with the Python object (as we now have exclusive access to it)
3. Drop the GIL

That's how both the Python interpreter and PyO3 currently do it.

Alternative Method

... also known as my random idea.

If the Python object stems from some other module (that doesn't use PyO3), the Python interpreter would still access the object like above:

1. Acquire GIL
2. Do something with the Python object
3. Drop the GIL

However, what I want to achieve (for objects coming from PyO3), is this:

1. Acquire GIL (the Python interpreter will always do that right now, no matter what)
2. Acquire "PyO3 lock" for Python object
3. Do something with the Python object
4. Drop the "PyO3 lock"
5. Drop the GIL

Accessing our own objects would then be as follows:

1. Acquire "PyO3 lock"
2. Do something with the Python object
3. Drop the "PyO3 lock"

That way the interpreter would need to synchronize with PyO3 instead of the other way around (PyO3 relying on the GIL).

I am really not sure how we could achieve this, though. Essentially, we would need:

1. A way to manage our own memory
   • Maybe just use Box<T> everywhere for now?
2. A threadsafe locking mechanism
   • Maybe just use an Arc<Mutex<T>> for the time being? But I'm not sure.
3. A way to enforce the locking mechanism when the Python interpreter accesses data owned by PyO3

Points 1 and 2 would probably be very trivial, as we could just use the things the Rust standard library already provides. Whether we do that on a per-PyObject level or something else is up to us, but I would prefer to keep it simple for now.

Point 3 is where it gets hairy: I have no idea how to tell Python that we got our own lock (yet). That's what I will look into in the coming days.

[Aequitosh]

I don't have much knowledge about the idea of no GIL, what is the planing? Can you give me some links besides the topic PyO3#2885 to I study about this?

Well, there are a couple places you could check out:

• The Python Wiki has a section dedicated to removing the GIL
• PEP 703 which proposes to make the GIL optional

That's more on Python's side, though. I'm afraid that if we don't want to rely on the GIL, we need to come up with our own solution.

[letalboy]

Yeah, sounds good! And also may work! Here is a diagram that visually represents what we are drafting:

graph TD

    MemoryOwnership --> DataOnPythonHeap
    DataOnPythonHeap --> CurrentMethod
    DataOnPythonHeap --> AlternativeMethod1
    DataOnPythonHeap --> AlternativeMethod2

    subgraph "Current Method"
    CurrentMethod --> AcquireGIL1
    AcquireGIL1 --> AccessPythonObject1
    AccessPythonObject1 --> DropGIL1
    end

    subgraph "Alternative Method (Non-PyO3)"
    AlternativeMethod1 --> AcquireGIL2
    AcquireGIL2 --> AcquirePyO3Lock1
    AcquirePyO3Lock1 --> AccessPythonObject2
    AccessPythonObject2 --> DropPyO3Lock1
    DropPyO3Lock1 --> DropGIL2
    end

    subgraph "Alternative Method (PyO3)"
    AlternativeMethod2 --> AcquirePyO3Lock2
    AcquirePyO3Lock2 --> AccessPythonObject3
    AccessPythonObject3 --> DropPyO3Lock2
    end

Loading

To start implementing this it is a good idea to start looking at the bridges between Rust and CPython and try to find where the most essential connections that our changes will impact are made in the PyO3 ffi module, and then we can start implementing some of these ideas, I also think that special the lock idea on the PyO3 side is a good idea since it will reduce the unknown states of the other side as I mentioned here:

removing the complexity of working with almost non-predictable states that emerges from this link between the sub compilers

In one way or another the max things that we can stop relying on in check states outside rust the best, since rust is memory safe if we do the things correctly it will be much more robust and much more reliable, so I think our start point is mapping the API links used at moment that our changes will impact on and see what we can start modding in it to reduce the dependence in external states, and then start implementing some of these ideas and see which one is the best for implementing and then also to maintain in the future that also is a very important thing to ensure. If the mods will be easy for future devs to maintain and if it is well documented

[letalboy]

I'm afraid that if we don't want to rely on the GIL, we need to come up with our own solution.

Relax, I think we can figure it out, it may sound complicated at first, but almost everything I have done so far looks like this in the start, as we are planing and drafting the solution will appear in the horizon, and we will do a great job!

Well, there are a couple places you could check out:

The Python Wiki has a section dedicated to removing the GIL
PEP 703 which proposes to make the GIL optional

By the way, tks for the refs, I will read and try to elaborate something, I'm also searching something's here too, if I find something that may help I send to you!

[letalboy]

I will start a new branch of our discussion here just to brainstorm a potential thing if the nogil becomes very, very hard to implement . and discuss if it makes any sense:

This idea was sneaking around in my mind since the start, however, I was not sure if it can work or not, however this line here:

(unlike threads within a process)

From this paragraph:

The GIL is a CPython implementation detail that limits multithreaded parallelism, so it might seem unintuitive to think of it as a usability issue. However, library authors frequently care a great deal about performance and will design APIs that support working around the GIL. These workaround frequently lead to APIs that are more difficult to use. Consequently, users of these APIs may experience the GIL as a usability issue and not just a performance issue.

For example, PyTorch exposes a multiprocessing-based API called DataLoader for building data input pipelines. It uses fork() on Linux because it is generally faster and uses less memory than spawn(), but this leads to additional challenges for users: creating a DataLoader after accessing a GPU can lead to confusing CUDA errors. Accessing GPUs within a DataLoader worker quickly leads to out-of-memory errors because processes do not share CUDA contexts (unlike threads within a process).

Of this PEP 703 that you mentioned makes me think about this idea again, basically, at least at the point that I know we can only have one GIL for session since this will lock the global interpreter and we will only be able to use it in a unique thread wright? What if we build a mechanism that create multiple sessions in Rust side and then we control each one from a head controler? This way each one will have their own GIL. It will looks like python default subprocess since the def of it says:

Python processes are whole instances of the Python interpreter that run independently. Each Python process has its own GIL and its own copy of the data to be worked on. That means that multiple Python processes can run in parallel on separate hardware cores.

So this will allow us to use most part of the built structure in a way that we only need to build a mechanism to parallelize it in multiple sessions then share (in the rust side) a secure state of each one, this will looks something like the following:

graph TD

subgraph Main[Main]
    style Main stroke:#2FFFFF, fill:#00FFFF ,fill-opacity:0.1
    XX[Head]
end

subgraph Controler[Controler]
    style Controler stroke:#a14FFF, fill:#a14FFF ,fill-opacity:0.1

    subgraph Process1[Process1]
        style Process1 stroke:#a14FFF, fill:#a14FFF ,fill-opacity:0.1

        A[Lock GIL]
        B[GIL]
        C[Aquire Python]
        D[Use Python]
        E[Drop Python]
        F[Drop Drop GIL]

        XX --> A

        A --> B
        B --> |GIL Lock| C
        C --> |Python| D
        B --> |GIL Lock| F
        D --> E
        C --> |Python| E
        E --> F

    end

 
    subgraph Process2[Process2]
        style Process2 stroke:#a14FFF, fill:#a14FFF ,fill-opacity:0.1
        N[Lock GIL]
        O[GIL]
        P[Aquire Python]
        Q[Use Python]
        R[Drop Python]
        S[Drop Drop GIL]

        XX --> N

        N --> O
        O --> |GIL Lock| P
        P --> |Python| Q
        P --> |Python| R
        Q --> R
        N --> |GIL Lock| S
        R --> S

    end

    subgraph Process3[Process3]
        style Process3 stroke:#a14FFF, fill:#a14FFF ,fill-opacity:0.1

        T[Lock GIL]
        U[GIL]
        V[Aquire Python]
        W[Use Python]
        X[Drop Python]
        Y[Drop Drop GIL]

        XX --> T

        T --> U
        U --> |GIL Lock| V
        V --> |Python| W
        W --> X
        V --> |Python| X
        X --> Y
        V --> |GIL Lock| Y

    end

end

Loading

I was trying to do something like this in my crate RustPyNet that I built to help the use of PyO3 in a batch sequence queue from multiple threads executing python task in various treads in an orchestrated way, this crate is in my public profile and in a branch named: rayon_parallelized_subprocess_experiment have theatrical code structure of a parallelized session based python tasks process distribution, I have a feeling that it may have some potential

[Aequitosh]

Hey there! Sorry for the absence; I was on holiday and university started again, too. I'm available more often again now.

That idea (and also the things you posted in the other thread) look fantastic! I will definitely explore this more.

Also, something worth noting: There's now a GIL per sub-interpreter since Python 3.12 which we could also leverage. We have to keep in mind though that if we build our solution on top of those new sub-interpreters, it will only work - well - for that kind of sub-interpreter. Still worth checking out, en my opinion!

Either way, I will look into what you posted above. This seems very promising.

[Aequitosh]

Ah, the link above was already posted in PyO3#3479, my bad. Still, something worth considering.

[Aequitosh]

So, I gave the thread you opened some consideration - I think what you proposed above could actually be developed as a separate extension / crate instead, building on top of PyO3. I really like it though, so after we're done with this, we could work on that together, maybe!

Either way, I'm going to continue my research. All we need is some synchronization mechanism, the question is which.

[letalboy]

Hey there! Apologies for the absence; I was on holiday and university has resumed as well.

No worries, I'm a bit busy here too. Given the substantial changes, it will take some time. The delay is somewhat beneficial as we now have numerous new elements in Python 3.12 to explore, providing alternative mechanisms to achieve our goals, at least on the Python side...

That idea (and also the things you posted in the other thread) looks fantastic! I will definitely explore this more.

I'm thrilled that you like it! Synchronizing all of that isn’t an easy task, but I've observed performance gains in my tests, so it's at least worth the effort, haha.

Also, something worth noting: There's now a GIL per sub-interpreter since Python 3.12 which we could also leverage. We have to keep in mind though that if we build our solution on top of those new sub-interpreters, it will only work - well - for that kind of sub-interpreter. Still worth checking out, in my opinion!

Absolutely! I agree. This release introduces a lot of exciting new things. I think your job will be significantly easier with these new features! We'll also have other possibilities and new mechanisms to explore. I tend to think that this release and new features have come at the perfect moment!

Either way, I will look into what you posted above. This seems very promising.

I think so too, especially regarding backward compatibility. However, the PyO3 team doesn't want to add a higher level of complexity to the crate, and this requires a bit more than what is currently implemented since it involves session syncs and a lot of shared data that need the extra complexity. They suggested continuing the development for users who need it as an extension module, not a core feature. But if we can simplify it, I think they might adopt the idea. Also, we can ensure true multithreading, genuinely processing things at once, which for many applications is a huge deal.

So, I gave the thread you opened some consideration - I think what you proposed above could actually be developed as a separate extension / crate instead, building on top of PyO3. I really like it though, so after we're done with this, we could work on that together, maybe!

Thank you for considering it! Let's dive in. I believe we can construct an FFI (Foreign Function Interface) integration while ensuring compatibility with PyO3, thereby establishing our own mechanism. This approach allows users, especially those requiring enhanced performance, to integrate it into their libraries developed with both PyO3 and CPython with relative ease.

For instance, I'm already utilizing a similar strategy in my network-based IPC (Inter-Process Communication) networks, akin to Apache mod_wsgi, to accommodate multiple clients while preserving necessary performance levels. Specifically, by employing a mechanism that efficiently handles client requests and manages server resources, I've observed a noticeable improvement in response times and resource utilization in high-traffic scenarios.

Undoubtedly, we can collaborate on this and deliver a potent tool that bridges Rust and Python, ensuring efficiency, reliability, and optimal hardware utilization. This will be particularly advantageous for AI training, where DataLoaders encounter a leakage issue due to multithreading, leading to memory overload. For example, during the training of deep learning models, I've noticed that the memory usage spikes significantly, impacting the overall training time and resource management.

Moreover, it’s also beneficial in fields like biochemistry, where I’ve heard there are challenges using Python for extensive molecular calculations. For instance, in simulations involving large molecular structures, the computational demand can be quite intensive, and optimizing these processes could significantly enhance the speed and efficiency of such calculations.

This solution has the potential to impact numerous areas positively, such as data analytics, scientific computing, and more, by providing a robust and efficient method for handling complex computations and managing resources. It will undoubtedly be a substantial contribution. After implementing the changes here, I’m in agreement to progress further, exploring additional functionalities and optimizations to further enhance its utility and performance in various applications.

This will undoubtedly be a valuable contribution to the community!

Either way, I'm going to continue my research. All we need is some synchronization mechanism, the question is which.

The problem with sync, as you mentioned earlier in this topic, is that we can't really confirm the status of the remote sub-interpreter. Maybe now we can in Python > 3.12, but it needs research. The only way is by checking every time, and if the link drops, we reestablish it again, but this will exchange safety for performance. So, we have to lay everything out on the table, make a list of the pros and cons, and ask the PyO3 team to help make a decision. The more brains on this, the better to have multiple sides of this problem covered and make the best decision. 😄

[Aequitosh]

FYI: I've decided to publish my development environment, so you can follow up on whatever I'm currently up to. I'll include a way to automatically use development builds of PyO3 in the near future to make everything a little easier for myself.

[letalboy]

Hello @Aequitosh how are you? Hope you are doing well! I see some activity in the other threads, how is going the research that you were doing in the PyCell subject? Any news related to that or something else? I'm waiting some feedback on that, so I can help in that development, pls share here or in other place the work being done, if you want I can help organize the project session in a way that becomes easy to people that want to contrib to this being able to do so, given the amount of work that we will need to do I think that's might be the way. I'm waiting a reply, say what you need to elaborate that idea, let's organize it ant put into practice into a fork with good tests to we evaluate the stability of the subinterpreters and the overall crate in this new conditions!

[Aequitosh]

Hi there! Sorry for not getting back to you; I'm doing pretty alright, and you?

I haven't got much news other than the things I had posted in that thread - I'm still working on and off on sorting out static data first. The whole PyCell thing is too much to tackle at once I feel, I wasn't really making any tangible progress in that regard. I also feel like once the static data stuff is solved, working on the synchronization primitives becomes a little easier.

So, in terms of work / research that's actually been done, I've mostly tried to implement what this example has been doing here into PyO3. I'll push my branch once I've gotten a module to work without bugs (and cleaned up my code). In the meantime, you can look into things regarding PyModule_GetState as well if you want!

At the moment it's a bit hard to really split up any work yet, as the things I'm currently working on can't really be split up, I feel? Though it would be nice if you could take a look at my branch once I got it ready! I'll also definitely need help with writing tests and such.

So, I'll let you know once I need a hand ;) Though I'm not stopping you if you wanna look into things yourself, you don't have to wait for me 👀

[letalboy]

Hi there! Sorry for not getting back to you; I'm doing pretty alright, and you?
I'm good too tks God! Nice to know that you are doing alright!

I haven't got much news other than the things I had posted in that thread
That's nice, looks like you are progressing in it, this is great news 🚀

I'm still working on and off on sorting out static data first. The whole PyCell thing is too much to tackle at once I feel, I wasn't really Making any tangible progress in that regard. I also feel like once the static data stuff is solved, working on the synchronization primitives becomes a little easier.

Yeah, when I take a look in this topic once you mentioned it couple months ago, I immediately see the amount of work needed to be done around it, I am with your decision, I think is the best way to go!

So, in terms of work / research that's actually been done, I've mostly tried to implement what this example has been doing here into PyO3. I'll push my branch once I've gotten a module to work without bugs (and cleaned up my code). In the meantime, you can look into things regarding PyModule_GetState as well if you want!

At the moment it's a bit hard to really split up any work yet, as the things I'm currently working on can't really be split up, I feel? Though it would be nice if you could take a look at my branch once I got it ready! I'll also definitely need help with writing tests and such.

I understand, this kind of thing I prefer to do solo too, when we try to split things that are complex, and we have an idea is hard to even try to explain it, some tings don't are worth to split, they are worth to be done the fast as we can do so then we can split other things that are more suitable for it. I already started looking in it at some time ago when I was looking the FFI module and the CPython things, I will resume my research in relation to that and try to see what we can do about that in my free time here, but I will stay waiting for the things that you are doing in your branch to see how we can integrate all. At the moment I don't have a clear image of the overall so I'm working to understand how to connect it all, I hope that once you publish the things that you are doing it will give a north of where we need to go!

So, I'll let you know once I need a hand ;) Though I'm not stopping you if you wanna look into things yourself, you don't have to wait for me 👀

Haha okay, but I will wait anyway, if I start here too then we will have two versions of same thing, I am studying some advanced rust concepts and doing some work related to multi threading send and sync structs and custom data wrappers, we will need something like that to sync all things anyway later, also to change the crate code it's better to do in a solid way when we go do that! So I will continue to work in the rust part of the sync mechanism while you work in the static part so then we have some models to try to build the PyCell thing and have a clear representation to create a way to send and sync py objects in a more safe way, I don't know if we will be able to make it completely safe any time soon, but we can try to improve it at least, by adding some safe mechanism on top of the API to CPython and the data shared with it, any improvment in it will make work with multiple sub interpreters much more easy.

[Aequitosh]

I'm back! See: PyO3#4162

Looking at the git diffs should hopefully help you get started - after cleaning everything up, the changes I made don't look that scary anymore 😄

[letalboy]

Nice man, that very nice! I will see this for sure, hope that this changes allow us to be a step closer to the sub-interpreters!