[Feature Request] Multithreaded env

[vmoens]

Motivation

We have a version of multiprocessed envs, but not multithreaded.
It would be nice to explore how fast these can be, and/or if that would be a viable solution for executing a list of environments simultaneously.

[Benjamin-eecs]

I don't think written in python is going to achieve that goal, you can give a look at EnvPool, maybe we can consider envpool integration for accelerated env backend.

[vmoens]

Thanks @Benjamin-eecs
Sure having a look won't hurt!
The issue is that we must use the spawn mp setting, which is much slower than fork (bc it launches independent python session on each process, rather than forking the current one -> each import needs to happen etc).
I was wondering if by using multithreading we could speed this up.
Does it clarify the issue?
Happy to look at EnvPool anyhow!

[Benjamin-eecs]

I was wondering if by using multithreading we could speed this up.

I think because of Python Global Interpreter Lock (GIL), we cannot achieve multi-thread written in Python (only one thread can be running at a time), maybe that's why there's only subproc vecenv written in python on the RL lib market. But maybe it can be faster than spawn mp setting.

[vmoens]

I think because of Python Global Interpreter Lock (GIL), we cannot achieve multi-thread written in Python (only one thread can be running at a time), maybe that's why there's only subproc vecenv written in python on the RL lib market. But maybe it can be faster than spawn mp setting.

Hence my "exploration" claim :)
Let's try to integrate EnvPool then!

[Benjamin-eecs]

Sure, 'd love to do that 😄

[vmoens]

Hey @Benjamin-eecs
We're not able to work on this atm because our team is working with M1 mcbooks, which I believe is not an arch that is supported by EnvPool. So this will be a pending issue for a little while, unless you want to take care of it yourself and/or make EnvPool compatible with M1 :)