Add documentation for Vectorized environments

[tristandeleu]

• Add documentation for vectorized environments (in Sphinx style).
  • Includes documentation to get started with gym.vector.make, AsyncVectorEnv, and SyncVectorEnv
  • Includes documentation about intermediate (shared memory in AsyncVectorEnv and exception handling) and advanced (custom spaces) usages of vectorized environments.
  • Includes multiple example snippets.
• Minor fixes in the docstrings of gym.vector.make, VectorEnv, AsyncVectorEnv, and SyncVectorEnv for automatic generation of the API reference.

[vwxyzjn]

This looks good to me! Could you add another section on recording the episodic returns with the vector env using the following code?

import gym

envs = gym.vector.make("CartPole-v1", num_envs=3)
envs = gym.wrappers.RecordEpisodeStatistics(envs)
envs.reset()
for i in range(100):
    _, _, _, infos = envs.step(envs.action_space.sample())
    for info in infos:
        if "episode" in info.keys():
            print(f"i, episode_reward={info['episode']['r']}")
            break

[vwxyzjn]

Hmm. What happens when the origin env's action space is MultiDiscrete, do you return a tuple of MultiDiscrete?

[tristandeleu]

The rules for batching all the standard Gym spaces are available here. For example a MultiDiscrete space becomes a Box with an appropriate shape. Do you think it is worth mentioning these conversion rules in the documentation?

[vwxyzjn]

Thank you!! This is super helpful. I'd suggest highlighting this section with a heading like

Working with the spaces of a sub-environment

[jkterry1]

Hey, I really appreciate all the time you put into this. However, per #2276 this isn't compatible with how the documentation website is going to be done. The way to include these docs would be to add elements of this to the api.md docs where applicable, and then to add the remainder to the dev_docs.md page once we have that (or just create it know and the person working on it can add the rest to that file).

[tristandeleu]

However, per #2276 this isn't compatible with how the documentation website is going to be done. The way to include these docs would be to add elements of this to the api.md docs where applicable, and then to add the remainder to the dev_docs.md page once we have that (or just create it know and the person working on it can add the rest to that file).

Is the documentation going to be just a collection of Markdown documents? How do you plan on having cross-references and auto-generated documentation from docstrings?

[tristandeleu]

Could you add another section on recording the episodic returns with the vector env using the following code?

I think a page on wrappers for vectorized environments (VectorEnvWrapper and Wrapper compatible with VectorEnv) would be very useful indeed, thanks for the suggestion!

[jkterry1]

There's a way of hooking the code into the MD files. When a PR for the new Gym website comes in a few weeks you'll see. I'll leave this PR open until then.

[tristandeleu]

Any progress on the documentation of vectorized environments? What is the status of the documentation?

[nikhilweee]

I was reading up on Vectorised Environments and I happened to stumble upon this PR. Any chance I can view the HTML version of the docs right now? Is there a beta version of the new documentation website that's mentioned in the README?

[tristandeleu]

@nikhilweee There is a HTML version of this documentation available here: https://tristandeleu.github.io/gym/vector/

[nikhilweee]

Thank you very much, just what I was looking for!