Add Gym 0.26 support

[araffin]

Description

See DLR-RM/stable-baselines3#780 for installation or usage with Gymnasium

List of things (probably incomplete) that need to be done. Add if you see others.

Note: I had to patch gym for pybullet and the TimeLimit wrapper

• Update step methods - The Env.step function returns 5 values instead of 4 previously (observations, reward, termination, truncation, info).
• Update reset - The Env.reset function returns two values (obs and info)
• Update seed - Env.seed no longer exists, instead, use gym.reset(seed=...)
• Update rendering - render arguments can be part of the environment's constructor i.e., gym.make("CartPole-v1", render_mode="human")
• Probably fix pybullet env (Fix registration of gym envs bulletphysics/bullet3#4332 and compatibility with Gym 0.26.0 bulletphysics/bullet3#4335)
• Fix timelimit wrapper ([Bug Report] TimeLimit wrapper logic has changed openai/gym#3102)
• Fix highway env (AttributeError: 'numpy.random._generator.Generator' object has no attribute 'randn' Farama-Foundation/HighwayEnv#379)
• Fix highway env render (Render mode not implemented for parking-v0 Farama-Foundation/HighwayEnv#400)
• Upgrade Panda gym pre-trained models
• Add step compat argument
• Fixes for gym/gymnasium transition

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[qgallouedec]

List of things (probably incomplete) that need to be done. Add if you see others.

• Update step methods - The Env.step function returns 5 values instead of 4 previously (observations, reward, termination, truncation, info).
• Update reset - The Env.reset function returns two values (obs and info)
• Update seed - Env.seed no longer exists, instead, use gym.reset(seed=...)
• Update rendering - render arguments can be part of the environment's constructor i.e., gym.make("CartPole-v1", render_mode="human")

EDIT: Outdated, fill the checklist in the description instead

[araffin]

I added the TODOs to the description ;)

[araffin]

@qgallouedec the API change is only for the wrapper, for the rest, VecEnv should work as before (except the render mode I think, that was already a bit hacky anyway with pybullet)

[qgallouedec]

a7bdc5e: panda-gym fixed in v3.0.1