Gymnasium Integration #789

Markus28 · 2022-12-29T17:32:23Z

Changes:

Disclaimer in README
Replaced all occurences of Gym with Gymnasium
Removed code that is now dead since we no longer need to support the old step API
Updated type hints to only allow new step API
Increased required version of envpool to support Gymnasium
Increased required version of PettingZoo to support Gymnasium
Updated PettingZooEnv to only use the new step API, removed hack to also support old API
I had to add some # type: ignore comments, due to new type hinting in Gymnasium. I'm not that familiar with type hinting but I believe that the issue is on the Gymnasium side and we are looking into it.
Had to update MyTestEnv to support options kwarg
Skip NNI tests because they still use OpenAI Gym
Also allow PettingZooEnv in vector environment
Updated doc page about ReplayBuffer to also talk about terminated and truncated flags.

Still need to do:

Update the Jupyter notebooks in docs
Check the entire code base for more dead code (from compatibility stuff)
Check the reset functions of all environments/wrappers in code base to make sure they use the options kwarg
Someone might want to check test_env_finite.py
Is it okay to allow PettingZooEnv in vector environments? Might need to update docs?

… to only use new step API

… API) and fixed FiniteVectorEnv

codecov-commenter · 2023-01-05T09:04:23Z

Codecov Report

Merging #789 (363250d) into master (6019406) will increase coverage by 0.20%.
The diff coverage is 89.10%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master     #789      +/-   ##
==========================================
+ Coverage   90.89%   91.10%   +0.20%     
==========================================
  Files          73       73              
  Lines        5109     5079      -30     
==========================================
- Hits         4644     4627      -17     
+ Misses        465      452      -13

Flag	Coverage Δ
unittests	`91.10% <89.10%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tianshou/env/worker/subproc.py	`57.54% <60.00%> (+0.24%)`	⬆️
tianshou/env/venvs.py	`90.34% <80.95%> (-3.82%)`	⬇️
tianshou/env/utils.py	`81.25% <83.33%> (-2.09%)`	⬇️
tianshou/data/collector.py	`93.30% <100.00%> (+3.00%)`	⬆️
tianshou/env/gym_wrappers.py	`97.72% <100.00%> (ø)`
tianshou/env/pettingzoo_env.py	`90.56% <100.00%> (+2.43%)`	⬆️
tianshou/env/venv_wrappers.py	`81.81% <100.00%> (+1.49%)`	⬆️
tianshou/env/worker/base.py	`65.45% <100.00%> (ø)`
tianshou/env/worker/dummy.py	`85.29% <100.00%> (ø)`
tianshou/env/worker/ray.py	`83.33% <100.00%> (+10.11%)`	⬆️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Trinkle23897 · 2023-01-06T00:12:10Z

Hey @Markus28, I'm actually thinking about a non-breaking change. If a user uses gym env, tianshou's VectorEnv itself will first detect the class of the env, and if it's gym env then we adapt shimmy wrapper on top of that. This is an automated process and won't degrade user-experience.

The current changes looks fine for me but it would be nice to add the above feature request!

Also, have you added shimmy usage in docs or readme so that people can know how to convert it to a gymnasium env without seeking other resources for help?

Markus28 · 2023-01-07T09:38:54Z

I can do that :)
Currently, I only have a disclaimer in the README, but I can add a section to the docs

Markus28 · 2023-01-20T13:54:35Z

So it turns out that there is unfortunately no "universal" shimmy wrapper that will take any Gym environment and give us an up-to-date Gymnasium environment. Shimmy implements different wrappers for the different API versions. I think we have the following options:

Just let the user do the conversion
Use different Shimmy wrappers depending on gym.__version__. That could lead to problems for people who are using environments with APIs that don't match their gym version
Implement a universal wrapper ourselves

Mark seems to believe that doing a universal wrapper would be quite complicated, but I don't see why it shouldn't be doable. I'm not sure whether it would then be part of Shimmy or Tianshou though.

Trinkle23897 · 2023-01-20T16:49:58Z

I personally in favor of the second option. This will reduce most of (90%) the issue user hit and will reduce the maintenance effort quite a lot.

Markus28 · 2023-01-20T18:09:15Z

tianshou/env/venvs.py

+        gym_version = packaging.version.parse(old_gym.__version__)
+        if gym_version >= packaging.version.parse("0.26.0"):
+            return shimmy.GymV26CompatibilityV0(env=env)
+        elif gym_version >= packaging.version.parse("0.22.0"):
+            return shimmy.GymV22CompatibilityV0(env=env)
+        else:
+            raise Exception(
+                f"Found OpenAI Gym version {gym.__version__}. "
+                f"Tianshou only supports OpenAI Gym environments of version>=0.22.0"
+            )


@pseudo-rnd-thoughts Can you confirm that this is how the shimmy wrappers are intended to be used? I.e. for any environment 0.26.0>gym.__version__>=0.22.0 we use GymV22CompatibilityV0" and for gym.__version__ >= 0.26.0 we use GymV26CompatibilityV0?

Markus28 · 2023-01-20T18:10:20Z

I implemented the second version. I will still need to update some documentation and the tutorials :) There are currently no tests for OpenAI Gym environments, let me know if I should add some.

nuance1979 · 2023-01-22T22:38:55Z

I implemented the second version. I will still need to update some documentation and the tutorials :) There are currently no tests for OpenAI Gym environments, let me know if I should add some.

Yes, I think we definitely need a test which imports gym and triggers the use of shimmy. I'd suggest making a copy of test_dqn.py to test_dqn_gym.py then only replace import gymnasium as gym with import gym.

setup.py

nuance1979 · 2023-01-25T16:36:22Z

@Trinkle23897 This PR is failing the doc spelling check but the error message is quite cryptic:

../../../../../../opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/tianshou/env/gym_wrappers.py:docstring of tianshou.env.gym_wrappers.ContinuousToDiscrete.action:4: : Spell check: Args: Args:.

I have no idea how to fix it. Can you take a look?

Other than that, I think this PR is ready to merge.

Trinkle23897 · 2023-01-25T16:46:47Z

will take a look tonight

Trinkle23897 · 2023-01-26T17:42:35Z

hmm on my local:

>>> gym.__version__, shimmy.__version__, gymnasium.__version__
('0.25.0', '0.2.0', '0.27.1')

$ python test_dqn_gym.py
Traceback (most recent call last):
  File "/Users/jiayi/github/tianshou/test/discrete/test_dqn_gym.py", line 172, in <module> 
    test_dqn(get_args())
  File "/Users/jiayi/github/tianshou/test/discrete/test_dqn_gym.py", line 64, in test_dqn
    train_envs = DummyVectorEnv(
  File "/Users/jiayi/github/tianshou/tianshou/env/venvs.py", line 428, in __init__
    super().__init__(env_fns, DummyEnvWorker, **kwargs)
  File "/Users/jiayi/github/tianshou/tianshou/env/venvs.py", line 148, in __init__
    self.workers = [worker_fn(_patch_env_generator(fn)) for fn in env_fns]
  File "/Users/jiayi/github/tianshou/tianshou/env/venvs.py", line 148, in <listcomp>
    self.workers = [worker_fn(_patch_env_generator(fn)) for fn in env_fns]
  File "/Users/jiayi/github/tianshou/tianshou/env/worker/dummy.py", line 13, in __init__
    self.env = env_fn()
  File "/Users/jiayi/github/tianshou/tianshou/env/venvs.py", line 77, in patched
    return shimmy.GymV22CompatibilityV0(env=env)
  File "/Users/jiayi/.virtualenvs/openai/lib/python3.9/site-packages/shimmy/openai_gym_compatibility.py", line 199, in __init__
    gym_env = _strip_default_wrappers(gym_env)
  File "/Users/jiayi/.virtualenvs/openai/lib/python3.9/site-packages/shimmy/openai_gym_compatibility.py", line 280, in _strip_default_wrappers
    gym.wrappers.render_collection.RenderCollection,
AttributeError: module 'gym.wrappers' has no attribute 'render_collection'

pseudo-rnd-thoughts · 2023-01-26T19:57:16Z

You need to update your gym version, but we will update shimmy to avoid this issue

Trinkle23897

leave render_collection as-is, will educate user to upgrade the gym version as temporary solution

Changes: - Disclaimer in README - Replaced all occurences of Gym with Gymnasium - Removed code that is now dead since we no longer need to support the old step API - Updated type hints to only allow new step API - Increased required version of envpool to support Gymnasium - Increased required version of PettingZoo to support Gymnasium - Updated `PettingZooEnv` to only use the new step API, removed hack to also support old API - I had to add some `# type: ignore` comments, due to new type hinting in Gymnasium. I'm not that familiar with type hinting but I believe that the issue is on the Gymnasium side and we are looking into it. - Had to update `MyTestEnv` to support `options` kwarg - Skip NNI tests because they still use OpenAI Gym - Also allow `PettingZooEnv` in vector environment - Updated doc page about ReplayBuffer to also talk about terminated and truncated flags. Still need to do: - Update the Jupyter notebooks in docs - Check the entire code base for more dead code (from compatibility stuff) - Check the reset functions of all environments/wrappers in code base to make sure they use the `options` kwarg - Someone might want to check test_env_finite.py - Is it okay to allow `PettingZooEnv` in vector environments? Might need to update docs?

Markus28 added 15 commits December 29, 2022 18:31

Replaced Gym with Gymnasium

10cc606

Updated docs

4e827c1

Added type ignores for gym wrappers

eaeb4cc

Changed envpool from gym to gymnasium

125069a

Require envpool>=0.7.0

201f0cd

Fixed MyTestEnv, removed __init__ that is unnecessary due to Gymnasium

e1d0c0e

Removed dead code that was necessary for old step API

5f0833a

Removed type hints about old step API

bea1559

Removed check for whether environments return info upon reset

3fc8c78

Increase required version of PZ, fix some CI issues

9f2f104

Added dummy info to reset of FiniteVectorEnv

1ea1b90

Fix log method to take terminated and truncated, update PettingZooEnv…

9c9a70b

… to only use new step API

Made some code more explicit (removed hack for compatibility with old…

28e5cb1

… API) and fixed FiniteVectorEnv

Fix FiniteVectorEnv

6be6f41

Fixed some type hints

803f543

Markus28 mentioned this pull request Jan 4, 2023

Transition to Gymnasium / Future Compatibility with Tianshou microsoft/nni#5308

Open

Try to fix FiniteVectorEnv

54e922f

Markus28 added 4 commits January 5, 2023 10:04

Skip NNI test, remove commented out code

d9a1feb

Fixed type errors

54515f7

Disclaimer in README

bfca5a9

Put type ignore in the right places

bc379b0

Also allow OpenAI gym environments, fixed documentation

266c64a

Markus28 commented Jan 20, 2023

View reviewed changes

Also allow PettingZooEnv in vector environment, fixed type hint

565c2e9

Markus28 added 3 commits January 21, 2023 08:47

Fixed import of PettingZooEnv

1402f1e

Fixed type hinting, updated README

2a5ff31

Updated documentation about ReplayBuffer

040d2af

nuance1979 reviewed Jan 22, 2023

View reviewed changes

setup.py Show resolved Hide resolved

Markus Krimmel added 2 commits January 23, 2023 13:06

Fixed gymnasium version, added shimmy to dev requirements

28475f4

Added test for conversion of OpenAI Gym environments

2be8591

araffin mentioned this pull request Jan 23, 2023

Add Gym 0.26 support DLR-RM/stable-baselines3#780

Closed

19 tasks

nuance1979 requested a review from Trinkle23897 January 25, 2023 16:37

fix spelling

654f9d8

pseudo-rnd-thoughts mentioned this pull request Jan 26, 2023

Check when importing gym wrappers Farama-Foundation/Shimmy#32

Closed

Trinkle23897 mentioned this pull request Jan 27, 2023

_setup_buf conflicting Dict types when using Gymnasium with FetchReach environment #799

Closed

8 tasks

Trinkle23897 marked this pull request as ready for review February 3, 2023 19:20

Merge branch 'master' into gymnasium_integration

363250d

Trinkle23897 approved these changes Feb 3, 2023

View reviewed changes

Trinkle23897 merged commit 6c6c872 into thu-ml:master Feb 3, 2023

nuance1979 mentioned this pull request Feb 28, 2023

removed gym as a dev dependency #813

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gymnasium Integration #789

Gymnasium Integration #789

Markus28 commented Dec 29, 2022 •

edited

Loading

codecov-commenter commented Jan 5, 2023 •

edited

Loading

Trinkle23897 commented Jan 6, 2023 •

edited

Loading

Markus28 commented Jan 7, 2023

Markus28 commented Jan 20, 2023

Trinkle23897 commented Jan 20, 2023

Markus28 Jan 20, 2023

Markus28 commented Jan 20, 2023 •

edited

Loading

nuance1979 commented Jan 22, 2023

nuance1979 commented Jan 25, 2023

Trinkle23897 commented Jan 25, 2023

Trinkle23897 commented Jan 26, 2023 •

edited

Loading

pseudo-rnd-thoughts commented Jan 26, 2023

Trinkle23897 left a comment

Gymnasium Integration #789

Gymnasium Integration #789

Conversation

Markus28 commented Dec 29, 2022 • edited Loading

codecov-commenter commented Jan 5, 2023 • edited Loading

Codecov Report

Trinkle23897 commented Jan 6, 2023 • edited Loading

Markus28 commented Jan 7, 2023

Markus28 commented Jan 20, 2023

Trinkle23897 commented Jan 20, 2023

Markus28 Jan 20, 2023

Choose a reason for hiding this comment

Markus28 commented Jan 20, 2023 • edited Loading

nuance1979 commented Jan 22, 2023

nuance1979 commented Jan 25, 2023

Trinkle23897 commented Jan 25, 2023

Trinkle23897 commented Jan 26, 2023 • edited Loading

pseudo-rnd-thoughts commented Jan 26, 2023

Trinkle23897 left a comment

Choose a reason for hiding this comment

Markus28 commented Dec 29, 2022 •

edited

Loading

codecov-commenter commented Jan 5, 2023 •

edited

Loading

Trinkle23897 commented Jan 6, 2023 •

edited

Loading

Markus28 commented Jan 20, 2023 •

edited

Loading

Trinkle23897 commented Jan 26, 2023 •

edited

Loading