[Bug]: Kwargs in record_video.py not preserved by SB3 Vector Environment wrappers

[anayebi]

🐛 Bug

The render_mode in record_video.py is not preserved after create_test_env is called here:

rl-baselines3-zoo/rl_zoo3/record_video.py

Lines 96 to 105 in 2d5db11

	env = create_test_env(
	env_name.gym_id,
	n_envs=n_envs,
	stats_path=maybe_stats_path,
	seed=seed,
	log_dir=None,
	should_render=not args.no_render,
	hyperparams=hyperparams,
	env_kwargs=env_kwargs,
	)

.

I have traced it to exactly this line in utils.py, where the VecNormalize wrapper of Stable-Baselines3 seems to set it back to None:

rl-baselines3-zoo/rl_zoo3/utils.py

Line 284 in 2d5db11

env = VecNormalize.load(path_, env)

.

My temporary solution was to set env.render_mode = 'rgb_array' back in record_video.py right after create_test_env:

rl-baselines3-zoo/rl_zoo3/record_video.py

Line 106 in 2d5db11

, but I think a more general solution should be developed to ensure kwargs are preserved by environment wrappers.

To Reproduce

python3 -m rl_zoo3.record_video --algo ppo --n-timesteps 1000 --env Swimmer-v3 --load-best

Relevant log output / Error message

Loading /om/weka/yanglab/anayebi/zfa/logs/ppo/Swimmer-v3_3/best_model.zip
Traceback (most recent call last):
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/rl_zoo3/record_video.py", line 141, in <module>
    env = VecVideoRecorder(
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 50, in __init__
    assert self.env.render_mode == "rgb_array", f"The render_mode must be 'rgb_array', not {self.env.render_mode}"
AssertionError: The render_mode must be 'rgb_array', not None
Exception ignored in: <function VecVideoRecorder.__del__ at 0x2b570cf051f0>
Traceback (most recent call last):
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in __del__
    self.close_video_recorder()
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 103, in close_video_recorder
    if self.recording:
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 392, in __getattr__
    return self.getattr_recursive(name)
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive
    attr = self.venv.getattr_recursive(name)
  File "/om2/user/anayebi/miniconda/envs/pong_env/lib/python3.9/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 417, in getattr_recursive
    attr = getattr(self.venv, name)
AttributeError: 'DummyVecEnv' object has no attribute 'recording'

System Info

Library was installed via pip.

({'OS': 'Linux-3.10.0-1062.el7.x86_64-x86_64-with-glibc2.17 # 1 SMP Wed Aug 7 18:08:02 UTC 2019',
  'Python': '3.9.7',
  'Stable-Baselines3': '2.1.0',
  'PyTorch': '2.0.0+cu117',
  'GPU Enabled': 'True',
  'Numpy': '1.21.2',
  'Cloudpickle': '2.0.0',
  'Gymnasium': '0.28.1',
  'OpenAI Gym': '0.26.2'},
 '- OS: Linux-3.10.0-1062.el7.x86_64-x86_64-with-glibc2.17 # 1 SMP Wed Aug 7 18:08:02 UTC 2019\n- Python: 3.9.7\n- Stable-Baselines3: 2.1.0\n- PyTorch: 2.0.0+cu117\n- GPU Enabled: True\n- Numpy: 1.21.2\n- Cloudpickle: 2.0.0\n- Gymnasium: 0.28.1\n- OpenAI Gym: 0.26.2\n')

Checklist

• I have checked that there is no similar issue in the repo
• I have read the SB3 documentation
• I have read the RL Zoo documentation
• I have provided a minimal and working example to reproduce the bug
• I've used the markdown code blocks for both code and stack traces.

[araffin]

Hello,

I guess your issue is a duplicate of #379 (see dicussion in DLR-RM/stable-baselines3#1525)

I thought the bug was fixed but it is still there apparently...

and yes env = VecNormalize.load(path_, env) is the issue, when the VecNormalize is unpickled, some attributes are not set properly (self.class_attributes doesn't contain render_mode).

A fix would be to do self.render_mode = self.venv.render_mode L166 of the VecNormalize (right after self.class_attributes = dict(inspect.getmembers(self.__class__)))

[araffin]

I've pushed a fix here: https://github.com/DLR-RM/stable-baselines3/pull/1671/files

[anayebi]

Great, thanks so much! Hopefully it gets merged to master soon :)

[araffin]

Released on master branch =)