[Feature Request] Turn off auto-reset

[jbuckman]

Motivation

The documentation says auto-reset is enabled by default, but for certain applications it is better to have it turned off.

Solution

When auto-reset is off, terminated environments should not be run. They should continue to return dummy state/action/reward/etc in syncronous mode, and they should never be included in the batch in async mode.

Checklist

• I have checked that there is no similar issue in the repo (required)

[Trinkle23897]

but for certain applications

Is that your own customized environment?

When auto-reset is off, terminated environments should not be run.

Actually it has already supported this feature to some extend: see https://github.com/sail-sg/envpool/blob/master/envpool/atari/atari_pretrain_test.py
This script first creates 100 environments to run 100 episodes. When an environment terminates, it will not step anymore. The idea is to use env_id to control the step. So the stepping num_envs will gradually decrease from 100 to 1.

BTW, you can also call reset(action, env_id) to continue the running for a set of envs.