Can we step a Gym environment faster than real time?

[stephane-caron]

I'm training a policy and having the simulation run at 1x the real time is a waste of computing time. Is it possible to step the Gym environment faster than real-time?

[stephane-caron]

Environments are CPU-based by choice (so that they run the same in sim and on the robot), but for training it is possible to step the simulation faster than real-time, provided the CPU is powerful enough for that. There are two sides to it:

1. Start the Bullet spine in "stepping" mode: ./tools/bazelisk run -c opt //spines:bullet -- --nb-substeps 5 --spine-frequency 1000.
   • Here the spine runs at 1000 Hz, i.e. with 1 ms time steps.
   • Every action it receives will forward the simulation by 5 steps == 5 ms.
2. Disable loop frequency regulation in the environment: gym.make("UpkieWheelsEnv-v3", frequency=None)

These two steps are illustrated in ppo_balancer/train.py.