VecNormalize vs Scaling

[lhorus]

From my understanding, VecNormalize scales the rewards or observations in real-time, that is, according to the running observances, i.e., mean and stddev get updated in real time. However, for the typical RL envs presented here, one actually knows the reward and observation ranges, so why not use scaling based on that ? (e.g. scikit's preprocessing framework)

[pzhokhov]

Hi @lhorus ! While the reward and observation ranges are known ahead of time, the actual achieved level of reward can vary quite a bit throughout training. Consider the humanoid runner, where the reward is the distance that the humanoid managed to run before falling - in the beginning, the rewards will be small (the humanoid cannot run and falls) and in this regime VecNormalize amplifies small reward signal. When the model trains to run / walk successfully, the rewards can be rather large - and here VecNormalize will imitate behaviour of scaling by max reward range.
That being said, VecNormalize is not always necessary (and has its own drawbacks such as serialization difficulty). By default we only apply VecNormalize to mujoco environments; and there are several environment wrappers that allow for constant reward and observation scaling (ScaledFloatFrame, RewardScaler)