Question: normalize_reward not subtracting mean

[ChenDRAG]

stable-baselines3/stable_baselines3/common/vec_env/vec_normalize.py

Line 177 in 237223f

def normalize_reward(self, reward: np.ndarray) -> np.ndarray:

    def normalize_reward(self, reward: np.ndarray) -> np.ndarray:
        """
        Normalize rewards using this VecNormalize's rewards statistics.
        Calling this method does not update statistics.
        """
        if self.norm_reward:
            reward = np.clip(reward / np.sqrt(self.ret_rms.var + self.epsilon), -self.clip_reward, self.clip_reward)
        return reward

I wonder why we do not subtract mean in normalize reward? I have done some experiment and it indeed shows that "subtract mean" will reduce performance, but chould you tell me why exactly. Are there researches or explanations validing this thought?

[araffin]

Hello,
you will find answers there:

• Some questions regarding VecNormalize hill-a/stable-baselines#698
• Gamma in VecNormalize for rms updates. openai/baselines#538
• VecNormalize vs Scaling openai/baselines#629
• [Question] DDPG return normalization hill-a/stable-baselines#234