Entropy Question in PPO code

[btnorman]

Hey Guys,

I'm really enjoying Brax! I have a question about entropy loss in the PPO code.

I have been logging the entropy loss, and for some training runs it becomes positive... which really confuses me as it seems to suggests that the distributions entropy is negative!

I have looked through the Brax code and have not managed to work out an explanation. I wonder if I have missed some constant being added somewhere, or if there is some other obvious explanation. I feel I must be missing something. Has anyone else experienced this?

Many thanks,
Ben

[m-orsini]

Hi!
Awesome to hear you're having fun with Brax!
The entropy is calculated in training/distribution.py. FYI, The distribution we're using is a tanh of a normal distribution. For entropy calculation, we don't use an exact formula but an approximation.
I believe the entropy being negative is normal as the entropy of a normal distribution can be negative as far as I understand: The formula we use is:
.5 + .5ln(2pi) + ln(sigma)

If sigma becomes close to 0 - which can definitely happen as actions are less random after training - then the entropy becomes negative.

I hope I didn't mix myself up - please tell me if you think there is a mistake.

[btnorman]

Thanks for clearing that up. I was thinking of Shannon entropy, rather than differentiable entropy!