pos_out > neg_out

[GloryyrolG]

Hi SW @swyoon ,

As far as I know, the expectation of EBM training at least should be pos_out <= neg_out, right? However, I found in the exp of CelebA 64, we end up early where pos_out > neg_out.

[swyoon]

Dear @GloryyrolG ,

Thank you for your observation.

You are right about pos_energy <= neg_energy in general energy-based models. However, in NAE, we regularize (neg_energy ** 2) in order to prevent a model from diverging. I think this regularization is responsible for a slightly smaller neg_energy. From my experience, it is usually fine as long as pos_energy stays not too far from neg_energy.

Please let me know if you observe any notable consequences.

Best,
Sangwoong.