tmmd not maximizing the test power

[ylfzr]

Hi,
Thanks for sharing. In the tmmd_model.py file, the optimizer seems just trys to minimize the ratio-loss, which should be the approximated test power mentioned in the paper. Shouldn't we increase the test power? I can't find any "-" there. Do I miss anything?

[djsutherland]

Hi @ylfzr! The generator indeed minimizes ratio_loss, i.e. the generator tries to minimize the distinguishability of the two distributions. The generator tries to change the model to make the two distributions look the same. (Note that kernel_optim only updates g_vars.) The discriminator would be the one trying to find a kernel to make the two distributions look different, but model_tmmd uses a fixed kernel.

We couldn't get kernel optimization to work for this paper (as mentioned on the bottom of page 7, and in footnote 7 there). It works if you add some kind of regularization, e.g. in MMD GAN: Towards Deeper Understanding of Moment Matching Network or our sequel paper Demystifying MMD GANs. Our recent followup On gradient regularizers for MMD GANs tries to explain a little more why this regularization is needed and derives alternatives that seem to work better. Overall, with hindsight it seems like the power-proxy maximization is not really important for generative models (though it can give you much more powerful tests, if you're actually testing).

[ylfzr]

Get it.
Thank you very much.