Evaluate the representation robustness is still a training process?

[zoeleeee]

I test the command under Evaluate the representation robustness, it is still a training process, instead of the statement "load a pre-trained model and test the mutual information between input and representation." So this cannot be a evaluation measure for any pre-trained model, right? or is there something I understand or use in a wrong way?

[schzhu]

Hi zoeleeee, thanks for your interest! It's indeed an optimization-based evaluation so it looks like another training process.

Since mutual information is difficult to estimate directly, by referring to MINE (https://arxiv.org/pdf/1801.04062.pdf) we represent the mutual information (KL-divergence) in terms of its Donsker-Varadhan representation. Yet for such representation to hold we need to find an optimal measurable function given two distributions, and the optimization process is just to find such function. You can refer to MINE (https://arxiv.org/pdf/1801.04062.pdf) and this note (https://web.stanford.edu/class/stats311/lecture-notes.pdf) for more information about the Donsker-Varadhan representation of KL-divergence.