Using own generator

[sidney1505]

Hi,

I trained my own generator model and tried to use it for creating counterfactuals for CelebA. However, I encounter some things that I do not understand:

1. The FID score of the downloaded generator is around 60, which is quite high - if I train it myself I can reach around 10.
2. Still, creating counterfactuals with that self-trained generator is not that easy probably because I do not set the hyperparameters for ACE correctly. Is there an intuition how to set them / what can cause such big differences despite good performance of the generator?

[guillaumejs2403]

Hi @sidney1505,

1. Is the downloaded generator the one I used, available here? My generator is not the best available. I merely used it for the constrained denoising. In fact, I never check for the FID of the generations from pure noise (which is different from the FID of the counterfactual), so I would not recommend it for generation from scratch. Is your own generator a DDPM?
2. I have two recommendations. (i) Check the $\tau$ such that the classification does not flip when applying the filtering process for most images, e.i. $C(x) =C(F(x))$. (ii) After you find $tau$, fix one distance loss, e.g. $ell_2$, and do some grid search multiples of 10 (1, 0.1, 0.01, 0.001). Try changing $\tau$ as well, but not that much. I recommend finding the best qualitative counterfactuals for a few instances (about 10-20, I would say), but you must ensure that they flip the classifier with a FR of at least ~90%.

If it is unclear, let me know :)

[sidney1505]

Hi,

and thanks for the fast answer!

1. I phrased it a bit wrong I guess, but it is highly impressive that your method works that well even when using a generator that is not tuned to the fullest, which is very benefitial if you want to apply it to low data domains.

2. Do you mean "not flip" or that it does flip? If it is really "not flip" I do not fully understand why I would want them to not flip, because I want them to flip when creating counterfactuals and by putting a higher L1 or L2 loss on them they will rather flip even less or am i wrong?

[guillaumejs2403]

We do NOT want them to flip when we apply the filtering process. Recall that the filtering adds some noise and denoise it with the DDPM. Note that the filtering does not include the attack process (when the CE is actually created). Hence, we want this because we want to preserve the image after the filtering process. If the DDPM produces random structures in the instance, the analysis of the CE will be harder. That is because we will not know if the flip is due to the random new structure or other variables. This is only to search a first $\tau$, you will still need to search your own $\tau$ :P.
Finally, yes: by putting a higher $\ell_p$, the FR will decrease when creating the explanation.