loss metric in search.py is off, augmentation policies for CIFAR10/SVHN seem random.

[sean-metzger]

Hello,

I have been trying FAA in my application.

I noticed that in search.py, line 116, you're basically taking the minimum loss over all the losses for each image.

Since your loss function defined in line 95 has no reduction, losses ends up being a vector (num_policies*batch_size,). Therefore you just get the minimum loss for a single image as your minimum loss for your metrics. Hence, if you are truly minimizing loss (as it says you do in the paper) using this code, then you're basically just getting mostly random noise as your reward_attr since there will probably always be at least one very good prediction giving low loss.

This may help explain why your CIFAR10 and SVHN policies are basically random. (I am using the policies from archive.py here:

Each augmentation appears roughly the same number of times, with almost uniform distributions of strength and probability of each. What's plotted is the normalized probability of each augmentation ((number of times it appears/total augmentations) * mean probability of the augmentation) on the y axis, vs average strength of each augmentation on the x axis. The same graph is true for SVHN.

On imagenet, you can see the distribution does seem a little bit less random - perhaps this is because loss will be a little more meaningful since there are 1000 classes so the minimum loss will be a slightly less noisy reward signal

By contrast, the augmentations from AutoAug:

This makes more sense to me: there should be some terrible operations that don't get used much, and some that are valuable and are used more...the fact they are roughly equal for cifar 10 is surprising.

So question: did you use top_1_valid to get the policies in archive? Or minus_loss? If it's the latter, was the code that is published being used?

Thanks!
Sean