Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model overfits with low test accuracy for higher epsilon values #4

Closed
chrissmiller opened this issue Mar 20, 2020 · 1 comment
Closed

Comments

@chrissmiller
Copy link

chrissmiller commented Mar 20, 2020

I'm using the FGSM approach to train a ResNet18 model on CIFAR10.

Using the values in the paper for epsilon=8/255 and alpha=10/255 works fine. But when I try to extend to an epsilon of 12 (and an alpha of 1.25*epsilon as outlined in the paper, so 15) to compare to other robust models, the model catastrophically overfits relatively early with very low clean example accuracy (50 to 60%). Has anyone had success using this approach with a higher epsilon than 8/255? Does alpha=1.25*epsilon not apply for other values of epsilon?

Thanks in advance for any help you can provide.

@chrissmiller chrissmiller changed the title Convergence for higher epsilon values Model overfits with low test accuracy for higher epsilon values Mar 20, 2020
@leslierice1
Copy link
Collaborator

Hey, thanks for your question. If you are experiencing catastrophic overfitting when using a higher epsilon, you can lower your step size until you no longer overfit. I ran our code for your particular example, epsilon=12/255, and found that with alpha=13/255 (rather than 15/255), the model does not catastrophically overfit, and gets 47% PGD accuracy, and 74% clean accuracy. Let me know if you have any further questions on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants