You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The PGD (and BIM) implementation in this repository is significantly less effective than as reported in prior work. In Table XIV PGD (or BIM) appears to succeed 82.4% (or 75.6%) of the time. When I run the code in the repository, I get a very similar result: 82.5% (or 74.2%).
This should be somewhat surprising given that prior work reports PGD and BIM succeeds nearly 100% of the time with the same distortion bound of 0.3. See for example Figure 4 of Madry et al. (2018), or Table IV of Carlini & Wagner (2017). Indeed, when I put a loop around my FGSM call (using the approach discussed in #3) I reach 100% attack success rate with both BIM and PGD.
I have not investigated the cause of this discrepancy further.
It is deeply concerning that now I have checked five results (FGSM/PGD/BIM/JSMA/PAT) and all of them have issues (#3 / this issue / #14 / #4). Did you cross-check the results of your attacks with any other libraries?
The text was updated successfully, but these errors were encountered:
The PGD (and BIM) implementation in this repository is significantly less effective than as reported in prior work. In Table XIV PGD (or BIM) appears to succeed 82.4% (or 75.6%) of the time. When I run the code in the repository, I get a very similar result: 82.5% (or 74.2%).
This should be somewhat surprising given that prior work reports PGD and BIM succeeds nearly 100% of the time with the same distortion bound of 0.3. See for example Figure 4 of Madry et al. (2018), or Table IV of Carlini & Wagner (2017). Indeed, when I put a loop around my FGSM call (using the approach discussed in #3) I reach 100% attack success rate with both BIM and PGD.
I have not investigated the cause of this discrepancy further.
It is deeply concerning that now I have checked five results (FGSM/PGD/BIM/JSMA/PAT) and all of them have issues (#3 / this issue / #14 / #4). Did you cross-check the results of your attacks with any other libraries?
The text was updated successfully, but these errors were encountered: