Performance on Pre-Activation ResNet

[S-Abdelnabi]

Hello,

Thank you so much for providing your code!

I had a question about pre-activation ResNet. I am trying to implement your attack on my problem using pre-act resNet18 and CIFAR-10, but the transferability is lower than the plain vanilla attack (or at most similar). The source is a pre-act resNet18 and also the target is a pre-act resNet18. The only thing that I did different from your code is to adjust the ReLU condition (since pre-act resnet doesn't have a ReLU before the residual blocks, so I removed the "not '0.relu' in name" condition).

I would very much appreciate it if you could let me know if you have tried this architecture, or if you have some insights about what could be going wrong (e.g. the source and target models are the same or the source model is not deep enough). Please let me know if you need more info about the architecture.

Thank you very much.

[arnav-gudibande]

Hi @S-Abdelnabi I also have the same question about how to change the backward hook to support Pre-Act ResNets. Do you or @csdongxian have any updates on this?

[csdongxian]

Sincerely sorry for the late reply. Since I was busy graduating and switched the email, I missed the issue.

I still remember that I applied SGM to ResNet-v2 (PreAct ResNet in slim of TensorFlow) and it worked. For adaptation to PreAct ResNet, there are two points:

(1) Are there modules belonging to ReLU? Since the implementation in SGM first finds some modules with "relu" in their name. However, some implementation (e.g., this repo uses torch.nn.functional.relu directly, and we cannot find any module called "relu".

(2) The code in Line 33 in utils_sgm.py is to avoid the gradient vanishing, and it should be conducted in the trunk path (i.e., all gradients flow through the same path). For example, we could register the hook for BasicBlock / Bottleneck in ResNet or PreActBlock / PreActBottleneck module in PreAct ResNet. Unfortunately, the original code only supports ResNet we used. When we apply other architectures, it is better to modify the code.