-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CarliniLInfMethod differs significantly from the original, resulting in lower attack success rate #1374
Comments
Hi @kaitokishi Thank you very much for raising this issue! I would be very interested to take a closer look. Would you be able to share your code used for the comparisons? That way I could start with the exactly same experiments. |
Thank you for your reply. I refactor my code and the result is slightly changed but the main result is not changed. My code consists of two files: 1. Make a simple neural network to classify MNISTUse requirements-original.txt
Run train_model.pyimport numpy as np
import keras
from keras import layers
from tensorflow.keras.datasets import mnist
# Load a dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train[:,:,:,np.newaxis] / 255. - 0.5 # x_train.shape is (60000, 28, 28, 1)
x_test = x_test[:,:,:,np.newaxis] / 255. - 0.5 # x_test.shape is (10000, 28, 28, 1)
# Make a model
model = keras.Sequential()
model.add(layers.Flatten())
model.add(layers.Dense(units=512, activation='relu', input_dim=28*28))
model.add(layers.Dense(units=512, activation='relu'))
model.add(layers.Dense(units=10))
loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(loss=loss, optimizer=keras.optimizers.SGD(), metrics=['accuracy'])
model.fit(x_train, y_train, batch_size=128, epochs=3)
model.save('mnist_model.h5') 2. Generate adversarial images by the original implementationDownload python attack.py original to attack the model by Carlini's original implementation. attack.pyimport sys
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.keras.datasets import mnist
def get_correct_predicted_indices(model):
indices = []
for i in range(len(x_test)):
if y_test[i] == np.argmax(model.predict(np.array([x_test[i]]))):
indices.append(i)
if len(indices) == 10:
break
print(f'Selected indices: {indices}')
return indices
# Load a dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train[:,:,:,np.newaxis] / 255. - 0.5 # x_train.shape is (60000, 28, 28, 1)
x_test = x_test[:,:,:,np.newaxis] / 255. - 0.5 # x_test.shape is (10000, 28, 28, 1)
if sys.argv[1] == 'original':
import keras
from li_attack import CarliniLi
class Model:
def __init__(self, restore):
self.num_channels = 1
self.image_size = 28
self.num_labels = 10
self.model = model
def predict(self, data):
return self.model(data)
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
# load model
model = keras.models.load_model('mnist_model.h5')
indices = get_correct_predicted_indices(model)
attack = CarliniLi(sess, Model(model), targeted=False, learning_rate=0.01, max_iterations=1000)
x_adv = attack.attack(x_test[indices], np.identity(10)[y_test[indices]]) # one-hot encoding is required
print(f'original, ASR: {np.sum(y_test[indices] != np.argmax(model.predict(x_adv), axis=1)) / 10}')
print(f'original, LInf norm: {np.max(np.abs(x_adv - x_test[indices]), axis=(1,2,3))}')
#for i, each_x_adv in zip(indices, x_adv): # Optional: we can save the images with `img` directory.
# plt.imshow(each_x_adv, cmap='gray')
# plt.savefig(f'img/original-{i}.png')
elif sys.argv[1] == 'art':
from tensorflow import keras
# load model
model = keras.models.load_model('mnist_model.h5')
indices = get_correct_predicted_indices(model)
from art.attacks.evasion import CarliniLInfMethod
from art.estimators.classification import TensorFlowV2Classifier
attack = CarliniLInfMethod(TensorFlowV2Classifier(model=model, nb_classes=10, input_shape=(28, 28, 1), loss_object=model.loss, clip_values=(-0.5,0.5)), targeted=False, learning_rate=0.01, max_iter=1000, eps=0.16)
x_adv = attack.generate(x_test[indices], y_test[indices])
print(f'ART, ASR: {np.sum(y_test[indices] != np.argmax(model.predict(x_adv), axis=1)) / 10}')
print(f'ART, LInf norm: {np.max(np.abs(x_adv - x_test[indices]), axis=(1,2,3))}')
#for i, each_x_adv in zip(indices, x_adv): # Optional: we can save the images with `img` directory.
# plt.imshow(each_x_adv, cmap='gray')
# plt.savefig(f'img/art-{i}.png') I got below result. Selected indices: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10]
...
original, ASR: 1.0
original, LInf norm: [0.13257101 0.06283894 0.07840255 0.15271175 0.04800587 0.09540931
0.06005499 0.03468984 0.04028573 0.07987416] This means the attack success rate is 100% and the perturbation size (i.e. 3. Generate adversarial images by CarliniLInfMethod with
|
Hi @kaitokishi Thank you very much! This is a really great documentation of your experiment! I will take a look immediately. |
Hi @kaitokishi I have been able to reproduce your results and you are right that we should add the missing minimisation of the perturbation. I would propose that we support this with ART 1.9. We'll definitely work on a solution, but if you would be interested to start working on it please let us know, you would be most welcome! |
Hi @kaitokishi I have pushed updates for
The perturbations and success rates are now similar to the original implementation. The attack runs slower, but supports all deep learning frameworks. We will test if we can reduce the number of gradient and loss calculations, apply Numba, etc. to accelerate the attack and I think it would be good to use your test case to also compare the attacks for L2 and L0. I have used the modified version of attack.py
import sys
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.keras.datasets import mnist
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
formatter = logging.Formatter("[%(levelname)s] %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
def get_correct_predicted_indices(model):
indices = []
for i in range(len(x_test)):
if y_test[i] == np.argmax(model.predict(np.array([x_test[i]]))):
indices.append(i)
if len(indices) == 10:
break
print(f'Selected indices: {indices}')
return indices
# Load a dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# x_train = x_train[:,:,:,np.newaxis] / 255. - 0.5 # x_train.shape is (60000, 28, 28, 1)
# x_test = x_test[:,:,:,np.newaxis] / 255. - 0.5 # x_test.shape is (10000, 28, 28, 1)
x_train = x_train[:,:,:,np.newaxis] / 255. #- 0.5 # x_train.shape is (60000, 28, 28, 1)
x_test = x_test[:,:,:,np.newaxis] / 255. #- 0.5 # x_test.shape is (10000, 28, 28, 1)
if sys.argv[1] == 'original':
import keras
from li_attack import CarliniLi
x_train = x_train - 0.5
x_test = x_test - 0.5
class Model:
def __init__(self, restore):
self.num_channels = 1
self.image_size = 28
self.num_labels = 10
self.model = model
def predict(self, data):
return self.model(data)
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
# load model
model = keras.models.load_model('mnist_model.h5')
indices = get_correct_predicted_indices(model)
attack = CarliniLi(sess, Model(model), targeted=False, learning_rate=0.01, max_iterations=1000)
x_adv = attack.attack(x_test[indices], np.identity(10)[y_test[indices]]) # one-hot encoding is required
print(f'original, ASR: {np.sum(y_test[indices] != np.argmax(model.predict(x_adv), axis=1)) / 10}')
print(f'original, LInf norm: {np.max(np.abs(x_adv - x_test[indices]), axis=(1,2,3))}')
#for i, each_x_adv in zip(indices, x_adv): # Optional: we can save the images with `img` directory.
# plt.imshow(each_x_adv, cmap='gray')
# plt.savefig(f'img/original-{i}.png')
elif sys.argv[1] == 'art':
from tensorflow import keras
# load model
model = keras.models.load_model('mnist_model.h5')
indices = get_correct_predicted_indices(model)
from art.attacks.evasion import CarliniLInfMethod
from art.estimators.classification import TensorFlowV2Classifier
classifier = TensorFlowV2Classifier(model=model,
nb_classes=10,
input_shape=(28, 28, 1),
loss_object=model.loss,
clip_values=(0.0, 1.0),
preprocessing=(0.5, 1.0))
attack = CarliniLInfMethod(classifier=classifier,
targeted=False,
learning_rate=0.01,
max_iter=1000,
confidence=0.0)
x_adv = attack.generate(x_test[indices], y_test[indices])
print(f'ART, ASR: {np.sum(y_test[indices] != np.argmax(classifier.predict(x_adv), axis=1)) / 10}')
print(f'ART, LInf norm: {np.max(np.abs(x_adv - x_test[indices]), axis=(1,2,3))}')
print(f'ART, adversarial: {y_test[indices] != np.argmax(classifier.predict(x_adv), axis=1)}') What do you think? |
The result and the modified code seems great! The original implementation only supports TensorFlow v1, so supporting all deep learning frameworks by ART is brilliant. I agree to use this |
Describe the bug
The loss used in
CarliniLInfMethod
differs from the paper. Therefore,CarliniLInfMethod
fails to generate adversarial images in smalleps
while the original implementation by Carlini can generate adversarial images in theeps
perturbation size.To reproduce
0.1
.CarliniLInfMethod
witheps=0.1
which is sufficient in the original implementation, and get lower attack success rate. I tried random 10 samples from MNIST and got 100% attack success rate (i.e. 10 samples out of 10) in the original implementation but got 60% attack success rate (i.e. 6 samples out of 10, which are the same as used in the original implementation) inCarliniLInfMethod
.Expected behavior
CarliniLInfMethod
should have high attack success rate comparable with Carlini's original implementation even ifeps
is small as in the original implementation.Cause of the bug
is
(see p.10 of the paper) while the loss implemented on
, so they are inconsistent.
The loss of Carlini & Wagner
CarliniLInfMethod
is onlySystem information (please complete the following information):
CarliniLInfMethod
:The text was updated successfully, but these errors were encountered: