Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndexError in CarliniL2Method #29

Closed
weitianli opened this Issue Jan 23, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@weitianli
Copy link

weitianli commented Jan 23, 2019

Describe the bug
I implement a simple resnet9 and want to test cw2 result. All is good when I set max_iter=10. But when I set max_iter=1000 that is same value as cw2 original version. The index error comes.

To Reproduce
Steps to reproduce the behavior:

  1. Set a simple CarliniL2Method.
  2. cl2m = CarliniL2Method(classifier=ptc, targeted=True, max_iter=1000)
  3. run the cw2 code like in the cw2unittest
  4. See error

Expected behavior
No index error and the result comes out.

System information (please complete the following information):

  • Linux 16.04
  • Python 3.6
  • ART version 0.4.0
  • PyTorch 0.4.1
# First attack
cl2m = CarliniL2Method(classifier=ptc, targeted=True, max_iter=1000)
params={'y': targets}
x_test_adv = cl2m.generate(inputs, **params)
y_pred_adv = np.argmax(ptc.predict(x_test_adv , batch_size=32), axis=1)
logger.info('CW2 Success Rate: %.2f', (sum(adv_label == y_pred_adv) / float(len(adv_label))))
logger.info('Model Acc: %.2f', (sum(adv_original_label == y_pred_adv) / float(len(adv_original_label))))

Traceback (most recent call last):
  File "cw_pytorch.py", line 172, in <module>
    x_test_adv = cl2m.generate(inputs, **params)
  File "/home/weitian/anaconda3/envs/xnor/lib/python3.6/site-packages/art/attacks/carlini.py", line 380, in generate
    x_adv_batch_tanh[active_and_update_adv] = x_adv_batch_tanh[update_adv] + \
IndexError: boolean index did not match indexed array along dimension 0; dimension is 18 but corresponding boolean dimension is 17

@mathsinn mathsinn self-assigned this Jan 23, 2019

@mathsinn

This comment has been minimized.

Copy link
Collaborator

mathsinn commented Jan 23, 2019

Thank you @weitianli I'm looking into it.

@ririnicolae ririnicolae added the bug label Jan 23, 2019

@ririnicolae

This comment has been minimized.

Copy link
Collaborator

ririnicolae commented Jan 23, 2019

Hi, @weitianli! Fix is now out, please install from clone for the issue to be solved. Let us know if you encounter any further problems!

@mathsinn

This comment has been minimized.

Copy link
Collaborator

mathsinn commented Jan 23, 2019

@weitianli Normally, with the new version of the attack, an max_iter value significantly smaller than 1000 should suffice. Before we were using gradient descent with decayed learning rate for minimizing the CWL2 objective, now we are using binary line search which should converge faster and be more stable. Please let us know if you have any issues with determining the right value for this parameter.

@weitianli

This comment has been minimized.

Copy link
Author

weitianli commented Jan 23, 2019

Thank you very much! I will try it!

@weitianli

This comment has been minimized.

Copy link
Author

weitianli commented Jan 25, 2019

I have a small problem about the max_iter. I found your tutorial code about cw set max_iter is 10. Is it the same effect like Cleverhans tutorial(max_iter=100) or CW(max_iter=1000) original version?
Also, I have a small problem about eps in FGSM calculation. Is eps formula in ART is 1*(1/255)?
Thank you very much!
@mathsinn @ririnicolae

@ririnicolae

This comment has been minimized.

Copy link
Collaborator

ririnicolae commented Jan 25, 2019

@weitianli What is the problem you encounter with max_iter in C&W? What about the one on the eps value in FGSM? I would say the number of iterations is not strictly equivalent to the other implementations, just that in ART a small number of iterations should suffice. The eps value depends on the data range of your classifier. For example, if your data is normalized between 0 and 1, eps should also be in that range. If your data is between 0 and 255, eps can take values up to 255.

@weitianli

This comment has been minimized.

Copy link
Author

weitianli commented Jan 26, 2019

I got it. Thank you for your kind response.

ririnicolae added a commit that referenced this issue Feb 1, 2019

Merge new features for release
  Changes:
  - Add spatial transformations evasion attack
  - Add elastic net (EAD) evasion attack
  - Add data generator support for multiple types of TensorFlow iterators
  - Add new function and property to the Classifier API that allow to explicitly control the learning phase (train/test)
  - Add reports for poisoning module
  - Add new parameter `batch_size`
  - Add `ExpectationOverTransformations` class
  - Add parameter `expectation` to evasion attacks
  - Update list of attacks supported by universarl perturbation
  - Update PyLint and Travis configs
  - Fix indexing error in C&W L_2 attack (issue #29)
  - Fix universal perturbation stop condition
  - Fix wrror with data subsampling in AdversarialTrainer

Conflicts:
	.pylintrc
	README.md
	art/attacks/__init__.py
	art/attacks/carlini.py
	art/attacks/carlini_unittest.py
	art/attacks/deepfool.py
	art/attacks/deepfool_unittest.py
	art/attacks/newtonfool_unittest.py
	art/attacks/saliency_map.py
	art/attacks/sampling_model_theft.py
	art/attacks/sampling_model_theft_unittest.py
	art/classifiers/classifier_unittest.py
	art/classifiers/ensemble.py
	art/classifiers/ensemble_unittest.py
	art/classifiers/keras.py
	art/classifiers/mxnet.py
	art/classifiers/pytorch.py
	art/classifiers/tensorflow_unittest.py
	art/data_generators.py
	art/data_generators_unittest.py
	art/defences/adversarial_trainer.py
	art/defences/reverse_sigmoid_unittest.py
	art/defences/variance_minimization.py
	art/defences/variance_minimization_unittest.py
	art/detection/detector_unittest.py
	art/poison_detection/activation_defence.py
	art/poison_detection/clustering_analyzer.py
	art/poison_detection/clustering_analyzer_unittest.py
	notebooks/ART_demo.ipynb
	readthedocs.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.