Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Foolbox for non-image inputs? #80

Closed
EdwardRaff opened this issue Nov 24, 2017 · 16 comments
Closed

Foolbox for non-image inputs? #80

EdwardRaff opened this issue Nov 24, 2017 · 16 comments

Comments

@EdwardRaff
Copy link

Hi, I read through the docs and issues, and couldn't find any information about this.

I want to generate adversarial examples for some non-image based problems. In my specific case, the inputs are fixed length sequence of integers which then goes into an embedding layer and into the network.

Is there a way to use foolbox in this scenario at the moment?

Thanks for your time!
@jonasrauber
Copy link
Member

Hi @EdwardRaff, sorry for not responding earlier. Somehow, I accidentally marked this issue as read.

In principle, it should be no problem to use Foolbox in this scenario. Image-specific things like the channel_axis argument can probably simply be ignored (they are only relevant for certain image-specific attacks). Same goes for arguments that have image-specific names (e.g. image, original_image), just use them as if they were named in a more generic way (we will fix this in a future version).

@jonasrauber
Copy link
Member

@EdwardRaff can this be closed?

@EdwardRaff
Copy link
Author

I don't think so. I just tried using the library as you state, and get an unhelpful error message.

Traceback (most recent call last):
  File "fool_malconv.py", line 108, in <module>
    adversarial = attack(batch, 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 89, in __call__
    find = Adversarial(model, criterion, image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 61, in __init__
    self.predictions(original_image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 235, in predictions
    assert not strict or self.in_bounds(image)
AssertionError

where the code that does the work

     fmodel = PyTorchModel(model, bounds=(0, 255), num_classes=2, cuda=args.num_gpus > 0)

     attack = foolbox.attacks.FGSM(fmodel)

     adversarial = attack(batch, 1)

@EdwardRaff
Copy link
Author

Looks like that error was for having too lose a bound, but still results in an error

Traceback (most recent call last):
  File "fool_malconv.py", line 108, in <module>
    adversarial = attack(batch, 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 89, in __call__
    find = Adversarial(model, criterion, image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 61, in __init__
    self.predictions(original_image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 238, in predictions
    predictions = self.__model.predictions(image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 122, in predictions
    return np.squeeze(self.batch_predictions(image[np.newaxis]), axis=0)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 64, in batch_predictions
    predictions = self._model(images)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/Development/SeveringMalConv/web/model.py", line 73, in forward
    x = self.lookup_table(x)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    )(input, self.weight)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 33, in forward
    assert indices.dim() <= 2
AssertionError

@jonasrauber
Copy link
Member

Could you please provide the full code or a minimal example to reproduce the problem.
Btw, you used bounds=(0, 255) which looks wrong if you have non-image inputs. What kind of data do you have?

Also, you are feeding a variable called batch to the attack, which sounds like it could be the reason for the above error message. When applying an attack, you should provide a single input and the corresponding label.

@EdwardRaff
Copy link
Author

My data is byte based, so there are 256 possible byte values (and a special EOF token that I think it was unhappy about).

Unfortunately, I'm not allowed to share the code at this moment in time.

I was doing batches but a batch size of 1. I used numpy.squeeze to remove the first dimension, and now get this error.

Traceback (most recent call last):
  File "fool_malconv.py", line 110, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 89, in __call__
    find = Adversarial(model, criterion, image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 61, in __init__
    self.predictions(original_image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 238, in predictions
    predictions = self.__model.predictions(image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 122, in predictions
    return np.squeeze(self.batch_predictions(image[np.newaxis]), axis=0)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 65, in batch_predictions
    predictions = predictions.data
AttributeError: 'tuple' object has no attribute 'data'

@jonasrauber
Copy link
Member

Which Foolbox and PyTorch version are you using?

Is your PyTorch model (the model you pass to foolbox.models.PyTorch) an instance of PyTorch's nn.Module.

It looks like the forward method of your nn.Module subclass does return a tuple instead of Tensor with a data attribute.

@jonasrauber
Copy link
Member

Foolbox currently assumes that the forward method returns a torch.autograd.Variable that holds the tensor that contains the predictions. Maybe this is a bit too strict on our side, but so far it worked fine. Foolbox will automatically convert inputs from NumPy to a PyTorch Tensor to a PyTorch Variable holding that Tensor, then apply the PyTorch model (effectively calling the forward method) and then unpacking the result again, i.e. it calls the PyTorch model with a PyTorch Variable and expects to get one back.

A very simple net (used in our test cases) can for example be implemented like this. It doesn't do anything useful, but shows that things should be fine as long as your forward method just does the usual transformations.

class Net(nn.Module):

        def __init__(self):
            super(Net, self).__init__()

        def forward(self, x):
            x = torch.mean(x, 3)
            x = torch.squeeze(x, dim=3)
            x = torch.mean(x, 2)
            x = torch.squeeze(x, dim=2)
            logits = x
            return logits

@EdwardRaff
Copy link
Author

I'm using PyTorch version 1.12 and the latest version of foolbox from pip.

The information about what is expected is very useful. Please consider adding it to the documentation. I'd also encourage some error messages that indicate what the problem is.

I've adjusted my code to match your stated exceptions. But I'm still having some trouble. If i return the logits as an array of shape (2) (1 dim, 2 values for the 2 classes), I get an error in batch_predictions asserting that the predictions.ndim == 2.

Traceback (most recent call last):
  File "fool_malconv.py", line 116, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 89, in __call__
    find = Adversarial(model, criterion, image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 61, in __init__
    self.predictions(original_image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 238, in predictions
    predictions = self.__model.predictions(image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 122, in predictions
    return np.squeeze(self.batch_predictions(image[np.newaxis]), axis=0)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 69, in batch_predictions
    assert predictions.ndim == 2
AssertionError

If I return an array of shape (1,2), i get an error in predictions_and_gradient asserting that the image dimension is == 3.

Traceback (most recent call last):
  File "fool_malconv.py", line 116, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 98, in __call__
    _ = self._apply(adversarial, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/gradientsign.py", line 22, in _apply
    gradient = a.gradient()
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 322, in gradient
    gradient = self.__model.gradient(image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 208, in gradient
    _, gradient = self.predictions_and_gradient(image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 89, in predictions_and_gradient
    assert image.ndim == 3
AssertionError

@jonasrauber
Copy link
Member

The second one is correct. The PyTorch model (i.e. the nn.Module subclass) should always deal with batches, i.e. expect batches and return batches, so in your case, it should return something with shape (1, 2).

The error you see in that case is indeed in image-specific assertion that should not be there. I will change this as soon as possible, but it will take a bit of time (running all the tests, publishing a new version on PyPI, etc.)… For now, it would be great if you could just open the file foolbox/models/pytorch.py and comment the lines 89 and 159 (both saying assert image.ndim == 3) out. You can find the location of that file by opening a Python interpreter, running import foolbox and then running foolbox. That should print the path where the foolbox package is located in your system. Something like this:

Python 3.6.3 (default, Oct  6 2017, 08:44:35) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import foolbox
>>> foolbox
<module 'foolbox' from 'SOME PATH TO foolbox/__init__.py'>

Just go to that path and edit the two lines I mentioned. Let me know if this fixed the problem.

Sorry for all the trouble. Originally, we developed Foolbox in a very image-centric way. As I said, this will change and for TensorFlow I think I verified that it works, but apparently, for PyTorch these assertions were still left.

Next time it would great if you could create a minimal example that reproduces the error – it doesn't need to be your actual secret code ;-)

Regarding the documentation: yes, it has to improve ;-)

@EdwardRaff
Copy link
Author

No worries, I appreciate all the help! I made the changes you mentioned, and it now gets to what I was afraid of: what to do about the embedding layer. This is the error the comes out

Traceback (most recent call last):
  File "fool_malconv.py", line 116, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 98, in __call__
    _ = self._apply(adversarial, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/gradientsign.py", line 22, in _apply
    gradient = a.gradient()
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 322, in gradient
    gradient = self.__model.gradient(image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 208, in gradient
    _, gradient = self.predictions_and_gradient(image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 95, in predictions_and_gradient
    predictions = self._model(images)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/Development/SeveringMalConv/web/model_infer.py", line 77, in forward
    x = self.lookup_table(x)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    )(input, self.weight)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 34, in forward
    assert not self.needs_input_grad[0], "Embedding doesn't " \
AssertionError: Embedding doesn't compute the gradient w.r.t. the indices

Yea, I know a simplified example would help. Unfortunately I don't get to fully dictate my time :-/ I'm working on getting approval to open source the code I'm working on as its not significant, but that takes time.

@EdwardRaff
Copy link
Author

I also tried the LocalSearchAttack since it seems to be non-gradient based, but it errors on line 82 asserting that the image has two axes. And the axes object is empty.

Traceback (most recent call last):
  File "fool_malconv.py", line 117, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 98, in __call__
    _ = self._apply(adversarial, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/localsearch.py", line 87, in _apply
    assert len(axes) == 2
AssertionError

@jonasrauber
Copy link
Member

Well, if your model isn't differentiable w.r.t. its inputs, it's hard to apply a gradient-based adversarial attack. The problem with the LocalSearchAttack is: it has built-in notion of pixels and is therefore image-specific. I do think it should be possible to generalize the attack to arbitrary inputs, but I haven't thought about it in detail. Maybe you want to give it a try? Alternatively, it might be possible to change the shape of your data to something that looks like an image, e.g. an image with width 1, height N and 1 color channel. Again, no guarantees, but I think that should be easy and might work. Finally, you can use a decision-based attack: it requires even less than the LocalSearchattack. To get started and make sure that your code works, I recommend something basic like the AdditiveUniformNoiseAttack. Once everything works, you can use our recently published BoundaryAttack. https://foolbox.readthedocs.io/en/latest/modules/attacks/decision.html

@EdwardRaff
Copy link
Author

Well, if your model isn't differentiable w.r.t. its inputs, it's hard to apply a gradient-based adversarial attack.

That is what I was asking about in my first post since there is an embedding layer. I wasn't sure how it would be handled.

I was able to get it to work with the AdditiveUniformNoiseAttack attack. I did try the boundary attack after, but got the following error. I'm not sure where its getting a non float input from.

Traceback (most recent call last):
  File "fool_malconv.py", line 119, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py", line 165, in __call__
    threaded_gen=threaded_gen)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 98, in __call__
    _ = self._apply(adversarial, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py", line 183, in _apply
    return self._apply_inner(pool, *args, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py", line 206, in _apply_inner
    assert external_dtype in [np.float32, np.float64]
AssertionError

I also tried it with the GPU since the error was so strange, and got this even more confusing error. It looks like it got what it was expecting.

Traceback (most recent call last):
  File "fool_malconv.py", line 119, in <module>
    adversarial = attack(np.squeeze(batch,axis=0), 1)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py", line 165, in __call__
    threaded_gen=threaded_gen)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/attacks/base.py", line 89, in __call__
    find = Adversarial(model, criterion, image, label)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 61, in __init__
    self.predictions(original_image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/adversarial.py", line 238, in predictions
    predictions = self.__model.predictions(image)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/base.py", line 122, in predictions
    return np.squeeze(self.batch_predictions(image[np.newaxis]), axis=0)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/foolbox/models/pytorch.py", line 64, in batch_predictions
    predictions = self._model(images)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/Development/SeveringMalConv/web/model_infer.py", line 79, in forward
    x = self.lookup_table(x)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    )(input, self.weight)
  File "/home/edraff/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 53, in forward
    output = torch.index_select(weight, 0, indices.view(-1))
TypeError: torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, torch.LongTensor), but expected (torch.cuda.FloatTensor source, int dim, torch.cuda.LongTensor index)

@jonasrauber
Copy link
Member

If your model requires inputs to be integers, it's hard to apply any of these attacks directly. Sorry if I missed that detail in your first message. I think it should still be possible to do what you want, but one first needs to define adversarials in your setting properly and then think about how they could be found. Technically, it might be possible that you define a model that accepts floats and rounds them to integers and then applying the attacks to that model. Without your code and a description of the scenario, I don't think I can be much of a help here. You might also want to get familiar with Foolbox in a more common setting (say images or even audio data) first and then think about the differences to your problem.

@jonasrauber jonasrauber mentioned this issue Feb 5, 2018
Closed
@EdwardRaff EdwardRaff mentioned this issue Feb 22, 2018
Closed
@15556535718
Copy link

Hi,@jonasrauber. I meet several problems that confused me. My python is Python 3.6.3 on Windows64, and my computer can not connect the Internet. So I need your help. Can you give me an example? This example shows how to use foolbox to generate adversarial example for MNIST. If you can give me the code is better. Hoping the code is shorter and shows explanation. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@EdwardRaff @jonasrauber @15556535718