149 implement robust dpatch attack #155
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t-robust-dpatch-attack
…erent detection model types
…-robust-dpatch-attack
deprit
reviewed
Jun 5, 2024
examples/src/armory/examples/object_detection/datasets/visdrone.py
Outdated
Show resolved
Hide resolved
examples/src/armory/examples/object_detection/datasets/visdrone.py
Outdated
Show resolved
Hide resolved
| # TODO non-zero min value | ||
| self.patch_min = 0 | ||
| self.patch_max = self.model.inputs_spec.scale.max | ||
| self.patch = torch.randint(0, 255, self.patch_shape) / 255 * self.patch_max |
There was a problem hiding this comment.
Let's define a static method for patch initialization to allow easy subclassing, something like the following.
@staticmethod
def _patch_init(shape: Sequence(int) ...) -> torch.Tensor:
...
| # x_1, y_1 = self.patch_location | ||
| x_2 = x_1 + self.patch_shape[1] | ||
| y_2 = y_1 + self.patch_shape[2] | ||
| inputs_with_patch = inputs.clone() |
There was a problem hiding this comment.
Is this clone operation necessary? Does the batch loader already create tensor copies of the batch?
Comment on lines
+63
to
+73
| # Apply patch to image | ||
| x_1 = random.randint(0, inputs.shape[3] - self.patch_shape[2]) | ||
| y_1 = random.randint(0, inputs.shape[2] - self.patch_shape[1]) | ||
| # x_1, y_1 = self.patch_location | ||
| x_2 = x_1 + self.patch_shape[1] | ||
| y_2 = y_1 + self.patch_shape[2] | ||
| inputs_with_patch = inputs.clone() | ||
| inputs_with_patch[:, :, x_1:x_2, y_1:y_2] = self.patch | ||
|
|
||
| # Apply random augmentations to images | ||
| inputs_with_augmentations = self.augmentation(inputs_with_patch) |
There was a problem hiding this comment.
Let's encapsulate these lines into a method that applies the patch to a batch of input images with augmentations.
def _apply_patch(self,
inputs: torch.Tensor,
patch: torch.Tensor,
location: torch.Tensor,
augmentations: kornia.augmentation.container.ImageSequential
) -> torch.Tensor:
...
examples/src/armory/examples/object_detection/visdrone_yolov5_robustdpatch.py
Outdated
Show resolved
Hide resolved
| resize = A.Compose( | ||
| [ | ||
| A.LongestMaxSize(max_size=max_size), | ||
| A.PadIfNeeded( |
There was a problem hiding this comment.
It may not be necessary to pad all input images to the max size, some models take a list of images as input (e.g. Faster-RCNN).
mwartell
approved these changes
Jun 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial implementation of Robust DPatch using a lightning module. This PR does not yet offer attack optimization as part of the armory-library API, but rather implements it entirely in user/example code.
Outside of editorial fixes or blatant errors, most comments on this PR will probably be addressed in a follow-on when the attack optimization is made generic and moved to be part of armory-library.