Implement customizable summary writer and indicators of attack failure #1416

beat-buesser · 2021-11-21T00:05:11Z

Description

This pull request introduces a new Summary Writer API which allows to collect metrics based on attack and estimator properties as function of iteration step in TensorBoard format. It provides a default summary writer collecting properties of loss gradients, loss values, adversarial patch, and indicators of attack failure. This summary writer will be available in all attacks in art.attacks as property self.summary_writer and selected attacks including Projected Gradient Descent and Adversarial Patch and Texture have been updated accordingly, more attacks will follow soon. Users can provide customized summary writers to the attacks to collect custom metrics. Introduces a new Union type ESTIMATOR_TYPE representing all estimators in ART.

Fixes #1325

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

codecov-commenter · 2021-11-21T00:12:00Z

Codecov Report

Merging #1416 (5d493c0) into dev_1.9.0 (b77725e) will increase coverage by 8.76%.
The diff coverage is 76.19%.

@@              Coverage Diff              @@
##           dev_1.9.0    #1416      +/-   ##
=============================================
+ Coverage      81.65%   90.42%   +8.76%     
=============================================
  Files            236      237       +1     
  Lines          19214    19386     +172     
  Branches        3387     3438      +51     
=============================================
+ Hits           15690    17530    +1840     
+ Misses          2815     1076    -1739     
- Partials         709      780      +71

Impacted Files	Coverage Δ
art/utils.py	`90.92% <ø> (+1.03%)`	⬆️
art/estimators/object_tracking/pytorch_goturn.py	`72.24% <6.25%> (-3.82%)`	⬇️
...ion/adversarial_patch/adversarial_patch_pytorch.py	`83.18% <50.00%> (+70.96%)`	⬆️
.../adversarial_patch/adversarial_patch_tensorflow.py	`81.81% <50.00%> (+68.22%)`	⬆️
...adient_descent/projected_gradient_descent_numpy.py	`87.50% <60.00%> (-1.66%)`	⬇️
...ient_descent/projected_gradient_descent_pytorch.py	`96.96% <60.00%> (-1.46%)`	⬇️
...escent/projected_gradient_descent_tensorflow_v2.py	`96.03% <60.00%> (-1.49%)`	⬇️
art/attacks/evasion/fast_gradient.py	`84.00% <71.42%> (-0.55%)`	⬇️
art/attacks/attack.py	`92.98% <80.00%> (+1.31%)`	⬆️
...adversarial_texture/adversarial_texture_pytorch.py	`88.37% <80.00%> (-0.66%)`	⬇️
... and 32 more

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

ambrishrawat

Please find comments in the code

ambrishrawat · 2021-12-08T16:19:37Z

art/attacks/evasion/adversarial_patch/adversarial_patch_tensorflow.py

@@ -458,24 +461,20 @@ def generate(self, x: np.ndarray, y: Optional[np.ndarray] = None, **kwargs) -> T
                for images, target, mask_i in dataset:
                    _ = self._train_step(images=images, target=target, mask=mask_i)

+            # Write summary
            if self.summary_writer is not None:  # pragma: no cover


the attack constructors accept “bool” for summary writer, how does that flow work here?

Good point, the super class replace any input type with an instance of SummaryWriter or None.

ambrishrawat · 2021-12-08T16:21:28Z

art/summary_writer.py

+                    global_step=global_step,
+                )
+
+        # Attack Failure Indicators


Are these failure indicator based on some paper? Is it possible to include information here for documentation?

Great suggestion, I will add the reference to Pintor et al. (https://arxiv.org/abs/2106.09947)

ambrishrawat · 2021-12-08T16:22:51Z

art/utils.py

    TENSORFLOWV2_ESTIMATOR_TYPE = Union[  # pylint: disable=C0103
        TensorFlowV2Classifier,
        TensorFlowV2Estimator,
    ]

+    ESTIMATOR_TYPE = Union[  # pylint: disable=C0103


Are the changes in utils.py independent of SummaryWriter? Maybe include an additional line in the PR description?

They are related, but I'll also update the PR description.

ambrishrawat · 2021-12-08T16:23:31Z

tests/test_summary_writer.py

+    yield x_train_mnist[:n_train], y_train_mnist[:n_train], x_test_mnist[:n_test], y_test_mnist[:n_test]
+
+
+def test_update(art_warning, fix_get_mnist_subset, image_dl_estimator):


Should there be a separate test case for boolean values of swd?

Good idea, I'll add additional tests.

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

…e_1325

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

ambrishrawat

Looks good, thanks

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

…e_1325

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

lgtm-com · 2021-12-11T01:56:25Z

This pull request introduces 1 alert when merging c2c33fa into b77725e - view on LGTM.com

new alerts:

1 for Module is imported more than once

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

lgtm-com · 2021-12-13T11:42:44Z

This pull request introduces 3 alerts when merging b91bad1 into b77725e - view on LGTM.com

new alerts:

3 for Module is imported more than once

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

lgtm-com · 2021-12-14T12:39:22Z

This pull request introduces 3 alerts when merging c28a1b3 into b77725e - view on LGTM.com

new alerts:

3 for Module is imported more than once

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

lgtm-com · 2021-12-14T14:34:55Z

This pull request introduces 3 alerts when merging 5d493c0 into b77725e - view on LGTM.com

new alerts:

3 for Module is imported more than once

Beat Buesser added 3 commits November 12, 2021 22:29

Abstract SummaryWriter for TensorBoard support

a51e9f5

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Updates for attack failure indicator 1 and 2

93b5fe3

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Add indicators 3 and 4

a9dd865

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

beat-buesser self-assigned this Nov 21, 2021

beat-buesser added the enhancement New feature or request label Nov 21, 2021

beat-buesser added this to Pull request open in ART 1.9.0 via automation Nov 21, 2021

beat-buesser added this to the ART 1.9.0 milestone Nov 21, 2021

beat-buesser linked an issue Nov 21, 2021 that may be closed by this pull request

integrate "indicators of attack failure" from Pintor et al 2021 #1325

Closed

Merge branch 'dev_1.9.0' into development_issue_1325

6d7361b

beat-buesser changed the title ~~Implement customizable summary writer and indicators of attack failue~~ Implement customizable summary writer and indicators of attack failure Nov 21, 2021

Beat Buesser added 5 commits November 22, 2021 23:12

Updates for style check

9b872e0

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Add class docstring

5014eff

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Change tensor_board to summary_writer

5414571

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update check_params for summary_writer

a024a12

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update tests

749568e

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

beat-buesser requested a review from ambrishrawat December 7, 2021 22:03

Beat Buesser and others added 5 commits December 7, 2021 23:00

Add unit tests for summary writer

dcd7c5e

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Merge branch 'dev_1.9.0' into development_issue_1325

1d1f8a2

Merge branch 'dev_1.9.0' into development_issue_1325

85a8611

Update tests

3a6c555

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update tests

805d64d

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

ambrishrawat reviewed Dec 8, 2021

View reviewed changes

Beat Buesser added 7 commits December 8, 2021 20:50

Address review comments

4528f3a

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Fix tests

f565dfa

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Exclude tensorflow2v1

f2b3f88

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Add reset for summary writer

d4e45db

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update directory names with initialisation

132f8cf

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Merge remote-tracking branch 'origin/dev_1.9.0' into development_issu…

0e0a68a

…e_1325

Update for no summary writer

0ec0d17

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Updates for object detectors in summary writer

61a70f8

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

ambrishrawat approved these changes Dec 10, 2021

View reviewed changes

Beat Buesser added 4 commits December 10, 2021 17:32

Fix pylint

de492fa

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Add summary writer test for object detectors

72fd3d7

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Merge remote-tracking branch 'origin/dev_1.9.0' into development_issu…

0ed10bf

…e_1325

Add summary writer to AdversarialTexturePyTorch

c2c33fa

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Fix style checks

b91bad1

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update indicators

c28a1b3

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

Update tests

5d493c0

Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com>

beat-buesser merged commit c114a49 into dev_1.9.0 Dec 14, 2021

ART 1.9.0 automation moved this from Pull request open to Pull request done Dec 14, 2021

beat-buesser deleted the development_issue_1325 branch December 14, 2021 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement customizable summary writer and indicators of attack failure #1416

Implement customizable summary writer and indicators of attack failure #1416

beat-buesser commented Nov 21, 2021 •

edited

Loading

codecov-commenter commented Nov 21, 2021 •

edited

Loading

ambrishrawat left a comment

ambrishrawat Dec 8, 2021

beat-buesser Dec 8, 2021

ambrishrawat Dec 8, 2021

beat-buesser Dec 8, 2021

ambrishrawat Dec 8, 2021

beat-buesser Dec 8, 2021

ambrishrawat Dec 8, 2021

beat-buesser Dec 8, 2021

ambrishrawat left a comment

lgtm-com bot commented Dec 11, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 14, 2021

		yield x_train_mnist[:n_train], y_train_mnist[:n_train], x_test_mnist[:n_test], y_test_mnist[:n_test]


		def test_update(art_warning, fix_get_mnist_subset, image_dl_estimator):

Implement customizable summary writer and indicators of attack failure #1416

Implement customizable summary writer and indicators of attack failure #1416

Conversation

beat-buesser commented Nov 21, 2021 • edited Loading

Description

Type of change

Checklist

codecov-commenter commented Nov 21, 2021 • edited Loading

Codecov Report

ambrishrawat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ambrishrawat left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Dec 11, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 14, 2021

beat-buesser commented Nov 21, 2021 •

edited

Loading

codecov-commenter commented Nov 21, 2021 •

edited

Loading