Transformer Object Detection #2047

kieranfraser · 2023-03-02T16:42:49Z

Description

Adding DEtection TRansformer (DETR) to ART to facilitate object detection using Transformers.

DETR estimator added to art/estimators/object_detection
requires additional method in art/estimators/pytorch.py which sets MultiheadAttention layer into train/eval mode
associated tests added
notebook demonstrating use of DETR estimator detecting objects and under adversarial patch attack added to notebooks/adversarial_patch/ and updated notebooks/ README

Paper: "End-to-End Object Detection with Transformers"

Fixes # (issue)
#1638

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

Test prediction of DETR
Test loss gradient
Test applying preprocessing defences
Test computation of losses (classification, bbox, giou)
Test DETR with PGD attack

Test Configuration:

OS
Python version
ART version or commit number
TensorFlow / Keras / PyTorch / MXNet version

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

codecov-commenter · 2023-03-02T16:49:45Z

Codecov Report

Merging #2047 (cb49cdd) into dev_1.15.0 (30e513c) will decrease coverage by 9.59%.
The diff coverage is 73.09%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@              Coverage Diff               @@
##           dev_1.15.0    #2047      +/-   ##
==============================================
- Coverage       75.97%   66.38%   -9.59%     
==============================================
  Files             297      307      +10     
  Lines           26516    27480     +964     
  Branches         4867     5047     +180     
==============================================
- Hits            20145    18243    -1902     
- Misses           5325     8198    +2873     
+ Partials         1046     1039       -7

Impacted Files	Coverage Δ
art/attacks/evasion/auto_conjugate_gradient.py	`86.52% <ø> (+0.94%)`	⬆️
art/attacks/evasion/brendel_bethge.py	`87.01% <ø> (ø)`
.../defences/trainer/ibp_certified_trainer_pytorch.py	`75.00% <0.00%> (-3.17%)`	⬇️
art/attacks/poisoning/bad_det/bad_det_rma.py	`96.42% <33.33%> (ø)`
...s/certification/randomized_smoothing/tensorflow.py	`89.83% <50.00%> (-1.55%)`	⬇️
...cks/poisoning/perturbations/image_perturbations.py	`88.46% <55.55%> (ø)`
...certification/derandomized_smoothing/tensorflow.py	`83.33% <66.66%> (-1.67%)`	⬇️
art/estimators/object_detection/pytorch_yolo.py	`77.57% <68.67%> (+64.27%)`	⬆️
.../object_detection/pytorch_detection_transformer.py	`69.93% <69.93%> (ø)`
art/estimators/classification/tensorflow.py	`82.92% <75.00%> (+6.88%)`	⬆️
... and 9 more

... and 77 files with indirect coverage changes

notebooks/vision/coco_eval.py

notebooks/vision/coco_utils.py

art/estimators/object_detection/pytorch_detr.py

beat-buesser

Hi @kieranfraser Thank you very much for your pull request! I have added a few review comments, please let me know what you think.

art/estimators/object_detection/pytorch_detr.py

beat-buesser · 2023-05-10T09:51:40Z

art/estimators/object_detection/pytorch_detr.py

+logger = logging.getLogger(__name__)
+
+
+def box_cxcywh_to_xyxy(x):


What is the license of these functions from https://github.com/facebookresearch/detr?

Can we add typing to these functions?

The license is Apache 2.0 which should be okay?

tests/estimators/object_detection/test_pytorch_detr.py

beat-buesser · 2023-05-10T09:59:29Z

tests/estimators/object_detection/test_pytorch_detr.py

@@ -0,0 +1,300 @@
+# MIT License
+#


We need to either add a pytest call to run_tests.sh or integrate into one existing or new dedicated testing runs for object detectors to make the tests run automatically.

Tried to add to run_tests.sh, but tests failing in setup with urllib.error.HTTPError: HTTP Error 500: Internal Server Error, which could be due to needing to pull in the pre-trained weights from torch.hub? The PyTorch Faster-RCNN test, which also pulls a pre-trained model from torch.hub, is called from a GitHub workflow (ci-pytorch-object-detectors.yml) - so that might be a better place to call?

art/estimators/object_detection/pytorch_detection_transformer.py

+        # Set device
+        self._device: torch.device
+        if device_type == "cpu" or not torch.cuda.is_available():
+            self._device = torch.device("cpu")


art/estimators/object_detection/pytorch_detection_transformer.py

+            self._device = torch.device("cpu")
+        else:  # pragma: no cover
+            cuda_idx = torch.cuda.current_device()
+            self._device = torch.device(f"cuda:{cuda_idx}")


Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

…utils from torchvision repo. pretrained vit output to be investigated after training. Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

…nd robust dpatch attack (for COCO image, still to check: IN images) Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Bumps [jax[cpu]](https://github.com/google/jax) from 0.4.6 to 0.4.8. - [Release notes](https://github.com/google/jax/releases) - [Changelog](https://github.com/google/jax/blob/main/CHANGELOG.md) - [Commits](google/jax@jax-v0.4.6...jax-v0.4.8) --- updated-dependencies: - dependency-name: jax[cpu] dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.2.2...7.3.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Bumps [pandas](https://github.com/pandas-dev/pandas) from 1.5.3 to 2.0.1. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Commits](pandas-dev/pandas@v1.5.3...v2.0.1) --- updated-dependencies: - dependency-name: pandas dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: signedqiu <10063108+signedqiu@user.noreply.gitee.com> Signed-off-by: signedqiu <chunlinqiu@whu.edu.cn> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Muhammad Zaid Hameed <Zaid.Hameed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

…tebook: validated attack with PGD. Updates to test: correct loss computation and PGD. Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

art/defences/trainer/adversarial_trainer_trades.py

+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def fit_generator(self, generator: DataGenerator, nb_epochs: int = 20, **kwargs):


beat-buesser · 2023-06-14T22:53:27Z

Continuing in #2192

beat-buesser self-requested a review March 2, 2023 16:45

beat-buesser self-assigned this Mar 2, 2023

beat-buesser added enhancement New feature or request notebooks Notebooks labels Mar 2, 2023

beat-buesser added this to Pull request open in ART 1.14.0 via automation Mar 2, 2023

beat-buesser added this to the ART 1.14.0 milestone Mar 2, 2023

beat-buesser linked an issue Mar 2, 2023 that may be closed by this pull request

Implement object detection estimators for Vision Transformers (ViT) #1638

Closed

github-advanced-security bot found potential problems Mar 6, 2023

View reviewed changes

notebooks/vision/coco_eval.py Fixed Show fixed Hide fixed

notebooks/vision/coco_eval.py Fixed Show fixed Hide fixed

notebooks/vision/coco_eval.py Fixed Show fixed Hide fixed

notebooks/vision/coco_eval.py Fixed Show fixed Hide fixed

notebooks/vision/coco_utils.py Fixed Show fixed Hide fixed

beat-buesser changed the base branch from main to dev_1.14.0 March 11, 2023 00:08

beat-buesser removed this from Pull request open in ART 1.14.0 Mar 15, 2023

beat-buesser modified the milestones: ART 1.14.0, ART 1.15.0 Mar 15, 2023

github-advanced-security bot found potential problems Mar 20, 2023

View reviewed changes

art/estimators/object_detection/pytorch_detr.py Fixed Show fixed Hide fixed

art/estimators/object_detection/pytorch_detr.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Apr 13, 2023

View reviewed changes

art/estimators/object_detection/pytorch_detr.py Fixed Show fixed Hide fixed

kieranfraser changed the title ~~ViT Object Detection~~ Transformer Object Detection Apr 13, 2023

kieranfraser force-pushed the dev_vit_object_detection branch from ee3e4a4 to 4ae0110 Compare April 17, 2023 16:18

kieranfraser marked this pull request as ready for review April 20, 2023 12:47

beat-buesser changed the base branch from dev_1.14.0 to dev_1.15.0 April 20, 2023 13:19

beat-buesser changed the base branch from dev_1.15.0 to dev_1.14.0 April 20, 2023 13:19

beat-buesser changed the base branch from dev_1.14.0 to dev_1.15.0 April 20, 2023 13:20

beat-buesser removed the notebooks Notebooks label May 10, 2023

beat-buesser requested changes May 10, 2023

View reviewed changes

beat-buesser added this to Pull request review in ART 1.15.0 May 10, 2023

github-advanced-security bot found potential problems May 11, 2023

View reviewed changes

kieranfraser added 5 commits June 14, 2023 09:35

ViT backbone object detector with fasterRCNN example

1fbd8b8

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

ViT backbone object detector with fasterRCNN example

06d405c

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

training pipeline for fasterrcnn with vit backbone working. requires …

903ff71

…utils from torchvision repo. pretrained vit output to be investigated after training. Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Adding pytorch detr. Working example demonstrating object detection a…

4d44b17

…nd robust dpatch attack (for COCO image, still to check: IN images) Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

DETR with original source methods attributed

23ead6d

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

f4str and others added 21 commits June 14, 2023 09:35

fix poison backdoor image trigger bug

98e7707

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

minor backdoor image insert optimizations

d9fbdb1

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Updated class name, added typing and other minor fixes.

83c63ea

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Updated class name, added typing and other minor fixes.

6e45ccf

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Added test call to github workflow

946f1ff

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

add optimizer parameter to tensorflow v2 classifier

4050eea

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

add default train step for tensorflow classifiers

af15252

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

refactor tensorflow tests to use default train step

0c8ef35

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

refactor tensorflow notebooks to use default train step

7852d2a

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

bug fix with new tensorflow optimizer parameter

224ab02

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

fix tensorflow model repr test case

9e7f392

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

address review comments

bae5ab1

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

fix Tensor Device Inconsistencies in pgd

d607169

Signed-off-by: signedqiu <10063108+signedqiu@user.noreply.gitee.com> Signed-off-by: signedqiu <chunlinqiu@whu.edu.cn> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

adding trades adversarial training

f27a778

Signed-off-by: Muhammad Zaid Hameed <Zaid.Hameed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

changes after review

0842c44

Signed-off-by: Muhammad Zaid Hameed <Zaid.Hameed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

changes for type checking style checks

1fdf552

Signed-off-by: Muhammad Zaid Hameed <Zaid.Hameed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

correcting parameters description in TRADES trainer base class

d6a29a2

Signed-off-by: Muhammad Zaid Hameed <Zaid.Hameed@ibm.com> Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

Updates to DETR: cleaned up resizing; correct clipping. Updates to no…

e0fe372

…tebook: validated attack with PGD. Updates to test: correct loss computation and PGD. Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

kieranfraser force-pushed the dev_vit_object_detection branch from 0687aec to e0fe372 Compare June 14, 2023 08:36

kieranfraser changed the base branch from dev_1.15.0 to dev_1.14.0 June 14, 2023 08:42

kieranfraser changed the base branch from dev_1.14.0 to dev_1.15.0 June 14, 2023 08:42

Fixing formatting

cb49cdd

Signed-off-by: Kieran Fraser <Kieran.Fraser@ibm.com>

kieranfraser requested a review from beat-buesser June 14, 2023 08:54

github-advanced-security bot found potential problems Jun 14, 2023

View reviewed changes

beat-buesser closed this Jun 14, 2023

ART 1.15.0 automation moved this from Pull request review to Pull request done Jun 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transformer Object Detection #2047

Transformer Object Detection #2047

kieranfraser commented Mar 2, 2023 •

edited

Loading

codecov-commenter commented Mar 2, 2023 •

edited

Loading

beat-buesser left a comment

beat-buesser May 10, 2023

beat-buesser May 10, 2023

kieranfraser May 11, 2023

beat-buesser May 10, 2023

kieranfraser May 11, 2023

beat-buesser commented Jun 14, 2023

		logger = logging.getLogger(__name__)


		def box_cxcywh_to_xyxy(x):

Transformer Object Detection #2047

Transformer Object Detection #2047

Conversation

kieranfraser commented Mar 2, 2023 • edited Loading

Description

Type of change

Testing

Checklist

codecov-commenter commented Mar 2, 2023 • edited Loading

Codecov Report

beat-buesser left a comment

Choose a reason for hiding this comment

beat-buesser May 10, 2023

Choose a reason for hiding this comment

beat-buesser May 10, 2023

Choose a reason for hiding this comment

kieranfraser May 11, 2023

Choose a reason for hiding this comment

beat-buesser May 10, 2023

Choose a reason for hiding this comment

kieranfraser May 11, 2023

Choose a reason for hiding this comment

beat-buesser commented Jun 14, 2023

kieranfraser commented Mar 2, 2023 •

edited

Loading

codecov-commenter commented Mar 2, 2023 •

edited

Loading