Add robustness benchmarking toolkit #1021
Conversation
ROBUSTNESS_BENCHMARKING.md
Outdated
|
|
||
| ```shell | ||
| # noise | ||
| python tools/test_c.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions noise |
There was a problem hiding this comment.
test_c.py has been renamed to test_robustness.py?
There was a problem hiding this comment.
Changed the name
mmdet/datasets/custom.py
Outdated
| @@ -11,6 +11,8 @@ | |||
| from .utils import to_tensor, random_scale | |||
| from .extra_aug import ExtraAugmentation | |||
|
|
|||
| from imagecorruptions import corrupt | |||
There was a problem hiding this comment.
Third-party imports may be moved before relative imports.
mmdet/datasets/custom.py
Outdated
| # corruption | ||
| if self.corruption is not None: | ||
| img = corrupt(img, severity=self.corruption_severity, | ||
| corruption_name=self.corruption) |
There was a problem hiding this comment.
These lines need to be formatted by yapf with the specified style configurations.
tools/robustness_eval.py
Outdated
| for j in range(len(eval_output[distortion][severity]))] | ||
| results[i, severity, :] = mAP | ||
| # if verbose > 0: | ||
| # print(distortion, severity, mAP) |
tools/robustness_eval.py
Outdated
| print("\nmodel: {}".format(osp.basename(filename))) | ||
| if metric is None: | ||
| if 'P' in prints: | ||
| print("Performance on Clean Data [P] ({})" |
There was a problem hiding this comment.
To be consistent with other files, single quote is preferred.
tools/robustness_eval.py
Outdated
| ] | ||
|
|
||
| with open(filename, "rb") as f: | ||
| eval_output = pickle.load(f) |
There was a problem hiding this comment.
These two lines can be simply replaced with mmcv.load(filename).
|
Hi there are some conflicts. Could you fix that? |
|
Sorry that I noted that there were some |
|
Give me a minute. I am currently fixing the yapf errors |
|
The merge conflicts and yapf errors should be fixed. |
* Add robust detection benchmark * Update readmes * Changed readmes for pull request * Ensure pep8 conformity * fixed formatting * Fix yapf errors * minor formatting * fix imports order
This pull request contains tools to benchmark the robustness of object detection models to common image corruptions. The corresponding robust detection benchmark was defined in "Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming".
We tried to design the code in a way that it minimally infer with the current workflow, while adding functionality to evaluate any model implemented in mmdetection for robustness. To corrupt and distort the images we uses the imagecorruptions package we separately developed and distribute through PyPI.