Armory 0.15.0

Changelog

User Experience

New Armory instrument module revamps existing metrics computation and provides users with substantially enhanced flexibility in recording/measuring experimental artifacts of interest (#1387)
Added ability to export images with predicted and ground-truth bounding boxes overlaid for CARLA object detection and video tracking (#1261). Also saves bounding box data as coco-formatted json (#1398)
Revamped logging including more user control (#1307)
Log more informative output path info (#1366)
Armory now saves log files to the per-run output directory beside the output json file (#1307)
Deprecation of TF1 (#1290)

Dependency/library updates

Updated to black 22.3 (#1383)
Updated to ART 1.10.1 (#1427)
Updated to Pytorch 1.10, TensorFlow 2.8, and CUDA 11.3 (#1337)
Updated to TFDS v4 (#1323)
Removed apache_beam and pandas dependencies in the main codebase, as well as moved pymongo from requirements to host-requirements (#1277)

Datasets

New CARLA dataset for model training including train/val splits; mitigated annotation issues present in version 1 (#1412)
New CARLA object detection dev set (#1431); mitigated annotation issues present in version 1 (#1449)
New CARLA video tracking dev set (#1431)
Move UCF101 data clipping to the beginning of the dataset preprocessing pipeline instead of the baseline model's forward pass (and implicitly, the gradient computation) (#1309)

Attacks

Added support for a dirty-label poisoning attack on Cifar10 (#1403)
Integrated a new clean-label poisoning attack from ART, Witches’ Brew (#1406)
New CARLA AdversarialPatchPyTorch attack with substantial speed improvements; also includes depth-channel perturbation (#1400)
Updated CARLA Robust DPatch attack to support depth-channel perturbation (#1409)
Updated CARLA video tracking attack Adversarial Texture to support per-frame perspective transform of patch (#1410)

Models

New model weights for CARLA object detection, both RGB and multi-modal (#1445)
Update DeepSpeech 2 to version 3 (PyTorch Lightning) (#1293)

Scenarios

Added a new scenario to support the Witches’ Brew poisoning attack (#1406)
Integration of CIFAR10 dataset and trigger for poisoning scenario (#1403)
Added example of using entailment-based adversarial target labels for ASR scenario (#1407)

Metrics

New Armory instrument module (#1387)
Integrated entailment metric for ASR (#1407)
Added mean success rate metric for CARLA video tracking (#1408)
Additional statistics measured for poisoning scenarios, particularly for filter-based defenses (#1393)
Added poisoning-specific metrics that attempt to characterize the fairness or bias of models and filters (#1360)

Documentation

Added Jupyter notebook tutorial for executing Armory scenarios step-by-step (#1441)
Updated metrics.md with instructions for using new Armory instrument feature (#1387)

Default Configs

"docker_image" field no longer requires version number (see scenario config files for examples)
Organized default configs by scenario and evaluation period (#1374)
New CIFAR10 poisoning config (#1403)
New config for ASR entailment (#1407)

Performance Improvement / Bug Fixes

Fixed bugs present when running Armory with the --jupyter flag (#1416)
Provide consistent ART data pathing (#1352)
Fixed issue with GPU memory usage in PyTorch containers (#1341)
Refactored Docker builds such that there is an Armory base image (twosixarmory/base) that will be built infrequently (likely quarterly) and pushed to dockerhub (#1307)
Refactor of sample exporting feature (#1297)
Properly save model weights to Armory data directory instead of tmp (#1281)