Auto-ART: Automated Adversarial Robustness Testing

A Python framework for comprehensive adversarial robustness evaluation of ML models and autonomous AI agents. Built on ART v1.20+, aligned with OWASP LLM Top 10 (2025) and NIST AI 100-2.

What It Does

• 43 attacks across evasion (28), poisoning (4), extraction (3), inference (3), regression (3), LLM (1), and agentic categories
• 28 defence modules spanning preprocessors, postprocessors, trainers, transformers, detectors, guardrails, and certification
• 9 ML frameworks -- PyTorch, TensorFlow, Keras, scikit-learn, XGBoost, LightGBM, CatBoost, GPy, Transformers
• YAML-driven orchestrator with 6 evaluation phases, real attack execution, and CI/CD gates
• SARIF output for GitHub Code Scanning, GitLab SAST, and Azure DevOps integration
• Adaptive attack selection -- memory-guided, tiered escalation inspired by AutoRedTeamer
• Parallel execution engine -- concurrent attack evaluation with configurable workers and timeouts
• Continuous red teaming with multi-turn adaptive attacks across 7 OWASP-mapped categories
• Robustness certification via Randomized Smoothing, DeRandomized Smoothing, DeepZ, IBP, and GREAT Score (ART 1.20+)
• AgentOps telemetry with state machine tracing and infinite loop detection

Install

pip install .                   # Core (includes PyYAML, ART 1.20+)
pip install ".[pytorch]"        # + PyTorch
pip install ".[agentic]"        # + Agentic security
pip install ".[dev]"            # + Dev tools

Quick Start

Evaluate a model:

from auto_art.core.evaluation.art_evaluator import ARTEvaluator
from auto_art.core.evaluation.config.evaluation_config import EvaluationConfig, ModelType, Framework

evaluator = ARTEvaluator(model_obj=None, config=EvaluationConfig(
    model_type=ModelType.CLASSIFICATION, framework=Framework.PYTORCH,
))
results = evaluator.evaluate_robustness_from_path("model.pt", "pytorch", num_samples=100)

Test agent resilience:

from auto_art.core.evaluation.attacks.agentic import AdvWebDOMAttack
result = AdvWebDOMAttack(max_injections=5).execute_agentic(agent=my_agent, environment=my_env)

One-command YAML evaluation:

from auto_art.core.orchestrator import Orchestrator
report = Orchestrator.from_yaml("eval_config.yaml").run(agent=my_agent)
report.to_markdown()  # Human-readable report
report.to_sarif()     # SARIF 2.1.0 for CI/CD

Adaptive attack selection:

from auto_art.core.adaptive import AdaptiveAttackSelector
selector = AdaptiveAttackSelector(budget_seconds=3600)
attacks = selector.select_attacks(model_arch="resnet50", max_attacks=10)

OWASP Coverage

90% coverage of OWASP LLM Top 10 (2025) -- 8 categories full, 2 partial. See detailed mapping.

Documentation

Full documentation lives in docs/:

Section	Contents
Getting Started	Installation, quick start, first evaluation
Architecture	System design, layers, data flow
Attacks	All 43 attacks with usage examples
Defences	All 28 defence modules with configuration
Orchestrator	YAML schema, 6 phases, SARIF output
Red Teaming	Continuous adversarial probing
OWASP Mapping	LLM Top 10 coverage tracking
CI/CD	Pipeline phases, SARIF integration, deployment gates
Telemetry	Agent tracing, loop detection
SOTA Roadmap	Gap analysis, research integration, future plans

Tests

pip install ".[dev]"
pytest tests/ -v              # 346+ tests

License

MIT

Acknowledgments

• Adversarial Robustness Toolbox (ART) by Trusted-AI / Linux Foundation AI & Data
• AutoRedTeamer for memory-guided adaptive attack selection
• IBM ARES for YAML orchestration patterns
• RobustBench for AutoAttack evaluation methodology
• OWASP LLM Top 10 and NIST AI 100-2 for threat taxonomy