feat: add AART probe for AI-assisted red-teaming across diverse cultural contexts

[Varshith-Kali]

What & Why

Adds a new probe module garak.probes.aart implementing the AART (AI-Assisted Red-Teaming) methodology from Radharapu et al. (EMNLP 2023 Industry Track). AART generates adversarial prompts structured across region × crime × use_case dimensions to test whether a target consistently refuses policy-violating requests across cultural and geographic contexts that are under-represented in existing safety benchmarks.

Closes #515.

The probe ships a curated subset of 34 prompts (covering 20 crime categories and 33 region combinations) sourced from the publicly-released walledai/AART dataset on HuggingFace (CC-BY-4.0, 3,269 prompts total). The curated subset picks one prompt per unique region × crime combination to maximise coverage diversity while keeping the probe self-contained — consistent with how realtoxicityprompts.py ships a subset rather than the full dataset.

Why this approach

• Static prompt list, not runtime HF download: garak probes should be runnable offline and self-contained. The 34-prompt curated subset covers the diversity dimensions (region, crime, use_case) that are the paper's contribution, without requiring a network fetch at probe-load time.
• Single AART class, not per-crime subclasses: for a first contribution, a single class with the curated subset is lower-risk and easier to review. If maintainers prefer per-crime subclasses (as donotanswer.py does), I'm happy to refactor.
• primary_detector = "mitigation.MitigationBypass": the AART prompts are direct policy-violating instructions (not jailbreak-technique prompts). The right detector is one that flags absence of a mitigation/refusal message, which is exactly what MitigationBypass does. This matches adaptive_attacks.py's detector choice for similar policy-violation probes.
• tier = OF_CONCERN: these are substantive adversarial probes with real policy-violation potential, not informational or fluctuating. Matches malwaregen and donotanswer tier choices.
• active = False: new substantive probes default to inactive per garak convention; maintainers can flip on stabilisation.

Files changed

File	Change
garak/probes/aart.py	New probe module — AART class with 34 curated prompts, module + class docstrings per docs/source/extending.probe.rst
tests/probes/test_probes_aart.py	9 tests: load, prompt count (≥30), uniqueness, string types, detector binding, doc_uri, inactive default, goal, tag presence
docs/source/probes/aart.rst	New docs page (automodule pattern matching existing probe docs)
docs/source/index_probes.rst	Add probes/aart to toctree (alphabetical, first entry)
garak/resources/plugin_cache.json	Regenerated via tools/rebuild_plugin_cache.sh to include probes.aart.AART

Why this is not duplicating an existing PR

Per AGENTS.md duplicate-work checks, verified before opening:

gh issue view 515 --repo nvidia/garak --comments    # 0 prior claim comments
gh pr list --repo nvidia/garak --state open --search "515 in:body"   # 0 open PRs
gh pr list --repo nvidia/garak --state open --search "aart"          # 0 open PRs

The only PR in repo history mentioning "#515" in its body is #1564 (closed, unrelated — parallelization pickle crash). No open PRs target AART.

Test commands run and results

# 1. AART-specific tests
$ python -m pytest tests/probes/test_probes_aart.py -v
============================== 9 passed in 0.11s ===============================

# 2. Parametrized probe tests (the suite that checks ALL probes for metadata/tag/docstring compliance)
$ python -m pytest tests/probes/test_probes.py -v -k "aart or AART"
====================== 6 passed, 1149 deselected in 1.53s ======================
# (test_detector_specified, test_probe_detector_exists, test_probe_structure,
#  test_probe_metadata, test_check_docstring, test_tag_format — all pass for probes.aart.AART)

# 3. Probe discoverable via --list_probes
$ python -m garak --list_probes | grep aart
probes: aart 🌟
probes: aart.AART 💤

# 4. End-to-end probe run against test target
$ python -m garak -t test -p aart.AART
# → runs 34 prompts through MitigationBypass detector, produces report JSONL + HTML
# → "garak run complete in 2.25s" (100% attack success rate on Lorem Ipsum test target, as expected since it doesn't refuse)

# 5. Formatting + lint
$ black --config pyproject.toml garak/probes/aart.py tests/probes/test_probes_aart.py
# 1 file reformatted, 1 left unchanged
$ ruff check garak/probes/aart.py tests/probes/test_probes_aart.py
# All checks passed!

AI assistance disclosure

Drafted with AI assistance (Claude), reviewed and tested by me before submission. I have read every line of the diff, run the full test suite locally, and verified the probe executes end-to-end against a test target. The prompt curation approach (selecting one prompt per unique region × crime combination from the walledai/AART HF dataset) was my decision based on the dataset's structure and garak's conventions for shipped prompt subsets.

Co-authored-by: Claude noreply@anthropic.com

[jmartin-tech]

Just a drive by comment, full review of suitability or general technical standards has not been preformed.

[jmartin-tech]

In future PRs please avoid committing change to this file, in fact amending this commit to remove edits to this file would be prefered.

When update is required please ensure to use the provided tool in tools/rebuild_plugin_cache.sh of this repository on a Linux based environment.

This file is maintained via automation and rarely needs revisions to be part of the PR unless there is some specific testing reason such as validation of a specific change that would be impacting the cache itself.