Unified run.spec selection grammar

[patriciapampanelli]

Summary

Introduces a single selection grammar — run.spec — for choosing probes and buffs, replacing the fragmented --probes / --probe_tags / --buffs flags and their config keys. One transport-agnostic engine parses both the CLI string (--run_spec) and the config form (run: { spec: { include: [...], exclude: [...] } }) into the same internal Spec, then resolves it to canonical plugin paths.

Resolves #1744

Grammar

• Plugin paths: probes.<module>[.<Class>], buffs.<module>[.<Class>], probes.*
• Empty selection: none (or probes.none) selects no probes — an explicit empty selection, distinct from an unspecified spec (which defaults to probes.*)
• Filters: tier:<N> (inclusive threshold — tiers 1..N), tag:<prefix> (membership)
• Exclusion: any selector prefixed with - (e.g. -probes.dan.DanInTheWild, -tier:3)
• Default (no probe plugin-path selector) → all active probes (probes.*), injected only at resolve time.

What changed

• New module garak/_spec.py — Selector / Spec / Resolution + parsing and resolution. _resolve_plugin_paths is the single shared resolution core. A none selector kind represents an explicit empty selection.

• garak/cli.py — adds --run_spec (underscore, consistent with garak's other flags such as --probe_tags / --list_probes); --probes/--probe_tags/--buffs are deprecated and map onto run.spec (with --run_spec winning on conflict).

• garak/command.py — print_buffs() gains an optional selected_buffs argument so --list_buffs matching --list_probes / --list_detectors. Previously the buff listing ignored any selection and always listed every buff; called without the argument, behaviour is unchanged (lists all).

• garak/_config.py — run.spec default; legacy config keys (plugins.probe_spec, plugins.buff_spec, run.probe_tags) map onto run.spec post-merge (mirroring model_type → target_type).

• Report snapshot now carries run.spec instead of the legacy keys.

• Downstream consumers (report_digest.py, aggregate_reports.py) read run.spec, with a fallback for reports predating this change.

• Bundled configs (bag.yaml, fast.json) migrated to run.spec.

• Docs updated (configurable.rst, usage.rst, faster.rst, new _spec.rst, cliref.rst regenerated from --help).

Selection of "none" vs unspecified (standardized)

A single rule governs empty vs default selection, consistently across CLI and config:

• none (e.g. --probes none or probe_spec: none) → explicit empty selection (probes.none); resolves to no probes. A run with no probes is a deliberate no-op ("No probes, nothing to do").
• vacuous values (omitted, empty, or auto) → unspecified; resolves to all active probes (probes.*).

This restores the legacy meaning of none (which selected nothing) while keeping the unified grammar's implicit default. The only behavior change vs the legacy flags is that --probes "" / --probes auto now default to all (they previously selected nothing); these are degenerate inputs and the new behavior is the more intuitive one.

Deprecation

--probes / --probe_tags / --buffs and plugins.probe_spec / plugins.buff_spec / run.probe_tags are deprecated (emit a deprecation notice). Detectors are out of scope here and keep their own plugins.detector_spec surface, to be migrated at #1789.

Examples

CLI

# all active probes (default — no selector needed)
garak -t test.Blank
# a whole probe family
garak -t test.Blank --run_spec "probes.dan"
# a single probe class
garak -t test.Blank --run_spec "probes.dan.DanInTheWild"
# family minus one class
garak -t test.Blank --run_spec "probes.dan, -probes.dan.DanInTheWild"
# tier threshold (tiers 1..2) filtered by tag, with one exclusion
garak -t test.Blank --run_spec "tier:2, tag:owasp:llm01, -probes.dan.AutoDAN"
# probes + a buff
garak -t test.Blank --run_spec "probes.dan, buffs.encoding"
# explicit empty selection (no probes)
garak -t test.Blank --run_spec "probes.none"

Config (YAML)

run:
  spec:
    include:
      - probes.dan
      - tag:owasp:llm01
      - buffs.encoding
    exclude:
      - probes.dan.DanInTheWild

Manual — selection resolution

• all active probes (no selector): garak --list_probes
• probe family: garak --list_probes --run_spec "probes.dan"
• explicit empty selection: garak --list_probes --run_spec "probes.none" → 0 probes (default still lists all)
• single class: garak --list_probes --run_spec "probes.dan.DanInTheWild"
• family minus one class: garak --list_probes --run_spec "probes.dan, -probes.dan.DanInTheWild"
• tier threshold (1..1): garak --list_probes --run_spec "tier:1"
• negative tier excludes exactly that tier (1..3 then drop tier 2 → {1,3}): garak --list_probes --run_spec "tier:3, -tier:2"
• tag filter: garak --list_probes --run_spec "tag:owasp:llm01"
• buff selection: garak --list_buffs --run_spec "buffs.encoding"

Manual — deprecation, conflict, errors

• --probes none is a deliberate no-op: garak -t test.Blank --probes none → "No probes, nothing to do" (does not run all probes)
• --probes deprecation maps to run.spec: garak --list_probes -p dan
• --probe_tags deprecation maps to tag:: garak --list_probes --probe_tags owasp:llm01
• --buffs deprecation maps to buffs.: garak --list_buffs -b encoding
• config probe_spec: none selects nothing (parity with CLI); auto/empty default to all
• --run_spec wins over legacy flag: garak --list_probes --run_spec "probes.dan" -p atkgen
• friendly error + exit 1 on invalid spec: garak --list_probes --run_spec "detectors.foo"
• aborts with empty-result reason: garak --list_probes --run_spec "probes.dan, -probes.dan"

[jmartin-tech]

Initial comments, testing and usage flow are still underway.

[jmartin-tech]

While I understand why this is here I think there may be some odd coupling created by this.

[jmartin-tech]

Is there a more generic way to organize this? Having probes and buffs attributes makes this a bit tightly coupled to the current supported plugin types.

[jmartin-tech]

Consider adding a fixer that can reformat a provided configuration into the new format.

[jmartin-tech]

While thinking on testing methods for this, I find myself thinking it would be nice to have a clean syntax to select all active probes plus a set of specific inactive ones. Something like:

run:
  spec:
    include:
      - probes.all
      - probes.fitd.FITD

Would it makes sense to have a reserved probes.all indicator for probes?

[patriciapampanelli]

Consider adding a fixer that can reformat a provided configuration into the new format.

The legacy keys are already converted to spec internally at load, and the report.jsonl persists the new spec grammar. Is your fixer suggestion specifically about surfacing this through --fix so users can convert their config to the new format? @jmartin-tech

[jmartin-tech]

Yes a fixer is meant to offer a method of converting an old configuration to the new format, this provides a direct way for users who are getting a deprecation notice to convert a config to the new format using the --fix cli option.

[jmartin-tech]

Some edge cases found in testing and a functionality that needs to be retained.

[jmartin-tech]

Recently added detection for probe families that are fully inactive is being lost here. The revisions need to account for flagging inactive_only_modules in a provided spec in both the new and old formats.