feat(cli): unified run.spec plugin selection grammar; deprecate --probes/--probe_tags/--buffs

Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>

Author: patriciapampanelli

docs/source/_spec.rst

@@ -0,0 +1,33 @@
+run.spec selection grammar
+===========================
+
+
+``garak/_spec.py`` implements the unified ``run.spec`` selection grammar: a
+single internal ``Spec`` (a list of ``Selector`` with explicit polarity) that
+both transports parse to.
+
+* CLI string (comma separated), via ``parse_spec_string``
+* config file form (YAML/JSON ``include``/``exclude`` lists), via ``parse_spec_file``
+
+Selectors carry a category-prefixed plugin path (``probes.<module>[.<Class>]``,
+``buffs.<module>[.<Class>]``) or a probe filter (``tag:<prefix>``,
+``tier:<N|name>``). A leading ``-`` excludes; ``tier:N`` is inclusive
+("log level": tiers ``1..N``). ``Spec.resolve()`` orchestrates probe and buff
+selection plus the filters; the single plugin-path resolution core,
+``_resolve_plugin_paths``, is shared with the ``parse_plugin_spec`` adapter used
+for detectors.
+
+See :doc:`configurable` for the user-facing grammar and examples.
+
+
+Code
+^^^^
+
+
+garak._spec
+-----------
+
+.. automodule:: garak._spec
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/cliref.rst

@@ -3,7 +3,7 @@ CLI reference for garak
 
 ::
 
-  garak LLM vulnerability scanner v0.15.1.pre1 ( https://github.com/NVIDIA/garak ) at 2026-05-01T13:01:25.835551
+  garak LLM vulnerability scanner v0.15.1.pre1 ( https://github.com/NVIDIA/garak ) at 2026-06-02T11:24:39.583036
   usage: python -m garak [-h] [--verbose] [--report_prefix REPORT_PREFIX]
                          [--narrow_output]
                          [--parallel_requests PARALLEL_REQUESTS]
@@ -12,15 +12,15 @@ CLI reference for garak
                          [--eval_threshold EVAL_THRESHOLD]
                          [--generations GENERATIONS] [--config CONFIG]
                          [--target_type TARGET_TYPE] [--target_name TARGET_NAME]
-                         [--probes PROBES] [--probe_tags PROBE_TAGS]
-                         [--detectors DETECTORS] [--extended_detectors]
-                         [--buffs BUFFS]
+                         [--run-spec RUN_SPEC] [--probes PROBES]
+                         [--probe_tags PROBE_TAGS] [--detectors DETECTORS]
+                         [--extended_detectors] [--buffs BUFFS]
                          [--buff_option_file BUFF_OPTION_FILE | --buff_options BUFF_OPTIONS]
                          [--detector_option_file DETECTOR_OPTION_FILE | --detector_options DETECTOR_OPTIONS]
                          [--generator_option_file GENERATOR_OPTION_FILE | --generator_options GENERATOR_OPTIONS]
                          [--harness_option_file HARNESS_OPTION_FILE | --harness_options HARNESS_OPTIONS]
                          [--probe_option_file PROBE_OPTION_FILE | --probe_options PROBE_OPTIONS]
-                         [--taxonomy TAXONOMY]
+                         [--intents INTENTS] [--taxonomy TAXONOMY]
                          [--confidence_interval_method {bootstrap,none}]
                          [--bootstrap_num_iterations BOOTSTRAP_NUM_ITERATIONS]
                          [--bootstrap_confidence_level BOOTSTRAP_CONFIDENCE_LEVEL]
@@ -29,9 +29,9 @@ CLI reference for garak
                          [--list_detectors] [--list_generators] [--list_buffs]
                          [--list_config] [--version] [--report REPORT]
                          [--interactive] [--fix]
-  
+
   LLM safety & security scanning tool
-  
+
   options:
     -h, --help            show this help message and exit
     --verbose, -v         add one or more times to increase verbosity of output
@@ -60,20 +60,28 @@ CLI reference for garak
     --target_name TARGET_NAME, --model_name TARGET_NAME, -n TARGET_NAME
                           name of the target, e.g.
                           'timdettmers/guanaco-33b-merged'
+    --run-spec RUN_SPEC, --run_spec RUN_SPEC
+                          unified selection spec, e.g. 'probes.dan,
+                          -dan.DanInTheWild, tag:owasp:llm01'. Selectors:
+                          probes.<module>[.<Class>], buffs.<module>[.<Class>],
+                          tag:<prefix>, tier:<N|name>; '-' excludes, tier:N is
+                          inclusive (tiers 1..N).
     --probes PROBES, -p PROBES
-                          list of probe names to use, or 'all' for all
-                          (default).
+                          DEPRECATED, use --run-spec. list of probe names to
+                          use, or 'all'.
     --probe_tags PROBE_TAGS
-                          only include probes with a tag that starts with this
-                          value (e.g. owasp:llm01)
+                          DEPRECATED, use --run-spec 'tag:<value>'. only include
+                          probes with a tag starting with this value (e.g.
+                          owasp:llm01)
     --detectors DETECTORS, -d DETECTORS
                           list of detectors to use, or 'all' for all. Default is
                           to use the probe's suggestion.
     --extended_detectors  If detectors aren't specified on the command line,
                           should we run all detectors? (default is just the
                           primary detector, if given, else everything)
     --buffs BUFFS, -b BUFFS
-                          list of buffs to use. Default is none
+                          DEPRECATED, use --run-spec 'buffs.<name>'. list of
+                          buffs to use. Default is none
     --buff_option_file BUFF_OPTION_FILE, -B BUFF_OPTION_FILE
                           path to JSON file containing options to pass to buff
     --buff_options BUFF_OPTIONS
@@ -97,6 +105,9 @@ CLI reference for garak
                           path to JSON file containing options to pass to probe
     --probe_options PROBE_OPTIONS
                           options to pass to probe, formatted as a JSON dict
+    --intents INTENTS, -i INTENTS
+                          comma-separated list of intents & intent prefixes to
+                          use. Default is empty, for all
     --taxonomy TAXONOMY   specify a MISP top-level taxonomy to be used for
                           grouping probes in reporting. e.g. 'avid-effect',
                           'owasp'
@@ -134,5 +145,5 @@ CLI reference for garak
     --fix                 Update provided configuration with fixer migrations;
                           requires one of --config / --*_option_file, /
                           --*_options
-  
+
   See https://github.com/NVIDIA/garak

docs/source/configurable.rst

@@ -57,17 +57,20 @@ Let's take a look at the core config.
         deprefix: true
         eval_threshold: 0.5
         generations: 5
-        probe_tags:
+        spec:
+            include:
+                - probes.dan
+                - tag: owasp:llm01
+            exclude:
+                - probes.dan.DanInTheWild
         user_agent: "garak/{version} (LLM vulnerability scanner https://garak.ai)"
         soft_probe_prompt_cap: 256
 
     plugins:
         target_type:
         target_name:
-        probe_spec: all
         detector_spec: auto
         extended_detectors: false
-        buff_spec:
         buffs_include_original_prompt: false
         buff_max:
         detectors: {}
@@ -123,7 +126,7 @@ Run Config Items
 """"""""""""""""
 
 * ``system_prompt`` -- If given and not overriden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality.
-* ``probe_tags`` - If given, the probe selection is filtered according to these tags; probes that don't match the tags are not selected
+* ``spec`` - The unified selection spec for probes and buffs (``run.spec``); see "Selecting probes and buffs with run.spec" below. If absent, the default is all active probes (``probes.*``)
 * ``generations`` - How many times to send each prompt for inference
 * ``deprefix`` - Remove the prompt from the start of the output (some models return the prompt as part of their output)
 * ``seed`` - An optional random seed
@@ -138,11 +141,15 @@ Plugins Config Items
 
 * ``target_type`` - The type of target generator, e.g. "nim" or "huggingface"
 * ``target_name`` - The specific name of the target to be used (optional - if blank, type-specific default is used)
-* ``probe_spec`` - A comma-separated list of probe modules or probe classnames (in ``module.classname``) format to be used. If a module is given, only ``active`` plugin in that module are chosen, this is equivalent to passing `-p` to the CLI
 * ``detector_spec`` - An optional spec of detectors to be used, if overriding those recommended in probes. Specifying ``detector_spec`` means the ``pxd`` harness will be used. This is equivalent to passing `-d` to the CLI
 * ``extended_detectors`` - Should just the primary detector be used per probe, or should the extended detectors also be run? The former is fast, the latter thorough.
-* ``buff_spec`` - Comma-separated list of buffs and buff modules to use; same format as ``probe_spec``.
 * ``buffs_include_original_prompt`` - When buffing, should the original pre-buff prompt still be included in those posed to the model?
+
+.. note::
+   ``plugins.probe_spec``, ``plugins.buff_spec`` and ``run.probe_tags`` are
+   **deprecated**. They still work (and are mapped onto ``run.spec`` with a
+   deprecation notice) but will be removed in a future release; use
+   ``run.spec`` instead (see below).
 * ``buff_max`` - Upper bound on how many items a buff should return
 * ``detectors`` - Root node for detector plugin configs
 * ``generators`` - Root note for generator plugin configs
@@ -153,6 +160,57 @@ Plugins Config Items
 For an example of how to use the ``detectors``, ``generators``, ``buffs``,
 ``harnesses``, and ``probes`` root entries, see :ref:`Configuring plugins with YAML <config_with_yaml>` below.
 
+Selecting probes and buffs with run.spec
+""""""""""""""""""""""""""""""""""""""""
+
+``run.spec`` is the single source of truth for selecting probes and buffs. It
+has two transports that parse to the same internal spec: a CLI string
+(``--run-spec``) and the config-file form (``include`` / ``exclude`` lists).
+
+Selectors (a category prefix is mandatory):
+
+* ``probes.*`` - all active probes (the default when no ``run.spec`` is given)
+* ``probes.<module>`` - an active family; ``probes.<module>.<Class>`` - one class
+* ``buffs.<module>[.<Class>]`` - selects buffs (no buffs are run by default)
+* ``tag:<prefix>`` - filters probes by tag (e.g. ``tag:owasp:llm01``)
+* ``tier:<N|name>`` - filters probes by tier; **inclusive** ("log level"): ``tier:N``
+  admits tiers ``1..N`` (``tier:1`` is the most critical). Names work too
+  (``tier:of_concern`` == ``tier:1``).
+
+Polarity: a bare selector (or ``+``) includes; a leading ``-`` removes. Note
+the asymmetry of ``tier``: ``tier:N`` is the inclusive filter, while ``-tier:N``
+removes *exactly* tier ``N``. Resolution applies excludes last (exclude wins),
+and if a spec resolves to no probes garak aborts with an actionable message.
+``tier:`` and ``tag:`` filters apply to the whole candidate set, including
+explicitly-named classes, so e.g. ``probes.foo.Bar, tier:1`` yields nothing when
+``foo.Bar`` is tier 3.
+
+.. code-block:: bash
+
+    # whole family minus one class
+    garak --run-spec "probes.dan, -probes.dan.DanInTheWild"
+    # family filtered by tag
+    garak --run-spec "probes.grandma, tag:owasp:llm06"
+    # all active buffs except one, over all active probes
+    garak --run-spec "probes.*, buffs.*, -buffs.paraphrase"
+    # tiers {1,3}: tier:3 admits 1..3, then -tier:2 removes exactly tier 2
+    garak --run-spec "+probes.*, +tier:3, -tier:2"
+
+.. code-block:: yaml
+
+    run:
+      spec:
+        include:
+          - probes.dan
+          - tag: owasp:llm01
+        exclude:
+          - probes.dan.DanInTheWild
+
+The deprecated ``--probes`` / ``--probe_tags`` / ``--buffs`` flags (and the
+``plugins.probe_spec`` / ``plugins.buff_spec`` / ``run.probe_tags`` config keys)
+are mapped onto ``run.spec`` with a deprecation notice; ``--run-spec`` wins if
+both are given.
+
 Reporting Config Items
 """"""""""""""""""""""
 
@@ -231,9 +289,9 @@ probes and run each prompt just once:
     ---
     run:
         generations: 1
-
-    plugins:
-        probe_spec: latentinjection
+        spec:
+            include:
+                - probes.latentinjection
 
 If we save this as ``latent1.yaml`` somewhere, then we can use it with ``garak --config latent1.yaml``.
 Note: YAML configs require the explicit ``.yaml`` or ``.yml`` extension (case-insensitive).
@@ -244,10 +302,11 @@ Note: YAML configs require the explicit ``.yaml`` or ``.yml`` extension (case-in
 
     {
       "run": {
-        "generations": 1
-      },
-      "plugins": {
-        "probe_spec": "latentinjection"
+        "generations": 1,
+        "spec": {
+          "include": ["probes.latentinjection"],
+          "exclude": []
+        }
       }
     }
 

docs/source/faster.rst

@@ -104,7 +104,7 @@ You can get help by running ``python -m garak.analyze.aggregate_reports``.
 Probe aggregation
 ^^^^^^^^^^^^^^^^^
 
-One way of achieving parallel probing is by splitting garak probing up into many jobs each with one probe given in ``plugins.probe_spec``.
+One way of achieving parallel probing is by splitting garak probing up into many jobs each selecting one probe via ``run.spec`` (e.g. ``--run-spec probes.dan.AutoDANCached``).
 Each job should write to a distinct report file.
 When complete, the resulting report JSONL files can be aggregated into one using the ``aggregate_reports`` tool.
 

docs/source/index.rst

@@ -69,6 +69,7 @@ Check out the :doc:`usage` section for further information, including :doc:`inst
    intents
    payloads
    _plugins
+   _spec
    cas
    report
    services

docs/source/usage.rst

@@ -15,8 +15,8 @@ You can list the probes by running the following command:
 
 You can limit the probes to run by specifying more arguments.
 
-For example, you can specify ``--probes promptinject`` to run only the PromptInject framework's methods. for example.
-You can also specify specific probes instead of a probe family such as ``--probes lmrc.SlurUsage`` to probe a model for generating slurs based on the Language Model Risk Cards framework.
+For example, you can specify ``--run-spec probes.promptinject`` to run only the PromptInject framework's methods.
+You can also specify specific probes instead of a probe family such as ``--run-spec probes.lmrc.SlurUsage`` to probe a model for generating slurs based on the Language Model Risk Cards framework.
 
 Running a Scan
 --------------
@@ -37,11 +37,11 @@ Probe a commercial model for encoding-based prompt injection (OSX/\*nix) (replac
 .. code-block:: console
 
     export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
-    garak --target_type openai --target_name gpt-5-nano --probes encoding
+    garak --target_type openai --target_name gpt-5-nano --run-spec probes.encoding
 
 
 Determine if the Hugging Face version of GPT2 is vulnerable to DAN 11.0:
 
 .. code-block:: console
 
-    garak --target_type huggingface --target_name gpt2 --probes dan.Dan_11_0
+    garak --target_type huggingface --target_name gpt2 --run-spec probes.dan.Dan_11_0