v0.16.0

What's Changed

New features

• Feature: technique and intent annotation and initial IntentProbe iteration by garak-maintainers in #1984

  "Power is in tearing attacks to pieces and putting them together again in shapes of your own choosing."

  This initial feature represents the first step in enabling users to provided their own context and requirements for target expectations and enable identification of attack vectors that show the edges of the underlying systems safeguards and runtime posture from new perspectives. This feature includes the initial groundwork for bringing user context into account during evaluation of a target. In its first iteration, it provides a new facet of information while keeping the existing probes primarily unchanged. The trait and intent concepts are explored, with policy introduced only as a reference definition. Further iteration and community feedback will guide how these new concepts are consumed, and influence how the broader Context Aware Scanning feature evolves.

  • arch: support & data for trait/intent, and for policy datatype by @leondz in #1421
  • cas feature: intent service by @leondz in #1481
  • cas: technique & intent minimal pilot by @leondz in #1598
  • cas: add technique tags to existing probes by @leondz in #1691
  • avoid stored probename side-effects in evaluation by @jmartin-tech in #1788
  • cas: intents for integrated probes by @leondz in #1693
  • Fix IntentProbe crash when intent has no detector mapping by @patriciapampanelli in #1798
  • cas: Add eval-level intents and digest technique/intent matrix by @patriciapampanelli in #1807
  • Unified run.spec selection grammar by @patriciapampanelli in #1831
  • feat(cas): expand intent->detector mapping coverage by @patriciapampanelli in #1861
  • probes.base: balance IntentProbe pruning across intents by @patriciapampanelli in #1838
  • Feature/intent run spec selector by @patriciapampanelli in #1866
  • report: enrich technique_intent_matrix with names and descriptions by @patriciapampanelli in #1890
  • fix: drop name fallback for stub-less intents by @patriciapampanelli in #1876
  • cas: donotanswer intents by @leondz in #1716
  • docs: cas user dev guide for technique and intent by @patriciapampanelli in #1874
  • Refactor/cas config to run by @patriciapampanelli in #1886
  • add probe summary entries and augment digest based on them by @jmartin-tech in #1932
  • Feat: report technique and intent tabs lists by @otavionvidia in #1879
  • double escape for 'dot' literal match in regex by @jmartin-tech in #1989

New plugins

• generator: add native Anthropic generator by @NishchayMahor in #1809
• probe: add simple adaptive attacks probe by @neerazz in #1742

Breaking changes

This release introduces breaking changes:

• introduces a new unified spec syntax to enable activation and suppression of probes, buffs and newly added intents
• cli plugin selection options (backwards compatible with existing options now marked as deprecated)
• configuration file formats:
  • probe_spec and buff_spec move into the run definition as part of a unified spec
• report.jsonl output structure changes:
  • new entry types "probe_summary"
  • expanded "digest" format
  • removed "digest" values for older *_spec configuration no longer consumed

Improved plugins

• fix: catch OpenAI AuthenticationError before multiprocessing pickle by @DevamShah in #1859
• Enhance error handling for Mistral API responses by @jka236 in #1769
• feat: add suppressed_params support to BedrockGenerator by @u7k4rs6 in #1842
• fix: repair dead blank-prompt guard in NVCF 400 handling by @anxkhn in #1901
• Fix: continuation probe not pruning triggers with prompts by @Osamaali313 in #1976
• Guard nvcf 500 error parsing against non-JSON bodies by @chuenchen309 in #1945
• escape regex metachars in StringDetector word matching by @bereketDeneke in #1880
• Fix: improve extra_params support in NeMoGuardrailsServer by @chuenchen309 in #1947
• feat(detectors): add Unicode normalisation to StringDetector by @PhinehasNarh in #1884
• Handle non-list LangServe /invoke output in langchain_serve generator by @WatchTree-19 in #1916
• web_injection: harden markdown exfil detectors against domain/extension/param evasion by @WatchTree-19 in #1928
• fix(rest): clear error when response_json_field matches a non-text response value by @xr843 in #1892

Documentation

See "Feature: technique and intent" references

Tuning & fixes

• test: reduce test-suite runtime by @patriciapampanelli in #1854
• test(detectors): mock data to manage avoid HF Hub 5xx instead of failing by @JakeBx in #1833
• test: add unit tests for glitch, snowball, and ansiescape probe modules by @nuthalapativarun in #1755
• fix(probes): download wordnet lexicon when it is missing from the database by @adityasingh2400 in #1820
• fix calibration pathlib paths by @ron1po in #1847
• Resolve symlinked install paths in data_path escape protections by @ABeltramo in #1860
• test(detectors): add unit tests for snowball detector module by @nuthalapativarun in #1824
• test(probes): un-shadow leakreplay probe structure test by @anxkhn in #1914
• tests: cover a completed parallel_attempts run by @EphraiemSarabamoun in #1816
• fix(analyze): guard analyze_log against zero total_evaluated by @anxkhn in #1941
• fix: reverse_translation_outputs reassembled in wrong order via list.pop() by @chuenchen309 in #1959
• test(config): assert aiohttp User-Agent test actually verifies the request by @anxkhn in #1933
• fix(probes): correct inverted target_lang_name_en guard message by @anxkhn in #1934
• fix(cli): report file path in plugin option-file errors by @anxkhn in #1951
• Fix: image support in OpenAICompatible._conversation_to_list by @chuenchen309 in #1944
• UX: Show probe name in detector progress bar by @feiiiiii5 in #1942
• fix: close prior httpx client before reload to stop FD leak in long fitd runs by @mvanhorn in #1787
• task: update npm and python deps by @jmartin-tech in #2007
• lock nltk due to incompatible path restrictions by @jmartin-tech in #2017
• fix: default config_root to _config in InjectAtbash by @anugram in #2006
• fix(detectors): honour config_root in goodside.RileyIsnt by @manunicholasjacob in #2014

New Contributors

• @NishchayMahor made their first contribution in #1809
• @nuthalapativarun made their first contribution in #1755
• @ron1po made their first contribution in #1847
• @DevamShah made their first contribution in #1859
• @ABeltramo made their first contribution in #1860
• @jka236 made their first contribution in #1769
• @neerazz made their first contribution in #1742
• @anxkhn made their first contribution in #1901
• @u7k4rs6 made their first contribution in #1842
• @EphraiemSarabamoun made their first contribution in #1816
• @chuenchen309 made their first contribution in #1959
• @Osamaali313 made their first contribution in #1976
• @bereketDeneke made their first contribution in #1880
• @PhinehasNarh made their first contribution in #1884
• @WatchTree-19 made their first contribution in #1916
• @feiiiiii5 made their first contribution in #1942
• @mvanhorn made their first contribution in #1787
• @xr843 made their first contribution in #1892
• @anugram made their first contribution in #2006
• @manunicholasjacob made their first contribution in #2014

Full Changelog: v0.15.1...v0.16.0