v0.11.0

What's Changed

New plugins

• Template / SQL injection probes by @erickgalinkin in #1138
• Feature/add mistral generator by @dimensi0n in #1135

New features

• feature: update default toxicity detector by @leondz in #1106
• feature: lightweight probe defaults by @leondz in #1116
• feature: max_workers / give kinda helpful message if too many open files by @leondz in #1110
• Multiprocess enabled logging config by @jmartin-tech in #1140
• Feature: multilingual machine translation by @SnowMasaya in #943
• Support stripping until end think token given empty skip_seq_start in config by @aishwaryap in #1185
• update: add probe tiers by @leondz in #1151
• update: promptinject detector now accepts multiple triggers by @leondz in #1148
• update: rename atkgen probe model to be clear about toxicity by @leondz in #1149
• update: remove ambiguous terms from slur_terms_en payload by @leondz in #1150
• reporting: update report aggregation funcs by @leondz in #1156
• script: qualitative review output by @leondz in #1144
• Add -no-cnv flag support to ggml generators by @IanYHChu in #1189
• reporting: add option for no group score by @leondz in #1194
• reporting: aggregate probe as min by @leondz in #1218
• reporting: add defcon lozenges for relative & absolute scores by @leondz in #1216
• Update/refactor specialwords by @leondz in #1178
• reporting: smooth z-score wildness by @leondz in #1212
• Task: 2025 Q2 scoring calibration by @jmartin-tech in #1231 (thanks to Vijil.ai for data contributions)
• update calibration data for additional probes by @jmartin-tech in #1236
• reporting: change default aggregation by @leondz in #1234

Documentation

• Fix typo in README for leak replay probe by @arjun-krishna1 in #1142
• docs: split 'extending' docs out from 'contributing' by @leondz in #1146
• doc file class corrections by @jmartin-tech in #1200
• docs: formatting fixes by @leondz in #1215

Tuning & fixes

• clear pip cached files by @jmartin-tech in #1129
• set a default soft_probe_prompt_cap in _config by @jmartin-tech in #1133
• enhance response type support from local NeMo-Guardrails by @jmartin-tech in #1131
• bugfix: encoding detection generating false positives by @leondz in #1130
• update: unify on attempt.notes["triggers"] by @leondz in #1147
• Bump datasets version by @JanetVictorious in #1137
• make all workflow permissions explicit by @jmartin-tech in #1162
• update: add soft prompt caps to encoding probes by @leondz in #1154
• update: rename bcp47 to lang by @leondz in #1164
• one detection result per output when testing regex based matches in exploitation by @jmartin-tech in #1167
• Removed detector prefix from eval records by @mrowebot in #1157
• bugfix: HF Detector exceptions now handled gracefully by default by @leondz in #1170
• cache workflow resources by @jmartin-tech in #1173
• refactor probe tier as enum with value in plugin cache by @jmartin-tech in #1159
• update: more meaningful values in tier enums by @leondz in #1176
• block failing litellm 1.67.2 by @leondz in #1179
• ux: give more verbose message for CLI typos by @leondz in #1182
• refactor LatentInjection by @leondz in #1152
• cap litellm max version to avoid their windows bug by @leondz in #1186
• update: rename Translator -> LangProvider and associated elements by @leondz in #1183
• bugfix: reduce latent optimisation permutation explosion by @leondz in #1181
• replicate generator pickle support improvements by @jmartin-tech in #1190
• Fix ambiguous series value error when running --report by @marcorosa in #1171
• add arm64 runner to Linux testing by @jmartin-tech in #1196
• Testing: storage reduction by @jmartin-tech in #1204
• remove unused tooling to free space by @jmartin-tech in #1206
• update deps away from insecure versions by @leondz in #1207
• update Tier impl by @leondz in #1205
• config: sync probe active defaults with default config used in practice by @leondz in #1214
• update: revert default _config.run.generations to 5 by @leondz in #1227
• fix: stop atkgen turn count variation in test relying on fixed turn count by @leondz in #1226
• fix plugin cache tests by @emmanuel-ferdman in #1229
• ux: move translator load msg into translator instantiation by @leondz in #1184
• extract text when processing multi-modal prompts by @jmartin-tech in #1228

New Contributors

• @JanetVictorious made their first contribution in #1137
• @SnowMasaya made their first contribution in #943
• @dimensi0n made their first contribution in #1135
• @mrowebot made their first contribution in #1157
• @aishwaryap made their first contribution in #1185
• @marcorosa made their first contribution in #1171
• @IanYHChu made their first contribution in #1189

Full Changelog: v0.10.3.1...v0.11.0

v0.10.3.1

What's Changed

Tuning & fixes

• defensive coding around capture results in xss content detector by @jmartin-tech #1120

Full Changelog: v0.10.3...v0.10.3.1

v0.10.3

What's Changed

New plugins

• Added watsonx.ai generator by @iamnotcj in #1058
• Additional XSS Exfil Probes by @erickgalinkin in #1060

New features

• Generators: add option to skip output btw given delimiters; add hook for postprocessing by @leondz in #1097
• Feature: align hallucinated package named with outputs by @leondz in #1076
• support module.classname config specification for plugins by @jmartin-tech in #1108

Documentation

• pedantic spelling by @dltemple in #1085
• update contributing checklist numbers to be ascending by @shane-rosse in #1115

Tuning & fixes

• update: reorder ditw prompts by @leondz in #1098
• update openai model list by @leondz in #1100
• update: Give clearer message when header encoding fails by @leondz in #1088

What's Changed

• uninclude donotanswer from default probes by @leondz in #1111
• stop forcing generation count for mini phrasing probes by @leondz in #1109

New Contributors

• @iamnotcj made their first contribution in #1058
• @dltemple made their first contribution in #1085
• @shane-rosse made their first contribution in #1115

Full Changelog: v0.10.2...v.0.10.3

v0.10.2

What's Changed

New plugins

• Detector: Shields for testing LLM Application Firewalls by @Eric-Hacker in #1059

New features

• Warn if api_key in Config by @erickgalinkin in #1049
• Feature: configuration based rest proxy support by @jmartin-tech in #1073
• Feature: configuration based rest ssl suppression by @jmartin-tech in #1074
• Detector: add startswith matching for StringDetector base class by @leondz in #1075

Documentation

• docs: update index.rst by @eltociear in #1053
• docs: update readme by @leondz in #1050
• adjust label ref syntax by @jmartin-tech in #1057

Tuning & fixes

• StringDetector.case_sensitive now operates as expected by @leondz in #1072
• guard against None in leakreplay attempt history management by @leondz in #1081
• init zscore with enough scope to cover its uses by @leondz in #1086
• Update for latest ollama support by @jmartin-tech in #1092

New Contributors

• @eltociear made their first contribution in #1053
• @Eric-Hacker made their first contribution in #1059

Full Changelog: v0.10.1...v0.10.2

v0.10.1

What's Changed

New plugins

• probe: ANSI terminal takeover by @leondz in #1025
• probe: Past Tense Vulnerability by @Shine-afk in #924
• generator: promote OpenAICompatible as first class generator by @jmartin-tech in #1021

New features

• feature: amend garak user-agent by @leondz in #960
• feature: reinforce trust_remote_code=False by @leondz in #979
• generators: add option to specify HTTP codes to skip generation on, for RestGenerator by @leondz in #999
• feature: add experimental features flag to be only accessible in core config by @leondz in #1003
• feature: enable configuration updates by @jmartin-tech in #1026
• feature: use hf chat support by @jmartin-tech in #1047
• feature: per-probe tags now adjustable based on payload selection by @leondz in #1031

Documentation

• doc: Update README.md to notify of migration by @leondz in #1002
• docs: add arxiv lozenge by @leondz in #1030

Tuning & fixes

• remove no longer needed skip-duplicate-actions by @jmartin-tech in #976
• Sanity test for pip install from repository by @jmartin-tech in #1000
• add CODEOWNERS by @jmartin-tech in #1001
• Update reference to reflect migration of org by @jmartin-tech in #1004
• use allowed action name by @jmartin-tech in #1005
• Fixing a few typos by @zoenolan in #1006
• Fix huggingface inference endpoint name by @jmartin-tech in #1011
• detect if tokenizer is not loaded and adjust by @jmartin-tech in #1012
• fix: correct argument in TAP by @harshraj172 in #1022
• refactor attempt to utilized property annotations by @jmartin-tech in #1027
• Restrict transformers version until MPS issue is addressed by @jmartin-tech in #1039
• pass device to transfomers pipeline by @jmartin-tech in #1042
• Fix tap.PAIR run error - run_tap param evaluator_model by @rafaelsandroni in #1045

New Contributors

• @zoenolan made their first contribution in #1006
• @harshraj172 made their first contribution in #1022
• @Shine-afk made their first contribution in #924
• @rafaelsandroni made their first contribution in #1045

Full Changelog: v0.10.0...v0.10.1

v0.10.0

What's Changed

New plugins

• detector: llm as a judge by @jmartin-tech in #956
• generator: update openai for o1 models by @leondz in #922
• generator: Ollama by @martinebl in #876
• generator: support of azure openai by @eric-therond in #817
• generator: vision nims by @leondz in #959
• probe: add whois injection to latentinjection by @leondz in #947

New features

• payload/probe separation by @leondz in #870
• data file override support with precedence by @jmartin-tech in #916
• cli: --parallel_attempts hint by @leondz in #932
• reporting: tidy config details, add payload audit info by @leondz in #936
• payloads: web-related, incl. probes.encoding migration by @leondz in #933

Documentation

• Document configuration of garak using YAML by @leondz in #911
• RST formatting updates to restgenerator page by @leondz in #912
• doc: give generatiors.rest list the space it wants and deserves by @leondz in #914
• latent injection doc fix by @leondz in #921
• Contributing Probes documentation by @erickgalinkin in #919
• docs: add example of using Configurable to specify a generator name for load_plugin by @leondz in #929
• docs: NVIDIA generators by @leondz in #917
• update bag reference by @emmanuel-ferdman in #953
• docs: update faq by @leondz in #957
• docs: describe top-level objectives in reference guide by @leondz in #958
• docs: centralise faq by @leondz in #965
• docs: make project scope clear and note that contributors should check it by @leondz in #964
• doc: update to use rst toctree href syntax by @leondz in #967
• docs: clarify "contributing" document by @leondz in #968
• docs: require core config top- and second-level params to be documented by @leondz in #966

Tuning & fixes

• plugin classes should not access DEFAULT_PARAMS by @jmartin-tech in #906
• add test status lozenges for lin/win/osx by @leondz in #903
• utilized the cache to filter base classes by @jmartin-tech in #905
• paraphrase fast consistent model device by @jmartin-tech in #898
• Force cache build in automation by @jmartin-tech in #907
• tweak: defer import of slow Llava* classes by @leondz in #908
• add issue and PR templates by @jmartin-tech in #909
• tweak: new latentinjection probe, add configurability to mini version by @leondz in #910
• bug: set match type correctly in riskywords detectors by @leondz in #918
• bugfix: enable latent injection mini latent jailbreak probe by default by @leondz in #923
• tweak: update migitation strings by @leondz in #925
• bugfix: use distinct report entry type for payload init by @leondz in #930
• add code coverage options by @jmartin-tech in #934
• reporting: update link to bag by @leondz in #935
• probe: missing newline in latentinjection.LatentInjectionFactSnippetEiffel by @leondz in #938
• update: extend calibration with more probes by @leondz in #939
• limit project actions to primary repo by @jmartin-tech in #940
• Add error handling for empty node_results to base.py. by @erickgalinkin in #942
• update: rename replay plugins to divergence by @leondz in #945
• update hint to reference --parallel_attempts by @jmartin-tech in #951
• Configurable TAP probe, refactor judge resources to shared red team by @jmartin-tech in #949
• architecture: factor HFCompatible out by @leondz in #954
• generators: fix rasa issues #961 & #962 by @rgstephens in #963
• fix: add garak/data to pip package by @jmartin-tech in #969

New Contributors

• @martinebl made their first contribution in #876
• @emmanuel-ferdman made their first contribution in #953
• @eric-therond made their first contribution in #817

Full Changelog: v0.9.0.16...v0.10.0

v0.9.0.16

What's Changed

New plugins

• probe: topic pushing by @leondz in #764
• probe: ruby package hallucination by @arjun-krishna1 in #851
• probe: Latent prompt injection by @leondz in #877
• probe: npm package hallucination by @arjun-krishna1 in #861
• probe: rust crate hallucination by @arjun-krishna1 in #873
• generator: Groq API by @mmilenkovic-groq in #896

New features

• extract generations param to be set on probes instead of generators by @jmartin-tech in #837
• secure garak HF assets by @leondz in #854
• Rename knownbadsignatures to av_spam_scanning by @leondz in #850
• enable tokenizer customization in HFDetector by @jmartin-tech in #855
• refactor calibration / z-score code, so z-scores can be shown in CLI at run time by @leondz in #847

Tuning & fixes

• add automation action for plugin_cache.json by @jmartin-tech in #819
• add automation action for plugin_cache.json by @jmartin-tech in #819
• Docs/readmefaqdc32 by @leondz in #828
• fix initialization bug for kwargs in function by @soumilinandi in #827
• suppress LiteLLM logging during import by @jmartin-tech in #834
• relax --probe_tags as supported when other probe options are passed by @jmartin-tech in #836
• rename harness test module by @leondz in #848
• Surface underlying exception msg at plugin load by @leondz in #846
• relax litellm provider constraint by @arjun-krishna1 in #820
• plugin metadata tests by @leondz in #849
• Add explanation of how we treat the word "vulnerability" by @leondz in #852
• add faq items by @leondz in #857
• test buff transform() methods, to get them to load and run by @leondz in #856
• expand doc string that fails test requirements for verbosity by @jmartin-tech in #859
• refresh tutorial on contributing a generator by @leondz in #858
• avoid lambda in _config module level code by @jmartin-tech in #860
• add logging dir, new generator questions to FAQ by @leondz in #866
• revise default generations value to 5 by @leondz in #868
• validate detector doc_uri by @leondz in #865
• fix output error when parallel_requests > 1 and supports_multiple_generations is False by @arjun-krishna1 in #864
• improved messaging & error handling around NIM names by @leondz in #867
• look at the right var for NIM name checking by @leondz in #871
• filter detector template in tests by @jmartin-tech in #874
• have CLI UI render 100% w/o using scientific notation by @leondz in #882
• Generator docs are now tested for as intended by @leondz in #897
• reporting tweaks by @leondz in #901

New Contributors

• @soumilinandi made their first contribution in #827
• @arjun-krishna1 made their first contribution in #820
• @mmilenkovic-groq made their first contribution in #896
• @greshake made their first contribution in #877

Full Changelog: v0.9.0.15...v0.9.0.16

v0.9.0.15

What's Changed

New plugins

• probe: assess 🤗 repos for potentially malicious files (fileformats) by @leondz in #767

New features

• use XDG paths for configuration data and caching by @jmartin-tech in #799
• Report comparative info for detector scores by @leondz in #814
• enable HFDetector model configuration with hf_args by @jmartin-tech in #810
• Flags for varying NIM seed and temperature every call by @leondz in #808

Tuning & fixes

• don't skip listing probes in 100%-pass-rate modules by @leondz in #781
• make default regex raw to remove python 3.12 warnings by @jmartin-tech in #809
• suppress HF warnings and lower during detector model instantiation by @leondz in #807
• validate cache consistency on first access by @jmartin-tech in #815
• clarify continuation probes by @leondz in #813
• print logfile path when invoked from CLI by @leondz in #811
• update slurprompts filename in encoding pkg by @leondz in #818

Full Changelog: v0.9.0.14...v0.9.0.15

v0.9.0.14

What's Changed

New plugins

• BEAST Attack Implementation by @erickgalinkin in #728
• add probe for "grandma" attack by @leondz in #740
• generator: lorem ipsum by @leondz in #776

New features

• Feature: configurable plugins by @jmartin-tech in #711
• Add JSON Path support for RestGenerator by @leondz in #651
• Make banner line test dynamic by @jmartin-tech in #670
• Tutorial for contributing a generator by @leondz in #688
• add 'complete' message to report at end of run by @leondz in #735
• support chat modality in attempts by @leondz in #644
• add grandma substance synth probe, update docs and var names by @leondz in #763
• update NVCF to support versioning, custom payload keys by @leondz in #772
• WINDOWS & OSX SUPPORT: test by default on linux, osx, win by @leondz in #793
• enable nested setting of plugin configs by @leondz in #773
• always log nvcf payload by @leondz in #778
• update ecoji for windows support by @jmartin-tech in #787
• remove extraneous default params for nims that expect conservative pa… by @leondz in #749
• require probes to match input modality of the generator by @jmartin-tech in #738
• Improve cloze data for leakreplay probe by @leondz in #708
• add CONTRIBUTING.md, update contributing guide in docs by @leondz in #706
• add DEFAULT_PARAMS to base types that are configurables by @leondz in #771
• atkgen - add configurability, allow config of which adversarial generator to use by @leondz in #777
• enhancement: enable lazy setting of nested dicts by @leondz in #775
• convert mutation_generator to use load_plugin by @jmartin-tech in #795
• Add support for gpt-4o, gpt-4o-mini by @erickgalinkin in #798
• bump transformers ver to min reqd for llama 3.1 by @leondz in #800
• add mitigation strings, add test for string-based mitigation skipping by @leondz in #762
• speedup report generation by using class attribs instead of instantiating by @leondz in #707
• cached plugin enum by @jmartin-tech in #768

Tuning & fixes

• add some actionable opinions about linting by @leondz in #668
• handle target model returning None in atkgen by @leondz in #678
• bump & fix black version by @leondz in #686
• Bump/black 24.4.2 -- and apply formatting by @leondz in #687
• convert module-level constant names to uppercase by @leondz in #685
• Guard cli run with invalid or incomplete config by @jmartin-tech in #677
• Catch generic exception from transformers in huggingface by @jmartin-tech in #690
• stablize openai parallel by @jmartin-tech in #689
• rest generator response must be a list by @jmartin-tech in #693
• Set generator _call_model() and _generate() type hints; amend functions by @leondz in #694
• typo in racial slur prompt by @leondz in #697
• Generator streamlining, docs by @leondz in #682
• Split NVCF generator into completion chat by @leondz in #696
• Update the quality taxonomy by @leondz in #703
• generator & detector structure validation by @leondz in #702
• doc & inheritance fixes to DanInTheWildMini by @leondz in #712
• Remove support for openai api v0 by @leondz in #718
• restore req_template_json_object lost in refactor by @jmartin-tech in #729
• utilize GarakException in cli by @jmartin-tech in #730
• ensure report is printed in utf-8 for Windows by @jmartin-tech in #732
• Linting - 20240612 by @leondz in #733
• add garak preprint paper by @leondz in #739
• only set generations if still default by @jmartin-tech in #741
• return value from batch_decode is already a list by @jmartin-tech in #743
• Handle openai 400 errors more gracefully by @leondz in #744
• Refactor huggingface config support by @jmartin-tech in #742
• rename class level probe doc_uri by @jmartin-tech in #759
• retry openaicompatible requests if invalid content received by @leondz in #761
• add colour theme config by @leondz in #769
• reduce zalgo-ness by @leondz in #770
• move REST generator exception into garak.exceptions by @leondz in #779
• defensive coding: allow python generators more places by @leondz in #782
• stop encoding.payloads accumulating each load by @leondz in #780
• correct sorting for set json in plugin_cache.json by @jmartin-tech in #794
• corrections to config format for TAP internal generators by @jmartin-tech in #796
• add missing comma, extra test case in probes.test.Test by @leondz in #797
• Bugfix/leakreplay output setting by @leondz in #790

Full Changelog: v0.9.0.13...v0.9.0.14

v0.9.0.13

What's Changed

garak's 1st birthday! 🎂

Headlines in this birthday release:

• Multimodal support! LLaVa + FigStep - HUGE thanks to @DavidLee528
• LiteLLM generator support - thanks to @Tien-Cheng!
• DoNotAnswer probe for prompts an LLM should decline - thanks to @AhsanAyub!
• LangChain Serve generator support - thanks to @GustavFredrikson!
• Support for NIM generators, NVIDIA Inference Microservices
• OpenAI abstraction with parallelisation support
• Windows compatibility enhanced, with optional testing in the workflow
• Hugging Face speedup

plugins

• LiteLLM generator by @Tien-Cheng in #572
• Support for using 'langchain serve' endpoints by @GustavFredrikson in #588
• Enable parallel OpenAI calls by @jmartin-tech in #645
• Multi-modal Jailbreaking Attack on LLaVA by @DavidLee528 in #587
• bump openai module version to match paths in latest litellm by @leondz in #664
• generator: NIM by @leondz in #637
• Probe: Do Not Answer by @AhsanAyub in #608

architecture

• change supported pythons to 3.10-3.12 by @leondz in #503
• add more detailed TAP docs by @leondz in #504
• add multiple-result aggregator by @leondz in #505
• add post buff hook by @erickgalinkin in #506
• Add fleshed-out docs to all probes by @leondz in #507
• add bibtex under citation info in readme by @leondz in #511
• define broad test, all probes, 1 gen per by @leondz in #514
• Feature/taxonomy payloads by @leondz in #519
• include paraphrasing in broad conf by @leondz in #521
• choose whether buffing will also include the original prompt by @leondz in #523
• add config var for capping max # buffed prompts to add per buff by @leondz in #526
• document Probe.probe(); skip a buff hook if no buffs by @leondz in #527
• add type hints to base.Probe; fix base probe rst by @leondz in #528
• Bump datasets package by @shubhobm in #536
• Add ConversationalPipeline for huggingface models by @erickgalinkin in #539
• add generator for supporting openai module v0.x by @leondz in #553
• Update README.md by @erickgalinkin in #558
• Minor typo in FAQ by @jmartin-tech in #562
• Add additional error message when doc is None type by @DavidLee528 in #566
• shared constant & string literal by @jmartin-tech in #571
• Spelling corrections for multiple locations by @jmartin-tech in #564
• Reduce Huggingface GPU utilization by @erickgalinkin in #567
• skip verbose flag in secondary parser by @jmartin-tech in #576
• Added project twitter link and corrected a grammatic error by @codebrain001 in #578
• Convert GGML to expect GGUF format by @jmartin-tech in #581
• Update workflows: CLA asst bump, PR & manual testing by @leondz in #591
• add test de-duping using skip-duplicate-actions by @leondz in #597
• Remove #! entries from files not intended as executables by @jmartin-tech in #612
• Further align shebangs with code that has executable entry points by @leondz in #613
• interactive mode intro by @leondz in #614
• add tests for ggml generator by @jmartin-tech in #618
• add var for generator context_len and populate this for some generators by @leondz in #616
• allow generators.Base.generate() to take an optional param specifying generation count by @leondz in #600
• Enable windows tests as github action by @jmartin-tech in #626
• add on-demand macos testing by @leondz in #631
• macOS test install from correct path by @jmartin-tech in #633
• consolidate test file cleanup by @jmartin-tech in #634
• bump discord link by @leondz in #648
• meta the arguments a bit for GET vs other request types by @jmartin-tech in #640

stability

• fail gracefully if nvcf rejects input; compact zalgo prompts by @leondz in #509
• log & skip past NVCF 4xx errors by @leondz in #533
• fix empty autodan prompts & poor detector behaviour by @leondz in #534
• Fix AutoDAN issues by @erickgalinkin in #537
• fix bad nonetype handling in atkgen probe by @leondz in #538
• Division by zero error fixed in HTML report generation by @CoderMayhem in #545
• cap cohere lib version by @leondz in #569
• rm deprecated model from example by @leondz in #575
• Attack fixes by @erickgalinkin in #555
• More regex as string literal by @jmartin-tech in #586
• Bugfix/action dedupe by @leondz in #598
• wrap cli exec to gracefully catch keyboard exit signal by @jmartin-tech in #603
• Enforce warning output for garak classes. by @jmartin-tech in #605
• bump hf transformers v to avoid transformers#30076 by @leondz in #636
• update avidtools to remove typing reference by @jmartin-tech in #639
• torch v bump by @leondz in #649
• Pause FigStepTiny by @leondz in #652
• Bugfix/visual jailbreak pause by @leondz in #653
• limit push test to main by @jmartin-tech in #661
• Update MANIFEST.in so all resources are installed by @JKL98ISR in #660
• handle extant but closed hitlogfile file by @leondz in #665

New Contributors

• @CoderMayhem made their first contribution in #545
• @jmartin-tech made their first contribution in #562
• @codebrain001 made their first contribution in #578
• @Tien-Cheng made their first contribution in #572
• @GustavFredrikson made their first contribution in #588
• @JKL98ISR made their first contribution in #660
• @AhsanAyub made their first contribution in #608

Full Changelog: v0.9.0.12...v0.9.0.13