giskard v3.0.0

What's Changed

• feat!: rewrite as v3 monorepo with giskard-core, giskard-agents, and giskard-checks by @kevinmessiaen in #2229
• fix: refactor release workflow by @henchaves in #2256
• chore: update readme by @mattbit in #2257
• chore(deps): update pep621 dependencies by @renovate[bot] in #2258
• feat(checks): ENG-1390 judges directory for llm and prompt reliant checks by @davidberenstein1957 in #2248
• refactor: introduce JsonPathStr type for JSONPath validation by @davidberenstein1957 in #2264
• feat: [OSS-20] Add support to annotations in scenario/trace by @henchaves in #2268
• feat(checks)!: [OSS-13] add RegexMatching check implementation by @davidberenstein1957 in #2247
• feat(agents,core): [ENG-1272]/[ENG-1271] refactor rate limiters and add generator retry/timeout policies by @kevinmessiaen in #2239
• refactor(checks)!: new 'Interact' API by @mattbit in #2270
• chore(ci): add Python 3.14 to test matrix by @mattbit in #2280
• feat: [OSS-22] Suite: run batches of scenarios with dynamic binding by @henchaves in #2283
• chore(deps): update astral-sh/setup-uv digest to 5a095e7 by @renovate[bot] in #2285
• feat(agents): add step-level type discriminator (GAP-003) by @Hartorn in #2275
• chore: consolidate cursor rules at workspace level and enforce Python 3.12+ by @Hartorn in #2286
• feat(generators): introduce personas and extended context for UserSimulator by @davidberenstein1957 in #2249
• feat(agents): add tool input coercion and output serialization (GAP-005) by @Hartorn in #2273
• refactor(agents): replace mixin-based retry/rate-limiting with middle… by @Hartorn in #2281
• refactor(checks): make Scenario mutable and remove scenario() factory by @kevinmessiaen in #2296
• refactor(rate-limiter): remove redundant locks, raise on duplicate id by default by @kevinmessiaen in #2298
• refactor(checks): replace Scenario sequence with step-based structure by @kevinmessiaen in #2297
• feat(agents): generator-as-protocol-adapter and tool coercion by @Hartorn in #2279
• chore(deps): bump tornado from 6.5.4 to 6.5.5 in the uv group across 1 directory by @dependabot[bot] in #2301
• fix(release): treat workflow dry_run input as string by @kevinmessiaen in #2303
• test(agents): skip OpenAI-dependent embedding functional test by @kevinmessiaen in #2304
• fix(tests): Uses gemini-embedding-001 by default by @kevinmessiaen in #2306
• chore(deps): update github-actions dependencies by @renovate[bot] in #2309
• fix(checks)!: make Interact injection name-based by @kevinmessiaen in #2302
• feat(agents): add metadata parameter to generator completion pipeline by @Hartorn in #2300
• feat: reexport resolve at root level by @Deepak8858 in #2311
• Make Suite.append() chainable (return self) by @Bingtagui404 in #2317
• feat(checks): add print_report to result models by @kevinmessiaen in #2314
• docs(giskard-agents): update README examples to match current API and improve clarity by @Hartorn in #2312
• fix(agents): use shallow copy in BaseGenerator.with_params() by @Hartorn in #2319
• chore(deps): update astral-sh/setup-uv digest to 37802ad by @renovate[bot] in #2322
• feat(checks): show scenario and check error details in suite report by @kevinmessiaen in #2323
• fix(giskard-checks): preserve concrete trace type on scenario results by @kevinmessiaen in #2326
• build(security): ignore pygments CVE-2026-4539 in pip-audit by @mattbit in #2333
• fix(giskard-agents): use jinja2 sandboxed environment to prevent template injection [ENG-1486] by @mattbit in #2332
• refactor(giskard-checks)!: remove Scenario.from_sequence by @kevinmessiaen in #2325
• docs: update README and repo docs for v3 by @davidberenstein1957 in #2335
• build(security): harden GitHub Actions workflows by @Hartorn in #2336
• fix(deps): bump requests to 2.33.0 for CVE-2026-25645 by @kevinmessiaen in #2341
• build(security): use GitHub App token for release push by @Hartorn in #2342
• security(deps): Upgrade pygments to fix CVE CVE-2026-4539 by @kevinmessiaen in #2351
• fix(checks): preserve generator and embedding model when running scenarios by @kevinmessiaen in #2293
• security: upgrade aiohttp to fix multiple cve by @kevinmessiaen in #2356
• Add Junix XML export for SuiteResult by @Mapalo90 in #2352
• docs: align PR template with uv.lock and NumPy docstrings by @kevinmessiaen in #2343
• security(giskard-agents)!: opt-in Jinja parsing for Workflow.chat by @kevinmessiaen in #2346
• docs(contributing): align guide with repo and fix links by @kevinmessiaen in #2344
• fix(release): sync uv.lock after package version bump by @kevinmessiaen in #2357
• chore(deps): update dependency litellm to v1.83.0 [security] by @renovate[bot] in #2359
• fix(giskard-checks): bound RegexMatching with regex timeout to mitigate ReDoS by @kevinmessiaen in #2381
• fix(security)!: drop support for templating in conformity rules by @kevinmessiaen in #2382
• feat(giskard-checks): add AllOf, AnyOf, Not check composition operators by @nuthalapativarun in #2390
• chore(deps): update actions/create-github-app-token digest to 1b10c78 by @renovate[bot] in #2394
• feat(giskard-checks): add AnswerRelevance LLM judge check by @nuthalapativarun in #2388
• feat: agents instructions by @kevinmessiaen in #2383
• chore(deps-dev): bump pytest from 9.0.2 to 9.0.3 in the uv group across 1 directory by @dependabot[bot] in #2402
• feat(telemetry): add optional PostHog analytics to giskard-core and checks by @kevinmessiaen in #2378
• feat(giskard-checks): add Toxicity LLM judge check by @Koushik-Salammagari in #2385
• feat: agents PR instructions by @kevinmessiaen in #2400
• chore(analytics): add GeoIP disabling option by @henchaves in #2407
• fix(deps): declare regex on giskard-checks by @kevinmessiaen in #2405
• Allow scenarios to run multiple times by @raghavtries in #2401
• chore(deps): update actions/github-script action to v9 by @renovate[bot] in #2415
• fix(deps): update pep621 dependencies (major) by @renovate[bot] in #2416
• chore(deps): bump python-dotenv from 1.0.1 to 1.2.2 in the uv group across 1 directory by @dependabot[bot] in #2424
• chore(deps): bump litellm from 1.83.0 to 1.83.14 by @kevinmessiaen in #2426
• feat: add giskard-llm as lean litellm replacement by @Hartorn in #2329
• deps: Upgrade giskard-core to 1.0.1b3 for giskard-agents and giskard-checks by @kevinmessiaen in #2428
• fix(tests): use provider-agnostic assertions by @kevinmessiaen in #2429
• ci(release): handle giskard-llm by @kevinmessiaen in #2430
• deps: Upgrade giskard-agents to 1.0.2b2 for giskard-checks by @kevinmessiaen in #2432
• fix(functional-test): Fix giskard-llm functional tests by @kevinmessiaen in #2437
• chore(deps): bump urllib3 from 2.6.3 to 2.7.0 in the uv group across 1 directory by @dependabot[bot] in #2446
• chore(deps): update astral-sh/setup-uv action to v8 by @renovate[bot] in #2444
• chore(deps): update github-actions dependencies by @renovate[bot] in #2443
• feat(giskard-checks): minimal OWASP LLM suite generator (LLM01 indirect injection) by @kevinmessiaen in #2438
• doc(scenariod): improve docstring of scenariod by @kevinmessiaen in #2379
• fix(docs): correct typos and improve formatting in README.md by @henchaves in #2448
• checks: support pydantic-compatible input types by @harsh21234i in #2450
• docs: add CLAUDE.md and improve agent workflow in AGENTS.md by @kevinmessiaen in #2452
• docs: restructure agent instruction files by @kevinmessiaen in #2453
• chore: add Claude Code worktree support and config by @kevinmessiaen in #2455
• fix(giskard-llm): correct phantom ResponseOutputFunctionCall in all by @yahyasaqban-lab in #2461
• docs: update workflow to use superpowers skills by default by @kevinmessiaen in #2462
• Add JSON validity check by @shusingh in #2460
• fix: upgrade idna 3.11 -> 3.15 (CVE-2026-45409) by @kevinmessiaen in #2469
• fix(giskard-checks): lazy-initialize default generator and embedding model by @kevinmessiaen in #2468
• feat(checks): infer trace type from target second parameter annotation by @kevinmessiaen in #2470
• ci(release): make functional tests optional via workflow input by @kevinmessiaen in #2476
• docs: make AGENTS.md cleaner by @kevinmessiaen in #2477
• docs: add AUTONOMOUS.md by @kevinmessiaen in #2479
• feat(checks): add parallel scenario execution to Suite.run by @harsh21234i in #2474
• fix(giskard-oss): PR template message outdated links by @educhesne in #2480
• chore(deps): update actions/create-github-app-token digest to bcd2ba4 by @renovate[bot] in #2463
• fix(deps): update dependency google-genai to v2 by @renovate[bot] in #2464
• chore(deps): update zizmorcore/zizmor-action action to v0.5.6 by @renovate[bot] in #2489
• feat(giskard-llm): pass through provider transport config by @Hartorn in #2486
• docs(giskard-llm): document Azure Foundry v1 via OpenAI by @Hartorn in #2487
• test: optional deps refactor by @kevinmessiaen in #2490
• feat(giskard-checks): adversarial scenario generator for stereotypes & discrimination by @kevinmessiaen in #2466
• chore: update model from gemini-2.0-flash to gemini-3.5-flash by @henchaves in #2502
• docs(giskard-checks): fix lowercase scenario import in README by @kevinmessiaen in #2491
• feat(giskard-checks): rego policy checks by @educhesne in #2478
• chore(giskard-scan): migrate generate_suite and generators from giskard-checks" by @kevinmessiaen in #2506
• chore(deps): bump aiohttp from 3.13.4 to 3.14.0 in the uv group across 1 directory by @dependabot[bot] in #2510
• feat(scan): [ENG-1302] add GOAT scenario generator by @pierlj in #2507
• chore(deps-dev): bump pytest-asyncio from 1.3.0 to 1.4.0 by @dependabot[bot] in #2508
• chore(deps-dev): bump pytest-cov from 6.1.0 to 7.1.0 by @dependabot[bot] in #2509
• chore(security): Upgrade pip to 26.1.2 PYSEC-2026-19 by @kevinmessiaen in #2528
• fix(docs): expose version via importlib.metadata and update bug template by @nuthalapativarun in #2516
• chore(deps): update github-actions dependencies to df4cb1c by @renovate[bot] in #2527
• feat(checks): add progress bar to suite runs by @henchaves in #2514
• fix: share schema-name sanitizer and simplify LiteLLM generator by @henchaves in #2522
• feat: Add crescendo-like scenarios generator by @pierlj in #2513
• feat: add scenario tag grouping in suite results by @kevinmessiaen in #2534
• feat(scan): add vulnerability scan api by @kevinmessiaen in #2538
• chore(deps): bump tornado from 6.5.5 to 6.5.6 in the uv group across 1 directory by @dependabot[bot] in #2540
• chore(deps): update github-actions dependencies by @renovate[bot] in #2542
• fix(checks): validate suite max concurrency by @kiku-jw in #2541
• deps: Upgrade aiohttp, cryptography and tornado due to CVEs by @kevinmessiaen in #2547
• fix(scan): read JSONL datasets as UTF-8 by @weijun-xia in #2544
• feat(checks): add suite field to SuiteResult by @kevinmessiaen in #2548
• fix(giskard-checks): use NOT_PROVIDED sentinel as default for expected_value by @nuthalapativarun in #2504
• chore(claude): auto-format hook + /check verification-gate skill by @kevinmessiaen in #2551
• fix(deps): upgrade msgpack to 1.2.1 for GHSA-6v7p-g79w-8964 by @kevinmessiaen in #2554
• test(scan): cover UTF-8 decoding of JSONL datasets by @weijun-xia in #2550
• chore(deps): update pep621 dependencies by @renovate[bot] in #2553
• feat(scan): Add quality scan and knowledge base scenario generator by @pierlj in #2539
• chore: fix license generation and update third-party notices by @kevinmessiaen in #2492
• feat: Add buildable giskard meta-package and release workflow support by @kevinmessiaen in #2404
• feat(scan): add target_mode flag for singleturn/multiturn scan by @kevinmessiaen in #2549
• feat(scan): add do-not-answer scan generator by @kevinmessiaen in #2552
• fix(checks): NOT_PROVIDED serialization roundtrip set {} by @kevinmessiaen in #2558
• feat(scan): add HarmBench dataset to vulnerability scan by @kevinmessiaen in #2559
• feat(scan): Add sycophantic scenario from knowledge base by @pierlj in #2557
• checks: add readability NLP metric check by @harsh21234i in #2412
• checks: make suite failure reporting configurable by @harsh21234i in #2418
• fix(agents): surface unknown workflow tool calls by @alexanderkreidich in #2414
• chore: update lockfile by @kevinmessiaen in #2566
• fix: formalize cross-lib public contracts by @kevinmessiaen in #2536
• chore(deps): update pep621 dependencies by @renovate[bot] in #2573
• chore(deps): update zizmorcore/zizmor-action action to v0.5.7 by @renovate[bot] in #2574
• feat: allow unlimited suite reporting by @kevinmessiaen in #2564
• feat: Fail-fast when instanciating checks with missing optional lib by @kevinmessiaen in #2565
• feat: [ENG-1568] add to_hub_format() in SuiteResult by @henchaves in #2560
• feat(scan): Complete and reorganize quality scan by @pierlj in #2563
• feat(scan): Add automatic recommendation from failure categories by @pierlj in #2567
• feat(scan): add GCG injection scenario generator by @kevinmessiaen in #2561
• fix(checks): rename LesserThan to LessThan with legacy aliases by @kevinmessiaen in #2580
• chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 by @dependabot[bot] in #2577
• chore: regenerate THIRD_PARTY_NOTICES.md by @kevinmessiaen in #2589
• chore(doc): Update giskard-scan status by @carloshenriquefbf in #2587
• fix(google): map Gemini finish_reason via enum value, not str() by @WatchTree-19 in #2595
• fix(checks): resolve JsonValid string parse/value ambiguity by @kevinmessiaen in #2575
• feat(checks): add match mode to comparison checks by @kevinmessiaen in #2579
• feat(checks): add pydantic-settings for default model and config by @kevinmessiaen in #2581
• chore(deps): update github-actions dependencies by @renovate[bot] in #2601
• fix(scan): GOAT/Crescendo generators crash on empty languages list by @chuenchen309 in #2602
• fix(checks): dot-wildcard/multi-field JSONPath returns scalar vs list depending on match count by @chuenchen309 in #2605
• fix(llm): use .text not .content in _validate_messages to avoid AttributeError on list content by @chuenchen309 in #2608
• fix(agents): guard on chat.messages not chat.last to avoid IndexError in _run_tools by @chuenchen309 in #2609
• chore(gitignore): ignore generated agent tooling state by @kevinmessiaen in #2617
• chore(deps): bump nltk from 3.9.4 to 3.10.0 in the uv group across 1 directory by @dependabot[bot] in #2619
• ci: opt in to fork PR checkout for integration tests by @kevinmessiaen in #2624
• fix(checks): record input generation errors by @harsh21234i in #2599
• fix: scan continue on generation error by @henchaves in #2625
• fix(security): upgrade pyasn1 to 0.6.4 to fix known vulnerabilities by @kevinmessiaen in #2630
• ci: pin ruff so check-format stops failing on main by @BILLKISHORE in #2686
• chore(deps): bump astral-sh/setup-uv from 8.3.0 to 9.0.0 by @dependabot[bot] in #2685
• chore(deps): bump actions/labeler from 6.2.0 to 7.0.0 by @dependabot[bot] in #2684
• chore(deps): bump zizmorcore/zizmor-action from 0.5.7 to 0.6.1 by @dependabot[bot] in #2683
• chore(deps): bump actions/checkout from 7.0.0 to 7.0.1 by @dependabot[bot] in #2682
• chore(scan): export default registry generators on the public API by @davidberenstein1957 in #2665
• docs(readme): install matrix, telemetry opt-out, runnable quickstarts by @davidberenstein1957 in #2674
• docs: clarify parallel execution vs scenario generation by @davidberenstein1957 in #2656
• chore(pyright): flag and remove unused ignore comments by @kevinmessiaen in #2690
• fix(llm): warn on unknown Anthropic params; detect bare refusals by @davidberenstein1957 in #2647
• fix(core): registry loses live rate limiters via WeakSet structural dedup by @davidberenstein1957 in #2649
• fix(checks): SemanticSimilarity embeds the repr of a list-valued key by @davidberenstein1957 in #2652
• fix(deps): upgrade aiohttp and cryptography for pip-audit CVEs by @kevinmessiaen in #2698
• refactor(scan): align quality_scan and vulnerability_scan option APIs by @davidberenstein1957 in #2679
• fix(deps): bump setuptools to 83.0.0 (CVE-2026-59890) by @davidberenstein1957 in #2694
• fix(checks): structural unevaluable outcomes return ERROR (Not silent-green) by @kevinmessiaen in #2693
• docs(agents): document Cursor Cloud dev environment setup by @kevinmessiaen in #2704
• fix(telemetry): fall back on an empty ~/.giskard/id file instead of returning "" by @chuenchen309 in #2612
• fix(agents): resolve string annotations in tools by @BILLKISHORE in #2687
• fix(checks): include readability in the all extra by @BILLKISHORE in #2705
• fix(google): convert Pydantic response_format to the Interactions SDK TextResponseFormatParam by @davidberenstein1957 in #2648
• feat(scan): integrate third party scan by @kevinmessiaen in #2578
• fix(checks): join list-valued judge inputs without Python repr by @davidberenstein1957 in #2650
• fix(checks): descendants JSONPath always returns a list by @davidberenstein1957 in #2644
• fix(checks): skip regorus extra on platforms without a wheel (Windows, linux/aarch64) by @davidberenstein1957 in #2646
• docs(readme): correct v2/v3 feature mapping — RAG now in giskard-scan by @kevinmessiaen in #2610
• fix(checks): ERROR when Contradiction/Groundedness keys miss [ENG-1672] by @kevinmessiaen in #2702
• feat(checks): require non-blank judge reason by @kevinmessiaen in #2703
• fix(checks): ERROR when AnswerRelevance keys miss; history is context only [ENG-1673] by @kevinmessiaen in #2706
• chore(deps): require giskard-core>=1.0.1b6 in direct dependents by @kevinmessiaen in #2707
• chore(deps): require giskard-llm>=1.0.0b6 in direct dependents by @kevinmessiaen in #2708
• chore(deps): require giskard-agents>=1.0.2b6 in direct dependents by @kevinmessiaen in #2709
• fix(prompts): fence untrusted content in judge and simulator prompts by @weijun-xia in #2543
• chore(deps): require giskard-checks>=1.0.2b6 in direct dependents by @kevinmessiaen in #2710
• chore(deps): update astral-sh/setup-uv action to v9 by @renovate[bot] in #2714
• fix(google): map content blocks to refusal by @BILLKISHORE in #2715
• fix(giskard-checks): catch TimeoutError in RegexMatching.run() by @kevinmessiaen in #2711
• refactor(checks)!: drop LesserThan and LesserThanEquals classes by @kevinmessiaen in #2720
• refactor(checks)!: rename GreaterEquals to GreaterThanEquals by @kevinmessiaen in #2747
• fix(checks): preserve partial trace on input generation failure by @dev404ai in #2718
• refactor(agents)!: remove deprecated set_prompts_path by @kevinmessiaen in #2738
• chore: fix package descriptions and remove empty legacy giskard shells by @davidberenstein1957 in #2662
• fix(checks): fence contradiction judge prompt by @kevinmessiaen in #2746
• refactor(agents)!: re-export Error from giskard.core by @kevinmessiaen in #2748
• fix(release): stop default inputs inflating version, add stable bump by @kevinmessiaen in #2722
• fix(checks): Toxicity errors on unresolvable output_key by @davidberenstein1957 in #2757
• fix(checks): AllOf no longer short-circuits on SKIP by @davidberenstein1957 in #2750
• fix(checks)!: return None from SuiteResult.pass_rate when nothing evaluated by @davidberenstein1957 in #2753
• test(giskard-checks): isolate tests from ambient GISKARD_* env by @davidberenstein1957 in #2749
• fix(checks): audit SKIP/ERROR consumers for false-PASS bugs by @davidberenstein1957 in #2759
• refactor(checks)!: drop builtin judge re-exports by @davidberenstein1957 in #2751
• fix(scan): anchor optional-extra collection ignores to the lib root by @kevinmessiaen in #2766
• fix(release): resolve PACKAGE=giskard to the whole workspace by @kevinmessiaen in #2764
• fix(meta): align root giskard-scan extras with member version b4 by @kevinmessiaen in #2737
• fix(agents): escape prompt template values by default by @davidberenstein1957 in #2760
• feat(types): export and tighten public Literal/status types by @davidberenstein1957 in #2755
• feat: ship PEP 561 py.typed for core, llm, checks, and scan by @kevinmessiaen in #2767
• feat: [OSS-138] Add more scenario and templates for prompt injection by @pierlj in #2770
• fix(release): tag the meta package as vX.Y.Z by @kevinmessiaen in #2771
• test(examples): keep README snippets and examples correct via CI by @davidberenstein1957 in #2676
• fix(google): map Interactions timeouts to LLMTimeoutError by @BILLKISHORE in #2765
• fix(google): count Gemini thinking tokens in usage.output_tokens by @davidberenstein1957 in #2653
• fix(checks)!: reject unknown fields across the persisted-scenario tree by @kevinmessiaen in #2768
• docs: correct two misleading docstrings by @davidberenstein1957 in #2772
• chore(deps): update github-actions dependencies by @renovate[bot] in #2713
• refactor(checks)!: name the value under test target_key on every check by @kevinmessiaen in #2769
• docs: add v3 migration changelog by @kevinmessiaen in #2774
• chore: pin downstream packages to giskard-core rc1 by @kevinmessiaen in #2775
• chore: pin downstream packages to giskard-llm rc1 by @kevinmessiaen in #2776
• chore: pin downstream packages to giskard-agents rc1 by @kevinmessiaen in #2777
• chore: pin downstream packages to giskard-checks rc1 by @kevinmessiaen in #2778
• chore: pin giskard to giskard-scan rc1 by @kevinmessiaen in #2779
• chore(ci): disable deployment tracking for tests and enforce base branch for integration tests by @mattbit in #2781
• chore(deps): upgrade pip to 26.2.1 for PYSEC-2026-3721 by @kevinmessiaen in #2788
• chore(deps): update astral-sh/setup-uv action to v10 by @renovate[bot] in #2789
• fix(deps): update dependency openai to v3 by @renovate[bot] in #2790
• Add enterprise welcome message by @alexcombessie in #2784
• fix(core): wrap enterprise welcome message for small screens by @henchaves in #2793
• chore: update third-party notices for boto3 1.43.80 by @kevinmessiaen in #2795
• chore: pin downstream packages to giskard-core 1.0.1 by @kevinmessiaen in #2796
• chore: pin downstream packages to giskard-llm 1.0.0 by @kevinmessiaen in #2797
• chore: pin downstream packages to giskard-agents 1.0.2 by @kevinmessiaen in #2798
• chore: pin downstream packages to giskard-checks 1.0.2 by @kevinmessiaen in #2799
• chore: pin giskard to giskard-scan 1.0.0 by @kevinmessiaen in #2800

New Contributors

• @renovate[bot] made their first contribution in #2258
• @Deepak8858 made their first contribution in #2311
• @Bingtagui404 made their first contribution in #2317
• @Mapalo90 made their first contribution in #2352
• @nuthalapativarun made their first contribution in #2390
• @Koushik-Salammagari made their first contribution in #2385
• @raghavtries made their first contribution in #2401
• @harsh21234i made their first contribution in #2450
• @yahyasaqban-lab made their first contribution in #2461
• @shusingh made their first contribution in #2460
• @educhesne made their first contribution in #2480
• @kiku-jw made their first contribution in #2541
• @weijun-xia made their first contribution in #2544
• @alexanderkreidich made their first contribution in #2414
• @carloshenriquefbf made their first contribution in #2587
• @WatchTree-19 made their first contribution in #2595
• @chuenchen309 made their first contribution in #2602
• @BILLKISHORE made their first contribution in #2686
• @dev404ai made their first contribution in #2718

Full Changelog: v2.19.1...v3.0.0