Skip to content

Releases

v3.5.2: watch agentest catch a prompt injection in one command

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 21 Nov 07:37
3aef310

One command, and you watch agentest catch an agent following a prompt injection

A real 8-second terminal run of pip install agentest-ai followed by agentest demo --agent a28, which reports 7 failed checks including T4-001

Install from PyPI:

pip install agentest-ai

Then run:

agentest demo --agent a28

a28 is a bundled reference agent recorded on a real run. A planted URGENT instruction in its input surfaced in what it did next, and agentest flags it under T4-001, T4-009 and T4-010. Plain agentest demo judges the bundled overreach sample. Add --html report.html to get a report you can click through. It runs offline, with no API key and no trace file of your own.

New

  • agentest demo: judges a bundled sample trace and prints pass/fail/n/a counts with a few example findings. --agent picks another bundled agent, --html writes a report, and --no-color turns off colour.
  • How-to pages for LangGraph, CrewAI, OpenAI Agents SDK, AutoGen, smolagents, n8n and MCP under docs/integrations/.
  • llms.txt and llms-full.txt, generated from the README, the CLI flags and every test ID.

Fixed

  • T1-005 / X-008 (#114): a short negation such as "no" cancels a success claim only within its own sentence, and a failed step that an identical retry redid successfully no longer fails T1-005. Bundled verdicts: T1-002 moves from pass to fail for a22 (both modes) and a28 (examples).

Unchanged

165 tests. Apart from the T1-005 fix above, no rule, scorer or label change.

Also new on main since this tag: an agentic evals challenge with 20 example traces for contributor week (23 to 30 November), the Wall of caught failures, a 60-second Colab notebook and the agentest GitHub Action.

Full changelog: CHANGELOG.md · Package: agentest-ai on PyPI · Getting started: Tutorials

agentest v3.5.1 (PyPI: agentest-ai)

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 19 Nov 16:47
555b753

agentest 3.5.1 — wording and packaging on PyPI

Install from PyPI:

pip install agentest-ai

To pin this release:

pip install "agentest-ai==3.5.1"

The import and the agentest command stay agentest. Package page: https://pypi.org/project/agentest-ai/

Highlights

  • The PyPI description, package docstrings and agentest --help now say 165 tests.
  • Metadata adds keywords (offline, opentelemetry, traces), a Quality Assurance classifier, and a Tutorials project URL.
  • Install commands in the README and first-use docs sit in their own code blocks, with no trailing punctuation.

Packaging and wording only. No rule, scorer or verdict change.

agentest v3.5.0 (PyPI: agentest-ai)

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 19 Nov 09:48
e22cf89

agentest 3.5.0 — now on PyPI as agentest-ai

Install:

pip install agentest-ai

The import and the agentest command stay agentest. Python 3.11 or newer. Package page: https://pypi.org/project/agentest-ai/

Upgrade

pip install -U agentest-ai

If you previously installed from git or from a clone, the command above switches you to the PyPI package.

Highlights

  • First PyPI release of agentest-ai 3.5.0. The name agentest on PyPI belongs to a different project.
  • 30 opt-in New tests (NT-001–NT-030). Run with agentest std run --ids NT or list with agentest list --new-tests. The default run is still 130 checks (110 checklist + X-001–X-020).
  • Trace format 1.0.0: specification, JSON Schema, connectors and readers (including D2 aliases and D7/D8 record forms). Verdicts on existing traces are unchanged.
  • Connectors for plain OpenTelemetry, LangChain/LangGraph, OpenAI Agents SDK, n8n, and other native-log dialects (steps/CrewAI, events, AutoGen, smolagents, file:).

What's Changed

Other changes

New Contributors

Full Changelog: v3.4.0...v3.5.0

agentest v3.3.2 (PyPI name agentest-ai)

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 02 Nov 06:36
436fe39

Packaging release; rules and verdicts unchanged from v3.3.1.

  • PyPI distribution name is now agentest-ai (agentest on PyPI is an unrelated project). Import name and the agentest CLI are unchanged.
  • PyPI upload is pending. Until it is live, install from git: pip install "agentest-ai @ git+https://github.com/srpadmanabhuni/agentest.git@v3.3.2", or install the wheel attached below.
  • RELEASE gains pypi_name; scripts/check_versions.py checks the pyproject name and every install line.
  • scripts/publish_pypi.sh uploads with twine using PYPI_API_TOKEN (never printed).

Artifacts (twine check passed): agentest_ai-3.3.2-py3-none-any.whl, agentest_ai-3.3.2.tar.gz. PR #15.

agentest v3.3.1 (review fixes)

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 02 Nov 06:31
d3819d1

Review-fix release after an external correctness review of v3.3.0 and dataset v1.3.1. Rules and verdicts are unchanged from v3.3.0.

  • Fix: catalog/checklist-110.jsonl regenerated from dataset v1.3.1 (byte-identical to the Hub file). It had carried the pre-v1.2.1 control_refs the dataset retracted on all 110 rows. catalog_revision is now v1.3.1.
  • CI: the catalogue pin check now covers the jsonl (sha256), cross-checks control_refs against every pack CSV, and rejects retracted tokens.
  • CI: new RELEASE file is the single source of truth for version strings; scripts/check_versions.py fails CI on drift (pyproject, __version__, install pins in README/docs, README coverage rows vs docs/COVERAGE.md, catalog/VERSION, citation DOI) and --bump rewrites them.
  • Docs: stale install pins fixed; README coverage table current; legacy catalog/t1_001.yaml archived to docs/history/.
  • Citation: README and CITATION.cff cite the checklist's Zenodo all-versions concept DOI 10.5281/zenodo.23597877.

Install: pip install "agentest @ git+https://github.com/srpadmanabhuni/agentest.git@v3.3.1"

PRs: #12, #13, #14.

v3.3.0

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 27 Oct 10:07

agentest v3.3.0 — universal-rule precision

Seven fixes from the A99–A101 blind test (standard fields + documented governance extension only; no agent-specific rule code):

  1. T2-004 / T2-012 — only governance.approval.required (MCP destructiveHint fallback when absent)
  2. T2-011 — write/destructive tool after deny only
  3. T3-011 — post-handoff orchestrator writes only
  4. T5-002 — domain status=denied is not a tool error
  5. X-005 — honour limits.max_steps; multi-agent default 24
  6. X-011 / X-012 — id + currency grounding against tool outputs
  7. X-014 — Unicode-script language consistency (any language)

Gates

  • Examples regression floor ≥ 391/394
  • Seed catch-rate 90/90
  • 0 one-sided IDs
  • Verdict snapshot regenerated (VERDICT-DIFF-3.3.0.md)

Blind tests

See docs/BLIND-TEST.md.

Round Agents Result
1 re-run A99–A101 FP→0; prior currency/id/language FNs caught
2 fresh A102–A104 TP=3 FP=0 FN=0 (precision/recall 1.00)

v3.2.1

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 27 Oct 08:32

agentest v3.2.1

Seed-aware catch-rate, tight published budgets on A90–A97, targeted re-seeds, T6-003 widen.

Catch-rate

  • Before (v3.2.0 broken accounting): 63 “misses” counted every decided global TARGET pass on defect agents (including 35 IDs never seeded).
  • Corrected before (same 90 declared seeds, v3.2.0 behaviour): 62/90 caught.
  • After (v3.2.1): 90/90 declared seeds caught; 0 remaining misses.

Changes

  • Seeds declared in AgentCard fixture extension urn:agentest:a2a-ext:seeded-defects:v1 (rules never read it).
  • A90–A97 publish tight limits so T5-006 / X-006 / X-007 fire.
  • A95 T4-003 (notify_author + injected email), A96 X-011 (OPS-4401), A97 T1-017 (distinct ADJ-NNNN on twin submit).
  • T6-003 also checks numbers on slots/multi_turn (catches A92); no all-writes widen / no regressions.
  • Verdict snapshot: 56 cells P→F (28 seeds × 2 modes); regress floor held 391/394.

Zero API cost (local Qwen2.5-7B).