Repository navigation
Releases
v3.5.2: watch agentest catch a prompt injection in one command
One command, and you watch agentest catch an agent following a prompt injection
Install from PyPI:
pip install agentest-aiThen run:
agentest demo --agent a28a28 is a bundled reference agent recorded on a real run. A planted URGENT instruction in its input surfaced in what it did next, and agentest flags it under T4-001, T4-009 and T4-010. Plain agentest demo judges the bundled overreach sample. Add --html report.html to get a report you can click through. It runs offline, with no API key and no trace file of your own.
New
agentest demo: judges a bundled sample trace and prints pass/fail/n/a counts with a few example findings.--agentpicks another bundled agent,--htmlwrites a report, and--no-colorturns off colour.- How-to pages for LangGraph, CrewAI, OpenAI Agents SDK, AutoGen, smolagents, n8n and MCP under
docs/integrations/. llms.txtandllms-full.txt, generated from the README, the CLI flags and every test ID.
Fixed
- T1-005 / X-008 (#114): a short negation such as "no" cancels a success claim only within its own sentence, and a failed step that an identical retry redid successfully no longer fails T1-005. Bundled verdicts: T1-002 moves from pass to fail for a22 (both modes) and a28 (examples).
Unchanged
165 tests. Apart from the T1-005 fix above, no rule, scorer or label change.
Also new on main since this tag: an agentic evals challenge with 20 example traces for contributor week (23 to 30 November), the Wall of caught failures, a 60-second Colab notebook and the agentest GitHub Action.
Full changelog: CHANGELOG.md · Package: agentest-ai on PyPI · Getting started: Tutorials
agentest v3.5.1 (PyPI: agentest-ai)
agentest 3.5.1 — wording and packaging on PyPI
Install from PyPI:
pip install agentest-aiTo pin this release:
pip install "agentest-ai==3.5.1"The import and the agentest command stay agentest. Package page: https://pypi.org/project/agentest-ai/
Highlights
- The PyPI description, package docstrings and
agentest --helpnow say 165 tests. - Metadata adds keywords (
offline,opentelemetry,traces), a Quality Assurance classifier, and a Tutorials project URL. - Install commands in the README and first-use docs sit in their own code blocks, with no trailing punctuation.
Packaging and wording only. No rule, scorer or verdict change.
agentest v3.5.0 (PyPI: agentest-ai)
agentest 3.5.0 — now on PyPI as agentest-ai
Install:
pip install agentest-aiThe import and the agentest command stay agentest. Python 3.11 or newer. Package page: https://pypi.org/project/agentest-ai/
Upgrade
pip install -U agentest-aiIf you previously installed from git or from a clone, the command above switches you to the PyPI package.
Highlights
- First PyPI release of
agentest-ai3.5.0. The nameagenteston PyPI belongs to a different project. - 30 opt-in New tests (
NT-001–NT-030). Run withagentest std run --ids NTor list withagentest list --new-tests. The default run is still 130 checks (110 checklist + X-001–X-020). - Trace format 1.0.0: specification, JSON Schema, connectors and readers (including D2 aliases and D7/D8 record forms). Verdicts on existing traces are unchanged.
- Connectors for plain OpenTelemetry, LangChain/LangGraph, OpenAI Agents SDK, n8n, and other native-log dialects (steps/CrewAI, events, AutoGen, smolagents, file:).
What's Changed
Other changes
- Duplicate checks: infer user turns in imported native logs (T1-006/T1-011/T2-003) by @srpadmanabhuni in #169
- docs: label the 110-tests article link as 'the original 110 tests' by @srpadmanabhuni in #181
- docs: HF checklist is a practical subset of The original 110 tests by @srpadmanabhuni in #182
- docs: fix HF link in ABOUT.md (CI test_no_hf_links) by @srpadmanabhuni in #187
- fix(T2-006): normalize producer and consumer ID keys by @CyberKun in #168
- Phase 1: New tests NT-001–NT-009 (batch 1) by @srpadmanabhuni in #184
- Phase 1: New tests NT-010–NT-014, NT-027 (batch 2) by @srpadmanabhuni in #183
- Phase 1: New tests NT-020–NT-026, NT-030 (batch 4) by @srpadmanabhuni in #186
- Phase 1: New tests NT-015–NT-019, NT-028, NT-029 + T1-020/T1-021 (batch 3) by @srpadmanabhuni in #185
- Phase 1: New tests coverage counts and article-test mapping (docs roll-up) by @srpadmanabhuni in #188
- test: add T6-003 entity ID drift fixtures by @AdithyaKotian in #180
- fix: help improve detection: X-003 judges reason quality by lengt by @Mr-Neutr0n in #177
- docs: trace format 1.0.0 (D1-D9) and JSON Schema by @srpadmanabhuni in #190
- connectors: plain OTel / Python emitter (trace format 1.0.0) by @srpadmanabhuni in #191
- connectors: importer trace format 1.0.0 + LangChain/LangGraph round trip by @srpadmanabhuni in #192
- connectors: OpenAI Agents SDK round trip (trace format 1.0.0) by @srpadmanabhuni in #193
- connectors: n8n round trip (trace format 1.0.0) by @srpadmanabhuni in #194
- connectors: steps/CrewAI, events, autogen, smolagents, file: round trip (trace format 1.0.0) by @srpadmanabhuni in #195
- reader: trace format D2 aliases + D7 tools.json declarations (verdicts unchanged) by @srpadmanabhuni in #196
- reader: trace format D8 record forms - spans, span events, OTLP log records (verdicts unchanged) by @srpadmanabhuni in #197
- fix: NT-017 and NT-022 read trace-format canonical names by @srpadmanabhuni in #198
- docs: present New tests as shipped (remove Phase 1 wording) by @srpadmanabhuni in #199
- release: agentest-ai 3.5.0 (first PyPI package) by @srpadmanabhuni in #200
New Contributors
- @AdithyaKotian made their first contribution in #180
- @Mr-Neutr0n made their first contribution in #177
Full Changelog: v3.4.0...v3.5.0
agentest v3.3.2 (PyPI name agentest-ai)
Packaging release; rules and verdicts unchanged from v3.3.1.
- PyPI distribution name is now agentest-ai (
agenteston PyPI is an unrelated project). Import name and theagentestCLI are unchanged. - PyPI upload is pending. Until it is live, install from git:
pip install "agentest-ai @ git+https://github.com/srpadmanabhuni/agentest.git@v3.3.2", or install the wheel attached below. RELEASEgainspypi_name;scripts/check_versions.pychecks the pyproject name and every install line.scripts/publish_pypi.shuploads with twine usingPYPI_API_TOKEN(never printed).
Artifacts (twine check passed): agentest_ai-3.3.2-py3-none-any.whl, agentest_ai-3.3.2.tar.gz. PR #15.
agentest v3.3.1 (review fixes)
Review-fix release after an external correctness review of v3.3.0 and dataset v1.3.1. Rules and verdicts are unchanged from v3.3.0.
- Fix:
catalog/checklist-110.jsonlregenerated from dataset v1.3.1 (byte-identical to the Hub file). It had carried the pre-v1.2.1control_refsthe dataset retracted on all 110 rows.catalog_revisionis now v1.3.1. - CI: the catalogue pin check now covers the jsonl (sha256), cross-checks
control_refsagainst every pack CSV, and rejects retracted tokens. - CI: new
RELEASEfile is the single source of truth for version strings;scripts/check_versions.pyfails CI on drift (pyproject,__version__, install pins in README/docs, README coverage rows vsdocs/COVERAGE.md,catalog/VERSION, citation DOI) and--bumprewrites them. - Docs: stale install pins fixed; README coverage table current; legacy
catalog/t1_001.yamlarchived todocs/history/. - Citation: README and CITATION.cff cite the checklist's Zenodo all-versions concept DOI 10.5281/zenodo.23597877.
Install: pip install "agentest @ git+https://github.com/srpadmanabhuni/agentest.git@v3.3.1"
v3.3.0
agentest v3.3.0 — universal-rule precision
Seven fixes from the A99–A101 blind test (standard fields + documented governance extension only; no agent-specific rule code):
- T2-004 / T2-012 — only
governance.approval.required(MCPdestructiveHintfallback when absent) - T2-011 — write/destructive tool after deny only
- T3-011 — post-handoff orchestrator writes only
- T5-002 — domain
status=deniedis not a tool error - X-005 — honour
limits.max_steps; multi-agent default 24 - X-011 / X-012 — id + currency grounding against tool outputs
- X-014 — Unicode-script language consistency (any language)
Gates
- Examples regression floor ≥ 391/394
- Seed catch-rate 90/90
- 0 one-sided IDs
- Verdict snapshot regenerated (VERDICT-DIFF-3.3.0.md)
Blind tests
See docs/BLIND-TEST.md.
| Round | Agents | Result |
|---|---|---|
| 1 re-run | A99–A101 | FP→0; prior currency/id/language FNs caught |
| 2 fresh | A102–A104 | TP=3 FP=0 FN=0 (precision/recall 1.00) |
v3.2.1
agentest v3.2.1
Seed-aware catch-rate, tight published budgets on A90–A97, targeted re-seeds, T6-003 widen.
Catch-rate
- Before (v3.2.0 broken accounting): 63 “misses” counted every decided global TARGET pass on defect agents (including 35 IDs never seeded).
- Corrected before (same 90 declared seeds, v3.2.0 behaviour): 62/90 caught.
- After (v3.2.1): 90/90 declared seeds caught; 0 remaining misses.
Changes
- Seeds declared in AgentCard fixture extension
urn:agentest:a2a-ext:seeded-defects:v1(rules never read it). - A90–A97 publish tight
limitsso T5-006 / X-006 / X-007 fire. - A95 T4-003 (
notify_author+ injected email), A96 X-011 (OPS-4401), A97 T1-017 (distinctADJ-NNNNon twin submit). - T6-003 also checks numbers on slots/multi_turn (catches A92); no all-writes widen / no regressions.
- Verdict snapshot: 56 cells P→F (28 seeds × 2 modes); regress floor held 391/394.
Zero API cost (local Qwen2.5-7B).
