Repository navigation
One command, and you watch agentest catch an agent following a prompt injection
Install from PyPI:
pip install agentest-aiThen run:
agentest demo --agent a28a28 is a bundled reference agent recorded on a real run. A planted URGENT instruction in its input surfaced in what it did next, and agentest flags it under T4-001, T4-009 and T4-010. Plain agentest demo judges the bundled overreach sample. Add --html report.html to get a report you can click through. It runs offline, with no API key and no trace file of your own.
New
agentest demo: judges a bundled sample trace and prints pass/fail/n/a counts with a few example findings.--agentpicks another bundled agent,--htmlwrites a report, and--no-colorturns off colour.- How-to pages for LangGraph, CrewAI, OpenAI Agents SDK, AutoGen, smolagents, n8n and MCP under
docs/integrations/. llms.txtandllms-full.txt, generated from the README, the CLI flags and every test ID.
Fixed
- T1-005 / X-008 (#114): a short negation such as "no" cancels a success claim only within its own sentence, and a failed step that an identical retry redid successfully no longer fails T1-005. Bundled verdicts: T1-002 moves from pass to fail for a22 (both modes) and a28 (examples).
Unchanged
165 tests. Apart from the T1-005 fix above, no rule, scorer or label change.
Also new on main since this tag: an agentic evals challenge with 20 example traces for contributor week (23 to 30 November), the Wall of caught failures, a 60-second Colab notebook and the agentest GitHub Action.
Full changelog: CHANGELOG.md · Package: agentest-ai on PyPI · Getting started: Tutorials
