Skip to content

v3.5.2: watch agentest catch a prompt injection in one command

Latest

Choose a tag to compare

@srpadmanabhuni srpadmanabhuni released this 21 Nov 07:37
3aef310

One command, and you watch agentest catch an agent following a prompt injection

A real 8-second terminal run of pip install agentest-ai followed by agentest demo --agent a28, which reports 7 failed checks including T4-001

Install from PyPI:

pip install agentest-ai

Then run:

agentest demo --agent a28

a28 is a bundled reference agent recorded on a real run. A planted URGENT instruction in its input surfaced in what it did next, and agentest flags it under T4-001, T4-009 and T4-010. Plain agentest demo judges the bundled overreach sample. Add --html report.html to get a report you can click through. It runs offline, with no API key and no trace file of your own.

New

  • agentest demo: judges a bundled sample trace and prints pass/fail/n/a counts with a few example findings. --agent picks another bundled agent, --html writes a report, and --no-color turns off colour.
  • How-to pages for LangGraph, CrewAI, OpenAI Agents SDK, AutoGen, smolagents, n8n and MCP under docs/integrations/.
  • llms.txt and llms-full.txt, generated from the README, the CLI flags and every test ID.

Fixed

  • T1-005 / X-008 (#114): a short negation such as "no" cancels a success claim only within its own sentence, and a failed step that an identical retry redid successfully no longer fails T1-005. Bundled verdicts: T1-002 moves from pass to fail for a22 (both modes) and a28 (examples).

Unchanged

165 tests. Apart from the T1-005 fix above, no rule, scorer or label change.

Also new on main since this tag: an agentic evals challenge with 20 example traces for contributor week (23 to 30 November), the Wall of caught failures, a 60-second Colab notebook and the agentest GitHub Action.

Full changelog: CHANGELOG.md · Package: agentest-ai on PyPI · Getting started: Tutorials