Non-invasive CLI testing framework for AI prompts, agents, and workflows.
trap treats any solution as a black box — it invokes it as a subprocess, captures outputs, then optionally scores them through a language-agnostic judge and grader. The solution doesn't need to import trap or know it exists.
# requires uv — https://docs.astral.sh/uv/getting-started/installation/
# from PyPI
uv tool install trap-cli
# from git (latest main)
uv tool install "git+https://github.com/trapstreet/trap.git"The command is tp.
trap runs your solution as a subprocess with one env var (TRAP_MANIFEST — an input dir and
an output dir), captures what it writes, then optionally scores it through a judge (per case)
and a grader (overall). See the full flow and IO contract.
Two roles, two directories, one IO contract:
- Solution author — writes
trap.yamland the solution code - Task author — writes
traptask.yaml,inputs/,expected/, and optional judge/grader scripts
# from examples/echo/solution/
tp run # run all cases
tp run -t smoke # run only cases tagged `smoke`MIT