Repository navigation
A lighter install and sturdier assertions.
Upgrading from 1.4: if you use the benchmark runner, framework adapters, fault injectors, tracer or
python -m agentprobe.demo, installagentprobe-toolkit[research]. Without it they raise an ImportError naming the extra.agentprobe.testingneeds no extra, and import paths are unchanged.
pip install -U agentprobe-toolkit # probe(), scenarios, assertions: no other packages
pip install -U "agentprobe-toolkit[research]" # benchmark, adapters, injectors, tracer, demoHighlights
- No required dependencies.
agentprobe.testingruns on the standard library. Extras:[langchain],[observability],[research],[all]. - Answer checks that tolerate varied wording:
assert_answer_matches(pattern),assert_answer_not_matches(pattern)andassert_answer(check, description). - Tool-call assertions:
assert_called_with,assert_call_order,assert_called_times,assert_called_at_most. Arguments are compared exactly against a copy taken when the call started. - Injected timeouts are
TimeoutErrors, soexcept TimeoutErrorcatches them (#114). - OTLP over TLS:
https://endpoints now use TLS;http://and scheme-less endpoints stay plaintext.insecure=Truewithhttps://raisesValueError. - Strict typing enforced in CI, with corrected async injector types.
- Docs site: https://agentprobe-toolkit.pages.dev/
Thanks to @Zlongyang for the tool-call assertions (#117) and the TimeoutError fix (#116).
Full details in CHANGELOG.md.
What's Changed
- Example: fault tests on a real LLM agent (local Ollama) by @chhayankjain in #112
- Case study: AgentProbe finds an unhandled tool exception in a LangGraph tutorial agent by @chhayankjain in #113
- Move research material under research/ by @chhayankjain in #115
- Add tool-call assertions for arguments, order and counts by @Zlongyang in #117
- Let agents catch injected timeouts with TimeoutError by @Zlongyang in #116
- Tool-call assertions: compare exact argument copies, match by parameter name by @chhayankjain in #118
- Case study: injected timeouts are TimeoutError since #114 by @chhayankjain in #119
- Docs site: getting started, task guides, reference, llms.txt by @chhayankjain in #120
- Docs site: neutral theme, agentprobe-toolkit name, hosting on Cloudflare Pages by @chhayankjain in #121
- Bump mkdocs-material from 9.7.6 to 9.7.7 by @dependabot[bot] in #122
- README: professional layout and the docs address by @chhayankjain in #123
- No required dependencies: agentprobe.testing runs on the standard library by @chhayankjain in #124
- Answer assertions for varied wording; OTLP tracing over TLS by @chhayankjain in #125
- Strict type checking passes and is enforced in CI by @chhayankjain in #126
- Tests: release hung agents so pytest exits promptly by @chhayankjain in #127
- Release 1.5.0 by @chhayankjain in #128
Full Changelog: v1.4.0...v1.5.0