Repository navigation
Releases
v1.5.0
A lighter install and sturdier assertions.
Upgrading from 1.4: if you use the benchmark runner, framework adapters, fault injectors, tracer or
python -m agentprobe.demo, installagentprobe-toolkit[research]. Without it they raise an ImportError naming the extra.agentprobe.testingneeds no extra, and import paths are unchanged.
pip install -U agentprobe-toolkit # probe(), scenarios, assertions: no other packages
pip install -U "agentprobe-toolkit[research]" # benchmark, adapters, injectors, tracer, demoHighlights
- No required dependencies.
agentprobe.testingruns on the standard library. Extras:[langchain],[observability],[research],[all]. - Answer checks that tolerate varied wording:
assert_answer_matches(pattern),assert_answer_not_matches(pattern)andassert_answer(check, description). - Tool-call assertions:
assert_called_with,assert_call_order,assert_called_times,assert_called_at_most. Arguments are compared exactly against a copy taken when the call started. - Injected timeouts are
TimeoutErrors, soexcept TimeoutErrorcatches them (#114). - OTLP over TLS:
https://endpoints now use TLS;http://and scheme-less endpoints stay plaintext.insecure=Truewithhttps://raisesValueError. - Strict typing enforced in CI, with corrected async injector types.
- Docs site: https://agentprobe-toolkit.pages.dev/
Thanks to @Zlongyang for the tool-call assertions (#117) and the TimeoutError fix (#116).
Full details in CHANGELOG.md.
What's Changed
- Example: fault tests on a real LLM agent (local Ollama) by @chhayankjain in #112
- Case study: AgentProbe finds an unhandled tool exception in a LangGraph tutorial agent by @chhayankjain in #113
- Move research material under research/ by @chhayankjain in #115
- Add tool-call assertions for arguments, order and counts by @Zlongyang in #117
- Let agents catch injected timeouts with TimeoutError by @Zlongyang in #116
- Tool-call assertions: compare exact argument copies, match by parameter name by @chhayankjain in #118
- Case study: injected timeouts are TimeoutError since #114 by @chhayankjain in #119
- Docs site: getting started, task guides, reference, llms.txt by @chhayankjain in #120
- Docs site: neutral theme, agentprobe-toolkit name, hosting on Cloudflare Pages by @chhayankjain in #121
- Bump mkdocs-material from 9.7.6 to 9.7.7 by @dependabot[bot] in #122
- README: professional layout and the docs address by @chhayankjain in #123
- No required dependencies: agentprobe.testing runs on the standard library by @chhayankjain in #124
- Answer assertions for varied wording; OTLP tracing over TLS by @chhayankjain in #125
- Strict type checking passes and is enforced in CI by @chhayankjain in #126
- Tests: release hung agents so pytest exits promptly by @chhayankjain in #127
- Release 1.5.0 by @chhayankjain in #128
Full Changelog: v1.4.0...v1.5.0
v1.4.0
What's Changed
- Rewrite roadmap around developer adoption by @chhayankjain in #93
- Test your own agent: probe(), deterministic scenarios, evidence that faults fired (v1.4) by @chhayankjain in #98
- Cleanup sweep: dead code, packaging, stale docs, stronger smoke checks by @chhayankjain in #99
- feat: add JSON output to the demo CLI by @Zlongyang in #100
- Request a maintainer review on every pull request by @chhayankjain in #101
- Roadmap: add #102–#106 by @chhayankjain in #107
- Point all links at the renamed repository by @chhayankjain in #108
- New social preview image by @chhayankjain in #109
- README: lead with the developer workflow by @chhayankjain in #110
- Release v1.4.0 by @chhayankjain in #111
New Contributors
- @Zlongyang made their first contribution in #100
Full Changelog: v1.3.0...v1.4.0
v1.3.0
What's Changed
- Replace ASCII architecture sketches with Mermaid diagrams by @chhayankjain in #58
- Provider-agnostic LLM backends with backend-error isolation (#26) by @chhayankjain in #59
- Add a no-model 60-second demo and social preview by @chhayankjain in #60
- Fix code-review findings (four review rounds) by @chhayankjain in #61
- Release 1.3.0 by @chhayankjain in #62
Full Changelog: v1.2.0...v1.3.0
v1.2.0
What's Changed
- Add CI/CD, security scanning, and open-source repo infrastructure by @chhayankjain in #23
- Consolidate public API and align docs with what the paper validated by @chhayankjain in #40
- Bump actions/setup-python from 5 to 7 by @dependabot[bot] in #42
- Bump github/codeql-action from 3 to 4 by @dependabot[bot] in #43
- Bump actions/cache from 4 to 6 by @dependabot[bot] in #44
- Bump actions/checkout from 4 to 7 by @dependabot[bot] in #45
- Prepare PyPI publishing as agentprobe-toolkit (v1.2.0) by @chhayankjain in #41
- Bump the github-actions group with 3 updates by @dependabot[bot] in #56
- Polish PyPI page: SPDX license expression and absolute README links by @chhayankjain in #57
New Contributors
- @dependabot[bot] made their first contribution in #42
Full Changelog: v1.1.0...v1.2.0
v1.1.0
What's Changed
Added
- Phase 2 experiment runner (
run_experiment_v2.py) with matched injection probability and answer-presence grading - Qualification runner (
run_qualification.py) for deterministic pipeline validation at p=1.0 - Model smoke test (
smoke_test_model.py) for quick tool-calling verification - AutoGen diagnostic (
autogen_diagnostic.py) comparing direct OpenAI client vs AutoGen shim - Cross-model analysis tooling (
analyze_cross_model.py) and data-driven figure generation - Checkpoint validation with exact record count, run ID range, and condition identity matching
- Resume safety: reject metadata mismatches (model, seed, n_runs) on experiment resume
- Full experiment data for Llama 3.1, Mistral 7B, and Qwen2.5 7B (Phase 2, qualification, diagnostic)
- ORCID identifiers in CITATION.cff
- CHANGELOG.md
Fixed
- AutoGen adapter: patch legacy
functions/function_callformat for Ollama compatibility - AutoGen output extraction: select correct assistant message instead of empty tool-call messages
- Numeric answer validation: word-boundary regex to prevent substring false positives
recovery_ratereturnsNone(not0.0) when no failures occur in a condition- P99 percentile: switched from floor-based to ceiling nearest-rank
Nonerecovery rate handling in all reporting scripts and plot generation- Pilot hung-invocation control flow and empty-output parsing
Changed
- Manuscript updated with two-phase study, cross-model validation, and MDPI/MAKE formatting
- Phase 1 latency ratio corrected from 7x to 11x
- Scope claims narrowed to experimentally validated fault categories
- Development status upgraded from Alpha to Beta
- Removed unused dependencies (httpx, typer) from core; moved pytest to dev, matplotlib/pandas to plotting
- Archived one-time utility scripts to
archive_v1/ - Deleted duplicate
experiments/results/data (kept inarchive_v1/)
Full Changelog: v1.0.0...v1.1.0
AgentProbe v1.0.0 — Initial Public Release
AgentProbe v1.0.0
Open-source fault-injection and observability toolkit for systematically evaluating the reliability of LLM-based intelligent agents.
Key Features
- Five-category failure taxonomy — 16 failure sub-types covering tool call faults, orchestration errors, context degradation, latency anomalies, and output consistency violations
- Configurable fault injectors — probabilistic injection with reproducible seeds
- OpenTelemetry-based tracing — distributed tracing with OTLP export to Jaeger
- 55 benchmark tasks — tool use (25), RAG (15), multi-agent coordination (15)
- Framework support — LangGraph, LangChain, AutoGen
Benchmark Results (1,050 runs)
- LangGraph/LangChain: 12.9% failure rate under injection, 79% recovery
- AutoGen: 0% apparent failures — silent tool-execution bypass discovered
- Statistically significant differences: p < 0.0001, Cohen's h = 0.73
Quick Start
pip install -e ".[ollama]"
agentprobe run --framework langgraph --task tool_use --failure-type timeoutCitation
See CITATION.cff or the "Cite this repository" button on the repo page.
Paper: Under review