Repository navigation
AgentProbe v1.0.0 — Initial Public Release
AgentProbe v1.0.0
Open-source fault-injection and observability toolkit for systematically evaluating the reliability of LLM-based intelligent agents.
Key Features
- Five-category failure taxonomy — 16 failure sub-types covering tool call faults, orchestration errors, context degradation, latency anomalies, and output consistency violations
- Configurable fault injectors — probabilistic injection with reproducible seeds
- OpenTelemetry-based tracing — distributed tracing with OTLP export to Jaeger
- 55 benchmark tasks — tool use (25), RAG (15), multi-agent coordination (15)
- Framework support — LangGraph, LangChain, AutoGen
Benchmark Results (1,050 runs)
- LangGraph/LangChain: 12.9% failure rate under injection, 79% recovery
- AutoGen: 0% apparent failures — silent tool-execution bypass discovered
- Statistically significant differences: p < 0.0001, Cohen's h = 0.73
Quick Start
pip install -e ".[ollama]"
agentprobe run --framework langgraph --task tool_use --failure-type timeoutCitation
See CITATION.cff or the "Cite this repository" button on the repo page.
Paper: Under review