Skip to content

AgentProbe v1.0.0 — Initial Public Release

Choose a tag to compare

@chhayankjain chhayankjain released this 13 Sep 21:37
· 80 commits to main since this release
07d5a33

AgentProbe v1.0.0

Open-source fault-injection and observability toolkit for systematically evaluating the reliability of LLM-based intelligent agents.

Key Features

  • Five-category failure taxonomy — 16 failure sub-types covering tool call faults, orchestration errors, context degradation, latency anomalies, and output consistency violations
  • Configurable fault injectors — probabilistic injection with reproducible seeds
  • OpenTelemetry-based tracing — distributed tracing with OTLP export to Jaeger
  • 55 benchmark tasks — tool use (25), RAG (15), multi-agent coordination (15)
  • Framework support — LangGraph, LangChain, AutoGen

Benchmark Results (1,050 runs)

  • LangGraph/LangChain: 12.9% failure rate under injection, 79% recovery
  • AutoGen: 0% apparent failures — silent tool-execution bypass discovered
  • Statistically significant differences: p < 0.0001, Cohen's h = 0.73

Quick Start

pip install -e ".[ollama]"
agentprobe run --framework langgraph --task tool_use --failure-type timeout

Citation

See CITATION.cff or the "Cite this repository" button on the repo page.

Paper: Under review