Skip to content

AI Agent Reliability Toolkit v1.0.0

Choose a tag to compare

View all tags
@MrRex168 MrRex168 released this 07 Sep 05:14
· 28 commits to main since this release
v1.0.0
714f744

AI Agent Reliability Toolkit v1.0.0

Your AI agent passed the demo. But will it pass 100 runs?

The first production-ready open-source release of AI Agent Reliability Toolkit.

Test AI agents repeatedly, expose intermittent failures, understand why they fail, compare versions, and catch regressions before production.

What's included

  • Repeated AI agent evaluation — catch failures that one-off demos hide
  • Reliability and consistency scoring — measure stochastic agent behavior across repeated runs
  • Failure classification and diagnostics — understand why an agent failed, not just that it failed
  • Structured output evaluation — validate required keys and exact JSON/object values
  • Regex evaluation — test outputs against expected patterns
  • LLM-as-a-judge — provider-agnostic semantic evaluation with configurable criteria and thresholds
  • Tool-call evaluation — detect wrong tools, arguments, call order, and execution failures
  • MCP trace evaluation — normalize and evaluate MCP-style tool calls without requiring an MCP SDK
  • Regression testing — compare agent versions and detect reliability drops
  • CI quality gates — fail CI when an agent gets worse
  • LangChain and LangGraph adapters — evaluate existing agent workflows with minimal integration
  • OpenTelemetry integration — optional tracing without locking into an observability vendor
  • Evaluation history — store results locally using SQLite
  • Historical comparison — compare reliability across agent versions
  • Local reliability dashboard — inspect scores, trends, failures, and individual runs
  • JSON reports — machine-readable results for CI and custom workflows

30-second quick start

Requires Python 3.10+.

git clone https://github.com/MrRex168/agent-reliability-toolkit.git
cd agent-reliability-toolkit

python -m venv .venv
source .venv/bin/activate

pip install -e .

agent-reliability eval examples/unreliable_cases.yaml --runs 20

The included demo agent intentionally fails intermittently, making it easy to see why repeated evaluation matters.

See the dashboard

pip install -e '.[dashboard]'
agent-reliability dashboard

Use the local dashboard to inspect reliability scores, task success, failed runs, failure categories, historical trends, and version comparisons.

Built for practical agent reliability testing

The toolkit is designed to stay:

  • Local-first
  • Open source
  • Provider-agnostic
  • Dependency-minimal
  • CI-friendly
  • Easy to understand and extend

No cloud account is required.

Feedback & contributions

This release establishes the open-source foundation.

The next priority is learning from real AI agent workflows: what developers are testing, where their agents fail, and which reliability problems matter most in production.

Bug reports, feature requests, real-world evaluation cases, and contributions are welcome.

Build the agent. Test it repeatedly. Know whether it actually works.

Assets 2
Loading