Skip to content

Releases: MrRex168/agent-reliability-toolkit

Release list

AI Agent Reliability Toolkit v1.0.1

AI Agent Reliability Toolkit v1.0.1 Latest
Latest

Choose a tag to compare

View all tags
@MrRex168 MrRex168 released this 11 Sep 09:26
v1.0.1
283548c

AI Agent Reliability Toolkit v1.0.1

A small maintenance release focused on reliability correctness, repository polish, and developer discoverability.

Fixed

  • Regex failure classification — regex mismatches and invalid regular expressions are now classified as OUTPUT_MISMATCH instead of falling through to UNKNOWN.
  • Added regression tests covering regex mismatch and invalid-regex failure classification.

Improved

  • Clearer README positioning and quick-start experience
  • Added common AI agent reliability use cases
  • Improved GitHub and package search discoverability
  • Added Python package keywords, classifiers, and project URLs
  • Added pull request guidance
  • Improved issue routing to GitHub Discussions
  • Added a security policy
  • Improved repository topics and release documentation
  • Polished the dashboard and onboarding experience

Release focus

This is intentionally a small post-v1.0 maintenance release.

No major new product surface, cloud infrastructure, authentication, billing, or unnecessary dependencies were added.

The priority remains simple:

Test AI agents repeatedly. Find intermittent failures. Catch regressions before production.

Assets 2
Loading
0 Join discussion

AI Agent Reliability Toolkit v1.0.0

AI Agent Reliability Toolkit v1.0.0

Choose a tag to compare

View all tags
@MrRex168 MrRex168 released this 07 Sep 05:14
v1.0.0
714f744

AI Agent Reliability Toolkit v1.0.0

Your AI agent passed the demo. But will it pass 100 runs?

The first production-ready open-source release of AI Agent Reliability Toolkit.

Test AI agents repeatedly, expose intermittent failures, understand why they fail, compare versions, and catch regressions before production.

What's included

  • Repeated AI agent evaluation — catch failures that one-off demos hide
  • Reliability and consistency scoring — measure stochastic agent behavior across repeated runs
  • Failure classification and diagnostics — understand why an agent failed, not just that it failed
  • Structured output evaluation — validate required keys and exact JSON/object values
  • Regex evaluation — test outputs against expected patterns
  • LLM-as-a-judge — provider-agnostic semantic evaluation with configurable criteria and thresholds
  • Tool-call evaluation — detect wrong tools, arguments, call order, and execution failures
  • MCP trace evaluation — normalize and evaluate MCP-style tool calls without requiring an MCP SDK
  • Regression testing — compare agent versions and detect reliability drops
  • CI quality gates — fail CI when an agent gets worse
  • LangChain and LangGraph adapters — evaluate existing agent workflows with minimal integration
  • OpenTelemetry integration — optional tracing without locking into an observability vendor
  • Evaluation history — store results locally using SQLite
  • Historical comparison — compare reliability across agent versions
  • Local reliability dashboard — inspect scores, trends, failures, and individual runs
  • JSON reports — machine-readable results for CI and custom workflows

30-second quick start

Requires Python 3.10+.

git clone https://github.com/MrRex168/agent-reliability-toolkit.git
cd agent-reliability-toolkit

python -m venv .venv
source .venv/bin/activate

pip install -e .

agent-reliability eval examples/unreliable_cases.yaml --runs 20

The included demo agent intentionally fails intermittently, making it easy to see why repeated evaluation matters.

See the dashboard

pip install -e '.[dashboard]'
agent-reliability dashboard

Use the local dashboard to inspect reliability scores, task success, failed runs, failure categories, historical trends, and version comparisons.

Built for practical agent reliability testing

The toolkit is designed to stay:

  • Local-first
  • Open source
  • Provider-agnostic
  • Dependency-minimal
  • CI-friendly
  • Easy to understand and extend

No cloud account is required.

Feedback & contributions

This release establishes the open-source foundation.

The next priority is learning from real AI agent workflows: what developers are testing, where their agents fail, and which reliability problems matter most in production.

Bug reports, feature requests, real-world evaluation cases, and contributions are welcome.

Build the agent. Test it repeatedly. Know whether it actually works.

Loading