Skip to content

Releases

v1.5.0

Choose a tag to compare

@github-actions github-actions released this 20 Nov 17:27

A lighter install and sturdier assertions.

Upgrading from 1.4: if you use the benchmark runner, framework adapters, fault injectors, tracer or python -m agentprobe.demo, install agentprobe-toolkit[research]. Without it they raise an ImportError naming the extra. agentprobe.testing needs no extra, and import paths are unchanged.

pip install -U agentprobe-toolkit            # probe(), scenarios, assertions: no other packages
pip install -U "agentprobe-toolkit[research]" # benchmark, adapters, injectors, tracer, demo

Highlights

  • No required dependencies. agentprobe.testing runs on the standard library. Extras: [langchain], [observability], [research], [all].
  • Answer checks that tolerate varied wording: assert_answer_matches(pattern), assert_answer_not_matches(pattern) and assert_answer(check, description).
  • Tool-call assertions: assert_called_with, assert_call_order, assert_called_times, assert_called_at_most. Arguments are compared exactly against a copy taken when the call started.
  • Injected timeouts are TimeoutErrors, so except TimeoutError catches them (#114).
  • OTLP over TLS: https:// endpoints now use TLS; http:// and scheme-less endpoints stay plaintext. insecure=True with https:// raises ValueError.
  • Strict typing enforced in CI, with corrected async injector types.
  • Docs site: https://agentprobe-toolkit.pages.dev/

Thanks to @Zlongyang for the tool-call assertions (#117) and the TimeoutError fix (#116).

Full details in CHANGELOG.md.

What's Changed

Full Changelog: v1.4.0...v1.5.0

v1.4.0

Choose a tag to compare

@github-actions github-actions released this 18 Nov 06:24

What's Changed

New Contributors

Full Changelog: v1.3.0...v1.4.0

v1.3.0

Choose a tag to compare

@github-actions github-actions released this 17 Nov 12:10
d8d15cf

What's Changed

Full Changelog: v1.2.0...v1.3.0

v1.2.0

Choose a tag to compare

@github-actions github-actions released this 16 Nov 04:14

What's Changed

  • Add CI/CD, security scanning, and open-source repo infrastructure by @chhayankjain in #23
  • Consolidate public API and align docs with what the paper validated by @chhayankjain in #40
  • Bump actions/setup-python from 5 to 7 by @dependabot[bot] in #42
  • Bump github/codeql-action from 3 to 4 by @dependabot[bot] in #43
  • Bump actions/cache from 4 to 6 by @dependabot[bot] in #44
  • Bump actions/checkout from 4 to 7 by @dependabot[bot] in #45
  • Prepare PyPI publishing as agentprobe-toolkit (v1.2.0) by @chhayankjain in #41
  • Bump the github-actions group with 3 updates by @dependabot[bot] in #56
  • Polish PyPI page: SPDX license expression and absolute README links by @chhayankjain in #57

New Contributors

Full Changelog: v1.1.0...v1.2.0

v1.1.0

Choose a tag to compare

@chhayankjain chhayankjain released this 14 Nov 10:31

What's Changed

Added

  • Phase 2 experiment runner (run_experiment_v2.py) with matched injection probability and answer-presence grading
  • Qualification runner (run_qualification.py) for deterministic pipeline validation at p=1.0
  • Model smoke test (smoke_test_model.py) for quick tool-calling verification
  • AutoGen diagnostic (autogen_diagnostic.py) comparing direct OpenAI client vs AutoGen shim
  • Cross-model analysis tooling (analyze_cross_model.py) and data-driven figure generation
  • Checkpoint validation with exact record count, run ID range, and condition identity matching
  • Resume safety: reject metadata mismatches (model, seed, n_runs) on experiment resume
  • Full experiment data for Llama 3.1, Mistral 7B, and Qwen2.5 7B (Phase 2, qualification, diagnostic)
  • ORCID identifiers in CITATION.cff
  • CHANGELOG.md

Fixed

  • AutoGen adapter: patch legacy functions/function_call format for Ollama compatibility
  • AutoGen output extraction: select correct assistant message instead of empty tool-call messages
  • Numeric answer validation: word-boundary regex to prevent substring false positives
  • recovery_rate returns None (not 0.0) when no failures occur in a condition
  • P99 percentile: switched from floor-based to ceiling nearest-rank
  • None recovery rate handling in all reporting scripts and plot generation
  • Pilot hung-invocation control flow and empty-output parsing

Changed

  • Manuscript updated with two-phase study, cross-model validation, and MDPI/MAKE formatting
  • Phase 1 latency ratio corrected from 7x to 11x
  • Scope claims narrowed to experimentally validated fault categories
  • Development status upgraded from Alpha to Beta
  • Removed unused dependencies (httpx, typer) from core; moved pytest to dev, matplotlib/pandas to plotting
  • Archived one-time utility scripts to archive_v1/
  • Deleted duplicate experiments/results/ data (kept in archive_v1/)

Full Changelog: v1.0.0...v1.1.0

AgentProbe v1.0.0 — Initial Public Release

Choose a tag to compare

@chhayankjain chhayankjain released this 13 Sep 21:37
07d5a33

AgentProbe v1.0.0

Open-source fault-injection and observability toolkit for systematically evaluating the reliability of LLM-based intelligent agents.

Key Features

  • Five-category failure taxonomy — 16 failure sub-types covering tool call faults, orchestration errors, context degradation, latency anomalies, and output consistency violations
  • Configurable fault injectors — probabilistic injection with reproducible seeds
  • OpenTelemetry-based tracing — distributed tracing with OTLP export to Jaeger
  • 55 benchmark tasks — tool use (25), RAG (15), multi-agent coordination (15)
  • Framework support — LangGraph, LangChain, AutoGen

Benchmark Results (1,050 runs)

  • LangGraph/LangChain: 12.9% failure rate under injection, 79% recovery
  • AutoGen: 0% apparent failures — silent tool-execution bypass discovered
  • Statistically significant differences: p < 0.0001, Cohen's h = 0.73

Quick Start

pip install -e ".[ollama]"
agentprobe run --framework langgraph --task tool_use --failure-type timeout

Citation

See CITATION.cff or the "Cite this repository" button on the repo page.

Paper: Under review