Skip to content

v1.5.0

Latest

Choose a tag to compare

@github-actions github-actions released this 20 Nov 17:27

A lighter install and sturdier assertions.

Upgrading from 1.4: if you use the benchmark runner, framework adapters, fault injectors, tracer or python -m agentprobe.demo, install agentprobe-toolkit[research]. Without it they raise an ImportError naming the extra. agentprobe.testing needs no extra, and import paths are unchanged.

pip install -U agentprobe-toolkit            # probe(), scenarios, assertions: no other packages
pip install -U "agentprobe-toolkit[research]" # benchmark, adapters, injectors, tracer, demo

Highlights

  • No required dependencies. agentprobe.testing runs on the standard library. Extras: [langchain], [observability], [research], [all].
  • Answer checks that tolerate varied wording: assert_answer_matches(pattern), assert_answer_not_matches(pattern) and assert_answer(check, description).
  • Tool-call assertions: assert_called_with, assert_call_order, assert_called_times, assert_called_at_most. Arguments are compared exactly against a copy taken when the call started.
  • Injected timeouts are TimeoutErrors, so except TimeoutError catches them (#114).
  • OTLP over TLS: https:// endpoints now use TLS; http:// and scheme-less endpoints stay plaintext. insecure=True with https:// raises ValueError.
  • Strict typing enforced in CI, with corrected async injector types.
  • Docs site: https://agentprobe-toolkit.pages.dev/

Thanks to @Zlongyang for the tool-call assertions (#117) and the TimeoutError fix (#116).

Full details in CHANGELOG.md.

What's Changed

Full Changelog: v1.4.0...v1.5.0