Skip to content

ASSERT v0.1.0 Initial release

Latest
Latest

Choose a tag to compare

View all tags
@tangym tangym released this 02 Jun 17:51
· 7 commits to main since this release
v0.1.0
78d35e0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ASSERT v0.1.0 Initial release

ASSERT stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing.

Local-first. Framework-agnostic. Trace-aware.

ASSERT turns your specified behaviors in natural language into structured, executable evaluations that can be reviewed, run, scored, and improved over time. From the natural language specification, the ASSERT pipeline derives behavior categories, generates single-turn and multi-turn test cases, inferences them against your target, and uses an LLM judge to score each conversation against your policies.

Install

pip install assert-ai
# optional extras
pip install "assert-ai[otel]"        # OpenTelemetry + Phoenix trace capture
pip install "assert-ai[langgraph]"   # LangGraph target adapter

Supports Python 3.11, 3.12, and 3.13.

Quickstart

assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml

Follow the full walkthrough: https://github.com/responsibleai/ASSERT/blob/main/docs/getting-started.md

What you get with ASSERT

  • Spec-driven coverage - test cases are generated from your product requirements and context, not a generic benchmark. You specify the behaviors that you want to test for
  • Test any model endpoint via integrations with LiteLLM, supporting 100+ model endpoints from platform providers such as Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM.
  • Test any agent or multi-agent system via integrations with OpenInference. Evaluate a LangGraph agent, a CrewAI / OpenAI Agents SDK / DSPy / LlamaIndex / AutoGen system, custom multi-agent orchestration, a Python callable, or a hosted model — without rewriting the evaluation orchestration pipeline.
  • Agent trace-grounded judgment - the recommended integration captures OpenTelemetry spans (Phoenix/OpenInference auto-instruments 33+ frameworks in two lines, or you can emit your own with the OTel SDK) so the judge can cite tool calls, routing, model calls, and latency as evidence — not just the final response.
  • Portable artifacts - every stage writes JSON/JSONL files locally for inspection, CI, and sharing.
  • Bundled local viewer - browse runs side-by-side, pin a baseline, drill into per-behavior dimension breakdowns, and read judge justifications cited against the captured traces.

Links

Risks and limitations

See the Concept Doc for the full risks and limitations section.

Telemetry

This project does not collect or send telemetry to Microsoft by default. Runs write local artifacts under artifacts/results/, and optional OpenTelemetry trace capture is controlled by your configuration and local collector setup, such as Phoenix.

Feedback

Issues and feedback welcome at https://github.com/responsibleai/ASSERT/issues.

Assets 4
Loading