Adaptive Spec-driven Scoring for Evaluation and Regression Testing
Local-first. Framework-agnostic. Trace-aware.
🚀 Get started | 🌐 Visit project website | 🔌 View supported targets | 📘 CLI Reference | 🧪 Examples
Most AI systems start with a specification: product requirements, policies, system prompts, or launch criteria describing what the system should and should not do.
But evaluation often starts elsewhere: generic scorers, predefined benchmarks, or manual test cases that drift from the original intent.
ASSERT closes that gap. It turns your specified behaviors in natural language into structured, executable evaluations that can be reviewed, run, scored, and improved over time.
From the natural language specification, the ASSERT pipeline derives behavior categories, generates single-turn and multi-turn test cases, inferences them against your target, and uses an LLM judge to score each conversation against your policies.
- Spec-driven coverage - test cases are generated from your product requirements and context, not a generic benchmark. You specify the behaviors that you want to test for
- Test any model endpoint via integrations with LiteLLM, supporting 100+ model endpoints from platform providers such as Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM.
- Test any agent or multi-agent system via integrations with OpenInference. Evaluate a LangGraph agent, a CrewAI / OpenAI Agents SDK / DSPy / LlamaIndex / AutoGen system, custom multi-agent orchestration, a Python callable, or a hosted model — without rewriting the evaluation orchestration pipeline.
- Agent trace-grounded judgment - the recommended integration captures OpenTelemetry spans (Phoenix/OpenInference auto-instruments 33+ frameworks in two lines, or you can emit your own with the OTel SDK) so the judge can cite tool calls, routing, model calls, and latency as evidence — not just the final response.
- Portable artifacts - every stage writes JSON/JSONL files locally for inspection, CI, and sharing.
- Bundled local viewer - browse runs side-by-side, pin a baseline, drill into per-behavior dimension breakdowns, and read judge justifications cited against the captured traces.
pip install -e ".[otel,langgraph]" # install
cp .env.example .env # add your provider key
assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml| 🌐 Project website ↗ | 📝 Technical blog ↗ | 🚀 Quickstart guide ↗ | 📚 Documentation ↗ |
|---|---|---|---|
| Learn about ASSERT | Read the Command Line post | Follow the full walkthrough | Browse concepts and guides |
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos is subject to those third party's policies.
This project does not collect or send telemetry to Microsoft by default. Runs write local artifacts under artifacts/results/, and optional OpenTelemetry trace capture is controlled by your configuration and local collector setup, such as Phoenix.
If you configure a target, judge, trace collector, or model provider to send data to an external service, the prompts, responses, traces, metadata, and other evaluation artifacts sent to that service are governed by that service's terms and your configuration.
See the full section in the Concept Doc.