Rubric 0.2 repositions around its core strength: testing agent behavior — what your agent did, not just what it said.
✨ New
- LangGraph / LangChain auto-capture —
run_langgraph(agent, scenarios)runs your agent and extracts tool calls, arguments, outputs, errors, the reasoning trace, latency, and token usage automatically.from_langgraph(result)converts an existinginvoke()result;from_messages()works on any OpenAI-format message list, no langchain install required. - Regression diffing —
rubric compare current.json --baseline baseline.jsonclassifies pass→fail regressions, score drops, fixes, and new/removed tests, with failing-metric reasons inline.--fail-on-regressionfor CI. - GitHub Action — runs your evals on every PR and posts the regression diff as a comment:
uses: Kareem-Rashed/rubric-eval@v0.2.0
Also in this release (previously unreleased)
HallucinationScoremetric (LLM judge + NLI modes)capture()/@trackfor zero-friction call recording- LangFuse & LangSmith trace importers
- Per-test metrics and LLM-judge flakiness detection
pip install rubric-eval — zero required dependencies, fully local, MIT.
Try the no-API-key demo: python examples/langgraph_eval.py