Skip to content

v0.2.0 — Agent behavior testing

Latest

Choose a tag to compare

@Kareem-Rashed Kareem-Rashed released this 12 Jun 20:48

Rubric 0.2 repositions around its core strength: testing agent behavior — what your agent did, not just what it said.

✨ New

  • LangGraph / LangChain auto-capturerun_langgraph(agent, scenarios) runs your agent and extracts tool calls, arguments, outputs, errors, the reasoning trace, latency, and token usage automatically. from_langgraph(result) converts an existing invoke() result; from_messages() works on any OpenAI-format message list, no langchain install required.
  • Regression diffingrubric compare current.json --baseline baseline.json classifies pass→fail regressions, score drops, fixes, and new/removed tests, with failing-metric reasons inline. --fail-on-regression for CI.
  • GitHub Action — runs your evals on every PR and posts the regression diff as a comment:
    uses: Kareem-Rashed/rubric-eval@v0.2.0

Also in this release (previously unreleased)

  • HallucinationScore metric (LLM judge + NLI modes)
  • capture() / @track for zero-friction call recording
  • LangFuse & LangSmith trace importers
  • Per-test metrics and LLM-judge flakiness detection

pip install rubric-eval — zero required dependencies, fully local, MIT.
Try the no-API-key demo: python examples/langgraph_eval.py