Skip to content

v1.5.0 - Evaluatuions

Choose a tag to compare

View all tags
@g8rswimmer g8rswimmer released this 15 Jun 16:35
· 16 commits to main since this release
v1.5.0
7885771
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Overview

The Workflow Evaluation feature adds automated quality testing to cogniflow. Authors define eval suites containing test cases with fixed inputs, optional node mocks to stub out side effects, and graders that assert correctness of workflow or node output. Running a suite executes each test case as a real workflow run and scores every grader. Results are persisted, browsable in the UI, and linkable back to the underlying workflow run for node-level debugging.

What Was Built

ME1 — Data Foundation & CRUD API

Introduced the full eval data model and REST API for authoring suites and test cases.

  • Four new database tables: eval_suites, eval_test_cases, eval_runs, eval_test_case_results. No foreign key constraints; referential integrity enforced at the application layer in the correct cascade order.
  • Full CRUD for eval suites (GET/POST/PUT/DELETE /v1/workflows/{workflow_id}/eval-suites, GET/PUT/DELETE /v1/eval-suites/{suite_id}).
  • Full CRUD + reorder for test cases (/v1/eval-suites/{suite_id}/test-cases).
  • Each test case carries: initial_data (the fixed inputs fed to the workflow), mocks (per-node output overrides), and graders (assertions to evaluate after the run).
  • Save-time validation rejects invalid mock node IDs (checked against the live workflow), malformed regex patterns, and invalid JSON Schema objects before anything is stored.
  • LLM grader api_key values are encrypted at rest using the same AES-256-GCM vault already used for node sensitive config. The plain-text key is never returned via the API — responses always show "***".
  • Suite-level pass_threshold (0.0–1.0, default 1.0) and max_concurrency fields control pass rate aggregation and parallel test case execution.

ME2 — Eval Execution & Deterministic Graders

Made suites runnable. Added three deterministic grader types.

  • POST /v1/eval-suites/{suite_id}/runs — triggers an async eval run and returns the run ID immediately (same non-blocking pattern as workflow runs).
  • Each test case triggers one real workflow execution. The triggered_by field on the resulting workflow run is set to "eval" for traceability.
  • Node mock interception — when a test case defines a mock for a node, the eval engine bypasses Execute() and returns the mock output directly. The node still emits node.succeeded on the event bus (with "mocked": true in its output) so the event stream stays consistent and downstream nodes receive their expected input.
  • Node outputs are captured from node.succeeded events and stored in eval_test_case_results.node_outputs for use by node-scoped graders and the debug UI.
  • String Match grader — exact, contains, or regex match against a gjson field path on the target output. Invalid regex patterns are rejected at save time.
  • Numeric Threshold grader — compares a numeric field against a threshold using ==, !=, >, >=, <, or <=.
  • JSON Schema grader — validates the target output (or a field within it) against a JSON Schema draft-07 document using github.com/santhosh-tekuri/jsonschema/v5.
  • All graders support scope: workflow (the run's final merged output) or scope: node (a specific node's captured output).
  • Pass rate is computed per test case as passed_graders / total_graders. A test case with zero graders that completes without error has a pass rate of 1.0.
  • GET /v1/eval-runs/{run_id} returns the full run with all TestCaseResult and GraderResult entries.

ME3 — LLM Graders

Added AI-evaluated grader types for subjective or open-ended assertions.

  • LLM Judge grader — sends a rubric and the target value (a specific field or the full output as JSON) to an LLM and expects a structured {"verdict": "pass"|"fail", "explanation": "..."} response. Supports OpenAI and Anthropic providers. Provider and model are configurable per grader instance. If the LLM call fails or the response cannot be parsed, the verdict is "error" with the error detail in explanation — the eval run is not aborted.
  • Checklist grader — evaluates N independent criteria in a single LLM call. Returns a score (fraction of criteria met), criteria_results (per-criterion met boolean and judge explanation), and an overall pass/fail verdict based on a per-grader pass_threshold. Partial scores are stored and displayed.
  • Both LLM grader types use the existing aiprovider.LLMClient interface; no new provider abstraction was required.
  • Mixed providers within the same suite are supported (e.g., one grader uses OpenAI, another uses Anthropic).

ME4 — Frontend: Suite & Test Case Authoring

Full browser UI for creating and configuring eval suites and test cases.

  • A new Eval Suites tab on the workflow editor page lists all suites for the workflow with name, pass threshold, and test case count.
  • The suite detail page shows ordered test cases and a collapsible run history panel.
  • The TestCaseEditor slide-over handles all grader types through type-switched form fields — no hand-written JSON required:
    • String Match — field path, match type (exact / contains / regex), expected value.
    • Numeric Threshold — field path, operator, threshold.
    • LLM Judge — provider, model, api_key (password input), rubric, optional field path.
    • JSON Schema — optional field path, schema textarea.
    • Checklist — dynamic criterion list, per-grader pass threshold, provider/model/api_key/field_path.
  • The MockEditor provides a node selector (workflow nodes by label) and a JSON textarea for the mock output.
  • Initial data is rendered as an RJSF form when the workflow declares an initial_data_schema, and falls back to a plain JSON textarea otherwise.
  • Test cases are reorderable within a suite.
  • Save-time validation errors from the backend (invalid regex, unknown node ID in mock) surface inline per field.
  • LLM grader api_key fields are masked (***) when an existing grader is loaded for editing.

ME5 — Frontend: Run & Observe

Browser UI for triggering runs and inspecting results at every level of detail.

  • Run Suite button on the suite detail page triggers a new eval run and navigates to the run detail page.
  • The run detail page polls GET /v1/eval-runs/{id} every 2 seconds until the run reaches a terminal status (completed or failed). Polling survives transient API errors — a network hiccup reschedules the next poll rather than halting the chain.
  • A summary tile row shows total / passed / failed / error counts and run duration.
  • EvalRunResultsTable — expandable accordion rows, one per test case:
    • Collapsed: test case name, workflow run status chip, "View Run →" link, overall pass/fail verdict.
    • Expanded: per-grader result rows. If the underlying workflow run failed before graders could evaluate, a red banner explains this.
  • GraderResultRow — verdict icon (green ✓ / red ✗ / amber ! for error), grader name, type chip, explanation text, and a collapsible "show value / hide value" toggle exposing the raw value the grader inspected.
  • ChecklistResultDetail — rendered inside the grader row without an additional click; shows a per-criterion table with met/unmet badges and the judge's explanation for each, plus a score summary ("4 of 5 criteria met — 80%").
  • Run History accordion on the suite detail page lists past runs with short run ID, timestamp, duration, pass summary, and status badge. Fetches lazily when opened.
  • "View Run →" link navigates to the existing workflow run detail page, showing node-level status and output for the exact execution the eval captured.

API Reference (summary)

Method Path Description
GET /v1/workflows/{workflow_id}/eval-suites List suites for a workflow
POST /v1/workflows/{workflow_id}/eval-suites Create suite
GET /v1/eval-suites/{suite_id} Get suite
PUT /v1/eval-suites/{suite_id} Update suite
DELETE /v1/eval-suites/{suite_id} Delete suite and all data
GET /v1/eval-suites/{suite_id}/test-cases List test cases
POST /v1/eval-suites/{suite_id}/test-cases Create test case
GET /v1/eval-suites/{suite_id}/test-cases/{case_id} Get test case
PUT /v1/eval-suites/{suite_id}/test-cases/{case_id} Replace test case
DELETE /v1/eval-suites/{suite_id}/test-cases/{case_id} Delete test case
PUT /v1/eval-suites/{suite_id}/test-cases/order Reorder test cases
POST /v1/eval-suites/{suite_id}/runs Trigger a new eval run
GET /v1/eval-suites/{suite_id}/runs List eval runs (?status=&limit=&offset=)
GET /v1/eval-runs/{eval_run_id} Get full run with all results
GET /v1/eval-runs/{eval_run_id}/test-case-results/{result_id} Get single result with node_outputs

Grader Types

Type Config fields Verdict logic
string_match field_path, match_type (exact / contains / regex), expected_value Coerces resolved value to string; compares
numeric_threshold field_path, operator (==, !=, >, >=, <, <=), threshold Requires numeric field; evaluates comparison
json_schema field_path (optional), schema Validates against JSON Schema draft-07
llm_judge provider, model, api_key, rubric, field_path (optional) LLM returns {"verdict":"pass"|"fail","explanation":"..."}
checklist provider, model, api_key, criteria (array), pass_threshold, field_path (optional) LLM evaluates each criterion; score = met/total; pass if score ≥ threshold

All graders support scope: workflow or scope: node. Node-scoped graders require a node_id; if that node did not execute, the verdict is error.

Database Migrations

Migration Description
0012_create_eval_tables Creates eval_suites, eval_test_cases, eval_runs, eval_test_case_results

Migrations run automatically on server startup. No manual steps required.

Configuration

No new environment variables are required. LLM provider credentials for graders are stored per-grader in the test case definition (encrypted at rest). The server's existing COGNIFLOW_ENCRYPTION_KEY is used for grader api_key encryption.

Known Limitations (v1)

  • Eval runs are triggered manually only. Scheduled runs (cron-triggered evals) are not yet supported.
  • There is no CI webhook for triggering evals from a pipeline; POST /v1/eval-suites/{id}/runs can be called from scripts with a tool like curl.
  • Dataset import (generating test cases from CSV or JSONL) is not supported.
  • Baseline comparison (diffing two eval run results to surface regressions) is not supported.
  • Eval result streaming via WebSocket is not supported; results are fetched by polling.
  • Custom grader plugins via gRPC are not supported.
Assets 2
Loading