v1.13.1

What's Changed

• Fix infinite reload loop on login page by @g8rswimmer in #41

Full Changelog: v1.13.0...v1.13.1

v1.13.0

What's Changed

• F03 auth by @g8rswimmer in #40

Full Changelog: v1.12.0...v1.13.0

v1.12.0

What's Changed

• F02 message queue trigger by @g8rswimmer in #39

Full Changelog: v1.11.0...v1.12.0

v1.11.0

What's Changed

• F04 workflow versioning by @g8rswimmer in #38

Full Changelog: v1.10.0...v1.11.0

v1.10.0

What's Changed

• Ef06 eval result streaming by @g8rswimmer in #36
• Ef05 custom grader plugins by @g8rswimmer in #37

Full Changelog: v1.9.0...v1.10.0

v1.9.0

What's Changed

• updated requirements by @g8rswimmer in #33
• F01 workflow loops by @g8rswimmer in #34
• Ef03 dataset import by @g8rswimmer in #35

Full Changelog: v1.8.0...v1.9.0

v1.8.0

What this does

Adds the ability to diff two completed eval runs from the same suite, surfacing regressions and improvements test-case by test-case. This is the EF-04 future consideration from REQUIREMENTS_EVAL.md.

No new database tables are needed — the comparison is computed in-memory from existing TestCaseResult rows using two calls to ListTestCaseResults.

---

Backend

A new endpoint is registered between the existing GetRun and GetTestCaseResult routes:

GET /v1/eval-runs/{eval_run_id}/compare?baseline_run_id={id}

Both runs must be completed and belong to the same eval suite. The handler fetches TestCaseResult rows for each run, builds lookup maps keyed by test_case_id, then iterates the union to classify each case as one of: regressed (passed → failed), improved (failed → passed), unchanged, new_case (head only), or missing (baseline only). Results are sorted with regressions first, then improvements, then new cases, missing cases, and unchanged — with TestCaseName and TestCaseID as tiebreakers for fully deterministic ordering.

The response includes aggregate counts (regressed_count, improved_count, etc.) alongside the per-case breakdown.

A new migration (0015) adds a UNIQUE KEY (eval_run_id, test_case_id) constraint to eval_test_case_results. This prevents a runner bug from inserting duplicate result rows that would cause non-deterministic comparison output.

---

Frontend

EvalRunDetailPage gains three new behaviours when the head run is completed:

A baseline selector dropdown appears above the summary panel, listing all other completed runs from the same suite. Selecting one updates the URL as ?baseline_run_id=, making the comparison deep-linkable and bookmarkable. Clearing the selection removes the parameter.

A delta stats banner renders between the summary panel and the results table showing the aggregate counts (-2 regressed, +1 improved, 3 unchanged, etc.).

Per-row change badges appear in EvalRunResultsTable alongside the existing pass/fail indicator — color-coded red for regressed, green for improved, gray for unchanged, indigo for new, and amber for missing.

WorkflowListPage gains an Evals button on each workflow row, linking directly to that workflow's eval suite list.

---

Code review fixes applied

After an initial implementation review, the following issues were addressed in a follow-up commit:

Stale fetch cancellation. The compare fetch effect now carries an alive flag (matching the pattern used by the existing poll effect). When the user clears the baseline selector before the in-flight fetch resolves, the response is silently dropped rather than overwriting the cleared state.

Failed run guard. The frontend previously fired compareEvalRuns for both completed and failed head runs. The backend rejects anything that is not completed, so this always produced a confusing error banner. The guard is now run?.status === 'completed' only.

Sibling loader pagination. listEvalRuns defaulted to 50 results. A suite with more than 50 completed runs would silently omit older baselines from the dropdown, even when a deep-linked ?baseline_run_id pointed to one of them. The sibling loader now passes limit=200.

Sibling loader freshness. The sibling loader effect previously depended only on run?.suite_id and runId, both of which are stable after initial load. Adding run?.status to the dependency array causes the loader to re-fetch when the head run transitions to completed, so a run that finishes in another tab appears in the selector without a page reload.

Sort stability. sort.Slice is not guaranteed stable. The comparator was upgraded to sort.SliceStable with TestCaseID added as a tertiary key, ensuring identical requests always return the same ordering. The changeOrder lookup map was also hoisted to package scope to avoid a heap allocation on every request.

---

Files changed

Backend

• backend/internal/eval/handler.go — CompareRuns handler, types, and constants
• backend/internal/api/router.go — route registration
• backend/internal/eval/compare_test.go (new) — 13 table-driven tests covering all change classifications and all validation error paths
• backend/internal/store/mysql/migrations/0015_unique_eval_test_case_result.up.sql (new)
• backend/internal/store/mysql/migrations/0015_unique_eval_test_case_result.down.sql (new)

Frontend

• frontend/src/api/types.ts — CompareChangeType, TestCaseComparison, EvalRunCompare
• frontend/src/hooks/useApi.ts — compareEvalRuns, updated listEvalRuns with optional limit
• frontend/src/pages/EvalRunDetailPage.tsx — baseline selector, compare effect, delta banner
• frontend/src/components/eval/EvalRunResultsTable.tsx — optional compareMap prop and change badges
• frontend/src/pages/WorkflowListPage.tsx — Evals button on each workflow row

Docs

• DEMO_EF04.md (new) — end-to-end walkthrough of all comparison scenarios using curl

v1.7.0

What's Changed

• F05 presistent per node execution data by @g8rswimmer in #30

Full Changelog: v1.6.0...v1.7.0

v1.6.0 - workflow settings, eval triggers

What's Changed

• Automated eval runs by @g8rswimmer in #28
• Workflow config settings by @g8rswimmer in #29

Full Changelog: v1.5.0...v1.6.0

v1.5.0 - Evaluatuions

Overview

The Workflow Evaluation feature adds automated quality testing to cogniflow. Authors define eval suites containing test cases with fixed inputs, optional node mocks to stub out side effects, and graders that assert correctness of workflow or node output. Running a suite executes each test case as a real workflow run and scores every grader. Results are persisted, browsable in the UI, and linkable back to the underlying workflow run for node-level debugging.

---

What Was Built

ME1 — Data Foundation & CRUD API

Introduced the full eval data model and REST API for authoring suites and test cases.

• Four new database tables: eval_suites, eval_test_cases, eval_runs, eval_test_case_results. No foreign key constraints; referential integrity enforced at the application layer in the correct cascade order.
• Full CRUD for eval suites (GET/POST/PUT/DELETE /v1/workflows/{workflow_id}/eval-suites, GET/PUT/DELETE /v1/eval-suites/{suite_id}).
• Full CRUD + reorder for test cases (/v1/eval-suites/{suite_id}/test-cases).
• Each test case carries: initial_data (the fixed inputs fed to the workflow), mocks (per-node output overrides), and graders (assertions to evaluate after the run).
• Save-time validation rejects invalid mock node IDs (checked against the live workflow), malformed regex patterns, and invalid JSON Schema objects before anything is stored.
• LLM grader api_key values are encrypted at rest using the same AES-256-GCM vault already used for node sensitive config. The plain-text key is never returned via the API — responses always show "***".
• Suite-level pass_threshold (0.0–1.0, default 1.0) and max_concurrency fields control pass rate aggregation and parallel test case execution.

ME2 — Eval Execution & Deterministic Graders

Made suites runnable. Added three deterministic grader types.

• POST /v1/eval-suites/{suite_id}/runs — triggers an async eval run and returns the run ID immediately (same non-blocking pattern as workflow runs).
• Each test case triggers one real workflow execution. The triggered_by field on the resulting workflow run is set to "eval" for traceability.
• Node mock interception — when a test case defines a mock for a node, the eval engine bypasses Execute() and returns the mock output directly. The node still emits node.succeeded on the event bus (with "mocked": true in its output) so the event stream stays consistent and downstream nodes receive their expected input.
• Node outputs are captured from node.succeeded events and stored in eval_test_case_results.node_outputs for use by node-scoped graders and the debug UI.
• String Match grader — exact, contains, or regex match against a gjson field path on the target output. Invalid regex patterns are rejected at save time.
• Numeric Threshold grader — compares a numeric field against a threshold using ==, !=, >, >=, <, or <=.
• JSON Schema grader — validates the target output (or a field within it) against a JSON Schema draft-07 document using github.com/santhosh-tekuri/jsonschema/v5.
• All graders support scope: workflow (the run's final merged output) or scope: node (a specific node's captured output).
• Pass rate is computed per test case as passed_graders / total_graders. A test case with zero graders that completes without error has a pass rate of 1.0.
• GET /v1/eval-runs/{run_id} returns the full run with all TestCaseResult and GraderResult entries.

ME3 — LLM Graders

Added AI-evaluated grader types for subjective or open-ended assertions.

• LLM Judge grader — sends a rubric and the target value (a specific field or the full output as JSON) to an LLM and expects a structured {"verdict": "pass"|"fail", "explanation": "..."} response. Supports OpenAI and Anthropic providers. Provider and model are configurable per grader instance. If the LLM call fails or the response cannot be parsed, the verdict is "error" with the error detail in explanation — the eval run is not aborted.
• Checklist grader — evaluates N independent criteria in a single LLM call. Returns a score (fraction of criteria met), criteria_results (per-criterion met boolean and judge explanation), and an overall pass/fail verdict based on a per-grader pass_threshold. Partial scores are stored and displayed.
• Both LLM grader types use the existing aiprovider.LLMClient interface; no new provider abstraction was required.
• Mixed providers within the same suite are supported (e.g., one grader uses OpenAI, another uses Anthropic).

ME4 — Frontend: Suite & Test Case Authoring

Full browser UI for creating and configuring eval suites and test cases.

• A new Eval Suites tab on the workflow editor page lists all suites for the workflow with name, pass threshold, and test case count.
• The suite detail page shows ordered test cases and a collapsible run history panel.
• The TestCaseEditor slide-over handles all grader types through type-switched form fields — no hand-written JSON required:
  • String Match — field path, match type (exact / contains / regex), expected value.
  • Numeric Threshold — field path, operator, threshold.
  • LLM Judge — provider, model, api_key (password input), rubric, optional field path.
  • JSON Schema — optional field path, schema textarea.
  • Checklist — dynamic criterion list, per-grader pass threshold, provider/model/api_key/field_path.
• The MockEditor provides a node selector (workflow nodes by label) and a JSON textarea for the mock output.
• Initial data is rendered as an RJSF form when the workflow declares an initial_data_schema, and falls back to a plain JSON textarea otherwise.
• Test cases are reorderable within a suite.
• Save-time validation errors from the backend (invalid regex, unknown node ID in mock) surface inline per field.
• LLM grader api_key fields are masked (***) when an existing grader is loaded for editing.

ME5 — Frontend: Run & Observe

Browser UI for triggering runs and inspecting results at every level of detail.

• Run Suite button on the suite detail page triggers a new eval run and navigates to the run detail page.
• The run detail page polls GET /v1/eval-runs/{id} every 2 seconds until the run reaches a terminal status (completed or failed). Polling survives transient API errors — a network hiccup reschedules the next poll rather than halting the chain.
• A summary tile row shows total / passed / failed / error counts and run duration.
• EvalRunResultsTable — expandable accordion rows, one per test case:
  • Collapsed: test case name, workflow run status chip, "View Run →" link, overall pass/fail verdict.
  • Expanded: per-grader result rows. If the underlying workflow run failed before graders could evaluate, a red banner explains this.
• GraderResultRow — verdict icon (green ✓ / red ✗ / amber ! for error), grader name, type chip, explanation text, and a collapsible "show value / hide value" toggle exposing the raw value the grader inspected.
• ChecklistResultDetail — rendered inside the grader row without an additional click; shows a per-criterion table with met/unmet badges and the judge's explanation for each, plus a score summary ("4 of 5 criteria met — 80%").
• Run History accordion on the suite detail page lists past runs with short run ID, timestamp, duration, pass summary, and status badge. Fetches lazily when opened.
• "View Run →" link navigates to the existing workflow run detail page, showing node-level status and output for the exact execution the eval captured.

---

API Reference (summary)

Method	Path	Description
GET	/v1/workflows/{workflow_id}/eval-suites	List suites for a workflow
POST	/v1/workflows/{workflow_id}/eval-suites	Create suite
GET	/v1/eval-suites/{suite_id}	Get suite
PUT	/v1/eval-suites/{suite_id}	Update suite
DELETE	/v1/eval-suites/{suite_id}	Delete suite and all data
GET	/v1/eval-suites/{suite_id}/test-cases	List test cases
POST	/v1/eval-suites/{suite_id}/test-cases	Create test case
GET	/v1/eval-suites/{suite_id}/test-cases/{case_id}	Get test case
PUT	/v1/eval-suites/{suite_id}/test-cases/{case_id}	Replace test case
DELETE	/v1/eval-suites/{suite_id}/test-cases/{case_id}	Delete test case
PUT	/v1/eval-suites/{suite_id}/test-cases/order	Reorder test cases
POST	/v1/eval-suites/{suite_id}/runs	Trigger a new eval run
GET	/v1/eval-suites/{suite_id}/runs	List eval runs (?status=&limit=&offset=)
GET	/v1/eval-runs/{eval_run_id}	Get full run with all results
GET	/v1/eval-runs/{eval_run_id}/test-case-results/{result_id}	Get single result with node_outputs

---

Grader Types

Type	Config fields	Verdict logic
string_match	field_path, match_type (exact / contains / regex), expected_value	Coerces resolved value to string; compares
numeric_threshold	field_path, operator (==, !=, >, >=, <, <=), threshold	Requires numeric field; evaluates comparison
json_schema	field_path (optional), schema	Validates against JSON Schema draft-07
llm_judge	provider, model, api_key, rubric, field_path (optional)	LLM returns {"verdict":"pass"|"fail","explanation":"..."}
checklist	provider, model, api_key, criteria (array), pass_threshold, field_path (optional)	LLM evaluates each criterion; score = met/total; pass if score ≥ threshold

All graders support scope: workflow or scope: node. Node-scoped graders require a node_id; if that node did not execute, the verdict is error.

---

Database Migrations

| Migration | ...