0.3.0

Features

• Introduce multi-turn conversation test cases — a multi-turn case is a single test case carrying an ordered multiTurnData array (mutually exclusive with data), executed as a sequential fail-fast turn loop that writes one result row and one eval summary per turn; try-it-out previews the whole conversation and a suite testCaseFilter matches only when every turn satisfies it #17 (#100, #132, #135, #140, #147, #133)
• Introduce multi-request test suites — the suite's own request becomes request #0 of an ordered chain and the new additionalRequests array (max 10, each with its own name, endpointRef, requestTemplate, responseColumns and inputBindings) supplies requests 1..N, executed sequentially against the same deployment so a setup call's output feeds the call under test; response columns form one flat namespace capped at 50 suite-wide, the JSONata frame accumulates monotonically across requests, any subset of the chain may be multi-turn, and a row's identity becomes the pair (request_index, turn_index) #98 (#114, #164)
• Try-out is chain-aware — both try-it-out endpoints now run the whole additionalRequests chain, threading one accumulated frame across requests and turns exactly as the run engine does, each history entry carrying its own requestIndex / turnIndex stamps plus its extractedColumns and extractionWarnings; GET …/resolved-request?requestIndex=N previews any position in the chain #162 (#163, #164)
• Request-template JSON bodies accept a JSONata expression via the new jsonataContent field — conversation history becomes a user-authored expression over the $_request / $_response frame bindings instead of hardcoded messages reassembly, so multi-turn works against non-chat-completions APIs and streaming works on every turn #17 (#113, #121)
• Add conditional metric execution — an optional per-metric-definition condition (JSONata, max 2000 chars) decides per result row whether that metric runs, evaluated over {data, response, turn:{index,total,last}, request:{index,total,last,name}} #17 (#100, #114, #164)
• Import externally evaluated test-case results — POST /api/v1/test-suites/{id}/runs/import accepts a results CSV and runs metric evaluation and score computation over it, so a deployment evaluated elsewhere can still be scored by the framework #93 (#99, #122)
• Add eval-cli, a standalone DB-free CLI — clone, fetch, run, import and the chaining evaluate command reuse existing suites and the shared Phase 1 engine to evaluate a deployment with no Evaluation Framework instance alongside it; fetch also pulls the bound dataset's test-case schema and the results CSV carries requestIndex / totalRequests / turnIndex / totalTurns, so multi-request and multi-turn suites import with correct row identity; ships with a Dockerfile recipe for CI runners #92 (#112, #117, #128, #154, #164)
• Add Api-Key header authentication as an alternative to OIDC/JWT — keys are introspected against DIAL Core's GET /v1/user/info, cached, and mapped to application authorities; off by default and active only in oidc security mode (#134)
• Add a run comparison endpoint — GET /api/v1/analytics/metric-scores/comparison?runIds=<a>,<b> recomputes per-metric statistics, overall and average execution duration over only the eval-summary rows the two runs have in common, and returns the ids of the rows that did not match #103 (#106)
• Measure metric evaluation time — each eval-summary row gains metricEvalDurationMs, the summed latency of that row's provider /evaluate calls, filterable and aggregatable through the Query DSL #123 (#150)
• Expose deployment-reference fields on the test_suites Query DSL entity — deployment_ref::id|name|version and mcp_deployment_ref::id|name|type as flat STRING fields for filter, select, sort and group_by #97 (#101)
• Metric provider entries gain a per-entry enabled flag and a stock second slot — METRIC_PROVIDERS_EXTRA_* configures a ready-to-use extra provider (disabled by default), and disabling an entry skips it in the sync job while its already-synced declarations stay in the catalog and remain evaluable during runs #167 (#166, #164)
• Make test-suite-run request timeouts configurable — TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT / TEST_SUITE_RUN_EXEC_MAX_TIMEOUT replace the hardcoded values, with generous defaults so slow streaming responses no longer trip a request timeout (#153)

Fixes

• Write eval summaries for runs with no metrics configured — listing, count, aggregate and CSV export of such a run returned an empty page or HTTP 404; "latest computation" now resolves from test_case_eval_summaries instead of the legitimately empty metric snapshots (#108)
• Carry the full DIAL Core deployment contract through listing — version, owner, timestamps, keywords, attachment types, model capabilities/limits/pricing, application properties and toolset transport were silently dropped, and an unrecognized object discriminator now degrades to a skippable entry instead of failing the whole fetch #130 (#131)
• Restore application and model listing after DIAL Core switched deployment names and descriptions to multilingual objects (#145)
• Allow running a suite whose testCaseFilter uses an array-field Contains condition — lower/upper over an ARRAY-bound field rendered lower(jsonb) and failed run creation with an unhandled HTTP 500; it now translates to case-insensitive whole-element JSONB containment #142 (#148)
• Stop nc / ne / not filters from silently dropping rows whose field is null or absent — negated comparisons are now total, so an absent value correctly satisfies "does not contain" #141 (#143)
• Ignore the legacy disabled_test_case_ids exclusion during run selection — suites created before run conditions existed reported "39 of 39 selected" while executing 4 cases, and adding a run condition failed POST /runs with 409 #151 (#152)
• Upgrade Jackson to 2.21.5 / 3.1.5 to address CVE-2026-59889, CVE-2026-54515 and GHSA-mhm7-754m-9p8w, and Netty to 4.2.17.Final; build-time constraints added for CVE-2026-54399, CVE-2026-54428, CVE-2026-64607, CVE-2025-48924 and CVE-2025-67030 (#155)
• Patch OS packages inherited from the container base image at build time — the runtime stage now runs apk --no-cache upgrade, picking up libssl3 / libcrypto3 advisories fixed after the base image was cut (#166, #164)
• Return a short projection from GET /api/v1/deployments — the full-contract listing added in 0.3.0-rc.0 inflated the payload; entries now carry only deploymentId, displayName and description (plus transport for toolsets), and the full representation is fetched per deployment (regression fix, affects pre-release users of the expanded deployment contract only) (#157)

Deployment Changes

New environment variables

Required: ✅ = must be supplied, the service will not start without it. ❌ = ships with a working default. Nothing in this release is required in a default deployment; ⚠️ in a description marks a variable that becomes mandatory once you switch on the feature it belongs to.

Variable	Required	Default	Description
API_KEY_ENABLED	❌	false	Enables DIAL API-Key authentication via the Api-Key request header. Applies only when config.rest.security.mode=oidc.
API_KEY_CORE_URL	❌	-	⚠️ Base URL of the DIAL Core instance used to introspect API keys via GET /v1/user/info. Empty by default; startup fails with …requires …core-url to be set when API_KEY_ENABLED=true.
API_KEY_CACHE_TTL_SECONDS	❌	60	Time-to-live, in seconds, for cached successful introspection results.
API_KEY_CACHE_MAX_SIZE	❌	10000	Maximum number of cached introspection results.
API_KEY_REQUEST_TIMEOUT_MS	❌	3000	Connect/read timeout, in milliseconds, for the introspection call to DIAL Core.
API_KEY_ROLES_MAPPING	❌	- (empty)	⚠️ JSON object mapping DIAL Core project-key role names to lists of this service's authority strings. When API_KEY_ENABLED=true, at least one of the two role mappings must be non-empty or startup fails.
API_KEY_DEFAULT_ROLES_MAPPING	❌	- (empty)	⚠️ JSON object mapping DIAL Core roles from the JWT-rooted per-request-key response shape (userClaims) to lists of this service's authority strings. When API_KEY_ENABLED=true, at least one of the two role mappings must be non-empty or startup fails.
API_KEY_USER_CLAIMS_ROLE_CLAIM	❌	roles	Claim name read out of the introspection response's userClaims object to obtain the caller's raw roles.
API_KEY_STARTUP_PROBE	❌	true	When true, the service calls DIAL Core's /v1/user/info at startup and fails to start if Core is unreachable or misconfigured.
ANALYTICS_COMPARISON_MAX_UNMATCHED_ROWS	❌	5000	Maximum number of non-matching eval-summary rows a single run comparison may report per run; exceeding it fails the request with HTTP 409. Minimum 1.
JSONATA_EVALUATION_TIMEOUT_MS	❌	10000	Maximum wall-clock time in milliseconds a single JSONata expression evaluation may run before it is aborted. Minimum 1.
JSONATA_MAX_RECURSION_DEPTH	❌	1000	Maximum call-stack recursion depth a single JSONata expression evaluation may reach before it is aborted. Minimum 1.
ANALYTICS_RESULTS_CSV_IMPORT_MAX_FILE_SIZE	❌	10MB	Maximum CSV file size for the eval-results import endpoint. Larger requests are rejected with HTTP 400 before parsing begins.
TEST_CASE_MULTI_TURN_MAX_TURNS	❌	10	Maximum number of turns a multi-turn test case may carry. An over-cap case is persisted but marked invalid with a warning, so it is excluded from runnable selection.
TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT	❌	600000	Default per-request timeout, in milliseconds, for deployment requests issued during a test suite run.
TEST_SUITE_RUN_EXEC_MAX_TIMEOUT	❌	1200000	Upper bound, in milliseconds, a run may request for its per-request timeout.
VIRTUAL_THREADS_ENABLED	❌	true	Toggles Spring's virtual-thread executor; set to false to aid troubleshooting.
METRIC_PROVIDERS_DIAL_ENABLED	❌	true	Whether the sync job processes the stock dial provider entry. @NotNull — every configured entry must define it.
METRIC_PROVIDERS_EXTRA_ENABLED	❌	false	Activates the stock second provider entry extra. Point METRIC_PROVIDERS_EXTRA_BASE_URL at a real service before flipping it to true.
METRIC_PROVIDERS_EXTRA_BASE_URL	❌	http://localhost:8087	⚠️ Base URL of the extra provider; GET /metrics is called against it. The default points at localhost, so it must be overridden whenever METRIC_PROVIDERS_EXTRA_ENABLED=true.
METRIC_PROVIDERS_EXTRA_CONNECT_TIMEOUT_MS	❌	5000	HTTP connection timeout, in milliseconds, for the extra provider.
METRIC_PROVIDERS_EXTRA_READ_TIMEOUT_MS	❌	150000	HTTP read timeout, in milliseconds, for the extra provider.

Behavioral changes

Note

These take effect automatically on upgrade — no operator action required, but existing suites may select or time out differently than they did on 0.2.0.

• Test suite run request timeouts — the default per-request timeout rises from 30s to 600s and the maximum from 600s to 1200s, so a slow streaming deployment no longer trips a request timeout (#153)
• Query DSL null polarity — nc, ne and not are now total over nulls: a row whose field is null or absent satisfies a negated predicate instead of being dropped. Suite testCaseFilters using these operators will select more cases than before; the previous behavior is expressible as and(ne(field, null), nc(field, "x")) #141 (#143)
• Run selection — the only rule is now is_valid = TRUE AND testCaseFilter. Suites carrying stale test_suites.disabled_test_case_ids values run every valid, filter-matching case; the column is retained but no longer read or written, with the drop deferred to a follow-up #151 (#152)
• Suite snapshots on imported runs — the snapshot phase now runs for every run, so an imported run scores against frozen suite config instead of resynthesizing from the live suite (#122)
• Result row volume on chained suites — a chain of R requests over T turns writes up to R × T result and eval-summary rows per test-case repetition where a single-request, single-turn suite wrote one. A large suite can now exceed ANALYTICS_RESULTS_BATCH_MAX_ITEMS (default 10000) on import and fail with HTTP 400; raise it deliberately before running such a suite. Single-request suites are byte-identical to 0.2.0 #98 (#114, #164)
• Custom metric provider entries must now declare enabled — ProviderEntry.enabled is @NotNull with no Java-side default. The stock dial (true) and extra (false) entries carry defaults in application.yml, so standard deployments are unaffected. A custom provider entry supplied through external configuration (mounted application.yml, SPRING_APPLICATION_JSON, Helm values, or the METRICPROVIDERS_PROVIDERS_<ID>_* environment form) had no enabled field on 0.2.0 and now fails startup with providers[<id>].enabled must not be null until the flag is added #167 (#166, #164)

Caution

Upgrade the backend before eval-cli. This CLI version emits the 19-column import contract (requestIndex, totalRequests, turnIndex, totalTurns among them). An older backend does not reject such a CSV — it silently ignores the four unknown headers and persists every row as a distinct single-request, single-turn row under a random identity. The import reports success and there is no error to alert on. A fetch bundle written by an older CLI still loads, but run refuses a multi-turn test case against it; re-run fetch first. (#164)

Database migrations

Note

Flyway applies these automatically on startup; the service account needs DDL privileges on both schemas. All column adds are backward compatible — nullable JSONB, or NOT NULL DEFAULT backfilled in a metadata-only statement with no table rewrite.

• meta — V1.26__AddConditionToTestSuiteMetricDefinitions.sql — adds the nullable condition VARCHAR(2000) column to test_suite_metric_definitions (#100)
• meta — V1.27__AddMultiTurnDataToTestCases.sql — adds the nullable multi_turn_data JSONB column plus a mutual-exclusivity CHECK to test_cases (#100)
• meta — V1.28__AddMultiTurnDataToTestCaseRunInputs.sql — adds the nullable multi_turn_data JSONB column to test_case_run_inputs for the suite-run snapshot (#100)
• meta — V1.29__AddAdditionalRequestsToTestSuites.sql — adds additional_requests JSONB NOT NULL DEFAULT '[]' and the nullable request_name VARCHAR(255) to test_suites, so every existing suite behaves as a one-element chain #98 (#114, #164)
• analytics — V1.13__AddTurnColumnsToTestCaseRunResults.sql — adds turn_index / total_turns to test_case_run_results and extends uq_results_run_case_index with turn_index (#100)
• analytics — V1.14__AddTurnColumnsToEvalSummaries.sql — adds turn_index / total_turns to test_case_eval_summaries and extends uq_eval_summaries_natural_key with turn_index (#100)
• analytics — V1.15__AddEvalSummariesRunComputedAtIndex.sql — adds a covering (test_suite_run_id, computed_at_ms DESC, computation_id) index so latest-computation resolution stays a top-1 index descent (#108)
• analytics — V1.16__AddMetricEvalDurationToEvalSummaries.sql — adds metric_eval_duration_ms BIGINT NOT NULL DEFAULT 0 to test_case_eval_summaries (#150)
• analytics — V1.17__AddRequestColumnsToTestCaseRunResults.sql — adds request_index / total_requests to test_case_run_results (backfilled 0/1) and extends uq_results_run_case_index with request_index #98 (#114, #164)
• analytics — V1.18__AddRequestColumnsToEvalSummaries.sql — adds request_index / total_requests to test_case_eval_summaries (backfilled 0/1) and extends uq_eval_summaries_natural_key with request_index #98 (#114, #164)

Caution

V1.13, V1.14, V1.17 and V1.18 drop and re-create the unique constraint / index on test_case_run_results and test_case_eval_summaries. On a large analytics deployment this takes a long exclusive lock — plan a maintenance window, or replace them with a non-transactional CREATE UNIQUE INDEX CONCURRENTLY + swap.

API / contract changes

Note

Full reference in Swagger UI (/swagger-ui.html).

Note

GET /api/v1/deployments no longer returns version, owner, createdAt/updatedAt, descriptionKeywords, inputAttachmentTypes, model capabilities/limits/pricing, or application applicationProperties/applicationTypeSchemaId/routes. Consumers needing those fields fetch the deployment individually by type and ID. Full reference in Swagger UI (/swagger-ui.html). (#157)

• New endpoints — POST /api/v1/test-suites/{id}/runs/import (#99) and GET /api/v1/analytics/metric-scores/comparison (#106)
• New query parameter — GET …/resolved-request accepts requestIndex (default 0) to preview any position in a suite's request chain; out of range returns HTTP 400 #98 #162 (#114, #163, #164)
• Breaking — metricEvalDurationMs is now @NotNull on EvalSummaryBatchWriteItemDto; external producers calling POST /api/v1/analytics/eval-summaries must send it or receive HTTP 400 (#150)
• Breaking — disabledTestCaseIds is removed from the test-suite request and response DTOs. Unknown request fields are ignored, so no coordinated client release is required #151 (#152)
• Breaking — the legacy hardcoded multi-turn executor is removed; multi-turn request templates must now be authored as JSONata via jsonataContent #17 (#113)
• Changed (affects pre-release users of the results-import endpoint only) — the import CSV contract is now 19 reserved columns; testCaseData becomes a required reserved column carrying the row's data as a JSON object, arbitrary headers no longer map to testCaseData fields (unmatched headers are ignored), and extractedColumns, extractionWarnings, requestIndex, totalRequests, turnIndex, totalTurns join the reserved set. Rows with no testCaseId derive one from testCaseName, so a repetition's rows group together #98 (#114, #164)
• Additive — multiTurnData on test-case DTOs, condition on test-suite metric-definition DTOs, turnIndex / totalTurns on analytics result and eval-summary DTOs, a nullable turnIndex on ValidationWarningDto, a history list on the try-it-out response, and a reserved turnIndex column in test-case CSV export/import (#100, #147)
• Additive — additionalRequests and requestName on test-suite DTOs; requestIndex / totalRequests on analytics result and eval-summary DTOs; requestIndex / totalRequests / requestName / turnIndex / totalTurns, extractedColumns, extractionWarnings, streamingStatus and truncationWarning on try-it-out responses; a request: {index, total, last, name} namespace for metric condition expressions #98 #162 (#114, #163, #164)
• New 409 INVALID_OPERATION — creating a run for, or trying out, an MCP suite bound to a dataset containing multi-turn test cases (#100, #147)
• New 400 — a non-empty additionalRequests on an MCP_TOOL suite; MCP chaining is model-ready but deferred #98 (#114, #164)