Releases · epam/ai-dial-admin-evaluation-framework-backend

Repository navigation

Release list

0.5.0 Latest

Latest

github-actions released this 14 Oct 11:50

0.5.0

f040bed

Features

Add an authenticated inbound MCP endpoint — stateless HTTP exposes list_deployments and get_deployment tools #209 (#208)
Add optional DIAL Application Route execution — per-request keys keep long evaluations authenticated and auto-share EF files (#236)
Query test-suite runs with deployment context, run counts, metric names, overall scores, and optional total costs through the structured Query DSL #197 (#201, #206, #221, #222)
Fetch total costs for up to 1,000 test-suite runs in one request — preserves request order and returns null when usage is absent (#205)
List suite metric definitions with declaration and schema-version details — clients no longer join them locally #224 (#226)

Fixes

Calculate average run costs per test case instead of per usage request — corrects multi-request, multi-turn, and multi-metric suites (#207)
Calculate test-case scores across all turns, requests, and reruns — skipped metrics are excluded and test cases remain equally weighted (#231)
Restore application and model target listings after DIAL Core changed model pricing — accepts both scalar and object prompt / completion values #243 (#244, #247)
Resolve API-key roles against issuer-specific claim paths, including array-wrapped issuer claims — multi-provider authentication now maps authorities correctly (#241, #248)
Preserve multi-turn data, schemas, file refs, and content types in ZIP round trips — imports are transactional and archive parsing is bounded #225 (#227, #228)
Identify the failed test case and broken file reference when ZIP export cannot download an attachment #225 (#232)
Return template variables per request for multi-request suites — extraction and try-out now use request-indexed maps #217 (#235)
Upgrade Jackson to 2.22.3 / 3.2.3 across the service, CLI, runner, and build tooling — addresses CVE-2026-19032, CVE-2026-68497, CVE-2026-83557, CVE-2026-89407, CVE-2026-89425, CVE-2026-91776, and CVE-2026-91777 #242 (#233, #234, #245)
Preserve test-case values when renaming dataset schema fields — stable IDs move stored data, while bound public datasets reject destructive updates and imports #246 (#249, #254)

Other

Move run metric snapshots to metadata storage — run deletion now cascades, existing rows are copied, and the old analytics route remains a deprecated alias #197 (#199)
Rename the API-key role-mapping variables to API_KEY_KEY_ROLES_MAPPING / API_KEY_JWT_ROLES_MAPPING — names now distinguish project-key from JWT-rooted per-request-key mappings; old names are no longer read (#256)

Deployment Changes

Removed environment variables

Caution

Renamed without fallback; affects every deployment with API_KEY_ENABLED=true since 0.3.0. Keeping both old names fails startup; renaming only one leaves the other mapping empty, so callers on that path get no authorities.

Variable	Reason
`API_KEY_ROLES_MAPPING`	Renamed to `API_KEY_KEY_ROLES_MAPPING` — project-key role mapping (#256)
`API_KEY_DEFAULT_ROLES_MAPPING`	Renamed to `API_KEY_JWT_ROLES_MAPPING` — JWT-rooted per-request-key (`userClaims`) role mapping (#256)

New environment variables

Variable	Required	Default	Description
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_ENABLED`	No	`false`	Adds `total_cost` to row-mode `test_suite_runs` query results.
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_TIMEOUT_SEC`	Conditional	`2`	End-to-end cost lookup deadline; minimum `1`.
`CSV_IMPORT_ZIP_MAX_ENTRIES`	No	`1000`	Maximum ZIP entries accepted during test-case import.
`CSV_IMPORT_ZIP_MAX_TOTAL_UNCOMPRESSED_SIZE`	No	`1GB`	Maximum total uncompressed ZIP size.
`CSV_IMPORT_ZIP_MAX_MANIFEST_SIZE`	No	`1MB`	Maximum `manifest.json` size.
`SPRING_AI_MCP_SERVER_ENABLED`	No	`false`	Enables the inbound MCP server; `/mcp` returns 404 when disabled.
`SPRING_AI_MCP_SERVER_STDIO`	No	`false`	Enables local stdio transport; HTTP is the deployment default.
`SPRING_AI_MCP_SERVER_NAME`	No	`ai-dial-admin-evaluation-framework-backend`	Server name returned by MCP initialization.
`SPRING_AI_MCP_SERVER_VERSION`	No	Project version	Server version returned by MCP initialization.
`SPRING_AI_MCP_SERVER_TYPE`	No	`SYNC`	MCP implementation type; `SYNC` preserves request-thread caller identity.
`SPRING_AI_MCP_SERVER_PROTOCOL`	No	`STATELESS`	HTTP protocol; `STREAMABLE` requires one replica or sticky sessions.
`SPRING_AI_MCP_SERVER_CAPABILITIES_TOOL`	No	`true`	Advertises MCP tools.
`SPRING_AI_MCP_SERVER_CAPABILITIES_RESOURCE`	No	`false`	Advertises MCP resources; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_PROMPT`	No	`false`	Advertises MCP prompts; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_COMPLETION`	No	`false`	Advertises MCP completions; none ship in this release.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_MCP_ENDPOINT`	No	`/mcp`	Sets the authenticated MCP HTTP path.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_DISALLOW_DELETE`	No	`false`	Disables session termination by `DELETE` in `STREAMABLE` mode.
`SPRING_AI_MCP_SERVER_ANNOTATION_SCANNER_ENABLED`	No	`true`	Discovers `@McpTool` methods.
`SPRING_AI_MCP_SERVER_INSTRUCTIONS`	No	Built-in workflow guidance	Instructions returned by MCP initialization.

Enabling Application mode for test suite runs

Application mode (see DIAL Configuration changes below) routes long-running evals through a DIAL
Application Route instead of the user's JWT, so a run no longer fails when the JWT expires mid-run.
It is off by default. To turn it on:

1. Enable DIAL API-Key authentication first — the internal run-execution endpoint is authenticated
via the Api-Key header, not the user's JWT, so Application mode cannot start without it:

Environment Variable	Default	Required	Description
`API_KEY_ENABLED`	`false`	Yes	Enables DIAL API-Key authentication via the `Api-Key` request header. Must be `true` before `DIAL_APP_PROXY_ENABLED` can be set — EF fails fast at startup otherwise.
`API_KEY_CORE_URL`	-	Yes	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`.
`API_KEY_KEY_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core project-key role names to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`. At least one of this or `API_KEY_JWT_ROLES_MAPPING` must be non-empty.
`API_KEY_JWT_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core roles from the JWT-rooted `userClaims` response shape to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	No	Claim name read from the introspection response's `userClaims` object to obtain the caller's raw roles. Set to `dial_roles` to match this claim naming in Core.

2. Turn on the Application Route flag:

Environment Variable	Default	Required	Description
`DIAL_APP_PROXY_ENABLED`	`false`	No	Enables DIAL Application Routes mode for run execution — eval runs are triggered via DIAL Core's Application Route using a per-request key instead of the user's JWT. Requires `API_KEY_ENABLED=true`.
`DIAL_APP_PROXY_DEPLOYMENT_NAME`	`EF`	No	DIAL Core application deployment name. Must match the `applications.<name>` key used in the DIAL Core config snippet in DIAL Configuration changes below.
`DIAL_APP_PROXY_HEARTBEAT_INTERVAL_MS`	`30000`	No	Heartbeat interval for the internal run SSE connection.
`DIAL_APP_PROXY_TRIGGER_READ_TIMEOUT_MS`	`43200000`	No	DIAL Application Route trigger timeout in milliseconds.

3. Recommended tuning (not required to enable the mode, but needed for long evals):

Environment Variable	Default	Required	Description
`TEST_SUITE_RUN_SSE_TIMEOUT_MINUTES`	`30`	No	SSE client connection timeout, in minutes. Raise this under Application mode (e.g. `720` for up to 12 hours) so the status stream doesn't time out before a long-running eval completes.

Behavioral changes

Note

These take effect automatically on upgrade unless noted otherwise.

API-key role mapping — with OIDC providers configured, introspection matches userClaims.iss and unions the provider's roleClaims; API_KEY_USER_CLAIMS_ROLE_CLAIM is now the fallback (#241, #248)
dial-adas compatibility — run-cost averages now call POST /v1/queries/execute-sql; verify the deployed dial-adas version exposes this endpoint (#207)

DIAL Configuration changes

Important

DIAL Core registration is required only when DIAL_APP_PROXY_ENABLED=true; direct JWT execution remains the default.

Required setup (#236):

Add the applications.EF block to DIAL Core configuration, or use the application name set by DIAL_APP_PROXY_DEPLOYMENT_NAME, and grant it access to the EF file bucket. Apply the snippet with http://eval-framework-url replacement:

 {                                                                                                                                                                                                                                 
   "applications": {                                                                                                                                                                                                               
     "EF...

Assets 2

0.5.0-rc.2 Pre-release

Pre-release

github-actions released this 14 Oct 08:43

0.5.0-rc.2

5e30933

Features

Add an authenticated inbound MCP endpoint — stateless HTTP exposes list_deployments and get_deployment tools #209 (#208)
Add optional DIAL Application Route execution — per-request keys keep long evaluations authenticated and auto-share EF files (#236)
Query test-suite runs with deployment context, run counts, metric names, overall scores, and optional total costs through the structured Query DSL #197 (#201, #206, #221, #222)
Fetch total costs for up to 1,000 test-suite runs in one request — preserves request order and returns null when usage is absent (#205)
List suite metric definitions with declaration and schema-version details — clients no longer join them locally #224 (#226)

Fixes

Calculate average run costs per test case instead of per usage request — corrects multi-request, multi-turn, and multi-metric suites (#207)
Calculate test-case scores across all turns, requests, and reruns — skipped metrics are excluded and test cases remain equally weighted (#231)
Restore application and model target listings after DIAL Core changed model pricing — accepts both scalar and object prompt / completion values #243 (#244, #247)
Resolve API-key roles against issuer-specific claim paths, including array-wrapped issuer claims — multi-provider authentication now maps authorities correctly (#241, #248)
Preserve multi-turn data, schemas, file refs, and content types in ZIP round trips — imports are transactional and archive parsing is bounded #225 (#227, #228)
Identify the failed test case and broken file reference when ZIP export cannot download an attachment #225 (#232)
Return template variables per request for multi-request suites — extraction and try-out now use request-indexed maps #217 (#235)
Upgrade Jackson to 2.22.3 / 3.2.3 across the service, CLI, runner, and build tooling — addresses CVE-2026-19032, CVE-2026-68497, CVE-2026-83557, CVE-2026-89407, CVE-2026-89425, CVE-2026-91776, and CVE-2026-91777 #242 (#233, #234, #245)
Preserve test-case values when renaming dataset schema fields — stable IDs move stored data, while bound public datasets reject destructive updates and imports #246 (#249, #254)

Other

Move run metric snapshots to metadata storage — run deletion now cascades, existing rows are copied, and the old analytics route remains a deprecated alias #197 (#199)
Rename the API-key role-mapping variables to API_KEY_KEY_ROLES_MAPPING / API_KEY_JWT_ROLES_MAPPING — names now distinguish project-key from JWT-rooted per-request-key mappings; old names are no longer read (#256)

Deployment Changes

Removed environment variables

Caution

Variable	Reason
`API_KEY_ROLES_MAPPING`	Renamed to `API_KEY_KEY_ROLES_MAPPING` — project-key role mapping (#256)
`API_KEY_DEFAULT_ROLES_MAPPING`	Renamed to `API_KEY_JWT_ROLES_MAPPING` — JWT-rooted per-request-key (`userClaims`) role mapping (#256)

New environment variables

Variable	Required	Default	Description
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_ENABLED`	No	`false`	Adds `total_cost` to row-mode `test_suite_runs` query results.
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_TIMEOUT_SEC`	Conditional	`2`	End-to-end cost lookup deadline; minimum `1`.
`CSV_IMPORT_ZIP_MAX_ENTRIES`	No	`1000`	Maximum ZIP entries accepted during test-case import.
`CSV_IMPORT_ZIP_MAX_TOTAL_UNCOMPRESSED_SIZE`	No	`1GB`	Maximum total uncompressed ZIP size.
`CSV_IMPORT_ZIP_MAX_MANIFEST_SIZE`	No	`1MB`	Maximum `manifest.json` size.
`SPRING_AI_MCP_SERVER_ENABLED`	No	`false`	Enables the inbound MCP server; `/mcp` returns 404 when disabled.
`SPRING_AI_MCP_SERVER_STDIO`	No	`false`	Enables local stdio transport; HTTP is the deployment default.
`SPRING_AI_MCP_SERVER_NAME`	No	`ai-dial-admin-evaluation-framework-backend`	Server name returned by MCP initialization.
`SPRING_AI_MCP_SERVER_VERSION`	No	Project version	Server version returned by MCP initialization.
`SPRING_AI_MCP_SERVER_TYPE`	No	`SYNC`	MCP implementation type; `SYNC` preserves request-thread caller identity.
`SPRING_AI_MCP_SERVER_PROTOCOL`	No	`STATELESS`	HTTP protocol; `STREAMABLE` requires one replica or sticky sessions.
`SPRING_AI_MCP_SERVER_CAPABILITIES_TOOL`	No	`true`	Advertises MCP tools.
`SPRING_AI_MCP_SERVER_CAPABILITIES_RESOURCE`	No	`false`	Advertises MCP resources; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_PROMPT`	No	`false`	Advertises MCP prompts; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_COMPLETION`	No	`false`	Advertises MCP completions; none ship in this release.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_MCP_ENDPOINT`	No	`/mcp`	Sets the authenticated MCP HTTP path.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_DISALLOW_DELETE`	No	`false`	Disables session termination by `DELETE` in `STREAMABLE` mode.
`SPRING_AI_MCP_SERVER_ANNOTATION_SCANNER_ENABLED`	No	`true`	Discovers `@McpTool` methods.
`SPRING_AI_MCP_SERVER_INSTRUCTIONS`	No	Built-in workflow guidance	Instructions returned by MCP initialization.

Enabling Application mode for test suite runs

1. Enable DIAL API-Key authentication first — the internal run-execution endpoint is authenticated
via the Api-Key header, not the user's JWT, so Application mode cannot start without it:

Environment Variable	Default	Required	Description
`API_KEY_ENABLED`	`false`	Yes	Enables DIAL API-Key authentication via the `Api-Key` request header. Must be `true` before `DIAL_APP_PROXY_ENABLED` can be set — EF fails fast at startup otherwise.
`API_KEY_CORE_URL`	-	Yes	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`.
`API_KEY_KEY_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core project-key role names to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`. At least one of this or `API_KEY_JWT_ROLES_MAPPING` must be non-empty.
`API_KEY_JWT_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core roles from the JWT-rooted `userClaims` response shape to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	No	Claim name read from the introspection response's `userClaims` object to obtain the caller's raw roles. Set to `dial_roles` to match this claim naming in Core.

2. Turn on the Application Route flag:

Environment Variable	Default	Required	Description
`DIAL_APP_PROXY_ENABLED`	`false`	No	Enables DIAL Application Routes mode for run execution — eval runs are triggered via DIAL Core's Application Route using a per-request key instead of the user's JWT. Requires `API_KEY_ENABLED=true`.
`DIAL_APP_PROXY_DEPLOYMENT_NAME`	`EF`	No	DIAL Core application deployment name. Must match the `applications.<name>` key used in the DIAL Core config snippet in DIAL Configuration changes below.
`DIAL_APP_PROXY_HEARTBEAT_INTERVAL_MS`	`30000`	No	Heartbeat interval for the internal run SSE connection.
`DIAL_APP_PROXY_TRIGGER_READ_TIMEOUT_MS`	`43200000`	No	DIAL Application Route trigger timeout in milliseconds.

3. Recommended tuning (not required to enable the mode, but needed for long evals):

Environment Variable	Default	Required	Description
`TEST_SUITE_RUN_SSE_TIMEOUT_MINUTES`	`30`	No	SSE client connection timeout, in minutes. Raise this under Application mode (e.g. `720` for up to 12 hours) so the status stream doesn't time out before a long-running eval completes.

Behavioral changes

Note

These take effect automatically on upgrade unless noted otherwise.

API-key role mapping — with OIDC providers configured, introspection matches userClaims.iss and unions the provider's roleClaims; API_KEY_USER_CLAIMS_ROLE_CLAIM is now the fallback (#241, #248)
dial-adas compatibility — run-cost averages now call POST /v1/queries/execute-sql; verify the deployed dial-adas version exposes this endpoint (#207)

DIAL Configuration changes

Important

DIAL Core registration is required only when DIAL_APP_PROXY_ENABLED=true; direct JWT execution remains the default.

Required setup (#236):

Add the applications.EF block to DIAL Core configuration, or use the application name set by DIAL_APP_PROXY_DEPLOYMENT_NAME, and grant it access to the EF file bucket. Apply the snippet with http://eval-framework-url replacement:

 {                                                                                                                                                                                                                                 
   "applications": {                                                                                                                                                                                                               
     "EF...

Assets 2

0.5.0-rc.1 Pre-release

Pre-release

github-actions released this 12 Oct 15:33

0.5.0-rc.1

59e9530

Features

Add an authenticated inbound MCP endpoint — stateless HTTP exposes list_deployments and get_deployment tools #209 (#208)
Add optional DIAL Application Route execution — per-request keys keep long evaluations authenticated and auto-share EF files (#236)
Query test-suite runs with deployment context, run counts, metric names, overall scores, and optional total costs through the structured Query DSL #197 (#201, #206, #221, #222)
Fetch total costs for up to 1,000 test-suite runs in one request — preserves request order and returns null when usage is absent (#205)
List suite metric definitions with declaration and schema-version details — clients no longer join them locally #224 (#226)

Fixes

Calculate average run costs per test case instead of per usage request — corrects multi-request, multi-turn, and multi-metric suites (#207)
Calculate test-case scores across all turns, requests, and reruns — skipped metrics are excluded and test cases remain equally weighted (#231)
Restore application and model target listings after DIAL Core changed model pricing — accepts both scalar and object prompt / completion values #243 (#244, #247)
Resolve API-key roles against issuer-specific claim paths, including array-wrapped issuer claims — multi-provider authentication now maps authorities correctly (#241, #248)
Preserve multi-turn data, schemas, file refs, and content types in ZIP round trips — imports are transactional and archive parsing is bounded #225 (#227, #228)
Identify the failed test case and broken file reference when ZIP export cannot download an attachment #225 (#232)
Return template variables per request for multi-request suites — extraction and try-out now use request-indexed maps #217 (#235)
Upgrade Jackson to 2.22.3 / 3.2.3 across the service, CLI, runner, and build tooling — addresses CVE-2026-19032, CVE-2026-68497, CVE-2026-83557, CVE-2026-89407, CVE-2026-89425, CVE-2026-91776, and CVE-2026-91777 #242 (#233, #234, #245)

Other

Move run metric snapshots to metadata storage — run deletion now cascades, existing rows are copied, and the old analytics route remains a deprecated alias #197 (#199)
Rename the API-key role-mapping variables to API_KEY_KEY_ROLES_MAPPING / API_KEY_JWT_ROLES_MAPPING — names now distinguish project-key from JWT-rooted per-request-key mappings; old names are no longer read (#256)

Deployment Changes

Removed environment variables

Caution

Variable	Reason
`API_KEY_ROLES_MAPPING`	Renamed to `API_KEY_KEY_ROLES_MAPPING` — project-key role mapping (#256)
`API_KEY_DEFAULT_ROLES_MAPPING`	Renamed to `API_KEY_JWT_ROLES_MAPPING` — JWT-rooted per-request-key (`userClaims`) role mapping (#256)

New environment variables

Variable	Required	Default	Description
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_ENABLED`	No	`false`	Adds `total_cost` to row-mode `test_suite_runs` query results.
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_TIMEOUT_SEC`	Conditional	`2`	End-to-end cost lookup deadline; minimum `1`.
`CSV_IMPORT_ZIP_MAX_ENTRIES`	No	`1000`	Maximum ZIP entries accepted during test-case import.
`CSV_IMPORT_ZIP_MAX_TOTAL_UNCOMPRESSED_SIZE`	No	`1GB`	Maximum total uncompressed ZIP size.
`CSV_IMPORT_ZIP_MAX_MANIFEST_SIZE`	No	`1MB`	Maximum `manifest.json` size.
`SPRING_AI_MCP_SERVER_ENABLED`	No	`false`	Enables the inbound MCP server; `/mcp` returns 404 when disabled.
`SPRING_AI_MCP_SERVER_STDIO`	No	`false`	Enables local stdio transport; HTTP is the deployment default.
`SPRING_AI_MCP_SERVER_NAME`	No	`ai-dial-admin-evaluation-framework-backend`	Server name returned by MCP initialization.
`SPRING_AI_MCP_SERVER_VERSION`	No	Project version	Server version returned by MCP initialization.
`SPRING_AI_MCP_SERVER_TYPE`	No	`SYNC`	MCP implementation type; `SYNC` preserves request-thread caller identity.
`SPRING_AI_MCP_SERVER_PROTOCOL`	No	`STATELESS`	HTTP protocol; `STREAMABLE` requires one replica or sticky sessions.
`SPRING_AI_MCP_SERVER_CAPABILITIES_TOOL`	No	`true`	Advertises MCP tools.
`SPRING_AI_MCP_SERVER_CAPABILITIES_RESOURCE`	No	`false`	Advertises MCP resources; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_PROMPT`	No	`false`	Advertises MCP prompts; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_COMPLETION`	No	`false`	Advertises MCP completions; none ship in this release.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_MCP_ENDPOINT`	No	`/mcp`	Sets the authenticated MCP HTTP path.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_DISALLOW_DELETE`	No	`false`	Disables session termination by `DELETE` in `STREAMABLE` mode.
`SPRING_AI_MCP_SERVER_ANNOTATION_SCANNER_ENABLED`	No	`true`	Discovers `@McpTool` methods.
`SPRING_AI_MCP_SERVER_INSTRUCTIONS`	No	Built-in workflow guidance	Instructions returned by MCP initialization.

Enabling Application mode for test suite runs

1. Enable DIAL API-Key authentication first — the internal run-execution endpoint is authenticated
via the Api-Key header, not the user's JWT, so Application mode cannot start without it:

Environment Variable	Default	Required	Description
`API_KEY_ENABLED`	`false`	Yes	Enables DIAL API-Key authentication via the `Api-Key` request header. Must be `true` before `DIAL_APP_PROXY_ENABLED` can be set — EF fails fast at startup otherwise.
`API_KEY_CORE_URL`	-	Yes	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`.
`API_KEY_KEY_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core project-key role names to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`. At least one of this or `API_KEY_JWT_ROLES_MAPPING` must be non-empty.
`API_KEY_JWT_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core roles from the JWT-rooted `userClaims` response shape to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	No	Claim name read from the introspection response's `userClaims` object to obtain the caller's raw roles. Set to `dial_roles` to match this claim naming in Core.

2. Turn on the Application Route flag:

Environment Variable	Default	Required	Description
`DIAL_APP_PROXY_ENABLED`	`false`	No	Enables DIAL Application Routes mode for run execution — eval runs are triggered via DIAL Core's Application Route using a per-request key instead of the user's JWT. Requires `API_KEY_ENABLED=true`.
`DIAL_APP_PROXY_DEPLOYMENT_NAME`	`EF`	No	DIAL Core application deployment name. Must match the `applications.<name>` key used in the DIAL Core config snippet in DIAL Configuration changes below.
`DIAL_APP_PROXY_HEARTBEAT_INTERVAL_MS`	`30000`	No	Heartbeat interval for the internal run SSE connection.
`DIAL_APP_PROXY_TRIGGER_READ_TIMEOUT_MS`	`43200000`	No	DIAL Application Route trigger timeout in milliseconds.

3. Recommended tuning (not required to enable the mode, but needed for long evals):

Environment Variable	Default	Required	Description
`TEST_SUITE_RUN_SSE_TIMEOUT_MINUTES`	`30`	No	SSE client connection timeout, in minutes. Raise this under Application mode (e.g. `720` for up to 12 hours) so the status stream doesn't time out before a long-running eval completes.

Behavioral changes

Note

These take effect automatically on upgrade unless noted otherwise.

API-key role mapping — with OIDC providers configured, introspection matches userClaims.iss and unions the provider's roleClaims; API_KEY_USER_CLAIMS_ROLE_CLAIM is now the fallback (#241, #248)
dial-adas compatibility — run-cost averages now call POST /v1/queries/execute-sql; verify the deployed dial-adas version exposes this endpoint (#207)

DIAL Configuration changes

Important

DIAL Core registration is required only when DIAL_APP_PROXY_ENABLED=true; direct JWT execution remains the default.

Required setup (#236):

Add the applications.EF block to DIAL Core configuration, or use the application name set by DIAL_APP_PROXY_DEPLOYMENT_NAME, and grant it access to the EF file bucket. Apply the snippet with http://eval-framework-url replacement:

 {                                                                                                                                                                                                                                 
   "applications": {                                                                                                                                                                                                               
     "EF": {                                                                                                                                                                                 ...

Assets 2

0.5.0-rc.0 Pre-release

Pre-release

github-actions released this 09 Oct 10:03

0.5.0-rc.0

221a5e5

Features

Add an authenticated inbound MCP endpoint — stateless HTTP exposes list_deployments and get_deployment tools #209 (#208)
Add optional DIAL Application Route execution — per-request keys keep long evaluations authenticated and auto-share EF files (#236)
Query test-suite runs with deployment context, run counts, metric names, overall scores, and optional total costs through the structured Query DSL #197 (#201, #206, #221, #222)
Fetch total costs for up to 1,000 test-suite runs in one request — preserves request order and returns null when usage is absent (#205)
List suite metric definitions with declaration and schema-version details — clients no longer join them locally #224 (#226)

Fixes

Calculate average run costs per test case instead of per usage request — corrects multi-request, multi-turn, and multi-metric suites (#207)
Calculate test-case scores across all turns, requests, and reruns — skipped metrics are excluded and test cases remain equally weighted (#231)
Restore application and model target listings after DIAL Core changed model pricing — accepts both scalar and object prompt / completion values #243 (#244, #247)
Resolve API-key roles against issuer-specific claim paths, including array-wrapped issuer claims — multi-provider authentication now maps authorities correctly (#241, #248)
Preserve multi-turn data, schemas, file refs, and content types in ZIP round trips — imports are transactional and archive parsing is bounded #225 (#227, #228)
Identify the failed test case and broken file reference when ZIP export cannot download an attachment #225 (#232)
Return template variables per request for multi-request suites — extraction and try-out now use request-indexed maps #217 (#235)
Upgrade Jackson to 2.22.3 / 3.2.3 across the service, CLI, runner, and build tooling — addresses CVE-2026-19032, CVE-2026-68497, CVE-2026-83557, CVE-2026-89407, CVE-2026-89425, CVE-2026-91776, and CVE-2026-91777 #242 (#233, #234, #245)

Other

Move run metric snapshots to metadata storage — run deletion now cascades, existing rows are copied, and the old analytics route remains a deprecated alias #197 (#199)

Deployment Changes

New environment variables

Variable	Required	Default	Description
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_ENABLED`	No	`false`	Adds `total_cost` to row-mode `test_suite_runs` query results.
`QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_TIMEOUT_SEC`	Conditional	`2`	End-to-end cost lookup deadline; minimum `1`.
`CSV_IMPORT_ZIP_MAX_ENTRIES`	No	`1000`	Maximum ZIP entries accepted during test-case import.
`CSV_IMPORT_ZIP_MAX_TOTAL_UNCOMPRESSED_SIZE`	No	`1GB`	Maximum total uncompressed ZIP size.
`CSV_IMPORT_ZIP_MAX_MANIFEST_SIZE`	No	`1MB`	Maximum `manifest.json` size.
`SPRING_AI_MCP_SERVER_ENABLED`	No	`false`	Enables the inbound MCP server; `/mcp` returns 404 when disabled.
`SPRING_AI_MCP_SERVER_STDIO`	No	`false`	Enables local stdio transport; HTTP is the deployment default.
`SPRING_AI_MCP_SERVER_NAME`	No	`ai-dial-admin-evaluation-framework-backend`	Server name returned by MCP initialization.
`SPRING_AI_MCP_SERVER_VERSION`	No	Project version	Server version returned by MCP initialization.
`SPRING_AI_MCP_SERVER_TYPE`	No	`SYNC`	MCP implementation type; `SYNC` preserves request-thread caller identity.
`SPRING_AI_MCP_SERVER_PROTOCOL`	No	`STATELESS`	HTTP protocol; `STREAMABLE` requires one replica or sticky sessions.
`SPRING_AI_MCP_SERVER_CAPABILITIES_TOOL`	No	`true`	Advertises MCP tools.
`SPRING_AI_MCP_SERVER_CAPABILITIES_RESOURCE`	No	`false`	Advertises MCP resources; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_PROMPT`	No	`false`	Advertises MCP prompts; none ship in this release.
`SPRING_AI_MCP_SERVER_CAPABILITIES_COMPLETION`	No	`false`	Advertises MCP completions; none ship in this release.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_MCP_ENDPOINT`	No	`/mcp`	Sets the authenticated MCP HTTP path.
`SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_DISALLOW_DELETE`	No	`false`	Disables session termination by `DELETE` in `STREAMABLE` mode.
`SPRING_AI_MCP_SERVER_ANNOTATION_SCANNER_ENABLED`	No	`true`	Discovers `@McpTool` methods.
`SPRING_AI_MCP_SERVER_INSTRUCTIONS`	No	Built-in workflow guidance	Instructions returned by MCP initialization.

Enabling Application mode for test suite runs

1. Enable DIAL API-Key authentication first — the internal run-execution endpoint is authenticated
via the Api-Key header, not the user's JWT, so Application mode cannot start without it:

Environment Variable	Default	Required	Description
`API_KEY_ENABLED`	`false`	Yes	Enables DIAL API-Key authentication via the `Api-Key` request header. Must be `true` before `DIAL_APP_PROXY_ENABLED` can be set — EF fails fast at startup otherwise.
`API_KEY_CORE_URL`	-	Yes	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`.
`API_KEY_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core project-key role names to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`. At least one of this or `API_KEY_DEFAULT_ROLES_MAPPING` must be non-empty.
`API_KEY_DEFAULT_ROLES_MAPPING`	- (empty)	Conditional	JSON object mapping DIAL Core roles from the JWT-rooted `userClaims` response shape to EF authority strings, e.g. `{"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}`.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	No	Claim name read from the introspection response's `userClaims` object to obtain the caller's raw roles. Set to `dial_roles` to match this claim naming in Core.

2. Turn on the Application Route flag:

Environment Variable	Default	Required	Description
`DIAL_APP_PROXY_ENABLED`	`false`	No	Enables DIAL Application Routes mode for run execution — eval runs are triggered via DIAL Core's Application Route using a per-request key instead of the user's JWT. Requires `API_KEY_ENABLED=true`.
`DIAL_APP_PROXY_DEPLOYMENT_NAME`	`EF`	No	DIAL Core application deployment name. Must match the `applications.<name>` key used in the DIAL Core config snippet in DIAL Configuration changes below.
`DIAL_APP_PROXY_HEARTBEAT_INTERVAL_MS`	`30000`	No	Heartbeat interval for the internal run SSE connection.
`DIAL_APP_PROXY_TRIGGER_READ_TIMEOUT_MS`	`43200000`	No	DIAL Application Route trigger timeout in milliseconds.

3. Recommended tuning (not required to enable the mode, but needed for long evals):

Environment Variable	Default	Required	Description
`TEST_SUITE_RUN_SSE_TIMEOUT_MINUTES`	`30`	No	SSE client connection timeout, in minutes. Raise this under Application mode (e.g. `720` for up to 12 hours) so the status stream doesn't time out before a long-running eval completes.

Behavioral changes

Note

These take effect automatically on upgrade unless noted otherwise.

API-key role mapping — with OIDC providers configured, introspection matches userClaims.iss and unions the provider's roleClaims; API_KEY_USER_CLAIMS_ROLE_CLAIM is now the fallback (#241, #248)
dial-adas compatibility — run-cost averages now call POST /v1/queries/execute-sql; verify the deployed dial-adas version exposes this endpoint (#207)

DIAL Configuration changes

Important

DIAL Core registration is required only when DIAL_APP_PROXY_ENABLED=true; direct JWT execution remains the default.

Required setup (#236):

Add the applications.EF block to DIAL Core configuration, or use the application name set by DIAL_APP_PROXY_DEPLOYMENT_NAME, and grant it access to the EF file bucket. Apply the snippet with http://eval-framework-url replacement:

 {                                                                                                                                                                                                                                 
   "applications": {                                                                                                                                                                                                               
     "EF": {                                                                                                                                                                                                                       
       "endpoint": "http://xxx",                                                                                                                                                                                                   
       "display_name": "EF",                                                                                                                                                                                                       
       "routes": {                                                                                                                                                                                                                 
         "runs": {                             ...

Assets 2

0.4.0

github-actions released this 22 Sep 12:17

0.4.0

f9e6463

Features

add endpoint to retrieve run costs (#160)
add overall score computation per test case (#178)
cancellation status update #187 #191 (#190)
deployment costs endpoint (#202)
deployment id validation #188 (#195)
extend metrics api #182 (#183)
extra metric provider env-driven config #167 (#166)
migrate deployment-by-id lookup to unified /v1/deployments/{id} endpoint (#194)
multi request test suites #98 (#114)
openai Responses API support #20 (#186)
query deployment without type (#180)
support Anthropic Messages API (#193)
support try-out for multi-request test suites #162 (#163)
username resolution for test suite's created_by #47 (#184)

Fixes

can save empty required tool argument #69 (#174)
raw sql in error message #22 (#172)

CI

(deps) bump epam/ai-dial-ci from 4.9.0 to 4.11.0 (#189)
(deps) bump step-security/harden-runner from 2.20.1 to 2.21.0 in the github-actions group across 1 directory (#156)
(deps) bump the ai-dial-ci group with 4 updates (#165)

Other

keep gradle up-to-date (#159)
move Unified Query DSL out of "experimental" (#161)
update openspec #169 (#170)
update spring boot version #171 (#173)

Assets 2

0.4.0-rc.0 Pre-release

Pre-release

github-actions released this 16 Sep 09:17

0.4.0-rc.0

8e271cc

Features

Add per-test-case overall score and pass/fail — each eval-summary row gains nullable score / passed, computed by grafting id IN (…) + GROUP BY id onto the existing Phase-3 scoring query so all three definition types (mean, weighted_mean, custom_function) work uniformly; the new optional suite field testCaseOverallScore lets per-row scoring use a different definition than the run-level overall, and both fields are filterable/sortable through the Query DSL #85 (#178)
Cancellation is now interrupt-driven and observable — POST /cancel synchronously writes the new CANCELLING status (RUNNING → CANCELLING → CANCELLED, idempotent, SSE-emitted, reconciled on restart), each run owns an executor shut down with shutdownNow() so in-flight calls abort immediately instead of draining a grace period, and an analytics write failure now correctly ends the run FAILED rather than CANCELLED #187 #191 (#190)
Add run cost analytics — GET /api/v1/costs/test-suite-run/{id} returns avgTestCaseCost and avgMetricEvalCost, the average per-call price of the execution and metric-evaluation phases read live from dial-adas usage logs and correlated through the eval.run.id / eval.phase OTel baggage the service already emits #124 (#160)
Add deployment cost analytics — GET /api/v1/costs/deployment/{id}?from&to returns totalTestCaseCost and totalMetricEvalCost over an inclusive epoch-millisecond range, answering "what has this deployment cost this month" across all runs; all cost logic moves into a dedicated CostService and GET /api/v1/test-suite-runs/{id}/costs stays as a backward-compatible alias #65 (#202)
Support the Anthropic Messages API — a suite targets a Claude-style deployment by setting endpointRef.relativeUrlPattern = "/anthropic/v1/messages", which passes through to DIAL Core's deployment-less route (the body's model field does the routing), and StreamingResponseAccumulator gains a third mode that reassembles message_start / content_block_delta / message_delta SSE events into a proper non-streaming response instead of a raw {"events":[…]} envelope #168 (#193)
Support the OpenAI Responses API — /openai/v1/responses paths now pass through DialCoreUrlBuilder verbatim, and a new ResponsesApiAccumulator uses the terminal response.completed/incomplete/failed event's response object verbatim, so response columns such as output[0].content[0].text resolve identically whether the deployment streamed or not #20 (#186)
Validate that a request body's model matches the suite's deployment — the fixed-path Responses and Messages APIs route on model, not the URL, so a mismatch silently executed against a different deployment; a single RequestModelValidator now enforces it statically at suite save (soft warning, marks the suite invalid) and again at the final pre-invocation boundary for runs, CLI --deployment-id overrides and try-it-out, per request in a chain and per turn #188 (#195)
Look up a deployment without knowing its type — GET /api/v1/deployments/all/{id} resolves any deployment by ID alone, backed by DIAL Core's unified GET /v1/deployments/{id} in a single call instead of three concurrent type probes, and a missing deployment now returns a real 404 instead of a generic 502 UPSTREAM_NOT_FOUND #179 #192 (#180, #194)
Add GET /api/v1/metric-declarations/versions/latest — returns every metric declaration joined with its latest schema version in one call, so clients no longer fetch versions per declaration; served by a new unique (metric_declaration_id, schema_version DESC) index #182 (#183)
Resolve createdBy to a display name — with SECURITY_JWT_RESOLVE_USER_NAME=true the raw JWT claim (an opaque id in most IdPs) is replaced at creation time by userDisplayName from DIAL Core's GET /v1/user/info, falling back to the claim on a missing name or any Core error so the request never fails #47 (#184)

Fixes

Warn when a required MCP tool argument is left empty — MCP_TOOL suite validation never consulted toolRef.inputSchema, so a blank constant, a missing key or a placeholder bound to a blank constant persisted as valid and failed only at run time inside the tool; a schema-driven coverage check now emits a REQUIRED warning at $.argumentTemplate.arguments and marks the suite invalid #69 (#174)
Stop leaking raw SQL in the private-dataset binding error — the PL/pgSQL guard's P0001 error had no Spring translation, so jOOQ's exception fell through to the generic 500 handler and echoed the failing statement to the client; binding a second suite to a PRIVATE dataset is now an app-level 409 PRIVATE_DATASET_ALREADY_BOUND with a fixed message, the DB message logged and never returned #22 (#172)

Other

Promote the Unified Query DSL out of experimental — com.epam.aidial.evaluation.experimental.query.* becomes com.epam.aidial.evaluation.query.* and docs/experimental/ becomes docs/query-dsl/, reflecting that the DSL is load-bearing production functionality; no REST API, DTO, database, configuration or runtime behavior change #115 (#161)
Upgrade Spring Boot to 4.1.1 and drop the build-time dependency pins it now subsumes — the explicit CVE-driven version constraints added in 0.3.0 are supplied by the Boot BOM #171 (#173)

Deployment Changes

New environment variables

Required: ✅ = must be supplied, the service will not start without it. ❌ = ships with a working default. Nothing in this release is required in a default deployment; ⚠️ in a description marks a variable that becomes mandatory once you switch on the feature it belongs to.

Variable	Required	Default	Description
`DIAL_ADAS_URL`	❌	`http://localhost:8087`	⚠️ Base URL for the dial-adas query-execute API, which backs both cost endpoints. The default points at localhost, so it must be overridden before `GET /api/v1/costs/**` returns anything useful.
`DIAL_ADAS_CONNECT_TIMEOUT_MS`	❌	`5000`	Connection timeout in milliseconds for the dial-adas client.
`DIAL_ADAS_READ_TIMEOUT_MS`	❌	`30000`	Read timeout in milliseconds for the dial-adas client.
`SECURITY_JWT_RESOLVE_USER_NAME`	✅	`true`	When `true`, `AuthorResolver` calls DIAL Core `GET /v1/user/info` with the caller's bearer token and stores `userDisplayName` in `createdBy` instead of the raw claim value. Applies only when `config.rest.security.mode=oidc`. The stored value is a snapshot taken at creation time.

Removed environment variables

Variable	Reason
`TEST_SUITE_RUN_EXECUTOR_CORE_POOL_SIZE`	The fixed run-execution thread pool is replaced by an unbounded thread-per-run `SimpleAsyncTaskExecutor`; concurrency is now governed solely by the run limits (#190)
`TEST_SUITE_RUN_EXECUTOR_MAX_POOL_SIZE`	Same as above (#190)
`TEST_SUITE_RUN_EXECUTOR_QUEUE_CAPACITY`	Same as above (#190)
`TEST_SUITE_RUN_EXECUTION_CANCELLATION_GRACE_PERIOD_MS`	Cancellation is interrupt-driven — there is no drain phase left to bound (#190)
`TEST_SUITE_RUN_CANCELLATION_GRACE_PERIOD_MS` / `CLI_RUN_CANCELLATION_GRACE_PERIOD_MS`	The `eval-cli` equivalents of the above (#190)

Note

All five are ignored if still set — a stale value will not fail startup.

Behavioral changes

Note

These take effect automatically on upgrade — no operator action required, but run concurrency and cancellation timing differ from 0.3.0.

Run concurrency — the fixed pool (core 5 / max 10 / queue 50) that capped real concurrency at 5 despite max-concurrent-runs-global = 20 is gone; a run now gets its own thread and the configured run limits are the only ceiling. Expect more runs genuinely executing in parallel, and size the database connection pools and deployment quotas accordingly (#190)
Cancellation timing and outcome — cancelling no longer waits out a grace period; the run's executor is interrupted immediately and the metric-provider client moves to JdkClientHttpRequestFactory so a call blocked on the 150 s read timeout aborts at once. A run whose analytics batch write fails now ends FAILED / ANALYTICS_WRITE_FAILED instead of being mislabelled CANCELLED, and a late cancel can never end COMPLETED #187 #191 (#190)
Deployment by-id lookup — GET /api/v1/deployments/all/{id} issues one request to DIAL Core's unified endpoint instead of three concurrent probes (3x less upstream traffic), and an unknown deployment returns 404 where it previously returned 502 UPSTREAM_NOT_FOUND #192 (#194)
Suite validity on model-selecting endpoints — a suite whose literal request body sets model to something other than its deploymentRef.id is now persisted as invalid with a REQUEST_BODY_VALIDATION_ERROR warning, so it is excluded from runnable selection. Suites saved as valid before this rule existed are caught at run time instead and produce an ERROR row #188 (#193, #195)
Private-dataset double binding — returns 409 PRIVATE_DATASET_ALREADY_BOUND from the service layer instead of a 500 carrying the driver message; the PL/pgSQL trigger is demoted to a race backstop #22 (#172)

Database migrations

Note

Flyway applies these automatically on startup; the service account needs DDL privileges on both schemas.

analytics — V1.19__CreateTestCaseEvalScoresTable.sql — creates test_case_eval_scores (eval_summary_id PK, score, passed, computed_at_ms) for per-row scoring; forward-only, historical rows read back as null (#178)
meta — V1.30__AddUniqueIndexToMetricDeclarationVersions.sql — drops the non-unique idx_metric_declaration_versions_declaration_version from V1.9 and replaces it with a unique uq_metric_declaration_versions_declaration_version on (metric_declaration_id, schema_version DESC) (#183)
meta — `V1.31__AddTestCaseOverallSco...

Assets 2

0.3.0

github-actions released this 26 Aug 14:40

0.3.0

500320f

Features

Introduce multi-turn conversation test cases — a multi-turn case is a single test case carrying an ordered multiTurnData array (mutually exclusive with data), executed as a sequential fail-fast turn loop that writes one result row and one eval summary per turn; try-it-out previews the whole conversation and a suite testCaseFilter matches only when every turn satisfies it #17 (#100, #132, #135, #140, #147, #133)
Introduce multi-request test suites — the suite's own request becomes request #0 of an ordered chain and the new additionalRequests array (max 10, each with its own name, endpointRef, requestTemplate, responseColumns and inputBindings) supplies requests 1..N, executed sequentially against the same deployment so a setup call's output feeds the call under test; response columns form one flat namespace capped at 50 suite-wide, the JSONata frame accumulates monotonically across requests, any subset of the chain may be multi-turn, and a row's identity becomes the pair (request_index, turn_index) #98 (#114, #164)
Try-out is chain-aware — both try-it-out endpoints now run the whole additionalRequests chain, threading one accumulated frame across requests and turns exactly as the run engine does, each history entry carrying its own requestIndex / turnIndex stamps plus its extractedColumns and extractionWarnings; GET …/resolved-request?requestIndex=N previews any position in the chain #162 (#163, #164)
Request-template JSON bodies accept a JSONata expression via the new jsonataContent field — conversation history becomes a user-authored expression over the $_request / $_response frame bindings instead of hardcoded messages reassembly, so multi-turn works against non-chat-completions APIs and streaming works on every turn #17 (#113, #121)
Add conditional metric execution — an optional per-metric-definition condition (JSONata, max 2000 chars) decides per result row whether that metric runs, evaluated over {data, response, turn:{index,total,last}, request:{index,total,last,name}} #17 (#100, #114, #164)
Import externally evaluated test-case results — POST /api/v1/test-suites/{id}/runs/import accepts a results CSV and runs metric evaluation and score computation over it, so a deployment evaluated elsewhere can still be scored by the framework #93 (#99, #122)
Add eval-cli, a standalone DB-free CLI — clone, fetch, run, import and the chaining evaluate command reuse existing suites and the shared Phase 1 engine to evaluate a deployment with no Evaluation Framework instance alongside it; fetch also pulls the bound dataset's test-case schema and the results CSV carries requestIndex / totalRequests / turnIndex / totalTurns, so multi-request and multi-turn suites import with correct row identity; ships with a Dockerfile recipe for CI runners #92 (#112, #117, #128, #154, #164)
Add Api-Key header authentication as an alternative to OIDC/JWT — keys are introspected against DIAL Core's GET /v1/user/info, cached, and mapped to application authorities; off by default and active only in oidc security mode (#134)
Add a run comparison endpoint — GET /api/v1/analytics/metric-scores/comparison?runIds=<a>,<b> recomputes per-metric statistics, overall and average execution duration over only the eval-summary rows the two runs have in common, and returns the ids of the rows that did not match #103 (#106)
Measure metric evaluation time — each eval-summary row gains metricEvalDurationMs, the summed latency of that row's provider /evaluate calls, filterable and aggregatable through the Query DSL #123 (#150)
Expose deployment-reference fields on the test_suites Query DSL entity — deployment_ref::id|name|version and mcp_deployment_ref::id|name|type as flat STRING fields for filter, select, sort and group_by #97 (#101)
Metric provider entries gain a per-entry enabled flag and a stock second slot — METRIC_PROVIDERS_EXTRA_* configures a ready-to-use extra provider (disabled by default), and disabling an entry skips it in the sync job while its already-synced declarations stay in the catalog and remain evaluable during runs #167 (#166, #164)
Make test-suite-run request timeouts configurable — TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT / TEST_SUITE_RUN_EXEC_MAX_TIMEOUT replace the hardcoded values, with generous defaults so slow streaming responses no longer trip a request timeout (#153)

Fixes

Write eval summaries for runs with no metrics configured — listing, count, aggregate and CSV export of such a run returned an empty page or HTTP 404; "latest computation" now resolves from test_case_eval_summaries instead of the legitimately empty metric snapshots (#108)
Carry the full DIAL Core deployment contract through listing — version, owner, timestamps, keywords, attachment types, model capabilities/limits/pricing, application properties and toolset transport were silently dropped, and an unrecognized object discriminator now degrades to a skippable entry instead of failing the whole fetch #130 (#131)
Restore application and model listing after DIAL Core switched deployment names and descriptions to multilingual objects (#145)
Allow running a suite whose testCaseFilter uses an array-field Contains condition — lower/upper over an ARRAY-bound field rendered lower(jsonb) and failed run creation with an unhandled HTTP 500; it now translates to case-insensitive whole-element JSONB containment #142 (#148)
Stop nc / ne / not filters from silently dropping rows whose field is null or absent — negated comparisons are now total, so an absent value correctly satisfies "does not contain" #141 (#143)
Ignore the legacy disabled_test_case_ids exclusion during run selection — suites created before run conditions existed reported "39 of 39 selected" while executing 4 cases, and adding a run condition failed POST /runs with 409 #151 (#152)
Upgrade Jackson to 2.21.5 / 3.1.5 to address CVE-2026-59889, CVE-2026-54515 and GHSA-mhm7-754m-9p8w, and Netty to 4.2.17.Final; build-time constraints added for CVE-2026-54399, CVE-2026-54428, CVE-2026-64607, CVE-2025-48924 and CVE-2025-67030 (#155)
Patch OS packages inherited from the container base image at build time — the runtime stage now runs apk --no-cache upgrade, picking up libssl3 / libcrypto3 advisories fixed after the base image was cut (#166, #164)
Return a short projection from GET /api/v1/deployments — the full-contract listing added in 0.3.0-rc.0 inflated the payload; entries now carry only deploymentId, displayName and description (plus transport for toolsets), and the full representation is fetched per deployment (regression fix, affects pre-release users of the expanded deployment contract only) (#157)

Deployment Changes

New environment variables

Required: ✅ = must be supplied, the service will not start without it. ❌ = ships with a working default. Nothing in this release is required in a default deployment; ⚠️ in a description marks a variable that becomes mandatory once you switch on the feature it belongs to.

Variable	Required	Default	Description
`API_KEY_ENABLED`	❌	`false`	Enables DIAL API-Key authentication via the `Api-Key` request header. Applies only when `config.rest.security.mode=oidc`.
`API_KEY_CORE_URL`	❌	-	⚠️ Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`. Empty by default; startup fails with `…requires …core-url to be set` when `API_KEY_ENABLED=true`.
`API_KEY_CACHE_TTL_SECONDS`	❌	`60`	Time-to-live, in seconds, for cached successful introspection results.
`API_KEY_CACHE_MAX_SIZE`	❌	`10000`	Maximum number of cached introspection results.
`API_KEY_REQUEST_TIMEOUT_MS`	❌	`3000`	Connect/read timeout, in milliseconds, for the introspection call to DIAL Core.
`API_KEY_ROLES_MAPPING`	❌	- (empty)	⚠️ JSON object mapping DIAL Core project-key role names to lists of this service's authority strings. When `API_KEY_ENABLED=true`, at least one of the two role mappings must be non-empty or startup fails.
`API_KEY_DEFAULT_ROLES_MAPPING`	❌	- (empty)	⚠️ JSON object mapping DIAL Core roles from the JWT-rooted per-request-key response shape (`userClaims`) to lists of this service's authority strings. When `API_KEY_ENABLED=true`, at least one of the two role mappings must be non-empty or startup fails.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	❌	`roles`	Claim name read out of the introspection response's `userClaims` object to obtain the caller's raw roles.
`API_KEY_STARTUP_PROBE`	❌	`true`	When `true`, the service calls DIAL Core's `/v1/user/info` at startup and fails to start if Core is unreachable or misconfigured.
`ANALYTICS_COMPARISON_MAX_UNMATCHED_ROWS`	❌	`5000`	Maximum number of non-matching eval-summary rows a single run comparison may report per run; exceeding it fails the request with HTTP 409. Minimum `1`.
`JSONATA_EVALUATION_TIMEOUT_MS`	❌	`10000`	Maximum wall-clock time in milliseconds a single JSONata expression evaluation may run before it is aborted. Minimum `1`.
`JSONATA_MAX_RECURSION_DEPTH`	❌	`1000`	Maximum call-stack recursion depth a single JSONata expression evaluation may reach before it is aborted. Minimum `1`.
`ANALYTICS_RESULTS_CSV_IMPORT_MAX_FILE_SIZE`	❌	`10MB`	Maximum CSV file size for the eval-results import endpoint. Larger requests are rejected with HTTP 400 before parsing begins.
`TEST_CASE_MULTI_TURN_MAX_TURNS`	❌	`10`	Maximum number of turns a multi-turn test case may carry. An over-cap case is persisted but marked invalid with a warning, so it is excluded from runnable selection.
`TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT`	❌	`600000`	Default per-request timeout, in milliseconds, for deployment requests issued during a test ...

Assets 2

0.3.0-rc.2 Pre-release

Pre-release

github-actions released this 26 Aug 13:40

0.3.0-rc.2

a97a25b

Features

Introduce multi-turn conversation test cases — a multi-turn case is a single test case carrying an ordered multiTurnData array (mutually exclusive with data), executed as a sequential fail-fast turn loop that writes one result row and one eval summary per turn; try-it-out previews the whole conversation and a suite testCaseFilter matches only when every turn satisfies it #17 (#100, #132, #135, #140, #147, #133)
Introduce multi-request test suites — the suite's own request becomes request #0 of an ordered chain and the new additionalRequests array (max 10, each with its own name, endpointRef, requestTemplate, responseColumns and inputBindings) supplies requests 1..N, executed sequentially against the same deployment so a setup call's output feeds the call under test; response columns form one flat namespace capped at 50 suite-wide, the JSONata frame accumulates monotonically across requests, any subset of the chain may be multi-turn, and a row's identity becomes the pair (request_index, turn_index) #98 (#114, #164)
Try-out is chain-aware — both try-it-out endpoints now run the whole additionalRequests chain, threading one accumulated frame across requests and turns exactly as the run engine does, each history entry carrying its own requestIndex / turnIndex stamps plus its extractedColumns and extractionWarnings; GET …/resolved-request?requestIndex=N previews any position in the chain #162 (#163, #164)
Request-template JSON bodies accept a JSONata expression via the new jsonataContent field — conversation history becomes a user-authored expression over the $_request / $_response frame bindings instead of hardcoded messages reassembly, so multi-turn works against non-chat-completions APIs and streaming works on every turn #17 (#113, #121)
Add conditional metric execution — an optional per-metric-definition condition (JSONata, max 2000 chars) decides per result row whether that metric runs, evaluated over {data, response, turn:{index,total,last}, request:{index,total,last,name}} #17 (#100, #114, #164)
Import externally evaluated test-case results — POST /api/v1/test-suites/{id}/runs/import accepts a results CSV and runs metric evaluation and score computation over it, so a deployment evaluated elsewhere can still be scored by the framework #93 (#99, #122)
Add eval-cli, a standalone DB-free CLI — clone, fetch, run, import and the chaining evaluate command reuse existing suites and the shared Phase 1 engine to evaluate a deployment with no Evaluation Framework instance alongside it; fetch also pulls the bound dataset's test-case schema and the results CSV carries requestIndex / totalRequests / turnIndex / totalTurns, so multi-request and multi-turn suites import with correct row identity; ships with a Dockerfile recipe for CI runners #92 (#112, #117, #128, #154, #164)
Add Api-Key header authentication as an alternative to OIDC/JWT — keys are introspected against DIAL Core's GET /v1/user/info, cached, and mapped to application authorities; off by default and active only in oidc security mode (#134)
Add a run comparison endpoint — GET /api/v1/analytics/metric-scores/comparison?runIds=<a>,<b> recomputes per-metric statistics, overall and average execution duration over only the eval-summary rows the two runs have in common, and returns the ids of the rows that did not match #103 (#106)
Measure metric evaluation time — each eval-summary row gains metricEvalDurationMs, the summed latency of that row's provider /evaluate calls, filterable and aggregatable through the Query DSL #123 (#150)
Expose deployment-reference fields on the test_suites Query DSL entity — deployment_ref::id|name|version and mcp_deployment_ref::id|name|type as flat STRING fields for filter, select, sort and group_by #97 (#101)
Metric provider entries gain a per-entry enabled flag and a stock second slot — METRIC_PROVIDERS_EXTRA_* configures a ready-to-use extra provider (disabled by default), and disabling an entry skips it in the sync job while its already-synced declarations stay in the catalog and remain evaluable during runs #167 (#166, #164)
Make test-suite-run request timeouts configurable — TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT / TEST_SUITE_RUN_EXEC_MAX_TIMEOUT replace the hardcoded values, with generous defaults so slow streaming responses no longer trip a request timeout (#153)

Fixes

Write eval summaries for runs with no metrics configured — listing, count, aggregate and CSV export of such a run returned an empty page or HTTP 404; "latest computation" now resolves from test_case_eval_summaries instead of the legitimately empty metric snapshots (#108)
Carry the full DIAL Core deployment contract through listing — version, owner, timestamps, keywords, attachment types, model capabilities/limits/pricing, application properties and toolset transport were silently dropped, and an unrecognized object discriminator now degrades to a skippable entry instead of failing the whole fetch #130 (#131)
Restore application and model listing after DIAL Core switched deployment names and descriptions to multilingual objects (#145)
Allow running a suite whose testCaseFilter uses an array-field Contains condition — lower/upper over an ARRAY-bound field rendered lower(jsonb) and failed run creation with an unhandled HTTP 500; it now translates to case-insensitive whole-element JSONB containment #142 (#148)
Stop nc / ne / not filters from silently dropping rows whose field is null or absent — negated comparisons are now total, so an absent value correctly satisfies "does not contain" #141 (#143)
Ignore the legacy disabled_test_case_ids exclusion during run selection — suites created before run conditions existed reported "39 of 39 selected" while executing 4 cases, and adding a run condition failed POST /runs with 409 #151 (#152)
Upgrade Jackson to 2.21.5 / 3.1.5 to address CVE-2026-59889, CVE-2026-54515 and GHSA-mhm7-754m-9p8w, and Netty to 4.2.17.Final; build-time constraints added for CVE-2026-54399, CVE-2026-54428, CVE-2026-64607, CVE-2025-48924 and CVE-2025-67030 (#155)
Patch OS packages inherited from the container base image at build time — the runtime stage now runs apk --no-cache upgrade, picking up libssl3 / libcrypto3 advisories fixed after the base image was cut (#166, #164)
Return a short projection from GET /api/v1/deployments — the full-contract listing added in 0.3.0-rc.0 inflated the payload; entries now carry only deploymentId, displayName and description (plus transport for toolsets), and the full representation is fetched per deployment (regression fix, affects pre-release users of the expanded deployment contract only) (#157)

Deployment Changes

New environment variables

Required: ✅ = must be supplied, the service will not start without it. ❌ = ships with a working default. Nothing in this release is required in a default deployment; ⚠️ in a description marks a variable that becomes mandatory once you switch on the feature it belongs to.

Variable	Required	Default	Description
`API_KEY_ENABLED`	❌	`false`	Enables DIAL API-Key authentication via the `Api-Key` request header. Applies only when `config.rest.security.mode=oidc`.
`API_KEY_CORE_URL`	❌	-	⚠️ Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`. Empty by default; startup fails with `…requires …core-url to be set` when `API_KEY_ENABLED=true`.
`API_KEY_CACHE_TTL_SECONDS`	❌	`60`	Time-to-live, in seconds, for cached successful introspection results.
`API_KEY_CACHE_MAX_SIZE`	❌	`10000`	Maximum number of cached introspection results.
`API_KEY_REQUEST_TIMEOUT_MS`	❌	`3000`	Connect/read timeout, in milliseconds, for the introspection call to DIAL Core.
`API_KEY_ROLES_MAPPING`	❌	- (empty)	⚠️ JSON object mapping DIAL Core project-key role names to lists of this service's authority strings. When `API_KEY_ENABLED=true`, at least one of the two role mappings must be non-empty or startup fails.
`API_KEY_DEFAULT_ROLES_MAPPING`	❌	- (empty)	⚠️ JSON object mapping DIAL Core roles from the JWT-rooted per-request-key response shape (`userClaims`) to lists of this service's authority strings. When `API_KEY_ENABLED=true`, at least one of the two role mappings must be non-empty or startup fails.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	❌	`roles`	Claim name read out of the introspection response's `userClaims` object to obtain the caller's raw roles.
`API_KEY_STARTUP_PROBE`	❌	`true`	When `true`, the service calls DIAL Core's `/v1/user/info` at startup and fails to start if Core is unreachable or misconfigured.
`ANALYTICS_COMPARISON_MAX_UNMATCHED_ROWS`	❌	`5000`	Maximum number of non-matching eval-summary rows a single run comparison may report per run; exceeding it fails the request with HTTP 409. Minimum `1`.
`JSONATA_EVALUATION_TIMEOUT_MS`	❌	`10000`	Maximum wall-clock time in milliseconds a single JSONata expression evaluation may run before it is aborted. Minimum `1`.
`JSONATA_MAX_RECURSION_DEPTH`	❌	`1000`	Maximum call-stack recursion depth a single JSONata expression evaluation may reach before it is aborted. Minimum `1`.
`ANALYTICS_RESULTS_CSV_IMPORT_MAX_FILE_SIZE`	❌	`10MB`	Maximum CSV file size for the eval-results import endpoint. Larger requests are rejected with HTTP 400 before parsing begins.
`TEST_CASE_MULTI_TURN_MAX_TURNS`	❌	`10`	Maximum number of turns a multi-turn test case may carry. An over-cap case is persisted but marked invalid with a warning, so it is excluded from runnable selection.
`TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT`	❌	`600000`	Default per-request timeout, in milliseconds, for deployment requests issued during a test ...

Assets 2

0.3.0-rc.1 Pre-release

Pre-release

github-actions released this 20 Aug 10:18

0.3.0-rc.1

03bc412

Features

Introduce multi-turn conversation test cases — a multi-turn case is a single test case carrying an ordered multiTurnData array (mutually exclusive with data), executed as a sequential fail-fast turn loop that writes one result row and one eval summary per turn; try-it-out previews the whole conversation and a suite testCaseFilter matches only when every turn satisfies it #17 (#100, #132, #135, #140, #147, #133)
Request-template JSON bodies accept a JSONata expression via the new jsonataContent field — conversation history becomes a user-authored expression over the $_request / $_response frame bindings instead of hardcoded messages reassembly, so multi-turn works against non-chat-completions APIs and streaming works on every turn #17 (#113, #121)
Add conditional metric execution — an optional per-metric-definition condition (JSONata, max 2000 chars) decides per result row whether that metric runs, evaluated over {data, response, turn:{index,total,last}} #17 (#100)
Import externally evaluated test-case results — POST /api/v1/test-suites/{id}/runs/import accepts a results CSV and runs metric evaluation and score computation over it, so a deployment evaluated elsewhere can still be scored by the framework #93 (#99, #122)
Add eval-cli, a standalone DB-free CLI — clone, fetch, run, import and the chaining evaluate command reuse existing suites and the shared Phase 1 engine to evaluate a deployment with no Evaluation Framework instance alongside it; ships with a Dockerfile recipe for CI runners #92 (#112, #117, #128, #154)
Add Api-Key header authentication as an alternative to OIDC/JWT — keys are introspected against DIAL Core's GET /v1/user/info, cached, and mapped to application authorities; off by default and active only in oidc security mode (#134)
Add a run comparison endpoint — GET /api/v1/analytics/metric-scores/comparison?runIds=<a>,<b> recomputes per-metric statistics, overall and average execution duration over only the eval-summary rows the two runs have in common, and returns the ids of the rows that did not match #103 (#106)
Measure metric evaluation time — each eval-summary row gains metricEvalDurationMs, the summed latency of that row's provider /evaluate calls, filterable and aggregatable through the Query DSL #123 (#150)
Expose deployment-reference fields on the test_suites Query DSL entity — deployment_ref::id|name|version and mcp_deployment_ref::id|name|type as flat STRING fields for filter, select, sort and group_by #97 (#101)
Make test-suite-run request timeouts configurable — TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT / TEST_SUITE_RUN_EXEC_MAX_TIMEOUT replace the hardcoded values, with generous defaults so slow streaming responses no longer trip a request timeout (#153)

Fixes

Write eval summaries for runs with no metrics configured — listing, count, aggregate and CSV export of such a run returned an empty page or HTTP 404; "latest computation" now resolves from test_case_eval_summaries instead of the legitimately empty metric snapshots (#108)
Carry the full DIAL Core deployment contract through listing — version, owner, timestamps, keywords, attachment types, model capabilities/limits/pricing, application properties and toolset transport were silently dropped, and an unrecognized object discriminator now degrades to a skippable entry instead of failing the whole fetch #130 (#131)
Restore application and model listing after DIAL Core switched deployment names and descriptions to multilingual objects (#145)
Allow running a suite whose testCaseFilter uses an array-field Contains condition — lower/upper over an ARRAY-bound field rendered lower(jsonb) and failed run creation with an unhandled HTTP 500; it now translates to case-insensitive whole-element JSONB containment #142 (#148)
Stop nc / ne / not filters from silently dropping rows whose field is null or absent — negated comparisons are now total, so an absent value correctly satisfies "does not contain" #141 (#143)
Ignore the legacy disabled_test_case_ids exclusion during run selection — suites created before run conditions existed reported "39 of 39 selected" while executing 4 cases, and adding a run condition failed POST /runs with 409 #151 (#152)
Upgrade Jackson to 2.21.5 / 3.1.5 to address CVE-2026-59889, CVE-2026-54515 and GHSA-mhm7-754m-9p8w, and Netty to 4.2.17.Final; build-time constraints added for CVE-2026-54399, CVE-2026-54428, CVE-2026-64607, CVE-2025-48924 and CVE-2025-67030 (#155)
Return a short projection from GET /api/v1/deployments — the full-contract listing added in 0.3.0-rc.0 inflated the payload; entries now carry only deploymentId, displayName and description (plus transport for toolsets), and the full representation is fetched per deployment (regression fix, affects pre-release users of the expanded deployment contract only) (#157)

Deployment Changes

New environment variables

Variable	Default	Description
`API_KEY_ENABLED`	`false`	Enables DIAL API-Key authentication via the `Api-Key` request header. Applies only when `config.rest.security.mode=oidc`.
`API_KEY_CORE_URL`	-	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`. Required when API-Key auth is enabled.
`API_KEY_CACHE_TTL_SECONDS`	`60`	Time-to-live, in seconds, for cached successful introspection results.
`API_KEY_CACHE_MAX_SIZE`	`10000`	Maximum number of cached introspection results.
`API_KEY_REQUEST_TIMEOUT_MS`	`3000`	Connect/read timeout, in milliseconds, for the introspection call to DIAL Core.
`API_KEY_ROLES_MAPPING`	- (empty)	JSON object mapping DIAL Core project-key role names to lists of this service's authority strings. Conditional — at least one of the two role mappings must be non-empty.
`API_KEY_DEFAULT_ROLES_MAPPING`	- (empty)	JSON object mapping DIAL Core roles from the JWT-rooted per-request-key response shape (`userClaims`) to lists of this service's authority strings.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	Claim name read out of the introspection response's `userClaims` object to obtain the caller's raw roles.
`API_KEY_STARTUP_PROBE`	`true`	When `true`, the service calls DIAL Core's `/v1/user/info` at startup and fails to start if Core is unreachable or misconfigured.
`ANALYTICS_COMPARISON_MAX_UNMATCHED_ROWS`	`5000`	Maximum number of non-matching eval-summary rows a single run comparison may report per run; exceeding it fails the request with HTTP 409. Minimum `1`.
`JSONATA_EVALUATION_TIMEOUT_MS`	`10000`	Maximum wall-clock time in milliseconds a single JSONata expression evaluation may run before it is aborted. Minimum `1`.
`JSONATA_MAX_RECURSION_DEPTH`	`1000`	Maximum call-stack recursion depth a single JSONata expression evaluation may reach before it is aborted. Minimum `1`.
`ANALYTICS_RESULTS_CSV_IMPORT_MAX_FILE_SIZE`	`10MB`	Maximum CSV file size for the eval-results import endpoint. Larger requests are rejected with HTTP 400 before parsing begins.
`TEST_CASE_MULTI_TURN_MAX_TURNS`	`10`	Maximum number of turns a multi-turn test case may carry. An over-cap case is persisted but marked invalid with a warning, so it is excluded from runnable selection.
`TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT`	`600000`	Default per-request timeout, in milliseconds, for deployment requests issued during a test suite run.
`TEST_SUITE_RUN_EXEC_MAX_TIMEOUT`	`1200000`	Upper bound, in milliseconds, a run may request for its per-request timeout.
`VIRTUAL_THREADS_ENABLED`	`true`	Toggles Spring's virtual-thread executor; set to `false` to aid troubleshooting.

Behavioral changes

Note

These take effect automatically on upgrade — no operator action required, but existing suites may select or time out differently than they did on 0.2.0.

Test suite run request timeouts — the default per-request timeout rises from 30s to 600s and the maximum from 600s to 1200s, so a slow streaming deployment no longer trips a request timeout (#153)
Query DSL null polarity — nc, ne and not are now total over nulls: a row whose field is null or absent satisfies a negated predicate instead of being dropped. Suite testCaseFilters using these operators will select more cases than before; the previous behavior is expressible as and(ne(field, null), nc(field, "x")) #141 (#143)
Run selection — the only rule is now is_valid = TRUE AND testCaseFilter. Suites carrying stale test_suites.disabled_test_case_ids values run every valid, filter-matching case; the column is retained but no longer read or written, with the drop deferred to a follow-up #151 (#152)
Suite snapshots on imported runs — the snapshot phase now runs for every run, so an imported run scores against frozen suite config instead of resynthesizing from the live suite (#122)

Database migrations

Note

Flyway applies these automatically on startup; the service account needs DDL privileges on both schemas. All column adds are backward compatible — nullable JSONB, or NOT NULL DEFAULT backfilled in a metadata-only statement with no table rewrite.

meta — V1.26__AddConditionToTestSuiteMetricDefinitions.sql — adds the nullable condition VARCHAR(2000) column to test_suite_metric_definitions (#100)
meta — V1.27__AddMultiTurnDataToTestCases.sql — adds the nullable multi_turn_data JSONB column plus a mutual-exclusivity CHECK to test_cases (#100)
meta — V1.28__AddMultiTurnDataToTestCaseRunInputs.sql — adds the nullable multi_turn_data JSONB column to test_case_run_inputs for the suite-run snapshot (#100)
analytics — V1.13__AddTurnColumnsToTestCaseRunResults.sql — adds turn_index / total_turns to test_case_run_results and extends `...

Assets 2

0.3.0-rc.0 Pre-release

Pre-release

github-actions released this 19 Aug 12:47

0.3.0-rc.0

57f4853

Features

Introduce multi-turn conversation test cases — a multi-turn case is a single test case carrying an ordered multiTurnData array (mutually exclusive with data), executed as a sequential fail-fast turn loop that writes one result row and one eval summary per turn; try-it-out previews the whole conversation and a suite testCaseFilter matches only when every turn satisfies it #17 (#100, #132, #135, #140, #147, #133)
Request-template JSON bodies accept a JSONata expression via the new jsonataContent field — conversation history becomes a user-authored expression over the $_request / $_response frame bindings instead of hardcoded messages reassembly, so multi-turn works against non-chat-completions APIs and streaming works on every turn #17 (#113, #121)
Add conditional metric execution — an optional per-metric-definition condition (JSONata, max 2000 chars) decides per result row whether that metric runs, evaluated over {data, response, turn:{index,total,last}} #17 (#100)
Import externally evaluated test-case results — POST /api/v1/test-suites/{id}/runs/import accepts a results CSV and runs metric evaluation and score computation over it, so a deployment evaluated elsewhere can still be scored by the framework #93 (#99, #122)
Add eval-cli, a standalone DB-free CLI — clone, fetch, run, import and the chaining evaluate command reuse existing suites and the shared Phase 1 engine to evaluate a deployment with no Evaluation Framework instance alongside it; ships with a Dockerfile recipe for CI runners #92 (#112, #117, #128, #154)
Add Api-Key header authentication as an alternative to OIDC/JWT — keys are introspected against DIAL Core's GET /v1/user/info, cached, and mapped to application authorities; off by default and active only in oidc security mode (#134)
Add a run comparison endpoint — GET /api/v1/analytics/metric-scores/comparison?runIds=<a>,<b> recomputes per-metric statistics, overall and average execution duration over only the eval-summary rows the two runs have in common, and returns the ids of the rows that did not match #103 (#106)
Measure metric evaluation time — each eval-summary row gains metricEvalDurationMs, the summed latency of that row's provider /evaluate calls, filterable and aggregatable through the Query DSL #123 (#150)
Expose deployment-reference fields on the test_suites Query DSL entity — deployment_ref::id|name|version and mcp_deployment_ref::id|name|type as flat STRING fields for filter, select, sort and group_by #97 (#101)
Make test-suite-run request timeouts configurable — TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT / TEST_SUITE_RUN_EXEC_MAX_TIMEOUT replace the hardcoded values, with generous defaults so slow streaming responses no longer trip a request timeout (#153)

Fixes

Write eval summaries for runs with no metrics configured — listing, count, aggregate and CSV export of such a run returned an empty page or HTTP 404; "latest computation" now resolves from test_case_eval_summaries instead of the legitimately empty metric snapshots (#108)
Carry the full DIAL Core deployment contract through listing — version, owner, timestamps, keywords, attachment types, model capabilities/limits/pricing, application properties and toolset transport were silently dropped, and an unrecognized object discriminator now degrades to a skippable entry instead of failing the whole fetch #130 (#131)
Restore application and model listing after DIAL Core switched deployment names and descriptions to multilingual objects (#145)
Allow running a suite whose testCaseFilter uses an array-field Contains condition — lower/upper over an ARRAY-bound field rendered lower(jsonb) and failed run creation with an unhandled HTTP 500; it now translates to case-insensitive whole-element JSONB containment #142 (#148)
Stop nc / ne / not filters from silently dropping rows whose field is null or absent — negated comparisons are now total, so an absent value correctly satisfies "does not contain" #141 (#143)
Ignore the legacy disabled_test_case_ids exclusion during run selection — suites created before run conditions existed reported "39 of 39 selected" while executing 4 cases, and adding a run condition failed POST /runs with 409 #151 (#152)
Upgrade Jackson to 2.21.5 / 3.1.5 to address CVE-2026-59889, CVE-2026-54515 and GHSA-mhm7-754m-9p8w, and Netty to 4.2.17.Final; build-time constraints added for CVE-2026-54399, CVE-2026-54428, CVE-2026-64607, CVE-2025-48924 and CVE-2025-67030 (#155)

Deployment Changes

New environment variables

Variable	Default	Description
`API_KEY_ENABLED`	`false`	Enables DIAL API-Key authentication via the `Api-Key` request header. Applies only when `config.rest.security.mode=oidc`.
`API_KEY_CORE_URL`	-	Base URL of the DIAL Core instance used to introspect API keys via `GET /v1/user/info`. Required when API-Key auth is enabled.
`API_KEY_CACHE_TTL_SECONDS`	`60`	Time-to-live, in seconds, for cached successful introspection results.
`API_KEY_CACHE_MAX_SIZE`	`10000`	Maximum number of cached introspection results.
`API_KEY_REQUEST_TIMEOUT_MS`	`3000`	Connect/read timeout, in milliseconds, for the introspection call to DIAL Core.
`API_KEY_ROLES_MAPPING`	- (empty)	JSON object mapping DIAL Core project-key role names to lists of this service's authority strings. Conditional — at least one of the two role mappings must be non-empty.
`API_KEY_DEFAULT_ROLES_MAPPING`	- (empty)	JSON object mapping DIAL Core roles from the JWT-rooted per-request-key response shape (`userClaims`) to lists of this service's authority strings.
`API_KEY_USER_CLAIMS_ROLE_CLAIM`	`roles`	Claim name read out of the introspection response's `userClaims` object to obtain the caller's raw roles.
`API_KEY_STARTUP_PROBE`	`true`	When `true`, the service calls DIAL Core's `/v1/user/info` at startup and fails to start if Core is unreachable or misconfigured.
`ANALYTICS_COMPARISON_MAX_UNMATCHED_ROWS`	`5000`	Maximum number of non-matching eval-summary rows a single run comparison may report per run; exceeding it fails the request with HTTP 409. Minimum `1`.
`JSONATA_EVALUATION_TIMEOUT_MS`	`10000`	Maximum wall-clock time in milliseconds a single JSONata expression evaluation may run before it is aborted. Minimum `1`.
`JSONATA_MAX_RECURSION_DEPTH`	`1000`	Maximum call-stack recursion depth a single JSONata expression evaluation may reach before it is aborted. Minimum `1`.
`ANALYTICS_RESULTS_CSV_IMPORT_MAX_FILE_SIZE`	`10MB`	Maximum CSV file size for the eval-results import endpoint. Larger requests are rejected with HTTP 400 before parsing begins.
`TEST_CASE_MULTI_TURN_MAX_TURNS`	`10`	Maximum number of turns a multi-turn test case may carry. An over-cap case is persisted but marked invalid with a warning, so it is excluded from runnable selection.
`TEST_SUITE_RUN_EXEC_DEFAULT_TIMEOUT`	`600000`	Default per-request timeout, in milliseconds, for deployment requests issued during a test suite run.
`TEST_SUITE_RUN_EXEC_MAX_TIMEOUT`	`1200000`	Upper bound, in milliseconds, a run may request for its per-request timeout.
`VIRTUAL_THREADS_ENABLED`	`true`	Toggles Spring's virtual-thread executor; set to `false` to aid troubleshooting.

Behavioral changes

Note

These take effect automatically on upgrade — no operator action required, but existing suites may select or time out differently than they did on 0.2.0.

Test suite run request timeouts — the default per-request timeout rises from 30s to 600s and the maximum from 600s to 1200s, so a slow streaming deployment no longer trips a request timeout (#153)
Query DSL null polarity — nc, ne and not are now total over nulls: a row whose field is null or absent satisfies a negated predicate instead of being dropped. Suite testCaseFilters using these operators will select more cases than before; the previous behavior is expressible as and(ne(field, null), nc(field, "x")) #141 (#143)
Run selection — the only rule is now is_valid = TRUE AND testCaseFilter. Suites carrying stale test_suites.disabled_test_case_ids values run every valid, filter-matching case; the column is retained but no longer read or written, with the drop deferred to a follow-up #151 (#152)
Suite snapshots on imported runs — the snapshot phase now runs for every run, so an imported run scores against frozen suite config instead of resynthesizing from the live suite (#122)

Database migrations

Note

meta — V1.26__AddConditionToTestSuiteMetricDefinitions.sql — adds the nullable condition VARCHAR(2000) column to test_suite_metric_definitions (#100)
meta — V1.27__AddMultiTurnDataToTestCases.sql — adds the nullable multi_turn_data JSONB column plus a mutual-exclusivity CHECK to test_cases (#100)
meta — V1.28__AddMultiTurnDataToTestCaseRunInputs.sql — adds the nullable multi_turn_data JSONB column to test_case_run_inputs for the suite-run snapshot (#100)
analytics — V1.13__AddTurnColumnsToTestCaseRunResults.sql — adds turn_index / total_turns to test_case_run_results and extends uq_results_run_case_index with turn_index (#100)
analytics — V1.14__AddTurnColumnsToEvalSummaries.sql — adds turn_index / total_turns to test_case_eval_summaries and extends uq_eval_summaries_natural_key with turn_index (#100)
analytics — V1.15__AddEvalSummariesRunComputedAtIndex.sql — adds a covering `(test_suite_run_id, computed_at_ms DESC, comput...

Assets 2

Releases: epam/ai-dial-admin-evaluation-framework-backend

Release list

0.5.0

Features

Fixes

Other

Deployment Changes

Removed environment variables

New environment variables

Enabling Application mode for test suite runs

Behavioral changes

DIAL Configuration changes

Uh oh!

0.5.0-rc.2

Features

Fixes

Other

Deployment Changes

Removed environment variables

New environment variables

Enabling Application mode for test suite runs

Behavioral changes

DIAL Configuration changes

Uh oh!

0.5.0-rc.1

Features

Fixes

Other

Deployment Changes

Removed environment variables

New environment variables

Enabling Application mode for test suite runs

Behavioral changes

DIAL Configuration changes

Uh oh!

0.5.0-rc.0

Features

Fixes

Other

Deployment Changes

New environment variables

Enabling Application mode for test suite runs

Behavioral changes

DIAL Configuration changes

Uh oh!

0.4.0

Features

Fixes

CI

Other

Uh oh!

0.4.0-rc.0

Features

Fixes

Other

Deployment Changes

New environment variables

Removed environment variables

Behavioral changes

Database migrations

Uh oh!

0.3.0

Features

Fixes

Deployment Changes

New environment variables

Uh oh!

0.3.0-rc.2

Features

Fixes

Deployment Changes

New environment variables

Uh oh!

0.3.0-rc.1

Features

Fixes

Deployment Changes

New environment variables

Behavioral changes

Database migrations