0.5.0

Features

• Add an authenticated inbound MCP endpoint — stateless HTTP exposes list_deployments and get_deployment tools #209 (#208)
• Add optional DIAL Application Route execution — per-request keys keep long evaluations authenticated and auto-share EF files (#236)
• Query test-suite runs with deployment context, run counts, metric names, overall scores, and optional total costs through the structured Query DSL #197 (#201, #206, #221, #222)
• Fetch total costs for up to 1,000 test-suite runs in one request — preserves request order and returns null when usage is absent (#205)
• List suite metric definitions with declaration and schema-version details — clients no longer join them locally #224 (#226)

Fixes

• Calculate average run costs per test case instead of per usage request — corrects multi-request, multi-turn, and multi-metric suites (#207)
• Calculate test-case scores across all turns, requests, and reruns — skipped metrics are excluded and test cases remain equally weighted (#231)
• Restore application and model target listings after DIAL Core changed model pricing — accepts both scalar and object prompt / completion values #243 (#244, #247)
• Resolve API-key roles against issuer-specific claim paths, including array-wrapped issuer claims — multi-provider authentication now maps authorities correctly (#241, #248)
• Preserve multi-turn data, schemas, file refs, and content types in ZIP round trips — imports are transactional and archive parsing is bounded #225 (#227, #228)
• Identify the failed test case and broken file reference when ZIP export cannot download an attachment #225 (#232)
• Return template variables per request for multi-request suites — extraction and try-out now use request-indexed maps #217 (#235)
• Upgrade Jackson to 2.22.3 / 3.2.3 across the service, CLI, runner, and build tooling — addresses CVE-2026-19032, CVE-2026-68497, CVE-2026-83557, CVE-2026-89407, CVE-2026-89425, CVE-2026-91776, and CVE-2026-91777 #242 (#233, #234, #245)
• Preserve test-case values when renaming dataset schema fields — stable IDs move stored data, while bound public datasets reject destructive updates and imports #246 (#249, #254)

Other

• Move run metric snapshots to metadata storage — run deletion now cascades, existing rows are copied, and the old analytics route remains a deprecated alias #197 (#199)
• Rename the API-key role-mapping variables to API_KEY_KEY_ROLES_MAPPING / API_KEY_JWT_ROLES_MAPPING — names now distinguish project-key from JWT-rooted per-request-key mappings; old names are no longer read (#256)

Deployment Changes

Removed environment variables

Caution

Renamed without fallback; affects every deployment with API_KEY_ENABLED=true since 0.3.0. Keeping both old names fails startup; renaming only one leaves the other mapping empty, so callers on that path get no authorities.

Variable	Reason
API_KEY_ROLES_MAPPING	Renamed to API_KEY_KEY_ROLES_MAPPING — project-key role mapping (#256)
API_KEY_DEFAULT_ROLES_MAPPING	Renamed to API_KEY_JWT_ROLES_MAPPING — JWT-rooted per-request-key (userClaims) role mapping (#256)

New environment variables

Variable	Required	Default	Description
QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_ENABLED	No	false	Adds total_cost to row-mode test_suite_runs query results.
QUERY_DSL_EXTENSION_TEST_SUITE_RUN_COST_TIMEOUT_SEC	Conditional	2	End-to-end cost lookup deadline; minimum 1.
CSV_IMPORT_ZIP_MAX_ENTRIES	No	1000	Maximum ZIP entries accepted during test-case import.
CSV_IMPORT_ZIP_MAX_TOTAL_UNCOMPRESSED_SIZE	No	1GB	Maximum total uncompressed ZIP size.
CSV_IMPORT_ZIP_MAX_MANIFEST_SIZE	No	1MB	Maximum manifest.json size.
SPRING_AI_MCP_SERVER_ENABLED	No	false	Enables the inbound MCP server; /mcp returns 404 when disabled.
SPRING_AI_MCP_SERVER_STDIO	No	false	Enables local stdio transport; HTTP is the deployment default.
SPRING_AI_MCP_SERVER_NAME	No	ai-dial-admin-evaluation-framework-backend	Server name returned by MCP initialization.
SPRING_AI_MCP_SERVER_VERSION	No	Project version	Server version returned by MCP initialization.
SPRING_AI_MCP_SERVER_TYPE	No	SYNC	MCP implementation type; SYNC preserves request-thread caller identity.
SPRING_AI_MCP_SERVER_PROTOCOL	No	STATELESS	HTTP protocol; STREAMABLE requires one replica or sticky sessions.
SPRING_AI_MCP_SERVER_CAPABILITIES_TOOL	No	true	Advertises MCP tools.
SPRING_AI_MCP_SERVER_CAPABILITIES_RESOURCE	No	false	Advertises MCP resources; none ship in this release.
SPRING_AI_MCP_SERVER_CAPABILITIES_PROMPT	No	false	Advertises MCP prompts; none ship in this release.
SPRING_AI_MCP_SERVER_CAPABILITIES_COMPLETION	No	false	Advertises MCP completions; none ship in this release.
SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_MCP_ENDPOINT	No	/mcp	Sets the authenticated MCP HTTP path.
SPRING_AI_MCP_SERVER_STREAMABLE_HTTP_DISALLOW_DELETE	No	false	Disables session termination by DELETE in STREAMABLE mode.
SPRING_AI_MCP_SERVER_ANNOTATION_SCANNER_ENABLED	No	true	Discovers @McpTool methods.
SPRING_AI_MCP_SERVER_INSTRUCTIONS	No	Built-in workflow guidance	Instructions returned by MCP initialization.

Enabling Application mode for test suite runs

Application mode (see DIAL Configuration changes below) routes long-running evals through a DIAL
Application Route instead of the user's JWT, so a run no longer fails when the JWT expires mid-run.
It is off by default. To turn it on:

1. Enable DIAL API-Key authentication first — the internal run-execution endpoint is authenticated
via the Api-Key header, not the user's JWT, so Application mode cannot start without it:

Environment Variable	Default	Required	Description
API_KEY_ENABLED	false	Yes	Enables DIAL API-Key authentication via the Api-Key request header. Must be true before DIAL_APP_PROXY_ENABLED can be set — EF fails fast at startup otherwise.
API_KEY_CORE_URL	-	Yes	Base URL of the DIAL Core instance used to introspect API keys via GET /v1/user/info.
API_KEY_KEY_ROLES_MAPPING	- (empty)	Conditional	JSON object mapping DIAL Core project-key role names to EF authority strings, e.g. {"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}. At least one of this or API_KEY_JWT_ROLES_MAPPING must be non-empty.
API_KEY_JWT_ROLES_MAPPING	- (empty)	Conditional	JSON object mapping DIAL Core roles from the JWT-rooted userClaims response shape to EF authority strings, e.g. {"admin":["FULL_ADMIN"],"default":["READ_ONLY_ADMIN"]}.
API_KEY_USER_CLAIMS_ROLE_CLAIM	roles	No	Claim name read from the introspection response's userClaims object to obtain the caller's raw roles. Set to dial_roles to match this claim naming in Core.

2. Turn on the Application Route flag:

Environment Variable	Default	Required	Description
DIAL_APP_PROXY_ENABLED	false	No	Enables DIAL Application Routes mode for run execution — eval runs are triggered via DIAL Core's Application Route using a per-request key instead of the user's JWT. Requires API_KEY_ENABLED=true.
DIAL_APP_PROXY_DEPLOYMENT_NAME	EF	No	DIAL Core application deployment name. Must match the applications.<name> key used in the DIAL Core config snippet in DIAL Configuration changes below.
DIAL_APP_PROXY_HEARTBEAT_INTERVAL_MS	30000	No	Heartbeat interval for the internal run SSE connection.
DIAL_APP_PROXY_TRIGGER_READ_TIMEOUT_MS	43200000	No	DIAL Application Route trigger timeout in milliseconds.

3. Recommended tuning (not required to enable the mode, but needed for long evals):

Environment Variable	Default	Required	Description
TEST_SUITE_RUN_SSE_TIMEOUT_MINUTES	30	No	SSE client connection timeout, in minutes. Raise this under Application mode (e.g. 720 for up to 12 hours) so the status stream doesn't time out before a long-running eval completes.

Behavioral changes

Note

These take effect automatically on upgrade unless noted otherwise.

• API-key role mapping — with OIDC providers configured, introspection matches userClaims.iss and unions the provider's roleClaims; API_KEY_USER_CLAIMS_ROLE_CLAIM is now the fallback (#241, #248)
• dial-adas compatibility — run-cost averages now call POST /v1/queries/execute-sql; verify the deployed dial-adas version exposes this endpoint (#207)

DIAL Configuration changes

Important

DIAL Core registration is required only when DIAL_APP_PROXY_ENABLED=true; direct JWT execution remains the default.

Required setup (#236):

1. Add the applications.EF block to DIAL Core configuration, or use the application name set by DIAL_APP_PROXY_DEPLOYMENT_NAME, and grant it access to the EF file bucket. Apply the snippet with http://eval-framework-url replacement:

 {                                                                                                                                                                                                                                 
   "applications": {                                                                                                                                                                                                               
     "EF": {                                                                                                                                                                                                                       
       "endpoint": "http://unused",                                                                                                                                                                                                   
       "display_name": "EF",                                                                                                                                                                                                       
       "routes": {                                                                                                                                                                                                                 
         "runs": {                                                                                                                                                                                                                 
           "name": "runs",                                                                                                                                                                                                         
           "rewritePath": true,                                                                                                                                                                                                    
           "paths": [                                                                                                                                                                                                              
             "/api/internal/runs/[^/]+/execute$"                                                                                                                                                                                   
           ],                                                                                                                                                                                                                      
           "methods": [                                                                                                                                                                                                            
             "POST"                                                                                                                                                                                                                
           ],                                                                                                                                                                                                                      
           "upstreams": [                                                                                                                                                                                                          
             {                                                                                                                                                                                                                     
               "endpoint": "http://eval-framework-url",                                                                                                                                 
               "weight": 1,                                                                                                                                                                                                        
               "tier": 0                                                                                                                                                                                                           
             }                                                                                                                                                                                                                     
           ],                                                                                                                                                                                                                      
           "maxRetryAttempts": 1,                                                                                                                                                                                                  
           "order": 2147483647,                                                                                                                                                                                                    
           "permissions": [],                                                                                                                                                                                                      
           "attachmentPaths": {                                                                                                                                                                                                    
             "requestBody": ["$.attachments[*].url"],                                                                                                                                                                              
             "responseBody": []                                                                                                                                                                                                    
           }                                                                                                                                                                                                                       
         }                                                                                                                                                                                                                         
       }                                                                                                                                                                                                                           
     }                                                                                                                                                                                                                             
   }                                                                                                                                                                                                                               
 } 

Database migrations

Note

Flyway applies these automatically on startup; the service account needs DDL privileges on both schemas, and the analytics datasource must be reachable while meta V1.33 copies historical snapshots.

• meta — V1.32__CreateRunMetricSnapshotsTable.sql — creates the canonical snapshot table with cascading run deletion #197 (#199)
• meta — V1_33__CopyRunMetricSnapshotsFromAnalytics.java — copies historical snapshots from analytics and skips orphaned rows #197 (#199)
• meta — V1.34__ReplaceRunMetricSnapshotsRunIndex.sql — replaces the run-only index with latest-computation ordering #197 (#199)
• meta — V1.35__AddMetricScoreAggregationToTestSuites.sql — adds metric_score_aggregation with default AVG (#231)
• meta — V1.36__AddIdsToDatasetSchemaFields.sql — assigns stable UUIDs to existing dataset schema fields without changing dataset versions #246 (#249)
• analytics — V1.20__CreateTestCaseMetricScoresAggregatedTable.sql — stores per-test-case metric aggregates by computation (#231)
• analytics — V1.21__AddRunCaseContextToTestCaseEvalScores.sql — rebuilds test-case scores as one row per case and computation (#231)

Caution

Analytics V1.21 drops and recreates test_case_eval_scores, deleting existing derived scores until recomputed. Meta V1.34 takes an access-exclusive lock while replacing its index; consider a maintenance window for large snapshot tables.

API / contract changes

Note

Coordinate consumers for the breaking items below; full REST reference is available in Swagger UI (/swagger-ui.html).

• Breaking — template variables — both template-variable GET endpoints now return request-indexed maps instead of flat arrays, and try-out variables is now a request-indexed map #217 (#235)
• Breaking — score read surface — eval-summary list/detail responses remove per-row score and passed; deduplicated scores now come from the test_case_eval_scores Query DSL entity (#231)
• New endpoints — authenticated POST /mcp #209 (#208), batch costs at POST /api/v1/costs/test-suite-runs (#205), and aggregated metrics at GET /api/v1/test-suites/{testSuiteId}/metric-definitions/aggregated #224 (#226)
• Run snapshots — GET|POST /api/v1/run-metric-snapshots is canonical; /api/v1/analytics/run-metric-snapshots remains a deprecated alias #197 (#199)
• Structured Query DSL — adds test_suite_runs, test_case_metric_scores, and test_case_eval_scores; test_suite_runs gains number_of_runs plus result-only score and optional cost keys #197 (#201, #206, #221, #222; #231)
• Test-suite scoring — request/response snapshots gain metricScoreAggregation (AVG, MIN, MAX), and custom_function is rejected for testCaseOverallScore (#231)
• ZIP archives — exports gain version-1 manifest.json; broken referenced files now return HTTP 502 JSON without a partial archive #225 (#227, #228, #232)
• Pricing payloads — full model-deployment responses may expose pricing.prompt and pricing.completion as either scalars or JSON objects; strict clients must accept both #243 (#244, #247)
• Dataset schema fields — responses now include a server-assigned id; echo it unchanged in PUT /api/v1/datasets/{id} to preserve identity across renames, and omit it for new fields #246 (#249)
• Bound public datasets — destructive schema updates and CSV/ZIP imports now return HTTP 409 with DATASET_FIELD_RENAME_FORBIDDEN or DATASET_FIELD_REMOVAL_FORBIDDEN #246 (#249, #254)