Skip to content

Continuous Test Trends

Jump to bottom
github-actions[bot] edited this page Sep 17, 2026 · 68 revisions

title: Continuous Test Trends description: CI test and evaluation measurements with immutable source-run links.

Scope

Updated automatically from completed trusted main-branch CI runs. Raw per-run aggregates are retained in trend-history/. Tables show the latest 50 attempts; each chart shows up to 12 available measurements. Missing data is never plotted as zero. Chart labels use V (validation) or R (release), workflow run number, and attempt. The table links each label to the full run ID. Failed and cancelled runs stay visible. Charts show measurements, not workflow status. Live tests use existing staging with synthetic fixtures; the test-code SHA is not the deployed-agent source SHA.

Recent Runs

Run / attempt UTC Workflow outcome Test SHA Agent version Tests pass / fail / skip Captures / policy failures Load success / errors p50 / p95 (s)
V47.1 / 35242424312 2026-09-17T15:46:36Z Continuous Validation: success cba59f5 3 314 / 0 / 6 8 / 0 5 / 0 30.687 / 36.303
V46.1 / 35241529916 2026-09-17T15:38:14Z Continuous Validation: success 813ba2a 3 314 / 0 / 6 8 / 0 5 / 0 24.256 / 34.67
V45.1 / 35232060244 2026-09-17T14:12:45Z Continuous Validation: success 2ff5b7a 3 314 / 0 / 6 8 / 0 5 / 0 24.122 / 26.906
V43.1 / 35231607573 2026-09-17T14:08:33Z Continuous Validation: success 85d36d8 3 314 / 0 / 6 8 / 0 5 / 0 23.722 / 27.216
V42.1 / 35230054613 2026-09-17T13:54:15Z Continuous Validation: success 0f2cbf6 3 305 / 0 / 6 8 / 0 5 / 0 26.715 / 29.729
V41.1 / 35104721737 2026-09-16T13:52:24Z Continuous Validation: success 3ebf297 3 305 / 0 / 6 8 / 0 5 / 0 27.406 / 29.558
V40.1 / 35102549412 2026-09-16T13:32:15Z Continuous Validation: success c297393 3 304 / 0 / 6 8 / 0 5 / 0 27.739 / 34.484
V39.1 / 35050590678 2026-09-16T03:07:18Z Continuous Validation: success 70a8665 3 303 / 0 / 6 8 / 0 5 / 0 21.622 / 23.157
V38.2 / 35046124027 2026-09-16T02:48:44Z Continuous Validation: success b8b7331 3 N/A / N/A / N/A 8 / 0 5 / 0 20.677 / 24.431
R53.2 / 35046127030 2026-09-16T02:25:31Z Deploy and Evaluate (Staging -> Production): success b8b7331 3 N/A / N/A / N/A 8 / 0 N/A / N/A N/A / N/A
R53.1 / 35046127030 2026-09-16T01:57:53Z Deploy and Evaluate (Staging -> Production): failure b8b7331 3 283 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V38.1 / 35046124027 2026-09-16T01:57:50Z Continuous Validation: failure b8b7331 3 303 / 0 / 6 8 / 0 5 / 0 23.159 / 26.105
R52.1 / 35044768604 2026-09-16T01:36:43Z Deploy and Evaluate (Staging -> Production): failure 73a9ed0 2 282 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V37.1 / 35044764212 2026-09-16T01:36:38Z Continuous Validation: failure 73a9ed0 2 302 / 0 / 6 8 / 0 5 / 0 24.717 / 26.269
V36.1 / 35038446737 2026-09-16T00:04:49Z Continuous Validation: failure 1e02667 14 296 / 0 / 6 8 / 0 0 / 5 N/A / N/A
V35.1 / 35038325051 2026-09-16T00:03:08Z Continuous Validation: failure c2362c6 14 296 / 0 / 6 8 / 0 0 / 5 N/A / N/A
R51.1 / 35035668932 2026-09-15T23:26:30Z Deploy and Evaluate (Staging -> Production): failure 5093ade N/A 269 / 0 / 6 N/A / N/A N/A / N/A N/A / N/A
V34.1 / 35035030915 2026-09-15T23:18:02Z Continuous Validation: cancelled 5093ade N/A 104 / 0 / 6 N/A / N/A N/A / N/A N/A / N/A
V33.1 / 35034717498 2026-09-15T23:13:55Z Continuous Validation: cancelled b2201da 13 287 / 0 / 6 1 / N/A N/A / N/A N/A / N/A
R50.1 / 35033599689 2026-09-15T22:59:28Z Deploy and Evaluate (Staging -> Production): cancelled 4a5499b N/A 173 / 0 / 6 N/A / N/A N/A / N/A N/A / N/A
V32.1 / 35033031776 2026-09-15T22:52:00Z Continuous Validation: cancelled 4a5499b N/A 287 / 0 / 6 N/A / N/A N/A / N/A N/A / N/A
V31.1 / 35029008923 2026-09-15T22:03:57Z Continuous Validation: success 7f08820 13 280 / 0 / 6 8 / 0 5 / 0 23.964 / 25.243
V30.1 / 35007892286 2026-09-15T18:30:19Z Continuous Validation: success 5e50c0a 13 263 / 0 / 6 8 / 0 5 / 0 21.833 / 27.488
R49.1 / 35002986722 2026-09-15T17:42:56Z Deploy and Evaluate (Staging -> Production): failure ab8f8c1 13 232 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V29.1 / 35002935926 2026-09-15T17:42:27Z Continuous Validation: success ab8f8c1 13 252 / 0 / 6 8 / 0 5 / 0 21.529 / 28.735
V28.1 / 34868218304 2026-09-14T16:22:36Z Continuous Validation: success 0104f9f 13 252 / 0 / 6 8 / 0 5 / 0 19.033 / 22.367
V27.1 / 34785480298 2026-09-13T22:00:35Z Continuous Validation: success 6d68b53 13 252 / 0 / 6 8 / 0 5 / 0 17.575 / 20.527
V26.1 / 34434805151 2026-09-10T03:49:34Z Continuous Validation: success 2e10a60 13 199 / 0 / 6 8 / 0 5 / 0 19.254 / 20.377
V25.1 / 34433959877 2026-09-10T03:36:14Z Continuous Validation: success 399fcfc 13 199 / 0 / 6 8 / 0 5 / 0 19.069 / 21.044
V24.1 / 34432576820 2026-09-10T03:14:40Z Continuous Validation: success 4f79dc4 13 199 / 0 / 6 8 / 0 5 / 0 21.078 / 22.169
V23.1 / 34432311647 2026-09-10T03:10:31Z Continuous Validation: success 2771843 13 199 / 0 / 6 8 / 0 5 / 0 19.049 / 21.642
R48.1 / 34427432731 2026-09-10T01:56:02Z Deploy and Evaluate (Staging -> Production): success 15098b7 13 183 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V22.1 / 34427429700 2026-09-10T01:55:59Z Continuous Validation: success 15098b7 13 199 / 0 / 6 8 / 0 5 / 0 19.576 / 20.892
V21.1 / 34426583987 2026-09-10T01:43:07Z Continuous Validation: success 5b39be1 13 198 / 0 / 6 8 / 0 5 / 0 19.054 / 22.151
R47.1 / 34424741660 2026-09-10T01:15:30Z Deploy and Evaluate (Staging -> Production): success 20a5a35 13 182 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V20.1 / 34424723263 2026-09-10T01:15:15Z Continuous Validation: failure 20a5a35 13 198 / 0 / 6 8 / 0 2 / 3 23.444 / 27.969
R46.1 / 34422215632 2026-09-10T00:39:27Z Deploy and Evaluate (Staging -> Production): failure a7149db 13 182 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V19.1 / 34422212544 2026-09-10T00:39:24Z Continuous Validation: success a7149db 13 198 / 0 / 6 8 / 0 5 / 0 20.665 / 23.489
R45.1 / 34420904417 2026-09-10T00:20:55Z Deploy and Evaluate (Staging -> Production): failure 6c5b1e9 13 182 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V18.1 / 34420899611 2026-09-10T00:20:51Z Continuous Validation: failure 6c5b1e9 13 198 / 0 / 6 8 / 0 5 / 0 42.638 / 47.645
R44.1 / 34418932490 2026-09-09T23:54:07Z Deploy and Evaluate (Staging -> Production): failure 873f637 N/A 182 / 0 / 6 N/A / N/A N/A / N/A N/A / N/A
V17.1 / 34418922726 2026-09-09T23:53:59Z Continuous Validation: failure 873f637 12 198 / 0 / 6 8 / 0 5 / 0 41.381 / 44.556
R43.1 / 34353473428 2026-09-09T12:51:24Z Deploy and Evaluate (Staging -> Production): cancelled 6567d98 11 180 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V16.1 / 34353469516 2026-09-09T12:51:21Z Continuous Validation: success 6567d98 11 196 / 0 / 6 8 / 0 5 / 0 21.322 / 22.497
R42.1 / 34353239992 2026-09-09T12:49:03Z Deploy and Evaluate (Staging -> Production): cancelled ba838d2 N/A N/A / N/A / N/A N/A / N/A N/A / N/A N/A / N/A
V15.1 / 34353234669 2026-09-09T12:49:00Z Continuous Validation: success ba838d2 11 190 / 0 / 6 8 / 0 5 / 0 19.235 / 26.643
R41.1 / 34350963625 2026-09-09T12:25:49Z Deploy and Evaluate (Staging -> Production): cancelled d8589a6 11 174 / 0 / 6 8 / 0 N/A / N/A N/A / N/A
V14.1 / 34350960377 2026-09-09T12:25:46Z Continuous Validation: success d8589a6 11 190 / 0 / 6 8 / 0 5 / 0 18.658 / 26.284
V13.1 / 34349179327 2026-09-09T12:06:57Z Continuous Validation: success 9d9ec8b 10 188 / 0 / 6 8 / 0 5 / 0 20.503 / 21.396
V12.1 / 34348779996 2026-09-09T12:02:46Z Continuous Validation: failure b3844fb 10 188 / 0 / 6 8 / 0 3 / 2 20.778 / 26.621

Test Suite Growth

Counts per run, not cumulative executions. Offline inventory includes skipped tests. Adding tests increases inventory; rerunning the same suite does not. Decreases remain visible. Total offline tests is the sum of the JUnit types, not an additional test type. Shell regression steps are tracked as job outcomes and are not included in JUnit counts. Live evaluation cases, judge checks and load requests are measured separately: these overlap or repeat scenarios and must not be added to the offline inventory. They reflect available results, not undiscovered or unexecuted cases. Historical records without a type breakdown show N/A until their retained artifacts are replayed.

Run / attempt Total offline tests Agent graph Deterministic evaluation Reporting and load contracts Live evaluation cases Judge checks Load requests
V47.1 / 35242424312 320 113 69 138 8 21 5
V46.1 / 35241529916 320 113 69 138 8 21 5
V45.1 / 35232060244 320 113 69 138 8 21 5
V43.1 / 35231607573 320 113 69 138 8 21 5
V42.1 / 35230054613 311 113 69 129 8 21 5
V41.1 / 35104721737 311 113 69 129 8 21 5
V40.1 / 35102549412 310 113 69 128 8 21 5
V39.1 / 35050590678 309 113 69 127 8 21 5
V38.2 / 35046124027 N/A N/A N/A N/A 8 21 5
R53.2 / 35046127030 N/A N/A N/A N/A 8 21 N/A
R53.1 / 35046127030 289 113 69 107 8 N/A N/A
V38.1 / 35046124027 309 113 69 127 8 N/A 5
R52.1 / 35044768604 288 113 69 106 8 N/A N/A
V37.1 / 35044764212 308 113 69 126 8 N/A 5
V36.1 / 35038446737 302 110 69 123 8 N/A 5
V35.1 / 35038325051 302 110 69 123 8 N/A 5
R51.1 / 35035668932 275 110 69 96 N/A N/A N/A
V34.1 / 35035030915 110 110 N/A N/A N/A N/A N/A
V33.1 / 35034717498 293 110 69 114 1 N/A N/A
R50.1 / 35033599689 179 110 69 N/A N/A N/A N/A
V32.1 / 35033031776 293 110 69 114 N/A N/A N/A
V31.1 / 35029008923 286 110 69 107 8 21 5
V30.1 / 35007892286 269 110 69 90 8 21 5
R49.1 / 35002986722 238 107 62 69 8 21 N/A
V29.1 / 35002935926 258 107 62 89 8 21 5
V28.1 / 34868218304 258 107 62 89 8 21 5
V27.1 / 34785480298 258 107 62 89 8 21 5
V26.1 / 34434805151 205 107 54 44 8 21 5
V25.1 / 34433959877 205 107 54 44 8 21 5
V24.1 / 34432576820 205 107 54 44 8 21 5
V23.1 / 34432311647 205 107 54 44 8 21 5
R48.1 / 34427432731 189 107 54 28 8 21 N/A
V22.1 / 34427429700 205 107 54 44 8 21 5
V21.1 / 34426583987 204 107 54 43 8 21 5
R47.1 / 34424741660 188 107 54 27 8 21 N/A
V20.1 / 34424723263 204 107 54 43 8 21 5
R46.1 / 34422215632 188 107 54 27 8 21 N/A
V19.1 / 34422212544 204 107 54 43 8 21 5
R45.1 / 34420904417 188 107 54 27 8 N/A N/A
V18.1 / 34420899611 204 107 54 43 8 N/A 5
R44.1 / 34418932490 188 108 54 26 N/A N/A N/A
V17.1 / 34418922726 204 108 54 42 8 N/A 5
R43.1 / 34353473428 186 106 54 26 8 21 N/A
V16.1 / 34353469516 202 106 54 42 8 21 5
R42.1 / 34353239992 N/A N/A N/A N/A N/A N/A N/A
V15.1 / 34353234669 196 106 54 36 8 21 5
R41.1 / 34350963625 180 106 54 20 8 21 N/A
V14.1 / 34350960377 196 106 54 36 8 21 5
V13.1 / 34349179327 194 106 54 34 8 21 5
V12.1 / 34348779996 194 106 54 34 8 21 5 
xychart-beta
  title "Total offline tests"
  x-axis ["V37.1", "R52.1", "V38.1", "R53.1", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 320
  bar [308, 288, 309, 289, 309, 310, 311, 311, 320, 320, 320, 320]
Loading 
xychart-beta
  title "Agent graph"
  x-axis ["V37.1", "R52.1", "V38.1", "R53.1", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 113
  bar [113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113]
Loading 
xychart-beta
  title "Deterministic evaluation"
  x-axis ["V37.1", "R52.1", "V38.1", "R53.1", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 69
  bar [69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69]
Loading 
xychart-beta
  title "Reporting and load contracts"
  x-axis ["V37.1", "R52.1", "V38.1", "R53.1", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 138
  bar [126, 106, 127, 107, 127, 128, 129, 129, 138, 138, 138, 138]
Loading 
xychart-beta
  title "Live evaluation cases"
  x-axis ["V38.1", "R53.1", "R53.2", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 8
  bar [8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]
Loading 
xychart-beta
  title "Judge checks"
  x-axis ["V30.1", "V31.1", "R53.2", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 21
  bar [21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21]
Loading 
xychart-beta
  title "Load requests"
  x-axis ["V36.1", "V37.1", "V38.1", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "count" 0 --> 5
  bar [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
Loading

Test Failures

xychart-beta
  title "JUnit failures"
  x-axis ["V37.1", "R52.1", "V38.1", "R53.1", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "failed tests" 0 --> 1
  bar [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Loading

Evaluation Trends

Judge rates are grouped by dataset and evaluator code hashes, judge deployment and environment. Refusals are deterministic, not judge passes.

Dataset d1f226514641 / evaluator 62550fa9fe7f

Full dataset SHA-256: d1f2265146418cdf64ac6eb37415949ec6338f75ad7acf7ca4a5dd1247d0920a. Evaluator SHA-256: 62550fa9fe7f933efb341f7e5b3ed3d554d22f2347369e2af3b413e82fd316cc. Judge: gpt-4o-mini. Environment: staging.

xychart-beta
  title "coherence"
  x-axis ["V12.1", "V13.1", "V14.1", "R41.1", "V15.1", "V16.1", "R43.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "groundedness"
  x-axis ["V12.1", "V13.1", "V14.1", "R41.1", "V15.1", "V16.1", "R43.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "task_adherence"
  x-axis ["V12.1", "V13.1", "V14.1", "R41.1", "V15.1", "V16.1", "R43.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading

Dataset d1f226514641 / evaluator 3862f198cdb0

Full dataset SHA-256: d1f2265146418cdf64ac6eb37415949ec6338f75ad7acf7ca4a5dd1247d0920a. Evaluator SHA-256: 3862f198cdb0ce80afd5ed723418019ae100a36330328cc57dbe7910cd645aef. Judge: gpt-4o-mini. Environment: staging.

xychart-beta
  title "coherence"
  x-axis ["R47.1", "V21.1", "V22.1", "R48.1", "V23.1", "V24.1", "V25.1", "V26.1", "V27.1", "V28.1", "V29.1", "R49.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "groundedness"
  x-axis ["R47.1", "V21.1", "V22.1", "R48.1", "V23.1", "V24.1", "V25.1", "V26.1", "V27.1", "V28.1", "V29.1", "R49.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "task_adherence"
  x-axis ["R47.1", "V21.1", "V22.1", "R48.1", "V23.1", "V24.1", "V25.1", "V26.1", "V27.1", "V28.1", "V29.1", "R49.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 85.714]
Loading

Dataset d1f226514641 / evaluator a8b9e63a9fb3

Full dataset SHA-256: d1f2265146418cdf64ac6eb37415949ec6338f75ad7acf7ca4a5dd1247d0920a. Evaluator SHA-256: a8b9e63a9fb36a3816df0ba6a136aaf4536d4c8ee612f1fa0b072050d6375519. Judge: gpt-4o-mini. Environment: staging.

xychart-beta
  title "coherence"
  x-axis ["V30.1", "V31.1", "R53.2", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "groundedness"
  x-axis ["V30.1", "V31.1", "R53.2", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading 
xychart-beta
  title "task_adherence"
  x-axis ["V30.1", "V31.1", "R53.2", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "pass percent" 0 --> 100
  bar [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]
Loading

Load Latency

Completed-text-v2 contract, five concurrent requests against staging. p50/p95 use successful requests only; see error counts above. Small-sample percentiles are not capacity or SLA evidence. Timings are withheld if the routed version changes or cannot be verified.

xychart-beta
  title "latency_p50_seconds"
  x-axis ["V31.1", "V37.1", "V38.1", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "seconds" 0 --> 31
  bar [23.964, 24.717, 23.159, 20.677, 21.622, 27.739, 27.406, 26.715, 23.722, 24.122, 24.256, 30.687]
Loading 
xychart-beta
  title "latency_p95_seconds"
  x-axis ["V31.1", "V37.1", "V38.1", "V38.2", "V39.1", "V40.1", "V41.1", "V42.1", "V43.1", "V45.1", "V46.1", "V47.1"]
  y-axis "seconds" 0 --> 37
  bar [25.243, 26.269, 26.105, 24.431, 23.157, 34.484, 29.558, 29.729, 27.216, 26.906, 34.67, 36.303]
Loading

Reporting Gaps

No malformed-evidence or route-verification gaps detected. N/A cells can still indicate skipped tests or missing artifacts.

title: Wiki Navigation description: Current implementation and historical investigation pages.

Clone this wiki locally