Skip to content

Continuous Test Trends

Jump to bottom
github-actions[bot] edited this page Oct 1, 2026 · 124 revisions

title: Continuous Test Trends description: CI test and evaluation measurements with immutable source-run links.

Scope

Updated automatically from completed trusted main-branch CI runs. Raw per-run aggregates are retained in trend-history/. Tables show the latest 50 attempts; each chart shows up to 12 available measurements. Missing data is never plotted as zero. Chart labels use V (validation) or R (release), workflow run number, and attempt. The table links each label to the full run ID. Failed and cancelled runs stay visible. Charts show measurements, not workflow status. This repository has no hosted-agent capture pipeline; LLM-judge evaluation is author-only and inert until a real endpoint exists and Gates G2/G3/G6 clear (see eval/run_judge_evaluation.py).

Recent Runs

Run / attempt UTC Workflow outcome Test SHA Agent version Tests pass / fail / skip Judge cases judged
V92.1 / 36878066062 2026-10-01T14:39:36Z Continuous Validation: success b4ceba0 N/A 147 / 0 / 36 N/A
V87.1 / 36506613884 2026-09-29T01:10:15Z Continuous Validation: success 0b0fd5c N/A 147 / 0 / 36 N/A
V86.1 / 36504744215 2026-09-29T00:46:48Z Continuous Validation: success 4cb08df N/A 147 / 0 / 36 N/A
V85.1 / 36499388241 2026-09-28T23:42:18Z Continuous Validation: success 46a4e72 N/A 147 / 0 / 36 N/A
V84.1 / 36498326911 2026-09-28T23:30:04Z Continuous Validation: success ce6f146 N/A 147 / 0 / 36 N/A
V83.1 / 36497276083 2026-09-28T23:18:17Z Continuous Validation: success f1a0d0b N/A 147 / 0 / 36 N/A
V82.1 / 36495924471 2026-09-28T23:03:30Z Continuous Validation: success 586a08c N/A 147 / 0 / 36 N/A
V81.1 / 36493132125 2026-09-28T22:33:53Z Continuous Validation: success 3182431 N/A 147 / 0 / 36 N/A
V80.1 / 36483834076 2026-09-28T21:06:20Z Continuous Validation: success 5d199e0 N/A 147 / 0 / 36 N/A
V78.1 / 36473896914 2026-09-28T19:40:47Z Continuous Validation: success 16a5884 N/A 147 / 0 / 36 N/A
V77.1 / 36430392553 2026-09-28T13:41:25Z Continuous Validation: success 0ea824d N/A 147 / 0 / 36 N/A
V76.1 / 36423818823 2026-09-28T12:44:52Z Continuous Validation: success 41e2d91 N/A 147 / 0 / 36 N/A
V75.1 / 36422715426 2026-09-28T12:34:39Z Continuous Validation: success 2107eb6 N/A 147 / 0 / 36 N/A
V74.1 / 36421180646 2026-09-28T12:20:15Z Continuous Validation: success ab51490 N/A 147 / 0 / 36 N/A
V15.1 / 35857156466 2026-09-23T11:53:23Z Reviewer App Build: success 6110695 N/A N/A / N/A / N/A N/A
V73.1 / 35857156355 2026-09-23T11:53:23Z Continuous Validation: success 6110695 N/A 147 / 0 / 36 N/A
V14.1 / 35684514017 2026-09-22T03:48:35Z Reviewer App Build: success 1417eeb N/A N/A / N/A / N/A N/A
V72.1 / 35684513943 2026-09-22T03:48:35Z Continuous Validation: success 1417eeb N/A 147 / 0 / 36 N/A
V13.1 / 35681014333 2026-09-22T02:51:39Z Reviewer App Build: success 23334f0 N/A N/A / N/A / N/A N/A
V71.1 / 35681014235 2026-09-22T02:51:39Z Continuous Validation: success 23334f0 N/A 147 / 0 / 36 N/A
V70.1 / 35530833684 2026-09-20T18:59:57Z Continuous Validation: success ae55373 N/A 143 / 0 / 34 N/A
V69.1 / 35518568599 2026-09-20T15:07:10Z Continuous Validation: success 653e688 N/A 143 / 0 / 34 N/A
V12.1 / 35481906317 2026-09-20T01:38:17Z Reviewer App Build: success 1cce07d N/A N/A / N/A / N/A N/A
V68.1 / 35481906290 2026-09-20T01:38:17Z Continuous Validation: success 1cce07d N/A 143 / 0 / 34 N/A
R15.1 / 35360464502 2026-09-18T15:07:18Z Hosted Agent CI/CD: success ae01751 N/A 143 / 0 / 34 10
V67.1 / 35359893121 2026-09-18T15:01:50Z Continuous Validation: success 443c310 N/A 143 / 0 / 34 N/A
V66.1 / 35348374521 2026-09-18T13:07:21Z Continuous Validation: success c2f710b N/A 143 / 0 / 34 N/A
R14.1 / 35346690448 2026-09-18T12:49:04Z Hosted Agent CI/CD: success bed99f9 N/A 143 / 0 / 34 10
V65.1 / 35346682834 2026-09-18T12:48:58Z Continuous Validation: success bed99f9 N/A 143 / 0 / 34 N/A
R13.1 / 35343631113 2026-09-18T12:14:38Z Hosted Agent CI/CD: success c7751c5 N/A 143 / 0 / 34 10
V11.1 / 35343626218 2026-09-18T12:14:35Z Reviewer App Build: success c7751c5 N/A N/A / N/A / N/A N/A
V64.1 / 35343626153 2026-09-18T12:14:35Z Continuous Validation: success c7751c5 N/A 143 / 0 / 34 N/A
V63.1 / 35339385233 2026-09-18T11:23:23Z Continuous Validation: success a5ba049 N/A 143 / 0 / 34 N/A
V10.1 / 35339385166 2026-09-18T11:23:23Z Reviewer App Build: success a5ba049 N/A N/A / N/A / N/A N/A
V62.1 / 35301281774 2026-09-18T02:56:10Z Continuous Validation: success 7a62328 N/A 143 / 0 / 34 N/A
R12.1 / 35299088833 2026-09-18T02:22:51Z Hosted Agent CI/CD: success 95696e7 N/A 143 / 0 / 34 10
V9.1 / 35299083272 2026-09-18T02:22:47Z Reviewer App Build: success 95696e7 N/A N/A / N/A / N/A N/A
V61.1 / 35299083243 2026-09-18T02:22:47Z Continuous Validation: success 95696e7 N/A 143 / 0 / 34 N/A
V60.1 / 35296470314 2026-09-18T01:43:22Z Continuous Validation: success b876092 N/A 143 / 0 / 34 N/A
R25.1 / 35292843329 2026-09-18T00:50:16Z Deploy and Evaluate (Staging -> Production): success ea88977 N/A 143 / 0 / 34 10
V59.1 / 35289855063 2026-09-18T00:07:36Z Continuous Validation: success ea88977 N/A 143 / 0 / 34 N/A
R24.1 / 35274334216 2026-09-17T21:02:01Z Deploy and Evaluate (Staging -> Production): success 47c0477 N/A 143 / 0 / 34 0
R23.1 / 35272075564 2026-09-17T20:38:59Z Deploy and Evaluate (Staging -> Production): failure 47c0477 N/A 143 / 0 / 34 N/A
V58.1 / 35271468167 2026-09-17T20:32:58Z Continuous Validation: success 47c0477 N/A 143 / 0 / 34 N/A
R22.1 / 35249247909 2026-09-17T16:52:21Z Deploy and Evaluate (Staging -> Production): failure 3e6520b N/A 143 / 0 / 34 N/A
R21.1 / 35248376664 2026-09-17T16:43:46Z Deploy and Evaluate (Staging -> Production): failure 3e6520b N/A 143 / 0 / 34 N/A
R20.1 / 35238867109 2026-09-17T15:13:59Z Deploy and Evaluate (Staging -> Production): success 3e6520b N/A 143 / 0 / 34 10
V57.1 / 35238063387 2026-09-17T15:06:36Z Continuous Validation: success 3e6520b N/A 143 / 0 / 34 N/A
R19.1 / 35235911956 2026-09-17T14:47:11Z Deploy and Evaluate (Staging -> Production): success bdf4931 N/A 143 / 0 / 34 0
V56.1 / 35235905795 2026-09-17T14:47:08Z Continuous Validation: success bdf4931 N/A 143 / 0 / 34 N/A

Test Suite Growth

Counts per run, not cumulative executions. Offline inventory includes skipped tests. Adding tests increases inventory; rerunning the same suite does not. Decreases remain visible. Total offline tests is the sum of the JUnit types, not an additional test type. Judge cases and judge checks are measured separately: these overlap or repeat scenarios and must not be added to the offline inventory. They reflect available results, not undiscovered or unexecuted cases. Historical records without a type breakdown show N/A until their retained artifacts are replayed.

Run / attempt Total offline tests Agent graph Deterministic evaluation Reporting and contract tests Judge cases judged Judge checks
V92.1 / 36878066062 183 125 26 32 N/A N/A
V87.1 / 36506613884 183 125 26 32 N/A N/A
V86.1 / 36504744215 183 125 26 32 N/A N/A
V85.1 / 36499388241 183 125 26 32 N/A N/A
V84.1 / 36498326911 183 125 26 32 N/A N/A
V83.1 / 36497276083 183 125 26 32 N/A N/A
V82.1 / 36495924471 183 125 26 32 N/A N/A
V81.1 / 36493132125 183 125 26 32 N/A N/A
V80.1 / 36483834076 183 125 26 32 N/A N/A
V78.1 / 36473896914 183 125 26 32 N/A N/A
V77.1 / 36430392553 183 125 26 32 N/A N/A
V76.1 / 36423818823 183 125 26 32 N/A N/A
V75.1 / 36422715426 183 125 26 32 N/A N/A
V74.1 / 36421180646 183 125 26 32 N/A N/A
V15.1 / 35857156466 N/A N/A N/A N/A N/A N/A
V73.1 / 35857156355 183 125 26 32 N/A N/A
V14.1 / 35684514017 N/A N/A N/A N/A N/A N/A
V72.1 / 35684513943 183 125 26 32 N/A N/A
V13.1 / 35681014333 N/A N/A N/A N/A N/A N/A
V71.1 / 35681014235 183 125 26 32 N/A N/A
V70.1 / 35530833684 177 119 26 32 N/A N/A
V69.1 / 35518568599 177 119 26 32 N/A N/A
V12.1 / 35481906317 N/A N/A N/A N/A N/A N/A
V68.1 / 35481906290 177 119 26 32 N/A N/A
R15.1 / 35360464502 177 119 26 32 10 30
V67.1 / 35359893121 177 119 26 32 N/A N/A
V66.1 / 35348374521 177 119 26 32 N/A N/A
R14.1 / 35346690448 177 119 26 32 10 30
V65.1 / 35346682834 177 119 26 32 N/A N/A
R13.1 / 35343631113 177 119 26 32 10 30
V11.1 / 35343626218 N/A N/A N/A N/A N/A N/A
V64.1 / 35343626153 177 119 26 32 N/A N/A
V63.1 / 35339385233 177 119 26 32 N/A N/A
V10.1 / 35339385166 N/A N/A N/A N/A N/A N/A
V62.1 / 35301281774 177 119 26 32 N/A N/A
R12.1 / 35299088833 177 119 26 32 10 30
V9.1 / 35299083272 N/A N/A N/A N/A N/A N/A
V61.1 / 35299083243 177 119 26 32 N/A N/A
V60.1 / 35296470314 177 119 26 32 N/A N/A
R25.1 / 35292843329 177 119 26 32 10 30
V59.1 / 35289855063 177 119 26 32 N/A N/A
R24.1 / 35274334216 177 119 26 32 0 N/A
R23.1 / 35272075564 177 119 26 32 N/A N/A
V58.1 / 35271468167 177 119 26 32 N/A N/A
R22.1 / 35249247909 177 119 26 32 N/A N/A
R21.1 / 35248376664 177 119 26 32 N/A N/A
R20.1 / 35238867109 177 119 26 32 10 N/A
V57.1 / 35238063387 177 119 26 32 N/A N/A
R19.1 / 35235911956 177 119 26 32 0 N/A
V56.1 / 35235905795 177 119 26 32 N/A N/A 
xychart-beta
  title "Total offline tests"
  x-axis ["V76.1", "V77.1", "V78.1", "V80.1", "V81.1", "V82.1", "V83.1", "V84.1", "V85.1", "V86.1", "V87.1", "V92.1"]
  y-axis "count" 0 --> 183
  bar [183, 183, 183, 183, 183, 183, 183, 183, 183, 183, 183, 183]
Loading 
xychart-beta
  title "Agent graph"
  x-axis ["V76.1", "V77.1", "V78.1", "V80.1", "V81.1", "V82.1", "V83.1", "V84.1", "V85.1", "V86.1", "V87.1", "V92.1"]
  y-axis "count" 0 --> 125
  bar [125, 125, 125, 125, 125, 125, 125, 125, 125, 125, 125, 125]
Loading 
xychart-beta
  title "Deterministic evaluation"
  x-axis ["V76.1", "V77.1", "V78.1", "V80.1", "V81.1", "V82.1", "V83.1", "V84.1", "V85.1", "V86.1", "V87.1", "V92.1"]
  y-axis "count" 0 --> 26
  bar [26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26]
Loading 
xychart-beta
  title "Reporting and contract tests"
  x-axis ["V76.1", "V77.1", "V78.1", "V80.1", "V81.1", "V82.1", "V83.1", "V84.1", "V85.1", "V86.1", "V87.1", "V92.1"]
  y-axis "count" 0 --> 32
  bar [32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32]
Loading 
xychart-beta
  title "Judge cases judged"
  x-axis ["R19.1", "R20.1", "R24.1", "R25.1", "R12.1", "R13.1", "R14.1", "R15.1"]
  y-axis "count" 0 --> 10
  bar [0, 10, 0, 10, 10, 10, 10, 10]
Loading 
xychart-beta
  title "Judge checks"
  x-axis ["R25.1", "R12.1", "R13.1", "R14.1", "R15.1"]
  y-axis "count" 0 --> 30
  bar [30, 30, 30, 30, 30]
Loading

Test Failures

xychart-beta
  title "JUnit failures"
  x-axis ["V76.1", "V77.1", "V78.1", "V80.1", "V81.1", "V82.1", "V83.1", "V84.1", "V85.1", "V86.1", "V87.1", "V92.1"]
  y-axis "failed tests" 0 --> 1
  bar [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Loading

Deterministic Gate

Real eval/evaluation_gate.py output for each run: dataset size, passed/failed record counts, and the bilingual-parity check. N/A means the deterministic-gate evidence was missing or malformed for that run, not zero passing records.

Run / attempt Dataset size Passed records Failed records Bilingual parity
V92.1 / 36878066062 13 13 0 True
V87.1 / 36506613884 13 13 0 True
V86.1 / 36504744215 13 13 0 True
V85.1 / 36499388241 13 13 0 True
V84.1 / 36498326911 13 13 0 True
V83.1 / 36497276083 13 13 0 True
V82.1 / 36495924471 13 13 0 True
V81.1 / 36493132125 13 13 0 True
V80.1 / 36483834076 13 13 0 True
V78.1 / 36473896914 13 13 0 True
V77.1 / 36430392553 13 13 0 True
V76.1 / 36423818823 13 13 0 True
V75.1 / 36422715426 13 13 0 True
V74.1 / 36421180646 13 13 0 True
V15.1 / 35857156466 N/A N/A N/A N/A
V73.1 / 35857156355 13 13 0 True
V14.1 / 35684514017 N/A N/A N/A N/A
V72.1 / 35684513943 13 13 0 True
V13.1 / 35681014333 N/A N/A N/A N/A
V71.1 / 35681014235 13 13 0 True
V70.1 / 35530833684 13 13 0 True
V69.1 / 35518568599 13 13 0 True
V12.1 / 35481906317 N/A N/A N/A N/A
V68.1 / 35481906290 13 13 0 True
R15.1 / 35360464502 13 13 0 True
V67.1 / 35359893121 13 13 0 True
V66.1 / 35348374521 13 13 0 True
R14.1 / 35346690448 13 13 0 True
V65.1 / 35346682834 13 13 0 True
R13.1 / 35343631113 13 13 0 True
V11.1 / 35343626218 N/A N/A N/A N/A
V64.1 / 35343626153 13 13 0 True
V63.1 / 35339385233 13 13 0 True
V10.1 / 35339385166 N/A N/A N/A N/A
V62.1 / 35301281774 13 13 0 True
R12.1 / 35299088833 13 13 0 True
V9.1 / 35299083272 N/A N/A N/A N/A
V61.1 / 35299083243 13 13 0 True
V60.1 / 35296470314 13 13 0 True
R25.1 / 35292843329 13 13 0 True
V59.1 / 35289855063 13 13 0 True
R24.1 / 35274334216 13 13 0 True
R23.1 / 35272075564 N/A N/A N/A N/A
V58.1 / 35271468167 13 13 0 True
R22.1 / 35249247909 N/A N/A N/A N/A
R21.1 / 35248376664 N/A N/A N/A N/A
R20.1 / 35238867109 13 13 0 True
V57.1 / 35238063387 13 13 0 True
R19.1 / 35235911956 13 13 0 True
V56.1 / 35235905795 13 13 0 True

Evaluation Trends

Judge rates are grouped by dataset and evaluator code hashes, judge deployment and environment. Refusals are deterministic, not judge passes.

No comparable evaluation lineage has been collected yet.

Reporting Gaps

No malformed-evidence or route-verification gaps detected. N/A cells can still indicate skipped tests or missing artifacts.

title: Wiki Navigation

Clone this wiki locally