Skip to content

Stop the served findings contradicting the evaluation - #8

Merged
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control
Aug 8, 2026
Merged

Stop the served findings contradicting the evaluation#8
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control

Conversation

@jadoon200

Copy link
Copy Markdown
Owner

Verifying the live deploy caught the shipped Q13 finding still asserting that daily
autocorrelation "predicted every outcome measured so far"
— the exact claim the Q13
experiment refuted hours earlier. The verdict string, the explainer and the docs were all
updated; the Finding record baked into the snapshot was not. The deployed site was making a
claim the project had already disproved.

Fixed

  • Q13 → refuted, with the serverless numbers, the non-monotone shape, the failed
    spectral-entropy replacement, and the low-frequency lead that does not survive Bonferroni.
  • Q4 quoted "19 of 24 settings", which no configuration of evaluate_frontier.py can
    produce — it emits three traces × six ratios. Replaced with what the corrected run gives:
    strictly better in 9 of 18, worse in 6, identical in 3 at the ratio where p95 and the
    derived target coincide by construction. The regime matters more than the count and is now
    stated: derived wins below 19:1, loses above.
  • Q10's evidence was one sentence with no numbers. It now carries the frontier figures.

Tested, not remembered

This is the third time a served string has drifted from docs/EVAL.md — after the
verdict text claiming nothing below 0.50 ever won, and the explainer claiming an unsupported
scope. Prose can't be fully pinned, but the specific disproved assertions can.

test_findings.py pins that Q13 does not reassert the disproved claim, that Q1 names the
percentile recommender rather than threshold HPA as what it lost to, that Q4 does not quote
the unreproducible count, and that no confirmed finding claims a win for forecasting without
naming the regime it holds in. It also catches thin evidence — which is what flagged Q10.

Live deploy verified

delphi-c5pp.onrender.com is up and serving from the current main: /health reports
snapshot_mode: demo, all three band states render, the interactive diagnostic computes
end-to-end (r=0.724 → verdict, q*=0.9500 from prices), /geoint/evidence is ARGUS-shaped,
zero console errors. README now links it.

Gate

ruff clean; mypy strict clean; 195 tests passing (was 190).

Verifying the live deploy caught the shipped Q13 finding still asserting that
daily autocorrelation "predicted every outcome measured so far" — the exact claim
the Q13 experiment refuted hours earlier. The verdict string, the explainer and
the docs were all updated; the Finding record baked into the snapshot was not, so
the deployed site was making a claim the project had already disproved.

Q13 is now recorded as refuted, with the serverless numbers, the non-monotone
shape, the failed spectral-entropy replacement and the low-frequency lead that
does not survive Bonferroni.

Q4 quoted "19 of 24 settings", which no configuration of evaluate_frontier.py can
produce — it emits three traces by six ratios. Replaced with the counts the
corrected run actually gives: strictly better in 9 of 18, worse in 6, identical in
3 at the ratio where p95 and the derived target coincide by construction. The
regime matters more than the count and is now stated: derived wins below 19:1 and
loses above.

Q10's evidence was one sentence with no numbers. It now carries the frontier
figures that support it.

This is the third time a served string has drifted from docs/EVAL.md, so it is now
tested rather than remembered. test_findings.py pins that Q13 does not reassert the
disproved claim, that Q1 names the percentile recommender rather than threshold
HPA as what it lost to, that Q4 does not quote the unreproducible count, and that
no confirmed finding claims a win for forecasting without naming the regime it
holds in. Thin evidence is also caught — which is what flagged Q10.

README now links the live deployment.
@jadoon200
jadoon200 merged commit 5800673 into main Aug 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant