Stop the served findings contradicting the evaluation - #8
Merged
Conversation
Verifying the live deploy caught the shipped Q13 finding still asserting that daily autocorrelation "predicted every outcome measured so far" — the exact claim the Q13 experiment refuted hours earlier. The verdict string, the explainer and the docs were all updated; the Finding record baked into the snapshot was not, so the deployed site was making a claim the project had already disproved. Q13 is now recorded as refuted, with the serverless numbers, the non-monotone shape, the failed spectral-entropy replacement and the low-frequency lead that does not survive Bonferroni. Q4 quoted "19 of 24 settings", which no configuration of evaluate_frontier.py can produce — it emits three traces by six ratios. Replaced with the counts the corrected run actually gives: strictly better in 9 of 18, worse in 6, identical in 3 at the ratio where p95 and the derived target coincide by construction. The regime matters more than the count and is now stated: derived wins below 19:1 and loses above. Q10's evidence was one sentence with no numbers. It now carries the frontier figures that support it. This is the third time a served string has drifted from docs/EVAL.md, so it is now tested rather than remembered. test_findings.py pins that Q13 does not reassert the disproved claim, that Q1 names the percentile recommender rather than threshold HPA as what it lost to, that Q4 does not quote the unreproducible count, and that no confirmed finding claims a win for forecasting without naming the regime it holds in. Thin evidence is also caught — which is what flagged Q10. README now links the live deployment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verifying the live deploy caught the shipped Q13 finding still asserting that daily
autocorrelation "predicted every outcome measured so far" — the exact claim the Q13
experiment refuted hours earlier. The verdict string, the explainer and the docs were all
updated; the
Findingrecord baked into the snapshot was not. The deployed site was making aclaim the project had already disproved.
Fixed
spectral-entropy replacement, and the low-frequency lead that does not survive Bonferroni.
evaluate_frontier.pycanproduce — it emits three traces × six ratios. Replaced with what the corrected run gives:
strictly better in 9 of 18, worse in 6, identical in 3 at the ratio where p95 and the
derived target coincide by construction. The regime matters more than the count and is now
stated: derived wins below 19:1, loses above.
Tested, not remembered
This is the third time a served string has drifted from
docs/EVAL.md— after theverdict text claiming nothing below 0.50 ever won, and the explainer claiming an unsupported
scope. Prose can't be fully pinned, but the specific disproved assertions can.
test_findings.pypins that Q13 does not reassert the disproved claim, that Q1 names thepercentile recommender rather than threshold HPA as what it lost to, that Q4 does not quote
the unreproducible count, and that no confirmed finding claims a win for forecasting without
naming the regime it holds in. It also catches thin evidence — which is what flagged Q10.
Live deploy verified
delphi-c5pp.onrender.comis up and serving from the currentmain:/healthreportssnapshot_mode: demo, all three band states render, the interactive diagnostic computesend-to-end (r=0.724 → verdict, q*=0.9500 from prices),
/geoint/evidenceis ARGUS-shaped,zero console errors. README now links it.
Gate
ruff clean; mypy strict clean; 195 tests passing (was 190).