fix(analytics): contract v0.8 + v0.9 — cost-correct aligned spread, and gate monotonicity on its N_eff CI - #74
Merged
Conversation
…e monotonicity gate
First authorized change to the frozen `analytics/eval/` contract layer. Two
defects surfaced by the ten factor reproductions run under the no-touch rule.
1. Hypothesis-aligned net spread added the cost back at sign=-1.
Four sites computed `sign * net` where `net` was already cost-subtracted, so
at sign=-1 the expression expanded to `-gross + cost`: the factor was handed
its own trading fees back as profit. The hypothesis decides WHICH LEG IS
LONG; cost is a drag in either direction, so the legs are flipped first and
the cost subtracted after:
cost = gross - net (= fee * multiplier * leg_turnover)
aligned_net = sign * gross - cost
Worked check (locked by test): gross 0.000125, net(1x) -0.001983 gives
cost 0.002108 and aligned -0.002233; the old form read +0.001983.
Two of the four sites are `_base_spread` / `_all_spreads_negative` in
verdict.py, which feed the Tradable axis PASS condition and FAIL reason — a
GATE defect, not a display one. It never bit because Tradable has been
NOT_ASSESSED in every run to date.
`VerdictInputs` gains `gross_long_short_mean`; an unknown gross makes a
sign=-1 aligned spread UNKNOWN, which per the existing rule neither convicts
nor passes. sign=+1 is bit-identical (`+1*gross - (gross-net) == net`, and
the implementation returns `net` directly rather than round-tripping it).
`net_long_short_by_cost` and `gross_long_short_mean` keep their plain,
sign-agnostic meaning and wording verbatim — existing reports rely on it.
2. `monotonicity_spearman` is magnitude-sensitive but gated a rank axis.
The pooled statistic correlates bucket index against cross-date arithmetic
MEAN bucket returns: unbounded, so a few extreme-return days concentrated in
one bucket can flip it while the daily-capped rank IC barely moves.
Adds `monotonicity_spearman_by_date`: each date's own Spearman across its
buckets (skipped when that date has fewer than 3 finite buckets, or when its
buckets are all tied), each bounded in [-1, 1] before the cross-date mean —
structurally parallel to the rank IC. The Predictive axis now gates on it.
The pooled field is KEPT as a reported value, unchanged in semantics, so
historical reports stay comparable. A missing/NaN per-date figure falls back
to the pooled one and DISCLOSES the substitution in the axis reasons, so a
pre-v0.8 IR stays judgeable without pretending it was judged on the new
statistic.
Both are UNVALIDATED-BY-DATA design corrections (design doc §11): reasoned
defects in the implementation, not thresholds tuned after seeing results.
Impact on the eleven factor reproductions is NOT assessed here — re-running them
and disclosing every verdict change is the separate second stage.
Gates: pytest 1668 passed (1649 baseline + 19 new), ruff clean, phase0
ic 0.9600 / annual 0.8408 unchanged, secret scan 0 hits.
…inputs The verdict now reads monotonicity_spearman_by_date (the gated statistic) and gross_long_short_mean (needed to align a net spread), but the documentation dict still listed only the two pre-v0.8 keys. Documentation-only: VERDICT_KEYS is re-exported and referenced in docstrings, never used for runtime validation.
…eff CI
v0.8 moved the Predictive axis' monotonicity gate onto a per-date rank
statistic — the right KIND of statistic for a rank axis. Re-running the
eleven factors then showed that statistic is heavily attenuated by daily
noise: a constructively perfect quantile ladder scores 1.000, while the
best real factors score 0.045-0.106 and two of them sit 0.021 apart
across a gate that was still a bare 0.0 with no dispersion estimate
anywhere in the decision. Comparing a noise-compressed mean against zero
is a coin flip wearing a criterion's clothes — the same defect v0.6 fixed
for the ICIR by gating on its N_eff lower bound.
The direction is now decided by the N_eff-based CI of the per-date series
and is THREE-VALUED (user-chosen semantics):
aligned CI low > bar -> direction HOLDS (may PASS)
aligned CI high < 0 -> direction CONTRADICTED (FAIL)
otherwise -> UNKNOWN (neither convicts
nor acquits)
UNKNOWN NEVER RESCUES. It is evaluated AFTER every other predictive
criterion: if ICIR / NW-t / win rate / out-of-sample sign consistency
fail, the axis still FAILs and the unknown direction is merely disclosed
alongside. Only when everything else has cleared and the direction ALONE
cannot be asserted does the axis become INSUFFICIENT_DATA. "We could not
tell" withholds a PASS; it never withholds a FAIL that other evidence
independently earned — the same asymmetry as "unknown never convicts",
read from the other side.
The FAIL side stays pinned at 0 rather than tracking the configurable
bar: "above the bar" is the positive claim and moves with the bar, but
"below zero" is REVERSAL, a fact about the sign. Pinning FAIL to a raised
bar would convict a correctly-ordered-but-weak factor of being reversed.
Reuses the existing machinery rather than growing a parallel one: the CI
is the same mean_ci() v0.6 built for the ICIR, and the aligned bounds
come from a single _aligned_bounds() that _aligned_lower_bound() now
delegates to, so the ICIR gate and the monotonicity gate cannot drift
apart on the min/max swap at sign=-1.
Three fallback levels, each DISCLOSED in the axis reasons: CI -> the bare
per-date point (v0.8, two-valued: with no dispersion estimate there is
nothing to be uncertain with, and inventing an UNKNOWN here would
silently upgrade every pre-v0.9 FAIL to INSUFFICIENT_DATA) -> the pooled
magnitude statistic (v0.7, wording kept verbatim). Levels 1 and 2 are
worded differently on purpose: one degrades WHICH STATISTIC, the other
WHICH ESTIMATOR OF ITS UNCERTAINTY, and a reader must be able to tell
which happened.
spearman_by_date is refactored into the mean of a new
spearman_series_by_date and is BIT-IDENTICAL to v0.8 — the mean
deliberately keeps the sequential sum()/len() rather than Series.mean(),
since the two can differ in the last bit and this is a published,
cross-run-comparable field. Locked against the v0.8 body frozen in the
test as an oracle.
Mutation-tested (standing rule after a prior PR shipped a toothless
invariance test): reverting the gate to the bare-sign form fails 8 of the
8 new verdict tests plus the v0.8 fallback-disclosure test; zero-filling
skipped dates instead of omitting them fails 5; swapping the sequential
sum for Series.mean() fails the bit-identity test alone.
Stage A only — no factor evaluation was re-run. Gates: pytest 1682 passed
(1668 baseline + 14 new, no pre-existing expectation changed), ruff
clean, phase0 ic 0.9600 / annual 0.8408 unchanged, secret scan 0 hits.
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The first authorized modification to the frozen evaluation contract, after eleven factor reproductions ran under a strict no-touch rule on
analytics/eval/. Those runs surfaced two defects; both are fixed here, each independently reviewed, and each reconciled by re-running all eleven factors.v0.8 defect 1 — the hypothesis-aligned spread added costs back (a GATE defect, not display-only)
Four sites computed
sign * netwherenetis already cost-subtracted, which atsign=−1expands to−gross + cost— the fees a factor paid were handed back to it as profit. Corrected toaligned = sign*gross − cost(flip the legs by hypothesis, then always subtract cost; cost is a drag in either direction). Two of the four sites are_base_spread/_all_spreads_negativein verdict.py, which feed the Tradable axis's PASS condition — it never bit only because Tradable has been NOT_ASSESSED in every run to date. Worked case: gross 0.000125, net 1× −0.001983 → cost 0.002108 → aligned −0.002233; the defective form gave +0.001983.sign=+1is bit-identical (regression-locked). Reconciliation confirmed the 2×cost signature on all four sign=−1 factors, with sign=+1 spreads unchanged. Reported annualized aligned spreads were badly inflated: jump_amount_corr +23.14% → −0.01%, minute_ideal_amplitude +64.07% → −0.27%, intraday_amp_cut +6.30% → −20.52%, ridge_minute_return +64.28% → −43.25%.v0.8 defect 2 — a magnitude-sensitive statistic gated a rank-based axis
monotonicity_spearmancorrelates bucket index against date-equal-weighted arithmetic means — unbounded, so a handful of extreme return-days in one bucket can flip it while the daily-capped rank IC barely moves. Addedmonotonicity_spearman_by_date(per-date Spearman, bounded [−1,1] before averaging — structurally parallel to the rank IC) and moved the gate onto it. The pooled field is retained and reported, with a disclosed fallback.v0.9 — the replacement statistic is noise-attenuated, so the bare 0.0 bar was inside the noise band
The eleven-factor re-run revealed the per-date statistic scores only 0.045–0.106 on factors with a perfect pooled ladder, while the direction gate still sat at a bare 0.0 with no dispersion estimate — two factors decided 0.021 apart. That is the same defect v0.6 fixed for the ICIR (a noisy point estimate must not buy a PASS). The gate now reads the N_eff-based CI of the per-date series (reusing the existing
mean_ci), three-valued:The critical routing rule: UNKNOWN never rescues a factor failing on other grounds. Monotonicity was removed from the point-signal boolean (folding a three-valued fact into an
andchain collapses it); if ICIR / NW-t / win-rate / OOS-consistency fail, the axis still FAILs and the UNKNOWN is merely disclosed alongside. Measured CI half-widths came in at 0.0291–0.0448, confirming the bare 0.0 bar was operating as an implicit ~0.035 bar nobody knew was there.Impact — all eleven factors re-run under each version
Three verdict cells changed, all traceable to the gate change and stated as such:
intraday_amp_cut(no_book)intraday_amp_cut(with_book)ridge_minute_return(no_book)ridge_minute_return's path across three contract versions is the clearest illustration: v0.7 Reject (pooled statistic read as reversed) → v0.8 Watch (per-date point +0.020) → v0.9 INSUFFICIENT-DATA (CI straddles 0). The first two were verdicts rendered on noise, differing only in which way the noise pointed; the third admits the direction is not determinable.Invariants held: IC mean, ICIR, NW-t, win rate, incremental ICIR, and BOTH monotonicity point statistics are bit-identical before and after, for all eleven — these fixes touch neither the IC path nor the incremental path.
Routing verified on live data:
amp_marginal_anomaly_vol(fails at the OOS-reversal step) andpeak_interval_kurtosis(fails on OOS inconsistency AND ICIR 0.006) both have genuinely UNKNOWN monotonicity yet both still Reject, with the UNKNOWN only disclosed. No leak.Test plan
analytics/eval/{report,sections,standard,stats,verdict}.py+ the two contract test files. The eleven factor modules, their tests, configs,qt/,runtime/,data/— all untouchedgrossinput the defective one did not) rather than outcome engineering