feat: factor evaluation contract layer (cdd) + StandardFactorEvaluator + research dashboard + PR-C jump-amount-corr reproduction - #63
Merged
Conversation
…or ABC + report + three-axis verdict + pre-registered CI criteria) Three independent axis-verdicts (Predictive / Incremental / Tradable), each PASS/FAIL/INSUFFICIENT_DATA/NOT_ASSESSED, plus a derived deployment label (Adopt/Watch/Reject/INSUFFICIENT-DATA). A factor that re-expresses a known factor FAILs the Incremental axis and is Rejected where a scalar raw-IC verdict passed it. Pre-registered success criteria + confidence intervals (change #3): - EvalConfig.success_criteria: a frozen VerdictThresholds declared BEFORE evaluate() runs is pre-registered by construction. None -> the documented global default. The report stamps criteria_source ('declared' | 'default') in the provenance box and JSON, so the bar cannot be tuned post-hoc without a visibly different report. The per-run object passes exactly the same VerdictThresholds validation. - The magnitude PASS test (ICIR, incremental ICIR) now gates on the N_eff-based LOWER confidence bound in the expected direction, not the naked point, via _clears_magnitude. A thin/noisy estimate -> wide CI -> low lower bound -> not a PASS. This does NOT duplicate the sample gate: the gate is the 'can we estimate a CI at all?' precondition (INSUFFICIENT below it); the lower-CI test is the 'is it convincingly above the bar?' PASS test above it. FAIL stays POINT-based and known-fact-only; a NaN CI bound never convicts and never PASSes. - config.py imports VerdictThresholds (analytics.eval.verdict has no back-import; no cycle). The standard layer computes the CIs; the contract consumes the bounds. v0.7 — two threshold refinements surfaced by the PR-C jump-amount-corr repro: - Incremental axis gets its OWN bar, min_incremental_abs_icir=0.15, SEPARATE from the raw predictive min_abs_icir=0.30. Orthogonalizing on the known-factor book removes variance, so the residual ICIR lives on a structurally smaller scale; reusing the raw 0.30 bar was a category error that silently failed genuinely partial-incremental factors. The raw predictive bar is unchanged (a 0.20 ICIR clears incremental but still fails predictive). - Monotonicity becomes a DIRECTION gate, not a strength bar: default min_monotonicity_spearman=0.0 means "aligned monotonicity must be strictly > 0" (buckets ordered the RIGHT way / not reversed), not "strongly monotone". The old 0.80 strength bar rejected tail-concentrated real factors whose power sits in one extreme (jump-amount-corr: ICIR -0.40, NW-t -14.8, yet Q3-humped). A reversed aligned monotonicity still fails. Raise the threshold to demand strength again. Both defaults remain UNVALIDATED-BY-DATA (design section 11).
…(three-axis + N_eff CIs) StandardFactorEvaluator fills the 8 mandatory sections by reducing the vectorized eval-IR built once per run; joint orthogonalized IC (Gram-Schmidt, loop-free over dates) drives the Incremental axis. N_eff confidence intervals (change #3): - stats.py: information_ratio_ci (Lo 2002 Sharpe SE, N->N_eff: SE(IR)=sqrt((1+0.5*IR^2)/N_eff)) and mean_ci (SE=std/sqrt(N_eff)) at 95%, reusing effective_sample_size. Degenerate inputs -> NaN, never a fabricated CI. - standard.py: predictive_power attaches the ICIR CI (and IC-mean CI); purity attaches the incremental ICIR CI computed on the ORTHOGONALIZED IC series' OWN N_eff; return_risk reports the base-spread CI (point-only gating, disclosed). The contract's Predictive/Incremental PASS gates on the LOWER bound, so an autocorrelated or noisy IC series -> wider CI -> lower bound below the bar -> not a PASS, where an i.i.d. series of the same raw length passes. - pre-registered EvalConfig.success_criteria flows through StandardFactorEvaluator end to end and is stamped (criteria_source) in the report.
…nel figure) A one-PNG visual report for a FactorEvalReport, styled after the Kaiyuan (开源证券) layered-backtest charts: thin per-quantile NAV curves + a thick dark-maroon long-short curve on a secondary axis, dense YYYYMMDD date axis, white canvas, top multi-column legend. analytics/eval/figures.py (pure rendering, matplotlib Agg, NOT re-exported from __init__ so importing the eval package never pulls a plotting backend): - render_factor_dashboard(report, ir, out_path) -> the dashboard. - DashboardData.from_report(report, ir): normalizes the section payloads + three-axis verdict (from the report) and the ic / quantile_returns series (from the StandardEvalIR) into one testable value. - Panels: header + three-axis verdict chips, hero layered quantile NAV + gross long-short, per-period & cumulative IC, quantile final NAV, OOS train/test sign consistency, IC decay, purity corr-with-book, cost-drag sensitivity, coverage. Every panel degrades to "not available" on a Skipped/absent section rather than crashing. - MANDATORY factor-definition band (contract constraint): a factor may not be shown without stating how it is computed. Sourced from the report's FactorSpec (description + inputs + expected sign + horizon/return basis + family + min-history + price basis + the minute block when intraday) - reusing existing spec fields, no new mandatory field. analytics/eval/standard.py: StandardFactorEvaluator.evaluate_with_ir returns (report, ir) by mirroring the frozen ABC evaluate() template step for step - the contract evaluate() returns only the report (frozen signature) but the dashboard needs the IR series; building the IR once here avoids a second pass and the report stays byte-identical to evaluate()'s (test-locked). Frozen contract files untouched; the long-short curve is GROSS (cost enters via the separate cost-drag panel), avoiding the aligned-net cost-sign question.
…dFactorEvaluator run (PR-C)
The runner now emits the research-style dashboard PNG alongside the Markdown +
JSON reports, for both the no-book and with-book runs:
- evaluate_two_runs switches evaluator.evaluate -> evaluate_with_ir so it keeps
the StandardEvalIR (per-period IC + quantile return series) the dashboard needs;
the reports are byte-identical to the evaluate() path.
- render_factor_dashboard writes {stem}_{no_book,with_book}_dashboard.png. The
with-book dashboard shows the purity panel + Incremental PASS; the no-book one
degrades that panel to "not available" (Incremental NOT_ASSESSED).
- _RunReports carries the two dashboard paths; the CLI prints them.
The dashboard's mandatory factor-definition band is sourced from this factor's
FactorSpec (its description already states the full computation recipe), so no
new spec field was needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
4 commits, one stacked feature chain — the contract-driven (cdd) factor-evaluation layer, iterated v0.1→v0.7 and closed end-to-end by reproducing a real research-report factor through it.
FactorSpec(mandatory Factor classattr, pre-registeredexpected_ic_sign, PIT contract, intraday block validation) +FactorEvaluatorABC (8 mandatory sections, template method, missing-section raises) +FactorEvalReport(deterministic md/json, secret-redacting renderer) + three-axis verdict (Predictive × Incremental × Tradable → Adopt/Watch/Reject/INSUFFICIENT-DATA; asymmetric gate — hard Rejects precede the sample gate; unknown never convicts) + pre-registered success criteria (EvalConfig.success_criteria,criteria_sourcestamped) + N_eff-based CIs (PASS gates on the lower CI bound in the expected direction). v0.7: separatemin_incremental_abs_icir=0.15(orthogonalization shrinks the residual scale — reusing the raw 0.30 bar was a category error) + monotonicity as a direction gate (default 0.0 = aligned strictly > 0, not a strength bar; the old 0.8 bar rejected tail-concentrated real factors). Both defaults remain UNVALIDATED-BY-DATA (design §11).FactorSpec— a factor may not be shown without stating how it is computed.evaluate_with_ir()mirrors the frozen ABC template (report byte-identical, test-locked); figures NOT re-exported from__init__(no matplotlib on package import).stk_mins_live_calls=0), no-book + with-book (value_ep/value_bp/volatility_20). Result: Watch / Watch — predictive PASS (ICIR −0.401, NW-t −14.8, lower CI −0.342 clears 0.30, OOS sign consistent), incremental PASS (orth ICIR −0.290, lower CI 0.229 > 0.15), tradable NOT_ASSESSED + exploratory cap. EXPLORATORY — not a return claim.Test plan
pytest1238 passed (contract 317 + standard/IR + figures 7 + runner 11 + all prior suites; phase0 demoic 0.9600 / annual 0.8408unchanged)ruffclean on the full changed set.config.json) in additions, reports, logs, dashboards