P4-2: persistent cache for universe + tradability endpoints - #25
Merged
Conversation
Extend the P4-1 endpoint-level read-through cache (market_daily + adj_factor) to five universe/tradability endpoints so real runs stop refetching them every time: index_weight, suspend_d, namechange, stk_limit, stock_basic. Disabled by default (data.cache.enabled=false) — existing configs run byte-for-byte as before; opt-in routes these endpoints through the cache. Cache engine (data/cache/tushare_cache.py) gains three planning shapes on one engine: - dense per-symbol date-range (suspend_d, stk_limit): reuse the P4-1 gap + recent-tail planner; - index_code-keyed date-range (index_weight): coverage keyed by index_code, each uncovered gap paged in <=90-day windows, raw snapshots stored; - snapshot/dimension (namechange per-symbol, stock_basic global): a refresh_dimension_days staleness policy (CoverageLedger.snapshot_fetched_at) plus force_refresh. Semantics preserved: index_weight stays PIT/as-of (370-day pre-start lookback flows into the cached request; latest-snapshot membership stays in the universe layer); stk_limit stays raw-price; stock_basic feeds only list_date (current-tag industry is never cached/used); suspend_d/namechange/ST interval shapes unchanged; the cache stores raw endpoint facts only, never a derived flag as source of truth. Coverage is calendar-range; empty counts as coverage, a failed fetch records none (retried later). No token or secret reaches the cache/ledger. Feeds (index_feed, tushare_flags, tushare_covariates) take an optional shared cache: cache-present reads through and applies the same finalizer as the direct path (cached == direct, locked by tests); cache=None is the unchanged direct path. The tushare client is built lazily inside the fetch closures, so a fully covered warm run reads no token and constructs no client. One shared cache instance is threaded through all four runners (run_phase0, phase2_baseline, oos_stability, subset_validation) via _build_cache, and a single 7-endpoint run-log line is emitted after every cached endpoint has run: data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=N index_weight_gap_fetches=N suspend_d_gap_fetches=N namechange_gap_fetches=N stk_limit_gap_fetches=N stock_basic_gap_fetches=N Config: data.cache.refresh_dimension_days (default 30, >=0). Tests: tests/test_tushare_cache_universe.py (17) — cold/warm/zero-calls and cached==direct for each endpoint, empty-as-coverage, failed-not-coverage, duplicate-upsert-by-key, force_refresh, cache-disabled-untouched, stats line, no-secret. pytest 428 passed; ruff clean; all 12 configs validate; run-phase0 (demo) OK. Real smoke (phase2 baseline, fresh temp cache root, cold -> warm): cold line has all seven endpoints non-zero; warm line shows all seven = 0 with the coverage ledger unchanged and report metrics byte-identical to the P4-1 cached baseline (IC 0.0083 / annual -10.19% / maxDD -16.52% / vol 16.59% / sharpe -0.5703 / turnover 1.0818 / cost 1.19%). Wall cold 960s -> warm 366s (daily_basic / index_member_all still live; those are P4-3). Secret scan: 0 occurrences.
StackOverFlow11
added a commit
that referenced
this pull request
Jun 13, 2026
docs: mark PR #25 (P4-2) merged in progress docs
This was referenced Jun 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the P4-1 endpoint-level read-through cache (market_daily + adj_factor) to five universe/tradability endpoints so real runs stop refetching them every time:
index_weight,suspend_d,namechange,stk_limit,stock_basic.Disabled by default (
data.cache.enabled=false) — existing configs run byte-for-byte as before; opt-in routes these endpoints through the cache.How
One engine, three planning shapes:
suspend_d,stk_limit): reuse the P4-1 gap + recent-tail planner;index_weight): coverage keyed byindex_code, each uncovered gap paged in ≤90-day windows, raw snapshots stored;namechangeper-symbol,stock_basicglobal): arefresh_dimension_daysstaleness policy (CoverageLedger.snapshot_fetched_at) +force_refresh.The three tushare feeds (
index_feed,tushare_flags,tushare_covariates) take an optional shared cache: cache-present reads through and applies the same finalizer as the direct path (cached == direct, locked by tests);cache=Noneis the unchanged direct path. The tushare client is built lazily inside the fetch closures, so a fully-covered warm run reads no token and constructs no client. One shared cache instance is threaded through all four runners (run_phase0,phase2_baseline,oos_stability,subset_validation).Correctness preserved
index_weightstays PIT/as-of (370-day pre-start lookback flows into the cached request; latest-snapshot membership stays in the universe layer);stk_limitstays raw-price (limit checks run before front-adjust);stock_basicfeeds onlylist_date(current-tagindustryis never cached/used);suspend_d/namechange/ ST interval shapes unchanged;Non-goals (P4-3):
daily_basic,fina_indicator,index_member_all. No change to factor / alpha / portfolio / execution / OOS slicing / report math /front_adjust.Observability
One concise run-log line after every cached endpoint has run:
Test plan
pytest -p no:cacheprovider— 428 passed (+17 new intests/test_tushare_cache_universe.py: cold/warm/zero-calls + cached==direct per endpoint, empty-as-coverage, failed-not-coverage, duplicate-upsert-by-key, force_refresh, cache-disabled-untouched, stats line, no-secret)ruff check .cleanconfig/*.yamlvalidate;run-phase0(demo) OKgit diff --checkclean; no conflict markers; AGENTS.md == CLAUDE.md