Skip to content

P4-2: persistent cache for universe + tradability endpoints - #25

Merged
StackOverFlow11 merged 1 commit into
mainfrom
data-cache-universe-tradability
Jun 13, 2026
Merged

P4-2: persistent cache for universe + tradability endpoints#25
StackOverFlow11 merged 1 commit into
mainfrom
data-cache-universe-tradability

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

What

Extends the P4-1 endpoint-level read-through cache (market_daily + adj_factor) to five universe/tradability endpoints so real runs stop refetching them every time:
index_weight, suspend_d, namechange, stk_limit, stock_basic.

Disabled by default (data.cache.enabled=false) — existing configs run byte-for-byte as before; opt-in routes these endpoints through the cache.

How

One engine, three planning shapes:

  • dense per-symbol date-range (suspend_d, stk_limit): reuse the P4-1 gap + recent-tail planner;
  • index_code-keyed date-range (index_weight): coverage keyed by index_code, each uncovered gap paged in ≤90-day windows, raw snapshots stored;
  • snapshot/dimension (namechange per-symbol, stock_basic global): a refresh_dimension_days staleness policy (CoverageLedger.snapshot_fetched_at) + force_refresh.

The three tushare feeds (index_feed, tushare_flags, tushare_covariates) take an optional shared cache: cache-present reads through and applies the same finalizer as the direct path (cached == direct, locked by tests); cache=None is the unchanged direct path. The tushare client is built lazily inside the fetch closures, so a fully-covered warm run reads no token and constructs no client. One shared cache instance is threaded through all four runners (run_phase0, phase2_baseline, oos_stability, subset_validation).

Correctness preserved

  • index_weight stays PIT/as-of (370-day pre-start lookback flows into the cached request; latest-snapshot membership stays in the universe layer);
  • stk_limit stays raw-price (limit checks run before front-adjust);
  • stock_basic feeds only list_date (current-tag industry is never cached/used);
  • suspend_d / namechange / ST interval shapes unchanged;
  • cache stores raw endpoint facts only, never a derived flag as source of truth;
  • coverage is calendar-range; empty counts as coverage, a failed fetch records none (retried later);
  • no token or secret reaches the cache / ledger / logs.

Non-goals (P4-3): daily_basic, fina_indicator, index_member_all. No change to factor / alpha / portfolio / execution / OOS slicing / report math / front_adjust.

Observability

One concise run-log line after every cached endpoint has run:

data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=N index_weight_gap_fetches=N suspend_d_gap_fetches=N namechange_gap_fetches=N stk_limit_gap_fetches=N stock_basic_gap_fetches=N

Test plan

  • pytest -p no:cacheprovider428 passed (+17 new in tests/test_tushare_cache_universe.py: cold/warm/zero-calls + cached==direct per endpoint, empty-as-coverage, failed-not-coverage, duplicate-upsert-by-key, force_refresh, cache-disabled-untouched, stats line, no-secret)
  • ruff check . clean
  • all 12 config/*.yaml validate; run-phase0 (demo) OK
  • git diff --check clean; no conflict markers; AGENTS.md == CLAUDE.md
  • Real smoke (phase2 baseline, fresh temp cache root, cold → warm): cold line all seven endpoints non-zero; warm line all seven = 0 with coverage ledger unchanged and report metrics byte-identical to the P4-1 cached baseline (IC 0.0083 / annual −10.19% / maxDD −16.52% / vol 16.59% / sharpe −0.5703 / turnover 1.0818 / cost 1.19%). Wall cold 960s → warm 366s. Secret scan: 0 occurrences across cache parquet, ledger, run log, report.

Extend the P4-1 endpoint-level read-through cache (market_daily + adj_factor)
to five universe/tradability endpoints so real runs stop refetching them every
time: index_weight, suspend_d, namechange, stk_limit, stock_basic. Disabled by
default (data.cache.enabled=false) — existing configs run byte-for-byte as
before; opt-in routes these endpoints through the cache.

Cache engine (data/cache/tushare_cache.py) gains three planning shapes on one
engine:
- dense per-symbol date-range (suspend_d, stk_limit): reuse the P4-1 gap +
  recent-tail planner;
- index_code-keyed date-range (index_weight): coverage keyed by index_code,
  each uncovered gap paged in <=90-day windows, raw snapshots stored;
- snapshot/dimension (namechange per-symbol, stock_basic global): a
  refresh_dimension_days staleness policy (CoverageLedger.snapshot_fetched_at)
  plus force_refresh.

Semantics preserved: index_weight stays PIT/as-of (370-day pre-start lookback
flows into the cached request; latest-snapshot membership stays in the universe
layer); stk_limit stays raw-price; stock_basic feeds only list_date (current-tag
industry is never cached/used); suspend_d/namechange/ST interval shapes
unchanged; the cache stores raw endpoint facts only, never a derived flag as
source of truth. Coverage is calendar-range; empty counts as coverage, a failed
fetch records none (retried later). No token or secret reaches the cache/ledger.

Feeds (index_feed, tushare_flags, tushare_covariates) take an optional shared
cache: cache-present reads through and applies the same finalizer as the direct
path (cached == direct, locked by tests); cache=None is the unchanged direct
path. The tushare client is built lazily inside the fetch closures, so a fully
covered warm run reads no token and constructs no client.

One shared cache instance is threaded through all four runners (run_phase0,
phase2_baseline, oos_stability, subset_validation) via _build_cache, and a
single 7-endpoint run-log line is emitted after every cached endpoint has run:
  data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=N
  index_weight_gap_fetches=N suspend_d_gap_fetches=N namechange_gap_fetches=N
  stk_limit_gap_fetches=N stock_basic_gap_fetches=N

Config: data.cache.refresh_dimension_days (default 30, >=0).

Tests: tests/test_tushare_cache_universe.py (17) — cold/warm/zero-calls and
cached==direct for each endpoint, empty-as-coverage, failed-not-coverage,
duplicate-upsert-by-key, force_refresh, cache-disabled-untouched, stats line,
no-secret. pytest 428 passed; ruff clean; all 12 configs validate; run-phase0
(demo) OK.

Real smoke (phase2 baseline, fresh temp cache root, cold -> warm): cold line has
all seven endpoints non-zero; warm line shows all seven = 0 with the coverage
ledger unchanged and report metrics byte-identical to the P4-1 cached baseline
(IC 0.0083 / annual -10.19% / maxDD -16.52% / vol 16.59% / sharpe -0.5703 /
turnover 1.0818 / cost 1.19%). Wall cold 960s -> warm 366s (daily_basic /
index_member_all still live; those are P4-3). Secret scan: 0 occurrences.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant