Skip to content

feat(cache): D4 coverage-ledger batch writes + in-process lookup cache - #49

Merged
StackOverFlow11 merged 1 commit into
mainfrom
feat/data-layer-d4-ledger-batch-scaling
Jun 19, 2026
Merged

feat(cache): D4 coverage-ledger batch writes + in-process lookup cache#49
StackOverFlow11 merged 1 commit into
mainfrom
feat/data-layer-d4-ledger-batch-scaling

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

Summary

Data Layer D4 — a behavior-preserving cache-internal scaling step that reduces repeated full-ledger read/write overhead in the daily and intraday coverage ledgers. Not D5 (no concurrency / schema registry / PanelStore work).

What changed

  • record_many(rows) on both CoverageLedger and IntradayCoverageLedger: one normalize + one atomic parquet write for a batch; empty input is a no-op; each row normalizes exactly like a single record; the written parquet is reindexed to the public column contract; no secret fields. record() now delegates to record_many([row]), so the single-row path is identical to the batch path.
  • In-process lookup cache: a per-instance frame cache + per-(endpoint, key[, freq]) memo so repeated covered_intervals / snapshot_fetched_at / covered_day_intervals do not re-read and re-filter the full parquet. Invalidated on the file's mtime change (an out-of-instance write is never served stale) and refreshed in place on writes through this instance. _load() runs before the memo check so invalidation always clears stale memos. read() returns a copy so callers cannot corrupt internal caches.
  • Cache wiring (TushareCache._record_gap_coverage): the not-ready gap split now builds its 1–2 coverage rows and writes them with one record_many (via _record_rows) instead of per-row writes; the not_ready counter and statuses are unchanged. Other call sites keep record() (a single-row batch) — successful gaps stay recorded as before, so a later failed fetch never un-records earlier successes.

Invariants preserved

Public constructors + methods (read, covered_intervals, snapshot_fetched_at, record, intraday covered_day_intervals); parquet paths (manifest/coverage.parquet, manifest/coverage_intraday.parquet); LEDGER_COLUMNS / INTRADAY_LEDGER_COLUMNS names+order; coverage semantics (only ok/empty count; failed/not_ready do not; snapshot freshness = latest successful fetched_at; empty-return still prevents refetch; failed still retryable; not_ready_days unchanged). No config knob, no cache-root layout change. Intraday cache (one row per whole fetched gap) unchanged — no obvious safe batch. phase0 anchor 0.9600/0.8408 unchanged.

Tests / gates

  • tests/test_coverage_ledger_scaling.py (network-free, 13 tests): record_many == repeated record (schema/order/values); record_many([]) no-op; covered_intervals counts only ok/empty (ignores failed/not_ready); snapshot_fetched_at = latest successful; intraday mirror; repeated lookups do not re-read the parquet (structural spy on the load path, not timing); out-of-instance write invalidates the cache; read() returns a copy; not-ready split writes in one batch; plain gap writes one row; ledger columns carry no secret field.
  • Full suite 656 passed (was 643; +13). ruff clean. All 17 configs validate-config. run-phase0 anchor 0.9600 / 0.8408. git diff --check clean; no merge markers. Secret scan over changed files + test-generated ledger artifacts: real token value 0 hits; ledger schema has no token/secret column.

Out of scope (not started)

D5 concurrency / thread pools / async / global rate limiter; endpoint schema registry; PanelStore append/partition; data-quality changes; any research/backtest work.

Live data-update against Tushare was not run (acceptance is network-free).

Behavior-preserving cache-internal scaling for the daily and intraday coverage
ledgers. Public constructors, methods, parquet paths, column names/order, and
coverage semantics (ok/empty count; failed/not_ready do not; snapshot freshness
= latest successful fetched_at; not_ready_days unchanged) are all preserved.

- CoverageLedger / IntradayCoverageLedger: add record_many(rows) — one
  normalize + one atomic parquet write for a batch; empty input is a no-op; each
  row normalizes exactly like record(); written parquet reindexed to the public
  column contract; no secret fields. record() now delegates to record_many([row])
  so the single-row path is identical to the batch path.
- in-process lookup cache: a per-instance frame cache + per-(endpoint,key[,freq])
  memo so repeated covered_intervals / snapshot_fetched_at / covered_day_intervals
  do not re-read and re-filter the full parquet. The cache is invalidated on the
  file's mtime change (an out-of-instance write is never served stale) and
  refreshed in place on writes through this instance; read() returns a copy so
  callers cannot corrupt internal caches. _load() runs before the memo check so
  invalidation always clears stale memos.
- TushareCache: the not-ready gap split (_record_gap_coverage) now builds its 1-2
  coverage rows and writes them with one record_many (via _record_rows) instead
  of per-row writes; the not_ready counter and statuses are unchanged. Other call
  sites keep record() (which batches a single row) — successful gaps stay
  recorded as before, so a later failed fetch never un-records earlier successes.

Intraday cache (one row per whole fetched gap) is unchanged — no obvious safe
batch there. No config knob, no cache-root layout change, no D5 concurrency /
schema registry / PanelStore work. phase0 anchor (0.9600/0.8408) unchanged.

tests/test_coverage_ledger_scaling.py (network-free): record_many == repeated
record (schema/order/values); record_many([]) no-op; covered_intervals counts
only ok/empty; snapshot_fetched_at = latest successful; intraday mirror; repeated
lookups do not re-read the parquet (spy on the load path); out-of-instance write
invalidates the cache; read() returns a copy; not-ready split writes in ONE batch;
plain gap writes one row; ledger columns carry no secret field.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant