feat(cache): D4 coverage-ledger batch writes + in-process lookup cache - #49
Merged
Merged
Conversation
Behavior-preserving cache-internal scaling for the daily and intraday coverage ledgers. Public constructors, methods, parquet paths, column names/order, and coverage semantics (ok/empty count; failed/not_ready do not; snapshot freshness = latest successful fetched_at; not_ready_days unchanged) are all preserved. - CoverageLedger / IntradayCoverageLedger: add record_many(rows) — one normalize + one atomic parquet write for a batch; empty input is a no-op; each row normalizes exactly like record(); written parquet reindexed to the public column contract; no secret fields. record() now delegates to record_many([row]) so the single-row path is identical to the batch path. - in-process lookup cache: a per-instance frame cache + per-(endpoint,key[,freq]) memo so repeated covered_intervals / snapshot_fetched_at / covered_day_intervals do not re-read and re-filter the full parquet. The cache is invalidated on the file's mtime change (an out-of-instance write is never served stale) and refreshed in place on writes through this instance; read() returns a copy so callers cannot corrupt internal caches. _load() runs before the memo check so invalidation always clears stale memos. - TushareCache: the not-ready gap split (_record_gap_coverage) now builds its 1-2 coverage rows and writes them with one record_many (via _record_rows) instead of per-row writes; the not_ready counter and statuses are unchanged. Other call sites keep record() (which batches a single row) — successful gaps stay recorded as before, so a later failed fetch never un-records earlier successes. Intraday cache (one row per whole fetched gap) is unchanged — no obvious safe batch there. No config knob, no cache-root layout change, no D5 concurrency / schema registry / PanelStore work. phase0 anchor (0.9600/0.8408) unchanged. tests/test_coverage_ledger_scaling.py (network-free): record_many == repeated record (schema/order/values); record_many([]) no-op; covered_intervals counts only ok/empty; snapshot_fetched_at = latest successful; intraday mirror; repeated lookups do not re-read the parquet (spy on the load path); out-of-instance write invalidates the cache; read() returns a copy; not-ready split writes in ONE batch; plain gap writes one row; ledger columns carry no secret field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Data Layer D4 — a behavior-preserving cache-internal scaling step that reduces repeated full-ledger read/write overhead in the daily and intraday coverage ledgers. Not D5 (no concurrency / schema registry / PanelStore work).
What changed
record_many(rows)on bothCoverageLedgerandIntradayCoverageLedger: one normalize + one atomic parquet write for a batch; empty input is a no-op; each row normalizes exactly like a singlerecord; the written parquet is reindexed to the public column contract; no secret fields.record()now delegates torecord_many([row]), so the single-row path is identical to the batch path.(endpoint, key[, freq])memo so repeatedcovered_intervals/snapshot_fetched_at/covered_day_intervalsdo not re-read and re-filter the full parquet. Invalidated on the file's mtime change (an out-of-instance write is never served stale) and refreshed in place on writes through this instance._load()runs before the memo check so invalidation always clears stale memos.read()returns a copy so callers cannot corrupt internal caches.TushareCache._record_gap_coverage): the not-ready gap split now builds its 1–2 coverage rows and writes them with onerecord_many(via_record_rows) instead of per-row writes; thenot_readycounter and statuses are unchanged. Other call sites keeprecord()(a single-row batch) — successful gaps stay recorded as before, so a later failed fetch never un-records earlier successes.Invariants preserved
Public constructors + methods (
read,covered_intervals,snapshot_fetched_at,record, intradaycovered_day_intervals); parquet paths (manifest/coverage.parquet,manifest/coverage_intraday.parquet);LEDGER_COLUMNS/INTRADAY_LEDGER_COLUMNSnames+order; coverage semantics (onlyok/emptycount;failed/not_readydo not; snapshot freshness = latest successfulfetched_at; empty-return still prevents refetch; failed still retryable;not_ready_daysunchanged). No config knob, no cache-root layout change. Intraday cache (one row per whole fetched gap) unchanged — no obvious safe batch. phase0 anchor0.9600/0.8408unchanged.Tests / gates
tests/test_coverage_ledger_scaling.py(network-free, 13 tests):record_many== repeatedrecord(schema/order/values);record_many([])no-op;covered_intervalscounts only ok/empty (ignores failed/not_ready);snapshot_fetched_at= latest successful; intraday mirror; repeated lookups do not re-read the parquet (structural spy on the load path, not timing); out-of-instance write invalidates the cache;read()returns a copy; not-ready split writes in one batch; plain gap writes one row; ledger columns carry no secret field.ruffclean. All 17 configsvalidate-config.run-phase0anchor0.9600 / 0.8408.git diff --checkclean; no merge markers. Secret scan over changed files + test-generated ledger artifacts: real token value 0 hits; ledger schema has no token/secret column.Out of scope (not started)
D5 concurrency / thread pools / async / global rate limiter; endpoint schema registry;
PanelStoreappend/partition; data-quality changes; any research/backtest work.Live
data-updateagainst Tushare was not run (acceptance is network-free).