feat(data): all-A historical backfill (chunked, resumable, failure-tolerant) - #61
Merged
Merged
Conversation
…lerant) Add `data-backfill`, a separate manual command that warms the same tushare read-through caches as the nightly `data-update`, but over a WIDE [backfill.start, today] window (and full 1min history) instead of the incremental lookback tail. The nightly incremental job is unchanged. - qt/data_backfill.py: run_data_backfill + BackfillResult + format_summary. Symbols are chunked into batches of backfill.chunk_size; each batch is warmed over the wide window. A persistent per-batch fetch failure is logged (secret-free: batch index + exception type only) and skipped, and the run continues — the failed batch's gaps stay uncovered and are retried on re-run (opposite of the fail-fast incremental job, by design). Resumability is inherent to the coverage ledgers (fixed window => stable re-runs). Minute history is warmed over the full window only when include_minute is true. A future backfill.start (config typo) raises readably instead of a silent no-op (an inverted window subtracts to zero gaps). The result/summary surface the concurrency mode (serial vs N workers + global rate limit). - qt/data_updater.py: extract behavior-preserving _build_scheduler / _build_caches helpers out of run_data_update (which now calls them) so backfill reuses the exact same cache + scheduler construction. Incremental behavior is byte-identical (existing data_update suites pass unchanged). - qt/config.py: BackfillCfg (start / chunk_size / include_minute, validated) as an all-defaults DataUpdateCfg.backfill sub-config, so every existing config still validates unchanged. - qt/cli.py: data-backfill subcommand (mirrors data-update; runs no factor/alpha/portfolio/backtest/PanelStore). - config/data_update_all_a.yaml: backfill block with an honest cost note (a cold all-A minute backfill is a long, many-hour, resumable operation). - docs/ops/data_update_schedule.md: Historical backfill section (run before enabling the nightly timer). - tests/test_data_backfill.py: 22 network-free tests (chunking, per-batch failure tolerance, minute full-window control, all-A resolution, config validation, future-start guard, real-cache resumability, and a joint failure+resume test proving a re-run re-fetches only the previously-failed symbol). Caches stay raw-only; no token is stored or logged; coverage semantics unchanged. phase0 anchor unchanged (ic_mean=0.9600, annual_return=0.8408).
StackOverFlow11
added a commit
that referenced
this pull request
Jul 16, 2026
docs: record all-A historical backfill (PR #61)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data-layer: all-A historical backfill (chunked, resumable, failure-tolerant)
PR-2 of the post-market all-A auto-fetch effort. PR-1 added the nightly INCREMENTAL warm + systemd timer; this PR adds the one-time HISTORICAL BACKFILL (
data-backfillCLI) to fill deep history for daily / financial / minute across the whole listed A-share market.What it adds (separate from the incremental job)
qt/data_backfill.py::run_data_backfill+data-backfillCLI subcommand. Chunks the universe intochunk_sizebatches, warms each over the WIDE[backfill.start, today]window (not the nightlytoday − lookbackwindow), and — wheninclude_minute— warms minutes over the full window (not the 7-day tail; the I2 intraday cache's ≤23-day paging is reused, not re-implemented). Runs NOTHING else (no factors/alpha/portfolio/backtest/PanelStore).except Exception, so Ctrl-C still propagates) and SKIPPED — its gaps stay uncovered (retryable on re-run); the run continues and tallies failed batches/symbols.start > todayguard: raises a readable error (an inverted window would otherwise be a SILENT no-op, sincesubtract_intervalstreats it as zero gaps).data_update.backfill: {start, chunk_size, include_minute}(all-defaults → every existing config validates unchanged); abackfill:block added toconfig/data_update_all_a.yamlwith an honest cost note.docs/ops/data_update_schedule.md: "Historical backfill" section (long manual resumable run; run BEFORE enabling the nightly timer).Invariants
run_data_updatepath is byte-identical: the only shared-code change is a verbatim extract of_build_scheduler/_build_caches(same kwargs, same construction order); the 49 existingdata_update*tests pass unchanged (independently confirmed via a main-vs-branch pytest diff: 771 → 793, exactly +22 new, 0 existing tests altered).ic_mean=0.9600, annual_return=0.8408.Acceptance (independently re-run)
pytest: 793 passed (+22) ·ruffclean · all 20 configs validate ·run-phase0:0.9600 / 0.8408"FAKE_TOKEN"tmp fixture)Review (2 rounds)
start > todaysilent-no-op guard, and a joint failure+resume test against a real cache; LOW — surfaced the concurrency mode + global rate limit in the backfill summary.Rollout (user-driven, Stage-1)
data-backfillis a long, resumable, manual run — do the history backfill first, then enable the nightly incremental timer (per the ops doc). Merging this is inert.