feat(data): add report-only data quality checks (D3) - #45
Merged
Conversation
A small, pure, report-only data-quality layer that flags suspicious upstream
daily-market / adj_factor / 1min-intraday data near ingestion. It NEVER mutates
data, filters rows, repairs values, alters cache coverage, or touches qfq /
factor / alpha / portfolio / runtime logic — it only surfaces findings.
New package data/quality/:
- report.py: QualityFinding (immutable) + make_finding (bounds examples to 5,
cleans values), findings_to_frame (stable, deterministically ordered),
render_report (deterministic Markdown; "no findings" on clean), has_hard,
clean_value. Findings carry only dataset/check metadata + bounded
{date/time, symbol, value} examples — no token, no secret path, no unbounded dump.
- market.py: daily checks — duplicate (date,symbol), non-positive OHLC, high<low,
close outside [low,high], negative volume/amount, invalid adj_factor (hard);
extreme close moves (|pct_change|>threshold) and missing dates vs an explicit
calendar (warning). run_market_checks / run_adj_factor_checks orchestrators.
- intraday.py: 1min checks — duplicate bars, non-monotonic timestamps,
non-positive OHLC, high<low, close outside range, negative volume/amount (hard);
missing minutes vs an explicit calendar (warning). run_intraday_checks.
- _frames.py: pure index-normalization + bounded example extraction helpers.
Pure functions: input DataFrame (raw cache-shaped columns or MultiIndex panel) ->
findings; inputs are never mutated. Clean input yields zero hard findings.
Tests (tests/test_data_quality_{market,intraday,report}.py): each daily/intraday
issue is caught; clean panels produce zero hard findings; examples are bounded
(<=5) and deterministic; rendering is deterministic and asserts no secret-looking
paths in output.
Integration deferred to D3b: this is the library + tests only; no data-update
hook, no config change, no extra Tushare calls. Non-goals (unchanged): cache
semantics, CoverageLedger storage, concurrency, schema registry, PanelStore,
factor/alpha/portfolio/runtime/fills/OOS/analytics math.
…ew fix) Codex review found a contract gap: render_report() (and stored findings) could emit a secret-looking config path or token key if a caller passed one in a finding note or example value — the checks never do, but the public report API did not defend against it. Add a small, deterministic report-safety guard in data/quality/report.py: - sanitize_text() redacts -> "[REDACTED]": any path ending in .config.json, the literal tushare.token, and obvious token=.../token: ... values. Idempotent; benign values (symbols, ISO dates, numbers) are untouched. - applied at the three string entry points: clean_value (string example values), make_finding (note), and _format_examples (rendered output, so a secret-looking example KEY is covered too). Update tests/test_data_quality_report.py to pass secret-looking inputs (a full .config.json path + tushare.token in example values AND in note) and assert they are redacted from both the stored finding and the rendered report, while benign symbols/ISO dates still render and examples stay bounded to 5. Report-only and scope-tight: no data-update, config, cache/ledger, feed, factor/ alpha/portfolio/runtime, or PanelStore change; D3 stays library + tests only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Data Layer D3 — a small, pure, report-only data-quality layer that flags
suspicious upstream daily-market / adj_factor / 1min-intraday data near ingestion.
It never mutates data, filters rows, repairs values, alters cache coverage,
changes qfq/front-adjust, or touches factor/alpha/portfolio/runtime — it only
surfaces findings. No cache / feed / backtest semantics changed.
This PR is the library + tests only; the optional
data-updateintegration isdeferred to D3b (the goal sanctions deferral) — no config change, no extra
Tushare calls.
Changed files (8, all new)
data/quality/__init__.py,data/quality/report.py,data/quality/market.py,data/quality/intraday.py,data/quality/_frames.pytests/test_data_quality_market.py,tests/test_data_quality_intraday.py,tests/test_data_quality_report.pyNo other file is touched.
Quality checks implemented
Daily market / adj_factor (
market.py): duplicate(date,symbol),non-positive OHLC,
high<low,closeoutside[low,high], negativevolume/amount, invalidadj_factor→ hard; extremecloseday-over-daymove (
|pct_change|>threshold, default 0.5) and missing dates vs an explicitcalendar → warning.
1min intraday (
intraday.py): duplicate bars(bar_end/time,symbol),non-monotonic timestamps, non-positive OHLC,
high<low,closeoutside range,negative
volume/amount→ hard; missing minutes vs an explicit calendar →warning.
Reporting (
report.py): immutableQualityFinding;make_findingboundsexamples to 5 and cleans values;
findings_to_frame(stable, deterministicallyordered, hard-first);
render_report(deterministic Markdown, explicit "nofindings" on clean input);
has_hard. Findings carry only dataset/check metadata{date/time, symbol, value}examples.Report-only confirmation
DataFrame→ findings; inputs never mutated.backtest. No integration hook in this PR (deferred to D3b, default-off when added).
Verification
pytest tests/test_data_quality_market.py tests/test_data_quality_intraday.py tests/test_data_quality_report.py→ all pass (34 tests).pytest -p no:cacheprovider→ 618 passed (584 prior + 34 new).ruff check .→ All checks passed!validate-configover all 17config/*.yaml→ all OK (no config added).run-phase0 --config config/example.yaml→ic_mean=0.9600, annual_return=0.8408(no behavior drift).git diff --check main...HEAD→ clean; merge-marker scan → 0.rg "TushareCache\(|CoverageLedger\(|PanelStore|simulate_fills|RollingICWeightAlpha|EqualWeightAlpha" data/quality tests) → 0 hits — the quality layer references no runtime/alpha/portfolio/cache internals.Secret scan
.config.jsontoken value over changed files → 0 hits..config.json/tushare.tokenliterals in the diff are thenegative-assertion scan targets in
test_data_quality_report.py::test_render_has_no_secret_looking_paths(thegoal-required test that asserts the renderer does not emit secret-looking
paths) — not leaked data, not in any rendered report/log.
Please review before merge.