feat(quality): flag adj_factor that decreases between consecutive dates - #77
Merged
Conversation
tushare's cumulative adjustment factor grows with each corporate action; it has no mechanism to shrink. front_adjust multiplies raw prices by it, so a published decrease propagates straight into every downstream return, and nothing in the quality layer looked for one. A real instance was found by hand while auditing daily ex-dividend correctness: 920627.BJ oscillates between two factor values, giving alternating -57.27% / +134.04% steps. Its +134% tail was briefly mistaken for a genuine corporate-action distribution before being traced to a single defective series. Across the full 5562-symbol cache only five symbols carry any change worse than -2% (920627.BJ 171 rows, 920130.BJ 2, 000545.SZ 1, 002898.SZ 1, 000998.SZ 1), and exactly one of them -- 000998.SZ, 2023-06-01, -2.64% with no matching raw-price move -- sits in an evaluated universe. Nothing was restated, but that was luck: no check would have named these. The threshold is deliberately 2%, not zero. tushare rounds the published factor, so ex-dates routinely carry basis-point negative steps: measured over CSI500 2021-07..2026-06, 1264 of 6932 factor changes (18.23%) are negative and essentially all lie between -0.0008% and -0.08%. Those are rounding, not corruption, and flagging them would bury the real defect in noise. WARNING severity, report-only: it never filters a panel, repairs a value, or touches front_adjust or cache coverage -- consistent with the rest of D3. Verified against the real cache: run_adj_factor_checks over the three defective shards plus two clean ones returns exactly one finding, count=174, with the clean names contributing nothing. Mutation evidence, each run: threshold 0.02 -> 0.0 => test_rounding_scale_negative_steps_are_not_flagged FAILS per-symbol grouping removed => test_..._does_not_bleed_across_symbols FAILS date sort removed => test_..._sorts_by_date_before_differencing FAILS
…he sweep
Review made two points that both hold up, and answering them moved the
threshold.
First, the docstring justified threshold != 0 with the CSI500 rounding-noise
band (-0.0008%..-0.08%), which on its own argues for something far TIGHTER than
2% and left the actual choice unexplained. The load-bearing evidence is a sweep
over every negative step in all 5562 cached symbols (7898 events, 4479 symbols),
now recorded in the docstring:
threshold events fired symbols fired that are NOT known-defective
0.01% 3653 2429 <- pure float noise
0.10% 179 2
0.50% 178 1 (603081.SH)
1.00% 177 0
2.00% 176 0
That puts the cut at 1%, not 2%. Below 1% the check names symbols whose status
cannot be decided -- 603081.SH's worst step is -0.776% against a raw close move
of -1.810% where a genuine action needs +0.782%, which has the SHAPE of a defect,
but at that magnitude an ordinary daily move swamps the implied one, so the price
test has no power and the finding would be uninterpretable. Above 1% a real event
is missed: 1% fires 177 events, 2% only 176, and the extra one belongs to a
known-defective symbol.
1% also leaves the smallest confirmed defect (000998.SZ, -2.636%) at 2.6x the bar
rather than 1.3x. Erring toward catching defects is right for a report-only
layer: a false positive costs one WARNING line, a false negative lets corrupted
prices through silently.
Verified on the real cache at the new threshold: 177 findings across exactly the
five known-defective symbols; 603081.SH, 600519.SH and 000001.SZ contribute none.
Second, review demonstrated by running it that swapping pct_change() for diff()
-- relative to absolute -- passed all six original tests, because every fixture
sat at factor ~1-11 where the two cuts coincide. Two fixtures at other scales
close that: 0.200 -> 0.195 is a real -2.5% break that an absolute cut misses
(-0.005 inside a 0.01 bar), and 500.0 -> 499.97 is -0.006% rounding that an
absolute cut wrongly fires on (-0.03 clears it). Both now die under the mutation;
they did not before.
Also records why this WARNING differs from check_extreme_returns': a >50% close
move really can be legitimate, whereas a cumulative factor has no way to shrink
at all. The severity reflects calibration risk at the threshold, not doubt about
the claim.
pytest 1736 passed (1734 + 2), ruff clean, phase0 unchanged.
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The user made daily ex-dividend backtest correctness the top priority. Auditing it surfaced a defect class the quality layer had no check for.
tushare's cumulative adjustment factor grows with each corporate action — it has no mechanism to shrink.
front_adjustmultiplies raw prices by it, so a published decrease propagates straight into every downstream return.A real instance existed and was found by hand:
920627.BJoscillates between two factor values, giving alternating −57.27% / +134.04% steps (1/(1−0.572723)−1 = +134.04%— the two tails are the same event). I briefly mistook that +134% tail for a genuine corporate-action distribution before tracing it to one defective series. A frame-level check would have named it immediately.Scope of the defect in the real cache
Full 5562-symbol cache, date-sorted and deduplicated. Only five symbols carry any change worse than −2%:
920627.BJ920130.BJ000545.SZ002898.SZ000998.SZThe one in-universe case is
000998.SZ, 2023-06-01: factor 11.267 → 10.970 (−2.64%) with the raw close moving +1.228% — no matching price discontinuity, and not a cache fetch boundary, so it is a defect in what tushare publishes rather than a caching artifact. Exposure is 1 pair out of ≈1.2M in the CSI500 window, so nothing is restated. But that was luck, not detection.Full investigation:
tmp/design/FINDING_adj_factor_seam_2026-07-21.md(the vintage-seam hypothesis was tested and rejected — 960 adjacent-day pairs straddle a fetch boundary, only 12 show any factor change, all positive, all on the annual ex-dividend peak, all with matching raw-price discontinuities).Why the threshold is 2% and not zero
tushare rounds the published factor, so ex-dates routinely carry basis-point negative steps. Measured over CSI500 2021-07 → 2026-06: 1264 of 6932 factor changes (18.23%) are negative, and essentially all lie between −0.0008% and −0.08%. Those are rounding, not corruption. A zero threshold would emit 1264 findings and bury the five real ones.
Behaviour
WARNING, report-only — consistent with the rest of D3. It never filters a panel, repairs a value, or touchesfront_adjustor cache coverage. Wired intorun_adj_factor_checksalongside the existing duplicate-key and positivity checks.Verification
Against the real cache (three defective shards + two clean ones, 7191 rows):
174 matches the hand scan exactly (176 across all five symbols, minus the two not loaded here). The clean names contribute nothing.
Mutation evidence (each actually run):
threshold0.02 → 0.0test_rounding_scale_negative_steps_are_not_flaggedtest_decreasing_adj_factor_does_not_bleed_across_symbolstest_decreasing_adj_factor_sorts_by_date_before_differencingGates
pytest1734 passed (1728 baseline + 6) ·ruffclean · phase0ic_mean=0.9600, annual_return=0.8408· secret scan 0