daily_append: hard-fail on missing macro keys + verify writes landed#37
Merged
Merged
Conversation
Root-cause fix for the silent-fail that let 2026-04-15 weekday pipeline runs report SUCCEEDED while macro/SPY sat 5 days stale (4/10 → 4/15) until the inference Lambda preflight finally caught it via the PR #28 deploy canary. ## The silent-fail Old macro update loop: for key in macro_keys: bar = closes.get(key) if bar and not np.isnan(...): macro_lib.update(key, new_row) If `closes.get(key)` returned None (upstream daily_closes collection gap for that ticker), the update silently skipped. No log. No raise. daily_append returned status="ok". weekly_collector returned status="ok". Python exited 0. Step Function marked SUCCEEDED. macro never updated. Combined with the "all stocks skipped because backfill wrote today's row" case, an entire daily_append invocation could produce ZERO writes and still report success. ## The fix 1. Track macro_missing_from_closes list. Any key absent from closes (or with NaN Close) goes in the list instead of being silently skipped. 2. After both loops, if the list is non-empty, raise RuntimeError naming the missing keys. Macro inputs are not optional — SPY feeds return_vs_spy_5d, VIX feeds vix_level, sector ETFs feed sector-relative features. Upstream collection must produce them. 3. Sector ETF iteration now uses an explicit ["XLB","XLC",...] list rather than filtering closes.keys() with startswith("XL"). Missing sector ETFs now surface as rejects instead of silently-not-iterating. 4. After update() calls complete, read back every updated key and verify its last date matches today. A pure belt-and-suspenders check — update() has no return value to confirm success, so a silent no-op in ArcticDB's commit layer could still slip through the no-exception path without this readback. If verification fails, raise with the specific (key, reason) tuples. 5. Summary log line now surfaces macro_updated + sector_updated counts alongside the stock counts, so "ok with zero writes" is visible in the logs instead of hidden behind stock-only metrics. ## Test plan tests/test_daily_append_semantics.py — 3 new source-level regression tests locking the hard-fail semantics: - Missing macro keys raise - Verification readback exists - Sector ETFs iterate explicit list Full suite: 46 passed (was 43 + 3 new). ## Follow-up Investigate WHY the upstream daily_closes collector was missing macro keys in the 2026-04-15 08:39 PT rerun despite loading 920 tickers with SPY/VIX/etc verified present in the parquet. The missing-keys branch should not trigger in a healthy run — its purpose is to loudly halt the pipeline when upstream collection is broken, so we find out and fix it instead of running on stale data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-cause fix for the silent-fail that let 2026-04-15 weekday pipeline runs report SUCCEEDED while macro/SPY sat 5 days stale (4/10 → 4/15) — the scenario that today's deploy canary finally surfaced after 5 days of silent daily failures.
Before: If `closes.get(key)` returned None for a macro ticker, `daily_append` silently skipped the update. Combined with "all stocks skipped because backfill wrote today's row," the pipeline could produce ZERO writes and still return status="ok".
After:
Test plan
Related
🤖 Generated with Claude Code