fix(polygon): per-ticker fallback for grouped-daily coverage gaps#131
Merged
Conversation
Polygon's grouped-daily endpoint returns inconsistent ticker subsets across calls. Observed 2026-05-02: two calls 4h apart against the same universe returned different 913-ticker subsets that differed by 8 real S&P 500/400 names (ASGN, GTM, HOLX, KMPR, LW, MOH, MTCH, PAYC). This tipped MorningEnrich's missing-from-closes hard-fail (12 missing > threshold of 5) and halted the Saturday SF a second time today. Fix: when the bulk grouped response is missing a requested stock ticker, fall through to polygon's per-ticker /aggs/ticker endpoint for that single ticker. Same source (polygon), same response schema, no silent yfinance fallback — the no-silent-fails contract is preserved. Each fallback hit costs one rate-limit slot; with the default 5 calls/min and ~10-15 typical grouped misses, MorningEnrich gains 2-3 minutes versus the prior <1 second hot path. Well within the SF DataPhase1 budget. FRED-handled indices (^TNX/^IRX/^VIX/^VIX3M) are excluded from the fallback loop — polygon has no coverage for them, the per-ticker call would just burn rate-limit slots returning nothing. Two layers tested in tests/test_polygon_per_ticker_fallback.py (12 new tests, 355 total): 1. PolygonClient.get_single_day_bar — happy path / no-data / 403 / missing-vwap. 2. _fetch_polygon_closes_per_ticker — recovery success / caret-prefix stripping / per-call exception isolation. 3. _fetch_polygon_closes integration — invokes fallback on grouped miss, skips for FRED indices, no fallback when grouped is complete (cost-quiet path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
cipher813
added a commit
that referenced
this pull request
May 2, 2026
* feat(preflight): add sf_preflight.py — Saturday SF dry-rehearsal Predicts whether the Saturday SF would succeed BEFORE launching a spot. Today's recovery cycle (5 SF redrives, ~5 polygon API calls each) burned free-tier quota and operator hours discovering bugs sequentially. This module simulates the critical pre-Phase-1 path against real S3 + ArcticDB state and reports per-step pass/fail in ~30s with 1 polygon call total. Eight independent checks, mapped to today's incident stack: PR #130 (backfill regression) → check_backfill_source_freshness PR #131 (polygon coverage flake) → check_polygon_grouped_coverage PR #132 (missing-from-closes scoping) → check_predicted_missing_from_closes PR #133 (freshness scan scoping) → check_universe_sample_freshness PR #134 (workflow ordering) → check_universe_drift PR #135 (return shape) → check_constituents_fetch Postflight contracts → check_postflight_contracts ArcticDB reachability → check_arctic_connectivity Each check is a pure function taking a PreflightContext, returning a CheckResult. The orchestrator runs them all (catching per-check exceptions so one fail doesn't abort the suite) and emits human or JSON output. Exit code 1 on any failure. Two macOS-specific design notes: 1. ArcticDB libs are initialized once in check_arctic_connectivity and reused across downstream checks via the context — re-initializing adb.Arctic() crashes Aws::S3::S3Client::S3Client on macOS. 2. Checks are ordered with arctic_connectivity FIRST so its bundled AWS SDK loads before boto3 (which gets pulled in by collectors imports). Polygon check skips gracefully (WARN, not FAIL) when POLYGON_API_KEY is unset — supports laptop-side preflight where the .env isn't loaded. On the spot the key is present and the check fires. 18 tests in tests/test_sf_preflight.py — happy path + each failure mode each check is designed to catch + orchestrator isolation. 394 tests total. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(sf_preflight): set POLYGON_API_KEY in polygon-coverage tests CI runs without POLYGON_API_KEY in env, so the no-key skip-to-WARN guard short-circuited the 3 polygon-coverage tests before they reached the mocked client. Set the env var via monkeypatch so the guard passes through to the polygon mock. Also add explicit test for the no-key path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/aggs/tickerendpoint for tickers the grouped response dropped. Same source (polygon), no silent yfinance fallback —feedback_no_silent_failspreserved.Test plan
tests/test_polygon_per_ticker_fallback.py— 12 new tests covering:get_single_day_bar(happy / no-data / 403 / missing-vwap);_fetch_polygon_closes_per_ticker(recovery / caret-prefix stripping / exception isolation);_fetch_polygon_closesintegration (invokes fallback on grouped miss / skips FRED indices / no fallback when grouped complete)🤖 Generated with Claude Code