feat(collectors): FRED date-range history fetcher (Stage 2.5b)#204
Merged
Conversation
Stage 2.5 of the regime-conditioning rebuild (plan doc:
alpha-engine-docs/private/regime-conditioning-260510.md). Adds two
FRED series to the daily_closes ingestion map:
- TWO → DGS2 (2Y constant-maturity treasury, percent)
- HYOAS → BAMLH0A0HYM2 (ICE BofA US HY Index Option-Adjusted Spread,
percent)
Why this matters: the predictor's macro feature set currently includes
yield curve slope as 10Y-3M (TNX-IRX), which is the cyclical curve.
Adding DGS2 enables the 10Y-2Y curve (recession-focused canonical) as
a parallel macro. HY OAS is a major regime indicator that VIX misses —
credit typically widens before vol spikes in macro-driven cycles, and
stays wide during recoveries when vol has already calmed. Real
institutional risk-factor models (AQR, Two Sigma, BlackRock Aladdin)
include credit spreads in their regime taxonomy.
Scope discipline: this PR is forward-only daily collection. Historical
backfill (which requires a new FRED history fetcher — current
``_fetch_fred_closes`` is single-latest-only) is gated as Stage 2.5b.
Predictor-side wiring (extending compute_features + regime_predictor +
MACRO_NORM_FEATURES with the new symbols) is gated as Stage 2c.
What ships:
- _FRED_INDEX_MAP: TWO + HYOAS entries
- config.yaml.example: TWO + HYOAS in always_download
- 5 tests locking the new mappings + the no-caret-for-FRED-only contract
Operator follow-up:
- Update production config.yaml to include TWO + HYOAS in always_download
(config.yaml is gitignored; this PR only updates the example file).
- Stage 2.5b PR will add FRED history fetcher + run historical backfill
before Stage 2c (predictor consumption) lands. Without that, predictor
training has no historical TWO/HYOAS data to learn from — only
forward-collected rows from this PR's deploy date onward.
633 tests pass + 1 skipped (5 new + 628 existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage 2.5b of the regime-conditioning rebuild (plan doc:
alpha-engine-docs/private/regime-conditioning-260510.md). Adds the
historical-fetch counterpart to ``daily_closes._fetch_fred_closes``
(single-latest-only) so FRED-only macro symbols (TWO, HYOAS, and any
future FRED series) can populate the 10-year price-cache parquets the
predictor reads.
Without this, Stage 2.5's daily ingestion of TWO + HYOAS only collects
data going forward — predictor training has no historical data to learn
from. This module unblocks Stage 2c-full (10Y-2Y curve + HY OAS macro
features in regime_predictor.build_features).
What ships:
- ``collectors/fred_history.py``:
- ``FRED_HISTORY_MAP``: TWO → DGS2, HYOAS → BAMLH0A0HYM2
- ``fetch_fred_history(series_id, period_years=10)``: date-range FRED
API call with retries + missing-value handling
- ``fred_history_to_ohlcv(df_fred)``: schema reshape so output
matches the parquet contract yfinance produces (Open/High/Low/Close
/Adj_Close/Volume/VWAP/source). FRED single-value → OHLC replicate,
Volume=0, VWAP=None, source="fred"
- ``backfill_to_s3(bucket, tickers, dry_run)``: orchestrator —
fetches each ticker, reshapes, writes parquet, uploads to S3.
Per-ticker error isolation; partial completion reported via
``status="partial"`` rather than raising
- CLI entry point: ``python -m collectors.fred_history --bucket ... --dry-run``
Tests:
- 20 new tests covering: FRED_HISTORY_MAP contract, fetch parse +
retries + missing handling + no-API-key error, OHLCV schema
invariant, backfill orchestration including dry-run, unknown-ticker
validation, per-ticker error isolation, S3 upload path.
Operator follow-up (post-merge, after Stage 2.5 PR #203 also lands):
- Run ``python -m collectors.fred_history --bucket alpha-engine-research``
to seed historical TWO.parquet + HYOAS.parquet in
s3://alpha-engine-research/predictor/price_cache/.
- Verify with ``aws s3 ls s3://alpha-engine-research/predictor/price_cache/ | grep -E "(TWO|HYOAS)"``.
- After backfill completes, Stage 2c-full can ship the predictor-side
consumption (yield_curve_10y_2y, hy_oas_level, hy_oas_change_21d).
Stacks on Stage 2.5 (alpha-engine-data PR #203). Merge order:
#203 first, then this PR. Both can also merge in either order — they
touch different files.
653 tests pass + 1 skipped (20 new + 633 existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
daily_closes._fetch_fred_closes(single-latest-only)python -m collectors.fred_history --bucket alpha-engine-researchPlan reference
Plan doc:
~/Development/alpha-engine-docs/private/regime-conditioning-260510.md(gitignored). ROADMAP entry in alpha-engine-config #103.Stacks on Stage 2.5
This PR is branched off
feat/dgs2-hyoas-ingestion-stage-2-5(PR #203). Either merge order works — the PRs touch different files (Stage 2.5 modifiesdaily_closes._FRED_INDEX_MAPfor daily fetch; Stage 2.5b adds a newcollectors/fred_history.pymodule for backfill).What ships
collectors/fred_history.py:FRED_HISTORY_MAP:TWO → DGS2,HYOAS → BAMLH0A0HYM2fetch_fred_history(series_id, period_years=10)— date-range FRED API call with 3-attempt retry + missing-value handlingfred_history_to_ohlcv(df_fred)— reshape FRED single-value series to the OHLCV-shape parquet contract yfinance produces. FRED value → OHLC replicate, Volume=0, VWAP=None, source="fred"backfill_to_s3(bucket, tickers, dry_run)— orchestrator with per-ticker error isolationStage sequencing
prod_vol_risk_augGBM) — independent track, can run anytime after feat(features): per-ticker risk features (Stage 2a regime-conditioning rebuild) #202Test plan
Operator follow-up (post-merge)
After both #203 and this PR merge:
python -m collectors.fred_history --bucket alpha-engine-research # Or first verify: python -m collectors.fred_history --bucket alpha-engine-research --dry-runVerify the backfilled parquets:
Then Stage 2c-full can ship the predictor-side consumption (
yield_curve_10y_2y,hy_oas_level,hy_oas_change_21dinregime_predictor.build_features+cfg.MACRO_NORM_FEATURES).🤖 Generated with Claude Code