feat(daily_closes): TWO + HYOAS forward-only daily ingestion (Stage 2.5)#203
Merged
Conversation
Stage 2.5 of the regime-conditioning rebuild (plan doc:
alpha-engine-docs/private/regime-conditioning-260510.md). Adds two
FRED series to the daily_closes ingestion map:
- TWO → DGS2 (2Y constant-maturity treasury, percent)
- HYOAS → BAMLH0A0HYM2 (ICE BofA US HY Index Option-Adjusted Spread,
percent)
Why this matters: the predictor's macro feature set currently includes
yield curve slope as 10Y-3M (TNX-IRX), which is the cyclical curve.
Adding DGS2 enables the 10Y-2Y curve (recession-focused canonical) as
a parallel macro. HY OAS is a major regime indicator that VIX misses —
credit typically widens before vol spikes in macro-driven cycles, and
stays wide during recoveries when vol has already calmed. Real
institutional risk-factor models (AQR, Two Sigma, BlackRock Aladdin)
include credit spreads in their regime taxonomy.
Scope discipline: this PR is forward-only daily collection. Historical
backfill (which requires a new FRED history fetcher — current
``_fetch_fred_closes`` is single-latest-only) is gated as Stage 2.5b.
Predictor-side wiring (extending compute_features + regime_predictor +
MACRO_NORM_FEATURES with the new symbols) is gated as Stage 2c.
What ships:
- _FRED_INDEX_MAP: TWO + HYOAS entries
- config.yaml.example: TWO + HYOAS in always_download
- 5 tests locking the new mappings + the no-caret-for-FRED-only contract
Operator follow-up:
- Update production config.yaml to include TWO + HYOAS in always_download
(config.yaml is gitignored; this PR only updates the example file).
- Stage 2.5b PR will add FRED history fetcher + run historical backfill
before Stage 2c (predictor consumption) lands. Without that, predictor
training has no historical TWO/HYOAS data to learn from — only
forward-collected rows from this PR's deploy date onward.
633 tests pass + 1 skipped (5 new + 628 existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
9 tasks
cipher813
added a commit
that referenced
this pull request
May 10, 2026
* feat(daily_closes): TWO + HYOAS forward-only daily ingestion (Stage 2.5)
Stage 2.5 of the regime-conditioning rebuild (plan doc:
alpha-engine-docs/private/regime-conditioning-260510.md). Adds two
FRED series to the daily_closes ingestion map:
- TWO → DGS2 (2Y constant-maturity treasury, percent)
- HYOAS → BAMLH0A0HYM2 (ICE BofA US HY Index Option-Adjusted Spread,
percent)
Why this matters: the predictor's macro feature set currently includes
yield curve slope as 10Y-3M (TNX-IRX), which is the cyclical curve.
Adding DGS2 enables the 10Y-2Y curve (recession-focused canonical) as
a parallel macro. HY OAS is a major regime indicator that VIX misses —
credit typically widens before vol spikes in macro-driven cycles, and
stays wide during recoveries when vol has already calmed. Real
institutional risk-factor models (AQR, Two Sigma, BlackRock Aladdin)
include credit spreads in their regime taxonomy.
Scope discipline: this PR is forward-only daily collection. Historical
backfill (which requires a new FRED history fetcher — current
``_fetch_fred_closes`` is single-latest-only) is gated as Stage 2.5b.
Predictor-side wiring (extending compute_features + regime_predictor +
MACRO_NORM_FEATURES with the new symbols) is gated as Stage 2c.
What ships:
- _FRED_INDEX_MAP: TWO + HYOAS entries
- config.yaml.example: TWO + HYOAS in always_download
- 5 tests locking the new mappings + the no-caret-for-FRED-only contract
Operator follow-up:
- Update production config.yaml to include TWO + HYOAS in always_download
(config.yaml is gitignored; this PR only updates the example file).
- Stage 2.5b PR will add FRED history fetcher + run historical backfill
before Stage 2c (predictor consumption) lands. Without that, predictor
training has no historical TWO/HYOAS data to learn from — only
forward-collected rows from this PR's deploy date onward.
633 tests pass + 1 skipped (5 new + 628 existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(collectors): FRED date-range history fetcher (Stage 2.5b)
Stage 2.5b of the regime-conditioning rebuild (plan doc:
alpha-engine-docs/private/regime-conditioning-260510.md). Adds the
historical-fetch counterpart to ``daily_closes._fetch_fred_closes``
(single-latest-only) so FRED-only macro symbols (TWO, HYOAS, and any
future FRED series) can populate the 10-year price-cache parquets the
predictor reads.
Without this, Stage 2.5's daily ingestion of TWO + HYOAS only collects
data going forward — predictor training has no historical data to learn
from. This module unblocks Stage 2c-full (10Y-2Y curve + HY OAS macro
features in regime_predictor.build_features).
What ships:
- ``collectors/fred_history.py``:
- ``FRED_HISTORY_MAP``: TWO → DGS2, HYOAS → BAMLH0A0HYM2
- ``fetch_fred_history(series_id, period_years=10)``: date-range FRED
API call with retries + missing-value handling
- ``fred_history_to_ohlcv(df_fred)``: schema reshape so output
matches the parquet contract yfinance produces (Open/High/Low/Close
/Adj_Close/Volume/VWAP/source). FRED single-value → OHLC replicate,
Volume=0, VWAP=None, source="fred"
- ``backfill_to_s3(bucket, tickers, dry_run)``: orchestrator —
fetches each ticker, reshapes, writes parquet, uploads to S3.
Per-ticker error isolation; partial completion reported via
``status="partial"`` rather than raising
- CLI entry point: ``python -m collectors.fred_history --bucket ... --dry-run``
Tests:
- 20 new tests covering: FRED_HISTORY_MAP contract, fetch parse +
retries + missing handling + no-API-key error, OHLCV schema
invariant, backfill orchestration including dry-run, unknown-ticker
validation, per-ticker error isolation, S3 upload path.
Operator follow-up (post-merge, after Stage 2.5 PR #203 also lands):
- Run ``python -m collectors.fred_history --bucket alpha-engine-research``
to seed historical TWO.parquet + HYOAS.parquet in
s3://alpha-engine-research/predictor/price_cache/.
- Verify with ``aws s3 ls s3://alpha-engine-research/predictor/price_cache/ | grep -E "(TWO|HYOAS)"``.
- After backfill completes, Stage 2c-full can ship the predictor-side
consumption (yield_curve_10y_2y, hy_oas_level, hy_oas_change_21d).
Stacks on Stage 2.5 (alpha-engine-data PR #203). Merge order:
#203 first, then this PR. Both can also merge in either order — they
touch different files.
653 tests pass + 1 skipped (20 new + 633 existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
daily_closes._FRED_INDEX_MAPfor forward-only daily ingestionTWO → DGS2(2Y constant-maturity treasury, percent) — enables 10Y-2Y curve slope (recession-focused canonical)HYOAS → BAMLH0A0HYM2(HY OAS credit spread, percent) — major regime indicator VIX misses_fetch_fred_closesis single-latest-only)Plan reference
Plan doc:
~/Development/alpha-engine-docs/private/regime-conditioning-260510.md(gitignored). ROADMAP entry in alpha-engine-config #103.Why credit spreads matter
The predictor's current macro set captures vol regime (VIX, vix_term_slope) and rates regime (10Y-3M curve via TNX-IRX), but is silent on credit regime. Credit typically widens before vol spikes in macro-driven cycles (HY OAS leads VIX), and stays wide during recoveries when vol has already calmed. AQR, Two Sigma, BlackRock Aladdin all include credit in their regime taxonomy. The 2Y treasury enables 10Y-2Y curve (recession-canonical) alongside the existing 10Y-3M (cyclical).
Stage decomposition
compute_features+regime_predictor.build_featuresto consume the new series; addyield_curve_10y_2y,hy_oas_level,hy_oas_change_21dtoMACRO_NORM_FEATURES. Also addsmarket_breadth_200d(alongside existing 50d) andvix_vix3m_ratio(alongside existing slope) — predictor-side macro additions queued from the original Stage 2a-extended discussion.What ships in this PR
_FRED_INDEX_MAPextended with TWO + HYOASconfig.yaml.examplealways_downloadlist extended (config.yaml is gitignored — operator must update production config separately, see follow-up below)Test plan
Operator follow-up (post-merge)
config.yamlto include TWO + HYOAS inalways_download(config.yaml is gitignored — this PR only touches the example).🤖 Generated with Claude Code