Treasury Yield Curve Data Pipeline

A Python pipeline for downloading, processing, and storing U.S. Treasury yield curve data from multiple sources.

Overview

This pipeline downloads yield curve data from the Federal Reserve (FRED) and NY Fed, processes it into standardized formats, and saves the output as Parquet files at multiple frequencies:

• Daily - All trading days
• Monthly - End of month values
• Quarterly - End of quarter values
• Quarterly Average - Daily averages over each quarter

Data Sources

St. Louis Fed (FRED)

Constant Maturity Treasury Yields:

• DGS1MO, DGS3MO, DGS6MO (1, 3, 6 month)
• DGS1, DGS2, DGS3, DGS5, DGS7, DGS10, DGS20, DGS30 (1-30 year)

Zero Coupon Yields:

• THREEFY1 through THREEFY10 (1-10 year fitted yields)

Other Series:

• DFF (Fed Funds Effective Rate)
• MORTGAGE30US (30-Year Fixed Rate Mortgage)
• GDPDEF (GDP Deflator)

NY Fed

ACM Term Premium:

• Risk-neutral yields and term premium decomposition from the Adrian, Crump, and Moench model

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/data_yields.git
   cd data_yields

2. Create and activate a virtual environment:

   python -m venv .venv
   .venv\Scripts\activate  # Windows
   source .venv/bin/activate  # macOS/Linux

3. Install dependencies:

   pip install -r requirements.txt

4. Set up your FRED API key:

   cp .env.example .env

   Edit .env and add your API key. Get one free at: https://fred.stlouisfed.org/docs/api/api_key.html

Usage

Run the full pipeline:

# Step 1: Download raw data from FRED and NY Fed
python 01_download.py

# Step 2: Process data and calculate forward rates
python 02_parse.py

# Step 3: Summarize and verify output files
python 03_summarize.py

Output Files

All processed files are saved to data/processed/ in Parquet format:

Constant Maturity Treasury Yields

File	Description
daily_yields.parquet	Daily yields and forward rates
monthly_yields.parquet	End-of-month yields
quarterly_yields.parquet	End-of-quarter yields
quarterly_avg_yields.parquet	Quarterly averages of daily data

Zero Coupon Yields

File	Description
daily_zero_coupon.parquet	Daily zero coupon yields and forward rates (1-10 year)
monthly_zero_coupon.parquet	End-of-month zero coupon yields and forward rates
quarterly_zero_coupon.parquet	End-of-quarter zero coupon yields and forward rates
quarterly_avg_zero_coupon.parquet	Quarterly averages of zero coupon yields and forward rates

ACM Risk-Neutral Yields

File	Description
daily_acm.parquet	Daily ACM risk-neutral yields and forward rates
monthly_acm.parquet	End-of-month ACM risk-neutral yields and forward rates
quarterly_acm.parquet	End-of-quarter ACM risk-neutral yields and forward rates
quarterly_avg_acm.parquet	Quarterly averages of ACM risk-neutral yields and forward rates

Data Dictionary

Yield Variables

All yields are stored as decimals (e.g., 0.05 = 5%).

Constant Maturity Yields

Variable	Description
y1m, y3m, y6m, y9m	Sub-annual yields
y1 through y30	Annual maturity yields
eff	Fed Funds Effective Rate

Zero Coupon Yields

Variable	Description
yzc1 through yzc10	Zero coupon yields (1-10 year maturities)

ACM Risk-Neutral Yields

Variable	Description
yrn3m, yrn6m, yrn9m	Sub-annual risk-neutral yields (interpolated)
yrn01 through yrn10	Risk-neutral yields (1-10 year maturities)
yrn20, yrn30	Long-term risk-neutral yields (extrapolated using CMT ratios)

Forward Rate Variables

All forward rates are 1-year forward rates unless otherwise noted.

Constant Maturity Forward Rates

Variable	Description
f1, f2, f3, f4, f5	1-year forward rates starting in years 0-4
f4_5	2-year forward starting in year 3
f5_10	5-year forward starting in year 5
f10_20	10-year forward starting in year 10
f10_30	20-year forward starting in year 10
f20_30	10-year forward starting in year 20

Zero Coupon Forward Rates

Variable	Description
fzc1 through fzc10	1-year forward rates starting in years 0-9
fzc5_10	5-year forward starting in year 5

ACM Risk-Neutral Forward Rates

Variable	Description
frn01 through frn10	1-year forward rates starting in years 0-9
frn5_10	5-year forward starting in year 5
frn10_20	10-year forward starting in year 10
frn20_30	10-year forward starting in year 20

Configuration

Edit config.py to modify:

• START_DATE - Data start date (default: 1980-01-01)
• MAX_DATE - Data end date (default: 2025Q2)
• Directory paths for raw and processed data

Project Structure

data_yields/
├── 01_download.py      # Download data from FRED and NY Fed
├── 02_parse.py         # Process and calculate forward rates
├── 03_summarize.py     # Verify and summarize output
├── config.py           # Configuration settings
├── requirements.txt    # Python dependencies
├── .env.example        # API key template
├── .gitignore
├── README.md
└── data/
    ├── raw/            # Downloaded raw data
    └── processed/      # Processed parquet files