us-market-data

If your use case requires high quality data, do not trust the data in this repo! The dataset is only good for crude buy-and-hold backtests. See the warnings section.

Provides daily U.S. market data going back to 1885. data/us_market_data.csv contains the data that gets updated to the present when the Jupyter Notebook or Python script are ran.

data/libor.csv contains old LIBOR data that is used in constructing the us_market_data.csv. This is not downloadable from the web anymore, hence why it is in the repo.

Dividends are added to price-index only data to approximate SPY as closely as possible. As such, by default, a 0.0945% expense ratio is applied throughout (SPY expense ratio as of January 27, 2024). Additionally, SPY adjusted close data is used to fill in dates since June 16, 2023.

The following data is contained in us_market_data.csv:

• Close - Close data without dividends or inflation adjustment
• Adjusted Close - Close data with added dividends in the same manner expected of SPY
• Real Return - Close data without dividends, adjusted for inflation
• Total Real Return - Close data with added dividends and adjusted for inflation
• Risk Free Rate - Risk free rate as a percentage
• Risk Free Return - The close prices that the risk free rate would give as an investment
• CPI - Consumer Price Index, All Urban Consumers
• Inflation - The annual inflation rate based on the CPI
• Swap Rate - The swap rate used by LETFs

Warnings (Important)

CPI/inflation data is monthly and interpolated to daily. Additionally, data in the most recent month will not be accurate for any inflation-related columns (Real Return, Total Real Return, CPI, and Inflation) since the FRED series used (CPIAUCNS) is monthly.

Stock market data from March 20, 1885 to December 30, 1927 is a Dow Jones composite portfolio, not the S&P 500. Additionally, adjusted close data from this period will be less accurate since dividend data was annual instead of quarterly.

Prior to 1952, the stock market was open on Saturdays. These dates are dropped from us_market_data.csv, but their effects on returns are still included.

Risk free rate data before July 1, 1926 is very crudely approximated as the 10-year treasury yield minus 1%, interpolated from monthly to daily. Additionally, data from July 1, 1926 to December 31, 1953 is 1-month T-Bills instead of the standard 3-month T-Bills. Swap rate data is exact same as the risk free rate data up to December 31, 1953.

Columbus/Indigenous People's Day and Veteran's Day are recognized by the Fed, but not by stock exchange, so some columns get interpolated even when the underlying data is daily.

There may be small differences in close data between runs due to floating point imprecision in yfinance (see ranaroussi/yfinance#1038).

---

Getting Started

This project uses uv for dependency management. If you don't have it installed, you can find instructions here.

Running the Python script

The Python script only updates us_market_data.csv. Use the Jupyter Notebook to fully reconstruct the dataset.

git clone https://github.com/SteelCerberus/us-market-data.git
cd us-market-data
uv run src/update_data.py

Running through Jupyter Lab

git clone https://github.com/SteelCerberus/us-market-data.git
cd us-market-data
uv run jupyter lab

---

Data

Stock Close Data Sources:

March 20, 1885 to December 30, 1927:

Dow Jones composite portfolio (industrial and railroad stocks) with dividends added

Dow Jones data source: https://www.billschwert.com/dstock.htm

Dividend data source: http://www.econ.yale.edu/~shiller/data.htm

January 3, 1928 to June 15, 1962:

S&P 500 price-only index data with divdends added

S&P 500 data source: https://www.billschwert.com/dstock.htm

Dividend data source: https://shillerdata.com/

June 18, 1962 to June 16, 2023:

S&P 500 price-only index data with divdends added

S&P 500 data source: https://finance.yahoo.com/quote/%5EGSPC/history

Dividend data source: https://shillerdata.com/

June 20, 2023 to Present:

SPY ETF data source: https://finance.yahoo.com/quote/SPY/history?p=SPY

Inflation Data Sources:

March 20, 1885 to December 31, 1912:

Shiller data: https://shillerdata.com/

January 2, 1913 to Present:

Consumer Price Index for All Urban Consumers: All Items in U.S. City Average (CPIAUCNS): https://fred.stlouisfed.org/series/CPIAUCNS

Risk Free Rate Data Sources:

March 20, 1885 to June 30, 1926:

Shiller 10 year yield minus 1%: https://shillerdata.com/

July 1, 1926 to December 31, 1953:

1-month T-Bill data: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/F-F_Research_Data_Factors_daily_CSV.zip

January 4, 1954 to Present:

3-month T-Bill data (DTB3): https://fred.stlouisfed.org/series/DTB3

Swap Rate Data Sources:

March 20, 1885 to June 30, 1926:

Shiller 10 year yield minus 1%: https://shillerdata.com/

July 1, 1926 to December 31, 1985:

1-month T-Bill data: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/F-F_Research_Data_Factors_daily_CSV.zip

January 2, 1986 to March 14, 2022:

1-month LIBOR data (no longer available for download, included in repository). Old link: http://iborate.com/usd-libor

March 15, 2022 to Present:

SOFR: https://fred.stlouisfed.org/series/SOFR