five-factor-universe-quant.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f4f2af90",
   "metadata": {
    "papermill": {
     "duration": 0.00682,
     "end_time": "2023-11-05T17:59:38.297125",
     "exception": false,
     "start_time": "2023-11-05T17:59:38.290305",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Building an Equity Universe with Fama-French 5 Factors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafa184f",
   "metadata": {
    "papermill": {
     "duration": 0.005778,
     "end_time": "2023-11-05T17:59:38.309438",
     "exception": false,
     "start_time": "2023-11-05T17:59:38.303660",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<a href=\"\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fef99d6",
   "metadata": {
    "papermill": {
     "duration": 0.005624,
     "end_time": "2023-11-05T17:59:38.321019",
     "exception": false,
     "start_time": "2023-11-05T17:59:38.315395",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<!-- @import \"[TOC]\" {cmd=\"toc\" depthFrom=1 depthTo=6 orderedList=false} -->\n",
    "\n",
    "![]()\n",
    "\n",
    "\n",
    "\n",
    "## Prepare your Environment\n",
    "\n",
    "Have a jupyter environment ready, and `pip install` these libraries:\n",
    "- numpy\n",
    "- pandas\n",
    "- yfinance\n",
    "\n",
    "You'll need access to [analysis_utils](./analysis_utils.py) library for common functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "6a827e09",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2023-11-05T17:59:38.335093Z",
     "iopub.status.busy": "2023-11-05T17:59:38.334737Z",
     "iopub.status.idle": "2023-11-05T18:01:02.444324Z",
     "shell.execute_reply": "2023-11-05T18:01:02.443308Z"
    },
    "papermill": {
     "duration": 84.119909,
     "end_time": "2023-11-05T18:01:02.447010",
     "exception": false,
     "start_time": "2023-11-05T17:59:38.327101",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dotenv extension is already loaded. To reload it, use:\n",
      "  %reload_ext dotenv\n",
      "Running Local...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'c:\\\\Users\\\\adamd\\\\workspace\\\\quant_research'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from tqdm import tqdm\n",
    "\n",
    "import dotenv\n",
    "%load_ext dotenv\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "IS_KAGGLE = os.getenv('IS_KAGGLE', 'True') == 'True'\n",
    "\n",
    "if IS_KAGGLE:\n",
    "    # Kaggle confgs\n",
    "    print('Running in Kaggle...')\n",
    "    %pip install yfinance\n",
    "    %pip install statsmodels\n",
    "    %pip install seaborn\n",
    "    %pip install itertools\n",
    "    %pip install scikit-learn\n",
    "\n",
    "    for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "        for filename in filenames:\n",
    "            print(os.path.join(dirname, filename))\n",
    "else:\n",
    "    print('Running Local...')\n",
    "\n",
    "import yfinance as yf\n",
    "from analysis_utils import load_ticker_prices_ts_df, load_ticker_ts_df\n",
    "\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Factors\n",
    "\n",
    "\n",
    "1. **Market Risk Premium (RM-RF)**: Excess return of a market portfolio over the risk-free rate.\n",
    "2. **Size (SMB - Small Minus Big)**: Return difference between small and large firms.\n",
    "3. **Value (HML - High Minus Low)**: Return difference between high and low book-to-market firms.\n",
    "4. **Profitability (RMW - Robust Minus Weak)**: Return difference between firms with robust and weak profitability.\n",
    "5. **Investment (CMA - Conservative Minus Aggressive)**: Return difference between firms with conservative and aggressive investments.\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "R_{it} - R_{ft} &= \\alpha_i + \\beta_{i,MktRF}(R_{Mt} - R_{ft}) + \\beta_{i,SMB}SMB_t + \\beta_{i,HML}HML_t + \\beta_{i,RMW}RMW_t + \\beta_{i,CMA}CMA_t + \\epsilon_{it} \\\\\n",
    "\\text{Where:} \\\\\n",
    "R_{it} &\\text{ is the return on asset } i \\text{ at time } t, \\\\\n",
    "R_{ft} &\\text{ is the risk-free rate at time } t, \\\\\n",
    "R_{Mt} &\\text{ is the return on the market portfolio at time } t, \\\\\n",
    "MktRF &\\text{ is the market risk premium (market return minus the risk-free rate)}, \\\\\n",
    "SMB &\\text{ represents the size premium (Small Minus Big)}, \\\\\n",
    "HML &\\text{ represents the value premium (High Minus Low)}, \\\\\n",
    "RMW &\\text{ represents the profitability premium (Robust Minus Weak)}, \\\\\n",
    "CMA &\\text{ represents the investment premium (Conservative Minus Aggressive)}, \\\\\n",
    "\\alpha_i, \\beta &\\text{ are coefficients}, \\\\\n",
    "\\epsilon_{it} &\\text{ is the error term}.\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "## Investable Universe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "38727fc7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-11-05T18:01:02.470424Z",
     "iopub.status.busy": "2023-11-05T18:01:02.469635Z",
     "iopub.status.idle": "2023-11-05T18:01:03.256388Z",
     "shell.execute_reply": "2023-11-05T18:01:03.255137Z"
    },
    "papermill": {
     "duration": 0.801372,
     "end_time": "2023-11-05T18:01:03.258937",
     "exception": false,
     "start_time": "2023-11-05T18:01:02.457565",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(             Adj Close                                                       \\\n",
       "                      A AABA    AAL AAMRQ        AAP        AAPL        ABBV   \n",
       " Date                                                                          \n",
       " 2003-01-02   11.612626  NaN    NaN   NaN  14.596296    0.224030         NaN   \n",
       " 2003-01-03   11.558020  NaN    NaN   NaN  14.370791    0.225543         NaN   \n",
       " 2003-01-06   12.110137  NaN    NaN   NaN  13.454131    0.225543         NaN   \n",
       " 2003-01-07   12.000923  NaN    NaN   NaN  13.530277    0.224787         NaN   \n",
       " 2003-01-08   11.418476  NaN    NaN   NaN  13.445341    0.220245         NaN   \n",
       " ...                ...  ...    ...   ...        ...         ...         ...   \n",
       " 2023-12-22  139.334351  NaN  14.31   NaN  61.250000  193.600006  154.940002   \n",
       " 2023-12-26  139.573929  NaN  14.11   NaN  60.919998  193.050003  154.619995   \n",
       " 2023-12-27  139.583923  NaN  13.99   NaN  61.560001  193.149994  154.880005   \n",
       " 2023-12-28  139.534012  NaN  13.98   NaN  61.490002  193.580002  154.750000   \n",
       " 2023-12-29  139.029999  NaN  13.74   NaN  61.029999  192.529999  154.970001   \n",
       " \n",
       "                           ...    Volume                                   \\\n",
       "            ABC ABI ABKFQ  ...       XOM     XRAY      XRX XTO        XYL   \n",
       " Date                      ...                                              \n",
       " 2003-01-02 NaN NaN   NaN  ...  12798800  1403200  1717769 NaN        NaN   \n",
       " 2003-01-03 NaN NaN   NaN  ...   9221900  1451600   586935 NaN        NaN   \n",
       " 2003-01-06 NaN NaN   NaN  ...  11925100  1551200  1209922 NaN        NaN   \n",
       " 2003-01-07 NaN NaN   NaN  ...  14600300  1538000  1201004 NaN        NaN   \n",
       " 2003-01-08 NaN NaN   NaN  ...  12677600  1032200   869472 NaN        NaN   \n",
       " ...         ..  ..   ...  ...       ...      ...      ...  ..        ...   \n",
       " 2023-12-22 NaN NaN   NaN  ...  12921800  1261200  1223100 NaN   829300.0   \n",
       " 2023-12-26 NaN NaN   NaN  ...  16835100  1335200  1154500 NaN   440300.0   \n",
       " 2023-12-27 NaN NaN   NaN  ...  14558800  1202800  1099000 NaN  1007700.0   \n",
       " 2023-12-28 NaN NaN   NaN  ...  16329300  1152300  1152200 NaN   709100.0   \n",
       " 2023-12-29 NaN NaN   NaN  ...  17741400  1330600  1692400 NaN   698900.0   \n",
       " \n",
       "                                                            \n",
       "                 YUM      ZBH     ZBRA     ZION        ZTS  \n",
       " Date                                                       \n",
       " 2003-01-02  3378461  1762845   800550   794400        NaN  \n",
       " 2003-01-03  3241308  1059870   924525   602300        NaN  \n",
       " 2003-01-06  2842926   841510   933750   684000        NaN  \n",
       " 2003-01-07  4670978   644574   982125   560000        NaN  \n",
       " 2003-01-08  3513944   598018  1422675   950200        NaN  \n",
       " ...             ...      ...      ...      ...        ...  \n",
       " 2023-12-22   991500  1337800   225900  1534500  1548400.0  \n",
       " 2023-12-26   627500  1870700   220000  1131600   814600.0  \n",
       " 2023-12-27  1050400  1058600   275700  1345100   766400.0  \n",
       " 2023-12-28   882500   662200   193900  1125900   880100.0  \n",
       " 2023-12-29  1196800   849600   251300  1420600  1007200.0  \n",
       " \n",
       " [5285 rows x 5694 columns],\n",
       " Date\n",
       " 2003-01-02    4.032\n",
       " 2003-01-03    4.038\n",
       " 2003-01-05      NaN\n",
       " 2003-01-06    4.066\n",
       " 2003-01-07    4.025\n",
       "               ...  \n",
       " 2023-12-25      NaN\n",
       " 2023-12-26    3.886\n",
       " 2023-12-27    3.789\n",
       " 2023-12-28    3.850\n",
       " 2023-12-29    3.866\n",
       " Name: Close, Length: 6421, dtype: float64)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "START_DATE = \"2003-01-01\"\n",
    "END_DATE = \"2023-12-31\"\n",
    "DATA_DIR = \"data\"\n",
    "\n",
    "uni_df = rf_df = None\n",
    "\n",
    "sp500_df = pd.read_csv(f\"./{DATA_DIR}/S&P500_curated_historical_components.csv\")\n",
    "sp500_df[\"date\"] = pd.to_datetime(sp500_df[\"date\"])\n",
    "\n",
    "sp500_df = sp500_df[(sp500_df[\"date\"] >= START_DATE) & (sp500_df[\"date\"] <= END_DATE)]\n",
    "\n",
    "tickers_set = set()\n",
    "sp500_df[\"tickers\"].str.split().apply(tickers_set.update)\n",
    "tickers = \" \".join(tickers_set)\n",
    "assert len(tickers) > 0\n",
    "\n",
    "cached_file_path = f\"{DATA_DIR}/cache/snp_comps.pkl\"\n",
    "try:\n",
    "    if os.path.exists(cached_file_path):\n",
    "        uni_df = pd.read_pickle(cached_file_path)\n",
    "    else:\n",
    "        uni_df = yf.download(tickers, start=START_DATE, end=END_DATE, keepna=True)\n",
    "        uni_df.to_pickle(cached_file_path)\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error downloading and caching or loading file\")\n",
    "\n",
    "cached_file_path = f\"{DATA_DIR}/cache/t10.pkl\"\n",
    "try:\n",
    "    if os.path.exists(cached_file_path):\n",
    "        rf_df = pd.read_pickle(cached_file_path)\n",
    "    else:\n",
    "        rf_df = yf.download(\"^TNX\", start=START_DATE, end=END_DATE, keepna=True)[\n",
    "            \"Close\"\n",
    "        ]\n",
    "        rf_df.to_pickle(cached_file_path)\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error downloading and caching or loading file\")\n",
    "\n",
    "uni_df, rf_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22234be0",
   "metadata": {},
   "source": [
    "## Market Risk Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "000b223d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"10\" halign=\"left\">Adj Close</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"10\" halign=\"left\">Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>AABA</th>\n",
       "      <th>AAMRQ</th>\n",
       "      <th>ABC</th>\n",
       "      <th>ABI</th>\n",
       "      <th>ABKFQ</th>\n",
       "      <th>ACAS</th>\n",
       "      <th>ADS</th>\n",
       "      <th>AGN</th>\n",
       "      <th>AKS</th>\n",
       "      <th>ALXN</th>\n",
       "      <th>...</th>\n",
       "      <th>XOM</th>\n",
       "      <th>XRAY</th>\n",
       "      <th>XRX</th>\n",
       "      <th>XTO</th>\n",
       "      <th>XYL</th>\n",
       "      <th>YUM</th>\n",
       "      <th>ZBH</th>\n",
       "      <th>ZBRA</th>\n",
       "      <th>ZION</th>\n",
       "      <th>ZTS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2003-01-02</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>12798800</td>\n",
       "      <td>1403200</td>\n",
       "      <td>1717769</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3378461</td>\n",
       "      <td>1762845</td>\n",
       "      <td>800550</td>\n",
       "      <td>794400</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003-01-03</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>9221900</td>\n",
       "      <td>1451600</td>\n",
       "      <td>586935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3241308</td>\n",
       "      <td>1059870</td>\n",
       "      <td>924525</td>\n",
       "      <td>602300</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003-01-06</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>11925100</td>\n",
       "      <td>1551200</td>\n",
       "      <td>1209922</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2842926</td>\n",
       "      <td>841510</td>\n",
       "      <td>933750</td>\n",
       "      <td>684000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003-01-07</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>14600300</td>\n",
       "      <td>1538000</td>\n",
       "      <td>1201004</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4670978</td>\n",
       "      <td>644574</td>\n",
       "      <td>982125</td>\n",
       "      <td>560000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003-01-08</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>12677600</td>\n",
       "      <td>1032200</td>\n",
       "      <td>869472</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3513944</td>\n",
       "      <td>598018</td>\n",
       "      <td>1422675</td>\n",
       "      <td>950200</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-12-22</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>12921800</td>\n",
       "      <td>1261200</td>\n",
       "      <td>1223100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>829300.0</td>\n",
       "      <td>991500</td>\n",
       "      <td>1337800</td>\n",
       "      <td>225900</td>\n",
       "      <td>1534500</td>\n",
       "      <td>1548400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-12-26</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>16835100</td>\n",
       "      <td>1335200</td>\n",
       "      <td>1154500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>440300.0</td>\n",
       "      <td>627500</td>\n",
       "      <td>1870700</td>\n",
       "      <td>220000</td>\n",
       "      <td>1131600</td>\n",
       "      <td>814600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-12-27</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>14558800</td>\n",
       "      <td>1202800</td>\n",
       "      <td>1099000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1007700.0</td>\n",
       "      <td>1050400</td>\n",
       "      <td>1058600</td>\n",
       "      <td>275700</td>\n",
       "      <td>1345100</td>\n",
       "      <td>766400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-12-28</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>16329300</td>\n",
       "      <td>1152300</td>\n",
       "      <td>1152200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>709100.0</td>\n",
       "      <td>882500</td>\n",
       "      <td>662200</td>\n",
       "      <td>193900</td>\n",
       "      <td>1125900</td>\n",
       "      <td>880100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-12-29</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>17741400</td>\n",
       "      <td>1330600</td>\n",
       "      <td>1692400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>698900.0</td>\n",
       "      <td>1196800</td>\n",
       "      <td>849600</td>\n",
       "      <td>251300</td>\n",
       "      <td>1420600</td>\n",
       "      <td>1007200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5285 rows × 6546 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Adj Close                                            ...    Volume  \\\n",
       "                AABA AAMRQ ABC ABI ABKFQ ACAS ADS AGN AKS ALXN  ...       XOM   \n",
       "Date                                                            ...             \n",
       "2003-01-02       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  12798800   \n",
       "2003-01-03       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...   9221900   \n",
       "2003-01-06       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  11925100   \n",
       "2003-01-07       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  14600300   \n",
       "2003-01-08       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  12677600   \n",
       "...              ...   ...  ..  ..   ...  ...  ..  ..  ..  ...  ...       ...   \n",
       "2023-12-22       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  12921800   \n",
       "2023-12-26       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  16835100   \n",
       "2023-12-27       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  14558800   \n",
       "2023-12-28       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  16329300   \n",
       "2023-12-29       NaN   NaN NaN NaN   NaN  NaN NaN NaN NaN  NaN  ...  17741400   \n",
       "\n",
       "                                                                        \\\n",
       "               XRAY      XRX XTO        XYL      YUM      ZBH     ZBRA   \n",
       "Date                                                                     \n",
       "2003-01-02  1403200  1717769 NaN        NaN  3378461  1762845   800550   \n",
       "2003-01-03  1451600   586935 NaN        NaN  3241308  1059870   924525   \n",
       "2003-01-06  1551200  1209922 NaN        NaN  2842926   841510   933750   \n",
       "2003-01-07  1538000  1201004 NaN        NaN  4670978   644574   982125   \n",
       "2003-01-08  1032200   869472 NaN        NaN  3513944   598018  1422675   \n",
       "...             ...      ...  ..        ...      ...      ...      ...   \n",
       "2023-12-22  1261200  1223100 NaN   829300.0   991500  1337800   225900   \n",
       "2023-12-26  1335200  1154500 NaN   440300.0   627500  1870700   220000   \n",
       "2023-12-27  1202800  1099000 NaN  1007700.0  1050400  1058600   275700   \n",
       "2023-12-28  1152300  1152200 NaN   709100.0   882500   662200   193900   \n",
       "2023-12-29  1330600  1692400 NaN   698900.0  1196800   849600   251300   \n",
       "\n",
       "                                \n",
       "               ZION        ZTS  \n",
       "Date                            \n",
       "2003-01-02   794400        NaN  \n",
       "2003-01-03   602300        NaN  \n",
       "2003-01-06   684000        NaN  \n",
       "2003-01-07   560000        NaN  \n",
       "2003-01-08   950200        NaN  \n",
       "...             ...        ...  \n",
       "2023-12-22  1534500  1548400.0  \n",
       "2023-12-26  1131600   814600.0  \n",
       "2023-12-27  1345100   766400.0  \n",
       "2023-12-28  1125900   880100.0  \n",
       "2023-12-29  1420600  1007200.0  \n",
       "\n",
       "[5285 rows x 6546 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_df = None\n",
    "tickers = uni_df.columns.get_level_values(1).unique()\n",
    "cached_file_path = f\"{DATA_DIR}/cache/snp_comps_hist.pkl\"\n",
    "try:\n",
    "    t_df = yf.Tickers(\" \".join(tickers))\n",
    "    if os.path.exists(cached_file_path):\n",
    "        t_hist_df = pd.read_pickle(cached_file_path)\n",
    "    else:\n",
    "        t_hist_df = t_df.history(start=START_DATE, end=END_DATE)\n",
    "        t_hist_df.to_pickle(cached_file_path)\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error downloading and caching or loading file\")\n",
    "\n",
    "t_hist_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "c02b62df",
   "metadata": {},
   "outputs": [],
   "source": [
    "mrk_cap = {}\n",
    "ptb = {}\n",
    "opm = {}\n",
    "roa = {}\n",
    "\n",
    "cached_file_path = f\"{DATA_DIR}/cache/fund_snp500.pkl\"\n",
    "try:\n",
    "    if os.path.exists(cached_file_path):\n",
    "        fund_df = pd.read_pickle(cached_file_path)\n",
    "    else:\n",
    "        for ticker in tqdm(tickers):\n",
    "            try:\n",
    "                ticker_info = t_df.tickers[ticker].info\n",
    "                if ticker_info is None:\n",
    "                    continue\n",
    "                mrk_cap[ticker] = ticker_info.get(\"marketCap\", np.nan)\n",
    "                ptb[ticker] = ticker_info.get(\"priceToBook\", np.nan)\n",
    "                opm[ticker] = ticker_info.get(\"operatingMargins\", np.nan)\n",
    "                roa[ticker] = ticker_info.get(\"returnOnAssets\", np.nan)\n",
    "            except Exception as e:\n",
    "                print(f\"Error processing ticker {ticker}: {e}\")\n",
    "\n",
    "        fund_df = pd.DataFrame(\n",
    "            {\n",
    "                \"MarketCap\": pd.Series(mrk_cap),\n",
    "                \"PriceToBook\": pd.Series(ptb),\n",
    "                \"OperatingMargins\": pd.Series(opm),\n",
    "                \"ReturnOnAssets\": pd.Series(roa),\n",
    "            }\n",
    "        )\n",
    "        fund_df.to_pickle(cached_file_path)\n",
    "except FileNotFoundError as fnf_error:\n",
    "    print(f\"File not found error: {fnf_error}\")\n",
    "except Exception as e:\n",
    "    print(f\"Error downloading and caching or loading file: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "b4f473e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A        4.073635e+10\n",
       "AABA              NaN\n",
       "AAL      8.979653e+09\n",
       "AAMRQ             NaN\n",
       "AAP      3.631407e+09\n",
       "             ...     \n",
       "YUM      3.662504e+10\n",
       "ZBH      2.543299e+10\n",
       "ZBRA     1.403812e+10\n",
       "ZION     6.499296e+09\n",
       "ZTS      9.061532e+10\n",
       "Name: MarketCap, Length: 933, dtype: float64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fund_df[\"MarketCap\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "6bbb2f61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2003-01-02   -4.031194\n",
       "2003-01-03   -4.037194\n",
       "2003-01-05         NaN\n",
       "2003-01-06   -4.065193\n",
       "2003-01-07   -4.024194\n",
       "                ...   \n",
       "2023-12-25         NaN\n",
       "2023-12-26   -3.885193\n",
       "2023-12-27   -3.788193\n",
       "2023-12-28   -3.849193\n",
       "2023-12-29   -3.865193\n",
       "Name: Close, Length: 6421, dtype: float64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mrk_cap = fund_df[\"MarketCap\"]\n",
    "\n",
    "mkt_weights = mrk_cap / mrk_cap.sum()\n",
    "\n",
    "\n",
    "weighted_mkt_rets_df = (uni_df[\"Adj Close\"].pct_change() * mkt_weights).sum(axis=1)\n",
    "mkt_df = weighted_mkt_rets_df.mean()\n",
    "\n",
    "\n",
    "rm_rf_df = mkt_df - rf_df\n",
    "\n",
    "rm_rf_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf868451",
   "metadata": {},
   "source": [
    "## Size Risk Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "5b1ce90e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "5694it [00:02, 2029.15it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(AAL     8.979653e+09\n",
       " AAP     3.631407e+09\n",
       " ABMD    1.718065e+10\n",
       " ACV     2.174619e+08\n",
       " ADCT    1.338665e+08\n",
       "             ...     \n",
       " X       1.085552e+10\n",
       " XRAY    7.540098e+09\n",
       " XRX     2.252867e+09\n",
       " ZBRA    1.403812e+10\n",
       " ZION    6.499296e+09\n",
       " Name: MarketCap, Length: 327, dtype: float64,\n",
       " A       4.073635e+10\n",
       " AAPL    2.994380e+12\n",
       " ABBV    2.736057e+11\n",
       " ABNB    8.725567e+10\n",
       " ABT     1.910881e+11\n",
       "             ...     \n",
       " XOM     3.995971e+11\n",
       " XYL     2.756968e+10\n",
       " YUM     3.662504e+10\n",
       " ZBH     2.543299e+10\n",
       " ZTS     9.061532e+10\n",
       " Name: MarketCap, Length: 327, dtype: float64)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for date, df in tqdm(uni_df.items()):\n",
    "    median_cap = mrk_cap.median()\n",
    "\n",
    "    small_companies = mrk_cap[mrk_cap < median_cap]\n",
    "    big_companies = mrk_cap[mrk_cap >= median_cap]\n",
    "\n",
    "small_companies, big_companies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2efe489a",
   "metadata": {},
   "source": [
    "## Value Risk Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "a2a89756",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(A        6.948373\n",
       " AAPL    48.168625\n",
       " ABBV    22.623358\n",
       " ABNB     9.535617\n",
       " ABT      5.098194\n",
       "           ...    \n",
       " WMT      5.343162\n",
       " WST      9.084858\n",
       " WU       7.124926\n",
       " ZBRA     4.659166\n",
       " ZTS     17.859922\n",
       " Name: PriceToBook, Length: 302, dtype: float64,\n",
       " AAP     1.362429\n",
       " ACGL    1.911958\n",
       " ADI     2.770630\n",
       " ADM     1.531545\n",
       " ADT     1.921127\n",
       "           ...   \n",
       " XRAY    2.233448\n",
       " XRX     0.807774\n",
       " XYL     2.773170\n",
       " ZBH     2.033077\n",
       " ZION    1.333151\n",
       " Name: PriceToBook, Length: 301, dtype: float64)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptb = fund_df[\"PriceToBook\"]\n",
    "\n",
    "median_rank = ptb.median()\n",
    "low_bm_companies = ptb[ptb < median_rank]\n",
    "high_bm_companies = ptb[ptb >= median_rank]\n",
    "\n",
    "\n",
    "high_bm_companies, low_bm_companies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35ba78e7",
   "metadata": {},
   "source": [
    "## Profitability Risk Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "71506c1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(A       0.26836\n",
       " AAPL    0.30134\n",
       " ABBV    0.17757\n",
       " ABNB    0.44039\n",
       " ABT     0.17441\n",
       "          ...   \n",
       " WU      0.19585\n",
       " XEL     0.26051\n",
       " YUM     0.36593\n",
       " ZION    0.31492\n",
       " ZTS     0.37099\n",
       " Name: OperatingMargins, Length: 327, dtype: float64,\n",
       " AAL     0.05504\n",
       " AAP    -0.01607\n",
       " ACN     0.16670\n",
       " ACV     0.00000\n",
       " ADCT   -2.58083\n",
       "          ...   \n",
       " XRAY    0.08237\n",
       " XRX     0.03148\n",
       " XYL     0.10212\n",
       " ZBH     0.17205\n",
       " ZBRA    0.05021\n",
       " Name: OperatingMargins, Length: 326, dtype: float64)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opm = fund_df[\"OperatingMargins\"]\n",
    "\n",
    "median_rank = opm.median()\n",
    "low_bm_companies = opm[opm < median_rank]\n",
    "high_bm_companies = opm[opm >= median_rank]\n",
    "\n",
    "high_bm_companies, low_bm_companies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f9b65b1",
   "metadata": {},
   "source": [
    "## Investment Risk Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "a8942c90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(AAL     0.04601\n",
       " AAP     0.02004\n",
       " ACGL    0.04064\n",
       " ADCT   -0.17433\n",
       " ADM     0.04543\n",
       "          ...   \n",
       " XRAY    0.02462\n",
       " XRX     0.02327\n",
       " XYL     0.03957\n",
       " ZBH     0.04243\n",
       " ZION    0.00954\n",
       " Name: ReturnOnAssets, Length: 324, dtype: float64,\n",
       " A       0.08253\n",
       " AAPL    0.20256\n",
       " ABBV    0.08654\n",
       " ABNB    0.07502\n",
       " ABT     0.05723\n",
       "          ...   \n",
       " WU      0.06029\n",
       " XOM     0.08879\n",
       " YUM     0.24863\n",
       " ZBRA    0.06470\n",
       " ZTS     0.13760\n",
       " Name: ReturnOnAssets, Length: 325, dtype: float64)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roa = fund_df[\"ReturnOnAssets\"]\n",
    "\n",
    "median_rank = roa.median()\n",
    "cons_companies = roa[roa < median_rank]\n",
    "aggro_companies = roa[roa >= median_rank]\n",
    "\n",
    "cons_companies, aggro_companies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "698a16a8",
   "metadata": {},
   "source": [
    "# Rank Factors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e9a9c5",
   "metadata": {
    "papermill": {
     "duration": 0.037345,
     "end_time": "2023-11-05T18:01:12.947046",
     "exception": false,
     "start_time": "2023-11-05T18:01:12.909701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "![]()\n",
    "\n",
    "## References\n",
    "\n",
    "- [Eugene F. Fama, Kenneth R. French,Common risk factors in the returns on stocks and bonds](https://www.sciencedirect.com/science/article/pii/0304405X93900235)\n",
    "- [Eugene F. Fama, Kenneth R. French, A five-factor asset pricing model](https://www.sciencedirect.com/science/article/pii/S0304405X14002323)\n",
    "- [YFinance Github](https://github.com/ranaroussi/yfinance)\n",
    "- [List of S&P 500 companies](https://en.wikipedia.org/wiki/List_of_S%26P_500_companies)\n",
    "\n",
    "\n",
    "## Github\n",
    "\n",
    "Article here is also available on [Github]()\n",
    "\n",
    "Kaggle notebook available [here]()\n",
    "\n",
    "\n",
    "## Media\n",
    "\n",
    "All media used (in the form of code or images) are either solely owned by me, acquired through licensing, or part of the Public Domain and granted use through Creative Commons License.\n",
    "\n",
    "## CC Licensing and Use\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" /></a><br />This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\">Creative Commons Attribution-NonCommercial 4.0 International License</a>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 98.905855,
   "end_time": "2023-11-05T18:01:13.607303",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-11-05T17:59:34.701448",
   "version": "2.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}