model_stock_price_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "18a819a0-322e-4d32-acd6-ea81fe8cc20f",
   "metadata": {},
   "source": [
    "### GROUP MEMBER:\n",
    "**1. Van Le:** vanle@buffalo.edu \n",
    "\n",
    "**2. Maria Anthony:** mariniv@buffalo.edu - CSE 587\n",
    "\n",
    "**3. Anushka:** atiwari4@buffalo.edu - CSE 587"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe5b248-34c2-45ed-97a3-cba108208f25",
   "metadata": {
    "tags": []
   },
   "source": [
    "# PHASE II  -  MODEL DEVELOPMENT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0008925e-b316-40a1-a5f2-470ab07f0b68",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e9d32a1-9961-412c-9bb3-557b9254667e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install pmdarima\n",
    "# !pip install xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d67dea9d-9fc1-401e-88a7-2f26b1f6a911",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import train_test_split\n",
    "from statsmodels.tsa.stattools import adfuller \n",
    "from statsmodels.tsa.arima.model import ARIMA\n",
    "from pmdarima import auto_arima\n",
    "import time\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestRegressor, StackingRegressor\n",
    "from sklearn.svm import SVR, LinearSVR\n",
    "import xgboost as xgb\n",
    "from xgboost import plot_importance, plot_tree\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score, mean_absolute_percentage_error, confusion_matrix\n",
    "from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from pandas.plotting import lag_plot\n",
    " \n",
    "# from tensorflow.keras.models import Sequential\n",
    "# from tensorflow.keras.layers import LSTM, Dense, Dropout\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2f4cc0-c614-4b58-a6b9-839843874ca0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Read the csv file we have from the Phase I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3db48aa0-d003-4bd3-b77c-000e738abb09",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df = pd.read_csv(\"stock-feature.csv\")\n",
    "merged_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "177b7ee2-3998-437e-a20a-3217f363a32a",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df['Date'] = pd.to_datetime(merged_df['Date'])\n",
    "\n",
    "# Filter the DataFrame for the years of 2017\n",
    "start_date = pd.to_datetime('2017-01-01')\n",
    "end_date = merged_df['Date'].max()\n",
    "filtered_df = merged_df[(merged_df['Date'] >= start_date) & (merged_df['Date'] <= end_date)]\n",
    "filtered_df = filtered_df.reset_index(drop=False)\n",
    "filtered_df = filtered_df.drop(['level_0', 'index'], axis=1)\n",
    "filtered_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "232ea025-e683-4ab5-b773-fd6405ce0321",
   "metadata": {},
   "outputs": [],
   "source": [
    "list(filtered_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "956fd175-4a04-450e-a8d4-3ad83fae6a8e",
   "metadata": {},
   "source": [
    "Considering our dataset contains over 4 million data rows, each comprising 30 features that encompass historical stock prices and technical indicators, we aim to extensively implement data scaling and examine the varying effects of different scaling techniques on dimension reduction via PCA. The transformed data from PCA will be supply as input features to several stock price prediction models, the integration of dimensionality reduction may enhance the accuracy of the forecasting model and gives more accurate prediction for the stock price in the future [2,3]."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71511003-795e-459d-8387-2c6e811adab8",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Comparision Scaling Techniques on PCA-based model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9894e95-e918-462a-a7ca-9c6a992f19c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_for_pca = ['Open','High','Low','Volume','Market Cap','% Change',\n",
    "                   'SMA','EMA','ADX','MACD','MACD Signal Line','RSI',\n",
    "                   'Bollinger_Bands_Middle','Bollinger_Bands_Upper','Bollinger_Bands_Lower',\n",
    "                   'KAMA','MFI','Tenkan_Sen','Kijun_Sen','Senkou_SpanA','Senkou_SpanB','Momentum',\n",
    "                   '%K','%D','Chaikin_AD','ROC','ATR','Normalized_ATR','OBV']\n",
    "\n",
    "data_for_pca = filtered_df[columns_for_pca]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1419c5be-111a-4ef2-b7fa-818bf5449f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create histograms for each feature\n",
    "data_for_pca.hist(figsize=(15, 12), bins=20)\n",
    "plt.subplots_adjust(hspace=0.5)\n",
    "plt.suptitle('Figure 1. Distribution of each feature', fontsize=16, color='b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b370d633-b74b-47bb-8b8b-8f863d988b00",
   "metadata": {},
   "source": [
    "**Observations:**\n",
    "From the figure 1 we have\n",
    "- Skewed Distributions: Certain features such as `% Change`, `MACD Signal Line`, `Momentum`, and `Chaikin A/D` are quite skewed. Skewed distributions are often better handled by scalers that are robust to outliers, we consider **Robust Scaler**.\n",
    "- Bimodal Distributions: Features like `Volume`, `MACD`, and `OBV` show two peaks. This could indicate that different scaling approaches may affect these features in significant ways. \n",
    "- Features with Outliers: `Open`, `High`, `Low`, `Market Cap`, and `ATR`, we want to diminish the influence of outliers, then **Min-Max Scaler** and **Robust Scaler** may be good options. \n",
    "- Normal-like Distributions: `KAMA`, `MFI`, and the `Bollinger Bands` appear to have a distribution close to normal, these indicators could be scaled effectively with the **Standard Scaler**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33db9a0-709a-4022-9106-c8fd66854b2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_table = pd.DataFrame(index=['PC1', 'PC2', 'PC3', 'PC4', 'PC5'])\n",
    "\n",
    "# Define different Scaler methods\n",
    "scalers = {\n",
    "    'Standard Scaler': StandardScaler(),\n",
    "    'MinMaxScaler': MinMaxScaler(),\n",
    "    'RobustScaler': RobustScaler()\n",
    "}\n",
    "\n",
    "pca_results = {}\n",
    "\n",
    "# Applying each method and perform PCA\n",
    "for name, scaler in scalers.items():\n",
    "    scaled_data = pd.DataFrame(scaler.fit_transform(data_for_pca))\n",
    "    num_components = 5\n",
    "    pca = PCA(n_components=num_components)\n",
    "    pca_fit = pca.fit_transform(scaled_data)\n",
    "\n",
    "    # Store the results in a dictionary\n",
    "    pca_results[name] = {\n",
    "        'scores': pca_fit, \n",
    "        'components': pca.components_,  \n",
    "        'explained_variance_ratio': pca.explained_variance_ratio_ \n",
    "    }\n",
    "    \n",
    "    pca_table[name] = pca.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8344678c-73e6-43b0-8d2e-26dbf4a2b913",
   "metadata": {},
   "outputs": [],
   "source": [
    "def biplot(score, coeff, labels=None, name=''):\n",
    "    plt.figure(figsize=(8,4))  \n",
    "    xs = score[:,0]\n",
    "    ys = score[:,1]\n",
    "    n = coeff.shape[0]\n",
    "    scale_x = 1.0/(xs.max() - xs.min())\n",
    "    scale_y = 1.0/(ys.max() - ys.min())\n",
    "    \n",
    "    plt.scatter(xs * scale_x,ys * scale_y)\n",
    "    \n",
    "    for i in range(n):\n",
    "        plt.arrow(0, 0, coeff[i,0], coeff[i,1],color='r',alpha=0.5)\n",
    "        if labels is None:\n",
    "            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, \"Var\"+str(i+1), color='g', ha='center', va='center')\n",
    "        else:\n",
    "            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, labels[i], color='g', ha='center', va='center')\n",
    "    \n",
    "    plt.xlabel(f\"PC{1}\")\n",
    "    plt.ylabel(f\"PC{2}\")\n",
    "    plt.title(f'Biplot using {name}', fontsize=14, color='b')\n",
    "    plt.grid()\n",
    "    plt.show()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4e0ce62-2b7a-432c-94c6-d622d1eec42e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the biplot for each scaler method\n",
    "for name, result in pca_results.items():\n",
    "    biplot(result['scores'][:,0:2], np.transpose(result['components'][0:2, :]), name=name)\n",
    "plt.suptitle('Figure 2. Biplot for each Scaler Method', fontsize=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d997df4e-8951-4e39-a2de-f5d9993dae51",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pca_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51b5e67c-6cdd-47d3-9742-70ccb9667426",
   "metadata": {},
   "source": [
    "**Observation - From biplots in Figure 2:**\n",
    "\n",
    "- StandardScaler: The plot shows a reasonable distribution of data points and component vectors (arrows), indicating a balanced representation of the dataset. The component vectors are well spread out, combining with `explained_variance_ratio_` with no single component overwhelmingly dominating. This might suggest that the data, when standardized, has its variance is captured effectively across multiple dimensions.\n",
    "- MinMaxScaler: The data points in this plot are compressed into a very narrow range on the x-axis. This could be because MinMaxScaler is sensitive to outliers, and when the data has outliers, it can cause a compression effect, which may not be ideal for PCA. \n",
    "- RobustScaler: In the last plot, a distribution that similar to StandardScaler, but with less extreme variation on the axes. The component vectors are also well distributed. RobustScaler is less sensitive to outliers, which suggests it's capturing the intrinsic spread of the data more effectively without allowing outliers to dictate the scale. Additionally, the first principal component `PC1` accounts for approximately 80% of the variance, which is much higher than with the other two methods. The remaining components contribute significantly less, which indicates that after mitigating the influence of outliers, the first component becomes highly dominant in explaining the variance.\n",
    "\n",
    "We can move forward with the $RobustScaler$ since it might be the most significant scaler for this PCA analysis because it appears to handle outliers better than other two scaler, ensuring that the principal components are not skewed by extreme values. The distribution of points can suggest that the inherent structure of the data is maintained. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5caf355a-ef46-438d-9831-cb540dde21a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the Robust Scaler transformation\n",
    "transformer = RobustScaler()\n",
    "scaled_arr = transformer.fit_transform(data_for_pca)\n",
    "scaled_df = pd.DataFrame(scaled_arr, columns=data_for_pca.columns)\n",
    "\n",
    "# Proceed with the PCA transformation\n",
    "num_components = 5\n",
    "pca = PCA(n_components=num_components)\n",
    "pca_result = pca.fit_transform(scaled_df)\n",
    "pca_result_df = pd.DataFrame(pca_result, columns=['Feature_1', 'Feature_2', 'Feature_3', 'Feature_4', 'Feature_5'])\n",
    "pca_result_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19fafe98-9127-4416-a294-b95f03839f06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Heatmap of Component Loadings\n",
    "plt.figure(figsize=(8,8))\n",
    "sns.heatmap(pca.components_,\n",
    "            cmap='magma',\n",
    "            yticklabels=[\"PC\"+str(x) for x in range(1,len(pca.components_)+1)],\n",
    "            xticklabels=columns_for_pca,\n",
    "            cbar_kws={\"orientation\": 'vertical'})\n",
    "plt.tight_layout()\n",
    "plt.title('Figure 3. Heatmap of Component Loadings', size=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b307ac25-011a-4063-addc-776f6bddb260",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = filtered_df[['Date', 'Ticker', 'Close']]\n",
    "target.index = pca_result_df.index\n",
    "transformed_df = pd.concat([pca_result_df, target], axis=1)\n",
    "transformed_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc5159e-bacb-4e42-8376-edbe2d7c6636",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "fig, axes = plt.subplots(num_components, num_components, figsize=(12, 12))\n",
    "for i, j in itertools.product(range(num_components), range(num_components)):\n",
    "    ax = axes[i, j]\n",
    "    if i != j:\n",
    "        ax.scatter(scaled_arr[:, i], scaled_arr[:, j])\n",
    "        ax.set_xlabel(f'PC{i+1}')\n",
    "        ax.set_ylabel(f'PC{j+1}')\n",
    "    else:\n",
    "        ax.annotate(f'PC{i+1}', (0.5, 0.5), xycoords='axes fraction', ha='center', va='center')\n",
    "        ax.axis('off')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.suptitle(\"Figure 4. Comparison of each Principal Component\", fontsize=14, color='r', y=1.05)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "405bb857-246b-4116-88a3-11cdf14bdaf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_df.to_csv('//Users//vanle//Downloads//pca_stock_result.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8219aed8-b422-42c2-a6ea-5a80fb5ce7dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f65f51bf-1559-4365-aead-b64f732f1832",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Read the csv file we have processed in the previous step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ba17a60e-f019-4095-8c06-0e965263daf7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature_1</th>\n",
       "      <th>Feature_2</th>\n",
       "      <th>Feature_3</th>\n",
       "      <th>Feature_4</th>\n",
       "      <th>Feature_5</th>\n",
       "      <th>Ticker</th>\n",
       "      <th>Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-01-03</th>\n",
       "      <td>-4.694621</td>\n",
       "      <td>0.050102</td>\n",
       "      <td>0.525245</td>\n",
       "      <td>-4.551044</td>\n",
       "      <td>1.058598</td>\n",
       "      <td>A</td>\n",
       "      <td>46.178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-04</th>\n",
       "      <td>-4.948428</td>\n",
       "      <td>0.293337</td>\n",
       "      <td>0.468583</td>\n",
       "      <td>-4.553871</td>\n",
       "      <td>1.065424</td>\n",
       "      <td>A</td>\n",
       "      <td>46.784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-05</th>\n",
       "      <td>-4.885045</td>\n",
       "      <td>0.113366</td>\n",
       "      <td>0.636895</td>\n",
       "      <td>-4.541259</td>\n",
       "      <td>1.020767</td>\n",
       "      <td>A</td>\n",
       "      <td>46.228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-06</th>\n",
       "      <td>-5.465603</td>\n",
       "      <td>0.731387</td>\n",
       "      <td>0.340322</td>\n",
       "      <td>-4.613235</td>\n",
       "      <td>1.154764</td>\n",
       "      <td>A</td>\n",
       "      <td>47.668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-09</th>\n",
       "      <td>-5.977897</td>\n",
       "      <td>1.172599</td>\n",
       "      <td>0.107106</td>\n",
       "      <td>-4.593020</td>\n",
       "      <td>1.132056</td>\n",
       "      <td>A</td>\n",
       "      <td>47.817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-11-06</th>\n",
       "      <td>-6.903199</td>\n",
       "      <td>-1.760332</td>\n",
       "      <td>-1.418393</td>\n",
       "      <td>4.088618</td>\n",
       "      <td>0.799909</td>\n",
       "      <td>ZYNE</td>\n",
       "      <td>11.190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-11-07</th>\n",
       "      <td>-6.796380</td>\n",
       "      <td>-1.978180</td>\n",
       "      <td>-1.165790</td>\n",
       "      <td>4.113964</td>\n",
       "      <td>0.734404</td>\n",
       "      <td>ZYNE</td>\n",
       "      <td>10.830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-11-08</th>\n",
       "      <td>-6.773013</td>\n",
       "      <td>-2.083958</td>\n",
       "      <td>-1.029962</td>\n",
       "      <td>4.123684</td>\n",
       "      <td>0.727150</td>\n",
       "      <td>ZYNE</td>\n",
       "      <td>10.900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-11-09</th>\n",
       "      <td>-7.115683</td>\n",
       "      <td>-1.690110</td>\n",
       "      <td>-1.202882</td>\n",
       "      <td>4.137377</td>\n",
       "      <td>0.783180</td>\n",
       "      <td>ZYNE</td>\n",
       "      <td>11.600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-11-10</th>\n",
       "      <td>-7.551065</td>\n",
       "      <td>-1.215361</td>\n",
       "      <td>-1.301172</td>\n",
       "      <td>4.140841</td>\n",
       "      <td>0.872513</td>\n",
       "      <td>ZYNE</td>\n",
       "      <td>12.460</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>386566 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Feature_1  Feature_2  Feature_3  Feature_4  Feature_5 Ticker  \\\n",
       "Date                                                                       \n",
       "2017-01-03  -4.694621   0.050102   0.525245  -4.551044   1.058598      A   \n",
       "2017-01-04  -4.948428   0.293337   0.468583  -4.553871   1.065424      A   \n",
       "2017-01-05  -4.885045   0.113366   0.636895  -4.541259   1.020767      A   \n",
       "2017-01-06  -5.465603   0.731387   0.340322  -4.613235   1.154764      A   \n",
       "2017-01-09  -5.977897   1.172599   0.107106  -4.593020   1.132056      A   \n",
       "...               ...        ...        ...        ...        ...    ...   \n",
       "2017-11-06  -6.903199  -1.760332  -1.418393   4.088618   0.799909   ZYNE   \n",
       "2017-11-07  -6.796380  -1.978180  -1.165790   4.113964   0.734404   ZYNE   \n",
       "2017-11-08  -6.773013  -2.083958  -1.029962   4.123684   0.727150   ZYNE   \n",
       "2017-11-09  -7.115683  -1.690110  -1.202882   4.137377   0.783180   ZYNE   \n",
       "2017-11-10  -7.551065  -1.215361  -1.301172   4.140841   0.872513   ZYNE   \n",
       "\n",
       "             Close  \n",
       "Date                \n",
       "2017-01-03  46.178  \n",
       "2017-01-04  46.784  \n",
       "2017-01-05  46.228  \n",
       "2017-01-06  47.668  \n",
       "2017-01-09  47.817  \n",
       "...            ...  \n",
       "2017-11-06  11.190  \n",
       "2017-11-07  10.830  \n",
       "2017-11-08  10.900  \n",
       "2017-11-09  11.600  \n",
       "2017-11-10  12.460  \n",
       "\n",
       "[386566 rows x 7 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stock_df = pd.read_csv('pca_stock_result.csv')\n",
    "stock_df['Date'] = pd.to_datetime(stock_df['Date'])\n",
    "stock_df.set_index('Date', inplace=True)\n",
    "stock_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f40d8c65-037a-48b6-8f7e-8ad0fedc9793",
   "metadata": {
    "tags": []
   },
   "source": [
    "## RobustScaler - PCA-based Stock Price Prediction Models Development "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14f12cb7-6941-4ba3-82f1-959720e4b2d0",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Splitting data into training and testing set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fdf347f5-6c6f-44e0-ba08-e6949e2abdb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X_train: (269495, 6)\n",
      "Shape of X_test: (58470, 6)\n",
      "Shape of X_valid: (58601, 6)\n",
      "Shape of y_train: (269495, 1)\n",
      "Shape of y_test: (58470, 1)\n",
      "Shape of y_valid: (58601, 1)\n"
     ]
    }
   ],
   "source": [
    "stock_df.sort_values(['Ticker', 'Date'], inplace=True)\n",
    "stock_df['row_num'] = stock_df.groupby('Ticker').cumcount() + 1\n",
    "\n",
    "# Calculating total count per partition\n",
    "partition_counts = stock_df.groupby('Ticker').size().reset_index(name='partition_count')\n",
    "\n",
    "# Merging the counts with the DataFrame\n",
    "stock_df_with_counts = pd.merge(stock_df, partition_counts, on='Ticker', how='left')\n",
    "\n",
    "# Calculate the thresholds for train, test, and valid partitions\n",
    "stock_df_with_counts['partition_threshold'] = stock_df_with_counts['partition_count'] * 0.7\n",
    "stock_df_with_counts['test_threshold'] = stock_df_with_counts['partition_count'] * 0.85\n",
    "\n",
    "# Filter the DataFrame per partition based on the thresholds\n",
    "train_data = stock_df_with_counts[stock_df_with_counts['row_num'] <= stock_df_with_counts['partition_threshold']]\n",
    "test_data = stock_df_with_counts[(stock_df_with_counts['row_num'] > stock_df_with_counts['partition_threshold']) &\n",
    "                                (stock_df_with_counts['row_num'] <= stock_df_with_counts['test_threshold'])]\n",
    "valid_data = stock_df_with_counts[stock_df_with_counts['row_num'] > stock_df_with_counts['test_threshold']]\n",
    "\n",
    "# Lower all column names \n",
    "train_data.columns = map(str.lower, train_data.columns)\n",
    "test_data.columns = map(str.lower, test_data.columns)\n",
    "valid_data.columns = map(str.lower, valid_data.columns)\n",
    "\n",
    "# Drop all unnessary columns \n",
    "train_data = train_data.drop(columns=['row_num', 'partition_count', 'partition_threshold', 'test_threshold'], axis=1)\n",
    "test_data = test_data.drop(columns=['row_num', 'partition_count', 'partition_threshold', 'test_threshold'], axis=1)\n",
    "valid_data = valid_data.drop(columns=['row_num', 'partition_count', 'partition_threshold', 'test_threshold'], axis=1)\n",
    "\n",
    "# Split into X and y \n",
    "X_train = train_data.drop('close', axis=1)\n",
    "X_test = test_data.drop('close', axis=1)\n",
    "X_valid = valid_data.drop('close', axis=1)\n",
    "y_train = train_data[['close']]\n",
    "y_test = test_data[['close']]\n",
    "y_valid = valid_data[['close']]\n",
    "print(\"Shape of X_train:\", X_train.shape)\n",
    "print(\"Shape of X_test:\", X_test.shape)\n",
    "print(\"Shape of X_valid:\", X_valid.shape)\n",
    "print(\"Shape of y_train:\", y_train.shape)\n",
    "print(\"Shape of y_test:\", y_test.shape)\n",
    "print(\"Shape of y_valid:\", y_valid.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "267d4c18-2517-40ce-b233-242e71ad1513",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (window._pyforest_update_imports_cell) { window._pyforest_update_imports_cell('from sklearn.preprocessing import LabelEncoder'); }\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train columns check: Index(['feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5'], dtype='object')\n",
      "X_test columns check: Index(['feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5'], dtype='object')\n",
      "X_valid columns check: Index(['feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5'], dtype='object')\n",
      "Sum of null value if exist in X_train: feature_1    0\n",
      "feature_2    0\n",
      "feature_3    0\n",
      "feature_4    0\n",
      "feature_5    0\n",
      "ticker       0\n",
      "dtype: int64\n",
      "Sum of null value if exist in X_test: feature_1    0\n",
      "feature_2    0\n",
      "feature_3    0\n",
      "feature_4    0\n",
      "feature_5    0\n",
      "ticker       0\n",
      "dtype: int64\n",
      "Sum of null value if exist in X_valid: feature_1    0\n",
      "feature_2    0\n",
      "feature_3    0\n",
      "feature_4    0\n",
      "feature_5    0\n",
      "ticker       0\n",
      "dtype: int64\n",
      "Shape of training features: (269495, 6)\n",
      "Shape of test features: (58470, 6)\n",
      "Shape of valid features: (58601, 6)\n",
      "Shape of training target: (269495, 1)\n",
      "Shape of test target: (58470, 1)\n",
      "Shape of valid target: (58601, 1)\n",
      "Train set: \n",
      "        feature_1  feature_2  feature_3  feature_4  feature_5  ticker\n",
      "269490   0.027960  -5.010235   0.437295   4.014645   0.559701    1871\n",
      "269491  -0.540750  -3.842529  -0.570200   4.067924   0.766602    1871\n",
      "269492   4.006511  -8.657852   2.992407   3.518763   1.263978    1871\n",
      "269493   5.356845  -9.007909   2.497050   4.147064   1.118489    1871\n",
      "269494   5.577453  -7.938635   1.213644   4.069383   1.544077    1871\n",
      "Test set:\n",
      "       feature_1  feature_2  feature_3  feature_4  feature_5  ticker\n",
      "58465  -1.996578  -2.767279  -3.950235   4.200832   0.965032    1866\n",
      "58466  -2.161328  -2.846956  -3.748161   4.197274   0.936594    1866\n",
      "58467  -2.304233  -2.927730  -3.572446   4.164762   0.933053    1866\n",
      "58468  -2.352760  -3.081475  -3.366854   4.185546   0.936339    1866\n",
      "58469  -2.399862  -3.202720  -3.206704   4.183147   0.920082    1866\n",
      "Target:\n",
      "        close\n",
      "386528   6.38\n",
      "386529   6.34\n",
      "386530   6.23\n",
      "386531   5.93\n",
      "386532   5.84\n"
     ]
    }
   ],
   "source": [
    "# Perform label encoder\n",
    "label_encoder = LabelEncoder()\n",
    "train_ticker_encoder = label_encoder.fit_transform(X_train['ticker'])\n",
    "train_ticker_encoder = pd.DataFrame(train_ticker_encoder, columns=['ticker'])\n",
    "test_ticker_encoder = label_encoder.fit_transform(X_test['ticker'])\n",
    "test_ticker_encoder = pd.DataFrame(test_ticker_encoder, columns=['ticker'])\n",
    "valid_ticker_encoder = label_encoder.fit_transform(X_valid['ticker'])\n",
    "valid_ticker_encoder = pd.DataFrame(valid_ticker_encoder, columns=['ticker'])\n",
    "\n",
    "# Drop the original 'ticker' column\n",
    "X_train.drop(['ticker'],axis=1,inplace=True)\n",
    "X_test.drop(['ticker'],axis=1,inplace=True)\n",
    "X_valid.drop(['ticker'],axis=1,inplace=True)\n",
    "print(\"X_train columns check:\", X_train.columns)\n",
    "print(\"X_test columns check:\", X_test.columns)\n",
    "print(\"X_valid columns check:\", X_valid.columns)\n",
    "\n",
    "# Reset the index of the training and testing data\n",
    "X_train = X_train.reset_index(drop=True)\n",
    "X_test = X_test.reset_index(drop=True)\n",
    "X_valid = X_valid.reset_index(drop=True)\n",
    "train_ticker_encoder = train_ticker_encoder.reset_index(drop=True)\n",
    "test_ticker_encoder = test_ticker_encoder.reset_index(drop=True)\n",
    "valid_ticker_encoder = valid_ticker_encoder.reset_index(drop=True)\n",
    "\n",
    "# Concatenate the encoded 'ticker' with the original DataFrames\n",
    "X_train = pd.concat([X_train, train_ticker_encoder], axis=1)\n",
    "X_test = pd.concat([X_test, test_ticker_encoder], axis=1)\n",
    "X_valid = pd.concat([X_valid, valid_ticker_encoder], axis=1)\n",
    "\n",
    "print(f\"Sum of null value if exist in X_train:\", X_train.isnull().sum())\n",
    "print(f\"Sum of null value if exist in X_test:\", X_test.isnull().sum())\n",
    "print(f\"Sum of null value if exist in X_valid:\", X_valid.isnull().sum())\n",
    "print(f\"Shape of training features:\", X_train.shape)\n",
    "print(f\"Shape of test features:\", X_test.shape)\n",
    "print(f\"Shape of valid features:\", X_valid.shape)\n",
    "print(f\"Shape of training target:\", y_train.shape)\n",
    "print(f\"Shape of test target:\", y_test.shape)\n",
    "print(f\"Shape of valid target:\", y_valid.shape)\n",
    "print(f\"Train set: \")\n",
    "print(X_train.tail())\n",
    "print(f\"Test set:\" )\n",
    "print(X_test.tail())\n",
    "print(f\"Target:\" )\n",
    "print(y_test.tail())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6438c181-c39b-4419-abbf-61a0d1f1e494",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n",
      "(269495,)\n",
      "<class 'numpy.ndarray'>\n",
      "(58470,)\n",
      "<class 'numpy.ndarray'>\n",
      "(58601,)\n"
     ]
    }
   ],
   "source": [
    "# Convert to a one-dimensional NumPy array\n",
    "y_train = y_train.values.ravel()\n",
    "y_test = y_test.values.ravel()\n",
    "y_valid = y_valid.values.ravel()\n",
    "print(type(y_train))\n",
    "print(y_train.shape)\n",
    "print(type(y_test))\n",
    "print(y_test.shape)\n",
    "print(type(y_valid))\n",
    "print(y_valid.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e4d42a6-53fb-455c-9add-234a366b77e0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 1. ARIMA Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bde65d4-b00e-444e-8236-5823fa736250",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Test for stationarity:\n",
    "The data is stationary if they do not have trend or any seasonal effects. And if the data is non-stationary, then we have to convert it to stationary data before fitting into the ARIMA model. To check whether the data is stationary, we will use Augmented Dicky-Fuller (ADF) test.\n",
    "\n",
    "The ADF test, also known as the “unit root test”, is a statistical test to inform the degree to which a null hypothesis can be rejected or fail to reject. The p-value below a threshold (1%, 5%, 10%) suggests we can reject the null hypothesis [2]. \n",
    "\n",
    "- Null Hypothesis \\(H<sub>0</sub>\\): If failed to be rejected, it suggests the time series has a unit root, meaning it is non-stationary.\n",
    "- Alternative Hypothesis \\(H<sub>1</sub>): The null hypothesis is rejected and suggests the time series does not have a unit root, meaning it is stationary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f6fd631-ffe0-4a35-8a52-cddbaf23529a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking the stationarity\n",
    "def adf_test(timeseries):\n",
    "    # # Determing rolling statistics\n",
    "    # moving_average = timeseries.rolling(12).mean()\n",
    "    # moving_std = timeseries.rolling(12).std()\n",
    "    # Perform Dickey-Fuller test:\n",
    "    adft = adfuller(timeseries, autolag='AIC')\n",
    "    # Extract and display test results in a Series\n",
    "    output = pd.Series(adft[0:4], \n",
    "                       index=['Test Statistics', 'p-value', 'No. of lags used', 'Number of observations used'])\n",
    "    \n",
    "    for key, value in adft[4].items():\n",
    "        output[\"Critical value (%s)\" % key] = value\n",
    "    print(output)\n",
    "\n",
    "#  Check if y_train is stationary \n",
    "adf_test(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a923032b-f6db-4e0d-aa48-b3e4df70503b",
   "metadata": {},
   "source": [
    "After performing Augmented Dickey-Fuller (ADF) test, we got the test statistics of $-42.395700$, which is much lower than the critical values at 1%, 5%, and 10% levels. The p-value is essentially 0, which is below the common alpha level of 0.05. This strongly suggests that we can reject the null hypothesis and conclude that our time series data is stationary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306846f8-36e3-4c30-af17-55bda4176ff3",
   "metadata": {},
   "source": [
    "We will apply `auto_arima` function from the `pmdarima` library since it is a useful tool for automatically determining the optimal parameters for an ARIMA model based on the provided time series data. Then, we will use the returned parameters to fit our ARIMA model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cab0ae51-44bd-4132-b1ed-77c805e89e7c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Fit the Auto-ARIMA Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "656bc803-8581-4def-a276-97bf4a14a40b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fit the auto_arima model\n",
    "def fit_auto_arima(train_data, initial_p, initial_q):\n",
    "    stepwise_fit = auto_arima(train_data, \n",
    "                              start_p=1, start_q=1,\n",
    "                              max_p=initial_p, max_q=initial_q, \n",
    "                              m=1, # we want to imply non-seasonal, so we use m=1 \n",
    "                              start_P=0, \n",
    "                              seasonal=False,\n",
    "                              d=None, \n",
    "                              D=0, \n",
    "                              trace=True,\n",
    "                              error_action='ignore',\n",
    "                              suppress_warnings=True,\n",
    "                              stepwise=True)\n",
    "    return stepwise_fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05c506bb-4aa9-40f8-941c-839e97ce5cff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "auto_arima_model = fit_auto_arima(y_train,5,5)\n",
    "print(auto_arima_model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51b4c48b-c492-4257-ae6f-b80f01f17103",
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_arima_model.plot_diagnostics(figsize=(14,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "def92417-2cba-44af-977c-caa4905c0ae4",
   "metadata": {},
   "source": [
    "ARIMA is a class of models used for time series forecasting and analysis. It combines autoregressive (AR), differencing (I), and moving average (MA) components to model and predict time series data.\n",
    "\n",
    "**5**: This is the number of autoregressive (AR) terms in the model. The autoregressive component captures the relationship between the current value in the time series and its past values. In this case, there are 5 lagged (previous) values of the time series that will be used as predictors in the model.\n",
    "\n",
    "**1**: This is the differencing order (I) in the model. The differencing component is used to make the time series stationary by taking differences between consecutive observations. A value of 1 indicates that first-order differencing is applied, which means that each value in the time series is replaced by the difference between it and the previous value.\n",
    "\n",
    "**1**: This is the number of moving average (MA) terms in the model. The moving average component models the relationship between the current value and past white noise (random) error terms. In this case, there is 1 lagged white noise error term included in the model.\n",
    "\n",
    "In summary, ARIMA(5,1,1) is 5 autoregressive terms, 1 order of differencing, and 1 moving average term. And all coefficients being statistically significant. \n",
    "\n",
    "<!-- There are signs of non-normality and heteroskedasticity in the residuals according to the Jarque-Bera and heteroskedasticity tests -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d211b3e7-29d0-4bc0-8725-e7d6d046df80",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_arima(train_data, p, d, q):\n",
    "    arima_model = ARIMA(train_data, order=(p, d, q))\n",
    "    arima_result = arima_model.fit()\n",
    "    return arima_result\n",
    "\n",
    "# Buid and fit ARIMA model with order (p,d,q)=(5,1,1)\n",
    "arima_result_1 = fit_arima(y_train, 5, 1, 1)\n",
    "print(arima_result_1.summary())\n",
    "\n",
    "# Forecasting stock prices on the test \n",
    "arima_forecast_1 = arima_result_1.get_forecast(steps=len(y_test))\n",
    "arima_forecast_values_1 = arima_forecast_1.predicted_mean  \n",
    "conf_int_1 = arima_forecast_1.conf_int()\n",
    "\n",
    "# Plot forecast results against actual data \n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.plot(y_test, label='Actual Test Data')  # Plot y_test directly\n",
    "plt.plot(y_test, arima_forecast_values_1, label='ARIMA(5,1,1) Forecast')\n",
    "plt.fill_between(\n",
    "    range(len(y_test)),\n",
    "    conf_int_1[:, 0],  \n",
    "    conf_int_1[:, 1],  \n",
    "    color='pink',\n",
    "    alpha=0.3,\n",
    "    label='95% Confidence Interval'\n",
    ")\n",
    "plt.title(\"Figure 5. ARIMA(5,1,1) Forecast vs Actual Test Data\", fontsize=14, color='r')\n",
    "plt.xlabel(\"Time Step\")\n",
    "plt.ylabel(\"Values\")\n",
    "plt.legend(loc='upper left', fontsize=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abeb681a-2816-4c65-a0f7-4dab44caf0ee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Since it's too large dataset, kernel was interruptted during middle of training \n",
    "auto_arima_model = fit_auto_arima(y_train,6,6)\n",
    "print(auto_arima_model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bef366f8-efbe-4da5-bf04-a2e977d3da78",
   "metadata": {},
   "source": [
    "Even kernel was interrupted, we still can see ARIMA(4,1,3) has smaller AIC than any other param values. We could fit in on our training to see if this model make any significant improvement on forecasting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d55e068-4fc5-41fe-b2a7-bf5144221f21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the ARIMA model with order (p,d,q)=(4,1,3)\n",
    "arima_result_2 = fit_arima(y_train, 4, 1, 3)\n",
    "print(arima_result_2.summary())\n",
    "\n",
    "# Make forecasts\n",
    "arima_forecast_2 = arima_result_2.get_forecast(steps=len(y_test))\n",
    "arima_forecast_values_2 = arima_forecast_2.predicted_mean  \n",
    "conf_int_2 = arima_forecast_2.conf_int()\n",
    "\n",
    "# Plot predict results against the actual results\n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.plot(y_test, label='Actual Test Data')  # Plot y_test directly\n",
    "plt.plot(y_test, arima_forecast_values_2, label='ARIMA(4,3,1) Forecast')\n",
    "plt.fill_between(\n",
    "    range(len(y_test)),\n",
    "    conf_int_2[:, 0],  \n",
    "    conf_int_2[:, 1],  \n",
    "    color='pink',\n",
    "    alpha=0.3,\n",
    "    label='95% Confidence Interval'\n",
    ")\n",
    "plt.title(\"Figure 6. ARIMA(4,3,1) Forecast vs Actual Test Data\", fontsize=14, color='r')\n",
    "plt.xlabel(\"Time Step\")\n",
    "plt.ylabel(\"Values\")\n",
    "plt.legend(loc='upper left', fontsize=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e4d3e16-10fc-4f51-ab8a-4b268ad423f8",
   "metadata": {
    "tags": []
   },
   "source": [
    "The best ARIMA model according to the Akaike Information Criterion (AIC), is an ARIMA(4,1,3) without a seasonal component. This model includes 4 autoregressive terms, a differencing order of 1, and 3 moving average term.\n",
    "\n",
    "- `IC`: The AIC value is 3270235.960, as the lower the AIC, the better the model fits time series data while penalizing for complexity.\n",
    "- `Coefficients`: All the coefficients for the autoregressive (AR) and moving average (MA) terms are significant (as $p-value < 0.05$), as indicated by the z-test.\n",
    "- `Log Likelihood`: The log likelihood of the model is quite large in the negative, indicating the likelihood of the observed data given the model.\n",
    "- `Ljung-Box Test`: The Ljung-Box test on residuals is a way to check for lack of fit. In this case, with a $p-value = 0.61$, there is no evidence of lack of fit.\n",
    "- `Jarque-Bera (JB) Test`: The JB test is extremely large, indicating that the residuals are not normally distributed. However, for large sample sizes, this test may always indicate non-normality.\n",
    "- `Heteroskedasticity (H)`: The test statistic is very high, indicating that there is heteroskedasticity in the residuals."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ffd91fc-77a0-41bd-ba4f-070b8e8ff4ab",
   "metadata": {},
   "source": [
    "**Comparision between ARIMA(5,1,1) vs. ARIMA(4,1,3)**\n",
    "\n",
    "- Based on these result tables, both models are relatively close in terms of fit, with ARIMA(4,1,3) having a slightly better log likelihood, AIC, and BIC compares to ARIMA(5,1,1). However, the differences are not significant, and the choice between these models may depend on other factors, such as the interpretability of the model coefficients, \n",
    "\n",
    "- In ARIMA(5,1,1), the AR coefficients (ar.L1 to ar.L5) are all negative, indicating a decreasing trend in autocorrelation with lag. While ARIMA(4,1,3), the AR coefficients (ar.L1 to ar.L4) are a mix of positive and negative values, suggesting a more complex relationship.\n",
    "\n",
    "- The standard errors in both models are relatively small, indicating that the parameter estimates are precise. \n",
    "- The z-scores in ARIMA(4,3,1) are very large in absolute terms, and even larger than ARIMA(5,1,1) indicating that the coefficients are highly significant.\n",
    "\n",
    "In overall, ARIMA(4,3,1) may have little better fit to the data. However, ARIMA(5,1,1) is simpler model so it might be easy for high variety dataset. We'll evaluate the metrics to get more insights from the result for both forecast models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0be3905f-1864-46c0-85fc-08bc4808729a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def forecast_bias(y_true, y_pred):\n",
    "    return np.mean(y_pred - y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d44a4a4c-0cea-456a-98c7-c068082883c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_absolute_percentage_error(y_true, y_pred):\n",
    "    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccba0273-40f1-4fde-aaac-d012fffda96d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate model performance: MAE, RMSE, MAPE\n",
    "mae1 = mean_absolute_error(y_test, arima_forecast_values_1)\n",
    "mae2 = mean_absolute_error(y_test, arima_forecast_values_2)\n",
    "\n",
    "rmse1 = np.sqrt(mean_squared_error(y_test, arima_forecast_values_1))\n",
    "rmse2 = np.sqrt(mean_squared_error(y_test, arima_forecast_values_2))\n",
    "\n",
    "mape1 = mean_absolute_percentage_error(y_test, arima_forecast_values_1)\n",
    "mape2 = mean_absolute_percentage_error(y_test, arima_forecast_values_2)\n",
    "\n",
    "bias1 = forecast_bias(y_test, arima_forecast_values_1)\n",
    "bias2 = forecast_bias(y_test, arima_forecast_values_2)\n",
    "\n",
    "# Calculate forecast accuracy\n",
    "accuracy1 = 1 - (mape1 / 100)\n",
    "accuracy2 = 1 - (mape2 / 100)\n",
    "\n",
    "# Create a comparision dataframe \n",
    "models = ['ARIMA(4,3,1)', 'ARIMA(5,1,1)']\n",
    "mae = [mae2, mae1]\n",
    "rmse = [rmse2, rmse1]\n",
    "mape = [mape2, mape1]\n",
    "bias = [bias2, bias1]\n",
    "accuracy = [accuracy2, accuracy1]\n",
    "\n",
    "metrics_dict = {\n",
    "    \"Model\": models,\n",
    "    \"MAE\": mae,\n",
    "    \"RMSE\": rmse,\n",
    "    \"MAPE\": mape,\n",
    "    \"Forecast Bias\": bias,\n",
    "    \"Forecast Accuracy\": accuracy\n",
    "}\n",
    "\n",
    "metrics_df = pd.DataFrame(metrics_dict)\n",
    "metrics_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "548b79ba-a765-425f-b6dc-982f5f891ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the last 50 points of y_test and the corresponding forecasted values and confidence interval\n",
    "y_test_last_50 = y_test[-100:]\n",
    "forecast_values_last_50 = arima_forecast_values_1[-100:]\n",
    "conf_int_last_50 = conf_int_1[-100:]\n",
    "\n",
    "# Create a range of values for the x-axis\n",
    "x_axis_last_50 = range(len(y_test_last_50))\n",
    "\n",
    "# Plot forecast results against actual data for the last 50 points\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(x_axis_last_50, y_test_last_50, label='Actual Test Data', color='blue', linewidth=2)\n",
    "plt.plot(x_axis_last_50, forecast_values_last_50, label='ARIMA(5,1,1) Forecast', color='red', linewidth=2)\n",
    "plt.fill_between(x_axis_last_50, \n",
    "                 conf_int_last_50[:, 0], \n",
    "                 conf_int_last_50[:, 1], \n",
    "                 color='pink', \n",
    "                 alpha=0.3, \n",
    "                 label='95% Confidence Interval')\n",
    "\n",
    "# Set y-axis limits to ignore outliers/extreme values\n",
    "plt.ylim([0,70]) \n",
    "plt.title(\"Figure 7. ARIMA(5,1,1) Forecast vs Actual Test Data (Last 50 Points)\", fontsize=14)\n",
    "plt.xlabel(\"Time Step\")\n",
    "plt.ylabel(\"Values\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee062747-93bb-4ab4-8fd4-cc2f042f96e2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 2. Random Forest Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "140fcc03-c083-435b-8b20-297172d8cd9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(max_depth=500, random_state=0)\n",
    "rf.fit(X_train,y_train)\n",
    "rf_pred = rf.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "rf_mse = mean_squared_error(y_test, rf_pred)\n",
    "rf_mae = mean_absolute_error(y_test, rf_pred)\n",
    "rf_rmse = np.sqrt(mean_squared_error(y_test, rf_pred))\n",
    "rf_mape = mean_absolute_percentage_error(y_test, rf_pred)\n",
    "rf_r2 = r2_score(y_test, rf_pred)\n",
    "\n",
    "print(\"MSE:\", rf_mse) \n",
    "print(\"MAE:\", rf_mae) \n",
    "print(\"RMSE:\", rf_rmse) \n",
    "print(\"MAPE:\", rf_mape) \n",
    "print(\"R-squared (Random Forest):\", rf_r2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7b35a40-c332-41b6-8ac3-86970baa5123",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the last 50 points of y_test and the corresponding forecasted values and confidence interval\n",
    "y_test_last_50 = y_test[-100:]\n",
    "forecast_values_last_50 = rf_pred[-100:]\n",
    "#conf_int_last_50 = conf_int_1[-100:]\n",
    "\n",
    "# Create a range of values for the x-axis\n",
    "x_axis_last_50 = range(len(y_test_last_50))\n",
    "\n",
    "# Plot forecast results against actual data for the last 50 points\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(x_axis_last_50, y_test_last_50, label='Actual Test Data', color='blue', linewidth=2)\n",
    "plt.plot(x_axis_last_50, forecast_values_last_50, label='Random Forest Regressor', color='red', linewidth=2)\n",
    "\n",
    "# Set y-axis limits to ignore outliers/extreme values\n",
    "plt.ylim([0,70])\n",
    "plt.title(\"Figure 8. Random Forest Regressor vs Actual Test Data (Last 50 Points)\", fontsize=14)\n",
    "plt.xlabel(\"Time Step\")\n",
    "plt.ylabel(\"Values\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3539fb1-c832-4b70-9903-bcf36622ff2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rf = RandomForestRegressor(random_state=0)\n",
    "\n",
    "# # Define parameter grid \n",
    "# param_grid = {\n",
    "#     'n_estimators': [100, 200, 300],\n",
    "#     'max_depth': [10, 20, 30, None],\n",
    "#     'min_samples_split': [2, 5, 10],\n",
    "#     'min_samples_leaf': [1, 2, 4],\n",
    "#     'max_features': ['auto', 'sqrt', 'log2']\n",
    "# }\n",
    "\n",
    "# # Create GridSearchCV object with cross-validation\n",
    "# grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, \n",
    "#                            scoring='neg_mean_squared_error', cv=5, n_jobs=-1)\n",
    "\n",
    "# grid_search.fit(X_valid, y_train)\n",
    "# cv_results = grid_search.cv_results_\n",
    "# best_params = grid_search.best_params_\n",
    "\n",
    "# # # Plot result from cv_results\n",
    "# # mean_test_scores = cv_results['mean_test_score']\n",
    "# # max_depths = [param['max_depth'] for param in cv_results['params']]\n",
    "# # n_estimators = [param['n_estimators'] for param in cv_results['params']]\n",
    "\n",
    "# # mean_test_scores_grid = np.array(mean_test_scores).reshape(len(max_depths), len(n_estimators))\n",
    "\n",
    "# # Create a heatmap to visualize MSE\n",
    "# plt.figure(figsize=(10, 8))\n",
    "# plt.imshow(mean_test_scores_grid, cmap='viridis', origin='lower')\n",
    "# plt.colorbar(label='Mean Squared Error (MSE)')\n",
    "# plt.xticks(np.arange(len(n_estimators)), n_estimators, rotation=45)\n",
    "# plt.yticks(np.arange(len(max_depths)), max_depths)\n",
    "# plt.xlabel('Number of Estimators (n_estimators)')\n",
    "# plt.ylabel('Maximum Depth (max_depth)')\n",
    "# plt.title('Grid Search Results for Random Forest Hyperparameters')\n",
    "# plt.show()\n",
    "\n",
    "# print(\"Best Hyperparameters:\", best_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b40fa7-5ee9-452e-be6b-c6c7b4dce379",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 3. SVR Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "749059ad-6862-499d-ab16-887818e08742",
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_model = SVR()\n",
    "svr_model.fit(X_train, y_train)\n",
    "svr_pred = svr_model.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "svr_mse = mean_squared_error(y_test, svr_pred)\n",
    "svr_mae = mean_absolute_error(y_test, svr_pred)\n",
    "svr_rmse = np.sqrt(mean_squared_error(y_test, svr_pred))\n",
    "svr_mape = mean_absolute_percentage_error(y_test, svr_pred)\n",
    "svr_r2 = r2_score(y_test, svr_pred)\n",
    "\n",
    "print(\"MSE:\", svr_mse) \n",
    "print(\"MAE:\", svr_mae) \n",
    "print(\"RMSE:\", svr_rmse) \n",
    "print(\"MAPE:\", svr_mape) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb20b2d8-fcf9-420c-b513-4e44b41333d8",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4. Linear SVR Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e76b0aa8-f756-47f2-9bd2-de9c36d1ddd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 53.25887128277389\n",
      "MAE: 1.1404649475441762\n",
      "RMSE: 7.297867584628669\n",
      "MAPE: 0.1311646933757877\n",
      "R-squared: 0.997204954161102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "linear_svr = LinearSVR(loss='epsilon_insensitive', random_state=0)\n",
    "linear_svr.fit(X_train, y_train)\n",
    "linear_svr_pred = linear_svr.predict(X_test)\n",
    "\n",
    "linear_svr_mse = mean_squared_error(y_test, linear_svr_pred)\n",
    "linear_svr_mae = mean_absolute_error(y_test, linear_svr_pred)\n",
    "linear_svr_rmse = np.sqrt(mean_squared_error(y_test, linear_svr_pred))\n",
    "linear_svr_mape = mean_absolute_percentage_error(y_test, linear_svr_pred)\n",
    "linear_svr_r2 = r2_score(y_test, linear_svr_pred)\n",
    "\n",
    "print(\"MSE:\", linear_svr_mse) \n",
    "print(\"MAE:\", linear_svr_mae) \n",
    "print(\"RMSE:\", linear_svr_rmse) \n",
    "print(\"MAPE:\", linear_svr_mape) \n",
    "print(\"R-squared:\", linear_svr_r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24d00533-b6d5-4f50-a022-e68c4d230332",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning for Linear SVR "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "302a2ee4-5cff-4176-a44c-61a576e7afe9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n",
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n",
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n",
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n",
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n",
      "/Users/vanle/opt/anaconda3/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "svr = LinearSVR() \n",
    "params = {\n",
    "    'C': [0.01, 0.1, 1],\n",
    "    'epsilon': [0.01, 0.1, 0.5],\n",
    "    'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive'],\n",
    "    'max_iter': [10000, 50000, 100000],\n",
    "    'tol': [0.01, 0.05, 0.1]\n",
    "}\n",
    "\n",
    "# Grid Search cross-validation\n",
    "grid_search = GridSearchCV(svr, params, cv=5, scoring='neg_mean_squared_error')\n",
    "grid_search.fit(X_valid, y_valid)\n",
    "best_svr = grid_search.best_estimator_\n",
    "\n",
    "# Print the best params\n",
    "print(\"Best Parameters:\", grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af125cef-a2ef-41a0-b73b-6a84f2f60541",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions with the best estimator\n",
    "best_svr_pred = best_svr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e276da4-31b2-44c4-bd58-46847deefa7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the best estimator\n",
    "best_svr_mse = mean_squared_error(y_test, best_svr_pred)\n",
    "best_svr_mae = mean_absolute_error(y_test, best_svr_pred)\n",
    "best_svr_rmse = np.sqrt(mean_squared_error(y_test, best_svr_pred))\n",
    "best_svr_mape = mean_absolute_percentage_error(y_test, best_svr_pred)\n",
    "best_svr_r2 = r2_score(y_test, best_svr_pred)\n",
    "\n",
    "# Print the evaluation metrics\n",
    "print(\"MSE:\", best_svr_mse) \n",
    "print(\"MAE:\", best_svr_mae) \n",
    "print(\"RMSE:\", best_svr_rmse) \n",
    "print(\"MAPE:\", best_svr_mape) \n",
    "print(\"R-squared:\", best_svr_r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e8eb79c-4842-4a80-b720-ec4c695241e5",
   "metadata": {},
   "source": [
    "**Observation:**\n",
    "\n",
    "The tuned LinearSVR model demonstrates outperforms compared to the default model in stock price prediction, as evidenced by its improved metrics across the board. Specifically, the tuned model shows a significant reduction in Mean Squared Error (MSE) from 53.259 to 39.157, indicating a more accurate prediction with fewer errors. The Mean Absolute Error (MAE) is almost halved, decreasing from 1.140 to 0.618, which reflects a more precise average prediction error. Furthermore, the Root Mean Squared Error (RMSE) is reduced from 7.298 to 6.258, demonstrating the tuned model's enhanced capability in handling large errors. The Mean Absolute Percentage Error (MAPE) also shows notable improvement, dropping from 13.12% to 9.15%, which implies better accuracy in terms of relative prediction error. \n",
    "\n",
    "Based on these metrics, the tuned LinearSVR model is the better choice as it shows significant improvements in prediction accuracy and error reduction. This makes it more reliable for stock price prediction, especially in a field where precision is crucial. However, it is important to consider the model's performance on unseen data to ensure its generalizability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c4d420-3f7d-44d8-a439-89727556003a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f2cbe65-83b6-4bb2-a685-691a0ee8823f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the LinearSVR model with the best parameters\n",
    "best_svr = LinearSVR(C=grid_search.best_params_['C'],\n",
    "                      epsilon=grid_search.best_params_['epsilon'],\n",
    "                      loss=grid_search.best_params_['loss'],\n",
    "                      max_iter=grid_search.best_params_['max_iter'],\n",
    "                      tol=grid_search.best_params_['tol'])\n",
    "\n",
    "best_svr.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions with the final model\n",
    "final_pred = best_svr.predict(X_test)\n",
    "\n",
    "# Evaluate the final model\n",
    "mse = mean_squared_error(y_test, final_pred)\n",
    "mae = mean_absolute_error(y_test, final_pred)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, final_pred))\n",
    "mape = mean_absolute_percentage_error(y_test, final_pred)\n",
    "r2 = r2_score(y_test, final_pred)\n",
    "\n",
    "# Print the evaluation metrics\n",
    "print(\"Final Model MSE:\", mse) \n",
    "print(\"Final Model MAE:\", mae) \n",
    "print(\"Final Model RMSE:\", rmse) \n",
    "print(\"Final Model MAPE:\", mape) \n",
    "print(\"Final Model R-squared:\", r2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e620a6a-73a6-435e-9a8f-a26c28933b39",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5. XGBoost Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "837e3497-0810-4a90-a61b-596add122dd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Baseline XGB Regressor\n",
    "xgb_reg = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=100)\n",
    "xgb_reg.fit(X_train, y_train)\n",
    "xgb_pred = xgb_reg.predict(X_test)\n",
    "\n",
    "xgb_mse = mean_squared_error(y_test, xgb_pred)\n",
    "xgb_mae = mean_absolute_error(y_test, xgb_pred)\n",
    "xgb_rmse = np.sqrt(mean_squared_error(y_test, xgb_pred))\n",
    "xgb_mape = mean_absolute_percentage_error(y_test, xgb_pred)\n",
    "\n",
    "print(\"MSE:\", xgb_mse) \n",
    "print(\"MAE:\", xgb_mae) \n",
    "print(\"RMSE:\", xgb_rmse) \n",
    "print(\"MAPE:\", xgb_mape) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c92dafc6-6329-4f7a-824d-ca99a180a676",
   "metadata": {},
   "source": [
    "#### Regularization \n",
    "\n",
    "Regularization is a technique used to prevent overfitting by penalizing models with extreme coefficient values. XGBoost offers two parameters for regularization [6]:\n",
    "\n",
    "`lambda`: L2 regularization term on weights, also known as reg_lambda. It's used to help prevent overfitting by adding a penalty for larger weights in the model.\n",
    "\n",
    "`alpha`: L1 regularization term on weights, also known as reg_alpha. It encourages sparsity, meaning it can set some weight coefficients in the model to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "172c14de-892e-4b96-819d-e37ddcbac463",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_reg = xgb.XGBRegressor(\n",
    "    objective='reg:squarederror',\n",
    "    n_estimators=100,\n",
    "    seed=0,\n",
    "    reg_lambda=1,  \n",
    "    reg_alpha=1    \n",
    ")\n",
    "\n",
    "xgb_reg.fit(X_train, y_train)\n",
    "xgb_pred = xgb_reg.predict(X_test)\n",
    "\n",
    "xgb_mse = mean_squared_error(y_test, xgb_pred)\n",
    "xgb_mae = mean_absolute_error(y_test, xgb_pred)\n",
    "xgb_rmse = np.sqrt(mean_squared_error(y_test, xgb_pred))\n",
    "xgb_mape = mean_absolute_percentage_error(y_test, xgb_pred)\n",
    "print(\"MSE:\", xgb_mse) \n",
    "print(\"MAE:\", xgb_mae) \n",
    "print(\"RMSE:\", xgb_rmse) \n",
    "print(\"MAPE:\", xgb_mape) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f380ba3-d03d-49d6-9299-eca7bad6e11d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Plot feature importance\n",
    "xgb.plot_importance(xgb_reg)\n",
    "plt.title(\"Figure 11. Feature Importance in XGBoost Regression after Regularization\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d77f5b97-beb2-4290-aae3-3f143be978f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize model\n",
    "xgb_reg = xgb.XGBRegressor()\n",
    "\n",
    "# Define the parameter grid\n",
    "param_grid = {\n",
    "    'n_estimators': [100, 200, 300, 400],\n",
    "    'learning_rate': [0.01, 0.1, 0.2, 0.4],\n",
    "    'reg_alpha': [1e-5, 1e-2, 0.1, 1, 100],\n",
    "    'reg_lambda': [1e-5, 1e-2, 0.1, 1, 100]\n",
    "}\n",
    "\n",
    "# Initialize grid search\n",
    "grid_search = GridSearchCV(estimator=xgb_reg, \n",
    "                           param_grid=param_grid, \n",
    "                           cv=5, \n",
    "                           scoring='neg_mean_squared_error'\n",
    "                          )\n",
    "\n",
    "grid_search.fit(X_valid, y_valid)\n",
    "print(\"Best parameters found: \", grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d67f72-6ecf-4310-abbe-9560fbfc2bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Train the final model with the best parameters\n",
    "# best_xgb = xgb.XGBRegressor(**grid_search.best_params_)\n",
    "# best_xgb.fit(X_train, y_train)\n",
    "# xgb_pred = final_model.predict(X_test)\n",
    "\n",
    "# # Evaluate\n",
    "# xgb_mse = mean_squared_error(y_test, xgb_pred)\n",
    "# xgb_mae = mean_absolute_error(y_test, xgb_pred)\n",
    "# xgb_rmse = np.sqrt(mean_squared_error(y_test, xgb_pred))\n",
    "# xgb_mape = mean_absolute_percentage_error(y_test, xgb_pred)\n",
    "# print(f\"Tuned XGBoost for Regression MSE:\", xgb_mse) \n",
    "# print(f\"Tuned XGBoost for Regression MAE:\", xgb_mae) \n",
    "# print(f\"Tuned XGBoost for Regression RMSE:\", xgb_rmse) \n",
    "# print(f\"Tuned XGBoost for Regression MAPE:\", xgb_mape) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "793b1bdc-ae94-4b6f-8b4d-5e5ed4b38bd1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8d6787d2-e8cb-437d-8cbe-c825fe375e19",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 5. LSTM Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17838d7e-c11f-473f-9f32-7ba3f12fea98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Training LSTM model\n",
    "# class LSTMModel:\n",
    "#     def __init__(self, input_size, lstm_units=50, epochs=50, batch_size=32):\n",
    "#         self.input_size = input_size\n",
    "#         self.lstm_units = lstm_units\n",
    "#         self.epochs = epochs\n",
    "#         self.batch_size = batch_size\n",
    "#         self.model = self.build_model()\n",
    "\n",
    "#     def lstm_model(self):\n",
    "#         model = Sequential()\n",
    "#         model.add(LSTM(units=self.lstm_units, input_size=self.input_size))\n",
    "#         model.add(Dense(units=1))\n",
    "#         model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "#         return model\n",
    "\n",
    "#     def train(self, X_train, y_train):\n",
    "#         X_train = np.reshape(X_train.values, (X_train.shape[0], 1, X_train.shape[1]))\n",
    "#         self.model.fit(X_train, y_train, epochs=self.epochs, batch_size=self.batch_size)\n",
    "\n",
    "#     def predict(self, X_test):\n",
    "#         X_test = np.reshape(X_test.values, (X_test.shape[0], 1, X_test.shape[1]))\n",
    "#         return self.model.predict(X_test)\n",
    "\n",
    "#     def evaluate(self, y_test, predictions):\n",
    "#         mse = mean_squared_error(y_test, predictions)\n",
    "#         return mse\n",
    "    \n",
    "#     def shutdown_cluster(self):\n",
    "#         if self.cluster is not None:\n",
    "#             self.cluster.shutdown() \n",
    "\n",
    "# predictor = LSTMModel(input_size=(1, X_train.shape[1]))\n",
    "# predictor.train(X_train, y_train)\n",
    "# predictions = predictor.predict(X_test)\n",
    "# lstm_mse = predictor.evaluate(y_test, predictions)\n",
    "# print(f\"MSE of LSTM: {lstm_mse}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b276d78-b8f4-4b47-8195-959a261ec098",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract numpy arrays from DataFrames\n",
    "X_train_values = X_train.values\n",
    "X_test_values = X_test.values\n",
    "X_valid_values = X_valid.values\n",
    "\n",
    "# Reshape input data to 3D [samples, timesteps, features] so it'll fit the LSTM layer\n",
    "# Use a timestep of 1.\n",
    "X_train_reshaped = np.reshape(X_train_values, (X_train_values.shape[0], 1, X_train_values.shape[1]))\n",
    "X_test_reshaped = np.reshape(X_test_values, (X_test_values.shape[0], 1, X_test_values.shape[1]))\n",
    "X_valid_reshaped = np.reshape(X_valid_values, (X_valid_values.shape[0], 1, X_valid_values.shape[1]))\n",
    "\n",
    "# LSTM model\n",
    "lstm = Sequential()\n",
    "lstm.add(LSTM(50, return_sequences=True, input_shape=(X_train_reshaped.shape[1], X_train_reshaped.shape[2])))\n",
    "lstm.add(Dropout(0.2))\n",
    "lstm.add(LSTM(50, return_sequences=False))\n",
    "lstm.add(Dropout(0.2))\n",
    "lstm.add(Dense(1))  # Prediction of the next closing value\n",
    "\n",
    "# Compile the model\n",
    "lstm.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Train the model\n",
    "lstm.fit(X_train_reshaped, y_train, epochs=50, batch_size=32, validation_data=(X_valid_reshaped, y_valid), verbose=1)\n",
    "\n",
    "# Predicting and inverse transforming the predictions \n",
    "y_pred = lstm.predict(X_test_reshaped)\n",
    "y_pred = scaler.inverse_transform(y_pred)\n",
    "\n",
    "# Calculate MSE for evaluation\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "print(f\"Mean Squared Error: {mse}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b48173f2-f4ce-4b35-8284-940f14fbb7a3",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Financial News Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8117df20-5857-4119-b205-1b87b2d78d6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/huggingface/transformers\n",
    "!pip install transformers\n",
    "!pip install tensorflow_probability==0.12.2\n",
    "!pip install contractions\n",
    "!pip install emoji\n",
    "!pip install emoticon_fix\n",
    "!pip install -U accelerate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a835e05-fa2c-4056-8d4e-b28862e8d958",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a pipeline as a high-level helper\n",
    "# from transformers import pipeline\n",
    "# pipe_sentiment = pipeline(\"text-classification\", model=\"mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis\")\n",
    "import contractions\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, PunktSentenceTokenizer\n",
    "from nltk.stem import PorterStemmer, SnowballStemmer, LancasterStemmer, WordNetLemmatizer\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "nltk.download('stopwords', quiet=True)\n",
    "nltk.download('wordnet', quiet=True)\n",
    "nltk.download('punkt', quiet=True)\n",
    "# nltk.download('omw-1.4')\n",
    "import re\n",
    "!pip install spacy\n",
    "import spacy\n",
    "import string\n",
    "import emoji\n",
    "from emoticon_fix import emoticon_fix\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b70d829-5c66-482f-96bf-5c622c251e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/Mydrive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f27c5d5-b06c-47c6-9942-395060894a49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df1=pd.read_csv('/content/Mydrive/MyDrive/raw_partner_headlines.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c47c0d9-252a-4475-bb13-a08684c2f4a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1_copy = df1.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23bb917f-6c6b-402c-94d9-ab25b7af21f6",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 1. Pre-trained Hugging Face Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "299b0172-8a36-4140-8dee-82215930cc0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a pipeline as a high-level helper\n",
    "from transformers import pipeline\n",
    "\n",
    "pipe_sentiment = pipeline(\"text-classification\", model=\"mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd35fc17-448a-4168-b93d-ddbb4d04ccc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = df1[(df1['date'] > '2017-01-01') & (df1['date'] < '2017-12-31')]\n",
    "df1 = df1.dropna(axis=0)\n",
    "df1.drop_duplicates(subset='headline', keep='first', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efa91219-b2a2-44e0-a387-1eb50c63e7a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1.columns\n",
    "df1 = df1.dropna(axis=0)\n",
    "df1.drop_duplicates(subset='headline', keep='first', inplace=True)\n",
    "df1.reset_index(drop=True)\n",
    "df1 = df1.reset_index()\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb8307ad-8b44-4d25-8273-e5f0a3c12925",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_paragraphs_from_url(url):\n",
    "    import requests\n",
    "    from bs4 import BeautifulSoup\n",
    "    # Send an HTTP GET request to the URL\n",
    "    response = requests.get(url)\n",
    "\n",
    "    if response.status_code == 200:\n",
    "        # Parse the HTML content using BeautifulSoup\n",
    "        soup = BeautifulSoup(response.text, 'html.parser')\n",
    "\n",
    "        # Find all the <p> elements and extract their text\n",
    "        paragraphs = [p.get_text() for p in soup.find_all('p')]\n",
    "\n",
    "        # Join the paragraphs into a single text\n",
    "        extracted_text = '\\n'.join(paragraphs)\n",
    "\n",
    "        return extracted_text\n",
    "    else:\n",
    "        flag=1\n",
    "        return flag\n",
    "\n",
    "# Example usage\n",
    "url = \"https://seekingalpha.com/article/4133429-agilent-technologies-good-business-total-return-pricey?source=partner_benzinga\"  # Replace with the URL you want to extract data from\n",
    "extracted_data = extract_paragraphs_from_url(url)\n",
    "\n",
    "if extracted_data:\n",
    "    # Print or store the extracted data as a paragraph\n",
    "    print(extracted_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ff65423-4622-458f-83cd-7cbbcdf20cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_data(text, lang):\n",
    "    # remove HTML\n",
    "    soup = BeautifulSoup(text, 'lxml')\n",
    "    text = soup.get_text()\n",
    "    #Replace Emoticon/Emoji with Text\n",
    "    text = emoji.demojize(text, language = lang )\n",
    "    text = emoticon_fix.emoticon_fix(text)\n",
    "    #6Decoding of abbreviations\n",
    "#     text = abbr_conversion(text)\n",
    "    # remove mentions\n",
    "    text = re.sub(\"@[A-Za-z0-9]+\",\"\", text)\n",
    "    # remove hashtags\n",
    "    text = re.sub(\"#[A-Za-z0-9_]+\",\"\", text)\n",
    "    # remove links\n",
    "    text = re.sub('https:\\/\\/\\S+', '', text)\n",
    "    # remove punctuation\n",
    "    text = re.sub('[%s]' % re.escape(string.punctuation), ' ', text)\n",
    "    # remove next line\n",
    "    text = re.sub(r'[^ \\w\\.]', '', text)\n",
    "    # remove words containing numbers\n",
    "    text = re.sub('\\w*\\d\\w*', '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5d832d3-b8cf-4ea9-9cfe-aa531aba9e1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_data = clean_data(extracted_data,'eng')\n",
    "cleaned_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3c3c676-6d44-42c2-b9d6-25b98250aca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1[\"Labels\"] = np.nan\n",
    "df1[\"Scores\"] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef2128b8-42c2-48a9-bacf-e0f542b4ea90",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1.columns\n",
    "df1 = df1.drop(['index','Unnamed: 0'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff40ca52-c2a5-4572-822b-2f90a308781d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # start_time = time.time()\n",
    "# for i in range(0,len(df1)):\n",
    "#     try:\n",
    "#         extracted_data = extract_paragraphs_from_url(df1['url'][i])\n",
    "#         if extracted_data == 1:\n",
    "#             extracted_data = clean_data(df1['headline'][i],'eng')\n",
    "#             sentiment1 = pipe_sentiment(extracted_data)\n",
    "#             df1['Labels'][i] = sentiment1[0]['label']\n",
    "#             df1['Scores'][i] = sentiment1[0]['score']\n",
    "#         else:\n",
    "#             cleaned_data = clean_data(extracted_data,'eng')\n",
    "#             sentiment2 = pipe_sentiment(cleaned_data,truncation=True)\n",
    "#             df1['Labels'][i] = sentiment2[0]['label']\n",
    "#             df1['Scores'][i] = sentiment2[0]['score']\n",
    "#         if i%100==0:\n",
    "#             df1.to_csv('Set.csv')\n",
    "\n",
    "#     except:\n",
    "#         pass\n",
    "# # print(\"--- %s seconds ---\" % (time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e008af7f-2f96-4613-b1d7-19aa6d8abbf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1.to_csv('Final_ouput.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d959e9a6-28fd-4d26-aa88-87212c278eac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# new_df = pd.read_csv('/content/Mydrive/MyDrive/Final_ouput.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0e7cae2-6f17-41fb-992d-85b13a9c1d7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# new_df = new_df.dropna(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfeb3cef-9b3c-4940-b70a-8f45248bdb11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# new_df.to_csv(\"SentimentAnalysis.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a29f875a-938b-4186-b330-1791dc456355",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2. Naive Bayes Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e353953-9471-4cb3-9be7-3e6ee5414707",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U spacy==3.*\n",
    "!python -m spacy download en_core_web_sm\n",
    "!python -m spacy info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "743fc8cb-1b64-4d5b-ac7e-8df88e27acd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, PunktSentenceTokenizer\n",
    "from nltk.stem import PorterStemmer, SnowballStemmer, LancasterStemmer, WordNetLemmatizer\n",
    "\n",
    "nltk.download('stopwords', quiet=True)\n",
    "nltk.download('wordnet', quiet=True)\n",
    "nltk.download('punkt', quiet=True)\n",
    "from nltk.corpus import stopwords\n",
    "import collections\n",
    "import spacy\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "from sklearn.metrics import RocCurveDisplay\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.sparse import coo_matrix\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f566f336-6fc8-40d6-a5cb-bc826a41430c",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('/content/Mydrive/MyDrive/SentimentAnalysis.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80b9f66e-2a07-4e16-9346-43ea53e54d2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_copy = data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56116a2d-fd75-4fe9-971b-6bc76f5bf337",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddafa437-468c-45a2-bcfa-42ebcf0d5fd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8baa1d21-3b2a-4139-b57c-799be30585ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.blank('en')\n",
    "\n",
    "# There should be no pipeline components.\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "130f1b15-7443-4deb-8bd5-81446abd240e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def spacy_tokenizer(doc):\n",
    "    return [t.text for t in nlp(doc) if \\\n",
    "            not t.is_punct and \\\n",
    "            not t.is_space and \\\n",
    "            t.is_alpha]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df1564d7-5652-4df6-b65d-22469700b1d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = LabelEncoder()\n",
    "labels = data['Labels'].values\n",
    "encoded_labels = encoder.fit_transform(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c61e6a83-feda-469a-bca8-8bfde20ef7c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(data['headline'], encoded_labels, stratify = encoded_labels,train_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7567d0fb-0eb0-4b41-9118-e4cac8942205",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "vectorizer = TfidfVectorizer(tokenizer=spacy_tokenizer)\n",
    "train_feature_vects = vectorizer.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68268ef4-6221-4a0c-8ba7-6f85f1cafd4f",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Train Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ce671a1-013a-4abd-b764-e123734bc914",
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_classifier = MultinomialNB()\n",
    "nb_classifier.fit(train_feature_vects, y_train)\n",
    "nb_classifier.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73086627-f38b-4c94-8e44-05087035a685",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_preds = nb_classifier.predict(train_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aefdb1ad-bbab-4ab7-a683-ece18d325d41",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_preds_score = nb_classifier.predict_proba(train_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4224926a-5758-42c6-aa59-4982a7edb583",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy score on  train set: {}'.format(metrics.accuracy_score(y_train, train_preds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5b70c8f-5510-481e-95d3-74b05973d99b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('F1 score on train set: {}'.format(metrics.f1_score(y_train, train_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72969000-67dd-41d3-8fdb-ef21d814ec4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Precision score on train set: {}'.format(metrics.precision_score(y_train, train_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bee7c7b2-20ef-4cfe-b8b2-d3db8725e38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Recall score on train set: {}'.format(metrics.recall_score(y_train, train_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0c76819-279b-4336-8cc7-fbb22d5c6fa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('ROC-AUC score on train set: {}'.format(metrics.roc_auc_score(y_train,train_preds_score, average='weighted',multi_class='ovr')))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52033ade-02ab-43b3-87fb-73d06ac9717b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "690aaf56-63c7-4974-8b29-236d5a9a024d",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_feature_vects = vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c4b6743-32fc-44ef-96e4-6e016939857f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = nb_classifier.predict(test_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fef31fb-037d-4a87-9888-b074267deaa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds_score = nb_classifier.predict_proba(test_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6be684ff-7661-4df5-b49c-aa4d3408946b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy score on test set: {}'.format(metrics.accuracy_score(y_test, test_preds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "480359ac-7744-458d-811d-92e1cd7b5f99",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('F1 score on test set: {}'.format(metrics.f1_score(y_test, test_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f1ce51c-c7a5-49d0-b034-22c5f4337aad",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Precision score on test set: {}'.format(metrics.precision_score(y_test, test_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23fad1f1-f665-466a-bee7-d5285084c177",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Recall score on test set: {}'.format(metrics.recall_score(y_test, test_preds, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "049499f0-01f3-4e4f-a29e-e6211920c9de",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('ROC-AUC score on test set: {}'.format(metrics.roc_auc_score(y_test, test_preds_score, average='weighted',multi_class='ovr')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc4c2ecf-d173-4e12-a68f-776dcb3a5c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr_NB = metrics.classification_report(y_test, test_preds)\n",
    "print(\"\\n\\nClassification Report\\n\")\n",
    "print(cr_NB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dcc7121-e389-45ca-9461-9f11beb8ca8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm_NB = metrics.confusion_matrix(y_test, test_preds)\n",
    "print(\"Confusion Matrix\\n\")\n",
    "print(cm_NB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f4d202f-4257-4fbc-8adc-638b2a5e0310",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.heatmap(cm_NB, annot=True,cmap='Blues')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c941b940-3881-4e8e-b078-94b3859f116c",
   "metadata": {},
   "source": [
    "##### This part of the code is commented which was made to run once for collecting csv with aggregated results of labels and scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ae7daa5-7a43-4db3-98b7-fe934765c8cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1_copy['Labels'] = np.nan\n",
    "# df1_copy['Scores'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3bb6b79-3e92-49e7-8c61-58501da1d5d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(10,len(X_train)):\n",
    "#   j=df1_copy.index[df1_copy['headline']==X_train.iloc[i]].tolist()[0]\n",
    "#   df1_copy['Labels'][j] = train_preds[i]\n",
    "#   df1_copy['Scores'][j] = np.max(train_preds_score[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e50af923-2903-4eae-9512-39557c5129de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(0,len(X_test)):\n",
    "#   j=df1_copy.index[df1_copy['headline']==X_test.iloc[i]].tolist()[0]\n",
    "#   df1_copy['Labels'][j] = test_preds[i]\n",
    "#   df1_copy['Scores'][j] = np.max(test_preds_score[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31895cd2-b007-4f54-a6bb-2bfaff47c3d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1.to_csv('Final-NB.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "086c6655-39ab-47b8-b123-ad2878e5ef59",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 3. KNN Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32071d20-846a-48f9-a762-21ce385d6220",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "model_knn = KNeighborsClassifier(n_neighbors=22,metric='cosine',weights='distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b365e1d-59f4-4328-931d-9530e1dd90f4",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Train Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30caa15e-d5a2-474f-ac73-ef1023a0faf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_knn.fit(train_feature_vects,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f6b1695-8100-4f71-a32c-f84eda0dd10a",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_train_knn = model_knn.predict(train_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d779813-2a4e-404c-a898-b901cf673347",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_train_scores_knn = model_knn.predict_proba(train_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88cf1bc3-9bbb-47fa-b082-0b21d1eee27e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy score on train set: {}'.format(metrics.accuracy_score(y_train,pred_train_knn)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b18aa5a0-453a-4565-8a4d-d382fdc2c8a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('F1 score on train set: {}'.format(metrics.f1_score(y_train,pred_train_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afc6147e-10fc-43a9-ba74-51e62e784b29",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Precision on train set: {}'.format(metrics.precision_score(y_train,pred_train_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "565b032f-45f0-4115-8f3a-d115915c6228",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Recall on train set: {}'.format(metrics.recall_score(y_train,pred_train_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80cf19c4-795c-4331-b131-d0f28e7d3c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('ROC-AUC score on train set: {}'.format(metrics.roc_auc_score(y_train,pred_train_scores_knn, average='weighted',multi_class='ovr')))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b9ff49-c44d-4499-91d9-dff87fb01839",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6e1f658-cf73-409c-b61e-c4e325970e5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_test_knn = model_knn.predict(test_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e1f23ea-6ceb-49c0-bcfc-282f55cb6e96",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_test_scores_knn = model_knn.predict_proba(test_feature_vects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9495cfa7-5f30-47a2-8119-ed1a03ef2936",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy score on test set: {}'.format(metrics.accuracy_score(y_test,pred_test_knn)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af9c2f47-ab0d-46e5-b463-4f63fc72fcf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('F1 score on test set: {}'.format(metrics.f1_score(y_test,pred_test_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e698e158-861e-40a1-8aae-58338bb2a753",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Precision on test set: {}'.format(metrics.precision_score(y_test,pred_test_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6d739c9-e586-411c-8b8a-a6735bd1dd0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Recall on test set: {}'.format(metrics.recall_score(y_test,pred_test_knn, average='weighted')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c41f81-68fb-4be4-82fe-be2dffc3142d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('AUC score on test set: {}'.format(metrics.roc_auc_score(y_test,pred_test_scores_knn, average='weighted',multi_class='ovr')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31be7c26-a16a-4cc3-864b-b08f19240d0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr_KNN = metrics.classification_report(y_test,pred_test_knn)\n",
    "print(\"\\n\\nClassification Report\\n\")\n",
    "print(cr_KNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea5f142e-d6be-4dd9-8a5a-4cb1e438bb0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm_KNN = metrics.confusion_matrix(y_test,pred_test_knn)\n",
    "print(\"Confusion Matrix\\n\")\n",
    "print(cm_KNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d003c39f-7a42-42ca-903f-c22613a336ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.heatmap(cm_KNN, annot=True,cmap='Blues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39acc8a9-bef3-4e80-8331-1d4a7adb04f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1_copy['Labels'] = np.nan\n",
    "# df1_copy['Scores'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e582e25d-dc4c-47f6-8b45-ce5380eac58e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(10,len(X_train)):\n",
    "#   j=df1_copy.index[df1_copy['headline']==X_train.iloc[i]].tolist()[0]\n",
    "#   df1_copy['Labels'][j] = pred_train_knn[i]\n",
    "#   df1_copy['Scores'][j] = np.max(pred_train_scores_knn[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3cc20cb-18e2-4b18-9cc2-24d1ec9fe876",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(0,len(X_test)):\n",
    "#   j=df1_copy.index[df1_copy['headline']==X_test.iloc[i]].tolist()[0]\n",
    "#   df1_copy['Labels'][j] = pred_test[i]\n",
    "#   df1_copy['Scores'][j] = np.max(pred_test_scores_knn[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6449645c-8416-4c93-b40a-d056bf6c56de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1.to_csv('Final-KNN.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb362ad-61bc-4870-b706-89995ab04300",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6c9414-05f9-49af-8a79-95c9be23913e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e73a875-b2cc-4d88-b9c9-817dbf63d141",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_binarizer1 = LabelBinarizer().fit(y_train)\n",
    "y_onehot_test = label_binarizer1.transform(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d644908-68f7-4333-aa76-d02995cdaf4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_binarizer1.transform([1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a605a7b5-568d-40b9-8507-89867a4faf59",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_of_interest0 = 0\n",
    "class_of_interest1 = 1\n",
    "class_of_interest2= 2\n",
    "class_id0 = np.flatnonzero(label_binarizer1.classes_ == 0)[0]\n",
    "class_id1 = np.flatnonzero(label_binarizer1.classes_ == 1)[0]\n",
    "class_id2 = np.flatnonzero(label_binarizer1.classes_ == 2)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3caad41c-9f09-471c-8660-b38ce92aa1d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id0],\n",
    "    test_preds_score[:, class_id0],\n",
    "    name=f\"{class_of_interest0} vs the rest\",\n",
    "    color=\"darkorange\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nNegative vs (Neutral& Positive)\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id1],\n",
    "    test_preds_score[:, class_id1],\n",
    "    name=f\"{class_of_interest1} vs the rest\",\n",
    "    color=\"green\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nNeutral vs (Negative& Positive)\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id2],\n",
    "    test_preds_score[:, class_id2],\n",
    "    name=f\"{class_of_interest2} vs the rest\",\n",
    "    color=\"blue\"\n",
    "\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nPositive vs (Negative& Neutral)\")\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f262b8ae-5fae-4954-a4a3-a1184921f33d",
   "metadata": {},
   "outputs": [],
   "source": [
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test.ravel(),\n",
    "    test_preds_score.ravel(),\n",
    "    name=\"micro-average OvR\",\n",
    "    color=\"darkorange\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"Micro-averaged One-vs-Rest\\nReceiver Operating Characteristic\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78369320-5a6f-4fa4-bc0d-b9227b26df08",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46386506-81ae-4c58-9b6b-3a2e153799f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id0],\n",
    "    pred_test_scores_knn[:, class_id0],\n",
    "    name=f\"{class_of_interest0} vs the rest\",\n",
    "    color=\"darkorange\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nNegative vs (Neutral& Positive)\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id1],\n",
    "    pred_test_scores_knn[:, class_id1],\n",
    "    name=f\"{class_of_interest1} vs the rest\",\n",
    "    color=\"green\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nNeutral vs (Negative& Positive)\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test[:, class_id2],\n",
    "    pred_test_scores_knn[:, class_id2],\n",
    "    name=f\"{class_of_interest2} vs the rest\",\n",
    "    color=\"blue\"\n",
    "\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"One-vs-Rest ROC curves:\\nPositive vs (Negative& Neutral)\")\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d15aa4cb-1a2f-4b54-9f53-3e18b5f76745",
   "metadata": {},
   "outputs": [],
   "source": [
    "RocCurveDisplay.from_predictions(\n",
    "    y_onehot_test.ravel(),\n",
    "    pred_test_scores_knn.ravel(),\n",
    "    name=\"micro-average OvR\",\n",
    "    color=\"darkorange\"\n",
    ")\n",
    "plt.axis(\"square\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"Micro-averaged One-vs-Rest\\nReceiver Operating Characteristic\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f7b9cb0-acd5-4dbd-b2db-67ffd4a7f9e5",
   "metadata": {},
   "source": [
    "#### Comparison of predictions between HuggingFace,Naive Bayes and KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f30591c0-c8e2-4caf-9f3f-bc2bec1dd3e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2)\n",
    "dense_matrix_test = test_feature_vects.toarray()\n",
    "pca_x_test = pca.fit_transform(dense_matrix_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "305553d1-eaf6-454f-9778-489430a237df",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(1, 3, 1)\n",
    "new_list_x=[]\n",
    "new_list_y=[]\n",
    "for j in range(0,len(y_test)):\n",
    "    if y_test[j]==0:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Negative'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,), edgecolor='k',s=40)   \n",
    "plt.legend()\n",
    "plt.xlabel(\"Features reduced\",fontsize=10)\n",
    "plt.ylabel(\"Features reduced\",fontsize=10)\n",
    "plt.title(\"Negative HF\")\n",
    "\n",
    "\n",
    "plt.subplot(1,3,2)\n",
    "for j in range(0,len(test_preds)):\n",
    "    if y_test[j]==0:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Negative'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Negative NB\")\n",
    "\n",
    "\n",
    "plt.subplot(1,3,3)\n",
    "for j in range(0,len(pred_test_knn)):\n",
    "    if y_test[j]==0:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Negative'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Negative KNN\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd4997d1-811e-4419-ba45-882e87f44e93",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(1, 3, 1)\n",
    "new_list_x=[]\n",
    "new_list_y=[]\n",
    "for j in range(0,len(y_test)):\n",
    "    if y_test[j]==1:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Neutral'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,), edgecolor='k',s=40)   \n",
    "plt.legend()\n",
    "plt.xlabel(\"Features reduced\",fontsize=10)\n",
    "plt.ylabel(\"Features reduced\",fontsize=10)\n",
    "plt.title(\"Neutral HF\")\n",
    "\n",
    "\n",
    "plt.subplot(1,3,2)\n",
    "for j in range(0,len(test_preds)):\n",
    "    if y_test[j]==1:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Neutral'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Neutral NB\")\n",
    "\n",
    "plt.subplot(1,3,3)\n",
    "for j in range(0,len(pred_test_knn)):\n",
    "    if y_test[j]==1:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Neutral'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Neutral KNN\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf862781-eeb5-4c6c-bad0-82c7dca49cab",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(1, 3, 1)\n",
    "new_list_x=[]\n",
    "new_list_y=[]\n",
    "for j in range(0,len(y_test)):\n",
    "    if y_test[j]==2:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Positive'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,), edgecolor='k',s=40)   \n",
    "plt.legend()\n",
    "plt.xlabel(\"Features reduced\",fontsize=10)\n",
    "plt.ylabel(\"Features reduced\",fontsize=10)\n",
    "plt.title(\"Positive HF\")\n",
    "\n",
    "\n",
    "plt.subplot(1,3,2)\n",
    "for j in range(0,len(test_preds)):\n",
    "    if y_test[j]==2:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Positive'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Positive NB\")\n",
    "\n",
    "plt.subplot(1,3,3)\n",
    "for j in range(0,len(pred_test_knn)):\n",
    "    if y_test[j]==2:\n",
    "        new_list_x.append(pca_x_test[j][0])\n",
    "        new_list_y.append(pca_x_test[j][1])\n",
    "        label_str = 'Positive'\n",
    "plt.scatter(new_list_x,new_list_y,label=f'{label_str}', c=np.random.rand(3,),edgecolor='k',s=30)   \n",
    "plt.legend()\n",
    "plt.title(\"Positive KNN\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97d994d7-c54d-43e6-8f30-f3107c039165",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8041d20-8ab1-4b93-b780-5838e9206267",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "819325b5-701a-4cb4-b58d-03f772c44578",
   "metadata": {},
   "source": [
    "## Content-based Collaborative Filtering (PHASE-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c20bcb43-1d51-4bec-a74c-83539c5335b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment_df = pd.read_csv('Final-KNN.csv')\n",
    "print(sentiment_df.columns)\n",
    "\n",
    "# Drop unwanted columns\n",
    "unwanted_columns = ['Unnamed: 0.1', 'index', 'Unnamed: 0', 'headline', 'url', 'publisher']\n",
    "sentiment_df = sentiment_df.drop(columns=unwanted_columns)\n",
    "sentiment_df.head()\n",
    "\n",
    "# Extract only yyyy-mm-dd\n",
    "sentiment_df['date'] = pd.to_datetime(sentiment_df['date']).dt.date\n",
    "\n",
    "# Sort df by 'ticker', 'date', and 'Scores' in descending order\n",
    "sentiment_df = sentiment_df.rename(columns={'stock': 'ticker'})\n",
    "sentiment_df = sentiment_df.sort_values(by=['ticker', 'date', 'Scores'], ascending=[True, True, False])\n",
    "\n",
    "# Get the highest score for each unique 'ticker' and 'date' combination\n",
    "sentiment_df = sentiment_df.drop_duplicates(subset=['ticker', 'date'], keep='first')\n",
    "sentiment_df = sentiment_df.reset_index(drop=True)\n",
    "sentiment_df.columns = sentiment_df.columns.str.lower()\n",
    "print(sentiment_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad276ce1-b94a-469d-941e-91358e18573f",
   "metadata": {},
   "outputs": [],
   "source": [
    "stock_df = pd.read_csv('pca_stock_result.csv')\n",
    "stock_df['Date'] = pd.to_datetime(stock_df['Date'])\n",
    "stock_df.set_index('Date', inplace=False)\n",
    "# Rename columns to lowercase\n",
    "stock_df = stock_df.rename(columns={\n",
    "    'Feature_1': 'feature_1',\n",
    "    'Feature_2': 'feature_2',\n",
    "    'Feature_3': 'feature_3',\n",
    "    'Feature_4': 'feature_4',\n",
    "    'Feature_5': 'feature_5',\n",
    "    'Date': 'date',\n",
    "    'Ticker': 'ticker',\n",
    "    'Close': 'close'\n",
    "})\n",
    "stock_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae56f2f-c87d-487e-96bc-2e90d6f7cba9",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Feature Representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26f51dc2-2b91-4160-bad7-2862f105343d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify the tickers match between data sets\n",
    "stock_tickers = stock_df['ticker'].unique()\n",
    "sentiment_tickers = sentiment_df['ticker'].unique()\n",
    "set(stock_tickers) == set(sentiment_tickers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0338b9e5-49a0-4c82-947c-cf67877e99b4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "stock_tickers = set(stock_df['ticker'].unique())\n",
    "sentiment_tickers = set(sentiment_df['ticker'].unique())\n",
    "\n",
    "missing_in_stock = stock_tickers.difference(sentiment_tickers)\n",
    "missing_in_sentiment = sentiment_tickers.difference(stock_tickers)\n",
    "\n",
    "print(\"Tickers missing in stock_df but present in sentiment_df:\", missing_in_stock)\n",
    "print(\"Tickers missing in sentiment_df but present in stock_df:\", missing_in_sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "103ccb41-a49e-4047-adcf-de59b7feb3bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find tickers in sentiment_df that are not in sentiment_df and delete those tickers from stock_df\n",
    "missing_tickers = set(stock_tickers) - set(sentiment_tickers)\n",
    "stock_df = stock_df[~stock_df['ticker'].isin(missing_tickers)]\n",
    "stock_df = stock_df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8d44faa-6fb8-4ec4-b266-15e0dc8cf0da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform an left merge on 'ticker' and 'date' columns, filling missing sentiment with NaN\n",
    "sentiment_df['date'] = pd.to_datetime(sentiment_df['date'])\n",
    "stock_df['date'] = pd.to_datetime(stock_df['date'])\n",
    "product_df = pd.merge(stock_df, sentiment_df, on=['ticker', 'date'], how='left')\n",
    "print(product_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "541e3831-f1e2-4127-a82b-369fe3614c89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill NaN values in the 'sentiment' column with forward filling\n",
    "product_df.sort_values(by=['ticker', 'date'], inplace=True)\n",
    "product_df['labels'] = merged_df.groupby('ticker')['labels'].fillna(method='ffill')\n",
    "product_df['scores'] = merged_df.groupby('ticker')['scores'].fillna(method='ffill')\n",
    "product_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8346367b-2b4b-4f4c-b907-5441f1b0aefc",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df['labels'].isnull().sum() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba93b3b2-aac7-4f0d-9f7b-2edf63fcd4e3",
   "metadata": {},
   "source": [
    "Since we use `filling forward` to fill missing `labels` and `scores`, meanwhile, in the very beginning of that ticker, if there was no data for sentiment, these missing values remaining NAN. We decide to remove 21,553 missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b80f6ee2-e9f7-4601-bd5b-6544b99c3877",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df = product_df.dropna()\n",
    "product_df = product_df.reset_index()\n",
    "product_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88c8ec2d-6420-41d7-9713-927349d82599",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df['date'] = pd.to_datetime(product_df['date'])\n",
    "# Label encoding for 'ticker'\n",
    "# label_encoder = LabelEncoder()\n",
    "# merged_df['ticker_encoded'] = label_encoder.fit_transform(merged_df['ticker'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260655e0-e67b-4491-a2de-14d31ce5a723",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df.drop(['index'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a5cb851-5fd8-4007-bff3-7dfec21a83a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da67dcfc-89ed-440a-a38c-936053fcc47a",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict_df = product_df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f131201-3b18-4dc1-a519-15b4f733295b",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict_df.to_csv(\"Data-Price-Pred-Testing.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04081846-74b6-433f-8fc9-10532ee8750e",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df.drop(['ticker'], axis=1, inplace=True)\n",
    "product_df.drop(['date'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cb63b66-0031-4880-bcf3-f7c9582452da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to numpy array\n",
    "ticker_features = np.array(product_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d314b51b-61b6-452a-8031-95d9f456a17b",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c485ba8d-5a93-418e-9765-677bba568407",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Similarity Matrix\n",
    "Calculate the similarity between stocks based on the PCA-transformed features and embedding sentiment features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c22cc97-a22b-438a-8245-0b121da5ad72",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate cosine similarity matrix\n",
    "cosine_sim_matrix = cosine_similarity(ticker_features[:, 1:], ticker_features[:, 1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21902b7d-5112-4c7d-a7bf-0cac89424c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate linear kernel \n",
    "kernel_matrix = linear_kernel(ticker_features[:, 1:], ticker_features[:, 1:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5f20b83-7b45-48b6-be44-675c2441df29",
   "metadata": {},
   "source": [
    "### Candidate Generation\n",
    "Rank stocks based on their similarity to the user's profile vector, as discussed earlier. Stocks with higher similarity scores are recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6848460-6c67-43ad-aef4-98f6baec4783",
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim_matrix = pd.read_csv(\"Cosine-similarity.csv\")\n",
    "cosine_sim_matrix.drop(columns=['Unnamed: 0'], inplace=True)\n",
    "kernel_matrix = pd.read_csv(\"kernel-matrix.csv\")\n",
    "kernel_matrix.drop(columns=['Unnamed: 0'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24611f65-553c-43c4-827c-772dac9d4216",
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim_matrix.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "580c26fb-0081-4e67-aa07-a31454ab1e57",
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim_matrix = cosine_sim_matrix.to_numpy()\n",
    "cosine_sim_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e36753-db0e-4257-a8b8-d97964913e11",
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_matrix = kernel_matrix.to_numpy()\n",
    "kernel_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ace263f5-49d3-46e6-b53c-df55a389f97b",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(cosine_sim_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ee4581f-bfff-4130-837d-eb906103d1d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output columns\n",
    "candidate_info = merged_df.reset_index(drop=True)\n",
    "titles = merged_df[['Product', 'Ingredients', 'Rating']]\n",
    "indices = pd.Series(df_cont.index, index=df_cont['Product'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "351c956b-6318-4e41-b860-a09645746012",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc83d967-c8c4-4731-8601-ed5e850b1b6f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "58a6c5fe-8466-4435-9ebf-989fbcfc220a",
   "metadata": {},
   "source": [
    "**Complete work in a function with pre-computed similarity matrix**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99e06220-4fa2-42b0-90b3-97afb0ae70c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_recommendation(ticker, top_n):\n",
    "    \"\"\"\n",
    "    Given a stock ticker, recommend the top n most similar stocks.\n",
    "    \"\"\"\n",
    "    \"\"\"\n",
    "    Parameters:\n",
    "    ticker: Stock ticker symbol\n",
    "    top_n: Number of recommendations to return\n",
    "    ------------\n",
    "    Returns: List of top n recommended stock ticker symbols\n",
    "    \"\"\"\n",
    "    # Validate inputs\n",
    "    if top_n < 1:\n",
    "        print(\"Invalid top_n, must be >= 1\")\n",
    "        return\n",
    "\n",
    "    # Check if ticker exists\n",
    "    if ticker not in product_df['ticker'].values:\n",
    "        print(f\"Ticker {ticker} not found in data\")\n",
    "        return\n",
    "\n",
    "    # Get index of ticker in similarity matrix\n",
    "    idx = list(product_df['ticker'].unique()).index(ticker)\n",
    "\n",
    "    # Lookup similarity scores for this ticker\n",
    "    sim_scores = similarity_matrix[idx]\n",
    "  \n",
    "    # Sort by similarity and take top n\n",
    "    sort_idx = np.argsort(sim_scores)[-1:-(top_n+1):-1]\n",
    "    top_tickers = merged_df['ticker'].unique()[sort_idx]\n",
    "  \n",
    "    return top_tickers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00ec7c66-a67a-4ef4-8764-20d81fc9218c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing\n",
    "print()\n",
    "user_input = input(f\"Enter ticker to get recommendations: \")\n",
    "top_n = input(f\"Insert number of your list (must be integer >= 1): \")\n",
    "generate_recommendation(user_input, top_n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66773d63-bc93-4f99-9ce9-8229c32bc300",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "16ed26f5-d3ba-4aae-909e-5712a8f4fa3d",
   "metadata": {},
   "source": [
    "## Reference:\n",
    "\n",
    "    [1] StandardScaler, MinMaxScaler and RobustScaler techniques - ML. (2020, July 15). GeeksforGeeks. https://www.geeksforgeeks.org/standardscaler-minmaxscaler-and-robustscaler-techniques-ml/\n",
    "    \n",
    "    \n",
    "    [2] https://towardsdatascience.com/why-does-stationarity-matter-in-time-series-analysis-e2fb7be74454\n",
    "\n",
    "    [3] Zheng, X., & XIONG, N. (2022). Stock price prediction based on PCA-LSTM model. https://doi.org/10.1145/3545839.3545852\n",
    "\n",
    "    [5] https://developers.google.com/machine-learning/recommendation/content-based/basics\n",
    "    \n",
    "    [6] Gulli, A. (2016, March 26). Complete guide to parameter tuning in XGBoost (with codes in Python). Analytics Vidhya. https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/#:~:text=XGBoost%20provides%20L1%20and%20L2,the%20objective%20function%20during%20training.\n",
    "\n",
    "    [7] https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis?library=true\n",
    "\n",
    "    [8] github.com/MD-Ryhan/NLP-Preprocesing/blob/main/NLP_Preprocessing.ipynbanva.com/download/mac/\n",
    "\n",
    "    [9] https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis?text=Morgan+Asset+Management+announces+the+liquidations+and+dissolving+six+exchange-traded+funds.+The+funds+include+among+them+the+popular+but+low-yielding+Diversire+Return+Europe+equity+ETF+Jed%2C+Long+Term+Short+ET+Jed+%2C+JPMorgan+Long+Shadow+ET+JPeu+%2C+JPMorgan+Commodity+Fund+J&library=true\n",
    "\n",
    "    [10] https://huggingface.co/pszemraj/led-large-book-summary?text=PR+Newswire+NEW+YORK+++May+++++NEW+YORK+++May+++++++PRNewswire+++++++J+++P+++Morgan+Asset+Management+today+announced+the+upcoming+liquidation+and+dissolution+of+six+exchange+traded+funds+++JPMorgan+Diversified+Return+Europe+Equity+ETF+++JPEU+++++JPMorgan+Long+++Short+ETF+++JPLS+++++JPMorgan+Managed+Futures+Strategy+ETF+++JPMF+++++JPMorgan+Diversified+Return+Global+Equity+ETF+++JPGE+++++JPMorgan+Diversified+Alternatives+ETF+++JPHF+++++and+JPMorgan+Event+Driven+ETF+++JPED+++++collectively+++the+++Funds+++++++Shareholders+of+the+Funds+may+sell+their+holdings+of+each+Fund+on+NYSE+Arca+++Inc+++++++NYSE+Arca+++++until+market+close+on+the+designated+last+day+of+trading+++transaction+fees+from+their+broker+dealer+may+be+incurred+++++ETF+Name+Ticker+Last+Day+of+Trading+Liquidation+Date+JPMorgan+Diversified+Return+Europe+Equity+ETF+JPEU+++++++++++JPMorgan+Long+++Short+ETF+JPLS+++++++++++JPMorgan+Managed+Futures+Strategy+ETF+JPMF+++++++++++JPMorgan+Diversified+Return+Global+Equity+ETF+JPGE+++++++++++JPMorgan+Diversified+Alternatives+ETF+JPHF+++++++++++JPMorgan+Event+Driven+ETF+JPED+++++++++++Shares+of+JPEU+++JPLS+and+JPMF+will+stop+accepting+creation+orders+from+authorized+participants+after+the+close+on+June+++++++and+will+be+delisted+ahead+of+market+open+on+June+++++++Additionally+++shares+of+JPGE+++JPHF+++and+JPED+will+stop+accepting+creation+orders+from+authorized+participants+after+the+close+on+June+++++++and+will+be+delisted+ahead+of+market+open+on+June+++++++Shareholders+who+continue+to+hold+shares+of+any+of+the+Funds+on+the+Funds+++designated+aforementioned+liquidation+date+will+receive+a+liquidating+distribution+of+cash+in+the+cash+portion+of+their+brokerage+accounts+equal+to+the+amount+of+the+net+asset+value+of+their+shares+++++We+regularly+monitor+and+evaluate+our+product+lineup+as+market+and+economic+conditions+evolve+++++said+Bryon+Lake+++Head+of+Americas+ETF+for+J+++P+++Morgan+Asset+Management+++++This+process+allows+us+to+optimize+and+scale+our+product+offerings+to+better+meet+client+objectives+and+market+demand+++++Shareholders+who+receive+a+liquidating+distribution+generally+will+recognize+a+capital+gain+or+loss+equal+to+the+amount+received+for+their+shares+over+their+adjusted+basis+in+such+shares+if+shares+are+held+in+taxable+account+++and+should+consult+their+tax+advisor+about+the+potential+tax+consequences+++About+J+++P+++Morgan+Asset+Management+J+++P+++Morgan+Asset+Management+++with+assets+under+management+of+USD+++trillion+++as+of++March++++++is+a+global+leader+in+investment+management+++J+++P+++Morgan+Asset+Management+s+clients+include+institutions+++retail+investors+and+high+net+worth+individuals+in+every+major+market+throughout+the+world+++J+++P+++Morgan+Asset+Management+offers+global+investment+management+in+equities+++fixed+income+++real+estate+++hedge+funds+++private+equity+and+liquidity+++JPMorgan+Chase+++Co+++++NYSE+++JPM+++is+a+leading+global+financial+services+firm+with+assets+of+USD+++trillion+++as+of++December++++and+operations+worldwide+++J+++P+++Morgan+Asset+Management+is+the+marketing+name+for+the+asset+management+businesses+of+JPMorgan+Chase+++Co+++and+its+affiliates+worldwide+++J+++P+++Morgan+ETFs+are+distributed+by+JPMorgan+Distribution+Services+++Inc+++++which+is+an+affiliate+of+JPMorgan+Chase+++Co+++Affiliates+of+JPMorgan+Chase+++Co+++receive+fees+for+providing+various+services+to+the+funds+++JPMorgan+Distribution+Services+++Inc+++is+a+member+of+FINRA+++Investors+should+carefully+consider+the+investment+objectives+and+risks+as+well+as+charges+and+expenses+of+an+ETF+before+investing+++The+summary+and+full+prospectuses+contain+this+and+other+information+about+the+ETF+and+should+be+read+carefully+before+investing+++To+obtain+a+prospectus+++Call+++++++ETF+++NOT+FDIC+INSURED+++NO+BANK+GUARANTEE+++MAY+LOSE+VALUE+View+original+content+++http+++www+prnewswire+com+news+releases+jp+morgan+asset+management+announces+liquidation+of+six+exchange+traded+funds++html+SOURCE+J+++P+++Morgan+Asset+Management+We+d+love+to+learn+more+about+your+experiences+on+GuruFocus+com+and+how+we+can+improve\n",
    "\n",
    "    [11] https://wandb.ai/ivangoncharov/FinBERT_Sentiment_Analysis_Project/reports/Financial-Sentiment-Analysis-on-Stock-Market-Headlines-With-FinBERT-HuggingFace--VmlldzoxMDQ4NjM0#:~:text=Financial%20news%20headlines%20are%20a,positive%2C%20negative%2C%20and%20neutral\n",
    "\n",
    "    [12] https://huggingface.co/docs/transformers/main_classes/pipelines\n",
    "\n",
    "    [13] https://huggingface.co/models?pipeline_tag=summarization&sort=trending\n",
    "\n",
    "    [14] https://www.analyticsvidhya.com/blog/2022/03/building-naive-bayes-classifier-from-scratch-to-perform-sentiment-analysis/\n",
    "\n",
    "    [15] https://colab.research.google.com/github/futuremojo/nlp-demystified/blob/main/notebooks/nlpdemystified_classification_naive_bayes.ipynb#scrollTo=uJ2lYTdW3HXP\n",
    "\n",
    "    [16] https://www.analyticsvidhya.com/blog/2022/03/building-naive-bayes-classifier-from-scratch-to-perform-sentiment-analysis/\n",
    "\n",
    "    [17] https://www.kaggle.com/code/carlosaguayo/text-clustering-with-unsupervised-learning\n",
    "\n",
    "    [18] https://www.kaggle.com/code/barishasdemir/classification-with-naive-bayes\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}