notebooks/forecasting-the-stock-market.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PREDICTING THE STOCK MARKET WITH WATSON\n",
    "\n",
    "## Part I: Introduction\n",
    "\n",
    "In this Jupyter Notebook you will learn step-by-step how to extract financial data from one of the most popular public databases for econometric data, as well as the cleansing and preparation processes of the data science traditional workflow.\n",
    "\n",
    "After preparing the data into an adequate format, structured into a pandas dataframe, we train a predictive model utilizing techniques such as curve-fitting and ARIMA (AutoRegressive Integrated Moving Average).\n",
    "\n",
    "Lastly, we create interactive plots to visualize the collected data, as well as the results of the forecasters trained.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "#### 1. Mining Stock Market Data\n",
    "* 1.1. Quandl API Setup\n",
    "* 1.2. Downloading the WIKI/PRICES Table\n",
    "* 1.3. Visualizing the Collected Data\n",
    "\n",
    "#### 2. Training a Predictive Model using Open-Source Libraries\n",
    "* 2.1. Fbprophet Setup\n",
    "* 2.2. Data Preparation\n",
    "* 2.3. The Machine Learning\n",
    "\n",
    "####  3. Analyzing the Results\n",
    "* 3.1. Analyzing Seasonalities\n",
    "* 3.2. Overlapping of Actual Values with Expected Values\n",
    "* 3.3. Trained Model Evaluation\n",
    "\n",
    "#### 4. Exporting Data\n",
    "* 4.1. Exporting Data to IBM Cloud Object Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1: Mining Stock Market Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data mining step is done using the Quandl API, a financial database (with both public and private tables) purchased by NASDAQ in 2018. It is not mandatory to use an API key to extract data, however it is recommended to generate one for free <a href=\"https://www.quandl.com/\">at the Quandl website</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1: Quandl API Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install quandl package (this can take some minutes)\n",
    "!pip install --user quandl==3.4.8 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import quandl package\n",
    "import quandl\n",
    "print(\"Quandl package imported.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you registered and generated an API key for free on the Quandl web page, type it in the variable in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quandl.ApiConfig.api_key = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2: Downloading the WIKI/PRICES Table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quandl has several public and private data tables available.\n",
    "\n",
    "In this project we will use the table <a href=\"https://www.quandl.com/databases/WIKIP\"> WIKI/PRICES</a>. This table has information on the daily closing and opening values of the shares of more than 3000 companies.\n",
    "\n",
    "The data download is done via an API call executed with the quandl library, using the `get_table` function. The arguments of this function are: the table name, the <a href=\"https://en.wikipedia.org/wiki/Ticker_symbol\"> ticker symbol </a> of the desired stock, and the desired columns of the table. The `paginate=True` parameter allows more than 10000 rows of data to be transferred with a single call.\n",
    "\n",
    "The result of this function call is a pandas dataframe with the desired stock data, ready to be transformed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# API Call example - Downloading financial data from IBM stocks\n",
    "data = quandl.get_table('WIKI/PRICES',\n",
    "                        ticker='IBM', \n",
    "                        qopts={'columns':['date', 'open', 'high', 'low', 'close']},\n",
    "                        paginate=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the <a href=\"https://www.nasdaq.com/symbol/\">NASDAQ page</a> you can see the ticker symbols for various stocks.\n",
    "\n",
    "You can also choose a ticker from the ones listed in the `nasdaq-ticker-table.csv` file, which can be accessed <a href =\"https://github.com/vanderleipf/ibmdegla-ws-projects/blob/master/EN-US/forecasting-the-stock-market/data/stock-ticker-table.csv\">here</a>.\n",
    "\n",
    "Choose the ticker of another company, or use the data already configured and saved in the dataframe `date` by the API call executed in the above code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming the pandas dataframe columns\n",
    "data.columns = ['Date', 'Open', 'High', 'Low', 'Close']\n",
    "\n",
    "# Reset the pandas dataframe index\n",
    "data = data.reset_index(drop=True)\n",
    "\n",
    "# Show the last five rows of the pandas dataframe\n",
    "data.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3: Visualizing the Collected Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the data, the `bokeh` package is used. This module is capable of generating interactive Javascript charts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user bokeh==1.0.4 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.plotting import figure, output_file, show\n",
    "from bokeh.models import ColumnDataSource\n",
    "from bokeh.embed import components\n",
    "from bokeh.io import output_notebook\n",
    "\n",
    "print('Packages imported.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load bokeh\n",
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bokeh Figure\n",
    "p = figure(plot_width=1200, plot_height=550, title='Stock Value Historical Data', x_axis_type=\"datetime\")\n",
    "\n",
    "# Plot Lines\n",
    "p.line(data.Date, data.Open, line_width=2, line_color=\"#0099ff\", legend='Open')\n",
    "p.line(data.Date, data.Close, line_width=2, line_color=\"#ff6699\", legend='Close')\n",
    "p.line(data.Date, data.High, line_width=1, line_color=\"#000000\", legend='High')\n",
    "p.line(data.Date, data.Low, line_width=1, line_color=\"#000000\", legend='Low')\n",
    "\n",
    "# Axis and Labels\n",
    "p.legend.orientation = \"vertical\"\n",
    "p.xaxis.axis_label = \"Date\"\n",
    "p.xaxis.axis_label_text_font_style = 'bold'\n",
    "p.xaxis.axis_label_text_font_size = '16pt'\n",
    "p.xaxis.major_label_text_font_size = '14pt'\n",
    "p.yaxis.axis_label = \"Value ($ USD)\"\n",
    "p.yaxis.axis_label_text_font_style = 'bold'\n",
    "p.yaxis.axis_label_text_font_size = '16pt'\n",
    "p.yaxis.major_label_text_font_size = '12pt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the figure below we can see and interact with the data extracted from the WIKI/PRICES table. In blue we have the opening value of the stock; in red the closing value of the stock; in black we have the maximum and minimum values (daily)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2: Training a Predictive Model using Open-Source Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the training process of our predictive model, we will use the `fbprophet` package, developed by Facebook for time series analysis.\n",
    "\n",
    "Fbprophet follows the same style of objects as `sklearn`, an extremely popular machine learning python library. An instance of the Prophet class is created and then the `fit` and` predict` methods are used.\n",
    "\n",
    "The training set of a Prophet model is a two-column pandas dataframe with the columns `ds` and` y`. The `ds` (datestamp) column must be a date in the format YYYY-MM-DD, or a timestamp in the format YYYY-MM-DD HH:MM:SS. The `y` column must be numeric, and represents the variable we wish to model in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1: Fbprophet Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Install required packages (this can take some minutes)\n",
    "# UPDATE(vnderlev): setuptools-git is now required for fbprophet 0.5. \n",
    "!pip install --user pystan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install fbprophet from pip (this will take some time, please be patient)\n",
    "!pip install fbprophet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import fbprophet\n",
    "import datetime as dt\n",
    "\n",
    "print('Packages imported.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2: Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, the previously collected dataframe needs to be prepared.\n",
    "\n",
    "The preparation includes the selection of a subset of the data for model training. In this Jupyter notebook we will only sample the latest data, starting from 2008.\n",
    "\n",
    "Of the four columns with stock values, only one column will be chosen as the variable of interest. We will choose the daily opening value of the shares (the `Open` column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select train sample\n",
    "df_train = data.copy()\n",
    "df_train = df_train[(df_train.Date > dt.datetime(2008,1,1))]\n",
    "df_train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format the dataframe for FBProphet\n",
    "df_train.rename(columns={'Open':'y', 'Date':'ds'}, inplace=True)\n",
    "df_train = df_train.filter(items=['ds', 'y'])\n",
    "df_train.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3: The Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As previously mentioned, Prophet uses the same interface style as the sklearn library.\n",
    "\n",
    "A Prophet class is instantiated, with the desired types of seasonality. For this work, we will consider annual and monthly seasonalities in stock values, and disregard daily and weekly effects.\n",
    "\n",
    "Usually the presented settings are the most adequate for stock prediction, but feel free to test different seasonalities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a fbprophet model\n",
    "model = fbprophet.Prophet(daily_seasonality=False,\n",
    "                weekly_seasonality=False, \n",
    "                yearly_seasonality=True,\n",
    "                changepoint_prior_scale=0.05,\n",
    "                changepoints=None)\n",
    "model.add_seasonality(name='monthly', period=30.5, fourier_order=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train model\n",
    "model.fit(df_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, the prediction runs in a period of 365 days (1 year) in the future. The result is a multi-column dataframe with information about trends, tolerances, and forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Execute forecasting algorithm (1 year into the future)\n",
    "future_data = model.make_future_dataframe(periods=365, freq='D')\n",
    "future_data = model.predict(future_data)\n",
    "future_data.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: Analyzing Seasonalities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "]In the figure below we have a quick plot of the results using the Prophet class `plot` method.\n",
    "\n",
    "The black dots are the actual data, the dark blue line is the modeled trend, and in light blue we have the tolerance (minimum and maximum values modeled)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig1 = model.plot(future_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `plot_components` function of the Prophet object, we can also analyze trend information, as well as the monthly and yearly seasonalities considered in the modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig2 = model.plot_components(future_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: Overlapping of Actual Values with Expected Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bokeh Figure\n",
    "p = figure(plot_width=1200, plot_height=550, title='Stock Value Historical Data', x_axis_type=\"datetime\")\n",
    "\n",
    "# Plot Lines\n",
    "p.line(data[data['Date'] > dt.datetime(2008,1,1)].Date, data[data['Date'] > dt.datetime(2008,1,1)].Open, line_width=2, line_color=\"#0099ff\", legend='Observed Open Value')\n",
    "p.line(future_data.ds, future_data.yhat, line_width=2, line_color=\"#2B0000\", legend='Modeled Open Value')\n",
    "p.line(future_data.ds, future_data.yhat_upper, line_width=0.5, line_color=\"#000099\", legend='Upper Estimates')\n",
    "p.line(future_data.ds, future_data.yhat_lower, line_width=0.5, line_color=\"#000099\", legend='Lower Estimates')\n",
    "\n",
    "# Axis and Labels\n",
    "p.legend.orientation = \"vertical\"\n",
    "p.xaxis.axis_label = \"Date\"\n",
    "p.xaxis.axis_label_text_font_style = 'bold'\n",
    "p.xaxis.axis_label_text_font_size = '16pt'\n",
    "p.xaxis.major_label_text_font_size = '14pt'\n",
    "p.yaxis.axis_label = \"Value ($ USD)\"\n",
    "p.yaxis.axis_label_text_font_style = 'bold'\n",
    "p.yaxis.axis_label_text_font_size = '16pt'\n",
    "p.yaxis.major_label_text_font_size = '12pt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell we can thoroughly analyze the quality of the trained model by comparing the modeled values (in black) with the actual observed values (light blue)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: Trained Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Prophet class also provides the means to perform an evaluation of the created model.\n",
    "\n",
    "The `cross-validation` method will be used to evaluate our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fbprophet.diagnostics import cross_validation\n",
    "df_cv = cross_validation(model, initial='730 days', period='180 days', horizon='365 days')\n",
    "df_cv.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fbprophet.diagnostics import performance_metrics\n",
    "df_p = performance_metrics(df_cv)\n",
    "df_p.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fbprophet.plot import plot_cross_validation_metric\n",
    "fig = plot_cross_validation_metric(df_cv, metric='mape')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzing the `MAPE` criterion projected in the cell above, it is noticed that the error for a forecast of up to 50 days turns around 5%, reaching 15% for a forecast of 365 days. This error is a relatively good metric if the objective is to construct an image of the market trends. For day-trading purposes, even 95% accuracy is not really adequate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4: Exporting Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: Exporting Data to IBM Cloud Object Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mined data, as well as the data produced by the model, can be exported as CSV files to IBM Cloud Object Storage, and eventually be used or published in other applications.\n",
    "\n",
    "Next we present the code necessary to perform this task in an automated way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_botocore.client import Config\n",
    "import ibm_boto3, os\n",
    "\n",
    "print('Packages imported.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up your IBM Cloud Object Storage credentials in the next cell. Easiest way to regenerate this is, click on the 0100 button on the top right corner here in Watson Studio, and select the `AAPL.csv` file and then `Insert to code -> Credentials`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following code contains the credentials for a file in your IBM Cloud Object Storage.\n",
    "# You might want to remove those credentials before you share your notebook.\n",
    "cos_credentials = {\n",
    "    'IAM_SERVICE_ID': '',\n",
    "    'IBM_API_KEY_ID': '',\n",
    "    'ENDPOINT': '',\n",
    "    'IBM_AUTH_ENDPOINT': '',\n",
    "    'BUCKET': '',\n",
    "    'FILE': ''\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `upload_data_to_ibm_cos` is a function created to upload pandas dataframes as CSV files in IBM COS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_data_to_ibm_cos(credentials, df, df_future, tick):\n",
    "    cos = ibm_boto3.client(service_name='s3', \n",
    "                           ibm_api_key_id=credentials['IBM_API_KEY_ID'], \n",
    "                           ibm_service_instance_id=credentials['IAM_SERVICE_ID'],\n",
    "                           ibm_auth_endpoint=credentials['IBM_AUTH_ENDPOINT'],\n",
    "                           config=Config(signature_version='oauth'),\n",
    "                           endpoint_url=credentials['ENDPOINT'])\n",
    "    df.to_csv('{}.csv'.format(tick), sep=',', encoding='utf-8')\n",
    "    df_future.to_csv('{}_future.csv'.format(tick), sep=',', encoding='utf-8')\n",
    "    try:\n",
    "        res=cos.upload_file(Filename='{}.csv'.format(tick), Bucket=credentials['BUCKET'], Key='{}.csv'.format(tick))\n",
    "        res=cos.upload_file(Filename='{}_future.csv'.format(tick), Bucket=credentials['BUCKET'], Key='{}_future.csv'.format(tick))\n",
    "    except Exception as e:\n",
    "        print(Exception, e)\n",
    "    else:\n",
    "        print(\"{} data uploaded to IBM COS.\".format(tick))\n",
    "        os.remove('{}.csv'.format(tick))\n",
    "        os.remove('{}_future.csv'.format(tick))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload mined and modeled data to IBM COS\n",
    "upload_data_to_ibm_cos(cos_credentials, data, future_data, 'IBM')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now access the `<stock_ticker>.csv` and `<stock_ticker>_future.csv` file anywhere using the IBM COS API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "\n",
    "## Want to Learn More?\n",
    "\n",
    "Watch the public video on how to replicate this analysis with SPSS Modeler, an analytics platform that has several embedded machine learning algorithms. This platform is designed to facilitate data analysis through a graphical programming language, and is integrated with Watson Studio.\n",
    "\n",
    "This notebook and its source code is made available under the terms of the <a href = \"https://github.com/IBM/watson-stock-market-predictor/blob/master/LICENSE\">Apache License 2.0</a>.\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thank you for completing this journey!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}