docs/source/notebooks/data_visualization_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c3dcce28",
   "metadata": {},
   "source": [
    "# Visualization examples\n",
    "\n",
    "## Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82b069c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../../')\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from numpy.random import random\n",
    "import statsmodels.api as sm\n",
    "\n",
    "from pytsviz import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2574f87",
   "metadata": {},
   "source": [
    "## Data preparation for forecasting\n",
    "\n",
    "Target series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efb53646",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_size=366"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39d48fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "t_axis = pd.date_range(start='1/1/2020', periods = dataset_size, freq=\"D\", name=\"t\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f0a0db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = (1 + random(dataset_size)) * np.linspace(6, 9, dataset_size) + \\\n",
    "    (1 + random(dataset_size)) * np.sin(np.linspace(0, 10*np.pi, dataset_size)) + \\\n",
    "    (1.5 + random(dataset_size)) * np.cos(np.linspace(0, 5.4*np.pi, dataset_size))\n",
    "ts = pd.Series(index=t_axis, data=y, name=\"y\")\n",
    "ts_df = ts.to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c22d24be",
   "metadata": {},
   "source": [
    "Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfcdbe0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_features = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7f3a8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_matrix = random((n_features, dataset_size))\n",
    "feat_df = pd.DataFrame(index=t_axis, data={f\"x_{i}\": feat_matrix[i] for i in range(n_features)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfc418b2",
   "metadata": {},
   "source": [
    "Forecast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d684d31b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ext_t_axis = pd.date_range(start='2/1/2020', periods = dataset_size, freq=\"D\", name=\"t\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b2e72c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat = y * (1 + 0.1 * random())\n",
    "\n",
    "forecast = pd.Series(index=ext_t_axis, data=y_hat, name=\"y_hat\")\n",
    "forecast_df = forecast.to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff3cc7d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "c_lower = [y_hat[i] - (0.001 * i) ** 1.2 for i in range(len(y_hat))]\n",
    "c_upper = [y_hat[i] + (0.001 * i) ** 1.2 for i in range(len(y_hat))]\n",
    "confidence_df = pd.DataFrame(index=ext_t_axis, data={\"lower confidence\": c_lower, \"upper confidence\": c_upper})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c000794",
   "metadata": {},
   "outputs": [],
   "source": [
    "total_df = pd.concat([ts, feat_df, forecast_df, confidence_df], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b456e80",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Time series plot\n",
    "\n",
    "First let's plot our target time series along with one of the features we want to use with forecasting.\n",
    "We can also display their log transformation along with (or excluding) the originals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9152693a",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ts(total_df, y_cols=[\"y\", \"x_0\"], tf=\"log\", tf_args = (np.e,), keep_original=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31b1d78b-e1d1-49eb-9b93-74eabfd73a35",
   "metadata": {},
   "source": [
    "There are other useful transformations, as well as the possibility of passing a custom one.\n",
    "Here we can see the Box-Cox transform applied to our target series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75169774-7ebd-425e-bc09-4c1cdda022a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ts(total_df, y_cols=[\"y\"], tf=\"Box-Cox\", keep_original=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77ce48ba",
   "metadata": {},
   "source": [
    "## Seasonal plot\n",
    "\n",
    "We can take our target series and inspect it, looking for seasonal components. For example, let's break it down into monthly components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c416475f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_seasonal_ts(ts_df, period=\"month\", subplots=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "437f8988-80f3-4d9b-8107-04c79a19d37e",
   "metadata": {},
   "source": [
    "We can also customize the grouping of our data, by specifying a couple of function to pass as period.\n",
    "\n",
    "These functions are used respectively to assign data points to period groups and to select the granularity with which we want to display the data.\n",
    "\n",
    "For example, let's say we want to inspect the time series by grouping it into weeks, and keeping a daily granularity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "444f2eac",
   "metadata": {},
   "outputs": [],
   "source": [
    "extract_week = lambda x : x.isocalendar().week\n",
    "extract_day = lambda x : x.isocalendar().day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92c20e2",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# noinspection PyTypeChecker\n",
    "plot_seasonal_ts(ts_df, period=(extract_week, extract_day), subplots=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d97cf138-7a9a-4d69-af30-92d7dd84a01d",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Notice that because of the high number of weeks in our dataset we had to set subplots = False."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebdf007e",
   "metadata": {},
   "source": [
    "## Decomposed plot\n",
    "\n",
    "One more thing we can do is to try and apply a classic time series decomposition method, such as STL decomposition, to our series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fb16387",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_decomposed_ts(ts_df, method = \"STL\", subplots = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f90df5ea",
   "metadata": {},
   "source": [
    "## Multiple forecasts\n",
    "\n",
    "We can display one or more forecasts together with their target series, and also show confidence bounds by specifying their columns in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb61ace4",
   "metadata": {},
   "outputs": [],
   "source": [
    "inverted_df = pd.DataFrame(total_df.values[::-1], total_df.index, total_df.columns)\n",
    "\n",
    "plot_forecast(total_df, y_col = \"y\", y_hat_cols = [\"y_hat\"], upper_col = \"upper confidence\", lower_col = \"lower confidence\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa211669-82e2-40ef-9cf3-cd4ad78c0e42",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat_2 = pd.Series(data=y_hat*0.9, index = ext_t_axis, name = \"y_hat_2\")\n",
    "multiple_fc_df = pd.concat([total_df, y_hat_2], axis = 1)\n",
    "plot_forecast(multiple_fc_df, y_col = \"y\", y_hat_cols = [\"y_hat\", \"y_hat_2\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6811fcf3-eebd-4cf7-97ea-a6ef5e1b6321",
   "metadata": {},
   "source": [
    "## Scatter matrix\n",
    "\n",
    "A useful inspection tool is the scatter matrix, that is a matrix of scatter plots between different series.\n",
    "\n",
    "We can include features, the target as well as lagged versions of both of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aeedb520",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_scatter_matrix(total_df, x_col=\"x_0\", y_col=\"y\", lags_x=[5, 10, 15])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f420b8f-e621-478b-b7d5-9c100aa5365e",
   "metadata": {},
   "source": [
    "## Fit\n",
    "\n",
    "By inspecting a single pair of time series in a scatter plot we can show the fit line, together with a verbose summary of the fit computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8637bf36",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_scatter_fit(total_df, \"x_0\", \"x_1\", fit = \"summary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e1f296f-5b0c-4e09-b4a8-fc76f86b0eb9",
   "metadata": {},
   "source": [
    "Extra arguments can be passed to modify the fit method of computation, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e79d9dfa-e39a-4c73-9952-2c2c4111651b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_scatter_fit(total_df, \"x_0\", \"x_1\",  trendline=\"lowess\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95d1796-c83b-430c-a9ec-3989d05dc344",
   "metadata": {},
   "source": [
    "## Auto-correlation\n",
    "\n",
    "We can produce auto-correlation (or partial auto-correlation) charts for a given time series, and optionally include both the significance correlation threshold and the confidence intervals for correlation values, to assess any significant lag in the series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e5fe0d1",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e4f2845",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_acf(ts_df, show_threshold = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "377c96d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_acf(ts_df, partial=True, alpha = 0.1, show_threshold = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "774372d4-d9bb-46f0-8ee6-e3d9dc289244",
   "metadata": {},
   "source": [
    "Another useful tool is the visualization of the distribution and the spectral decomposition of a time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0a31bcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_distribution(total_df, bins=100, title=\"Distribution\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f887298b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_psd(ts_df, scaling=\"spectrum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e681a5-1eea-4b39-8a83-6533f2a98595",
   "metadata": {},
   "source": [
    "## Roots of an ARMA process\n",
    "\n",
    "Finally, let's take a look at the roots of a given ARMA process; they can be used to find out is the process is stationary, among other things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56d5c687",
   "metadata": {},
   "outputs": [],
   "source": [
    "arparams = np.array([.75, -.25])\n",
    "maparams = np.array([.65, .35])\n",
    "ar = np.r_[1, -arparams] # add zero-lag and negate\n",
    "ma = np.r_[1, maparams] # add zero-lag\n",
    "arma_process = sm.tsa.ArmaProcess(ar, ma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c97bf57",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_inverse_arma_roots(arma_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "146263a1-562a-4d69-842c-019fb1057572",
   "metadata": {},
   "source": [
    "## Composite visualizations\n",
    "\n",
    "To conclude, here are some common composite plots that can be produced all at once, in order to provide a comprehensive qualitative view of a (set of) time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16410a0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_extended_scatter_matrix(total_df.dropna(), y_cols=[\"y\", \"x_1\", \"x_2\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4520df40",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ts_overview(ts_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02d7206b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ts_analysis(ts_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ce70d66",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_gof(total_df, \"y\", \"y_hat\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}