01.TimeSeries-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "chicken-infection",
   "metadata": {},
   "source": [
    "# Time Series Analysis and Forecasting\n",
    "\n",
    "This is a series of notebooks to support lectures on _Time series analysis and forecast_.\n",
    "\n",
    "![](https://user-images.githubusercontent.com/49638680/154160268-cf39a1ec-3557-4940-8853-d06fc7a79acf.png)\n",
    "\n",
    "## Time series Analysis\n",
    "\n",
    "We start by exploring pandas method to treat time series. \n",
    "In paticular, over this notebook we are going to analyse a dataframe and how to deal with pandas time index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tight-buddy",
   "metadata": {},
   "source": [
    "### Import data\n",
    "\n",
    "Let's have a look at the structure of a real-life time series. We will use both known and new Pandas tools, as well as Matplotlib and Seaborn.\n",
    "Once we are able to analyse a time series, we will use the information we have inferred to predict its future.\n",
    "In the meantime, here and there, we will think about the relationship with supervised learning.\n",
    "\n",
    "![img](https://raw.githubusercontent.com/fbagattini/Lezioni/master/img/OPSD.png)\n",
    "\n",
    "[OPSD database](https://open-power-system-data.org/) is a dataset provided by the famous open source project OPSD that provides data on installed generation capacity by country/technology, individual power plants (conventional and renewable), and time series data. The latter includes electricity consumption, spot prices, and wind and solar generation, both measured and derived from weather models.\n",
    "\n",
    "The aim of this very cool project is to make life easier for power system modelers. The goal is to help researchers focus on research and avoid redundant work when collecting, preparing, and aggregating data.\n",
    "\n",
    "In particular, in this lecture we are going to import the OPSD_Germany_consumption dataset. This reports Germany's electric energy consumption (in GWh) from 2006 to 2017. These data are part of the OPSD project.\n",
    "\n",
    "Let's start by loading the dataset as a DataFrame. The only mandatory parameter of the read_csv function is the path of the .csv file we are loading. The head and tail functions respectively show the first and last records (5 by default) of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "secondary-acrylic",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy\n",
    "\n",
    "import statsmodels.api as sm\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = \"retina\"\n",
    "plt.rcParams['figure.figsize'] = (15.0, 10.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "southwest-desktop",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>1069.184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2006-01-02</td>\n",
       "      <td>1380.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2006-01-03</td>\n",
       "      <td>1442.533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2006-01-04</td>\n",
       "      <td>1457.217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2006-01-05</td>\n",
       "      <td>1477.131</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  Consumption\n",
       "0  2006-01-01     1069.184\n",
       "1  2006-01-02     1380.521\n",
       "2  2006-01-03     1442.533\n",
       "3  2006-01-04     1457.217\n",
       "4  2006-01-05     1477.131"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_url = 'https://raw.githubusercontent.com/fbagattini/Lezioni/master/data/OPSD_Germany_consumption.csv'\n",
    "df = pd.read_csv(data_url)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "peaceful-lloyd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4383, 2)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affiliated-valuation",
   "metadata": {},
   "source": [
    "Note how the `DataFrame` object in pandas _inherits_ the attribute `shape` from numpy. \n",
    "The dataframe contains $4383$ records: each row reports, in addition to the consumption, the date at which the record is observed.\n",
    "\n",
    "Pandas, at loading time, provides the dataframe with an indexing structure. \n",
    "Via the synthax\n",
    "\n",
    "```python\n",
    "df.loc[position]\n",
    "```\n",
    "\n",
    "it is possible to access the element whose index is position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dying-tobacco",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           2006-01-08\n",
       "Consumption      1207.985\n",
       "Name: 7, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[7]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "known-violence",
   "metadata": {},
   "source": [
    "Whe are used to associate the columns of a dataset to the features. What about `Date`? Is that a special feature? Which role could it have within supervised learning?\n",
    "\n",
    "Let's have a look at its content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "healthy-stephen",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordinary-statement",
   "metadata": {},
   "source": [
    "In Pandas, type 'O' stands for Object: a generic type, for our purposes we can consider it more broadly as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "negative-garbage",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2006-01-01'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].loc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "noticed-segment",
   "metadata": {},
   "source": [
    "We want to give these data a structure. Meaning we want to be able to define an operation like\n",
    "\n",
    "```python\n",
    "df['Date'].loc[0] + year\n",
    "```\n",
    "\n",
    "having '2007-01-01' as result.\n",
    "\n",
    "Hence, we aim at providing these records with a temporal structure.\n",
    "\n",
    "### Time indexing\n",
    "\n",
    "We are going to explore a pandas method to perform such conversion `to_datetime`. This will result in a new type of object: `Timestamp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "gross-large",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2009-10-02 00:00:00')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('10-02-09')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respiratory-edwards",
   "metadata": {},
   "source": [
    "Note how we got not only the date, but also the time for free.\n",
    "\n",
    "The function has attemped to autonomously infer the date format: October 2, 2009. Through optional arguments we can drive this transformation.\n",
    "\n",
    "Can you tell the difference between the two following cells?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "magnetic-parameter",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2010-02-09 00:00:00')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('10-02-09', yearfirst=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "modular-advice",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2009-02-10 00:00:00')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('10-02-09', dayfirst=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "parallel-apache",
   "metadata": {},
   "source": [
    "The `to_datetime` method can handle several formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "killing-consensus",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('1990-06-07 00:00:00')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('7th of June 1990')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "central-psychiatry",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('1990-02-10 00:00:00')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('Feb 10 1990')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collaborative-idaho",
   "metadata": {},
   "source": [
    "As with individual strings, we can convert lists of strings; in our instance, we can transform the whole `Date` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "spanish-damages",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Date = pd.to_datetime(df.Date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "floating-question",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>1069.18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2006-01-02</td>\n",
       "      <td>1380.52100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2006-01-03</td>\n",
       "      <td>1442.53300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2006-01-04</td>\n",
       "      <td>1457.21700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2006-01-05</td>\n",
       "      <td>1477.13100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4378</th>\n",
       "      <td>2017-12-27</td>\n",
       "      <td>1263.94091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4379</th>\n",
       "      <td>2017-12-28</td>\n",
       "      <td>1299.86398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4380</th>\n",
       "      <td>2017-12-29</td>\n",
       "      <td>1295.08753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4381</th>\n",
       "      <td>2017-12-30</td>\n",
       "      <td>1215.44897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4382</th>\n",
       "      <td>2017-12-31</td>\n",
       "      <td>1107.11488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4383 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date  Consumption\n",
       "0    2006-01-01   1069.18400\n",
       "1    2006-01-02   1380.52100\n",
       "2    2006-01-03   1442.53300\n",
       "3    2006-01-04   1457.21700\n",
       "4    2006-01-05   1477.13100\n",
       "...         ...          ...\n",
       "4378 2017-12-27   1263.94091\n",
       "4379 2017-12-28   1299.86398\n",
       "4380 2017-12-29   1295.08753\n",
       "4381 2017-12-30   1215.44897\n",
       "4382 2017-12-31   1107.11488\n",
       "\n",
       "[4383 rows x 2 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "senior-swaziland",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           2006-01-08 00:00:00\n",
       "Consumption               1207.985\n",
       "Name: 7, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "legendary-safety",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           datetime64[ns]\n",
       "Consumption           float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulated-current",
   "metadata": {},
   "source": [
    "It is noteworthy that python stores temporal objects as 64-bits integers, so that to handle nanoseconds (ns) precision.\n",
    "\n",
    "**Recap**: we have transformed a column of the dataframe into a special one, equipped with temporal logic.\n",
    "\n",
    "### DateTimeIndex\n",
    "\n",
    "As you might have noticed, we have one example per day (this is called _frequency_). Furthermore, we have already talked about the definition of a time series, _i.e._ a _realisation_ of a stochastic process thanks to its projection on a time interval.\n",
    "\n",
    "$$\\{x_1, \\ldots , x_n \\vert x_k = x(t_k) \\} \\, .$$\n",
    "\n",
    "Hence, it is natural to use the time variable (the `Date` in this case) as an index. Pandas allows us to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "individual-madagascar",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-01</th>\n",
       "      <td>1069.18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-02</th>\n",
       "      <td>1380.52100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-03</th>\n",
       "      <td>1442.53300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>1457.21700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>1477.13100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-27</th>\n",
       "      <td>1263.94091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28</th>\n",
       "      <td>1299.86398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29</th>\n",
       "      <td>1295.08753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-30</th>\n",
       "      <td>1215.44897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-31</th>\n",
       "      <td>1107.11488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4383 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Consumption\n",
       "Date                   \n",
       "2006-01-01   1069.18400\n",
       "2006-01-02   1380.52100\n",
       "2006-01-03   1442.53300\n",
       "2006-01-04   1457.21700\n",
       "2006-01-05   1477.13100\n",
       "...                 ...\n",
       "2017-12-27   1263.94091\n",
       "2017-12-28   1299.86398\n",
       "2017-12-29   1295.08753\n",
       "2017-12-30   1215.44897\n",
       "2017-12-31   1107.11488\n",
       "\n",
       "[4383 rows x 1 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('Date')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proof-missouri",
   "metadata": {},
   "source": [
    "Great! let's call our dataset again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "constitutional-vatican",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>1069.18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2006-01-02</td>\n",
       "      <td>1380.52100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2006-01-03</td>\n",
       "      <td>1442.53300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2006-01-04</td>\n",
       "      <td>1457.21700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2006-01-05</td>\n",
       "      <td>1477.13100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4378</th>\n",
       "      <td>2017-12-27</td>\n",
       "      <td>1263.94091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4379</th>\n",
       "      <td>2017-12-28</td>\n",
       "      <td>1299.86398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4380</th>\n",
       "      <td>2017-12-29</td>\n",
       "      <td>1295.08753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4381</th>\n",
       "      <td>2017-12-30</td>\n",
       "      <td>1215.44897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4382</th>\n",
       "      <td>2017-12-31</td>\n",
       "      <td>1107.11488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4383 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date  Consumption\n",
       "0    2006-01-01   1069.18400\n",
       "1    2006-01-02   1380.52100\n",
       "2    2006-01-03   1442.53300\n",
       "3    2006-01-04   1457.21700\n",
       "4    2006-01-05   1477.13100\n",
       "...         ...          ...\n",
       "4378 2017-12-27   1263.94091\n",
       "4379 2017-12-28   1299.86398\n",
       "4380 2017-12-29   1295.08753\n",
       "4381 2017-12-30   1215.44897\n",
       "4382 2017-12-31   1107.11488\n",
       "\n",
       "[4383 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-miami",
   "metadata": {},
   "source": [
    "Wait! 😲 \n",
    "\n",
    "Nothing has happened to the dataframe. Why? Where did we go wrong?\n",
    "\n",
    "Often, Pandas functions return a new dataframe, without modifying the original one. In such cases, we have to either overwrite it (option `inplace=True`) or create a new one (assigning a new variable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "under-engineer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-01</th>\n",
       "      <td>1069.184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-02</th>\n",
       "      <td>1380.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-03</th>\n",
       "      <td>1442.533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>1457.217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>1477.131</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Consumption\n",
       "Date                   \n",
       "2006-01-01     1069.184\n",
       "2006-01-02     1380.521\n",
       "2006-01-03     1442.533\n",
       "2006-01-04     1457.217\n",
       "2006-01-05     1477.131"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.set_index('Date')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "auburn-pointer",
   "metadata": {},
   "source": [
    "Now `df` is correctly indexed by `Date`. Let's have a look at the new index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "prostate-ordering",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2006-01-01', '2006-01-02', '2006-01-03', '2006-01-04',\n",
       "               '2006-01-05', '2006-01-06', '2006-01-07', '2006-01-08',\n",
       "               '2006-01-09', '2006-01-10',\n",
       "               ...\n",
       "               '2017-12-22', '2017-12-23', '2017-12-24', '2017-12-25',\n",
       "               '2017-12-26', '2017-12-27', '2017-12-28', '2017-12-29',\n",
       "               '2017-12-30', '2017-12-31'],\n",
       "              dtype='datetime64[ns]', name='Date', length=4383, freq=None)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immune-treat",
   "metadata": {},
   "source": [
    "It is not the usual `Index`, it is something more specific `DatetimeIndex`. Indeed, it has specific properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "numerical-chain",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([2006, 2006, 2006, 2006, 2006, 2006, 2006, 2006, 2006, 2006,\n",
       "            ...\n",
       "            2017, 2017, 2017, 2017, 2017, 2017, 2017, 2017, 2017, 2017],\n",
       "           dtype='int64', name='Date', length=4383)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "spiritual-lyric",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10,\n",
       "            ...\n",
       "            22, 23, 24, 25, 26, 27, 28, 29, 30, 31],\n",
       "           dtype='int64', name='Date', length=4383)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index.day"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "educational-young",
   "metadata": {},
   "source": [
    "**Recap**: we have used the time information provided by our data (the date on which each record is observed) to index a dataframe.\n",
    "\n",
    "### Indexing at loading time\n",
    "\n",
    "Since the greatest force moving a developer is lazyness, we do not want to work on our dataframes in order to index them correctly. In other words we do not want to repeat the operations we have done above, _i.e._ converting to the right datetime format, creating a copy of our dataframe, changing the index to `Datetimeindex`, etc.\n",
    "\n",
    "We can achieve the same goal in a compact way. Among the arguments of `read_csv`, there are:\n",
    "\n",
    "* `parse_dates` \n",
    "* `index_col`\n",
    "\n",
    "The former one accepts a boolean as value (hence`True` or `False`) setting whether to try parsing dates as `Datetime` objects, on the other hand the latter accepts a string or a list of strings indicating which columns should be used as index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dangerous-budget",
   "metadata": {},
   "source": [
    "![](https://media.makeameme.org/created/ta-da.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "stuffed-shakespeare",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-01</th>\n",
       "      <td>1069.184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-02</th>\n",
       "      <td>1380.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-03</th>\n",
       "      <td>1442.533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>1457.217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>1477.131</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Consumption\n",
       "Date                   \n",
       "2006-01-01     1069.184\n",
       "2006-01-02     1380.521\n",
       "2006-01-03     1442.533\n",
       "2006-01-04     1457.217\n",
       "2006-01-05     1477.131"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(data_url,\n",
    "                 parse_dates=True,\n",
    "                 index_col='Date')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vocal-prairie",
   "metadata": {},
   "source": [
    "__Useful remark__: recall that it is possible to access the documentation of a function `func` by placing the cursor to the right of a function name and pressing\n",
    "```\n",
    "Shift Tab (Tab Tab)\n",
    "```\n",
    "\n",
    "Alternatively, we can run a cell like\n",
    "```\n",
    "func?\n",
    "```\n",
    "\n",
    "Let's now print some info about our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "amazing-maximum",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 4383 entries, 2006-01-01 to 2017-12-31\n",
      "Data columns (total 1 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   Consumption  4383 non-null   float64\n",
      "dtypes: float64(1)\n",
      "memory usage: 68.5 KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amino-touch",
   "metadata": {},
   "source": [
    "**Recap**: with just one line of code we have loaded the OPSD_Germany_consumption dataset and set Date column as its time index.\n",
    "\n",
    "### String and partial-string indexing\n",
    "\n",
    "Now that records have a time reference attached to, we can easily access them through string indexing.\n",
    "\n",
    "Let's suppose we want to know the electric consumption of Christmas 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "placed-notebook",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Consumption    1047.277\n",
       "Name: 2015-12-25 00:00:00, dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2015-12-25']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "institutional-hungary",
   "metadata": {},
   "source": [
    "And we want to compare this record with the average consumption of December 2015. It is quite easy to do so in pandas.\n",
    "\n",
    "As with slicing in Python and NumPy, string indexing can be used with ':' to access time intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "numerous-amsterdam",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-12-01</th>\n",
       "      <td>1588.021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-02</th>\n",
       "      <td>1585.308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-03</th>\n",
       "      <td>1577.457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-04</th>\n",
       "      <td>1570.318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-05</th>\n",
       "      <td>1337.095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-06</th>\n",
       "      <td>1232.073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-07</th>\n",
       "      <td>1536.251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-08</th>\n",
       "      <td>1572.740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-09</th>\n",
       "      <td>1586.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-10</th>\n",
       "      <td>1596.593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-11</th>\n",
       "      <td>1578.220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-12</th>\n",
       "      <td>1334.376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-13</th>\n",
       "      <td>1242.101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-14</th>\n",
       "      <td>1554.947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-15</th>\n",
       "      <td>1567.900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-16</th>\n",
       "      <td>1562.404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-17</th>\n",
       "      <td>1512.095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-18</th>\n",
       "      <td>1462.443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-19</th>\n",
       "      <td>1255.073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-20</th>\n",
       "      <td>1165.159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-21</th>\n",
       "      <td>1396.454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-22</th>\n",
       "      <td>1379.031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-23</th>\n",
       "      <td>1312.974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-24</th>\n",
       "      <td>1121.944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-25</th>\n",
       "      <td>1047.277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-26</th>\n",
       "      <td>1049.349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-27</th>\n",
       "      <td>1068.214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-28</th>\n",
       "      <td>1219.075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-29</th>\n",
       "      <td>1233.660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-30</th>\n",
       "      <td>1238.863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-31</th>\n",
       "      <td>1158.103</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Consumption\n",
       "Date                   \n",
       "2015-12-01     1588.021\n",
       "2015-12-02     1585.308\n",
       "2015-12-03     1577.457\n",
       "2015-12-04     1570.318\n",
       "2015-12-05     1337.095\n",
       "2015-12-06     1232.073\n",
       "2015-12-07     1536.251\n",
       "2015-12-08     1572.740\n",
       "2015-12-09     1586.393\n",
       "2015-12-10     1596.593\n",
       "2015-12-11     1578.220\n",
       "2015-12-12     1334.376\n",
       "2015-12-13     1242.101\n",
       "2015-12-14     1554.947\n",
       "2015-12-15     1567.900\n",
       "2015-12-16     1562.404\n",
       "2015-12-17     1512.095\n",
       "2015-12-18     1462.443\n",
       "2015-12-19     1255.073\n",
       "2015-12-20     1165.159\n",
       "2015-12-21     1396.454\n",
       "2015-12-22     1379.031\n",
       "2015-12-23     1312.974\n",
       "2015-12-24     1121.944\n",
       "2015-12-25     1047.277\n",
       "2015-12-26     1049.349\n",
       "2015-12-27     1068.214\n",
       "2015-12-28     1219.075\n",
       "2015-12-29     1233.660\n",
       "2015-12-30     1238.863\n",
       "2015-12-31     1158.103"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2015-12-1':'2015-12-31']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bright-boards",
   "metadata": {},
   "source": [
    "**Important note**: A noteworthy difference between pandas and plain python slicing is that in pandas both extrema of the interval are included in the result.\n",
    "\n",
    "Another temporal indexing technique is partial-string indexing. By specifying the month and year, we extract only the records of December 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "little-estate",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Consumption</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-12-01</th>\n",
       "      <td>1588.021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-02</th>\n",
       "      <td>1585.308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-03</th>\n",
       "      <td>1577.457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-04</th>\n",
       "      <td>1570.318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-05</th>\n",
       "      <td>1337.095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-06</th>\n",
       "      <td>1232.073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-07</th>\n",
       "      <td>1536.251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-08</th>\n",
       "      <td>1572.740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-09</th>\n",
       "      <td>1586.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-10</th>\n",
       "      <td>1596.593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-11</th>\n",
       "      <td>1578.220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-12</th>\n",
       "      <td>1334.376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-13</th>\n",
       "      <td>1242.101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-14</th>\n",
       "      <td>1554.947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-15</th>\n",
       "      <td>1567.900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-16</th>\n",
       "      <td>1562.404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-17</th>\n",
       "      <td>1512.095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-18</th>\n",
       "      <td>1462.443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-19</th>\n",
       "      <td>1255.073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-20</th>\n",
       "      <td>1165.159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-21</th>\n",
       "      <td>1396.454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-22</th>\n",
       "      <td>1379.031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-23</th>\n",
       "      <td>1312.974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-24</th>\n",
       "      <td>1121.944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-25</th>\n",
       "      <td>1047.277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-26</th>\n",
       "      <td>1049.349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-27</th>\n",
       "      <td>1068.214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-28</th>\n",
       "      <td>1219.075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-29</th>\n",
       "      <td>1233.660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-30</th>\n",
       "      <td>1238.863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-12-31</th>\n",
       "      <td>1158.103</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Consumption\n",
       "Date                   \n",
       "2015-12-01     1588.021\n",
       "2015-12-02     1585.308\n",
       "2015-12-03     1577.457\n",
       "2015-12-04     1570.318\n",
       "2015-12-05     1337.095\n",
       "2015-12-06     1232.073\n",
       "2015-12-07     1536.251\n",
       "2015-12-08     1572.740\n",
       "2015-12-09     1586.393\n",
       "2015-12-10     1596.593\n",
       "2015-12-11     1578.220\n",
       "2015-12-12     1334.376\n",
       "2015-12-13     1242.101\n",
       "2015-12-14     1554.947\n",
       "2015-12-15     1567.900\n",
       "2015-12-16     1562.404\n",
       "2015-12-17     1512.095\n",
       "2015-12-18     1462.443\n",
       "2015-12-19     1255.073\n",
       "2015-12-20     1165.159\n",
       "2015-12-21     1396.454\n",
       "2015-12-22     1379.031\n",
       "2015-12-23     1312.974\n",
       "2015-12-24     1121.944\n",
       "2015-12-25     1047.277\n",
       "2015-12-26     1049.349\n",
       "2015-12-27     1068.214\n",
       "2015-12-28     1219.075\n",
       "2015-12-29     1233.660\n",
       "2015-12-30     1238.863\n",
       "2015-12-31     1158.103"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2015-12']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amateur-radius",
   "metadata": {},
   "source": [
    "And here is the average consumption of December 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "gorgeous-terrain",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Consumption    1375.545516\n",
       "dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2015-12'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alert-understanding",
   "metadata": {},
   "source": [
    "**Recap**: we have learnt how to access the elements of a time indexed dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eligible-acting",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Along with electric energy consumption, the [OPSD_Germany_all](https://raw.githubusercontent.com/fbagattini/Lezioni/master/data/OPSD_Germany_all.csv) dataset reports the daily production of solar and wind energy.\n",
    "\n",
    "1. Load the dataset as a dataframe, using Date as time index. Solar energy production is not available until December 31, 2011; from the dataframe, select only the records following this date.\n",
    "\n",
    "2. Create a column Renewable as the sum of solar and wind energy.\n",
    "\n",
    "3. Compute the ratio between a) the total renewable production of September 2014 and b) the total electric consumption of the same month.\n",
    "\n",
    "4. Using the indexing properties of Date (hint: `index.day_name()`), create the column 'Weekday' containing, for each record, the day of the week at which it's been observed (Monday, Tuesday,...).\n",
    "\n",
    "5. Create the dataframe `df_sunday_wind` containing (only) Sunday wind production.\n",
    "\n",
    "6. Compute the average Sunday wind production between January and March 2017 (included)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lectures",
   "language": "python",
   "name": "lectures"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}