processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    <td style=\"background-color:#EBF5FB; border: 1px solid #CFCFCF\">\n",
    "      <b>National generation capacity: Processing notebook</b>\n",
    "      <ul>\n",
    "          <li><a href=\"main.ipynb\">Main notebook</a></li>\n",
    "          <li>Processing notebook (this)</li>\n",
    "          <li><a href=\"tests.ipynb\">Check notebook</a></li>\n",
    "      </ul>\n",
    "      <br>This Notebook is part of the <a href=\"http://data.open-power-system-data.org/national_generation_capacity\">National Generation Capacity Datapackage</a> of <a href=\"http://open-power-system-data.org\">Open Power System Data</a>.\n",
    "    </td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* [1. Introductory notes](#1.-Introductory-notes)\n",
    "* [2. Script setup](#2.-Script-setup)\n",
    "* [3. Data download and processing](#3.-Data-download-and-processing)\n",
    "\t* [3.1 Manually compiled dataset](#3.1-Manually-compiled-dataset)\n",
    "\t* [3.2 EUROSTAT data](#3.2-EUROSTAT-data)\n",
    "\t* [3.3 ENTSO-E data](#3.3-ENTSO-E-data)\n",
    "\t\t* [3.3.1 ENTSO-E statistical data](#3.3.1-ENTSO-E-statistical-data)\n",
    "\t\t* [3.3.2 ENTSO-E SO&AF data](#3.3.2-ENTSO-E-SO&AF-data)\n",
    "        * [3.3.3 ENTSO-E Transparency Plaftform](#3.3.1-ENTSO-E-Transparency-Platform)\n",
    "\t\t* [3.3.4 ENTSO-E Power Statistics](#3.3.4-ENTSO-E-Power-Statistics)\n",
    "\t* [3.4 Merge data sources](#3.4-Merge-data-sources)\n",
    "* [4. Convert stacked data to crosstable format](#4.-Convert-stacked-data-to-crosstable-format)\n",
    "* [5. Output](#5.-Output)\n",
    "\t* [5.1 Write results to file](#5.1-Write-results-to-file)\n",
    "\t* [5.2 Formatting of Excel tables](#5.2-Formatting-of-Excel-tables)\n",
    "\t* [5.3 Write checksums](#5.3-Write-checksums)\n",
    "* [6. Documentation of the data package](#6.-Documentation-of-the-data-package)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introductory notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script processes the compiled nationally aggregated generation capacity for European countries. Due to varying formats and data specifications of references for national generation capacity, the script firstly focuses on rearranging the manually compiled data. Thus, the script itself does not collect, select, download or manage data from original sources. Secondly, international data sources, such as EUROSTAT and ENTSO-E, are directly downloaded from original web sources and complement the initial data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Script setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some functions and classes that are defined in seperate files\n",
    "import functions.helper_functions as func\n",
    "import functions.soaf as soaf\n",
    "\n",
    "# core packages\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# packages to copy files, write sqllite data bases and manipulate excel files\n",
    "import shutil\n",
    "import sqlite3\n",
    "import openpyxl\n",
    "from openpyxl.styles import PatternFill, colors, Font, Alignment\n",
    "from openpyxl.utils import get_column_letter\n",
    "\n",
    "import yaml\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data download and processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compile data from different national and international sources. Firstly, national data sources are manually compiled due to varying data formats and specifications. Secondly, international sources are compiled directly and appended to the compiled data set. The international data sources comprise:\n",
    "- [EUROSTAT](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view)\n",
    "- [ENTSO-E Statistical data](https://www.entsoe.eu/data/data-portal/miscellaneous/Pages/default.aspx)\n",
    "- [ENTSO-E System Outlook and Adequacy Forecast](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx)\n",
    "- [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/)\n",
    "- [ENTSO-E Power Statistics](https://www.entsoe.eu/data/power-stats/)\n",
    "\n",
    "In the following section, the data sets are downloaded and uploaded to Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Manually compiled dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The manually compiled dataset is imported and rearranged to a DataFrame for further processing. The dataset comprises for each European country and specified generation technology different data entries, which are based on different sources. As these sources differ by country and year, information on the corresponding reference are directly given with the data entry. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>technology</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>Energy source levels and technology</td>\n",
       "      <td>Energy source levels and technology</td>\n",
       "      <td>Energy source levels and technology</td>\n",
       "      <td>Energy source levels and technology</td>\n",
       "      <td>Energy source levels and technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>Total generation capacity</td>\n",
       "      <td>Generation capacity by energy type (fossil, nu...</td>\n",
       "      <td>Generation capacity by fuel or energy source (...</td>\n",
       "      <td>Generation capacity by fuel or energy source a...</td>\n",
       "      <td>Generation capacity by fuel and technology (e....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              1  \\\n",
       "technology                                        \n",
       "NaN         Energy source levels and technology   \n",
       "NaN                   Total generation capacity   \n",
       "NaN                                         NaN   \n",
       "NaN                                         NaN   \n",
       "NaN                                         NaN   \n",
       "\n",
       "                                                            2  \\\n",
       "technology                                                      \n",
       "NaN                       Energy source levels and technology   \n",
       "NaN         Generation capacity by energy type (fossil, nu...   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "\n",
       "                                                            3  \\\n",
       "technology                                                      \n",
       "NaN                       Energy source levels and technology   \n",
       "NaN         Generation capacity by fuel or energy source (...   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "\n",
       "                                                            4  \\\n",
       "technology                                                      \n",
       "NaN                       Energy source levels and technology   \n",
       "NaN         Generation capacity by fuel or energy source a...   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "NaN                                                       NaN   \n",
       "\n",
       "                                                            5  \n",
       "technology                                                     \n",
       "NaN                       Energy source levels and technology  \n",
       "NaN         Generation capacity by fuel and technology (e....  \n",
       "NaN                                                       NaN  \n",
       "NaN                                                       NaN  \n",
       "NaN                                                       NaN  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_file = 'National_Generation_Capacities.xlsx'\n",
    "filepath = os.path.join('input', data_file)\n",
    "\n",
    "# Read data into pandas\n",
    "data_raw = pd.read_excel(filepath,\n",
    "                         sheet_name='Summary',\n",
    "                         header=None,\n",
    "                         na_values=['-'],\n",
    "                         skiprows=0)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Deal with merged cells from Excel: fill first three rows with information\n",
    "data_raw.iloc[0:2] = data_raw.iloc[0:2].fillna(method='ffill', axis=1)\n",
    "\n",
    "# Set index for rows\n",
    "data_raw = data_raw.set_index([0])\n",
    "data_raw.index.name = 'technology'\n",
    "\n",
    "# Extract energylevels from raw data for later use\n",
    "energylevels_raw = data_raw.iloc[:, 0:5]\n",
    "\n",
    "energylevels_raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th colspan=\"6\" halign=\"left\">AT</th>\n",
       "      <th colspan=\"4\" halign=\"left\">BE</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SI</th>\n",
       "      <th colspan=\"7\" halign=\"left\">SK</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>type</th>\n",
       "      <th colspan=\"6\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th colspan=\"4\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"3\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th colspan=\"7\" halign=\"left\">Installed capacity in MW</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <th>2014.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>2019.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>...</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2013.0</th>\n",
       "      <th>2014.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>...</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source_type</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>...</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weblink</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>...</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>...</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>technology</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Fossil fuels</th>\n",
       "      <td>7243.63</td>\n",
       "      <td>7059.07</td>\n",
       "      <td>7323</td>\n",
       "      <td>6469</td>\n",
       "      <td>6492</td>\n",
       "      <td>6041</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>1442.97</td>\n",
       "      <td>1450.5</td>\n",
       "      <td>1223</td>\n",
       "      <td>3348</td>\n",
       "      <td>2907</td>\n",
       "      <td>2586</td>\n",
       "      <td>3164.51</td>\n",
       "      <td>3266.97</td>\n",
       "      <td>2581.47</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>- Lignite</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>- Hard coal</th>\n",
       "      <td>1171</td>\n",
       "      <td>1171</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1012</td>\n",
       "      <td>1012</td>\n",
       "      <td>598</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>- Oil</th>\n",
       "      <td>333.736</td>\n",
       "      <td>173.636</td>\n",
       "      <td>NaN</td>\n",
       "      <td>168</td>\n",
       "      <td>168</td>\n",
       "      <td>164</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>140</td>\n",
       "      <td>138</td>\n",
       "      <td>138</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>- Natural gas</th>\n",
       "      <td>4888.66</td>\n",
       "      <td>4820.19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4853</td>\n",
       "      <td>4876</td>\n",
       "      <td>4429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1016</td>\n",
       "      <td>1022</td>\n",
       "      <td>959</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 157 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "country                                                                                                                                        AT  \\\n",
       "type                                                                                                                     Installed capacity in MW   \n",
       "year                                                                                                                                       2014.0   \n",
       "source                                                                                                                                  e-control   \n",
       "source_type                                                                                                                  Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538   \n",
       "capacity_definition                                                                                                                Gross capacity   \n",
       "technology                                                                                                                                          \n",
       "Fossil fuels                                                   7243.63                                                                              \n",
       "- Lignite                                                            0                                                                              \n",
       "- Hard coal                                                       1171                                                                              \n",
       "- Oil                                                          333.736                                                                              \n",
       "- Natural gas                                                  4888.66                                                                              \n",
       "\n",
       "country                                                                                                                                            \\\n",
       "type                                                                                                                                                \n",
       "year                                                                                                                                       2015.0   \n",
       "source                                                                                                                                  e-control   \n",
       "source_type                                                                                                                  Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538   \n",
       "capacity_definition                                                                                                                Gross capacity   \n",
       "technology                                                                                                                                          \n",
       "Fossil fuels                                                   7059.07                                                                              \n",
       "- Lignite                                                            0                                                                              \n",
       "- Hard coal                                                       1171                                                                              \n",
       "- Oil                                                          173.636                                                                              \n",
       "- Natural gas                                                  4820.19                                                                              \n",
       "\n",
       "country                                                                                                                                            \\\n",
       "type                                                                                                                                                \n",
       "year                                                                                                                                       2016.0   \n",
       "source                                                                                                                                  e-control   \n",
       "source_type                                                                                                                  Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538   \n",
       "capacity_definition                                                                                                                Gross capacity   \n",
       "technology                                                                                                                                          \n",
       "Fossil fuels                                                      7323                                                                              \n",
       "- Lignite                                                          NaN                                                                              \n",
       "- Hard coal                                                        NaN                                                                              \n",
       "- Oil                                                              NaN                                                                              \n",
       "- Natural gas                                                      NaN                                                                              \n",
       "\n",
       "country                                                                                                                                            \\\n",
       "type                                                                                                                                                \n",
       "year                                                                                                                                       2017.0   \n",
       "source                                                                                                                                  e-control   \n",
       "source_type                                                                                                                  Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538   \n",
       "capacity_definition                                                                                                                Gross capacity   \n",
       "technology                                                                                                                                          \n",
       "Fossil fuels                                                      6469                                                                              \n",
       "- Lignite                                                          NaN                                                                              \n",
       "- Hard coal                                                       1012                                                                              \n",
       "- Oil                                                              168                                                                              \n",
       "- Natural gas                                                     4853                                                                              \n",
       "\n",
       "country                                                                                                                                            \\\n",
       "type                                                                                                                                                \n",
       "year                                                                                                                                       2018.0   \n",
       "source                                                                                                                                  e-control   \n",
       "source_type                                                                                                                  Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538   \n",
       "capacity_definition                                                                                                                Gross capacity   \n",
       "technology                                                                                                                                          \n",
       "Fossil fuels                                                      6492                                                                              \n",
       "- Lignite                                                          NaN                                                                              \n",
       "- Hard coal                                                       1012                                                                              \n",
       "- Oil                                                              168                                                                              \n",
       "- Natural gas                                                     4876                                                                              \n",
       "\n",
       "country                                                                                                                                                  \\\n",
       "type                                                                                                                                                      \n",
       "year                                                                                                                                             2019.0   \n",
       "source                                                                                                                                        e-control   \n",
       "source_type                                                                                                                        Regulatory Authority   \n",
       "weblink             https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390   \n",
       "capacity_definition                                                                                                                      Gross capacity   \n",
       "technology                                                                                                                                                \n",
       "Fossil fuels                                                      6041                                                                                    \n",
       "- Lignite                                                          NaN                                                                                    \n",
       "- Hard coal                                                        598                                                                                    \n",
       "- Oil                                                              164                                                                                    \n",
       "- Natural gas                                                     4429                                                                                    \n",
       "\n",
       "country                                                                                  BE  \\\n",
       "type                                                               Installed capacity in MW   \n",
       "year                                                                                 2015.0   \n",
       "source                                                                                 ELIA   \n",
       "source_type                                                    Transmission System Operator   \n",
       "weblink             https://www.elia.be/en/grid-data/power-generation/generating-facilities   \n",
       "capacity_definition                                                                 Unknown   \n",
       "technology                                                                                    \n",
       "Fossil fuels                                                       NaN                        \n",
       "- Lignite                                                          NaN                        \n",
       "- Hard coal                                                        NaN                        \n",
       "- Oil                                                              NaN                        \n",
       "- Natural gas                                                      NaN                        \n",
       "\n",
       "country                                                                                      \\\n",
       "type                                                                                          \n",
       "year                                                                                 2016.0   \n",
       "source                                                                                 ELIA   \n",
       "source_type                                                    Transmission System Operator   \n",
       "weblink             https://www.elia.be/en/grid-data/power-generation/generating-facilities   \n",
       "capacity_definition                                                                 Unknown   \n",
       "technology                                                                                    \n",
       "Fossil fuels                                                       NaN                        \n",
       "- Lignite                                                          NaN                        \n",
       "- Hard coal                                                        NaN                        \n",
       "- Oil                                                              NaN                        \n",
       "- Natural gas                                                      NaN                        \n",
       "\n",
       "country                                                                                      \\\n",
       "type                                                                                          \n",
       "year                                                                                 2017.0   \n",
       "source                                                                                 ELIA   \n",
       "source_type                                                    Transmission System Operator   \n",
       "weblink             https://www.elia.be/en/grid-data/power-generation/generating-facilities   \n",
       "capacity_definition                                                                 Unknown   \n",
       "technology                                                                                    \n",
       "Fossil fuels                                                       NaN                        \n",
       "- Lignite                                                          NaN                        \n",
       "- Hard coal                                                        NaN                        \n",
       "- Oil                                                              NaN                        \n",
       "- Natural gas                                                      NaN                        \n",
       "\n",
       "country                                                                                      \\\n",
       "type                                                                                          \n",
       "year                                                                                 2018.0   \n",
       "source                                                                                 ELIA   \n",
       "source_type                                                    Transmission System Operator   \n",
       "weblink             https://www.elia.be/en/grid-data/power-generation/generating-facilities   \n",
       "capacity_definition                                                                 Unknown   \n",
       "technology                                                                                    \n",
       "Fossil fuels                                                       NaN                        \n",
       "- Lignite                                                          NaN                        \n",
       "- Hard coal                                                        NaN                        \n",
       "- Oil                                                              NaN                        \n",
       "- Natural gas                                                      NaN                        \n",
       "\n",
       "country              ...  \\\n",
       "type                 ...   \n",
       "year                 ...   \n",
       "source               ...   \n",
       "source_type          ...   \n",
       "weblink              ...   \n",
       "capacity_definition  ...   \n",
       "technology           ...   \n",
       "Fossil fuels         ...   \n",
       "- Lignite            ...   \n",
       "- Hard coal          ...   \n",
       "- Oil                ...   \n",
       "- Natural gas        ...   \n",
       "\n",
       "country                                                                                                                                             SI  \\\n",
       "type                                                                                                                          Installed capacity in MW   \n",
       "year                                                                                                                                            2015.0   \n",
       "source                                                                                                                            Agencija za energijo   \n",
       "source_type                                                                                                                                   Ministry   \n",
       "weblink             https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043   \n",
       "capacity_definition                                                                                                                       Net capacity   \n",
       "technology                                                                                                                                               \n",
       "Fossil fuels                                                   1442.97                                                                                   \n",
       "- Lignite                                                          NaN                                                                                   \n",
       "- Hard coal                                                        NaN                                                                                   \n",
       "- Oil                                                              NaN                                                                                   \n",
       "- Natural gas                                                      NaN                                                                                   \n",
       "\n",
       "country                                                                                                                                                 \\\n",
       "type                                                                                                                                                     \n",
       "year                                                                                                                                            2016.0   \n",
       "source                                                                                                                            Agencija za energijo   \n",
       "source_type                                                                                                                                   Ministry   \n",
       "weblink             https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5   \n",
       "capacity_definition                                                                                                                       Net capacity   \n",
       "technology                                                                                                                                               \n",
       "Fossil fuels                                                    1450.5                                                                                   \n",
       "- Lignite                                                          NaN                                                                                   \n",
       "- Hard coal                                                        NaN                                                                                   \n",
       "- Oil                                                              NaN                                                                                   \n",
       "- Natural gas                                                      NaN                                                                                   \n",
       "\n",
       "country                                                                                                  \\\n",
       "type                                                                                                      \n",
       "year                                                                                             2017.0   \n",
       "source                                                                             Agencija za energijo   \n",
       "source_type                                                                                    Ministry   \n",
       "weblink             https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94   \n",
       "capacity_definition                                                                        Net capacity   \n",
       "technology                                                                                                \n",
       "Fossil fuels                                                      1223                                    \n",
       "- Lignite                                                          NaN                                    \n",
       "- Hard coal                                                        NaN                                    \n",
       "- Oil                                                              NaN                                    \n",
       "- Natural gas                                                      NaN                                    \n",
       "\n",
       "country                                         SK  \\\n",
       "type                      Installed capacity in MW   \n",
       "year                                        2013.0   \n",
       "source              Statistical Office of Slovakia   \n",
       "source_type                     Statistical Office   \n",
       "weblink                           Link unavailable   \n",
       "capacity_definition                        Unknown   \n",
       "technology                                           \n",
       "Fossil fuels                                  3348   \n",
       "- Lignite                                      NaN   \n",
       "- Hard coal                                    NaN   \n",
       "- Oil                                          140   \n",
       "- Natural gas                                 1016   \n",
       "\n",
       "country                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \\\n",
       "type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "year                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               2014.0   \n",
       "source                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Statistical Office of Slovakia   \n",
       "source_type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Statistical Office   \n",
       "weblink             https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/   \n",
       "capacity_definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Unknown   \n",
       "technology                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "Fossil fuels                                                      2907                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Lignite                                                          NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Hard coal                                                        NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Oil                                                              138                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Natural gas                                                     1022                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "\n",
       "country                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \\\n",
       "type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "year                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               2015.0   \n",
       "source                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Statistical Office of Slovakia   \n",
       "source_type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Statistical Office   \n",
       "weblink             https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/   \n",
       "capacity_definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Unknown   \n",
       "technology                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "Fossil fuels                                                      2586                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Lignite                                                          NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Hard coal                                                        NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Oil                                                              138                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Natural gas                                                      959                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "\n",
       "country                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \\\n",
       "type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "year                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               2016.0   \n",
       "source                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Statistical Office of Slovakia   \n",
       "source_type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Statistical Office   \n",
       "weblink             https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/   \n",
       "capacity_definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Unknown   \n",
       "technology                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "Fossil fuels                                                   3164.51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Lignite                                                          NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Hard coal                                                        NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Oil                                                              NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Natural gas                                                      NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "\n",
       "country                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \\\n",
       "type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "year                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               2017.0   \n",
       "source                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Statistical Office of Slovakia   \n",
       "source_type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Statistical Office   \n",
       "weblink             https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/   \n",
       "capacity_definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Unknown   \n",
       "technology                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "Fossil fuels                                                   3266.97                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Lignite                                                          NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Hard coal                                                        NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Oil                                                              NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "- Natural gas                                                      NaN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "\n",
       "country                                                                                                                   \\\n",
       "type                                                                                                                       \n",
       "year                                                                                                              2018.0   \n",
       "source                                                                                    Statistical Office of Slovakia   \n",
       "source_type                                                                                           Statistical Office   \n",
       "weblink             https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa   \n",
       "capacity_definition                                                                                              Unknown   \n",
       "technology                                                                                                                 \n",
       "Fossil fuels                                                   2581.47                                                     \n",
       "- Lignite                                                          NaN                                                     \n",
       "- Hard coal                                                        NaN                                                     \n",
       "- Oil                                                              NaN                                                     \n",
       "- Natural gas                                                      NaN                                                     \n",
       "\n",
       "country                   \n",
       "type                      \n",
       "year                 NaN  \n",
       "source                    \n",
       "source_type          NaN  \n",
       "weblink                   \n",
       "capacity_definition  NaN  \n",
       "technology                \n",
       "Fossil fuels         NaN  \n",
       "- Lignite            NaN  \n",
       "- Hard coal          NaN  \n",
       "- Oil                NaN  \n",
       "- Natural gas        NaN  \n",
       "\n",
       "[5 rows x 157 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete definition of energy levels from raw data\n",
    "data_raw.drop(data_raw.columns[[0, 1, 2, 3, 4, 5]], axis=1, inplace=True)\n",
    "\n",
    "level_names = ['country', 'type', 'year', 'source',\n",
    "               'source_type', 'weblink', 'capacity_definition']\n",
    "# Set multiindex column names\n",
    "data_raw.columns = pd.MultiIndex.from_arrays(data_raw[:7].values,\n",
    "                                             names=level_names)\n",
    "\n",
    "# Remove 3 rows which are already used as column names\n",
    "data_raw = data_raw[pd.notnull(data_raw.index)]\n",
    "\n",
    "# Extract the ordering of technologies\n",
    "technology_order = data_raw.index.str.replace('- ', '').values.tolist()\n",
    "\n",
    "data_raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex([], names=['country', 'type', 'year', 'source', 'source_type', 'weblink', 'capacity_definition'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_raw.columns[data_raw.columns.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th colspan=\"6\" halign=\"left\">AT</th>\n",
       "      <th colspan=\"4\" halign=\"left\">BE</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SI</th>\n",
       "      <th colspan=\"7\" halign=\"left\">SK</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>type</th>\n",
       "      <th colspan=\"6\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th colspan=\"4\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"3\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th colspan=\"7\" halign=\"left\">Installed capacity in MW</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <th>2014.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>2019.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>...</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2013.0</th>\n",
       "      <th>2014.0</th>\n",
       "      <th>2015.0</th>\n",
       "      <th>2016.0</th>\n",
       "      <th>2017.0</th>\n",
       "      <th>2018.0</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>e-control</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>ELIA</th>\n",
       "      <th>...</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Agencija za energijo</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th>Statistical Office of Slovakia</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source_type</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Regulatory Authority</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>Transmission System Operator</th>\n",
       "      <th>...</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Ministry</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weblink</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
       "      <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
       "      <th>...</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5</th>\n",
       "      <th>https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
       "      <th>https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Gross capacity</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>...</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Net capacity</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>technology</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 157 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [(AT, Installed capacity in MW, 2014.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2015.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2016.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2017.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2018.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2019.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390, Gross capacity), (BE, Installed capacity in MW, 2015.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2016.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2017.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2018.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2019.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BG, Installed capacity in MW, 2012.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/fileObj.php?oid=99, Unknown), (BG, Installed capacity in MW, 2019.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/?did=380, Unknown), (CH, Installed capacity in MW, 2012.0, UN Statistical Office, Statistical Office, http://data.un.org/Data.aspx?d=EDATA&f=cmID%3AEC, Net capacity), (CH, Installed capacity in MW, 2012.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzA3Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2014.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzkyNS5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2015.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODM0My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2016.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODY5My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2017.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTM5Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2018.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTc0OC5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2019.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik.exturl.html/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvMTAxMTI=.html, Unknown), (CZ, Installed capacity in MW, 2013.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2013.pdf/34a35d27-9c58-4c79-99d1-f0fbc5eac06a, Unknown), (CZ, Installed capacity in MW, 2014.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2014.pdf/f23d80b5-668a-42c0-9d04-1a19556c9c58, Unknown), (CZ, Installed capacity in MW, 2015.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2015.pdf/2548f33b-7964-4537-8b97-3943c7d83854, Unknown), (CZ, Installed capacity in MW, 2016.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2016.pdf/bfbc88b7-a814-498f-99bf-98b50148c805, Unknown), (CZ, Installed capacity in MW, 2017.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/4580209/Yearly_report_electricity_2017.pdf/a4042a98-8750-4524-bb3c-f172ecba2f3b, Unknown), (CZ, Installed capacity in MW, 2018.0, ERU, Regulatory Authority, http://www.eru.cz/documents/10540/4580207/Yearly_report_electricity_2018.pdf/f25a55d8-6730-4521-8e40-96d8e5f00c70, Unknown), (DE, Installed capacity in MW, 2013.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2014.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2015.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2016.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2017.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2018.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2019.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DK, Installed capacity in MW, 2013.0, DEA, Ministry, Link unavailable, Unknown), (DK, Installed capacity in MW, 2014.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2014.pdf, Unknown), (DK, Installed capacity in MW, 2015.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2015.pdf, Unknown), (DK, Installed capacity in MW, 2016.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2016.pdf, Unknown), (DK, Installed capacity in MW, 2017.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2017.pdf, Unknown), (DK, Installed capacity in MW, 2018.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2018.pdf, Unknown), (EE, Installed capacity in MW, 2013.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2014.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2015.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2016.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2017.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2018.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2019.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (ES, Installed capacity in MW, 2013.0, REE, Transmission System Operator, Link unavailable, Unknown), (ES, Installed capacity in MW, 2014.0, REE, Transmission System Operator, nan, Unknown), (ES, Installed capacity in MW, 2015.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2016.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2017.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2018.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2019.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2020.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/agosto_2020/6_Potencia_instalada_08_2020.xlsm, Unknown), (FR, Installed capacity in MW, 2014.0, RTE, Transmission System Operator, https://www.rte-france.com/sites/default/files/bilan_electrique_2014_en.pdf, Unknown), (FR, Installed capacity in MW, 2015.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2016.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2017.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2018.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2019.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2020.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FI, Installed capacity in MW, 2012.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2013.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2014.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2015.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (GB, Installed capacity in MW, 2015.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2016.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2017.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2018.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2019.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/904812/DUKES_5.7.xls, Unknown), (GR, Installed capacity in MW, 2012.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2015, Unknown), (GR, Installed capacity in MW, 2013.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2016, Unknown), (GR, Installed capacity in MW, 2014.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2017, Unknown), (GR, Installed capacity in MW, 2015.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2016.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2017.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2018.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2019, Unknown), (HR, Installed capacity in MW, 2013.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/qyxRItZNXQL84ygVm4NbnE/publications/Temeljni_podaci_2013.pdf, Unknown), (HR, Installed capacity in MW, 2014.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/vHpFOrmCvaEk2wfzW8lBZ6/publications/Temeljni_podaci_za_2014.pdf, Unknown), (HR, Installed capacity in MW, 2015.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/PsIM6mazLoTDVjKzm4NbnE/publications/Temeljni_podaci_2015.pdf, Unknown), (HR, Installed capacity in MW, 2016.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/wEfz2PYHVQLQIygVm4NbnE/publications/Temeljni_podaci_2016.pdf, Unknown), (HR, Installed capacity in MW, 2017.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/KzqT4wOZhIFGvbKzm4t0IF/annual-report-hops/HOPS_GI_2017_-__za_web.pdf, Unknown), (HR, Installed capacity in MW, 2018.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/x2VUqipcFKEnEgg50EV391/annual-report-hops/GI%202018%20unutarnje%20-%209-7-19%20web.pdf, Unknown), (HU, Installed capacity in MW, 2013.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/160362/Planned+Installed+generation+capacity+for+2010_2011_2012_2013+++4.3.1.pdf/84e86e8b-3b7c-4377-8020-c4beec92c914, Unknown), (HU, Installed capacity in MW, 2014.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2015.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2016.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2017.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2018.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/230112823/VER+BT_20181231.pdf/20c127e3-bb04-8b84-925f-2548761a41d4, Unknown), (IE, Installed capacity in MW, 2014.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2015.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2019.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/EirGrid-Group-All-Island-Generation-Capacity-Statement-2019-2028.pdf, Unknown), (IE, Installed capacity in MW, 2020.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/All-Island-Generation-Capacity-Statement-2020-2029.pdf, Unknown), (IT, Installed capacity in MW, 2014.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0775/97.PDF, Gross capacity), (IT, Installed capacity in MW, 2015.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0995/02.PDF, Gross capacity), (IT, Installed capacity in MW, 2016.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/1112/78.PDF, Gross capacity), (IT, Installed capacity in MW, 2017.0, Terna, Transmission System Operator, https://download.terna.it/terna/2017_Eng_3-POWER_PLANTS_8d73131c70720ed.pdf, Gross capacity), (IT, Installed capacity in MW, 2018.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), (IT, Installed capacity in MW, 2019.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), ...]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 157 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_raw[data_raw.index.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>type</th>\n",
       "      <th>year</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>weblink</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>capacity</th>\n",
       "      <th>comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>2014</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7243.634</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>2015</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7059.065</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>2016</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7323.000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>2017</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>6469.000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>2018</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>6492.000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     technology country                      type  year     source  \\\n",
       "0  Fossil fuels      AT  Installed capacity in MW  2014  e-control   \n",
       "1  Fossil fuels      AT  Installed capacity in MW  2015  e-control   \n",
       "2  Fossil fuels      AT  Installed capacity in MW  2016  e-control   \n",
       "3  Fossil fuels      AT  Installed capacity in MW  2017  e-control   \n",
       "4  Fossil fuels      AT  Installed capacity in MW  2018  e-control   \n",
       "\n",
       "            source_type                                            weblink  \\\n",
       "0  Regulatory Authority  https://www.e-control.at/documents/1785851/181...   \n",
       "1  Regulatory Authority  https://www.e-control.at/documents/1785851/181...   \n",
       "2  Regulatory Authority  https://www.e-control.at/documents/1785851/181...   \n",
       "3  Regulatory Authority  https://www.e-control.at/documents/1785851/181...   \n",
       "4  Regulatory Authority  https://www.e-control.at/documents/1785851/181...   \n",
       "\n",
       "  capacity_definition  capacity comment  \n",
       "0      Gross capacity  7243.634     NaN  \n",
       "1      Gross capacity  7059.065     NaN  \n",
       "2      Gross capacity  7323.000     NaN  \n",
       "3      Gross capacity  6469.000     NaN  \n",
       "4      Gross capacity  6492.000     NaN  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reshape dataframe to list\n",
    "data_opsd = pd.DataFrame(data_raw.stack(level=level_names))\n",
    "\n",
    "# Reset index for dataframe\n",
    "data_opsd.reset_index(inplace=True)\n",
    "data_opsd['technology'] = data_opsd['technology'].str.replace('- ', '')\n",
    "\n",
    "data_opsd.rename(columns={0: 'capacity'}, inplace=True)\n",
    "data_opsd['capacity'] = pd.to_numeric(data_opsd['capacity'], errors='coerce')\n",
    "\n",
    "# For some source, permission to publish data\n",
    "banlist = ['ELIA', 'BMWi', 'Mavir']\n",
    "davail = 'data available, but cannot be provided'\n",
    "data_opsd.loc[data_opsd['source'].isin(banlist), 'comment'] = davail\n",
    "\n",
    "data_opsd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 EUROSTAT data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EUROSTAT publishes annual structural data on national electricity generation capacities for European countries. The dataset is available in the EUROSTAT database within the category 'Environment and Energy' ([nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unit</th>\n",
       "      <th>product</th>\n",
       "      <th>indic_nrg</th>\n",
       "      <th>geo\\time</th>\n",
       "      <th>2016</th>\n",
       "      <th>2015</th>\n",
       "      <th>2014</th>\n",
       "      <th>2013</th>\n",
       "      <th>2012</th>\n",
       "      <th>2011</th>\n",
       "      <th>...</th>\n",
       "      <th>1999</th>\n",
       "      <th>1998</th>\n",
       "      <th>1997</th>\n",
       "      <th>1996</th>\n",
       "      <th>1995</th>\n",
       "      <th>1994</th>\n",
       "      <th>1993</th>\n",
       "      <th>1992</th>\n",
       "      <th>1991</th>\n",
       "      <th>1990</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AL</td>\n",
       "      <td>98</td>\n",
       "      <td>98</td>\n",
       "      <td>98</td>\n",
       "      <td>98</td>\n",
       "      <td>97</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AT</td>\n",
       "      <td>5561</td>\n",
       "      <td>5964</td>\n",
       "      <td>6136</td>\n",
       "      <td>6326</td>\n",
       "      <td>6347</td>\n",
       "      <td>6339</td>\n",
       "      <td>...</td>\n",
       "      <td>5089</td>\n",
       "      <td>4917</td>\n",
       "      <td>5199</td>\n",
       "      <td>5031</td>\n",
       "      <td>5028</td>\n",
       "      <td>5079</td>\n",
       "      <td>5061</td>\n",
       "      <td>5133</td>\n",
       "      <td>4750</td>\n",
       "      <td>4743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BA</td>\n",
       "      <td>2065</td>\n",
       "      <td>1765</td>\n",
       "      <td>1765</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BE</td>\n",
       "      <td>7182</td>\n",
       "      <td>7199</td>\n",
       "      <td>7202</td>\n",
       "      <td>7559</td>\n",
       "      <td>8182</td>\n",
       "      <td>9128</td>\n",
       "      <td>...</td>\n",
       "      <td>7746</td>\n",
       "      <td>7650</td>\n",
       "      <td>6945</td>\n",
       "      <td>7126</td>\n",
       "      <td>7211</td>\n",
       "      <td>7298</td>\n",
       "      <td>6433</td>\n",
       "      <td>6394</td>\n",
       "      <td>6445</td>\n",
       "      <td>6490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BG</td>\n",
       "      <td>3635</td>\n",
       "      <td>3854</td>\n",
       "      <td>4410</td>\n",
       "      <td>4636</td>\n",
       "      <td>4849</td>\n",
       "      <td>4495</td>\n",
       "      <td>...</td>\n",
       "      <td>5172</td>\n",
       "      <td>4977</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  unit  product   indic_nrg geo\\time  2016   2015   2014   2013   2012   \\\n",
       "0   MW     9007  12_1176011       AL    98     98     98     98     97    \n",
       "1   MW     9007  12_1176011       AT  5561   5964   6136   6326   6347    \n",
       "2   MW     9007  12_1176011       BA  2065   1765   1765      0      0    \n",
       "3   MW     9007  12_1176011       BE  7182   7199   7202   7559   8182    \n",
       "4   MW     9007  12_1176011       BG  3635   3854   4410   4636   4849    \n",
       "\n",
       "   2011   ...  1999   1998   1997   1996   1995   1994   1993   1992   1991   \\\n",
       "0     0   ...     0      0      0      0      0      0      0      0      0    \n",
       "1  6339   ...  5089   4917   5199   5031   5028   5079   5061   5133   4750    \n",
       "2     0   ...     0      0      0      0      0      0      0      0      0    \n",
       "3  9128   ...  7746   7650   6945   7126   7211   7298   6433   6394   6445    \n",
       "4  4495   ...  5172   4977      0      0      0      0      0      0      0    \n",
       "\n",
       "   1990  \n",
       "0     0  \n",
       "1  4743  \n",
       "2     0  \n",
       "3  6490  \n",
       "4     0  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_eurostat = 'Link unavailable'\n",
    "\n",
    "filepath_eurostat = os.path.join('input', 'Eurostat', 'Eurostat.tsv.gz')\n",
    "\n",
    "data_eurostat = pd.read_csv(filepath_eurostat,\n",
    "                               compression='gzip',\n",
    "                               sep='\\t|,',\n",
    "                               engine='python'\n",
    "                               )\n",
    "data_eurostat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unit</th>\n",
       "      <th>product</th>\n",
       "      <th>indic_nrg</th>\n",
       "      <th>geo\\time</th>\n",
       "      <th>year</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AL</td>\n",
       "      <td>2016</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>5561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BA</td>\n",
       "      <td>2016</td>\n",
       "      <td>2065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BE</td>\n",
       "      <td>2016</td>\n",
       "      <td>7182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BG</td>\n",
       "      <td>2016</td>\n",
       "      <td>3635</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  unit  product   indic_nrg geo\\time   year  value\n",
       "0   MW     9007  12_1176011       AL  2016     98 \n",
       "1   MW     9007  12_1176011       AT  2016   5561 \n",
       "2   MW     9007  12_1176011       BA  2016   2065 \n",
       "3   MW     9007  12_1176011       BE  2016   7182 \n",
       "4   MW     9007  12_1176011       BG  2016   3635 "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_vars = ['unit', 'product','indic_nrg', 'geo\\\\time']\n",
    "data_eurostat = pd.melt(data_eurostat, id_vars=id_vars,\n",
    "                        var_name='year', value_name='value')\n",
    "\n",
    "data_eurostat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_definition = pd.read_csv(os.path.join('input', 'definition_EUROSTAT_indic.txt'),\n",
    "                              header=None,\n",
    "                              names=['indic', 'description',\n",
    "                                     'energy source'],\n",
    "                              sep='\\t')\n",
    "\n",
    "data_eurostat = data_eurostat.merge(data_definition,\n",
    "                                    how='left',\n",
    "                                    left_on='indic_nrg',\n",
    "                                    right_on='indic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classification of generation capacities in the EUROSTAT dataset is specified in [Regulation (EC) No 1099/2008](http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32008R1099&from=EN) (Annex B, 3.3). The available EUROSTAT dataset [nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view) covers the following indicators:\n",
    "\n",
    "|indic_nrg   | Description  | Technology in OPSD |\n",
    "|---|---|---|\n",
    "|**12_1176011**|\t**Electrical capacity, main activity producers - Combustible Fuels**|\tFossil fuels & bioenergy|\n",
    "|**12_1176012**|\t**Electrical capacity, autoproducers - Combustible Fuels**|\tFossil fuels & bioenergy|\n",
    "|*12_1176061*|\t*Electrical capacity, main activity producers - Mixed plants*|\t|\n",
    "|*12_1176101*|\t*Electrical capacity, main activity producers - Other Sources*|\t|\n",
    "|*12_1176102*|\t*Electrical capacity, autoproducers - Other Sources*|\t|\n",
    "|*12_1176111*|\t*Electrical capacity, main activity producers - Steam*||\n",
    "|*12_1176112*|\t*Electrical capacity, autoproducers - Steam*||\n",
    "|*12_1176121*|\t*Electrical capacity, main activity producers - Gas Turbine*||\n",
    "|*12_1176122*|\t*Electrical capacity, autoproducers - Gas Turbine*||\n",
    "|*12_1176131*|\t*Electrical capacity, main activity producers - Combined Cycle*||\n",
    "|*12_1176132*|\t*Electrical capacity, autoproducers - Combined Cycle*||\n",
    "|*12_1176141*|\t*Electrical capacity, main activity producers - Internal Combustion*||\n",
    "|*12_1176142*|\t*Electrical capacity, autoproducers - Internal Combustion*||\n",
    "|*12_1176401*|\t*Electrical capacity, main activity producers - Other Type of Generation*|\t|\n",
    "|*12_1176402*|\t*Electrical capacity, autoproducers - Other Type of Generation*|\t|\n",
    "|12_1176253|\tNet maximum capacity - Municipal Wastes|\tNon-renewable waste|\n",
    "|12_1176263|\tNet maximum capacity - Wood/Wood Wastes/Other Solid Wastes|\tOther bioenergy and renewable waste|\n",
    "|12_1176273|\tNet maximum capacity - Biogases|\tBiomass and biogas|\n",
    "|12_1176283|\tNet maximum capacity - Industrial Wastes (non-renewable)|\tNon-renewable waste|\n",
    "|12_1176343|\tNet maximum capacity - Liquid Biofuels|\tBiomass and biogas|\n",
    "|**12_1176031**|\t**Electrical capacity, main activity producers - Nuclear**|\tNuclear|\n",
    "|**12_1176032**|\t**Electrical capacity, autoproducers - Nuclear**|\tNuclear|\n",
    "|**12_1176051**|\t**Electrical capacity, main activity producers - Hydro**|\tHydro|\n",
    "|**12_1176052**|\t**Electrical capacity, autoproducers - Hydro**|\tHydro|\n",
    "|12_1176071|\tNet electrical capacity, main activity producers - Pure Pumped Hydro|\tPumped storage|\n",
    "|12_1176072|\tNet electrical capacity, autoproducers - Pure Pumped Hydro|\tPumped storage|\n",
    "|*12_117615*|\t*Net maximum capacity - Hydro <1 MW*|\t|\n",
    "|*12_117616*|\t*Net maximum capacity - Hydro >= 1 MW and <= 10 MW*|\t|\n",
    "|*12_117617*|\t*Net maximum capacity - Hydro 10 MW and over*|\t|\n",
    "|**12_1176301**|\t**Electrical capacity, main activity producers - Tide, wave and ocean**|\tMarine|\n",
    "|**12_1176302**|\t**Electrical capacity, autoproducers - Tide, wave and ocean**|\tMarine|\n",
    "|*12_1176303*|\t*Net maximum capacity - Tide, Wave, Ocean*||\n",
    "|**12_1176081**|\t**Electrical capacity, main activity producers - Geothermal**|\tGeothermal|\n",
    "|**12_1176082**|\t**Electrical capacity, autoproducers - Geothermal**|\tGeothermal|\n",
    "|*12_1176083*|\t*Net maximum capacity - Geothermal*|\t|\n",
    "|**12_1176091**|\t**Electrical capacity, main activity producers - Wind**|\tWind|\n",
    "|**12_1176092**|\t**Electrical capacity, autoproducers - Wind**|\tWind|\n",
    "|**12_1176233**|\t**Net maximum capacity - Solar Photovoltaic**|\tPhotovoltaics|\n",
    "|**12_1176243**|\t**Net maximum capacity - Solar Thermal Electric**|\tConcentrated solar power|\n",
    "\n",
    "**Bold** rows indicate top level classes within the EUROSTAT classification, whereas normal and *italic* rows cover different kinds of subclassifications. Especially within the top level 'Combustible fuels' different kinds of subcategorizations based on fuel or technology are available. Simarily, 'Hydro' is differentiated by type (e.g. pumped-hydro storage) or capacity classes. *Italic* rows are not further considered within the OPSD dataset due to the mismatch with existing technology classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unit</th>\n",
       "      <th>product</th>\n",
       "      <th>indic_nrg</th>\n",
       "      <th>geo\\time</th>\n",
       "      <th>year</th>\n",
       "      <th>value</th>\n",
       "      <th>indic</th>\n",
       "      <th>description</th>\n",
       "      <th>energy source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AL</td>\n",
       "      <td>2016</td>\n",
       "      <td>98.0</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>Electrical capacity, main activity producers -...</td>\n",
       "      <td>Fossil fuels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>5561.0</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>Electrical capacity, main activity producers -...</td>\n",
       "      <td>Fossil fuels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BA</td>\n",
       "      <td>2016</td>\n",
       "      <td>2065.0</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>Electrical capacity, main activity producers -...</td>\n",
       "      <td>Fossil fuels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BE</td>\n",
       "      <td>2016</td>\n",
       "      <td>7182.0</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>Electrical capacity, main activity producers -...</td>\n",
       "      <td>Fossil fuels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MW</td>\n",
       "      <td>9007</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>BG</td>\n",
       "      <td>2016</td>\n",
       "      <td>3635.0</td>\n",
       "      <td>12_1176011</td>\n",
       "      <td>Electrical capacity, main activity producers -...</td>\n",
       "      <td>Fossil fuels</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  unit  product   indic_nrg geo\\time  year   value       indic  \\\n",
       "0   MW     9007  12_1176011       AL  2016    98.0  12_1176011   \n",
       "1   MW     9007  12_1176011       AT  2016  5561.0  12_1176011   \n",
       "2   MW     9007  12_1176011       BA  2016  2065.0  12_1176011   \n",
       "3   MW     9007  12_1176011       BE  2016  7182.0  12_1176011   \n",
       "4   MW     9007  12_1176011       BG  2016  3635.0  12_1176011   \n",
       "\n",
       "                                         description energy source  \n",
       "0  Electrical capacity, main activity producers -...  Fossil fuels  \n",
       "1  Electrical capacity, main activity producers -...  Fossil fuels  \n",
       "2  Electrical capacity, main activity producers -...  Fossil fuels  \n",
       "3  Electrical capacity, main activity producers -...  Fossil fuels  \n",
       "4  Electrical capacity, main activity producers -...  Fossil fuels  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_eurostat = data_eurostat[data_eurostat['energy source'].isnull() == False]\n",
    "\n",
    "values_as_string = data_eurostat['value'].astype(str)\n",
    "string_values = values_as_string.str.split(' ', 1).str[0]\n",
    "string_values.replace(':', np.nan, inplace=True)\n",
    "subset_nan = string_values.isnull()\n",
    "\n",
    "data_eurostat['value'] = string_values\n",
    "data_eurostat['year'] = data_eurostat['year'].astype(int)\n",
    "data_eurostat['value'] = data_eurostat['value'].astype(float)\n",
    "\n",
    "data_eurostat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_eurostat = data_eurostat.drop(['unit', 'product', 'indic_nrg',\n",
    "                                    'indic', 'description'], axis=1)\n",
    "\n",
    "data_eurostat = data_eurostat.rename(columns={'geo\\\\time': 'country',\n",
    "                                              'energy source': 'technology',\n",
    "                                              'value': 'capacity'})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>capacity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>1990</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>1990</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>1990</td>\n",
       "      <td>BA</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>1990</td>\n",
       "      <td>BE</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>1990</td>\n",
       "      <td>BG</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           technology  year country  capacity\n",
       "0  Biomass and biogas  1990      AL       0.0\n",
       "1  Biomass and biogas  1990      AT       0.0\n",
       "2  Biomass and biogas  1990      BA       0.0\n",
       "3  Biomass and biogas  1990      BE       1.0\n",
       "4  Biomass and biogas  1990      BG       0.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_eurostat['country'].replace({'UK': 'GB', 'EL': 'GR'}, inplace=True)\n",
    "drop_list = data_eurostat[data_eurostat['country'].isin(['EU28','EA19'])].index\n",
    "data_eurostat.drop(drop_list, inplace=True)\n",
    "\n",
    "by_columns = ['technology', 'year', 'country']\n",
    "data_eurostat = pd.DataFrame(data_eurostat.groupby(by_columns)['capacity'].sum())\n",
    "data_eurostat_isnull = data_eurostat['capacity'].isnull() == True\n",
    "data_eurostat.reset_index(inplace=True)\n",
    "\n",
    "data_eurostat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Concentrated solar power</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Geothermal</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Marine</th>\n",
       "      <th>Non-renewable waste</th>\n",
       "      <th>Nuclear</th>\n",
       "      <th>Other bioenergy and renewable waste</th>\n",
       "      <th>Photovoltaics</th>\n",
       "      <th>Pumped storage</th>\n",
       "      <th>Wind</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AL</th>\n",
       "      <th>1990</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1991</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1993</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "technology    Biomass and biogas  Concentrated solar power  Fossil fuels  \\\n",
       "country year                                                               \n",
       "AL      1990                 0.0                       0.0           0.0   \n",
       "        1991                 0.0                       0.0           0.0   \n",
       "        1992                 0.0                       0.0           0.0   \n",
       "        1993                 0.0                       0.0           0.0   \n",
       "        1994                 0.0                       0.0           0.0   \n",
       "\n",
       "technology    Geothermal   Hydro  Marine  Non-renewable waste  Nuclear  \\\n",
       "country year                                                             \n",
       "AL      1990         0.0     0.0     0.0                  0.0      0.0   \n",
       "        1991         0.0     0.0     0.0                  0.0      0.0   \n",
       "        1992         0.0  1453.0     0.0                  0.0      0.0   \n",
       "        1993         0.0  1453.0     0.0                  0.0      0.0   \n",
       "        1994         0.0  1453.0     0.0                  0.0      0.0   \n",
       "\n",
       "technology    Other bioenergy and renewable waste  Photovoltaics  \\\n",
       "country year                                                       \n",
       "AL      1990                                  0.0            0.0   \n",
       "        1991                                  0.0            0.0   \n",
       "        1992                                  0.0            0.0   \n",
       "        1993                                  0.0            0.0   \n",
       "        1994                                  0.0            0.0   \n",
       "\n",
       "technology    Pumped storage  Wind  \n",
       "country year                        \n",
       "AL      1990             0.0   0.0  \n",
       "        1991             0.0   0.0  \n",
       "        1992             0.0   0.0  \n",
       "        1993             0.0   0.0  \n",
       "        1994             0.0   0.0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eurostat_pivot = data_eurostat.pivot_table(values='capacity',\n",
    "                                 index=['country','year'],\n",
    "                                 columns='technology')\n",
    "\n",
    "eurostat_pivot.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "01 Oct 2020 16:07:59 - numexpr.utils - INFO - NumExpr defaulting to 4 threads.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Concentrated solar power</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Geothermal</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Marine</th>\n",
       "      <th>Non-renewable waste</th>\n",
       "      <th>Nuclear</th>\n",
       "      <th>Other bioenergy and renewable waste</th>\n",
       "      <th>Photovoltaics</th>\n",
       "      <th>Pumped storage</th>\n",
       "      <th>Wind</th>\n",
       "      <th>Differently categorized solar</th>\n",
       "      <th>Solar</th>\n",
       "      <th>Differently categorized wind</th>\n",
       "      <th>Bioenergy and renewable waste</th>\n",
       "      <th>Renewable energy sources</th>\n",
       "      <th>Differently categorized fossil fuels</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AL</th>\n",
       "      <th>1990</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1991</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1993</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1453.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "technology    Biomass and biogas  Concentrated solar power  Fossil fuels  \\\n",
       "country year                                                               \n",
       "AL      1990                 0.0                       0.0           0.0   \n",
       "        1991                 0.0                       0.0           0.0   \n",
       "        1992                 0.0                       0.0           0.0   \n",
       "        1993                 0.0                       0.0           0.0   \n",
       "        1994                 0.0                       0.0           0.0   \n",
       "\n",
       "technology    Geothermal   Hydro  Marine  Non-renewable waste  Nuclear  \\\n",
       "country year                                                             \n",
       "AL      1990         0.0     0.0     0.0                  0.0      0.0   \n",
       "        1991         0.0     0.0     0.0                  0.0      0.0   \n",
       "        1992         0.0  1453.0     0.0                  0.0      0.0   \n",
       "        1993         0.0  1453.0     0.0                  0.0      0.0   \n",
       "        1994         0.0  1453.0     0.0                  0.0      0.0   \n",
       "\n",
       "technology    Other bioenergy and renewable waste  Photovoltaics  \\\n",
       "country year                                                       \n",
       "AL      1990                                  0.0            0.0   \n",
       "        1991                                  0.0            0.0   \n",
       "        1992                                  0.0            0.0   \n",
       "        1993                                  0.0            0.0   \n",
       "        1994                                  0.0            0.0   \n",
       "\n",
       "technology    Pumped storage  Wind  Differently categorized solar  Solar  \\\n",
       "country year                                                               \n",
       "AL      1990             0.0   0.0                              0    0.0   \n",
       "        1991             0.0   0.0                              0    0.0   \n",
       "        1992             0.0   0.0                              0    0.0   \n",
       "        1993             0.0   0.0                              0    0.0   \n",
       "        1994             0.0   0.0                              0    0.0   \n",
       "\n",
       "technology    Differently categorized wind  Bioenergy and renewable waste  \\\n",
       "country year                                                                \n",
       "AL      1990                           0.0                            0.0   \n",
       "        1991                           0.0                            0.0   \n",
       "        1992                           0.0                            0.0   \n",
       "        1993                           0.0                            0.0   \n",
       "        1994                           0.0                            0.0   \n",
       "\n",
       "technology    Renewable energy sources  Differently categorized fossil fuels  \\\n",
       "country year                                                                   \n",
       "AL      1990                       0.0                                   0.0   \n",
       "        1991                       0.0                                   0.0   \n",
       "        1992                    1453.0                                   0.0   \n",
       "        1993                    1453.0                                   0.0   \n",
       "        1994                    1453.0                                   0.0   \n",
       "\n",
       "technology     Total  \n",
       "country year          \n",
       "AL      1990     0.0  \n",
       "        1991     0.0  \n",
       "        1992  1453.0  \n",
       "        1993  1453.0  \n",
       "        1994  1453.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eurostat_pivot['Differently categorized solar'] = 0\n",
    "eurostat_pivot['Solar'] = eurostat_pivot[['Photovoltaics', 'Concentrated solar power']].sum(axis=1)\n",
    "\n",
    "eurostat_pivot['Differently categorized wind'] = eurostat_pivot['Wind']\n",
    "\n",
    "bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n",
    "eurostat_pivot['Bioenergy and renewable waste'] = eurostat_pivot[bio_arr].sum(axis=1)\n",
    "\n",
    "res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste']\n",
    "eurostat_pivot['Renewable energy sources'] = eurostat_pivot[res_arr].sum(axis=1)\n",
    "\n",
    "\n",
    "eurostat_pivot['Fossil fuels'] = eurostat_pivot['Fossil fuels'] - eurostat_pivot['Bioenergy and renewable waste']\n",
    "eurostat_pivot['Differently categorized fossil fuels'] = eurostat_pivot['Fossil fuels']\\\n",
    "                                                         - eurostat_pivot['Non-renewable waste']\n",
    "\n",
    "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n",
    "eurostat_pivot['Total'] = eurostat_pivot[total_arr].sum(axis=1)\n",
    "\n",
    "eurostat_pivot.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>type</th>\n",
       "      <th>weblink</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AL</td>\n",
       "      <td>1990</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>0.0</td>\n",
       "      <td>EUROSTAT</td>\n",
       "      <td>Statistical Office</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Link unavailable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AL</td>\n",
       "      <td>1990</td>\n",
       "      <td>Concentrated solar power</td>\n",
       "      <td>0.0</td>\n",
       "      <td>EUROSTAT</td>\n",
       "      <td>Statistical Office</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Link unavailable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AL</td>\n",
       "      <td>1990</td>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>0.0</td>\n",
       "      <td>EUROSTAT</td>\n",
       "      <td>Statistical Office</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Link unavailable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AL</td>\n",
       "      <td>1990</td>\n",
       "      <td>Geothermal</td>\n",
       "      <td>0.0</td>\n",
       "      <td>EUROSTAT</td>\n",
       "      <td>Statistical Office</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Link unavailable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AL</td>\n",
       "      <td>1990</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>0.0</td>\n",
       "      <td>EUROSTAT</td>\n",
       "      <td>Statistical Office</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Link unavailable</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year                technology  capacity    source  \\\n",
       "0      AL  1990        Biomass and biogas       0.0  EUROSTAT   \n",
       "1      AL  1990  Concentrated solar power       0.0  EUROSTAT   \n",
       "2      AL  1990              Fossil fuels       0.0  EUROSTAT   \n",
       "3      AL  1990                Geothermal       0.0  EUROSTAT   \n",
       "4      AL  1990                     Hydro       0.0  EUROSTAT   \n",
       "\n",
       "          source_type capacity_definition                      type  \\\n",
       "0  Statistical Office             Unknown  Installed capacity in MW   \n",
       "1  Statistical Office             Unknown  Installed capacity in MW   \n",
       "2  Statistical Office             Unknown  Installed capacity in MW   \n",
       "3  Statistical Office             Unknown  Installed capacity in MW   \n",
       "4  Statistical Office             Unknown  Installed capacity in MW   \n",
       "\n",
       "            weblink  \n",
       "0  Link unavailable  \n",
       "1  Link unavailable  \n",
       "2  Link unavailable  \n",
       "3  Link unavailable  \n",
       "4  Link unavailable  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_eurostat = eurostat_pivot.stack().reset_index().rename(columns={0: 'capacity'})\n",
    "\n",
    "data_eurostat['source'] = 'EUROSTAT'\n",
    "data_eurostat['source_type'] = 'Statistical Office'\n",
    "data_eurostat['capacity_definition'] = 'Unknown'\n",
    "data_eurostat['type'] = 'Installed capacity in MW'\n",
    "data_eurostat['weblink'] = url_eurostat\n",
    "\n",
    "data_eurostat.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 ENTSO-E data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ENTSO-E publishes annual data on national generation capacites in different specifications and formats. We use two relevant data sources from the ENTSOE-E, which comprises firstly statistical data within the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx) or [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/), and secondly datasets compiled within the [ENTSO-E System Outlook & Adequacy Forecast (SO&AF)](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx). The ENTSO-E Transparency Platform is currently not implemented as a data source for national generation capacities.\n",
    "\n",
    "The advantage of the ENTSO-E SO&AF is the higher granularity of the data with respect to the main fuel or technology. However, as the SO&AF provides a forecast on future system conditions in particular peak hours, the dataset also accounts for expected capacity changes throughout the years. Therefore, we only consider years which are closest to the publication year of the respective SO&AF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.1 ENTSO-E Statistical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, we use the statistical data available in the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "01 Oct 2020 16:07:59 - root - INFO - Downloading file Statistics.xls\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>year</th>\n",
       "      <th>hydro</th>\n",
       "      <th>of which storage</th>\n",
       "      <th>of which run of river</th>\n",
       "      <th>of which pumped storage</th>\n",
       "      <th>nuclear</th>\n",
       "      <th>renewable</th>\n",
       "      <th>of which wind</th>\n",
       "      <th>of which solar</th>\n",
       "      <th>of which biomass</th>\n",
       "      <th>fossil_fueals</th>\n",
       "      <th>other</th>\n",
       "      <th>representativity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2010</td>\n",
       "      <td>12665.0</td>\n",
       "      <td>7205.0</td>\n",
       "      <td>5241.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1031.0</td>\n",
       "      <td>1002.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7389.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2011</td>\n",
       "      <td>13200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1542.0</td>\n",
       "      <td>1056.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7818.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2012</td>\n",
       "      <td>13350.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1906.0</td>\n",
       "      <td>1307.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>427.0</td>\n",
       "      <td>7834.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2013</td>\n",
       "      <td>13427.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2305.0</td>\n",
       "      <td>1555.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>426.0</td>\n",
       "      <td>7847.0</td>\n",
       "      <td>244.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>13569.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3413.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>589.0</td>\n",
       "      <td>512.0</td>\n",
       "      <td>7243.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Country  year    hydro  of which storage  of which run of river  \\\n",
       "0      AT  2010  12665.0            7205.0                 5241.0   \n",
       "1      AT  2011  13200.0               0.0                    0.0   \n",
       "2      AT  2012  13350.0               0.0                    0.0   \n",
       "3      AT  2013  13427.0               0.0                    0.0   \n",
       "4      AT  2014  13569.0               0.0                    0.0   \n",
       "\n",
       "   of which pumped storage  nuclear  renewable  of which wind  of which solar  \\\n",
       "0                      0.0        0     1031.0         1002.0             0.0   \n",
       "1                      0.0        0     1542.0         1056.0            55.0   \n",
       "2                      0.0        0     1906.0         1307.0           172.0   \n",
       "3                      0.0        0     2305.0         1555.0           324.0   \n",
       "4                      0.0        0     3413.0         2110.0           589.0   \n",
       "\n",
       "   of which biomass  fossil_fueals  other  representativity  \n",
       "0               0.0         7389.0    0.0             100.0  \n",
       "1               0.0         7818.0   68.0             100.0  \n",
       "2             427.0         7834.0   74.0             100.0  \n",
       "3             426.0         7847.0  244.0             100.0  \n",
       "4             512.0         7243.0    0.0             100.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_entsoe = 'https://docstore.entsoe.eu/Documents/Publications/Statistics/NGC_2010-2015.xlsx'\n",
    "\n",
    "filepath_entsoe = func.downloadandcache(url_entsoe, 'Statistics.xls',\n",
    "                                     os.path.join('ENTSO-E','Data Portal 2010-2015')\n",
    "                                     )\n",
    "\n",
    "data_entsoe_raw = pd.read_excel(filepath_entsoe)\n",
    "\n",
    "data_entsoe_raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Reservoir</th>\n",
       "      <th>Run-of-river</th>\n",
       "      <th>Pumped storage</th>\n",
       "      <th>Nuclear</th>\n",
       "      <th>renewable</th>\n",
       "      <th>Wind</th>\n",
       "      <th>Solar</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Other or unspecified energy sources</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2010</td>\n",
       "      <td>12665.0</td>\n",
       "      <td>7205.0</td>\n",
       "      <td>5241.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1031.0</td>\n",
       "      <td>1002.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7389.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2011</td>\n",
       "      <td>13200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1542.0</td>\n",
       "      <td>1056.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7818.0</td>\n",
       "      <td>68.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2012</td>\n",
       "      <td>13350.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1906.0</td>\n",
       "      <td>1307.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>427.0</td>\n",
       "      <td>7834.0</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2013</td>\n",
       "      <td>13427.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2305.0</td>\n",
       "      <td>1555.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>426.0</td>\n",
       "      <td>7847.0</td>\n",
       "      <td>244.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>13569.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3413.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>589.0</td>\n",
       "      <td>512.0</td>\n",
       "      <td>7243.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year    Hydro  Reservoir  Run-of-river  Pumped storage  Nuclear  \\\n",
       "0      AT  2010  12665.0     7205.0        5241.0             0.0        0   \n",
       "1      AT  2011  13200.0        0.0           0.0             0.0        0   \n",
       "2      AT  2012  13350.0        0.0           0.0             0.0        0   \n",
       "3      AT  2013  13427.0        0.0           0.0             0.0        0   \n",
       "4      AT  2014  13569.0        0.0           0.0             0.0        0   \n",
       "\n",
       "   renewable    Wind  Solar  Biomass and biogas  Fossil fuels  \\\n",
       "0     1031.0  1002.0    0.0                 0.0        7389.0   \n",
       "1     1542.0  1056.0   55.0                 0.0        7818.0   \n",
       "2     1906.0  1307.0  172.0               427.0        7834.0   \n",
       "3     2305.0  1555.0  324.0               426.0        7847.0   \n",
       "4     3413.0  2110.0  589.0               512.0        7243.0   \n",
       "\n",
       "   Other or unspecified energy sources  \n",
       "0                                  0.0  \n",
       "1                                 68.0  \n",
       "2                                 74.0  \n",
       "3                                244.0  \n",
       "4                                  0.0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_energy_source = {'hydro': 'Hydro',\n",
    "                      'of which storage': 'Reservoir',\n",
    "                      'of which run of river': 'Run-of-river',\n",
    "                      'of which pumped storage': 'Pumped storage',\n",
    "                      'nuclear': 'Nuclear',\n",
    "                      'of which wind': 'Wind',\n",
    "                      'of which solar': 'Solar',\n",
    "                      'of which biomass': 'Biomass and biogas',\n",
    "                      'fossil_fuels': 'Fossil fuels',\n",
    "                      'other': 'Other or unspecified energy sources',\n",
    "                      \"Country\": \"country\",\n",
    "                      'fossil_fueals': 'Fossil fuels'}\n",
    "\n",
    "data_entsoe_raw.rename(columns=dict_energy_source,\n",
    "                       inplace=True)\n",
    "\n",
    "data_entsoe_raw.drop(columns='representativity', inplace=True)\n",
    "\n",
    "data_entsoe_raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2010</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>12665.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2011</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2012</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13350.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2013</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13427.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13569.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year technology  capacity\n",
       "0      AT  2010      Hydro   12665.0\n",
       "1      AT  2011      Hydro   13200.0\n",
       "2      AT  2012      Hydro   13350.0\n",
       "3      AT  2013      Hydro   13427.0\n",
       "4      AT  2014      Hydro   13569.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "data_entsoe_raw['Differently categorized solar'] = data_entsoe_raw['Solar']\n",
    "data_entsoe_raw['Differently categorized wind'] = data_entsoe_raw['Wind']\n",
    "data_entsoe_raw['Bioenergy and renewable waste'] = data_entsoe_raw['Biomass and biogas']\n",
    "data_entsoe_raw['Differently categorized fossil fuels'] = data_entsoe_raw['Fossil fuels']\n",
    "\n",
    "\n",
    "data_entsoe_raw['Differently categorized hydro'] = (\n",
    "        data_entsoe_raw['Hydro']\n",
    "        - data_entsoe_raw['Run-of-river']\n",
    "        - data_entsoe_raw['Reservoir']\n",
    "        - data_entsoe_raw['Pumped storage'])\n",
    "\n",
    "data_entsoe_raw['Differently categorized renewable energy sources'] = (\n",
    "        data_entsoe_raw['renewable']\n",
    "        - data_entsoe_raw['Wind']\n",
    "        - data_entsoe_raw['Solar']\n",
    "        - data_entsoe_raw['Biomass and biogas'])\n",
    "\n",
    "data_entsoe_raw.drop(columns='renewable', inplace=True)\n",
    "\n",
    "data_entsoe_raw['Renewable energy sources'] = (\n",
    "        data_entsoe_raw['Hydro']\n",
    "        + data_entsoe_raw['Wind']\n",
    "        + data_entsoe_raw['Solar']\n",
    "        + data_entsoe_raw['Bioenergy and renewable waste']\n",
    "        + data_entsoe_raw['Differently categorized renewable energy sources'])\n",
    "\n",
    "data_entsoe_raw['Total'] = (\n",
    "        data_entsoe_raw['Renewable energy sources']\n",
    "        + data_entsoe_raw['Nuclear']\n",
    "        + data_entsoe_raw['Fossil fuels']\n",
    "        + data_entsoe_raw['Other or unspecified energy sources'])\n",
    "\n",
    "data_entsoe = pd.melt(data_entsoe_raw,\n",
    "                      id_vars=['country', 'year'],\n",
    "                      var_name='technology',\n",
    "                      value_name='capacity')\n",
    "\n",
    "data_entsoe.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2010</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>12665.0</td>\n",
       "      <td>ENTSO-E Data Portal</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2011</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13200.0</td>\n",
       "      <td>ENTSO-E Data Portal</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2012</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13350.0</td>\n",
       "      <td>ENTSO-E Data Portal</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2013</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13427.0</td>\n",
       "      <td>ENTSO-E Data Portal</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>13569.0</td>\n",
       "      <td>ENTSO-E Data Portal</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year technology  capacity               source        source_type  \\\n",
       "0      AT  2010      Hydro   12665.0  ENTSO-E Data Portal  Other association   \n",
       "1      AT  2011      Hydro   13200.0  ENTSO-E Data Portal  Other association   \n",
       "2      AT  2012      Hydro   13350.0  ENTSO-E Data Portal  Other association   \n",
       "3      AT  2013      Hydro   13427.0  ENTSO-E Data Portal  Other association   \n",
       "4      AT  2014      Hydro   13569.0  ENTSO-E Data Portal  Other association   \n",
       "\n",
       "  capacity_definition                      type  \n",
       "0        Net capacity  Installed capacity in MW  \n",
       "1        Net capacity  Installed capacity in MW  \n",
       "2        Net capacity  Installed capacity in MW  \n",
       "3        Net capacity  Installed capacity in MW  \n",
       "4        Net capacity  Installed capacity in MW  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_entsoe['country'].replace('NI', 'GB', inplace=True)\n",
    "# set negative capacities to zero\n",
    "data_entsoe.loc[data_entsoe['capacity'] < 0, 'capacity'] = 0\n",
    "\n",
    "data_entsoe['source'] = 'ENTSO-E Data Portal'\n",
    "data_entsoe['source_type'] = 'Other association'\n",
    "data_entsoe['capacity_definition'] = 'Net capacity'\n",
    "data_entsoe['type'] = 'Installed capacity in MW'\n",
    "\n",
    "data_entsoe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.2 ENTSO-E SO&AF data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "01 Oct 2020 16:08:00 - root - INFO - Downloading file SO_AF_2011_-_2025_.zip\n",
      "01 Oct 2020 16:08:01 - root - INFO - Extracting SO_AF_2011_-_2025_.zip into the directory download\\ENTSO-E\\SO&AF\\2011\n",
      "01 Oct 2020 16:08:01 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2011\\SO&AF 2011 - 2025 Scenario B.xls and transforming into a DataFrame\n",
      "01 Oct 2020 16:08:02 - root - INFO - Downloading file 120705_SOAF_2012_Dataset.zip\n",
      "01 Oct 2020 16:08:02 - root - INFO - Extracting 120705_SOAF_2012_Dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2012\n",
      "01 Oct 2020 16:08:02 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2012\\SOAF 2012 Scenario B.xls and transforming into a DataFrame\n",
      "01 Oct 2020 16:08:03 - root - INFO - Downloading file 130403_SOAF_2013-2030_dataset.zip\n",
      "01 Oct 2020 16:08:04 - root - INFO - Extracting 130403_SOAF_2013-2030_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2013\n",
      "01 Oct 2020 16:08:04 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2013\\ScB.xls and transforming into a DataFrame\n",
      "01 Oct 2020 16:08:05 - root - INFO - Downloading file 140602_SOAF%202014_dataset.zip\n",
      "01 Oct 2020 16:08:05 - root - INFO - Extracting 140602_SOAF%202014_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2014\n",
      "01 Oct 2020 16:08:05 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2014\\ScB.xlsx and transforming into a DataFrame\n",
      "01 Oct 2020 16:08:06 - root - INFO - Downloading file SO_AF_2015_dataset.zip\n",
      "01 Oct 2020 16:08:07 - root - INFO - Extracting SO_AF_2015_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2016\n",
      "01 Oct 2020 16:08:07 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2016\\SO&AF 2015 dataset\\ScB_publication.xlsx and transforming into a DataFrame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Lignite</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hard coal</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Natural gas</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     technology  capacity country  year\n",
       "1       Nuclear       0.0      AL  2011\n",
       "2  Fossil fuels       0.0      AL  2011\n",
       "3       Lignite       0.0      AL  2011\n",
       "4     Hard coal       0.0      AL  2011\n",
       "5   Natural gas       0.0      AL  2011"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soafs = [soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/SO_AF_2011_-_2025_.zip',\n",
    "                          'SO_AF_2011_-_2025_.zip',\n",
    "                          'SO&AF 2011 - 2025 Scenario B.xls',\n",
    "                          2011),\n",
    "\n",
    "        soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/120705_SOAF_2012_Dataset.zip',\n",
    "                         '120705_SOAF_2012_Dataset.zip',\n",
    "                         'SOAF 2012 Scenario B.xls',\n",
    "                         2012),\n",
    "                             \n",
    "        soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/publications/entsoe/So_AF_2013-2030/130403_SOAF_2013-2030_dataset.zip',\n",
    "                         '130403_SOAF_2013-2030_dataset.zip',\n",
    "                         'ScB.xls',\n",
    "                         2013),\n",
    "                             \n",
    "        soaf.SoafDataRaw('https://www.entsoe.eu/Documents/SDC%20documents/SOAF/140602_SOAF%202014_dataset.zip',\n",
    "                         '140602_SOAF%202014_dataset.zip',\n",
    "                         'ScB.xlsx',\n",
    "                         2014),\n",
    "                             \n",
    "        soaf.SoafDataRaw('https://www.entsoe.eu/Documents/Publications/SDC/data/SO_AF_2015_dataset.zip',\n",
    "                         'SO_AF_2015_dataset.zip',\n",
    "                         os.path.join('SO&AF 2015 dataset', 'ScB_publication.xlsx'),\n",
    "                         2016)]\n",
    "\n",
    "\n",
    "data_soaf = pd.concat([s.transformed_df for s in soafs])\n",
    "\n",
    "# Correct that in the Soaf2015 datatset the year column is 2016 instead of 2015\n",
    "data_soaf['year'].replace({2016 : 2015}, inplace=True)\n",
    "\n",
    "data_soaf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Hard coal</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Lignite</th>\n",
       "      <th>Mixed fossil fuels</th>\n",
       "      <th>Natural gas</th>\n",
       "      <th>Nuclear</th>\n",
       "      <th>Offshore</th>\n",
       "      <th>Oil</th>\n",
       "      <th>...</th>\n",
       "      <th>Solar</th>\n",
       "      <th>Wind</th>\n",
       "      <th>Differently categorized solar</th>\n",
       "      <th>Differently categorized wind</th>\n",
       "      <th>Differently categorized hydro</th>\n",
       "      <th>Bioenergy and renewable waste</th>\n",
       "      <th>Differently categorized renewable energy sources</th>\n",
       "      <th>Differently categorized fossil fuels</th>\n",
       "      <th>Renewable energy sources</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AL</th>\n",
       "      <th>2011</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1860.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1860.0</td>\n",
       "      <td>1960.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "technology    Biomass and biogas  Fossil fuels  Hard coal   Hydro  Lignite  \\\n",
       "country year                                                                 \n",
       "AL      2011                 0.0           0.0        0.0     0.0      0.0   \n",
       "        2012                 0.0           0.0        0.0     0.0      0.0   \n",
       "        2013                 0.0           0.0        0.0     0.0      0.0   \n",
       "        2014                 0.0           0.0        0.0     0.0      0.0   \n",
       "        2015                 0.0         100.0        0.0  1860.0      0.0   \n",
       "\n",
       "technology    Mixed fossil fuels  Natural gas  Nuclear  Offshore  Oil  ...  \\\n",
       "country year                                                           ...   \n",
       "AL      2011                 0.0          0.0      0.0       0.0  0.0  ...   \n",
       "        2012                 0.0          0.0      0.0       0.0  0.0  ...   \n",
       "        2013                 0.0          0.0      0.0       0.0  0.0  ...   \n",
       "        2014                 0.0          0.0      0.0       0.0  0.0  ...   \n",
       "        2015                 0.0        100.0      0.0       0.0  0.0  ...   \n",
       "\n",
       "technology    Solar  Wind  Differently categorized solar  \\\n",
       "country year                                               \n",
       "AL      2011    0.0   0.0                            0.0   \n",
       "        2012    0.0   0.0                            0.0   \n",
       "        2013    0.0   0.0                            0.0   \n",
       "        2014    0.0   0.0                            0.0   \n",
       "        2015    0.0   0.0                            0.0   \n",
       "\n",
       "technology    Differently categorized wind  Differently categorized hydro  \\\n",
       "country year                                                                \n",
       "AL      2011                           0.0                            0.0   \n",
       "        2012                           0.0                            0.0   \n",
       "        2013                           0.0                            NaN   \n",
       "        2014                           0.0                            NaN   \n",
       "        2015                           0.0                            NaN   \n",
       "\n",
       "technology    Bioenergy and renewable waste  \\\n",
       "country year                                  \n",
       "AL      2011                            0.0   \n",
       "        2012                            0.0   \n",
       "        2013                            0.0   \n",
       "        2014                            0.0   \n",
       "        2015                            0.0   \n",
       "\n",
       "technology    Differently categorized renewable energy sources  \\\n",
       "country year                                                     \n",
       "AL      2011                                               0.0   \n",
       "        2012                                               0.0   \n",
       "        2013                                               0.0   \n",
       "        2014                                               0.0   \n",
       "        2015                                               0.0   \n",
       "\n",
       "technology    Differently categorized fossil fuels  Renewable energy sources  \\\n",
       "country year                                                                   \n",
       "AL      2011                                   0.0                       0.0   \n",
       "        2012                                   0.0                       0.0   \n",
       "        2013                                   0.0                       0.0   \n",
       "        2014                                   0.0                       0.0   \n",
       "        2015                                   0.0                    1860.0   \n",
       "\n",
       "technology     Total  \n",
       "country year          \n",
       "AL      2011     0.0  \n",
       "        2012     0.0  \n",
       "        2013     0.0  \n",
       "        2014     0.0  \n",
       "        2015  1960.0  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soaf_unstacked = func.unstackData(data_soaf)\n",
    "\n",
    "soaf_unstacked['Differently categorized solar'] = soaf_unstacked['Solar']\n",
    "\n",
    "soaf_unstacked['Differently categorized wind'] = soaf_unstacked['Wind']\\\n",
    "                                                - soaf_unstacked['Offshore']\\\n",
    "                                                - soaf_unstacked['Onshore']\n",
    "\n",
    "soaf_unstacked['Differently categorized hydro'] = soaf_unstacked['Hydro']\\\n",
    "                                                - soaf_unstacked['Run-of-river']\\\n",
    "                                                - soaf_unstacked['Reservoir including pumped storage']\n",
    "                                                \n",
    "\n",
    "soaf_unstacked['Bioenergy and renewable waste'] = soaf_unstacked['Biomass and biogas']\n",
    "                                                \n",
    "soaf_unstacked['Differently categorized renewable energy sources'] = (\n",
    "                                          soaf_unstacked['renewable']\n",
    "                                        - soaf_unstacked['Wind']\n",
    "                                        - soaf_unstacked['Solar']\n",
    "                                        - soaf_unstacked['Biomass and biogas'])\n",
    "\n",
    "soaf_unstacked.drop(columns='renewable', inplace=True)\n",
    "\n",
    "subtract_fossils_arr = ['Lignite','Hard coal','Oil','Natural gas','Mixed fossil fuels']\n",
    "\n",
    "soaf_unstacked['Differently categorized fossil fuels'] = soaf_unstacked['Fossil fuels']\\\n",
    "                                                        - soaf_unstacked[subtract_fossils_arr].sum(axis=1)\n",
    "\n",
    "\n",
    "res_arr = ['Solar','Wind','Bioenergy and renewable waste','Hydro',\n",
    "           'Differently categorized renewable energy sources']\n",
    "\n",
    "soaf_unstacked['Renewable energy sources'] = soaf_unstacked[res_arr].sum(axis=1)\n",
    "\n",
    "total_arr = ['Renewable energy sources','Fossil fuels','Nuclear',\n",
    "             'Other or unspecified energy sources']\n",
    "\n",
    "soaf_unstacked['Total'] = soaf_unstacked[total_arr].sum(axis=1)\n",
    "\n",
    "soaf_unstacked.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>source</th>\n",
       "      <th>type</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>source_type</th>\n",
       "      <th>weblink</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ENTSO-E SOAF</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Other association</td>\n",
       "      <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ENTSO-E SOAF</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Other association</td>\n",
       "      <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "      <td>Hard coal</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ENTSO-E SOAF</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Other association</td>\n",
       "      <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "      <td>Hydro</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ENTSO-E SOAF</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Other association</td>\n",
       "      <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AL</td>\n",
       "      <td>2011</td>\n",
       "      <td>Lignite</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ENTSO-E SOAF</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Other association</td>\n",
       "      <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year          technology  capacity        source  \\\n",
       "0      AL  2011  Biomass and biogas       0.0  ENTSO-E SOAF   \n",
       "1      AL  2011        Fossil fuels       0.0  ENTSO-E SOAF   \n",
       "2      AL  2011           Hard coal       0.0  ENTSO-E SOAF   \n",
       "3      AL  2011               Hydro       0.0  ENTSO-E SOAF   \n",
       "4      AL  2011             Lignite       0.0  ENTSO-E SOAF   \n",
       "\n",
       "                       type capacity_definition        source_type  \\\n",
       "0  Installed capacity in MW        Net capacity  Other association   \n",
       "1  Installed capacity in MW        Net capacity  Other association   \n",
       "2  Installed capacity in MW        Net capacity  Other association   \n",
       "3  Installed capacity in MW        Net capacity  Other association   \n",
       "4  Installed capacity in MW        Net capacity  Other association   \n",
       "\n",
       "                                             weblink  \n",
       "0  https://docstore.entsoe.eu/Documents/Publicati...  \n",
       "1  https://docstore.entsoe.eu/Documents/Publicati...  \n",
       "2  https://docstore.entsoe.eu/Documents/Publicati...  \n",
       "3  https://docstore.entsoe.eu/Documents/Publicati...  \n",
       "4  https://docstore.entsoe.eu/Documents/Publicati...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_soaf = func.restackData(soaf_unstacked)\n",
    "\n",
    "data_soaf.loc[data_soaf['capacity'] < 0, 'capacity'] = 0\n",
    "\n",
    "data_soaf['source'] = 'ENTSO-E SOAF'\n",
    "data_soaf['type'] = 'Installed capacity in MW'\n",
    "data_soaf['capacity_definition'] = 'Net capacity'\n",
    "data_soaf['source_type'] = 'Other association'\n",
    "data_soaf['weblink'] = url_entsoe\n",
    "\n",
    "data_soaf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.3 ENTSO-E Transparency Platform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>DateTime</th>\n",
       "      <th>ResolutionCode</th>\n",
       "      <th>areacode</th>\n",
       "      <th>AreaTypeCode</th>\n",
       "      <th>AreaName</th>\n",
       "      <th>MapCode</th>\n",
       "      <th>ProductionType</th>\n",
       "      <th>AggregatedInstalledCapacity</th>\n",
       "      <th>DeletedFlag</th>\n",
       "      <th>UpdateTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 00:00:00.000</td>\n",
       "      <td>P1Y</td>\n",
       "      <td>10YAT-APG------L</td>\n",
       "      <td>CTA</td>\n",
       "      <td>APG CA</td>\n",
       "      <td>AT</td>\n",
       "      <td>Other</td>\n",
       "      <td>22.8</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-09-13 15:14:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 00:00:00.000</td>\n",
       "      <td>P1Y</td>\n",
       "      <td>10YAT-APG------L</td>\n",
       "      <td>CTA</td>\n",
       "      <td>APG CA</td>\n",
       "      <td>AT</td>\n",
       "      <td>Wind Offshore</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-09-13 15:14:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 00:00:00.000</td>\n",
       "      <td>P1Y</td>\n",
       "      <td>10YAT-APG------L</td>\n",
       "      <td>CTA</td>\n",
       "      <td>APG CA</td>\n",
       "      <td>AT</td>\n",
       "      <td>Fossil Brown coal/Lignite</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-09-13 15:14:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 00:00:00.000</td>\n",
       "      <td>P1Y</td>\n",
       "      <td>10YAT-APG------L</td>\n",
       "      <td>CTA</td>\n",
       "      <td>APG CA</td>\n",
       "      <td>AT</td>\n",
       "      <td>Nuclear</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-09-13 15:14:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 00:00:00.000</td>\n",
       "      <td>P1Y</td>\n",
       "      <td>10YAT-APG------L</td>\n",
       "      <td>CTA</td>\n",
       "      <td>APG CA</td>\n",
       "      <td>AT</td>\n",
       "      <td>Fossil Hard coal</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-09-13 15:14:19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Month  Day                 DateTime ResolutionCode          areacode  \\\n",
       "0  2015      1    1  2015-01-01 00:00:00.000            P1Y  10YAT-APG------L   \n",
       "1  2015      1    1  2015-01-01 00:00:00.000            P1Y  10YAT-APG------L   \n",
       "2  2015      1    1  2015-01-01 00:00:00.000            P1Y  10YAT-APG------L   \n",
       "3  2015      1    1  2015-01-01 00:00:00.000            P1Y  10YAT-APG------L   \n",
       "4  2015      1    1  2015-01-01 00:00:00.000            P1Y  10YAT-APG------L   \n",
       "\n",
       "  AreaTypeCode AreaName MapCode             ProductionType  \\\n",
       "0          CTA   APG CA      AT                      Other   \n",
       "1          CTA   APG CA      AT              Wind Offshore   \n",
       "2          CTA   APG CA      AT  Fossil Brown coal/Lignite   \n",
       "3          CTA   APG CA      AT                    Nuclear   \n",
       "4          CTA   APG CA      AT           Fossil Hard coal   \n",
       "\n",
       "   AggregatedInstalledCapacity  DeletedFlag           UpdateTime  \n",
       "0                         22.8            0  2017-09-13 15:14:19  \n",
       "1                          0.0            0  2017-09-13 15:14:19  \n",
       "2                          0.0            0  2017-09-13 15:14:19  \n",
       "3                          0.0            0  2017-09-13 15:14:19  \n",
       "4                       1171.0            0  2017-09-13 15:14:19  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# file pattern for the single years\n",
    "filenamepattern = '_1_InstalledGenerationCapacityAggregated.csv'\n",
    "list_of_data_tables = [] # list to append\n",
    "\n",
    "# iterate over the years from 2015 to 2020\n",
    "for i in range(2015,2021):\n",
    "    filepath = os.path.join('input',\n",
    "                            'ENTSO-E',\n",
    "                            'Transparency',\n",
    "                            'InstalledGenerationCapacityAggregated',\n",
    "                            str(i) + filenamepattern)\n",
    "    \n",
    "    list_of_data_tables.append(pd.read_csv(filepath, delimiter=\"\\t\", encoding = \"UTF-16\"))\n",
    "\n",
    "# merge the datasets of the single of files into one pandas dataframe\n",
    "data_transparency = pd.concat(list_of_data_tables, ignore_index=True)\n",
    "\n",
    "data_transparency.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Other</td>\n",
       "      <td>22.8</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Wind Offshore</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fossil Brown coal/Lignite</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fossil Hard coal</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  technology  capacity country  year\n",
       "0                      Other      22.8      AT  2015\n",
       "1              Wind Offshore       0.0      AT  2015\n",
       "2  Fossil Brown coal/Lignite       0.0      AT  2015\n",
       "3                    Nuclear       0.0      AT  2015\n",
       "4           Fossil Hard coal    1171.0      AT  2015"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename columns according to the opsd standards\n",
    "data_transparency.rename(columns={'ProductionType': 'technology',\n",
    "                                  'AggregatedInstalledCapacity': 'capacity',\n",
    "                                  'MapCode': 'country',\n",
    "                                  'Year': 'year'}, inplace=True)\n",
    "\n",
    "# drop non relevant columns\n",
    "data_transparency = data_transparency.filter(items=['technology','capacity','country','year'], axis=1)\n",
    "# drop countries that are not part of opsd\n",
    "data_transparency = data_transparency[data_transparency['country'].isin(data_opsd.country.unique())]\n",
    "\n",
    "data_transparency.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Other or unspecified energy sources</td>\n",
       "      <td>22.8</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Offshore</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Lignite</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>0.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hard coal</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            technology  capacity country  year\n",
       "0  Other or unspecified energy sources      22.8      AT  2015\n",
       "1                             Offshore       0.0      AT  2015\n",
       "2                              Lignite       0.0      AT  2015\n",
       "3                              Nuclear       0.0      AT  2015\n",
       "4                            Hard coal    1171.0      AT  2015"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# adapt energy source notation\n",
    "dict_energy_source = {'Biomass': 'Biomass and biogas',\n",
    "                      'Fossil Brown coal/Lignite': 'Lignite',\n",
    "                      'Fossil Coal-derived gas': 'Mixed fossil fuels',\n",
    "                      'Fossil Gas': 'Natural gas',\n",
    "                      'Fossil Hard coal': 'Hard coal',\n",
    "                      'Fossil Oil': 'Oil',\n",
    "                      'Fossil Oil shale': 'Oil',\n",
    "                      'Fossil Peat': 'Other fossil fuels',\n",
    "                      'Hydro Pumped Storage': 'Pumped storage',\n",
    "                      'Hydro Run-of-river and poundage': 'Run-of-river',\n",
    "                      'Hydro Water Reservoir': 'Reservoir',\n",
    "                      'Other': 'Other or unspecified energy sources',\n",
    "                      'Other renewable': 'Differently categorized renewable energy sources',\n",
    "                      'Waste': 'Other bioenergy and renewable waste',\n",
    "                      'Wind Offshore': 'Offshore',\n",
    "                      'Wind Onshore': 'Onshore',\n",
    "                      ' ': np.nan}\n",
    "\n",
    "data_transparency['technology'].replace(dict_energy_source, inplace=True)\n",
    "\n",
    "\n",
    "data_transparency.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Differently categorized renewable energy sources</th>\n",
       "      <th>Geothermal</th>\n",
       "      <th>Hard coal</th>\n",
       "      <th>Lignite</th>\n",
       "      <th>Marine</th>\n",
       "      <th>Mixed fossil fuels</th>\n",
       "      <th>Natural gas</th>\n",
       "      <th>...</th>\n",
       "      <th>Differently categorized solar</th>\n",
       "      <th>Differently categorized natural gas</th>\n",
       "      <th>Non-renewable waste</th>\n",
       "      <th>Differently categorized fossil fuels</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Wind</th>\n",
       "      <th>Bioenergy and renewable waste</th>\n",
       "      <th>Renewable energy sources</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "      <td>464.2</td>\n",
       "      <td>31.3</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4501.2</td>\n",
       "      <td>...</td>\n",
       "      <td>587.0</td>\n",
       "      <td>4501.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11478.7</td>\n",
       "      <td>2120.9</td>\n",
       "      <td>608.3</td>\n",
       "      <td>14827.1</td>\n",
       "      <td>5761.15</td>\n",
       "      <td>20611.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>473.6</td>\n",
       "      <td>32.5</td>\n",
       "      <td>0.9</td>\n",
       "      <td>766.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4465.8</td>\n",
       "      <td>...</td>\n",
       "      <td>723.0</td>\n",
       "      <td>4465.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11510.7</td>\n",
       "      <td>2497.4</td>\n",
       "      <td>617.7</td>\n",
       "      <td>15382.2</td>\n",
       "      <td>5320.75</td>\n",
       "      <td>20725.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2017</td>\n",
       "      <td>473.8</td>\n",
       "      <td>32.5</td>\n",
       "      <td>0.9</td>\n",
       "      <td>598.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4465.8</td>\n",
       "      <td>...</td>\n",
       "      <td>1031.0</td>\n",
       "      <td>4465.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11946.8</td>\n",
       "      <td>2696.0</td>\n",
       "      <td>617.9</td>\n",
       "      <td>16325.1</td>\n",
       "      <td>5152.75</td>\n",
       "      <td>21500.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2018</td>\n",
       "      <td>491.0</td>\n",
       "      <td>42.3</td>\n",
       "      <td>0.9</td>\n",
       "      <td>598.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4467.7</td>\n",
       "      <td>...</td>\n",
       "      <td>1193.0</td>\n",
       "      <td>4467.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11990.4</td>\n",
       "      <td>2887.0</td>\n",
       "      <td>640.8</td>\n",
       "      <td>16754.4</td>\n",
       "      <td>5154.65</td>\n",
       "      <td>21931.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2019</td>\n",
       "      <td>499.8</td>\n",
       "      <td>42.3</td>\n",
       "      <td>0.9</td>\n",
       "      <td>598.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4463.2</td>\n",
       "      <td>...</td>\n",
       "      <td>1192.7</td>\n",
       "      <td>4463.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11117.5</td>\n",
       "      <td>3034.6</td>\n",
       "      <td>649.8</td>\n",
       "      <td>16037.8</td>\n",
       "      <td>5150.15</td>\n",
       "      <td>21210.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "technology country  year  Biomass and biogas  \\\n",
       "0               AT  2015               464.2   \n",
       "1               AT  2016               473.6   \n",
       "2               AT  2017               473.8   \n",
       "3               AT  2018               491.0   \n",
       "4               AT  2019               499.8   \n",
       "\n",
       "technology  Differently categorized renewable energy sources  Geothermal  \\\n",
       "0                                                       31.3         0.9   \n",
       "1                                                       32.5         0.9   \n",
       "2                                                       32.5         0.9   \n",
       "3                                                       42.3         0.9   \n",
       "4                                                       42.3         0.9   \n",
       "\n",
       "technology  Hard coal  Lignite  Marine  Mixed fossil fuels  Natural gas  ...  \\\n",
       "0              1171.0      0.0     0.0                 0.0       4501.2  ...   \n",
       "1               766.0      0.0     0.0                 0.0       4465.8  ...   \n",
       "2               598.0      0.0     0.0                 0.0       4465.8  ...   \n",
       "3               598.0      0.0     0.0                 0.0       4467.7  ...   \n",
       "4               598.0      0.0     0.0                 0.0       4463.2  ...   \n",
       "\n",
       "technology  Differently categorized solar  \\\n",
       "0                                   587.0   \n",
       "1                                   723.0   \n",
       "2                                  1031.0   \n",
       "3                                  1193.0   \n",
       "4                                  1192.7   \n",
       "\n",
       "technology  Differently categorized natural gas  Non-renewable waste  \\\n",
       "0                                        4501.2                    0   \n",
       "1                                        4465.8                    0   \n",
       "2                                        4465.8                    0   \n",
       "3                                        4467.7                    0   \n",
       "4                                        4463.2                    0   \n",
       "\n",
       "technology  Differently categorized fossil fuels    Hydro    Wind  \\\n",
       "0                                              0  11478.7  2120.9   \n",
       "1                                              0  11510.7  2497.4   \n",
       "2                                              0  11946.8  2696.0   \n",
       "3                                              0  11990.4  2887.0   \n",
       "4                                              0  11117.5  3034.6   \n",
       "\n",
       "technology  Bioenergy and renewable waste  Renewable energy sources  \\\n",
       "0                                   608.3                   14827.1   \n",
       "1                                   617.7                   15382.2   \n",
       "2                                   617.9                   16325.1   \n",
       "3                                   640.8                   16754.4   \n",
       "4                                   649.8                   16037.8   \n",
       "\n",
       "technology  Fossil fuels     Total  \n",
       "0                5761.15  20611.05  \n",
       "1                5320.75  20725.75  \n",
       "2                5152.75  21500.65  \n",
       "3                5154.65  21931.85  \n",
       "4                5150.15  21210.75  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add missing categories\n",
    "transparency_pivot = data_transparency.pivot_table(values='capacity',\n",
    "                                                   index=['country','year'],\n",
    "                                                   columns='technology')\n",
    "\n",
    "# technology level\n",
    "transparency_pivot['Differently categorized solar'] = transparency_pivot['Solar']\n",
    "transparency_pivot['Differently categorized natural gas'] = transparency_pivot['Natural gas']\n",
    "transparency_pivot['Non-renewable waste'] = 0\n",
    "transparency_pivot['Differently categorized fossil fuels'] = 0\n",
    "\n",
    "# level 3\n",
    "hydro_arr = ['Pumped storage', 'Reservoir', 'Run-of-river']\n",
    "transparency_pivot['Hydro'] = transparency_pivot[hydro_arr].sum(axis=1)\n",
    "\n",
    "wind_arr = ['Onshore', 'Offshore']\n",
    "transparency_pivot['Wind'] = transparency_pivot[wind_arr].sum(axis=1)\n",
    "\n",
    "# level 2\n",
    "bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n",
    "transparency_pivot['Bioenergy and renewable waste'] = transparency_pivot[bio_arr].sum(axis=1)\n",
    "\n",
    "#level 1\n",
    "res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine',\n",
    "           'Bioenergy and renewable waste', 'Differently categorized renewable energy sources']\n",
    "transparency_pivot['Renewable energy sources'] = transparency_pivot[res_arr].sum(axis=1)\n",
    "\n",
    "fossil_arr = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Mixed fossil fuels', 'Other fossil fuels']\n",
    "transparency_pivot['Fossil fuels'] = transparency_pivot[fossil_arr].sum(axis=1)\n",
    "\n",
    "# level 0\n",
    "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources', 'Other or unspecified energy sources']\n",
    "transparency_pivot['Total'] = transparency_pivot[total_arr].sum(axis=1)\n",
    "\n",
    "transparency_pivot.reset_index(inplace=True)\n",
    "\n",
    "transparency_pivot.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>type</th>\n",
       "      <th>weblink</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>464.2</td>\n",
       "      <td>ENTSO-E Transparency Platform</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>473.6</td>\n",
       "      <td>ENTSO-E Transparency Platform</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2017</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>473.8</td>\n",
       "      <td>ENTSO-E Transparency Platform</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2018</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>491.0</td>\n",
       "      <td>ENTSO-E Transparency Platform</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2019</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>499.8</td>\n",
       "      <td>ENTSO-E Transparency Platform</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year          technology  capacity                         source  \\\n",
       "0      AT  2015  Biomass and biogas     464.2  ENTSO-E Transparency Platform   \n",
       "1      AT  2016  Biomass and biogas     473.6  ENTSO-E Transparency Platform   \n",
       "2      AT  2017  Biomass and biogas     473.8  ENTSO-E Transparency Platform   \n",
       "3      AT  2018  Biomass and biogas     491.0  ENTSO-E Transparency Platform   \n",
       "4      AT  2019  Biomass and biogas     499.8  ENTSO-E Transparency Platform   \n",
       "\n",
       "         source_type capacity_definition                      type  \\\n",
       "0  Other association        Net capacity  Installed capacity in MW   \n",
       "1  Other association        Net capacity  Installed capacity in MW   \n",
       "2  Other association        Net capacity  Installed capacity in MW   \n",
       "3  Other association        Net capacity  Installed capacity in MW   \n",
       "4  Other association        Net capacity  Installed capacity in MW   \n",
       "\n",
       "                                             weblink  \n",
       "0  https://transparency.entsoe.eu/generation/r2/i...  \n",
       "1  https://transparency.entsoe.eu/generation/r2/i...  \n",
       "2  https://transparency.entsoe.eu/generation/r2/i...  \n",
       "3  https://transparency.entsoe.eu/generation/r2/i...  \n",
       "4  https://transparency.entsoe.eu/generation/r2/i...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_transparency = pd.melt(transparency_pivot,\n",
    "                      id_vars=['country', 'year'],\n",
    "                      var_name='technology',\n",
    "                      value_name='capacity')\n",
    "\n",
    "data_transparency = data_transparency.loc[data_transparency[\"year\"] < 2020, :]\n",
    "\n",
    "data_transparency['source'] = 'ENTSO-E Transparency Platform'\n",
    "data_transparency['source_type'] = 'Other association'\n",
    "data_transparency['capacity_definition'] = 'Net capacity'\n",
    "data_transparency['type'] = 'Installed capacity in MW'\n",
    "data_transparency['weblink'] = ('https://transparency.entsoe.eu/generation'\n",
    "                                '/r2/installedGenerationCapacityAggregation/show')\n",
    "\n",
    "data_transparency.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.4 ENTSO-E Power Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>capacity</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Non-Renewable</td>\n",
       "      <td>AL</td>\n",
       "      <td>0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>AL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Non-renwable hydro</td>\n",
       "      <td>AL</td>\n",
       "      <td>0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Of which hydro pure pumped storage</td>\n",
       "      <td>AL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Of which Hydro mixed pumped storage (non renew...</td>\n",
       "      <td>AL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          technology country capacity  year\n",
       "0                                      Non-Renewable      AL        0  2014\n",
       "1                                            Nuclear      AL      NaN  2014\n",
       "2                                 Non-renwable hydro      AL        0  2014\n",
       "3                 Of which hydro pure pumped storage      AL      NaN  2014\n",
       "4  Of which Hydro mixed pumped storage (non renew...      AL      NaN  2014"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_of_year = {2014: 9,\n",
    "               2015: 53,\n",
    "               2016: 97,\n",
    "               2017: 141,\n",
    "               2018: 185}\n",
    "\n",
    "dataframes = []\n",
    "\n",
    "for year, row in row_of_year.items():\n",
    "    # read the dataframe for each year\n",
    "    power_statistics_raw = pd.read_excel(os.path.join('input',\n",
    "                                                      'ENTSO-E',\n",
    "                                                      'Power Statistics',\n",
    "                                                      'NGC.xlsx'),\n",
    "                                         header=[0,1], \n",
    "                                         sheet_name='NGC',\n",
    "                                         skiprows=row,\n",
    "                                         nrows=42)\n",
    "    \n",
    "    # drop non relevant columns\n",
    "    power_statistics_raw.drop(columns='Coverage ratio in %', level=1, inplace=True)\n",
    "    power_statistics_raw.drop(columns=['Unnamed: 1_level_1','Unnamed: 2_level_1','Unnamed: 3_level_1','Unnamed: 4_level_1'], level=1, inplace=True)\n",
    "\n",
    "    # get rid of multi index\n",
    "    df = power_statistics_raw.set_index(year).stack().reset_index().drop('level_1', axis=1)\n",
    "    \n",
    "    # remove leftovers of multi index in the index column\n",
    "    df[\"technology\"] = df[year].apply(lambda x: x[0])\n",
    "    df.drop(columns=year, inplace=True)\n",
    "    \n",
    "    # stack df to the opsd standard format\n",
    "    stacked_df = df.melt(id_vars='technology', var_name='country', value_name='capacity')\n",
    "    \n",
    "    # add information about the year\n",
    "    stacked_df['year'] = year\n",
    "    \n",
    "    # append to the main list of dataframes\n",
    "    dataframes.append(stacked_df)\n",
    "    \n",
    "power_statistics = pd.concat(dataframes)\n",
    "power_statistics.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>capacity</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Of which hydro pure pumped storage</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Of which Hydro mixed pumped storage (non renew...</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Of which Fossil Brown coal/Lignite</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Of which Fossil Coal-derived gas</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           technology country  capacity  year\n",
       "41                                            Nuclear      AT       0.0  2014\n",
       "43                 Of which hydro pure pumped storage      AT       0.0  2014\n",
       "44  Of which Hydro mixed pumped storage (non renew...      AT       0.0  2014\n",
       "46                 Of which Fossil Brown coal/Lignite      AT       0.0  2014\n",
       "47                   Of which Fossil Coal-derived gas      AT       0.0  2014"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop countries that are not covered in opsd\n",
    "opsd_countries = data_opsd.country.unique()\n",
    "drop_list_country = power_statistics.loc[~power_statistics['country'].isin(opsd_countries)].index.to_list()\n",
    "power_statistics.drop(drop_list_country, inplace=True)\n",
    "\n",
    "# technology classes to be dropped\n",
    "tech_to_drop = ['Non-Renewable', 'Fossil fuels', 'Renewable','Non-renwable hydro',\n",
    "                'Total Waste', 'Bio', 'Renewable Hydro', 'Comments', 'Total NGC']\n",
    "\n",
    "drop_list_tech = power_statistics.loc[power_statistics['technology'].isin(tech_to_drop)].index.to_list()\n",
    "power_statistics.drop(drop_list_tech, inplace=True)\n",
    "\n",
    "# replace string with values that can be used in math operations \n",
    "power_statistics['capacity'].replace(to_replace='Not Expected', value=0, inplace=True)\n",
    "power_statistics['capacity'].replace(to_replace='Not Available', value=np.nan, inplace=True)\n",
    "\n",
    "power_statistics.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>capacity</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Nuclear</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Pumped storage</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Pumped storage</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Lignite</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Differently categorized fossil fuels</td>\n",
       "      <td>AT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              technology country  capacity  year\n",
       "41                               Nuclear      AT       0.0  2014\n",
       "43                        Pumped storage      AT       0.0  2014\n",
       "44                        Pumped storage      AT       0.0  2014\n",
       "46                               Lignite      AT       0.0  2014\n",
       "47  Differently categorized fossil fuels      AT       0.0  2014"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Not included because already categorized in OPSD standard:\n",
    "# Nuclear, Solar, Geothermal, Wind\n",
    "dict_energy_source = {'Of which hydro pure pumped storage':'Pumped storage',\n",
    "                      'Of which Hydro mixed pumped storage (non renewable part)':'Pumped storage',\n",
    "                      'Of which Fossil Brown coal/Lignite':'Lignite',\n",
    "                      'Of which Fossil Coal-derived gas':'Differently categorized fossil fuels',\n",
    "                      'Of which Fossil Gas':'Natural gas',\n",
    "                      'Of which Fossil Hard coal':'Hard coal', \n",
    "                      'Of which Fossil Oil':'Oil',\n",
    "                      'Of which Fossil Oil shale':'Oil', \n",
    "                      'Of which Fossil Peat':'Differently categorized fossil fuels',\n",
    "                      'Of which Mixed fuels':'Mixed fossil fuels', \n",
    "                      'Of which Other fossil fuels':'Other fossil fuels',\n",
    "                      'Non-renewable Waste':'Non-renewable waste', \n",
    "                      'Other non-renewable':'Differently categorized fossil fuels',                      \n",
    "                      'Of which Wind offshore':'Offshore', \n",
    "                      'Of which Wind onshore':'Onshore', \n",
    "                      'Of which Solar PV':'Photovoltaics', \n",
    "                      'Of which Solar Thermal':'Differently categorized solar', \n",
    "                      'Of which Biomass':'Biomass and biogas', \n",
    "                      'Of which Biogas':'Biomass and biogas', \n",
    "                      'Renewable Waste':'Other bioenergy and renewable waste', \n",
    "                      'Of which Hydro Pure storage':'Reservoir',\n",
    "                      'Of which Hydro Run-of-river and pondage':'Run-of-river',\n",
    "                      'Of which Hydro mixed pumped storage (renewable part)':'Pumped storage',\n",
    "                      'Of which Hydro Marine (tidal/wave)':'Marine',\n",
    "                      'Other renewable (not listed)':'Differently categorized renewable energy sources',\n",
    "                      'Non identified (other not listed)':'Other or unspecified energy sources', \n",
    "                      'Total Hydro':'Hydro'}\n",
    "\n",
    "power_statistics[\"technology\"].replace(dict_energy_source, inplace=True)\n",
    "power_statistics.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Differently categorized fossil fuels</th>\n",
       "      <th>Differently categorized renewable energy sources</th>\n",
       "      <th>Differently categorized solar</th>\n",
       "      <th>Geothermal</th>\n",
       "      <th>Hard coal</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Lignite</th>\n",
       "      <th>...</th>\n",
       "      <th>Onshore</th>\n",
       "      <th>Other bioenergy and renewable waste</th>\n",
       "      <th>Other fossil fuels</th>\n",
       "      <th>Other or unspecified energy sources</th>\n",
       "      <th>Photovoltaics</th>\n",
       "      <th>Pumped storage</th>\n",
       "      <th>Reservoir</th>\n",
       "      <th>Run-of-river</th>\n",
       "      <th>Solar</th>\n",
       "      <th>Wind</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>302.685</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>13568.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>21.96</td>\n",
       "      <td>414.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>586.12</td>\n",
       "      <td>2656.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5599.0</td>\n",
       "      <td>586.12</td>\n",
       "      <td>2110.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "      <td>298.000</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>13656.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2489.0</td>\n",
       "      <td>23.00</td>\n",
       "      <td>414.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>732.00</td>\n",
       "      <td>3997.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5662.0</td>\n",
       "      <td>732.00</td>\n",
       "      <td>2489.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>286.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14116.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>22.00</td>\n",
       "      <td>414.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>4212.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5692.0</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>2730.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2017</td>\n",
       "      <td>286.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14116.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>22.00</td>\n",
       "      <td>414.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>4212.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5692.0</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>2730.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2018</td>\n",
       "      <td>293.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2887.0</td>\n",
       "      <td>22.00</td>\n",
       "      <td>414.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1193.00</td>\n",
       "      <td>4218.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5714.0</td>\n",
       "      <td>1193.00</td>\n",
       "      <td>2887.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "technology country  year  Biomass and biogas  \\\n",
       "0               AT  2014             302.685   \n",
       "1               AT  2015             298.000   \n",
       "2               AT  2016             286.000   \n",
       "3               AT  2017             286.000   \n",
       "4               AT  2018             293.000   \n",
       "\n",
       "technology  Differently categorized fossil fuels  \\\n",
       "0                                           29.0   \n",
       "1                                           30.0   \n",
       "2                                           35.0   \n",
       "3                                           35.0   \n",
       "4                                           35.0   \n",
       "\n",
       "technology  Differently categorized renewable energy sources  \\\n",
       "0                                                        0.0   \n",
       "1                                                        NaN   \n",
       "2                                                        NaN   \n",
       "3                                                        NaN   \n",
       "4                                                        NaN   \n",
       "\n",
       "technology  Differently categorized solar  Geothermal  Hard coal    Hydro  \\\n",
       "0                                     0.0         1.0     1171.0  13568.0   \n",
       "1                                     0.0         1.0     1171.0  13656.0   \n",
       "2                                     0.0         1.0      598.0  14116.0   \n",
       "3                                     0.0         1.0      598.0  14116.0   \n",
       "4                                     0.0         1.0      598.0  14150.0   \n",
       "\n",
       "technology  Lignite  ...  Onshore  Other bioenergy and renewable waste  \\\n",
       "0               0.0  ...   2110.0                                21.96   \n",
       "1               0.0  ...   2489.0                                23.00   \n",
       "2               0.0  ...   2730.0                                22.00   \n",
       "3               0.0  ...   2730.0                                22.00   \n",
       "4               0.0  ...   2887.0                                22.00   \n",
       "\n",
       "technology  Other fossil fuels  Other or unspecified energy sources  \\\n",
       "0                        414.0                                  0.0   \n",
       "1                        414.0                                  NaN   \n",
       "2                        414.0                                  NaN   \n",
       "3                        414.0                                  NaN   \n",
       "4                        414.0                                  NaN   \n",
       "\n",
       "technology  Photovoltaics  Pumped storage  Reservoir  Run-of-river    Solar  \\\n",
       "0                  586.12     2656.333333        0.0        5599.0   586.12   \n",
       "1                  732.00     3997.000000        NaN        5662.0   732.00   \n",
       "2                 1031.00     4212.000000        NaN        5692.0  1031.00   \n",
       "3                 1031.00     4212.000000        NaN        5692.0  1031.00   \n",
       "4                 1193.00     4218.000000        NaN        5714.0  1193.00   \n",
       "\n",
       "technology    Wind  \n",
       "0           2110.0  \n",
       "1           2489.0  \n",
       "2           2730.0  \n",
       "3           2730.0  \n",
       "4           2887.0  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "powerstats_pivot = power_statistics.pivot_table(values='capacity',\n",
    "                                                   index=['country','year'],\n",
    "                                                   columns='technology').reset_index()\n",
    "powerstats_pivot.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>technology</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>Biomass and biogas</th>\n",
       "      <th>Differently categorized fossil fuels</th>\n",
       "      <th>Differently categorized renewable energy sources</th>\n",
       "      <th>Differently categorized solar</th>\n",
       "      <th>Geothermal</th>\n",
       "      <th>Hard coal</th>\n",
       "      <th>Hydro</th>\n",
       "      <th>Lignite</th>\n",
       "      <th>...</th>\n",
       "      <th>Pumped storage</th>\n",
       "      <th>Reservoir</th>\n",
       "      <th>Run-of-river</th>\n",
       "      <th>Solar</th>\n",
       "      <th>Wind</th>\n",
       "      <th>Differently categorized natural gas</th>\n",
       "      <th>Bioenergy and renewable waste</th>\n",
       "      <th>Fossil fuels</th>\n",
       "      <th>Renewable energy sources</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>302.685</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>13568.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2656.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5599.0</td>\n",
       "      <td>586.12</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>4888.0</td>\n",
       "      <td>324.645</td>\n",
       "      <td>7105.0</td>\n",
       "      <td>16589.765</td>\n",
       "      <td>23694.765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "      <td>298.000</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1171.0</td>\n",
       "      <td>13656.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>3997.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5662.0</td>\n",
       "      <td>732.00</td>\n",
       "      <td>2489.0</td>\n",
       "      <td>4820.0</td>\n",
       "      <td>321.000</td>\n",
       "      <td>7002.0</td>\n",
       "      <td>17199.000</td>\n",
       "      <td>24201.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>286.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14116.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4212.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5692.0</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>4841.0</td>\n",
       "      <td>308.000</td>\n",
       "      <td>6407.0</td>\n",
       "      <td>18186.000</td>\n",
       "      <td>24593.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2017</td>\n",
       "      <td>286.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14116.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4212.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5692.0</td>\n",
       "      <td>1031.00</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>4841.0</td>\n",
       "      <td>308.000</td>\n",
       "      <td>6407.0</td>\n",
       "      <td>18186.000</td>\n",
       "      <td>24593.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2018</td>\n",
       "      <td>293.000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>598.0</td>\n",
       "      <td>14150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4218.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5714.0</td>\n",
       "      <td>1193.00</td>\n",
       "      <td>2887.0</td>\n",
       "      <td>4853.0</td>\n",
       "      <td>315.000</td>\n",
       "      <td>6420.0</td>\n",
       "      <td>18546.000</td>\n",
       "      <td>24966.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "technology country  year  Biomass and biogas  \\\n",
       "0               AT  2014             302.685   \n",
       "1               AT  2015             298.000   \n",
       "2               AT  2016             286.000   \n",
       "3               AT  2017             286.000   \n",
       "4               AT  2018             293.000   \n",
       "\n",
       "technology  Differently categorized fossil fuels  \\\n",
       "0                                           29.0   \n",
       "1                                           30.0   \n",
       "2                                           35.0   \n",
       "3                                           35.0   \n",
       "4                                           35.0   \n",
       "\n",
       "technology  Differently categorized renewable energy sources  \\\n",
       "0                                                        0.0   \n",
       "1                                                        NaN   \n",
       "2                                                        NaN   \n",
       "3                                                        NaN   \n",
       "4                                                        NaN   \n",
       "\n",
       "technology  Differently categorized solar  Geothermal  Hard coal    Hydro  \\\n",
       "0                                     0.0         1.0     1171.0  13568.0   \n",
       "1                                     0.0         1.0     1171.0  13656.0   \n",
       "2                                     0.0         1.0      598.0  14116.0   \n",
       "3                                     0.0         1.0      598.0  14116.0   \n",
       "4                                     0.0         1.0      598.0  14150.0   \n",
       "\n",
       "technology  Lignite  ...  Pumped storage  Reservoir  Run-of-river    Solar  \\\n",
       "0               0.0  ...     2656.333333        0.0        5599.0   586.12   \n",
       "1               0.0  ...     3997.000000        NaN        5662.0   732.00   \n",
       "2               0.0  ...     4212.000000        NaN        5692.0  1031.00   \n",
       "3               0.0  ...     4212.000000        NaN        5692.0  1031.00   \n",
       "4               0.0  ...     4218.000000        NaN        5714.0  1193.00   \n",
       "\n",
       "technology    Wind  Differently categorized natural gas  \\\n",
       "0           2110.0                               4888.0   \n",
       "1           2489.0                               4820.0   \n",
       "2           2730.0                               4841.0   \n",
       "3           2730.0                               4841.0   \n",
       "4           2887.0                               4853.0   \n",
       "\n",
       "technology  Bioenergy and renewable waste  Fossil fuels  \\\n",
       "0                                 324.645        7105.0   \n",
       "1                                 321.000        7002.0   \n",
       "2                                 308.000        6407.0   \n",
       "3                                 308.000        6407.0   \n",
       "4                                 315.000        6420.0   \n",
       "\n",
       "technology  Renewable energy sources      Total  \n",
       "0                          16589.765  23694.765  \n",
       "1                          17199.000  24201.000  \n",
       "2                          18186.000  24593.000  \n",
       "3                          18186.000  24593.000  \n",
       "4                          18546.000  24966.000  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# technology level\n",
    "powerstats_pivot['Differently categorized natural gas'] = powerstats_pivot['Natural gas']\n",
    "\n",
    "# level 2\n",
    "powerstats_pivot['Bioenergy and renewable waste'] = (\n",
    "          powerstats_pivot['Biomass and biogas'] +\n",
    "          powerstats_pivot['Other bioenergy and renewable waste'])\n",
    "\n",
    "#level 1\n",
    "fossil_techs = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Non-renewable waste',\n",
    "               'Mixed fossil fuels', 'Other fossil fuels', 'Differently categorized fossil fuels']\n",
    "\n",
    "powerstats_pivot['Fossil fuels'] = powerstats_pivot[fossil_techs].sum(axis=1)\n",
    "\n",
    "res_tech = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste',\n",
    "           'Differently categorized renewable energy sources']\n",
    "\n",
    "powerstats_pivot['Renewable energy sources'] = powerstats_pivot[res_tech].sum(axis=1)\n",
    "\n",
    "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n",
    "powerstats_pivot['Total'] = powerstats_pivot[total_arr].sum(axis=1)\n",
    "\n",
    "powerstats_pivot.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>technology</th>\n",
       "      <th>capacity</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>type</th>\n",
       "      <th>weblink</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AT</td>\n",
       "      <td>2014</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>302.685</td>\n",
       "      <td>ENTSO-E Power Statistics</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AT</td>\n",
       "      <td>2015</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>298.000</td>\n",
       "      <td>ENTSO-E Power Statistics</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AT</td>\n",
       "      <td>2016</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>286.000</td>\n",
       "      <td>ENTSO-E Power Statistics</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AT</td>\n",
       "      <td>2017</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>286.000</td>\n",
       "      <td>ENTSO-E Power Statistics</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AT</td>\n",
       "      <td>2018</td>\n",
       "      <td>Biomass and biogas</td>\n",
       "      <td>293.000</td>\n",
       "      <td>ENTSO-E Power Statistics</td>\n",
       "      <td>Other association</td>\n",
       "      <td>Net capacity</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country  year          technology  capacity                    source  \\\n",
       "0      AT  2014  Biomass and biogas   302.685  ENTSO-E Power Statistics   \n",
       "1      AT  2015  Biomass and biogas   298.000  ENTSO-E Power Statistics   \n",
       "2      AT  2016  Biomass and biogas   286.000  ENTSO-E Power Statistics   \n",
       "3      AT  2017  Biomass and biogas   286.000  ENTSO-E Power Statistics   \n",
       "4      AT  2018  Biomass and biogas   293.000  ENTSO-E Power Statistics   \n",
       "\n",
       "         source_type capacity_definition                      type  \\\n",
       "0  Other association        Net capacity  Installed capacity in MW   \n",
       "1  Other association        Net capacity  Installed capacity in MW   \n",
       "2  Other association        Net capacity  Installed capacity in MW   \n",
       "3  Other association        Net capacity  Installed capacity in MW   \n",
       "4  Other association        Net capacity  Installed capacity in MW   \n",
       "\n",
       "                                             weblink  \n",
       "0  https://www.entsoe.eu/data/power-stats/net-gen...  \n",
       "1  https://www.entsoe.eu/data/power-stats/net-gen...  \n",
       "2  https://www.entsoe.eu/data/power-stats/net-gen...  \n",
       "3  https://www.entsoe.eu/data/power-stats/net-gen...  \n",
       "4  https://www.entsoe.eu/data/power-stats/net-gen...  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_power_statistics = powerstats_pivot.melt(id_vars=['country', 'year'],\n",
    "                                         var_name='technology',\n",
    "                                         value_name='capacity')\n",
    "\n",
    "data_power_statistics['source'] = 'ENTSO-E Power Statistics'\n",
    "data_power_statistics['source_type'] = 'Other association'\n",
    "data_power_statistics['capacity_definition'] = 'Net capacity'\n",
    "data_power_statistics['type'] = 'Installed capacity in MW'\n",
    "data_power_statistics['weblink'] = ('https://www.entsoe.eu/data/'\n",
    "                                    'power-stats/net-gen-capacity/')\n",
    "\n",
    "data_power_statistics.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Merge data sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>technology</th>\n",
       "      <th>source</th>\n",
       "      <th>source_type</th>\n",
       "      <th>weblink</th>\n",
       "      <th>year</th>\n",
       "      <th>type</th>\n",
       "      <th>country</th>\n",
       "      <th>capacity_definition</th>\n",
       "      <th>capacity</th>\n",
       "      <th>comment</th>\n",
       "      <th>energy_source_level_0</th>\n",
       "      <th>energy_source_level_1</th>\n",
       "      <th>energy_source_level_2</th>\n",
       "      <th>energy_source_level_3</th>\n",
       "      <th>technology_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>2014</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>AT</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7243.634</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>2015</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>AT</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7059.065</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>2016</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>AT</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>7323.000</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>2017</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>AT</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>6469.000</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fossil fuels</td>\n",
       "      <td>e-control</td>\n",
       "      <td>Regulatory Authority</td>\n",
       "      <td>https://www.e-control.at/documents/1785851/181...</td>\n",
       "      <td>2018</td>\n",
       "      <td>Installed capacity in MW</td>\n",
       "      <td>AT</td>\n",
       "      <td>Gross capacity</td>\n",
       "      <td>6492.000</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     technology     source           source_type  \\\n",
       "0  Fossil fuels  e-control  Regulatory Authority   \n",
       "1  Fossil fuels  e-control  Regulatory Authority   \n",
       "2  Fossil fuels  e-control  Regulatory Authority   \n",
       "3  Fossil fuels  e-control  Regulatory Authority   \n",
       "4  Fossil fuels  e-control  Regulatory Authority   \n",
       "\n",
       "                                             weblink  year  \\\n",
       "0  https://www.e-control.at/documents/1785851/181...  2014   \n",
       "1  https://www.e-control.at/documents/1785851/181...  2015   \n",
       "2  https://www.e-control.at/documents/1785851/181...  2016   \n",
       "3  https://www.e-control.at/documents/1785851/181...  2017   \n",
       "4  https://www.e-control.at/documents/1785851/181...  2018   \n",
       "\n",
       "                       type country capacity_definition  capacity comment  \\\n",
       "0  Installed capacity in MW      AT      Gross capacity  7243.634           \n",
       "1  Installed capacity in MW      AT      Gross capacity  7059.065           \n",
       "2  Installed capacity in MW      AT      Gross capacity  7323.000           \n",
       "3  Installed capacity in MW      AT      Gross capacity  6469.000           \n",
       "4  Installed capacity in MW      AT      Gross capacity  6492.000           \n",
       "\n",
       "   energy_source_level_0  energy_source_level_1  energy_source_level_2  \\\n",
       "0                  False                   True                  False   \n",
       "1                  False                   True                  False   \n",
       "2                  False                   True                  False   \n",
       "3                  False                   True                  False   \n",
       "4                  False                   True                  False   \n",
       "\n",
       "   energy_source_level_3  technology_level  \n",
       "0                  False             False  \n",
       "1                  False             False  \n",
       "2                  False             False  \n",
       "3                  False             False  \n",
       "4                  False             False  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframes = [data_opsd, data_eurostat, data_soaf, data_entsoe, data_transparency, data_power_statistics]\n",
    "data = pd.concat(dataframes, sort=False)\n",
    "data['comment'] = data['comment'].fillna('').astype(str)\n",
    "\n",
    "col_order = ['technology', 'source', 'source_type', 'weblink', 'year', 'type',\n",
    "             'country', 'capacity_definition', 'capacity', 'comment']\n",
    "\n",
    "data = data[col_order]\n",
    "\n",
    "\n",
    "energy_source_mapping = pd.read_csv(os.path.join('input','energy_source_mapping.csv'),\n",
    "                                    index_col ='name')\n",
    "energy_source_mapping.replace({0: False, 1: True}, inplace=True)\n",
    "\n",
    "data = data.merge(energy_source_mapping,\n",
    "                  left_on='technology',\n",
    "                  right_index=True,\n",
    "                  how='left')\n",
    "\n",
    "new_level_names = {\"Level 0\": \"energy_source_level_0\",\n",
    "                   \"Level 1\": \"energy_source_level_1\",\n",
    "                   \"Level 2\": \"energy_source_level_2\",\n",
    "                   \"Level 3\": \"energy_source_level_3\",\n",
    "                   \"Technology level\": \"technology_level\"}\n",
    "\n",
    "data.rename(columns=new_level_names, inplace=True)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Convert stacked data to crosstable format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>Country (ISO code)</th>\n",
       "      <th colspan=\"10\" halign=\"left\">AL</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"10\" halign=\"left\">XK</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type of data</th>\n",
       "      <th colspan=\"10\" halign=\"left\">Installed capacity in MW</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"10\" halign=\"left\">Installed capacity in MW</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th>1990</th>\n",
       "      <th>1991</th>\n",
       "      <th>1992</th>\n",
       "      <th>1993</th>\n",
       "      <th>1994</th>\n",
       "      <th>1995</th>\n",
       "      <th>1996</th>\n",
       "      <th>1997</th>\n",
       "      <th>1998</th>\n",
       "      <th>1999</th>\n",
       "      <th>...</th>\n",
       "      <th>2007</th>\n",
       "      <th>2008</th>\n",
       "      <th>2009</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Source</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>...</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "      <th>EUROSTAT</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type of source</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>...</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "      <th>Statistical Office</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Weblink</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>...</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "      <th>Link unavailable</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Capacity definition (net, gross, unknown)</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>...</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Unknown</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Fossil fuels</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>915.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lignite</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hard coal</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Oil</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Natural gas</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1633 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Country (ISO code)                                              AL  \\\n",
       "Type of data                              Installed capacity in MW   \n",
       "Year                                                          1990   \n",
       "Source                                                    EUROSTAT   \n",
       "Type of source                                  Statistical Office   \n",
       "Weblink                                           Link unavailable   \n",
       "Capacity definition (net, gross, unknown)                  Unknown   \n",
       "Fossil fuels                                                   0.0   \n",
       "Lignite                                                        NaN   \n",
       "Hard coal                                                      NaN   \n",
       "Oil                                                            NaN   \n",
       "Natural gas                                                    NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1991   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1992   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1993   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1994   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1995   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1996   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1997   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    1998   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                             0.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            ...  \\\n",
       "Type of data                                                  ...   \n",
       "Year                                                    1999  ...   \n",
       "Source                                              EUROSTAT  ...   \n",
       "Type of source                            Statistical Office  ...   \n",
       "Weblink                                     Link unavailable  ...   \n",
       "Capacity definition (net, gross, unknown)            Unknown  ...   \n",
       "Fossil fuels                                             0.0  ...   \n",
       "Lignite                                                  NaN  ...   \n",
       "Hard coal                                                NaN  ...   \n",
       "Oil                                                      NaN  ...   \n",
       "Natural gas                                              NaN  ...   \n",
       "\n",
       "Country (ISO code)                                              XK  \\\n",
       "Type of data                              Installed capacity in MW   \n",
       "Year                                                          2007   \n",
       "Source                                                    EUROSTAT   \n",
       "Type of source                                  Statistical Office   \n",
       "Weblink                                           Link unavailable   \n",
       "Capacity definition (net, gross, unknown)                  Unknown   \n",
       "Fossil fuels                                                 915.0   \n",
       "Lignite                                                        NaN   \n",
       "Hard coal                                                      NaN   \n",
       "Oil                                                            NaN   \n",
       "Natural gas                                                    NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2008   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2009   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2010   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2011   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2012   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2013   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2014   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \\\n",
       "Type of data                                                   \n",
       "Year                                                    2015   \n",
       "Source                                              EUROSTAT   \n",
       "Type of source                            Statistical Office   \n",
       "Weblink                                     Link unavailable   \n",
       "Capacity definition (net, gross, unknown)            Unknown   \n",
       "Fossil fuels                                           915.0   \n",
       "Lignite                                                  NaN   \n",
       "Hard coal                                                NaN   \n",
       "Oil                                                      NaN   \n",
       "Natural gas                                              NaN   \n",
       "\n",
       "Country (ISO code)                                            \n",
       "Type of data                                                  \n",
       "Year                                                    2016  \n",
       "Source                                              EUROSTAT  \n",
       "Type of source                            Statistical Office  \n",
       "Weblink                                     Link unavailable  \n",
       "Capacity definition (net, gross, unknown)            Unknown  \n",
       "Fossil fuels                                           915.0  \n",
       "Lignite                                                  NaN  \n",
       "Hard coal                                                NaN  \n",
       "Oil                                                      NaN  \n",
       "Natural gas                                              NaN  \n",
       "\n",
       "[5 rows x 1633 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['technology', 'source', 'source_type', 'weblink','year',\n",
    "        'type', 'country', 'capacity_definition', 'capacity']\n",
    "\n",
    "data_crosstable = pd.pivot_table(data[cols],\n",
    "                                 index=['technology'],\n",
    "                                 columns=['country', 'type', 'year',\n",
    "                                          'source', 'source_type',\n",
    "                                          'weblink', 'capacity_definition'],\n",
    "                                 values='capacity')\n",
    "\n",
    "# Apply initial ordering of technologies\n",
    "data_crosstable = data_crosstable.reindex(technology_order)\n",
    "\n",
    "# Delete index naming\n",
    "data_crosstable.index.name = None\n",
    "data_crosstable.columns.names = ('Country (ISO code)',\n",
    "                                 'Type of data', 'Year',\n",
    "                                 'Source', 'Type of source',\n",
    "                                 'Weblink',\n",
    "                                 'Capacity definition (net, gross, unknown)')\n",
    "\n",
    "data_crosstable.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>Country (ISO code)</th>\n",
       "      <th colspan=\"5\" halign=\"left\">Energy source levels and technology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Description</th>\n",
       "      <th>Total generation capacity</th>\n",
       "      <th>Generation capacity by energy type (fossil, nuclear, renewable, other)</th>\n",
       "      <th>Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind)</th>\n",
       "      <th>Generation capacity by fuel or energy source and detailed fuel for bioenergy</th>\n",
       "      <th>Generation capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Level</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "      <th>NaN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NaN</th>\n",
       "      <td>Level 0</td>\n",
       "      <td>Level 1</td>\n",
       "      <td>Level 2</td>\n",
       "      <td>Level 3</td>\n",
       "      <td>Technology level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fossil fuels</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lignite</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hard coal</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Oil</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Country (ISO code) Energy source levels and technology  \\\n",
       "Description                  Total generation capacity   \n",
       "                                                   NaN   \n",
       "                                                   NaN   \n",
       "                                                   NaN   \n",
       "Level                                              NaN   \n",
       "NaN                                            Level 0   \n",
       "Fossil fuels                                         0   \n",
       "Lignite                                              0   \n",
       "Hard coal                                            0   \n",
       "Oil                                                  0   \n",
       "\n",
       "Country (ISO code)                                                                         \\\n",
       "Description        Generation capacity by energy type (fossil, nuclear, renewable, other)   \n",
       "                                                                                      NaN   \n",
       "                                                                                      NaN   \n",
       "                                                                                      NaN   \n",
       "Level                                                                                 NaN   \n",
       "NaN                                                           Level 1                       \n",
       "Fossil fuels                                                        1                       \n",
       "Lignite                                                             0                       \n",
       "Hard coal                                                           0                       \n",
       "Oil                                                                 0                       \n",
       "\n",
       "Country (ISO code)                                                                                                  \\\n",
       "Description        Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind)   \n",
       "                                                                                                               NaN   \n",
       "                                                                                                               NaN   \n",
       "                                                                                                               NaN   \n",
       "Level                                                                                                          NaN   \n",
       "NaN                                                           Level 2                                                \n",
       "Fossil fuels                                                        0                                                \n",
       "Lignite                                                             1                                                \n",
       "Hard coal                                                           1                                                \n",
       "Oil                                                                 1                                                \n",
       "\n",
       "Country (ISO code)                                                                               \\\n",
       "Description        Generation capacity by fuel or energy source and detailed fuel for bioenergy   \n",
       "                                                                                            NaN   \n",
       "                                                                                            NaN   \n",
       "                                                                                            NaN   \n",
       "Level                                                                                       NaN   \n",
       "NaN                                                           Level 3                             \n",
       "Fossil fuels                                                        0                             \n",
       "Lignite                                                             1                             \n",
       "Hard coal                                                           1                             \n",
       "Oil                                                                 1                             \n",
       "\n",
       "Country (ISO code)                                                                                                                         \n",
       "Description        Generation capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind)  \n",
       "                                                                                                                                      NaN  \n",
       "                                                                                                                                      NaN  \n",
       "                                                                                                                                      NaN  \n",
       "Level                                                                                                                                 NaN  \n",
       "NaN                                                  Technology level                                                                      \n",
       "Fossil fuels                                                        0                                                                      \n",
       "Lignite                                                             1                                                                      \n",
       "Hard coal                                                           1                                                                      \n",
       "Oil                                                                 1                                                                      "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "energylevels_table = energylevels_raw[6:]\n",
    "energylevels_table.columns = pd.MultiIndex.from_arrays(energylevels_raw[:6].values,\n",
    "                                                       names=['country', 'type', 'year',\n",
    "                                                              'source', 'source_type',\n",
    "                                                              'capacity_definition'\n",
    "                                                              ])\n",
    "\n",
    "energylevels_table = energylevels_table.reset_index()\n",
    "energylevels_table['technology'] = energylevels_table['technology'].str.replace('- ', '')\n",
    "energylevels_table = energylevels_table.set_index('technology')\n",
    "\n",
    "# Delete index naming\n",
    "energylevels_table.index.name = None\n",
    "energylevels_table.columns.names = ('Country (ISO code)',\n",
    "                                    'Description', None,\n",
    "                                    None, None,\n",
    "                                    'Level')\n",
    "\n",
    "energylevels_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete downloaded zip files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted: download\\ENTSO-E\\SO&AF\\2011\\SO_AF_2011_-_2025_.zip\n",
      "Deleted: download\\ENTSO-E\\SO&AF\\2012\\120705_SOAF_2012_Dataset.zip\n",
      "Deleted: download\\ENTSO-E\\SO&AF\\2013\\130403_SOAF_2013-2030_dataset.zip\n",
      "Deleted: download\\ENTSO-E\\SO&AF\\2014\\140602_SOAF%202014_dataset.zip\n",
      "Deleted: download\\ENTSO-E\\SO&AF\\2016\\SO_AF_2015_dataset.zip\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(\"download\"):\n",
    "    for file in files:\n",
    "        item = os.path.join(root, file)\n",
    "        if item.endswith(\".zip\"):\n",
    "            os.remove(item)\n",
    "            print(\"Deleted: \" + item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy input files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "orig_data_path = os.path.join('output', 'original_data')\n",
    "shutil.rmtree(orig_data_path, ignore_errors=True)\n",
    "func.copydir(os.path.join('input'), orig_data_path)\n",
    "func.copydir(os.path.join('download'), orig_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Write results to file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write stacked data to formats: csv, xls and sql."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the result to file\n",
    "data.to_csv(os.path.join('output', 'national_generation_capacity_stacked.csv'),\n",
    "            encoding='utf-8', index_label='ID')\n",
    "\n",
    "# Write the results to excel file\n",
    "data.to_excel(os.path.join('output', 'national_generation_capacity_stacked.xlsx'),\n",
    "              sheet_name='output', index_label='ID')\n",
    "\n",
    "# Write the results to sql database\n",
    "data.to_sql('national_generation_capacity_stacked',\n",
    "            sqlite3.connect(os.path.join('output',\n",
    "                                         'national_generation_capacity.sqlite')),\n",
    "            if_exists=\"replace\", index_label='ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write data in human readable form to excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write crosstable data to excel file\n",
    "writer = pd.ExcelWriter(os.path.join('output', 'national_generation_capacity.xlsx'))\n",
    "data_crosstable.to_excel(writer, sheet_name='output')\n",
    "energylevels_table.to_excel(writer, sheet_name='technology levels')\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Formatting of Excel tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputxls = openpyxl.load_workbook(os.path.join('output',\n",
    "                                          'national_generation_capacity.xlsx'))\n",
    "\n",
    "ws1 = outputxls['output']\n",
    "ws2 = outputxls['technology levels']\n",
    "ws1_rows, ws1_cols = data_crosstable.shape\n",
    "amount_cols = ws1_cols + 1 # correct 0 index\n",
    "\n",
    "ws1.column_dimensions['A'].width = 50\n",
    "ws2.column_dimensions['A'].width = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "blackfont = Font(color=colors.BLACK, italic=False, bold=False)\n",
    "blackfontitalic = Font(color=colors.BLACK, italic=True, bold=False)\n",
    "blackfontbold = Font(color=colors.BLACK, italic=False, bold=True)\n",
    "\n",
    "align0 = Alignment(horizontal='left', indent=0)\n",
    "align1 = Alignment(horizontal='left', indent=1)\n",
    "align2 = Alignment(horizontal='left', indent=2)\n",
    "\n",
    "# darkest grey\n",
    "colour = \"{0:02X}{1:02X}{2:02X}\".format(166, 166, 166)\n",
    "grey166 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
    "\n",
    "# darker grey\n",
    "colour = \"{0:02X}{1:02X}{2:02X}\".format(191, 191, 191)\n",
    "grey191 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
    "\n",
    "# lighter grey\n",
    "colour = \"{0:02X}{1:02X}{2:02X}\".format(217, 217, 217)\n",
    "grey217 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
    "\n",
    "# lightest grey\n",
    "colour = \"{0:02X}{1:02X}{2:02X}\".format(242, 242, 242)\n",
    "grey242 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in range(2, amount_cols+1):\n",
    "    colname = openpyxl.utils.cell.get_column_letter(col)\n",
    "    ws1.column_dimensions[colname].width = 16\n",
    "\n",
    "\n",
    "for col in range(1, amount_cols+1):\n",
    "    # format column name block\n",
    "    for row in range(2,8):\n",
    "        \n",
    "        ws1.cell(row=row, column=col).font = blackfont\n",
    "        \n",
    "\n",
    "    # format cells that contain the values\n",
    "    for row in range(9, 48):\n",
    "        ws1.cell(row=row, column=col).fill = grey242\n",
    "        ws1.cell(row=row, column=1).font = blackfontitalic\n",
    "        ws1.cell(row=row, column=1).alignment = align2\n",
    "    \n",
    "    \n",
    "    # format row 'Total' with dark grey\n",
    "    ws1.cell(row=47, column=col).fill = grey166\n",
    "    ws1.cell(row=47, column=col).font = blackfontbold\n",
    "    \n",
    "    \n",
    "    # format level 1\n",
    "    for row in [9, 22, 23, 46]:\n",
    "        ws1.cell(row=row, column=col).fill = grey191\n",
    "        ws1.cell(row=row, column=col).font = blackfontbold\n",
    "        ws1.cell(row=row, column=1).alignment = align0\n",
    "        \n",
    "    \n",
    "    # format level 2\n",
    "    for row in [10, 11, 12, 13, 18, 19, 20, 21, 24, 31, 35, 39, 40, 41, 45]:\n",
    "        ws1.cell(row=row, column=col).fill = grey217\n",
    "        ws1.cell(row=row, column=1).alignment = align1\n",
    "\n",
    "    \n",
    "ws1.cell(row=47, column=1).alignment = align0\n",
    "ws1.freeze_panes = ws1['B8'] #freeze first column and header rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the same for the second worksheet 'technology levels' \n",
    "for col in range(1, 7):\n",
    "    colname = get_column_letter(col + 1)\n",
    "    ws2.column_dimensions[colname].width = 25\n",
    "    \n",
    "    for row in range(2, 8):\n",
    "        ws2.cell(row=row, column=col).font = blackfont\n",
    "        \n",
    "        \n",
    "    for row in range(9, 48):     \n",
    "        ws2.cell(row=row, column=col).fill = grey242\n",
    "        ws2.cell(row=row, column=1).font = blackfontitalic\n",
    "        ws2.cell(row=row, column=1).alignment = align2\n",
    "        \n",
    "        \n",
    "    # format row 'Total' with dark grey\n",
    "    ws2.cell(row=46, column=col).fill = grey166\n",
    "    \n",
    "    \n",
    "    # format level 1\n",
    "    for row in [8, 21, 22, 45]:\n",
    "        ws2.cell(row=row, column=col).fill = grey191\n",
    "        ws2.cell(row=row, column=col).font = blackfontbold\n",
    "        ws2.cell(row=row, column=1).font = blackfontbold\n",
    "        ws2.cell(row=row, column=1).alignment = align0\n",
    "        \n",
    "    \n",
    "    # format level 2\n",
    "    for row in [9, 10, 11, 12, 17, 18, 19, 20, 23, 30, 34, 38, 39, 40, 44]:\n",
    "        ws2.cell(row=row, column=col).fill = grey217\n",
    "        ws2.cell(row=row, column=1).alignment = align1\n",
    "\n",
    "    \n",
    "ws2.cell(row=46, column=1).alignment = align0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "additional_notes = openpyxl.load_workbook(os.path.join('input',\n",
    "                                          'National_Generation_Capacities.xlsx'))['Additional notes']\n",
    "# copy additional notes to output file\n",
    "for col in range(1, 3):\n",
    "    for row in range(1, 10):\n",
    "        add_notes_value = additional_notes.cell(row=row, column=col).value\n",
    "        ws1.cell(row=row + 50, column=col).value = add_notes_value\n",
    "        ws1.cell(row=51, column=1).font = blackfontbold\n",
    "        ws1.cell(row=row + 51, column=1).font = blackfontitalic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputxls.save(os.path.join('output', 'national_generation_capacity.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Write checksums"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = ['national_generation_capacity.xlsx',\n",
    "        'national_generation_capacity_stacked.csv',\n",
    "        'national_generation_capacity_stacked.xlsx',\n",
    "        'national_generation_capacity.sqlite']\n",
    "\n",
    "hash_dict = {}\n",
    "filesize_dict = {}\n",
    "\n",
    "with open('checksums.txt', 'w') as f:\n",
    "    for file_name in files:\n",
    "        path = os.path.join('output', file_name)\n",
    "        file_hash = func.get_sha_hash(path)\n",
    "        hash_dict[file_name] = file_hash\n",
    "        filesize_dict[file_name] = os.path.getsize(path)\n",
    "        f.write('{},{}\\n'.format(file_name, file_hash))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Documentation of the data package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We document the data packages meta data in the specific format JSON as proposed by the Open Knowledge Foundation. See the Frictionless Data project by OKFN (http://data.okfn.org/) and the Data Package specifications (http://dataprotocols.org/data-packages/) for more details.\n",
    "\n",
    "In order to keep the notebook more readable, we first formulate the metadata in the human-readable YAML format using a multi-line string. We then parse the string into a Python dictionary and save that to disk as a JSON file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join('input', 'metadata.yml'), 'r') as f:\n",
    "    metadata = yaml.load(f.read(), Loader=yaml.BaseLoader)\n",
    "    \n",
    "metadata['resources'][0]['hash'] = hash_dict['national_generation_capacity.xlsx']\n",
    "metadata['resources'][1]['hash'] = hash_dict['national_generation_capacity_stacked.csv']\n",
    "metadata['resources'][0]['bytes'] = filesize_dict['national_generation_capacity.xlsx']\n",
    "metadata['resources'][1]['bytes'] = filesize_dict['national_generation_capacity_stacked.csv']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "datapackage_json = json.dumps(metadata, indent=4, separators=(',', ': '))\n",
    "    \n",
    "# Write the information of the metadata\n",
    "with open(os.path.join('output', 'datapackage.json'), 'w') as f:\n",
    "    f.write(datapackage_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "End of script."
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}