2_imputations_concatenating.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imputing missing values\n",
    "\n",
    "We work with the data set how it is present now and apply a common machine learning methods to compute the imputations for missing values: \n",
    "\n",
    "The **weighted k nearest neighbour (w-kNN)** algorithm, which imputes missing values with weights equal to the inverse Euclidean distance. \n",
    "\n",
    "**Assumptions:** we believe the values missing lie in between the boundaries of the highest and lowest value present in the data set. This might work well for most, but it could also be that the missing values are outliers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note on `countries`\n",
    "\n",
    "We want to **remind** ourselves that our list `groupings` does not only contain countries, it also contains groups of countries like continents, economic zones, etc. and islands which belong to certain countries, but could somehow be very different as e.g. territories. We save all these entries in a new list `countries`. \n",
    "\n",
    "Furthermore, some sub-indicators are of ordinal type, i.e. they are defined as 'number of countries which...'. For all countries, we have here either a 1 or a 0 for 'yes' or 'no', respectively. For all other groupings, we have there most likely a number larger than 1. This could be tricky for our imputations later, because these are based on the similarity of two groupings and these similarities could be strong between, say, France and the World Trade Organisation (WTO). If the WTO had a missing value for an ordinal sub-indicator, the imputation would most likely be very similar to the one of France, so 1 or 0. But this is unrealistic, and all other countries being member of the WTO and behave almost as similar as France, would make the imputation just closer to 1 and not larger than 1.\n",
    "\n",
    "Therefore, we focus on `countries` only from hereon. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import math\n",
    "import os\n",
    "import pickle\n",
    "import copy\n",
    "import itertools\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from tqdm.notebook import tqdm\n",
    "from sklearn.preprocessing import scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading original and standardised data set\n",
    "dict_all = pickle.load(open('utils/data/dict_all_wb.pkl', 'rb'))\n",
    "dict_all_std = pickle.load(open('utils/data/dict_all_wb_std.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original values: \n",
      "1990          NaN\n",
      "1991          NaN\n",
      "1992          NaN\n",
      "1993          NaN\n",
      "1994          NaN\n",
      "1995          NaN\n",
      "1996          NaN\n",
      "1997    64.083333\n",
      "1998          NaN\n",
      "1999          NaN\n",
      "2000          NaN\n",
      "2001          NaN\n",
      "2002    56.125000\n",
      "2003          NaN\n",
      "2004          NaN\n",
      "2005          NaN\n",
      "2006          NaN\n",
      "2007    51.783333\n",
      "2008          NaN\n",
      "2009          NaN\n",
      "2010          NaN\n",
      "2011          NaN\n",
      "2012    50.016667\n",
      "2013          NaN\n",
      "2014          NaN\n",
      "2015          NaN\n",
      "2016          NaN\n",
      "2017          NaN\n",
      "2018          NaN\n",
      "2019          NaN\n",
      "Name: ER.H2O.FWTL.ZS, dtype: float64\n",
      "--------\n",
      "Standardised values: \n",
      "1990         NaN\n",
      "1991         NaN\n",
      "1992         NaN\n",
      "1993         NaN\n",
      "1994         NaN\n",
      "1995         NaN\n",
      "1996         NaN\n",
      "1997    1.580306\n",
      "1998         NaN\n",
      "1999         NaN\n",
      "2000         NaN\n",
      "2001         NaN\n",
      "2002    0.114715\n",
      "2003         NaN\n",
      "2004         NaN\n",
      "2005         NaN\n",
      "2006         NaN\n",
      "2007   -0.684838\n",
      "2008         NaN\n",
      "2009         NaN\n",
      "2010         NaN\n",
      "2011         NaN\n",
      "2012   -1.010183\n",
      "2013         NaN\n",
      "2014         NaN\n",
      "2015         NaN\n",
      "2016         NaN\n",
      "2017         NaN\n",
      "2018         NaN\n",
      "2019         NaN\n",
      "Name: ER.H2O.FWTL.ZS, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print('Original values: ')\n",
    "print(dict_all['Belgium'].loc['ER.H2O.FWTL.ZS'])\n",
    "\n",
    "print('--------')\n",
    "\n",
    "print('Standardised values: ')\n",
    "print(dict_all_std['Belgium'].loc['ER.H2O.FWTL.ZS'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the total number of values we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "850591\n"
     ]
    }
   ],
   "source": [
    "# number of values\n",
    "s = 0\n",
    "for country in dict_all_std.keys():\n",
    "    s += np.sum(dict_all_std[country].count())\n",
    "\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we open the `csv` file in a GUI and delete the groupings which are *not* countries or part of countries. We call these non-country groupings and examples are North America, Western Asia, Least Developed Countries (LDC), Land Locked Developing Countries (LLDC), Small Island Developing States (SIDS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Afghanistan',\n",
       " 'Albania',\n",
       " 'Algeria',\n",
       " 'Angola',\n",
       " 'Antigua and Barbuda',\n",
       " 'Argentina',\n",
       " 'Armenia',\n",
       " 'Australia',\n",
       " 'Austria',\n",
       " 'Azerbaijan',\n",
       " 'Bahamas, The',\n",
       " 'Bahrain',\n",
       " 'Bangladesh',\n",
       " 'Barbados',\n",
       " 'Belarus',\n",
       " 'Belgium',\n",
       " 'Belize',\n",
       " 'Benin',\n",
       " 'Bhutan',\n",
       " 'Bolivia',\n",
       " 'Bosnia and Herzegovina',\n",
       " 'Botswana',\n",
       " 'Brazil',\n",
       " 'Brunei Darussalam',\n",
       " 'Bulgaria',\n",
       " 'Burkina Faso',\n",
       " 'Burundi',\n",
       " 'Cambodia',\n",
       " 'Cameroon',\n",
       " 'Canada',\n",
       " 'Central African Republic',\n",
       " 'Chad',\n",
       " 'Chile',\n",
       " 'China',\n",
       " 'Colombia',\n",
       " 'Comoros',\n",
       " 'Congo, Dem. Rep.',\n",
       " 'Congo, Rep.',\n",
       " 'Costa Rica',\n",
       " \"Cote d'Ivoire\",\n",
       " 'Croatia',\n",
       " 'Cuba',\n",
       " 'Cyprus',\n",
       " 'Czech Republic',\n",
       " 'Denmark',\n",
       " 'Djibouti',\n",
       " 'Dominica',\n",
       " 'Dominican Republic',\n",
       " 'Ecuador',\n",
       " 'Egypt, Arab Rep.',\n",
       " 'El Salvador',\n",
       " 'Equatorial Guinea',\n",
       " 'Eritrea',\n",
       " 'Estonia',\n",
       " 'Ethiopia',\n",
       " 'Fiji',\n",
       " 'Finland',\n",
       " 'France',\n",
       " 'Gabon',\n",
       " 'Gambia, The',\n",
       " 'Georgia',\n",
       " 'Germany',\n",
       " 'Ghana',\n",
       " 'Greece',\n",
       " 'Grenada',\n",
       " 'Guatemala',\n",
       " 'Guinea',\n",
       " 'Guinea-Bissau',\n",
       " 'Guyana',\n",
       " 'Haiti',\n",
       " 'Honduras',\n",
       " 'Hungary',\n",
       " 'Iceland',\n",
       " 'India',\n",
       " 'Indonesia',\n",
       " 'Iran, Islamic Rep.',\n",
       " 'Iraq',\n",
       " 'Ireland',\n",
       " 'Israel',\n",
       " 'Italy',\n",
       " 'Jamaica',\n",
       " 'Japan',\n",
       " 'Jordan',\n",
       " 'Kazakhstan',\n",
       " 'Kenya',\n",
       " 'Kiribati',\n",
       " 'Korea, Rep.',\n",
       " 'Kuwait',\n",
       " 'Kyrgyz Republic',\n",
       " 'Lao PDR',\n",
       " 'Latvia',\n",
       " 'Lebanon',\n",
       " 'Lesotho',\n",
       " 'Liberia',\n",
       " 'Libya',\n",
       " 'Lithuania',\n",
       " 'Luxembourg',\n",
       " 'Madagascar',\n",
       " 'Malawi',\n",
       " 'Malaysia',\n",
       " 'Maldives',\n",
       " 'Mali',\n",
       " 'Malta',\n",
       " 'Mauritania',\n",
       " 'Mauritius',\n",
       " 'Mexico',\n",
       " 'Micronesia, Fed. Sts.',\n",
       " 'Moldova',\n",
       " 'Mongolia',\n",
       " 'Montenegro',\n",
       " 'Morocco',\n",
       " 'Mozambique',\n",
       " 'Myanmar',\n",
       " 'Namibia',\n",
       " 'Nepal',\n",
       " 'Netherlands',\n",
       " 'New Zealand',\n",
       " 'Nicaragua',\n",
       " 'Niger',\n",
       " 'Nigeria',\n",
       " 'Norway',\n",
       " 'Oman',\n",
       " 'Pakistan',\n",
       " 'Palau',\n",
       " 'Panama',\n",
       " 'Papua New Guinea',\n",
       " 'Paraguay',\n",
       " 'Peru',\n",
       " 'Philippines',\n",
       " 'Poland',\n",
       " 'Portugal',\n",
       " 'Puerto Rico',\n",
       " 'Qatar',\n",
       " 'Romania',\n",
       " 'Russian Federation',\n",
       " 'Rwanda',\n",
       " 'Samoa',\n",
       " 'Sao Tome and Principe',\n",
       " 'Saudi Arabia',\n",
       " 'Senegal',\n",
       " 'Serbia',\n",
       " 'Seychelles',\n",
       " 'Sierra Leone',\n",
       " 'Singapore',\n",
       " 'Slovak Republic',\n",
       " 'Slovenia',\n",
       " 'Solomon Islands',\n",
       " 'Somalia',\n",
       " 'South Africa',\n",
       " 'South Sudan',\n",
       " 'Spain',\n",
       " 'Sri Lanka',\n",
       " 'Sudan',\n",
       " 'Suriname',\n",
       " 'Sweden',\n",
       " 'Switzerland',\n",
       " 'Syrian Arab Republic',\n",
       " 'Tajikistan',\n",
       " 'Tanzania',\n",
       " 'Thailand',\n",
       " 'Timor-Leste',\n",
       " 'Togo',\n",
       " 'Tonga',\n",
       " 'Trinidad and Tobago',\n",
       " 'Tunisia',\n",
       " 'Turkey',\n",
       " 'Turkmenistan',\n",
       " 'Tuvalu',\n",
       " 'Uganda',\n",
       " 'Ukraine',\n",
       " 'United Arab Emirates',\n",
       " 'United Kingdom',\n",
       " 'United States',\n",
       " 'Uruguay',\n",
       " 'Uzbekistan',\n",
       " 'Vanuatu',\n",
       " 'Venezuela, RB',\n",
       " 'Vietnam',\n",
       " 'Yemen, Rep.',\n",
       " 'Zambia',\n",
       " 'Zimbabwe']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read amended csv file\n",
    "c = pd.read_csv('utils/countries_wb.csv', dtype=str, delimiter=';', header=None)\n",
    "countries = list(c[0])\n",
    "\n",
    "# check\n",
    "countries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We append these temperature series to the data frame of indicators. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = pickle.load(open('utils/data/temp.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_years = temp['Germany']['YEAR'].astype(str).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Hint:* When the data are downloaded, the names of countries do not match exactly. For example 'Republic of Serbia' is the name in the temperature data set, whereas 'Serbia' is the name in the SDG data set. These are aligned manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# countries in dict_all_std: 181\n",
      "# countries in temp: 183\n"
     ]
    }
   ],
   "source": [
    "print('# countries in dict_all_std:', len(dict_all_std.keys()))\n",
    "print('# countries in temp:', len(temp.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Micronesia, Fed. Sts.\n",
      "--------\n",
      "Greenland\n",
      "Liechtenstein\n",
      "Korea, Dem. People's Rep.\n"
     ]
    }
   ],
   "source": [
    "# which country is in dict_all_std but not in temp?\n",
    "for key in dict_all_std.keys():\n",
    "    if key not in temp.keys():\n",
    "        print(key)\n",
    "\n",
    "print('--------')\n",
    "\n",
    "for key in temp.keys():\n",
    "    if key not in dict_all_std.keys():\n",
    "        print(key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>YEAR</th>\n",
       "      <th>JAN</th>\n",
       "      <th>FEB</th>\n",
       "      <th>MAR</th>\n",
       "      <th>APR</th>\n",
       "      <th>MAY</th>\n",
       "      <th>JUN</th>\n",
       "      <th>JUL</th>\n",
       "      <th>AUG</th>\n",
       "      <th>SEP</th>\n",
       "      <th>OCT</th>\n",
       "      <th>NOV</th>\n",
       "      <th>DEC</th>\n",
       "      <th>MAM</th>\n",
       "      <th>JJA</th>\n",
       "      <th>SON</th>\n",
       "      <th>DJF</th>\n",
       "      <th>ANN</th>\n",
       "      <th>AVG</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1901   -10.5   -10.1</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>12.1</td>\n",
       "      <td>16.7</td>\n",
       "      <td>19.4</td>\n",
       "      <td>20.9</td>\n",
       "      <td>15.1</td>\n",
       "      <td>8.7</td>\n",
       "      <td>-0.3   -10.4</td>\n",
       "      <td>5.6</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7.8   -10.5</td>\n",
       "      <td>5.5</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.228571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1902   -12.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>-0.9</td>\n",
       "      <td>5.8</td>\n",
       "      <td>11.5</td>\n",
       "      <td>16.2</td>\n",
       "      <td>19.3</td>\n",
       "      <td>19.3</td>\n",
       "      <td>14.9</td>\n",
       "      <td>8.8</td>\n",
       "      <td>1.5</td>\n",
       "      <td>-7.3</td>\n",
       "      <td>5.5</td>\n",
       "      <td>18.3</td>\n",
       "      <td>8.4</td>\n",
       "      <td>-8.6</td>\n",
       "      <td>5.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1903   -10.5</td>\n",
       "      <td>-7.9</td>\n",
       "      <td>-0.6</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11.7</td>\n",
       "      <td>16.6</td>\n",
       "      <td>19.8</td>\n",
       "      <td>21.1</td>\n",
       "      <td>15.8</td>\n",
       "      <td>7.4</td>\n",
       "      <td>-1.1   -10.0</td>\n",
       "      <td>6.1</td>\n",
       "      <td>19.2</td>\n",
       "      <td>7.4   -10.4</td>\n",
       "      <td>5.8</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.085714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1904   -12.9</td>\n",
       "      <td>-8.5</td>\n",
       "      <td>-2.7</td>\n",
       "      <td>6.6</td>\n",
       "      <td>12.1</td>\n",
       "      <td>17.6</td>\n",
       "      <td>21.0</td>\n",
       "      <td>21.3</td>\n",
       "      <td>14.9</td>\n",
       "      <td>7.3</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-8.6</td>\n",
       "      <td>5.3</td>\n",
       "      <td>20.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>-8.9</td>\n",
       "      <td>5.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.971429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1905</td>\n",
       "      <td>-8.3</td>\n",
       "      <td>-9.9</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>4.9</td>\n",
       "      <td>11.4</td>\n",
       "      <td>16.8</td>\n",
       "      <td>20.3</td>\n",
       "      <td>19.9</td>\n",
       "      <td>14.9</td>\n",
       "      <td>8.2</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-7.2</td>\n",
       "      <td>4.8</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7.4   -10.5</td>\n",
       "      <td>5.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.757143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>2015</td>\n",
       "      <td>-9.6</td>\n",
       "      <td>-6.5</td>\n",
       "      <td>0.1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>21.2</td>\n",
       "      <td>21.5</td>\n",
       "      <td>16.2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>-0.0</td>\n",
       "      <td>-6.5</td>\n",
       "      <td>7.4</td>\n",
       "      <td>20.2</td>\n",
       "      <td>8.4</td>\n",
       "      <td>-8.6</td>\n",
       "      <td>7.1</td>\n",
       "      <td>10.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>2016   -11.9</td>\n",
       "      <td>-7.3</td>\n",
       "      <td>0.7</td>\n",
       "      <td>8.3</td>\n",
       "      <td>14.2</td>\n",
       "      <td>18.1</td>\n",
       "      <td>21.6</td>\n",
       "      <td>22.4</td>\n",
       "      <td>16.4</td>\n",
       "      <td>8.6</td>\n",
       "      <td>-1.2</td>\n",
       "      <td>-6.8</td>\n",
       "      <td>7.8</td>\n",
       "      <td>20.7</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-7.8</td>\n",
       "      <td>6.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.528571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>2017</td>\n",
       "      <td>-9.7</td>\n",
       "      <td>-6.8</td>\n",
       "      <td>0.4</td>\n",
       "      <td>9.1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>21.1</td>\n",
       "      <td>16.2</td>\n",
       "      <td>8.8</td>\n",
       "      <td>-0.8</td>\n",
       "      <td>-9.7</td>\n",
       "      <td>8.0</td>\n",
       "      <td>20.7</td>\n",
       "      <td>8.1   -10.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.328571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>2018   -12.3</td>\n",
       "      <td>-9.4</td>\n",
       "      <td>0.5</td>\n",
       "      <td>8.5</td>\n",
       "      <td>13.5</td>\n",
       "      <td>18.5</td>\n",
       "      <td>23.2</td>\n",
       "      <td>22.0</td>\n",
       "      <td>15.5</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1.1</td>\n",
       "      <td>-8.3</td>\n",
       "      <td>7.5</td>\n",
       "      <td>21.3</td>\n",
       "      <td>8.2</td>\n",
       "      <td>-7.7</td>\n",
       "      <td>6.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.528571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>2019</td>\n",
       "      <td>-8.5</td>\n",
       "      <td>-6.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>7.5</td>\n",
       "      <td>14.9</td>\n",
       "      <td>17.5</td>\n",
       "      <td>21.8</td>\n",
       "      <td>21.8</td>\n",
       "      <td>16.9</td>\n",
       "      <td>9.8</td>\n",
       "      <td>0.3</td>\n",
       "      <td>-7.3</td>\n",
       "      <td>7.9</td>\n",
       "      <td>20.4</td>\n",
       "      <td>9.0  -999.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.242857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>119 rows × 19 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     YEAR   JAN  FEB   MAR   APR   MAY   JUN   JUL   AUG  \\\n",
       "0    1901   -10.5   -10.1  -2.0  6.7  12.1  16.7  19.4  20.9  15.1   8.7   \n",
       "1            1902   -12.0  -9.0 -0.9   5.8  11.5  16.2  19.3  19.3  14.9   \n",
       "2            1903   -10.5  -7.9 -0.6   7.2  11.7  16.6  19.8  21.1  15.8   \n",
       "3            1904   -12.9  -8.5 -2.7   6.6  12.1  17.6  21.0  21.3  14.9   \n",
       "4                    1905  -8.3 -9.9  -2.1   4.9  11.4  16.8  20.3  19.9   \n",
       "..                    ...   ...  ...   ...   ...   ...   ...   ...   ...   \n",
       "114                  2015  -9.6 -6.5   0.1   8.0  14.0  18.0  21.2  21.5   \n",
       "115          2016   -11.9  -7.3  0.7   8.3  14.2  18.1  21.6  22.4  16.4   \n",
       "116                  2017  -9.7 -6.8   0.4   9.1  14.5  18.0  23.0  21.1   \n",
       "117          2018   -12.3  -9.4  0.5   8.5  13.5  18.5  23.2  22.0  15.5   \n",
       "118                  2019  -8.5 -6.2   1.4   7.5  14.9  17.5  21.8  21.8   \n",
       "\n",
       "              SEP           OCT   NOV           DEC           MAM   JJA  \\\n",
       "0    -0.3   -10.4           5.6  19.0   7.8   -10.5           5.5  None   \n",
       "1             8.8           1.5  -7.3           5.5          18.3   8.4   \n",
       "2             7.4  -1.1   -10.0   6.1          19.2   7.4   -10.4   5.8   \n",
       "3             7.3          -1.0  -8.6           5.3          20.0   7.1   \n",
       "4            14.9           8.2  -1.0          -7.2           4.8  19.0   \n",
       "..            ...           ...   ...           ...           ...   ...   \n",
       "114          16.2           9.0  -0.0          -6.5           7.4  20.2   \n",
       "115           8.6          -1.2  -6.8           7.8          20.7   8.0   \n",
       "116          16.2           8.8  -0.8          -9.7           8.0  20.7   \n",
       "117           8.1           1.1  -8.3           7.5          21.3   8.2   \n",
       "118          16.9           9.8   0.3          -7.3           7.9  20.4   \n",
       "\n",
       "              SON  DJF  ANN        AVG  \n",
       "0            None  NaN  NaN  14.228571  \n",
       "1            -8.6  5.7  NaN  12.300000  \n",
       "2            None  NaN  NaN  13.085714  \n",
       "3            -8.9  5.6  NaN  12.971429  \n",
       "4     7.4   -10.5  5.7  NaN   8.757143  \n",
       "..            ...  ...  ...        ...  \n",
       "114           8.4 -8.6  7.1  10.900000  \n",
       "115          -7.8  6.9  NaN  14.528571  \n",
       "116   8.1   -10.5  7.0  NaN  11.328571  \n",
       "117          -7.7  6.7  NaN  14.528571  \n",
       "118   9.0  -999.0  7.5  NaN  11.242857  \n",
       "\n",
       "[119 rows x 19 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# removes countries in-place\n",
    "countries.remove('Micronesia, Fed. Sts.')\n",
    "dict_all_std.pop('Micronesia, Fed. Sts.', None)\n",
    "temp.pop('Greenland', None)\n",
    "temp.pop('Liechtenstein', None)\n",
    "temp.pop(\"Korea, Dem. People's Rep.\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "temp['Serbia'] = temp.pop('Republic of Serbia')\n",
    "temp['Montenegro'] = temp.pop('Republic of Montenegro')\n",
    "temp['Timor-Leste'] = temp.pop('Timor Leste')\n",
    "temp['Slovak Republic'] = temp.pop('Slovakia')\n",
    "temp['Micronesia, Fed. Sts.'] = temp.pop('Federated States of Micronesia')\n",
    "temp['Yemen, Rep.'] = temp.pop('Yemen')\n",
    "temp['Syrian Arab Republic'] = temp.pop('Syria')\n",
    "temp.pop('Swaziland')\n",
    "temp['Egypt, Arab Rep.'] = temp.pop('Egypt')\n",
    "temp['Myanmar'] = temp.pop('Myanmar (Burma)')\n",
    "temp['Congo, Dem. Rep.'] = temp.pop('Congo (Democratic Republic of the)')\n",
    "temp['Bahamas, The'] = temp.pop('Bahamas')\n",
    "temp.pop('Northern Mariana Islands')\n",
    "temp.pop('Marshall Islands')\n",
    "temp.pop('Monaco')\n",
    "temp.pop('St. Vincent and the Grenadines')\n",
    "temp.pop('St. Lucia')\n",
    "temp.pop('Andorra')\n",
    "temp.pop('Faroe Islands')\n",
    "temp.pop('Cape Verde')\n",
    "temp.pop('Macedonia')\n",
    "temp['Congo, Rep.'] = temp.pop('Congo (Republic of the)')\n",
    "temp['Iran, Islamic Rep.'] = temp.pop('Iran')\n",
    "temp['Brunei Darussalam'] = temp.pop('Brunei')\n",
    "temp.pop('St. Kitts and Nevis')\n",
    "temp['Kyrgyz Republic'] = temp.pop('Kyrgyzstan')\n",
    "temp['Venezuela, RB'] = temp.pop('Venezuela')\n",
    "temp.pop('New Caledonia')\n",
    "temp['Lao PDR'] = temp.pop('Laos')\n",
    "temp['Russian Federation'] = temp.pop('Russia')\n",
    "temp['Korea, Dem. People\\'s Rep.'] = temp.pop('Korea')\n",
    "temp['Gambia, The'] = temp.pop('Gambia')\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Afghanistan\n",
      "Albania\n",
      "Algeria\n",
      "Angola\n",
      "Argentina\n",
      "Armenia\n",
      "Australia\n",
      "Austria\n",
      "Azerbaijan\n",
      "Bahrain\n",
      "Bangladesh\n",
      "Barbados\n",
      "Belarus\n",
      "Belgium\n",
      "Belize\n",
      "Benin\n",
      "Bhutan\n",
      "Bolivia\n",
      "Botswana\n",
      "Brazil\n",
      "Bulgaria\n",
      "Burundi\n",
      "Cambodia\n",
      "Cameroon\n",
      "Canada\n",
      "Chad\n",
      "Chile\n",
      "China\n",
      "Colombia\n",
      "Comoros\n",
      "Croatia\n",
      "Cuba\n",
      "Cyprus\n",
      "Denmark\n",
      "Djibouti\n",
      "Dominica\n",
      "Ecuador\n",
      "Eritrea\n",
      "Estonia\n",
      "Ethiopia\n",
      "Fiji\n",
      "Finland\n",
      "France\n",
      "Gabon\n",
      "Georgia\n",
      "Germany\n",
      "Ghana\n",
      "Greece\n",
      "Grenada\n",
      "Guatemala\n",
      "Guinea\n",
      "Guinea-Bissau\n",
      "Guyana\n",
      "Haiti\n",
      "Honduras\n",
      "Hungary\n",
      "Iceland\n",
      "India\n",
      "Indonesia\n",
      "Iraq\n",
      "Ireland\n",
      "Israel\n",
      "Italy\n",
      "Jamaica\n",
      "Japan\n",
      "Jordan\n",
      "Kazakhstan\n",
      "Kenya\n",
      "Kiribati\n",
      "Kuwait\n",
      "Latvia\n",
      "Lebanon\n",
      "Lesotho\n",
      "Liberia\n",
      "Libya\n",
      "Lithuania\n",
      "Luxembourg\n",
      "Madagascar\n",
      "Malawi\n",
      "Malaysia\n",
      "Maldives\n",
      "Mali\n",
      "Malta\n",
      "Mauritania\n",
      "Mauritius\n",
      "Mexico\n",
      "Moldova\n",
      "Mongolia\n",
      "Montenegro\n",
      "Morocco\n",
      "Mozambique\n",
      "Myanmar\n",
      "Namibia\n",
      "Nepal\n",
      "Netherlands\n",
      "Nicaragua\n",
      "Niger\n",
      "Nigeria\n",
      "Norway\n",
      "Oman\n",
      "Pakistan\n",
      "Palau\n",
      "Panama\n",
      "Paraguay\n",
      "Peru\n",
      "Philippines\n",
      "Poland\n",
      "Portugal\n",
      "Qatar\n",
      "Romania\n",
      "Rwanda\n",
      "Samoa\n",
      "Senegal\n",
      "Serbia\n",
      "Seychelles\n",
      "Singapore\n",
      "Slovenia\n",
      "Somalia\n",
      "Spain\n",
      "Sudan\n",
      "Suriname\n",
      "Sweden\n",
      "Switzerland\n",
      "Tajikistan\n",
      "Tanzania\n",
      "Thailand\n",
      "Togo\n",
      "Tonga\n",
      "Tunisia\n",
      "Turkey\n",
      "Turkmenistan\n",
      "Tuvalu\n",
      "Uganda\n",
      "Ukraine\n",
      "Uruguay\n",
      "Uzbekistan\n",
      "Vietnam\n",
      "Zambia\n",
      "Zimbabwe\n",
      "Antigua and Barbuda\n",
      "Bahamas, The\n",
      "Bosnia and Herzegovina\n",
      "Brunei Darussalam\n",
      "Burkina Faso\n",
      "Central African Republic\n",
      "Congo, Dem. Rep.\n",
      "Congo, Rep.\n",
      "Costa Rica\n",
      "Cote d'Ivoire\n",
      "Czech Republic\n",
      "Dominican Republic\n",
      "Egypt, Arab Rep.\n",
      "El Salvador\n",
      "Equatorial Guinea\n",
      "Gambia, The\n",
      "Iran, Islamic Rep.\n",
      "Korea, Rep.\n",
      "Kyrgyz Republic\n",
      "Lao PDR\n",
      "New Zealand\n",
      "Papua New Guinea\n",
      "Puerto Rico\n",
      "Russian Federation\n",
      "Sao Tome and Principe\n",
      "Saudi Arabia\n",
      "Sierra Leone\n",
      "Slovak Republic\n",
      "Solomon Islands\n",
      "South Africa\n",
      "South Sudan\n",
      "Sri Lanka\n",
      "Syrian Arab Republic\n",
      "Timor-Leste\n",
      "Trinidad and Tobago\n",
      "United Arab Emirates\n",
      "United Kingdom\n",
      "United States\n",
      "Vanuatu\n",
      "Venezuela, RB\n",
      "Yemen, Rep.\n"
     ]
    }
   ],
   "source": [
    "for country in temp.keys():\n",
    "    print(country)\n",
    "    temp[country] = temp[country].set_index('YEAR').T\n",
    "    temp[country].columns = temp[country].columns.astype(str)\n",
    "    temp[country].rename(index={'AVG': 'Temperature'}, inplace=True)\n",
    "    temp[country] = temp[country].loc['Temperature'] #, period]\n",
    "    temp[country] = pd.Series(scale(temp[country]), index=temp_years, name='Temperature')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1901    0.486906\n",
       "1902   -0.394132\n",
       "1903   -0.415886\n",
       "1904   -0.937982\n",
       "1905   -0.600795\n",
       "          ...   \n",
       "2015    1.726886\n",
       "2016    0.976372\n",
       "2017    1.259174\n",
       "2018    2.499153\n",
       "2019    1.944426\n",
       "Name: Temperature, Length: 119, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "temp['Azerbaijan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_all_t = {}\n",
    "\n",
    "for country in temp.keys():\n",
    "    temp[country].index = temp[country].index.astype(int).astype(str)\n",
    "    dict_all_t[country] = dict_all_std[country].append(temp[country])\n",
    "    dict_all_std[country] = dict_all_t[country][temp_years]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_to_drop = list(set(list(dict_all_std.keys())).difference(list(temp.keys())))\n",
    "\n",
    "for c in countries_to_drop:\n",
    "    dict_all_std.pop(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# 1) weighted k nearest neighbour (w-kNN)\n",
    "\n",
    "The w-kNN algorithm is straightforward: we calculate how similar countries are in each given year $y$ with the standardised Euclidean distance $E_y$ and take the inverse of the absolute $|E_y|$ as the weight to impute missing values in any given country $c_i$ in each given year $y$ for each given sub-indicator $j$.\n",
    "\n",
    "A good start to understand this algorithm is to understand high-dimensional space: https://youtu.be/wvsE8jm1GzE\n",
    "\n",
    "### Euclidean distance\n",
    "The Euclidean distance $e_y$ for year $y$ for any given pair of countries $(c_i, c_k)$ for any given sub-indicator $j$ is calculated by:\n",
    "$$ e_y(c_{i}, c_{k}) = \\lVert c_{i}, c_{k} \\rVert_2 = \\sqrt{ \\sum_{j=1}^J(c_{ij} - c_{kj})^2} $$\n",
    "\n",
    "We calculate the squared distances between any given pair of countries $(c_i, c_k)$, but do not consider the country $k+1$ which has the largest distance $e_y$ to country $i$. We do so for any given sub-indicator $j$ and take the square root of it. $c_{ij}$ is the sub-indicator $j$ of country $i$, and $i \\neq k$. Thus, any unique pair of countries $i$ and $k$, $i \\neq k$, has **one** Euclidean distance $e_y$ for year $y$ only.\n",
    "\n",
    "Afterwards, we normalise this with respect to the country $k+1$ which has the largest distance $e_y$ to country $i$ by the following equation:\n",
    "\n",
    "$$ E_y(c_{i}, c_{k}) = \\frac{e_y(c_{i}, c_{k})}{e_y(c_{i}, c_{k+1})} $$\n",
    "\n",
    "This can be seen as equivalent to the well-known normalisation equation:\n",
    "\n",
    "$$\n",
    "x_n = \\frac{x - x_{min}}{x_{max}-x_{min}}\n",
    "$$\n",
    "\n",
    "since $x_{min}$ is always 0, because the distance \"between\" the same country is 0.\n",
    "\n",
    "### Imputations\n",
    "We want that our imputations $x^{j}_{i,y}$ for missing sub-indicator $j$ in country $i$ in year $y$ are similar to sub-indicators $j$ of countries $k$ which have a **small** Euclidean distance $E_y$ and dissimilar to sub-indicators $j$ of countries $k$ which have a **large** Euclidean distance $E_y$. Consequently, the imputations $x^{j}_{i,y}$ are the *weighted* averages where the *weights* are equal to the inverse standardised Euclidean distance $\\frac{1}{|E_y(c_{i}, c_{k})|}$.\n",
    "\n",
    "First, we compute $E_y$ for all available pairs of sub-indicators $j$ amongst two countries $i$ and $k$. Since countries have different amounts of available data points, we average by multiplying the sum by $1/J$, where $J$ is the total number of sub-indicators taken into account here. Note, this does not necessarily be 375, because we have missing values for many sub-indicators. Second, we sum over $k$ to add together all weighted $x^j_{k,y}$ of each unique pair of countries $i$ and $k$ and compute its average by dividing by $K$.\n",
    "\n",
    "$$ x^{j}_{i,y} = \\frac{1}{K} \\sum_k \\frac{1}{|E_y(c_{i}, c_{k})|} \\cdot x^j_{k,y} $$\n",
    "\n",
    "**Assumptions:** we calculate how similar countries are according to their values for *all* sub-indicators in a given year. We assume that the specific sub-indicators which do not have values in this given year are exactly as similar as the ones which we can calculate a distance for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Let's have a final check before we start our w-kNN algorithm to compute all missing values. We have here negative values which might sound confusing, but bear in mind that we have standardised the data before, i.e. the data distribution has mean 0 and standard deviation 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1901</th>\n",
       "      <th>1902</th>\n",
       "      <th>1903</th>\n",
       "      <th>1904</th>\n",
       "      <th>1905</th>\n",
       "      <th>1906</th>\n",
       "      <th>1907</th>\n",
       "      <th>1908</th>\n",
       "      <th>1909</th>\n",
       "      <th>1910</th>\n",
       "      <th>...</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "      <th>2017</th>\n",
       "      <th>2018</th>\n",
       "      <th>2019</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EG.CFT.ACCS.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.641392</td>\n",
       "      <td>0.732207</td>\n",
       "      <td>0.799586</td>\n",
       "      <td>0.877218</td>\n",
       "      <td>0.925555</td>\n",
       "      <td>0.976822</td>\n",
       "      <td>1.011976</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.783718</td>\n",
       "      <td>-0.942031</td>\n",
       "      <td>0.694039</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>0.369409</td>\n",
       "      <td>0.768609</td>\n",
       "      <td>1.190220</td>\n",
       "      <td>1.461546</td>\n",
       "      <td>1.449148</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.RU.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.755535</td>\n",
       "      <td>-1.354031</td>\n",
       "      <td>0.870741</td>\n",
       "      <td>-0.154748</td>\n",
       "      <td>0.206070</td>\n",
       "      <td>0.637647</td>\n",
       "      <td>1.126894</td>\n",
       "      <td>1.450451</td>\n",
       "      <td>1.542347</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.UR.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.621267</td>\n",
       "      <td>1.023083</td>\n",
       "      <td>-0.269228</td>\n",
       "      <td>0.565059</td>\n",
       "      <td>0.822012</td>\n",
       "      <td>0.971654</td>\n",
       "      <td>1.017390</td>\n",
       "      <td>1.023083</td>\n",
       "      <td>0.606209</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FX.OWN.TOTL.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.744066</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.669492</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.413558</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDAN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDLN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDYN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_MISS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Temperature</th>\n",
       "      <td>0.266421</td>\n",
       "      <td>-0.32966</td>\n",
       "      <td>-1.037506</td>\n",
       "      <td>-0.863649</td>\n",
       "      <td>-0.876068</td>\n",
       "      <td>-0.776721</td>\n",
       "      <td>-1.112017</td>\n",
       "      <td>-1.037506</td>\n",
       "      <td>-0.304824</td>\n",
       "      <td>-0.739466</td>\n",
       "      <td>...</td>\n",
       "      <td>3.234407</td>\n",
       "      <td>0.254002</td>\n",
       "      <td>1.458583</td>\n",
       "      <td>0.800410</td>\n",
       "      <td>1.669695</td>\n",
       "      <td>2.004990</td>\n",
       "      <td>1.769041</td>\n",
       "      <td>1.744205</td>\n",
       "      <td>2.986040</td>\n",
       "      <td>1.992572</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>400 rows × 119 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       1901     1902      1903      1904      1905      1906  \\\n",
       "EG.CFT.ACCS.ZS          NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.ZS          NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.RU.ZS       NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.UR.ZS       NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "FX.OWN.TOTL.ZS          NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "...                     ...      ...       ...       ...       ...       ...   \n",
       "VC_DSR_PDAN             NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDLN             NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDYN             NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_MISS             NaN      NaN       NaN       NaN       NaN       NaN   \n",
       "Temperature        0.266421 -0.32966 -1.037506 -0.863649 -0.876068 -0.776721   \n",
       "\n",
       "                       1907      1908      1909      1910  ...      2010  \\\n",
       "EG.CFT.ACCS.ZS          NaN       NaN       NaN       NaN  ...  0.641392   \n",
       "EG.ELC.ACCS.ZS          NaN       NaN       NaN       NaN  ... -0.783718   \n",
       "EG.ELC.ACCS.RU.ZS       NaN       NaN       NaN       NaN  ... -0.755535   \n",
       "EG.ELC.ACCS.UR.ZS       NaN       NaN       NaN       NaN  ... -0.621267   \n",
       "FX.OWN.TOTL.ZS          NaN       NaN       NaN       NaN  ...       NaN   \n",
       "...                     ...       ...       ...       ...  ...       ...   \n",
       "VC_DSR_PDAN             NaN       NaN       NaN       NaN  ...       NaN   \n",
       "VC_DSR_PDLN             NaN       NaN       NaN       NaN  ...       NaN   \n",
       "VC_DSR_PDYN             NaN       NaN       NaN       NaN  ...       NaN   \n",
       "VC_DSR_MISS             NaN       NaN       NaN       NaN  ...       NaN   \n",
       "Temperature       -1.112017 -1.037506 -0.304824 -0.739466  ...  3.234407   \n",
       "\n",
       "                       2011      2012      2013      2014      2015      2016  \\\n",
       "EG.CFT.ACCS.ZS     0.732207  0.799586  0.877218  0.925555  0.976822  1.011976   \n",
       "EG.ELC.ACCS.ZS    -0.942031  0.694039  0.003795  0.369409  0.768609  1.190220   \n",
       "EG.ELC.ACCS.RU.ZS -1.354031  0.870741 -0.154748  0.206070  0.637647  1.126894   \n",
       "EG.ELC.ACCS.UR.ZS  1.023083 -0.269228  0.565059  0.822012  0.971654  1.017390   \n",
       "FX.OWN.TOTL.ZS    -0.744066       NaN       NaN -0.669492       NaN       NaN   \n",
       "...                     ...       ...       ...       ...       ...       ...   \n",
       "VC_DSR_PDAN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDLN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDYN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_MISS             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "Temperature        0.254002  1.458583  0.800410  1.669695  2.004990  1.769041   \n",
       "\n",
       "                       2017      2018      2019  \n",
       "EG.CFT.ACCS.ZS          NaN       NaN       NaN  \n",
       "EG.ELC.ACCS.ZS     1.461546  1.449148       NaN  \n",
       "EG.ELC.ACCS.RU.ZS  1.450451  1.542347       NaN  \n",
       "EG.ELC.ACCS.UR.ZS  1.023083  0.606209       NaN  \n",
       "FX.OWN.TOTL.ZS     1.413558       NaN       NaN  \n",
       "...                     ...       ...       ...  \n",
       "VC_DSR_PDAN             NaN  0.000000       NaN  \n",
       "VC_DSR_PDLN             NaN  0.000000       NaN  \n",
       "VC_DSR_PDYN             NaN  0.000000       NaN  \n",
       "VC_DSR_MISS             NaN       NaN       NaN  \n",
       "Temperature        1.744205  2.986040  1.992572  \n",
       "\n",
       "[400 rows x 119 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_all_std['Iraq']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHECKPOINT\n",
    "dict_e = pickle.load(open('utils/data/distances_unstd.pkl', 'rb'))\n",
    "dict_E = pickle.load(open('utils/data/distances_std.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check\n",
    "print('unstandardised distance e:', dict_e['2000', 'Afghanistan', 'Colombia'])\n",
    "print('standardised distance E:  ', dict_E['2000', 'Afghanistan', 'Colombia'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Calculating the Euclidean distance\n",
    "\n",
    "We can calculate the standardised distance $E_y(c_{i}, c_{k})$ after having prepared everything. We do this for each unique pair of two *countries* in each year. In other words, we do not want to calculate $E_y(c_{i}, c_{k})$ for $i = k$ and $E_y(c_{i}, c_{k}) = E_y(c_{k}, c_{i})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The python package <code>itertools</code> can help us generating the unique pairs of countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Afghanistan', 'Albania'),\n",
       " ('Afghanistan', 'Algeria'),\n",
       " ('Afghanistan', 'Angola'),\n",
       " ('Afghanistan', 'Antigua and Barbuda'),\n",
       " ('Afghanistan', 'Argentina'),\n",
       " ('Afghanistan', 'Armenia'),\n",
       " ('Afghanistan', 'Australia'),\n",
       " ('Afghanistan', 'Austria'),\n",
       " ('Afghanistan', 'Azerbaijan'),\n",
       " ('Afghanistan', 'Bahamas, The'),\n",
       " ('Afghanistan', 'Bahrain'),\n",
       " ('Afghanistan', 'Bangladesh'),\n",
       " ('Afghanistan', 'Barbados'),\n",
       " ('Afghanistan', 'Belarus'),\n",
       " ('Afghanistan', 'Belgium'),\n",
       " ('Afghanistan', 'Belize'),\n",
       " ('Afghanistan', 'Benin'),\n",
       " ('Afghanistan', 'Bhutan'),\n",
       " ('Afghanistan', 'Bolivia'),\n",
       " ('Afghanistan', 'Bosnia and Herzegovina'),\n",
       " ('Afghanistan', 'Botswana'),\n",
       " ('Afghanistan', 'Brazil'),\n",
       " ('Afghanistan', 'Brunei Darussalam'),\n",
       " ('Afghanistan', 'Bulgaria'),\n",
       " ('Afghanistan', 'Burkina Faso'),\n",
       " ('Afghanistan', 'Burundi'),\n",
       " ('Afghanistan', 'Cambodia'),\n",
       " ('Afghanistan', 'Cameroon'),\n",
       " ('Afghanistan', 'Canada'),\n",
       " ('Afghanistan', 'Central African Republic'),\n",
       " ('Afghanistan', 'Chad'),\n",
       " ('Afghanistan', 'Chile'),\n",
       " ('Afghanistan', 'China'),\n",
       " ('Afghanistan', 'Colombia'),\n",
       " ('Afghanistan', 'Comoros'),\n",
       " ('Afghanistan', 'Congo, Dem. Rep.'),\n",
       " ('Afghanistan', 'Congo, Rep.'),\n",
       " ('Afghanistan', 'Costa Rica'),\n",
       " ('Afghanistan', \"Cote d'Ivoire\"),\n",
       " ('Afghanistan', 'Croatia'),\n",
       " ('Afghanistan', 'Cuba'),\n",
       " ('Afghanistan', 'Cyprus'),\n",
       " ('Afghanistan', 'Czech Republic'),\n",
       " ('Afghanistan', 'Denmark'),\n",
       " ('Afghanistan', 'Djibouti'),\n",
       " ('Afghanistan', 'Dominica'),\n",
       " ('Afghanistan', 'Dominican Republic'),\n",
       " ('Afghanistan', 'Ecuador'),\n",
       " ('Afghanistan', 'Egypt, Arab Rep.'),\n",
       " ('Afghanistan', 'El Salvador'),\n",
       " ('Afghanistan', 'Equatorial Guinea'),\n",
       " ('Afghanistan', 'Eritrea'),\n",
       " ('Afghanistan', 'Estonia'),\n",
       " ('Afghanistan', 'Ethiopia'),\n",
       " ('Afghanistan', 'Fiji'),\n",
       " ('Afghanistan', 'Finland'),\n",
       " ('Afghanistan', 'France'),\n",
       " ('Afghanistan', 'Gabon'),\n",
       " ('Afghanistan', 'Gambia, The'),\n",
       " ('Afghanistan', 'Georgia'),\n",
       " ('Afghanistan', 'Germany'),\n",
       " ('Afghanistan', 'Ghana'),\n",
       " ('Afghanistan', 'Greece'),\n",
       " ('Afghanistan', 'Grenada'),\n",
       " ('Afghanistan', 'Guatemala'),\n",
       " ('Afghanistan', 'Guinea'),\n",
       " ('Afghanistan', 'Guinea-Bissau'),\n",
       " ('Afghanistan', 'Guyana'),\n",
       " ('Afghanistan', 'Haiti'),\n",
       " ('Afghanistan', 'Honduras'),\n",
       " ('Afghanistan', 'Hungary'),\n",
       " ('Afghanistan', 'Iceland'),\n",
       " ('Afghanistan', 'India'),\n",
       " ('Afghanistan', 'Indonesia'),\n",
       " ('Afghanistan', 'Iran, Islamic Rep.'),\n",
       " ('Afghanistan', 'Iraq'),\n",
       " ('Afghanistan', 'Ireland'),\n",
       " ('Afghanistan', 'Israel'),\n",
       " ('Afghanistan', 'Italy'),\n",
       " ('Afghanistan', 'Jamaica'),\n",
       " ('Afghanistan', 'Japan'),\n",
       " ('Afghanistan', 'Jordan'),\n",
       " ('Afghanistan', 'Kazakhstan'),\n",
       " ('Afghanistan', 'Kenya'),\n",
       " ('Afghanistan', 'Kiribati'),\n",
       " ('Afghanistan', 'Korea, Rep.'),\n",
       " ('Afghanistan', 'Kuwait'),\n",
       " ('Afghanistan', 'Kyrgyz Republic'),\n",
       " ('Afghanistan', 'Lao PDR'),\n",
       " ('Afghanistan', 'Latvia'),\n",
       " ('Afghanistan', 'Lebanon'),\n",
       " ('Afghanistan', 'Lesotho'),\n",
       " ('Afghanistan', 'Liberia'),\n",
       " ('Afghanistan', 'Libya'),\n",
       " ('Afghanistan', 'Lithuania'),\n",
       " ('Afghanistan', 'Luxembourg'),\n",
       " ('Afghanistan', 'Madagascar'),\n",
       " ('Afghanistan', 'Malawi'),\n",
       " ('Afghanistan', 'Malaysia'),\n",
       " ('Afghanistan', 'Maldives'),\n",
       " ('Afghanistan', 'Mali'),\n",
       " ('Afghanistan', 'Malta'),\n",
       " ('Afghanistan', 'Mauritania'),\n",
       " ('Afghanistan', 'Mauritius'),\n",
       " ('Afghanistan', 'Mexico'),\n",
       " ('Afghanistan', 'Moldova'),\n",
       " ('Afghanistan', 'Mongolia'),\n",
       " ('Afghanistan', 'Montenegro'),\n",
       " ('Afghanistan', 'Morocco'),\n",
       " ('Afghanistan', 'Mozambique'),\n",
       " ('Afghanistan', 'Myanmar'),\n",
       " ('Afghanistan', 'Namibia'),\n",
       " ('Afghanistan', 'Nepal'),\n",
       " ('Afghanistan', 'Netherlands'),\n",
       " ('Afghanistan', 'New Zealand'),\n",
       " ('Afghanistan', 'Nicaragua'),\n",
       " ('Afghanistan', 'Niger'),\n",
       " ('Afghanistan', 'Nigeria'),\n",
       " ('Afghanistan', 'Norway'),\n",
       " ('Afghanistan', 'Oman'),\n",
       " ('Afghanistan', 'Pakistan'),\n",
       " ('Afghanistan', 'Palau'),\n",
       " ('Afghanistan', 'Panama'),\n",
       " ('Afghanistan', 'Papua New Guinea'),\n",
       " ('Afghanistan', 'Paraguay'),\n",
       " ('Afghanistan', 'Peru'),\n",
       " ('Afghanistan', 'Philippines'),\n",
       " ('Afghanistan', 'Poland'),\n",
       " ('Afghanistan', 'Portugal'),\n",
       " ('Afghanistan', 'Puerto Rico'),\n",
       " ('Afghanistan', 'Qatar'),\n",
       " ('Afghanistan', 'Romania'),\n",
       " ('Afghanistan', 'Russian Federation'),\n",
       " ('Afghanistan', 'Rwanda'),\n",
       " ('Afghanistan', 'Samoa'),\n",
       " ('Afghanistan', 'Sao Tome and Principe'),\n",
       " ('Afghanistan', 'Saudi Arabia'),\n",
       " ('Afghanistan', 'Senegal'),\n",
       " ('Afghanistan', 'Serbia'),\n",
       " ('Afghanistan', 'Seychelles'),\n",
       " ('Afghanistan', 'Sierra Leone'),\n",
       " ('Afghanistan', 'Singapore'),\n",
       " ('Afghanistan', 'Slovak Republic'),\n",
       " ('Afghanistan', 'Slovenia'),\n",
       " ('Afghanistan', 'Solomon Islands'),\n",
       " ('Afghanistan', 'Somalia'),\n",
       " ('Afghanistan', 'South Africa'),\n",
       " ('Afghanistan', 'South Sudan'),\n",
       " ('Afghanistan', 'Spain'),\n",
       " ('Afghanistan', 'Sri Lanka'),\n",
       " ('Afghanistan', 'Sudan'),\n",
       " ('Afghanistan', 'Suriname'),\n",
       " ('Afghanistan', 'Sweden'),\n",
       " ('Afghanistan', 'Switzerland'),\n",
       " ('Afghanistan', 'Syrian Arab Republic'),\n",
       " ('Afghanistan', 'Tajikistan'),\n",
       " ('Afghanistan', 'Tanzania'),\n",
       " ('Afghanistan', 'Thailand'),\n",
       " ('Afghanistan', 'Timor-Leste'),\n",
       " ('Afghanistan', 'Togo'),\n",
       " ('Afghanistan', 'Tonga'),\n",
       " ('Afghanistan', 'Trinidad and Tobago'),\n",
       " ('Afghanistan', 'Tunisia'),\n",
       " ('Afghanistan', 'Turkey'),\n",
       " ('Afghanistan', 'Turkmenistan'),\n",
       " ('Afghanistan', 'Tuvalu'),\n",
       " ('Afghanistan', 'Uganda'),\n",
       " ('Afghanistan', 'Ukraine'),\n",
       " ('Afghanistan', 'United Arab Emirates'),\n",
       " ('Afghanistan', 'United Kingdom'),\n",
       " ('Afghanistan', 'United States'),\n",
       " ('Afghanistan', 'Uruguay'),\n",
       " ('Afghanistan', 'Uzbekistan'),\n",
       " ('Afghanistan', 'Vanuatu'),\n",
       " ('Afghanistan', 'Venezuela, RB'),\n",
       " ('Afghanistan', 'Vietnam'),\n",
       " ('Afghanistan', 'Yemen, Rep.'),\n",
       " ('Afghanistan', 'Zambia'),\n",
       " ('Afghanistan', 'Zimbabwe'),\n",
       " ('Albania', 'Algeria'),\n",
       " ('Albania', 'Angola'),\n",
       " ('Albania', 'Antigua and Barbuda'),\n",
       " ('Albania', 'Argentina'),\n",
       " ('Albania', 'Armenia'),\n",
       " ('Albania', 'Australia'),\n",
       " ('Albania', 'Austria'),\n",
       " ('Albania', 'Azerbaijan'),\n",
       " ('Albania', 'Bahamas, The'),\n",
       " ('Albania', 'Bahrain'),\n",
       " ('Albania', 'Bangladesh'),\n",
       " ('Albania', 'Barbados'),\n",
       " ('Albania', 'Belarus'),\n",
       " ('Albania', 'Belgium'),\n",
       " ('Albania', 'Belize'),\n",
       " ('Albania', 'Benin'),\n",
       " ('Albania', 'Bhutan'),\n",
       " ('Albania', 'Bolivia'),\n",
       " ('Albania', 'Bosnia and Herzegovina'),\n",
       " ('Albania', 'Botswana'),\n",
       " ('Albania', 'Brazil'),\n",
       " ('Albania', 'Brunei Darussalam'),\n",
       " ('Albania', 'Bulgaria'),\n",
       " ('Albania', 'Burkina Faso'),\n",
       " ('Albania', 'Burundi'),\n",
       " ('Albania', 'Cambodia'),\n",
       " ('Albania', 'Cameroon'),\n",
       " ('Albania', 'Canada'),\n",
       " ('Albania', 'Central African Republic'),\n",
       " ('Albania', 'Chad'),\n",
       " ('Albania', 'Chile'),\n",
       " ('Albania', 'China'),\n",
       " ('Albania', 'Colombia'),\n",
       " ('Albania', 'Comoros'),\n",
       " ('Albania', 'Congo, Dem. Rep.'),\n",
       " ('Albania', 'Congo, Rep.'),\n",
       " ('Albania', 'Costa Rica'),\n",
       " ('Albania', \"Cote d'Ivoire\"),\n",
       " ('Albania', 'Croatia'),\n",
       " ('Albania', 'Cuba'),\n",
       " ('Albania', 'Cyprus'),\n",
       " ('Albania', 'Czech Republic'),\n",
       " ('Albania', 'Denmark'),\n",
       " ('Albania', 'Djibouti'),\n",
       " ('Albania', 'Dominica'),\n",
       " ('Albania', 'Dominican Republic'),\n",
       " ('Albania', 'Ecuador'),\n",
       " ('Albania', 'Egypt, Arab Rep.'),\n",
       " ('Albania', 'El Salvador'),\n",
       " ('Albania', 'Equatorial Guinea'),\n",
       " ('Albania', 'Eritrea'),\n",
       " ('Albania', 'Estonia'),\n",
       " ('Albania', 'Ethiopia'),\n",
       " ('Albania', 'Fiji'),\n",
       " ('Albania', 'Finland'),\n",
       " ('Albania', 'France'),\n",
       " ('Albania', 'Gabon'),\n",
       " ('Albania', 'Gambia, The'),\n",
       " ('Albania', 'Georgia'),\n",
       " ('Albania', 'Germany'),\n",
       " ('Albania', 'Ghana'),\n",
       " ('Albania', 'Greece'),\n",
       " ('Albania', 'Grenada'),\n",
       " ('Albania', 'Guatemala'),\n",
       " ('Albania', 'Guinea'),\n",
       " ('Albania', 'Guinea-Bissau'),\n",
       " ('Albania', 'Guyana'),\n",
       " ('Albania', 'Haiti'),\n",
       " ('Albania', 'Honduras'),\n",
       " ('Albania', 'Hungary'),\n",
       " ('Albania', 'Iceland'),\n",
       " ('Albania', 'India'),\n",
       " ('Albania', 'Indonesia'),\n",
       " ('Albania', 'Iran, Islamic Rep.'),\n",
       " ('Albania', 'Iraq'),\n",
       " ('Albania', 'Ireland'),\n",
       " ('Albania', 'Israel'),\n",
       " ('Albania', 'Italy'),\n",
       " ('Albania', 'Jamaica'),\n",
       " ('Albania', 'Japan'),\n",
       " ('Albania', 'Jordan'),\n",
       " ('Albania', 'Kazakhstan'),\n",
       " ('Albania', 'Kenya'),\n",
       " ('Albania', 'Kiribati'),\n",
       " ('Albania', 'Korea, Rep.'),\n",
       " ('Albania', 'Kuwait'),\n",
       " ('Albania', 'Kyrgyz Republic'),\n",
       " ('Albania', 'Lao PDR'),\n",
       " ('Albania', 'Latvia'),\n",
       " ('Albania', 'Lebanon'),\n",
       " ('Albania', 'Lesotho'),\n",
       " ('Albania', 'Liberia'),\n",
       " ('Albania', 'Libya'),\n",
       " ('Albania', 'Lithuania'),\n",
       " ('Albania', 'Luxembourg'),\n",
       " ('Albania', 'Madagascar'),\n",
       " ('Albania', 'Malawi'),\n",
       " ('Albania', 'Malaysia'),\n",
       " ('Albania', 'Maldives'),\n",
       " ('Albania', 'Mali'),\n",
       " ('Albania', 'Malta'),\n",
       " ('Albania', 'Mauritania'),\n",
       " ('Albania', 'Mauritius'),\n",
       " ('Albania', 'Mexico'),\n",
       " ('Albania', 'Moldova'),\n",
       " ('Albania', 'Mongolia'),\n",
       " ('Albania', 'Montenegro'),\n",
       " ('Albania', 'Morocco'),\n",
       " ('Albania', 'Mozambique'),\n",
       " ('Albania', 'Myanmar'),\n",
       " ('Albania', 'Namibia'),\n",
       " ('Albania', 'Nepal'),\n",
       " ('Albania', 'Netherlands'),\n",
       " ('Albania', 'New Zealand'),\n",
       " ('Albania', 'Nicaragua'),\n",
       " ('Albania', 'Niger'),\n",
       " ('Albania', 'Nigeria'),\n",
       " ('Albania', 'Norway'),\n",
       " ('Albania', 'Oman'),\n",
       " ('Albania', 'Pakistan'),\n",
       " ('Albania', 'Palau'),\n",
       " ('Albania', 'Panama'),\n",
       " ('Albania', 'Papua New Guinea'),\n",
       " ('Albania', 'Paraguay'),\n",
       " ('Albania', 'Peru'),\n",
       " ('Albania', 'Philippines'),\n",
       " ('Albania', 'Poland'),\n",
       " ('Albania', 'Portugal'),\n",
       " ('Albania', 'Puerto Rico'),\n",
       " ('Albania', 'Qatar'),\n",
       " ('Albania', 'Romania'),\n",
       " ('Albania', 'Russian Federation'),\n",
       " ('Albania', 'Rwanda'),\n",
       " ('Albania', 'Samoa'),\n",
       " ('Albania', 'Sao Tome and Principe'),\n",
       " ('Albania', 'Saudi Arabia'),\n",
       " ('Albania', 'Senegal'),\n",
       " ('Albania', 'Serbia'),\n",
       " ('Albania', 'Seychelles'),\n",
       " ('Albania', 'Sierra Leone'),\n",
       " ('Albania', 'Singapore'),\n",
       " ('Albania', 'Slovak Republic'),\n",
       " ('Albania', 'Slovenia'),\n",
       " ('Albania', 'Solomon Islands'),\n",
       " ('Albania', 'Somalia'),\n",
       " ('Albania', 'South Africa'),\n",
       " ('Albania', 'South Sudan'),\n",
       " ('Albania', 'Spain'),\n",
       " ('Albania', 'Sri Lanka'),\n",
       " ('Albania', 'Sudan'),\n",
       " ('Albania', 'Suriname'),\n",
       " ('Albania', 'Sweden'),\n",
       " ('Albania', 'Switzerland'),\n",
       " ('Albania', 'Syrian Arab Republic'),\n",
       " ('Albania', 'Tajikistan'),\n",
       " ('Albania', 'Tanzania'),\n",
       " ('Albania', 'Thailand'),\n",
       " ('Albania', 'Timor-Leste'),\n",
       " ('Albania', 'Togo'),\n",
       " ('Albania', 'Tonga'),\n",
       " ('Albania', 'Trinidad and Tobago'),\n",
       " ('Albania', 'Tunisia'),\n",
       " ('Albania', 'Turkey'),\n",
       " ('Albania', 'Turkmenistan'),\n",
       " ('Albania', 'Tuvalu'),\n",
       " ('Albania', 'Uganda'),\n",
       " ('Albania', 'Ukraine'),\n",
       " ('Albania', 'United Arab Emirates'),\n",
       " ('Albania', 'United Kingdom'),\n",
       " ('Albania', 'United States'),\n",
       " ('Albania', 'Uruguay'),\n",
       " ('Albania', 'Uzbekistan'),\n",
       " ('Albania', 'Vanuatu'),\n",
       " ('Albania', 'Venezuela, RB'),\n",
       " ('Albania', 'Vietnam'),\n",
       " ('Albania', 'Yemen, Rep.'),\n",
       " ('Albania', 'Zambia'),\n",
       " ('Albania', 'Zimbabwe'),\n",
       " ('Algeria', 'Angola'),\n",
       " ('Algeria', 'Antigua and Barbuda'),\n",
       " ('Algeria', 'Argentina'),\n",
       " ('Algeria', 'Armenia'),\n",
       " ('Algeria', 'Australia'),\n",
       " ('Algeria', 'Austria'),\n",
       " ('Algeria', 'Azerbaijan'),\n",
       " ('Algeria', 'Bahamas, The'),\n",
       " ('Algeria', 'Bahrain'),\n",
       " ('Algeria', 'Bangladesh'),\n",
       " ('Algeria', 'Barbados'),\n",
       " ('Algeria', 'Belarus'),\n",
       " ('Algeria', 'Belgium'),\n",
       " ('Algeria', 'Belize'),\n",
       " ('Algeria', 'Benin'),\n",
       " ('Algeria', 'Bhutan'),\n",
       " ('Algeria', 'Bolivia'),\n",
       " ('Algeria', 'Bosnia and Herzegovina'),\n",
       " ('Algeria', 'Botswana'),\n",
       " ('Algeria', 'Brazil'),\n",
       " ('Algeria', 'Brunei Darussalam'),\n",
       " ('Algeria', 'Bulgaria'),\n",
       " ('Algeria', 'Burkina Faso'),\n",
       " ('Algeria', 'Burundi'),\n",
       " ('Algeria', 'Cambodia'),\n",
       " ('Algeria', 'Cameroon'),\n",
       " ('Algeria', 'Canada'),\n",
       " ('Algeria', 'Central African Republic'),\n",
       " ('Algeria', 'Chad'),\n",
       " ('Algeria', 'Chile'),\n",
       " ('Algeria', 'China'),\n",
       " ('Algeria', 'Colombia'),\n",
       " ('Algeria', 'Comoros'),\n",
       " ('Algeria', 'Congo, Dem. Rep.'),\n",
       " ('Algeria', 'Congo, Rep.'),\n",
       " ('Algeria', 'Costa Rica'),\n",
       " ('Algeria', \"Cote d'Ivoire\"),\n",
       " ('Algeria', 'Croatia'),\n",
       " ('Algeria', 'Cuba'),\n",
       " ('Algeria', 'Cyprus'),\n",
       " ('Algeria', 'Czech Republic'),\n",
       " ('Algeria', 'Denmark'),\n",
       " ('Algeria', 'Djibouti'),\n",
       " ('Algeria', 'Dominica'),\n",
       " ('Algeria', 'Dominican Republic'),\n",
       " ('Algeria', 'Ecuador'),\n",
       " ('Algeria', 'Egypt, Arab Rep.'),\n",
       " ('Algeria', 'El Salvador'),\n",
       " ('Algeria', 'Equatorial Guinea'),\n",
       " ('Algeria', 'Eritrea'),\n",
       " ('Algeria', 'Estonia'),\n",
       " ('Algeria', 'Ethiopia'),\n",
       " ('Algeria', 'Fiji'),\n",
       " ('Algeria', 'Finland'),\n",
       " ('Algeria', 'France'),\n",
       " ('Algeria', 'Gabon'),\n",
       " ('Algeria', 'Gambia, The'),\n",
       " ('Algeria', 'Georgia'),\n",
       " ('Algeria', 'Germany'),\n",
       " ('Algeria', 'Ghana'),\n",
       " ('Algeria', 'Greece'),\n",
       " ('Algeria', 'Grenada'),\n",
       " ('Algeria', 'Guatemala'),\n",
       " ('Algeria', 'Guinea'),\n",
       " ('Algeria', 'Guinea-Bissau'),\n",
       " ('Algeria', 'Guyana'),\n",
       " ('Algeria', 'Haiti'),\n",
       " ('Algeria', 'Honduras'),\n",
       " ('Algeria', 'Hungary'),\n",
       " ('Algeria', 'Iceland'),\n",
       " ('Algeria', 'India'),\n",
       " ('Algeria', 'Indonesia'),\n",
       " ('Algeria', 'Iran, Islamic Rep.'),\n",
       " ('Algeria', 'Iraq'),\n",
       " ('Algeria', 'Ireland'),\n",
       " ('Algeria', 'Israel'),\n",
       " ('Algeria', 'Italy'),\n",
       " ('Algeria', 'Jamaica'),\n",
       " ('Algeria', 'Japan'),\n",
       " ('Algeria', 'Jordan'),\n",
       " ('Algeria', 'Kazakhstan'),\n",
       " ('Algeria', 'Kenya'),\n",
       " ('Algeria', 'Kiribati'),\n",
       " ('Algeria', 'Korea, Rep.'),\n",
       " ('Algeria', 'Kuwait'),\n",
       " ('Algeria', 'Kyrgyz Republic'),\n",
       " ('Algeria', 'Lao PDR'),\n",
       " ('Algeria', 'Latvia'),\n",
       " ('Algeria', 'Lebanon'),\n",
       " ('Algeria', 'Lesotho'),\n",
       " ('Algeria', 'Liberia'),\n",
       " ('Algeria', 'Libya'),\n",
       " ('Algeria', 'Lithuania'),\n",
       " ('Algeria', 'Luxembourg'),\n",
       " ('Algeria', 'Madagascar'),\n",
       " ('Algeria', 'Malawi'),\n",
       " ('Algeria', 'Malaysia'),\n",
       " ('Algeria', 'Maldives'),\n",
       " ('Algeria', 'Mali'),\n",
       " ('Algeria', 'Malta'),\n",
       " ('Algeria', 'Mauritania'),\n",
       " ('Algeria', 'Mauritius'),\n",
       " ('Algeria', 'Mexico'),\n",
       " ('Algeria', 'Moldova'),\n",
       " ('Algeria', 'Mongolia'),\n",
       " ('Algeria', 'Montenegro'),\n",
       " ('Algeria', 'Morocco'),\n",
       " ('Algeria', 'Mozambique'),\n",
       " ('Algeria', 'Myanmar'),\n",
       " ('Algeria', 'Namibia'),\n",
       " ('Algeria', 'Nepal'),\n",
       " ('Algeria', 'Netherlands'),\n",
       " ('Algeria', 'New Zealand'),\n",
       " ('Algeria', 'Nicaragua'),\n",
       " ('Algeria', 'Niger'),\n",
       " ('Algeria', 'Nigeria'),\n",
       " ('Algeria', 'Norway'),\n",
       " ('Algeria', 'Oman'),\n",
       " ('Algeria', 'Pakistan'),\n",
       " ('Algeria', 'Palau'),\n",
       " ('Algeria', 'Panama'),\n",
       " ('Algeria', 'Papua New Guinea'),\n",
       " ('Algeria', 'Paraguay'),\n",
       " ('Algeria', 'Peru'),\n",
       " ('Algeria', 'Philippines'),\n",
       " ('Algeria', 'Poland'),\n",
       " ('Algeria', 'Portugal'),\n",
       " ('Algeria', 'Puerto Rico'),\n",
       " ('Algeria', 'Qatar'),\n",
       " ('Algeria', 'Romania'),\n",
       " ('Algeria', 'Russian Federation'),\n",
       " ('Algeria', 'Rwanda'),\n",
       " ('Algeria', 'Samoa'),\n",
       " ('Algeria', 'Sao Tome and Principe'),\n",
       " ('Algeria', 'Saudi Arabia'),\n",
       " ('Algeria', 'Senegal'),\n",
       " ('Algeria', 'Serbia'),\n",
       " ('Algeria', 'Seychelles'),\n",
       " ('Algeria', 'Sierra Leone'),\n",
       " ('Algeria', 'Singapore'),\n",
       " ('Algeria', 'Slovak Republic'),\n",
       " ('Algeria', 'Slovenia'),\n",
       " ('Algeria', 'Solomon Islands'),\n",
       " ('Algeria', 'Somalia'),\n",
       " ('Algeria', 'South Africa'),\n",
       " ('Algeria', 'South Sudan'),\n",
       " ('Algeria', 'Spain'),\n",
       " ('Algeria', 'Sri Lanka'),\n",
       " ('Algeria', 'Sudan'),\n",
       " ('Algeria', 'Suriname'),\n",
       " ('Algeria', 'Sweden'),\n",
       " ('Algeria', 'Switzerland'),\n",
       " ('Algeria', 'Syrian Arab Republic'),\n",
       " ('Algeria', 'Tajikistan'),\n",
       " ('Algeria', 'Tanzania'),\n",
       " ('Algeria', 'Thailand'),\n",
       " ('Algeria', 'Timor-Leste'),\n",
       " ('Algeria', 'Togo'),\n",
       " ('Algeria', 'Tonga'),\n",
       " ('Algeria', 'Trinidad and Tobago'),\n",
       " ('Algeria', 'Tunisia'),\n",
       " ('Algeria', 'Turkey'),\n",
       " ('Algeria', 'Turkmenistan'),\n",
       " ('Algeria', 'Tuvalu'),\n",
       " ('Algeria', 'Uganda'),\n",
       " ('Algeria', 'Ukraine'),\n",
       " ('Algeria', 'United Arab Emirates'),\n",
       " ('Algeria', 'United Kingdom'),\n",
       " ('Algeria', 'United States'),\n",
       " ('Algeria', 'Uruguay'),\n",
       " ('Algeria', 'Uzbekistan'),\n",
       " ('Algeria', 'Vanuatu'),\n",
       " ('Algeria', 'Venezuela, RB'),\n",
       " ('Algeria', 'Vietnam'),\n",
       " ('Algeria', 'Yemen, Rep.'),\n",
       " ('Algeria', 'Zambia'),\n",
       " ('Algeria', 'Zimbabwe'),\n",
       " ('Angola', 'Antigua and Barbuda'),\n",
       " ('Angola', 'Argentina'),\n",
       " ('Angola', 'Armenia'),\n",
       " ('Angola', 'Australia'),\n",
       " ('Angola', 'Austria'),\n",
       " ('Angola', 'Azerbaijan'),\n",
       " ('Angola', 'Bahamas, The'),\n",
       " ('Angola', 'Bahrain'),\n",
       " ('Angola', 'Bangladesh'),\n",
       " ('Angola', 'Barbados'),\n",
       " ('Angola', 'Belarus'),\n",
       " ('Angola', 'Belgium'),\n",
       " ('Angola', 'Belize'),\n",
       " ('Angola', 'Benin'),\n",
       " ('Angola', 'Bhutan'),\n",
       " ('Angola', 'Bolivia'),\n",
       " ('Angola', 'Bosnia and Herzegovina'),\n",
       " ('Angola', 'Botswana'),\n",
       " ('Angola', 'Brazil'),\n",
       " ('Angola', 'Brunei Darussalam'),\n",
       " ('Angola', 'Bulgaria'),\n",
       " ('Angola', 'Burkina Faso'),\n",
       " ('Angola', 'Burundi'),\n",
       " ('Angola', 'Cambodia'),\n",
       " ('Angola', 'Cameroon'),\n",
       " ('Angola', 'Canada'),\n",
       " ('Angola', 'Central African Republic'),\n",
       " ('Angola', 'Chad'),\n",
       " ('Angola', 'Chile'),\n",
       " ('Angola', 'China'),\n",
       " ('Angola', 'Colombia'),\n",
       " ('Angola', 'Comoros'),\n",
       " ('Angola', 'Congo, Dem. Rep.'),\n",
       " ('Angola', 'Congo, Rep.'),\n",
       " ('Angola', 'Costa Rica'),\n",
       " ('Angola', \"Cote d'Ivoire\"),\n",
       " ('Angola', 'Croatia'),\n",
       " ('Angola', 'Cuba'),\n",
       " ('Angola', 'Cyprus'),\n",
       " ('Angola', 'Czech Republic'),\n",
       " ('Angola', 'Denmark'),\n",
       " ('Angola', 'Djibouti'),\n",
       " ('Angola', 'Dominica'),\n",
       " ('Angola', 'Dominican Republic'),\n",
       " ('Angola', 'Ecuador'),\n",
       " ('Angola', 'Egypt, Arab Rep.'),\n",
       " ('Angola', 'El Salvador'),\n",
       " ('Angola', 'Equatorial Guinea'),\n",
       " ('Angola', 'Eritrea'),\n",
       " ('Angola', 'Estonia'),\n",
       " ('Angola', 'Ethiopia'),\n",
       " ('Angola', 'Fiji'),\n",
       " ('Angola', 'Finland'),\n",
       " ('Angola', 'France'),\n",
       " ('Angola', 'Gabon'),\n",
       " ('Angola', 'Gambia, The'),\n",
       " ('Angola', 'Georgia'),\n",
       " ('Angola', 'Germany'),\n",
       " ('Angola', 'Ghana'),\n",
       " ('Angola', 'Greece'),\n",
       " ('Angola', 'Grenada'),\n",
       " ('Angola', 'Guatemala'),\n",
       " ('Angola', 'Guinea'),\n",
       " ('Angola', 'Guinea-Bissau'),\n",
       " ('Angola', 'Guyana'),\n",
       " ('Angola', 'Haiti'),\n",
       " ('Angola', 'Honduras'),\n",
       " ('Angola', 'Hungary'),\n",
       " ('Angola', 'Iceland'),\n",
       " ('Angola', 'India'),\n",
       " ('Angola', 'Indonesia'),\n",
       " ('Angola', 'Iran, Islamic Rep.'),\n",
       " ('Angola', 'Iraq'),\n",
       " ('Angola', 'Ireland'),\n",
       " ('Angola', 'Israel'),\n",
       " ('Angola', 'Italy'),\n",
       " ('Angola', 'Jamaica'),\n",
       " ('Angola', 'Japan'),\n",
       " ('Angola', 'Jordan'),\n",
       " ('Angola', 'Kazakhstan'),\n",
       " ('Angola', 'Kenya'),\n",
       " ('Angola', 'Kiribati'),\n",
       " ('Angola', 'Korea, Rep.'),\n",
       " ('Angola', 'Kuwait'),\n",
       " ('Angola', 'Kyrgyz Republic'),\n",
       " ('Angola', 'Lao PDR'),\n",
       " ('Angola', 'Latvia'),\n",
       " ('Angola', 'Lebanon'),\n",
       " ('Angola', 'Lesotho'),\n",
       " ('Angola', 'Liberia'),\n",
       " ('Angola', 'Libya'),\n",
       " ('Angola', 'Lithuania'),\n",
       " ('Angola', 'Luxembourg'),\n",
       " ('Angola', 'Madagascar'),\n",
       " ('Angola', 'Malawi'),\n",
       " ('Angola', 'Malaysia'),\n",
       " ('Angola', 'Maldives'),\n",
       " ('Angola', 'Mali'),\n",
       " ('Angola', 'Malta'),\n",
       " ('Angola', 'Mauritania'),\n",
       " ('Angola', 'Mauritius'),\n",
       " ('Angola', 'Mexico'),\n",
       " ('Angola', 'Moldova'),\n",
       " ('Angola', 'Mongolia'),\n",
       " ('Angola', 'Montenegro'),\n",
       " ('Angola', 'Morocco'),\n",
       " ('Angola', 'Mozambique'),\n",
       " ('Angola', 'Myanmar'),\n",
       " ('Angola', 'Namibia'),\n",
       " ('Angola', 'Nepal'),\n",
       " ('Angola', 'Netherlands'),\n",
       " ('Angola', 'New Zealand'),\n",
       " ('Angola', 'Nicaragua'),\n",
       " ('Angola', 'Niger'),\n",
       " ('Angola', 'Nigeria'),\n",
       " ('Angola', 'Norway'),\n",
       " ('Angola', 'Oman'),\n",
       " ('Angola', 'Pakistan'),\n",
       " ('Angola', 'Palau'),\n",
       " ('Angola', 'Panama'),\n",
       " ('Angola', 'Papua New Guinea'),\n",
       " ('Angola', 'Paraguay'),\n",
       " ('Angola', 'Peru'),\n",
       " ('Angola', 'Philippines'),\n",
       " ('Angola', 'Poland'),\n",
       " ('Angola', 'Portugal'),\n",
       " ('Angola', 'Puerto Rico'),\n",
       " ('Angola', 'Qatar'),\n",
       " ('Angola', 'Romania'),\n",
       " ('Angola', 'Russian Federation'),\n",
       " ('Angola', 'Rwanda'),\n",
       " ('Angola', 'Samoa'),\n",
       " ('Angola', 'Sao Tome and Principe'),\n",
       " ('Angola', 'Saudi Arabia'),\n",
       " ('Angola', 'Senegal'),\n",
       " ('Angola', 'Serbia'),\n",
       " ('Angola', 'Seychelles'),\n",
       " ('Angola', 'Sierra Leone'),\n",
       " ('Angola', 'Singapore'),\n",
       " ('Angola', 'Slovak Republic'),\n",
       " ('Angola', 'Slovenia'),\n",
       " ('Angola', 'Solomon Islands'),\n",
       " ('Angola', 'Somalia'),\n",
       " ('Angola', 'South Africa'),\n",
       " ('Angola', 'South Sudan'),\n",
       " ('Angola', 'Spain'),\n",
       " ('Angola', 'Sri Lanka'),\n",
       " ('Angola', 'Sudan'),\n",
       " ('Angola', 'Suriname'),\n",
       " ('Angola', 'Sweden'),\n",
       " ('Angola', 'Switzerland'),\n",
       " ('Angola', 'Syrian Arab Republic'),\n",
       " ('Angola', 'Tajikistan'),\n",
       " ('Angola', 'Tanzania'),\n",
       " ('Angola', 'Thailand'),\n",
       " ('Angola', 'Timor-Leste'),\n",
       " ('Angola', 'Togo'),\n",
       " ('Angola', 'Tonga'),\n",
       " ('Angola', 'Trinidad and Tobago'),\n",
       " ('Angola', 'Tunisia'),\n",
       " ('Angola', 'Turkey'),\n",
       " ('Angola', 'Turkmenistan'),\n",
       " ('Angola', 'Tuvalu'),\n",
       " ('Angola', 'Uganda'),\n",
       " ('Angola', 'Ukraine'),\n",
       " ('Angola', 'United Arab Emirates'),\n",
       " ('Angola', 'United Kingdom'),\n",
       " ('Angola', 'United States'),\n",
       " ('Angola', 'Uruguay'),\n",
       " ('Angola', 'Uzbekistan'),\n",
       " ('Angola', 'Vanuatu'),\n",
       " ('Angola', 'Venezuela, RB'),\n",
       " ('Angola', 'Vietnam'),\n",
       " ('Angola', 'Yemen, Rep.'),\n",
       " ('Angola', 'Zambia'),\n",
       " ('Angola', 'Zimbabwe'),\n",
       " ('Antigua and Barbuda', 'Argentina'),\n",
       " ('Antigua and Barbuda', 'Armenia'),\n",
       " ('Antigua and Barbuda', 'Australia'),\n",
       " ('Antigua and Barbuda', 'Austria'),\n",
       " ('Antigua and Barbuda', 'Azerbaijan'),\n",
       " ('Antigua and Barbuda', 'Bahamas, The'),\n",
       " ('Antigua and Barbuda', 'Bahrain'),\n",
       " ('Antigua and Barbuda', 'Bangladesh'),\n",
       " ('Antigua and Barbuda', 'Barbados'),\n",
       " ('Antigua and Barbuda', 'Belarus'),\n",
       " ('Antigua and Barbuda', 'Belgium'),\n",
       " ('Antigua and Barbuda', 'Belize'),\n",
       " ('Antigua and Barbuda', 'Benin'),\n",
       " ('Antigua and Barbuda', 'Bhutan'),\n",
       " ('Antigua and Barbuda', 'Bolivia'),\n",
       " ('Antigua and Barbuda', 'Bosnia and Herzegovina'),\n",
       " ('Antigua and Barbuda', 'Botswana'),\n",
       " ('Antigua and Barbuda', 'Brazil'),\n",
       " ('Antigua and Barbuda', 'Brunei Darussalam'),\n",
       " ('Antigua and Barbuda', 'Bulgaria'),\n",
       " ('Antigua and Barbuda', 'Burkina Faso'),\n",
       " ('Antigua and Barbuda', 'Burundi'),\n",
       " ('Antigua and Barbuda', 'Cambodia'),\n",
       " ('Antigua and Barbuda', 'Cameroon'),\n",
       " ('Antigua and Barbuda', 'Canada'),\n",
       " ('Antigua and Barbuda', 'Central African Republic'),\n",
       " ('Antigua and Barbuda', 'Chad'),\n",
       " ('Antigua and Barbuda', 'Chile'),\n",
       " ('Antigua and Barbuda', 'China'),\n",
       " ('Antigua and Barbuda', 'Colombia'),\n",
       " ('Antigua and Barbuda', 'Comoros'),\n",
       " ('Antigua and Barbuda', 'Congo, Dem. Rep.'),\n",
       " ('Antigua and Barbuda', 'Congo, Rep.'),\n",
       " ('Antigua and Barbuda', 'Costa Rica'),\n",
       " ('Antigua and Barbuda', \"Cote d'Ivoire\"),\n",
       " ('Antigua and Barbuda', 'Croatia'),\n",
       " ('Antigua and Barbuda', 'Cuba'),\n",
       " ('Antigua and Barbuda', 'Cyprus'),\n",
       " ('Antigua and Barbuda', 'Czech Republic'),\n",
       " ('Antigua and Barbuda', 'Denmark'),\n",
       " ('Antigua and Barbuda', 'Djibouti'),\n",
       " ('Antigua and Barbuda', 'Dominica'),\n",
       " ('Antigua and Barbuda', 'Dominican Republic'),\n",
       " ('Antigua and Barbuda', 'Ecuador'),\n",
       " ('Antigua and Barbuda', 'Egypt, Arab Rep.'),\n",
       " ('Antigua and Barbuda', 'El Salvador'),\n",
       " ('Antigua and Barbuda', 'Equatorial Guinea'),\n",
       " ('Antigua and Barbuda', 'Eritrea'),\n",
       " ('Antigua and Barbuda', 'Estonia'),\n",
       " ('Antigua and Barbuda', 'Ethiopia'),\n",
       " ('Antigua and Barbuda', 'Fiji'),\n",
       " ('Antigua and Barbuda', 'Finland'),\n",
       " ('Antigua and Barbuda', 'France'),\n",
       " ('Antigua and Barbuda', 'Gabon'),\n",
       " ('Antigua and Barbuda', 'Gambia, The'),\n",
       " ('Antigua and Barbuda', 'Georgia'),\n",
       " ('Antigua and Barbuda', 'Germany'),\n",
       " ('Antigua and Barbuda', 'Ghana'),\n",
       " ('Antigua and Barbuda', 'Greece'),\n",
       " ('Antigua and Barbuda', 'Grenada'),\n",
       " ('Antigua and Barbuda', 'Guatemala'),\n",
       " ('Antigua and Barbuda', 'Guinea'),\n",
       " ('Antigua and Barbuda', 'Guinea-Bissau'),\n",
       " ('Antigua and Barbuda', 'Guyana'),\n",
       " ('Antigua and Barbuda', 'Haiti'),\n",
       " ('Antigua and Barbuda', 'Honduras'),\n",
       " ('Antigua and Barbuda', 'Hungary'),\n",
       " ('Antigua and Barbuda', 'Iceland'),\n",
       " ('Antigua and Barbuda', 'India'),\n",
       " ('Antigua and Barbuda', 'Indonesia'),\n",
       " ('Antigua and Barbuda', 'Iran, Islamic Rep.'),\n",
       " ('Antigua and Barbuda', 'Iraq'),\n",
       " ('Antigua and Barbuda', 'Ireland'),\n",
       " ('Antigua and Barbuda', 'Israel'),\n",
       " ('Antigua and Barbuda', 'Italy'),\n",
       " ('Antigua and Barbuda', 'Jamaica'),\n",
       " ('Antigua and Barbuda', 'Japan'),\n",
       " ('Antigua and Barbuda', 'Jordan'),\n",
       " ('Antigua and Barbuda', 'Kazakhstan'),\n",
       " ('Antigua and Barbuda', 'Kenya'),\n",
       " ('Antigua and Barbuda', 'Kiribati'),\n",
       " ('Antigua and Barbuda', 'Korea, Rep.'),\n",
       " ('Antigua and Barbuda', 'Kuwait'),\n",
       " ('Antigua and Barbuda', 'Kyrgyz Republic'),\n",
       " ('Antigua and Barbuda', 'Lao PDR'),\n",
       " ('Antigua and Barbuda', 'Latvia'),\n",
       " ('Antigua and Barbuda', 'Lebanon'),\n",
       " ('Antigua and Barbuda', 'Lesotho'),\n",
       " ('Antigua and Barbuda', 'Liberia'),\n",
       " ('Antigua and Barbuda', 'Libya'),\n",
       " ('Antigua and Barbuda', 'Lithuania'),\n",
       " ('Antigua and Barbuda', 'Luxembourg'),\n",
       " ('Antigua and Barbuda', 'Madagascar'),\n",
       " ('Antigua and Barbuda', 'Malawi'),\n",
       " ('Antigua and Barbuda', 'Malaysia'),\n",
       " ('Antigua and Barbuda', 'Maldives'),\n",
       " ('Antigua and Barbuda', 'Mali'),\n",
       " ('Antigua and Barbuda', 'Malta'),\n",
       " ('Antigua and Barbuda', 'Mauritania'),\n",
       " ('Antigua and Barbuda', 'Mauritius'),\n",
       " ('Antigua and Barbuda', 'Mexico'),\n",
       " ('Antigua and Barbuda', 'Moldova'),\n",
       " ('Antigua and Barbuda', 'Mongolia'),\n",
       " ('Antigua and Barbuda', 'Montenegro'),\n",
       " ('Antigua and Barbuda', 'Morocco'),\n",
       " ('Antigua and Barbuda', 'Mozambique'),\n",
       " ('Antigua and Barbuda', 'Myanmar'),\n",
       " ('Antigua and Barbuda', 'Namibia'),\n",
       " ('Antigua and Barbuda', 'Nepal'),\n",
       " ('Antigua and Barbuda', 'Netherlands'),\n",
       " ('Antigua and Barbuda', 'New Zealand'),\n",
       " ('Antigua and Barbuda', 'Nicaragua'),\n",
       " ('Antigua and Barbuda', 'Niger'),\n",
       " ('Antigua and Barbuda', 'Nigeria'),\n",
       " ('Antigua and Barbuda', 'Norway'),\n",
       " ('Antigua and Barbuda', 'Oman'),\n",
       " ('Antigua and Barbuda', 'Pakistan'),\n",
       " ('Antigua and Barbuda', 'Palau'),\n",
       " ('Antigua and Barbuda', 'Panama'),\n",
       " ('Antigua and Barbuda', 'Papua New Guinea'),\n",
       " ('Antigua and Barbuda', 'Paraguay'),\n",
       " ('Antigua and Barbuda', 'Peru'),\n",
       " ('Antigua and Barbuda', 'Philippines'),\n",
       " ('Antigua and Barbuda', 'Poland'),\n",
       " ('Antigua and Barbuda', 'Portugal'),\n",
       " ('Antigua and Barbuda', 'Puerto Rico'),\n",
       " ('Antigua and Barbuda', 'Qatar'),\n",
       " ('Antigua and Barbuda', 'Romania'),\n",
       " ('Antigua and Barbuda', 'Russian Federation'),\n",
       " ('Antigua and Barbuda', 'Rwanda'),\n",
       " ('Antigua and Barbuda', 'Samoa'),\n",
       " ('Antigua and Barbuda', 'Sao Tome and Principe'),\n",
       " ('Antigua and Barbuda', 'Saudi Arabia'),\n",
       " ('Antigua and Barbuda', 'Senegal'),\n",
       " ('Antigua and Barbuda', 'Serbia'),\n",
       " ('Antigua and Barbuda', 'Seychelles'),\n",
       " ('Antigua and Barbuda', 'Sierra Leone'),\n",
       " ('Antigua and Barbuda', 'Singapore'),\n",
       " ('Antigua and Barbuda', 'Slovak Republic'),\n",
       " ('Antigua and Barbuda', 'Slovenia'),\n",
       " ('Antigua and Barbuda', 'Solomon Islands'),\n",
       " ('Antigua and Barbuda', 'Somalia'),\n",
       " ('Antigua and Barbuda', 'South Africa'),\n",
       " ('Antigua and Barbuda', 'South Sudan'),\n",
       " ('Antigua and Barbuda', 'Spain'),\n",
       " ('Antigua and Barbuda', 'Sri Lanka'),\n",
       " ('Antigua and Barbuda', 'Sudan'),\n",
       " ('Antigua and Barbuda', 'Suriname'),\n",
       " ('Antigua and Barbuda', 'Sweden'),\n",
       " ('Antigua and Barbuda', 'Switzerland'),\n",
       " ('Antigua and Barbuda', 'Syrian Arab Republic'),\n",
       " ('Antigua and Barbuda', 'Tajikistan'),\n",
       " ('Antigua and Barbuda', 'Tanzania'),\n",
       " ('Antigua and Barbuda', 'Thailand'),\n",
       " ('Antigua and Barbuda', 'Timor-Leste'),\n",
       " ('Antigua and Barbuda', 'Togo'),\n",
       " ('Antigua and Barbuda', 'Tonga'),\n",
       " ('Antigua and Barbuda', 'Trinidad and Tobago'),\n",
       " ('Antigua and Barbuda', 'Tunisia'),\n",
       " ('Antigua and Barbuda', 'Turkey'),\n",
       " ('Antigua and Barbuda', 'Turkmenistan'),\n",
       " ('Antigua and Barbuda', 'Tuvalu'),\n",
       " ('Antigua and Barbuda', 'Uganda'),\n",
       " ('Antigua and Barbuda', 'Ukraine'),\n",
       " ('Antigua and Barbuda', 'United Arab Emirates'),\n",
       " ('Antigua and Barbuda', 'United Kingdom'),\n",
       " ('Antigua and Barbuda', 'United States'),\n",
       " ('Antigua and Barbuda', 'Uruguay'),\n",
       " ('Antigua and Barbuda', 'Uzbekistan'),\n",
       " ('Antigua and Barbuda', 'Vanuatu'),\n",
       " ('Antigua and Barbuda', 'Venezuela, RB'),\n",
       " ('Antigua and Barbuda', 'Vietnam'),\n",
       " ('Antigua and Barbuda', 'Yemen, Rep.'),\n",
       " ('Antigua and Barbuda', 'Zambia'),\n",
       " ('Antigua and Barbuda', 'Zimbabwe'),\n",
       " ('Argentina', 'Armenia'),\n",
       " ('Argentina', 'Australia'),\n",
       " ('Argentina', 'Austria'),\n",
       " ('Argentina', 'Azerbaijan'),\n",
       " ('Argentina', 'Bahamas, The'),\n",
       " ('Argentina', 'Bahrain'),\n",
       " ('Argentina', 'Bangladesh'),\n",
       " ('Argentina', 'Barbados'),\n",
       " ('Argentina', 'Belarus'),\n",
       " ('Argentina', 'Belgium'),\n",
       " ('Argentina', 'Belize'),\n",
       " ('Argentina', 'Benin'),\n",
       " ('Argentina', 'Bhutan'),\n",
       " ('Argentina', 'Bolivia'),\n",
       " ('Argentina', 'Bosnia and Herzegovina'),\n",
       " ('Argentina', 'Botswana'),\n",
       " ('Argentina', 'Brazil'),\n",
       " ('Argentina', 'Brunei Darussalam'),\n",
       " ('Argentina', 'Bulgaria'),\n",
       " ('Argentina', 'Burkina Faso'),\n",
       " ('Argentina', 'Burundi'),\n",
       " ('Argentina', 'Cambodia'),\n",
       " ('Argentina', 'Cameroon'),\n",
       " ('Argentina', 'Canada'),\n",
       " ('Argentina', 'Central African Republic'),\n",
       " ('Argentina', 'Chad'),\n",
       " ('Argentina', 'Chile'),\n",
       " ('Argentina', 'China'),\n",
       " ('Argentina', 'Colombia'),\n",
       " ('Argentina', 'Comoros'),\n",
       " ('Argentina', 'Congo, Dem. Rep.'),\n",
       " ('Argentina', 'Congo, Rep.'),\n",
       " ('Argentina', 'Costa Rica'),\n",
       " ('Argentina', \"Cote d'Ivoire\"),\n",
       " ('Argentina', 'Croatia'),\n",
       " ('Argentina', 'Cuba'),\n",
       " ('Argentina', 'Cyprus'),\n",
       " ('Argentina', 'Czech Republic'),\n",
       " ('Argentina', 'Denmark'),\n",
       " ('Argentina', 'Djibouti'),\n",
       " ('Argentina', 'Dominica'),\n",
       " ('Argentina', 'Dominican Republic'),\n",
       " ('Argentina', 'Ecuador'),\n",
       " ('Argentina', 'Egypt, Arab Rep.'),\n",
       " ('Argentina', 'El Salvador'),\n",
       " ('Argentina', 'Equatorial Guinea'),\n",
       " ('Argentina', 'Eritrea'),\n",
       " ('Argentina', 'Estonia'),\n",
       " ('Argentina', 'Ethiopia'),\n",
       " ('Argentina', 'Fiji'),\n",
       " ('Argentina', 'Finland'),\n",
       " ('Argentina', 'France'),\n",
       " ('Argentina', 'Gabon'),\n",
       " ('Argentina', 'Gambia, The'),\n",
       " ('Argentina', 'Georgia'),\n",
       " ('Argentina', 'Germany'),\n",
       " ('Argentina', 'Ghana'),\n",
       " ('Argentina', 'Greece'),\n",
       " ('Argentina', 'Grenada'),\n",
       " ('Argentina', 'Guatemala'),\n",
       " ('Argentina', 'Guinea'),\n",
       " ('Argentina', 'Guinea-Bissau'),\n",
       " ('Argentina', 'Guyana'),\n",
       " ('Argentina', 'Haiti'),\n",
       " ('Argentina', 'Honduras'),\n",
       " ('Argentina', 'Hungary'),\n",
       " ('Argentina', 'Iceland'),\n",
       " ('Argentina', 'India'),\n",
       " ('Argentina', 'Indonesia'),\n",
       " ('Argentina', 'Iran, Islamic Rep.'),\n",
       " ('Argentina', 'Iraq'),\n",
       " ('Argentina', 'Ireland'),\n",
       " ('Argentina', 'Israel'),\n",
       " ('Argentina', 'Italy'),\n",
       " ('Argentina', 'Jamaica'),\n",
       " ('Argentina', 'Japan'),\n",
       " ('Argentina', 'Jordan'),\n",
       " ('Argentina', 'Kazakhstan'),\n",
       " ('Argentina', 'Kenya'),\n",
       " ('Argentina', 'Kiribati'),\n",
       " ('Argentina', 'Korea, Rep.'),\n",
       " ('Argentina', 'Kuwait'),\n",
       " ('Argentina', 'Kyrgyz Republic'),\n",
       " ('Argentina', 'Lao PDR'),\n",
       " ('Argentina', 'Latvia'),\n",
       " ('Argentina', 'Lebanon'),\n",
       " ('Argentina', 'Lesotho'),\n",
       " ('Argentina', 'Liberia'),\n",
       " ('Argentina', 'Libya'),\n",
       " ('Argentina', 'Lithuania'),\n",
       " ('Argentina', 'Luxembourg'),\n",
       " ('Argentina', 'Madagascar'),\n",
       " ('Argentina', 'Malawi'),\n",
       " ('Argentina', 'Malaysia'),\n",
       " ('Argentina', 'Maldives'),\n",
       " ('Argentina', 'Mali'),\n",
       " ('Argentina', 'Malta'),\n",
       " ('Argentina', 'Mauritania'),\n",
       " ('Argentina', 'Mauritius'),\n",
       " ('Argentina', 'Mexico'),\n",
       " ('Argentina', 'Moldova'),\n",
       " ('Argentina', 'Mongolia'),\n",
       " ('Argentina', 'Montenegro'),\n",
       " ('Argentina', 'Morocco'),\n",
       " ('Argentina', 'Mozambique'),\n",
       " ('Argentina', 'Myanmar'),\n",
       " ('Argentina', 'Namibia'),\n",
       " ('Argentina', 'Nepal'),\n",
       " ('Argentina', 'Netherlands'),\n",
       " ('Argentina', 'New Zealand'),\n",
       " ('Argentina', 'Nicaragua'),\n",
       " ('Argentina', 'Niger'),\n",
       " ('Argentina', 'Nigeria'),\n",
       " ('Argentina', 'Norway'),\n",
       " ('Argentina', 'Oman'),\n",
       " ...]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create list out of all unique combinations\n",
    "countrycombinations = list(itertools.combinations(countries, 2))\n",
    "countrycombinations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Afghanistan'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "countrycombinations[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Here, we calculate the standardised distance $E_y(c_{i}, c_{k})$ for each unique pair of two countries in each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.spatial import distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "First, we compute the (not standardised) distances $e_y$ and insert them into a new dictionary `dict_e`.\n",
    "\n",
    "While exploring the data, we see that nearly no data are available for the years `1990` to `1999`. Consequently, imputations in those years will be based on very weak foundations and we do not consider these years for now. For our similarity investigations later, it does not matter much how many data points we have totally available per country, it is more important that all countries have the same amount of data points. For now, we also omit data for the year `2019`, because it seems not all countries have reported their data yet. Hence, there aren't too many data points available neither.\n",
    "\n",
    "We set the `period` of years we want to consider in our computations for $e_y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "period = ['2000', '2001', '2002', '2003', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017', '2018', '2019']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "399\n",
      "400\n"
     ]
    }
   ],
   "source": [
    "# call seriescodes again\n",
    "info = pd.read_csv('utils/wb_info.csv', header=None, dtype=str)\n",
    "info.drop_duplicates(subset=[2], inplace=True)\n",
    "#seriescodes = list(info['Series Code'])\n",
    "seriescodes = list(dict_all['Germany'].index)\n",
    "print(len(seriescodes))\n",
    "print(len(info))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------\n",
      "Temperature\n"
     ]
    }
   ],
   "source": [
    "# finding which seriescodes are not in other list and adding them\n",
    "for seriescode in seriescodes:\n",
    "    if seriescode not in list(info[0]):\n",
    "        print(seriescode)\n",
    "\n",
    "print('-------')\n",
    "\n",
    "for seriescode in list(info[0]):\n",
    "    if seriescode not in seriescodes:\n",
    "        print(seriescode)\n",
    "        seriescodes.append(seriescode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>FB.CBK.BRCH.P5</td>\n",
       "      <td>Financial Sector: Access</td>\n",
       "      <td>Commercial bank branches (per 100,000 adults)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>225</th>\n",
       "      <td>FX.OWN.TOTL.40.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>FX.OWN.TOTL.60.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>FX.OWN.TOTL.FE.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>FX.OWN.TOTL.MA.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>FX.OWN.TOTL.OL.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>FX.OWN.TOTL.PL.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>FX.OWN.TOTL.SO.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>FX.OWN.TOTL.YG.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>FX.OWN.TOTL.ZS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Account ownership at a financial institution o...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>IC.BUS.NDNS.ZS</td>\n",
       "      <td>Private Sector &amp; Trade: Business environment</td>\n",
       "      <td>New business density (new registrations per 1,...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>NV.AGR.EMPL.KD</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Agriculture, value added per worker (constant ...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>NV.IND.EMPL.KD</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Industry, value added per worker (constant 201...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>NV.SRV.EMPL.KD</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Services, value added per worker (constant 201...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>NY.GDP.MKTP.KD.ZG</td>\n",
       "      <td>Economic Policy &amp; Debt: National accounts: Gro...</td>\n",
       "      <td>GDP growth (annual %)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>NY.GDP.PCAP.KD.ZG</td>\n",
       "      <td>Economic Policy &amp; Debt: National accounts: Gro...</td>\n",
       "      <td>GDP per capita growth (annual %)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>SL.AGR.EMPL.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in agriculture, female (% of female...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>SL.AGR.EMPL.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in agriculture, male (% of male emp...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>242</th>\n",
       "      <td>SL.AGR.EMPL.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in agriculture (% of total employment)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>SL.EMP.WORK.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Wage and salaried workers, female (% of female...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>SL.EMP.WORK.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Wage and salaried workers, male (% of male emp...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245</th>\n",
       "      <td>SL.EMP.WORK.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Wage and salaried workers, total (% of total e...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>SL.GDP.PCAP.EM.KD</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>GDP per person employed (constant 2011 PPP $)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>SL.IND.EMPL.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in industry, female (% of female em...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>SL.IND.EMPL.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in industry, male (% of male employ...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>SL.IND.EMPL.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in industry (% of total employment)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>SL.ISV.IFRM.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Informal employment, female (% of total non-ag...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>SL.ISV.IFRM.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Informal employment, male (% of total non-agri...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>SL.ISV.IFRM.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Informal employment (% of total non-agricultur...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>SL.SRV.EMPL.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in services, female (% of female em...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>SL.SRV.EMPL.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in services, male (% of male employ...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>SL.SRV.EMPL.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Employment in services (% of total employment)</td>\n",
       "      <td>8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>SL.TLF.0714.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Children in employment, female (% of female ch...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>257</th>\n",
       "      <td>SL.TLF.0714.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Children in employment, male (% of male childr...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>SL.TLF.0714.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Economic activity</td>\n",
       "      <td>Children in employment, total (% of children a...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>SL.UEM.1524.FE.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth female (% of female labor ...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>SL.UEM.1524.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth female (% of female labor ...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>SL.UEM.1524.MA.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth male (% of male labor forc...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>SL.UEM.1524.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth male (% of male labor forc...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>263</th>\n",
       "      <td>SL.UEM.1524.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth total (% of total labor fo...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <td>SL.UEM.1524.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, youth total (% of total labor fo...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>SL.UEM.NEET.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Share of youth not in education, employment or...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.6</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>SL.UEM.NEET.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Share of youth not in education, employment or...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.6</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>267</th>\n",
       "      <td>SL.UEM.NEET.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Share of youth not in education, employment or...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.6</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>SL.UEM.TOTL.FE.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, female (% of female labor force)...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>SL.UEM.TOTL.FE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, female (% of female labor force)...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>SL.UEM.TOTL.MA.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, male (% of male labor force) (na...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>SL.UEM.TOTL.MA.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, male (% of male labor force) (mo...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>SL.UEM.TOTL.NE.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, total (% of total labor force) (...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>SL.UEM.TOTL.ZS</td>\n",
       "      <td>Social Protection &amp; Labor: Unemployment</td>\n",
       "      <td>Unemployment, total (% of total labor force) (...</td>\n",
       "      <td>8</td>\n",
       "      <td>8.5</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        0                                                  1  \\\n",
       "224        FB.CBK.BRCH.P5                           Financial Sector: Access   \n",
       "225     FX.OWN.TOTL.40.ZS                                                NaN   \n",
       "226     FX.OWN.TOTL.60.ZS                                                NaN   \n",
       "227     FX.OWN.TOTL.FE.ZS                                                NaN   \n",
       "228     FX.OWN.TOTL.MA.ZS                                                NaN   \n",
       "229     FX.OWN.TOTL.OL.ZS                                                NaN   \n",
       "230     FX.OWN.TOTL.PL.ZS                                                NaN   \n",
       "231     FX.OWN.TOTL.SO.ZS                                                NaN   \n",
       "232     FX.OWN.TOTL.YG.ZS                                                NaN   \n",
       "233        FX.OWN.TOTL.ZS                                                NaN   \n",
       "234        IC.BUS.NDNS.ZS       Private Sector & Trade: Business environment   \n",
       "235        NV.AGR.EMPL.KD                                                NaN   \n",
       "236        NV.IND.EMPL.KD                                                NaN   \n",
       "237        NV.SRV.EMPL.KD                                                NaN   \n",
       "238     NY.GDP.MKTP.KD.ZG  Economic Policy & Debt: National accounts: Gro...   \n",
       "239     NY.GDP.PCAP.KD.ZG  Economic Policy & Debt: National accounts: Gro...   \n",
       "240     SL.AGR.EMPL.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "241     SL.AGR.EMPL.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "242        SL.AGR.EMPL.ZS       Social Protection & Labor: Economic activity   \n",
       "243     SL.EMP.WORK.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "244     SL.EMP.WORK.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "245        SL.EMP.WORK.ZS       Social Protection & Labor: Economic activity   \n",
       "246     SL.GDP.PCAP.EM.KD       Social Protection & Labor: Economic activity   \n",
       "247     SL.IND.EMPL.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "248     SL.IND.EMPL.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "249        SL.IND.EMPL.ZS       Social Protection & Labor: Economic activity   \n",
       "250     SL.ISV.IFRM.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "251     SL.ISV.IFRM.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "252        SL.ISV.IFRM.ZS       Social Protection & Labor: Economic activity   \n",
       "253     SL.SRV.EMPL.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "254     SL.SRV.EMPL.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "255        SL.SRV.EMPL.ZS       Social Protection & Labor: Economic activity   \n",
       "256     SL.TLF.0714.FE.ZS       Social Protection & Labor: Economic activity   \n",
       "257     SL.TLF.0714.MA.ZS       Social Protection & Labor: Economic activity   \n",
       "258        SL.TLF.0714.ZS       Social Protection & Labor: Economic activity   \n",
       "259  SL.UEM.1524.FE.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "260     SL.UEM.1524.FE.ZS            Social Protection & Labor: Unemployment   \n",
       "261  SL.UEM.1524.MA.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "262     SL.UEM.1524.MA.ZS            Social Protection & Labor: Unemployment   \n",
       "263     SL.UEM.1524.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "264        SL.UEM.1524.ZS            Social Protection & Labor: Unemployment   \n",
       "265     SL.UEM.NEET.FE.ZS            Social Protection & Labor: Unemployment   \n",
       "266     SL.UEM.NEET.MA.ZS            Social Protection & Labor: Unemployment   \n",
       "267        SL.UEM.NEET.ZS            Social Protection & Labor: Unemployment   \n",
       "268  SL.UEM.TOTL.FE.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "269     SL.UEM.TOTL.FE.ZS            Social Protection & Labor: Unemployment   \n",
       "270  SL.UEM.TOTL.MA.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "271     SL.UEM.TOTL.MA.ZS            Social Protection & Labor: Unemployment   \n",
       "272     SL.UEM.TOTL.NE.ZS            Social Protection & Labor: Unemployment   \n",
       "273        SL.UEM.TOTL.ZS            Social Protection & Labor: Unemployment   \n",
       "\n",
       "                                                     2  3    4   5  \n",
       "224      Commercial bank branches (per 100,000 adults)  8  8.1   1  \n",
       "225  Account ownership at a financial institution o...  8  8.1   1  \n",
       "226  Account ownership at a financial institution o...  8  8.1   1  \n",
       "227  Account ownership at a financial institution o...  8  8.1   1  \n",
       "228  Account ownership at a financial institution o...  8  8.1   1  \n",
       "229  Account ownership at a financial institution o...  8  8.1   1  \n",
       "230  Account ownership at a financial institution o...  8  8.1   1  \n",
       "231  Account ownership at a financial institution o...  8  8.1   1  \n",
       "232  Account ownership at a financial institution o...  8  8.1   1  \n",
       "233  Account ownership at a financial institution o...  8  8.1   1  \n",
       "234  New business density (new registrations per 1,...  8  8.3   1  \n",
       "235  Agriculture, value added per worker (constant ...  8  8.2   1  \n",
       "236  Industry, value added per worker (constant 201...  8  8.2   1  \n",
       "237  Services, value added per worker (constant 201...  8  8.2   1  \n",
       "238                              GDP growth (annual %)  8  8.1   1  \n",
       "239                   GDP per capita growth (annual %)  8  8.1   1  \n",
       "240  Employment in agriculture, female (% of female...  8  8.2   1  \n",
       "241  Employment in agriculture, male (% of male emp...  8  8.2   1  \n",
       "242  Employment in agriculture (% of total employment)  8  8.2   1  \n",
       "243  Wage and salaried workers, female (% of female...  8  8.5   1  \n",
       "244  Wage and salaried workers, male (% of male emp...  8  8.5   1  \n",
       "245  Wage and salaried workers, total (% of total e...  8  8.5   1  \n",
       "246      GDP per person employed (constant 2011 PPP $)  8  8.2   1  \n",
       "247  Employment in industry, female (% of female em...  8  8.2   1  \n",
       "248  Employment in industry, male (% of male employ...  8  8.2   1  \n",
       "249     Employment in industry (% of total employment)  8  8.2   1  \n",
       "250  Informal employment, female (% of total non-ag...  8  8.3  -1  \n",
       "251  Informal employment, male (% of total non-agri...  8  8.3  -1  \n",
       "252  Informal employment (% of total non-agricultur...  8  8.3  -1  \n",
       "253  Employment in services, female (% of female em...  8  8.2   1  \n",
       "254  Employment in services, male (% of male employ...  8  8.2   1  \n",
       "255     Employment in services (% of total employment)  8  8.2   1  \n",
       "256  Children in employment, female (% of female ch...  8  8.7  -1  \n",
       "257  Children in employment, male (% of male childr...  8  8.7  -1  \n",
       "258  Children in employment, total (% of children a...  8  8.7  -1  \n",
       "259  Unemployment, youth female (% of female labor ...  8  8.5  -1  \n",
       "260  Unemployment, youth female (% of female labor ...  8  8.5  -1  \n",
       "261  Unemployment, youth male (% of male labor forc...  8  8.5  -1  \n",
       "262  Unemployment, youth male (% of male labor forc...  8  8.5  -1  \n",
       "263  Unemployment, youth total (% of total labor fo...  8  8.5  -1  \n",
       "264  Unemployment, youth total (% of total labor fo...  8  8.5  -1  \n",
       "265  Share of youth not in education, employment or...  8  8.6  -1  \n",
       "266  Share of youth not in education, employment or...  8  8.6  -1  \n",
       "267  Share of youth not in education, employment or...  8  8.6  -1  \n",
       "268  Unemployment, female (% of female labor force)...  8  8.5  -1  \n",
       "269  Unemployment, female (% of female labor force)...  8  8.5  -1  \n",
       "270  Unemployment, male (% of male labor force) (na...  8  8.5  -1  \n",
       "271  Unemployment, male (% of male labor force) (mo...  8  8.5  -1  \n",
       "272  Unemployment, total (% of total labor force) (...  8  8.5  -1  \n",
       "273  Unemployment, total (% of total labor force) (...  8  8.5  -1  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking if '8.10' became 8.1 (8 is only SDG with a target number 10 in the current version of data)\n",
    "# could be amended manually, but does not matter as long as analyses are done on goal-level\n",
    "info[info[3]=='8']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 119)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "dict_all_std['Iraq'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing Euclidean distances computed with vectors of different dimensions\n",
    "\n",
    "Simply computing the Euclidean distance between all the countries we have won't give us the results we like, because each of the pairs of countries we calculate the Euclidean distance for has different many measurements to take into account.\n",
    "\n",
    "We consider this by counting how many measurements `j` we have for each pair of countries and setting the weight `w` as the invesre of the number of measurements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000\n",
      "2001\n",
      "2002\n",
      "2003\n",
      "2004\n",
      "2005\n",
      "2006\n",
      "2007\n",
      "2008\n",
      "2009\n",
      "2010\n",
      "2011\n",
      "2012\n",
      "2013\n",
      "2014\n",
      "2015\n",
      "2016\n",
      "2017\n",
      "2018\n",
      "2019\n"
     ]
    }
   ],
   "source": [
    "# ~ one hour computing time\n",
    "# no need to run every time again, just see CHECKPOINT above and load pickle file\n",
    "\n",
    "dict_e = {}    \n",
    "\n",
    "for year in period:\n",
    "    print(year)\n",
    "    \n",
    "    for countrycombination in countrycombinations:\n",
    "        \n",
    "        country0_e = []    # create two empty lists for the two groupings we consider at the moment\n",
    "        country1_e = []    # these lists contain series codes with data available in both groupings\n",
    "        j = 0    # counter\n",
    "        \n",
    "        for seriescode in seriescodes:\n",
    "            # we can only consider sub-indicators with data available in both groupings\n",
    "            if pd.isna(dict_all_std[countrycombination[0]].loc[seriescode, year]) is False and pd.isna(dict_all_std[countrycombination[1]].loc[seriescode, year]) is False:\n",
    "                country0_e.append(dict_all_std[countrycombination[0]].loc[seriescode, year])\n",
    "                country1_e.append(dict_all_std[countrycombination[1]].loc[seriescode, year])\n",
    "                \n",
    "                j += 1\n",
    "        \n",
    "        #print('number of data points available: ', j)    # check\n",
    "        if j > 0:\n",
    "            e = distance.euclidean(country0_e, country1_e, w=1/j)\n",
    "        else:\n",
    "            e = np.nan    # make NaN\n",
    "            \n",
    "        #print('e in {} between {} and {}:'.format(year, countrycombination[0], countrycombination[1]), e)\n",
    "        \n",
    "        dict_e[year, countrycombination[1], countrycombination[0]] = e\n",
    "        dict_e[year, countrycombination[0], countrycombination[1]] = dict_e[year, countrycombination[1], countrycombination[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "f = open('utils/data/distances_unstd.pkl', 'wb')\n",
    "pickle.dump(dict_e, f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHECKPOINT\n",
    "dict_e = pickle.load(open('utils/data/distances_unstd.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.819235652376648\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print(dict_e['2011', 'Switzerland', 'Iraq'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalise these distances $e_y$ and save them in `dict_E`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000\n",
      "2001\n",
      "2002\n",
      "2003\n",
      "2004\n",
      "2005\n",
      "2006\n",
      "2007\n",
      "2008\n",
      "2009\n",
      "2010\n",
      "2011\n",
      "2012\n",
      "2013\n",
      "2014\n",
      "2015\n",
      "2016\n",
      "2017\n",
      "2018\n",
      "2019\n"
     ]
    }
   ],
   "source": [
    "dict_E = {}\n",
    "\n",
    "for year in period:\n",
    "    print(year)\n",
    "    \n",
    "    max_e = 0    # maximum value per year\n",
    "    min_e = 0\n",
    "    dict_e_year = {}    # auxiliary dictionary with all distances per year\n",
    "    \n",
    "    for k in dict_e.keys():\n",
    "        if year in k:\n",
    "            dict_e_year[k] = dict_e[k]            \n",
    "\n",
    "    max_e = np.nanmax(list(dict_e_year.values()))\n",
    "\n",
    "    #print('------------')\n",
    "    #print('max_e in {}'.format(year), max_e)\n",
    "    #print('------------')\n",
    "       \n",
    "    for k in dict_e_year.keys():\n",
    "        #print(k)\n",
    "        if np.isnan(dict_e_year[k]) == False:\n",
    "            #print('unstandardised distance e:', dict_e_year[k])\n",
    "            dict_E[k] = dict_e_year[k] / max_e    # standardise distance\n",
    "            \n",
    "        else:\n",
    "            dict_E[k] = np.nan    # keep as NaN\n",
    "        \n",
    "        #print('standardised distance E:', dict_E[k])\n",
    "        #print('------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unstandardised distance e: 1.0770541228951773\n",
      "standardised distance E:   0.5196050842596447\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print('unstandardised distance e:', dict_e['2016', 'Afghanistan', 'Colombia'])\n",
    "print('normalised distance E:  ', dict_E['2016', 'Afghanistan', 'Colombia'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# check (both should be False)\n",
    "print(0 in dict_e.values())\n",
    "print(0 in dict_E.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check \n",
    "min_value = 0.1\n",
    "\n",
    "for key, value in dict_E.items():\n",
    "    if 0 < value < min_value:\n",
    "        min_value = value\n",
    "        print('smallest:', key, value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "f = open('utils/data/distances_std.pkl', 'wb')\n",
    "pickle.dump(dict_E, f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Imputations for countries\n",
    "\n",
    "Now, we impute the missing values according to the equation we previously derived:\n",
    "\n",
    "$$ x^{j'}_{i,y} = \\frac{1}{K} \\sum_k \\frac{1}{|E_y(c_{i}, c_{k})|} \\cdot x^j_{k,y} $$\n",
    "\n",
    "To recap, our imputations $x^{j'}_{i,y}$ for missing sub-indicator $j'$ in country $i$ in year $y$ should be similar to sub-indicators $j$ of country $k$, according to the inverse standardised Euclidean distance $\\frac{1}{|E_y(c_{i}, c_{k})|}$ between $i$ and $k$. \n",
    "\n",
    "As aforementioned and shown in the equation of $E_y$, $E_y$ is dependent on the number of pairs we have in both countries data for. Our `dict_E` has already entries for $E$ normalised according to this number of available pairs of data. We multiply our weight for each imputation, i.e. the inverse standardised Euclidean distance $\\frac{1}{|E_y(c_{i}, c_{k})|}$ between $i$ and $k$, by the value $x^j_{k,y}$ of the other country $k$ in year $y$ for sub-indicator $j$. We sum over $k$ to add together all weighted $x^j_{k,y}$ of each unique pair of countries $i$ and $k$ and compute its average by dividing by $K$. $K$ is the number of countries which have values available for the sub-indicator $x^{j'}_{k,y}$ to be computed. \n",
    "\n",
    "We also know that some indicators are **binary**, i.e. 1 for 'yes' and 0 for 'no', and ordinal for groupings of countries. These indicators start usually with 'number of countries which...'. The imputations for these must be handled differently: we round the imputed value to an integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in UN data set exist binary indicators\n",
    "binary = ['1.5.3', '5.1.1', '5.6.2', '5.a.2', '5.c.1', '8.b.1', '10.7.2', '11.b.1', '12.1.1', '12.7.1', '13.1.2', '13.2.1', '13.3.1', '13.3.2', '13.b.1', '14.c.1', '15.6.1', '15.8.1', '16.10.2', '17.5.1', '17.14.1', '17.16.1', '17.18.2', '17.18.3', '17.19.2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info_binary = info.loc[info['Indicator'].isin(binary)]\n",
    "binary_seriescodes = list(info_binary['SeriesCode'])\n",
    "\n",
    "binary_seriescodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a8afec8218c4d328276aa999dcc0cb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=180.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# ~30 minutes\n",
    "\n",
    "dict_all_i = {}\n",
    "\n",
    "for country in tqdm(countries):\n",
    "    \n",
    "    dict_all_i[country] = pd.DataFrame(index=seriescodes, columns=period)\n",
    "    \n",
    "    not_countries = [c for c in countries if c != country]\n",
    "    \n",
    "    for seriescode in seriescodes:\n",
    "        for year in period:            \n",
    "            if pd.isna(dict_all_std[country].loc[seriescode, year]) is True:\n",
    "                K = 0\n",
    "                all_k = []\n",
    "                \n",
    "                for not_country in not_countries: \n",
    "                    if pd.isna(dict_all_std[not_country].loc[seriescode, year]) is False and pd.isna(dict_E[(year, country, not_country)]) is False: # and dict_E[(year, country, not_country)]!=0:    # not_country can also have NaN -> exclude those\n",
    "                        K += 1\n",
    "                        # print('value:', dict_all_std[not_country].loc[seriescode, year])\n",
    "                        # print('distance:', dict_E[(year, country, not_country)])\n",
    "                        k = (dict_all_std[not_country].loc[seriescode, year]) / (dict_E[(year, country, not_country)])\n",
    "                        # print('k =', k)\n",
    "                        all_k.append(k)\n",
    "                        \n",
    "                sum_k = np.sum(all_k)\n",
    "                    \n",
    "                #print('K =', K)\n",
    "                    \n",
    "                if K > 0:\n",
    "                    # print('sum k =', sum_k)\n",
    "                    \n",
    "                    \"\"\"\n",
    "                    # only UN data set has binary seriescodes\n",
    "                    if seriescode in binary_seriescodes:\n",
    "                        dict_all_i[country].loc[seriescode, year] = np.around(sum_k / K)    # round to have binary\n",
    "                    else:\n",
    "                        dict_all_i[country].loc[seriescode, year] = sum_k / K\n",
    "                    \"\"\"\n",
    "                    dict_all_i[country].loc[seriescode, year] = sum_k / K\n",
    "                            \n",
    "                else:\n",
    "                    dict_all_i[country].loc[seriescode, year] = np.nan    # only impute when data of other countries is available, 0 cannot be imputed because time-series are non-stationary\n",
    "                \n",
    "                #print('Imputation for {} in {} in {}'.format(seriescode, country, year), dict_all_i[country].loc[seriescode, year])        \n",
    "                \n",
    "            else:\n",
    "                dict_all_i[country].loc[seriescode, year] = dict_all_std[country].loc[seriescode, year]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to delete all keys which are not part of the list of countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of keys to delete\n",
    "delete_keys = []\n",
    "\n",
    "for key in dict_all_i.keys():\n",
    "    if key not in countries:\n",
    "        delete_keys.append(key)\n",
    "        \n",
    "# delete\n",
    "for dk in delete_keys:\n",
    "    dict_all_i.pop(dk, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "delete_keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.8232424225359845"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check \n",
    "max_values = []\n",
    "\n",
    "for c in dict_all_i.keys():\n",
    "    max_values.append(dict_all_i[c].max().max())\n",
    "    \n",
    "max(max_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NaN here nan\n",
      "Imputed value -0.4647248878236216\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print('NaN here', dict_all_std['Afghanistan'].loc['SE.SEC.UNER.LO.ZS', '2016'])\n",
    "print('Imputed value', dict_all_i['Afghanistan'].loc['SE.SEC.UNER.LO.ZS', '2016'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000     0.699648\n",
       "2001     0.812976\n",
       "2002    0.0895247\n",
       "2003    0.0136518\n",
       "2004    -0.338087\n",
       "2005    -0.394271\n",
       "2006    -0.320402\n",
       "2007    -0.281633\n",
       "2008    -0.177067\n",
       "2009    0.0878838\n",
       "2010   -0.0788393\n",
       "2011    -0.186653\n",
       "2012    -0.277259\n",
       "2013    -0.393072\n",
       "2014     -0.46631\n",
       "2015    -0.936052\n",
       "2016    -0.464725\n",
       "2017     -0.47778\n",
       "2018    -0.778671\n",
       "2019     -2.99926\n",
       "Name: SE.SEC.UNER.LO.ZS, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "dict_all_i['Afghanistan'].loc['SE.SEC.UNER.LO.ZS']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to save the imputations to have another checkpoint here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "# as csv files\n",
    "if not os.path.exists('csv_imputed'):\n",
    "    os.mkdir('csv_imputed')\n",
    "\n",
    "for c in countries:\n",
    "    dict_all_i[c].to_csv(r'csv_imputed/{}_wb.csv'.format(c))\n",
    "    \n",
    "# as pkl files\n",
    "imp = open('utils/data/dict_all_i_wb.pkl', 'wb')\n",
    "pickle.dump(dict_all_i, imp)\n",
    "imp.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHECKPOINT\n",
    "dict_all_i = pickle.load(open('utils/data/dict_all_i_wb.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the total number of values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total # values before imputations: 850591\n",
      "Total # values after imputations: 1172340\n",
      "How many have been imputed? 321749\n",
      "This accounts for 37.83 % of the total data available now\n"
     ]
    }
   ],
   "source": [
    "# number of values\n",
    "s_imp = 0\n",
    "for country in dict_all_i.keys():\n",
    "    s_imp += np.sum(dict_all_i[country].count())\n",
    "\n",
    "print('Total # values before imputations:', s)\n",
    "print('Total # values after imputations:', s_imp)\n",
    "print('How many have been imputed?', s_imp - s)\n",
    "print('This accounts for', round(100*(s_imp-s)/s, 2), '% of the total data available now')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Averaging and concatenating data to higher levels\n",
    "\n",
    "### *(UN data set only)* Averaging and concatenating series codes data to indicator-level\n",
    "\n",
    "1. We can average all series codes, i.e. sub-indicators, belonging to one indicator to this indicator.\n",
    "2. We can see the series codes as multiple samples of the same indicator and concatenate the series codes into indicators. Consequently, we have more than one measurement per time point for any indicators having more than one series code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indicators = list(info.Indicator)\n",
    "\n",
    "dict_indicators = {}\n",
    "\n",
    "for indicator in indicators:\n",
    "    i = info['SeriesCode'].where(info['Indicator'] == indicator)\n",
    "\n",
    "    dict_indicators[indicator] = [s for s in i if str(s) != 'nan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check\n",
    "np.isnan(dict_all_i['Germany'].loc['SI_POV_DAY1', '2001'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#indicators_values = {}\n",
    "#indicators_values_std = {}\n",
    "indicators_values_i = {}\n",
    "\n",
    "for country in countries:\n",
    "    #print(country)\n",
    "    \n",
    "    #indicators_values[country] = pd.DataFrame(columns=period, index=indicators)\n",
    "    #indicators_values_std[country] = pd.DataFrame(columns=period, index=indicators)\n",
    "    indicators_values_i[country] = pd.DataFrame(columns=period, index=list(dict_indicators.keys()))\n",
    "    \n",
    "    for year in period:\n",
    "        \n",
    "        for indicator in dict_indicators.keys():\n",
    "            #list_subindicators_values = []\n",
    "            #list_subindicators_values_std = []\n",
    "            list_subindicators_values_i = []\n",
    "    \n",
    "            for subindicator in list(dict_indicators[indicator]):\n",
    "                if np.isnan(dict_all_i[country].loc[subindicator, year]):\n",
    "                    pass\n",
    "                else:\n",
    "                    #list_subindicators_values.append(dict_all[country].loc[subindicator, year])\n",
    "                    #list_subindicators_values_std.append(dict_all_std[country].loc[subindicator, year])\n",
    "                    list_subindicators_values_i.append(dict_all_i[country].loc[subindicator, year])\n",
    "            \n",
    "            # 1. averaging\n",
    "            #indicators_values[country].loc[indicator, year] = np.nanmean(list_subindicators_values)\n",
    "            #indicators_values_std[country].loc[indicator, year] = np.nanmean(list_subindicators_values_std)\n",
    "            #indicators_values_i[country].loc[indicator, year] = np.nanmean(list_subindicators_values_i)\n",
    "            \n",
    "            # 2. concatenating\n",
    "            array_subindicators_values_i = np.asarray(list_subindicators_values_i)\n",
    "            indicators_values_i[country].loc[indicator, year] = array_subindicators_values_i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# check (duplicate indicator labels should be in index)\n",
    "indicators_values_i['Germany']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "#ind_val = open('utils/data/indicators_values.pkl', 'wb')\n",
    "#ind_val_std = open('utils/data/indicators_values_std.pkl', 'wb')\n",
    "ind_val_i = open('utils/data/indicators_values_i.pkl', 'wb')\n",
    "#pickle.dump(indicators_values, ind_val)\n",
    "#pickle.dump(indicators_values_std, ind_val_std)\n",
    "pickle.dump(indicators_values_i, ind_val_i)\n",
    "#ind_val.close()\n",
    "#ind_val_std.close()\n",
    "ind_val_i.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *(UN data set)* Averaging and concatenating indicator data to target-level\n",
    "We must also generate two lists of indicators which are meant to increase and decrease over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gone though all targets by hand and checked which indicators are meant to increase and which are meant to decrease over time.\n",
    "increase = ['1.3.1', '1.4.1', '1.4.2', '1.5.3', '1.5.4', '1.a.1', '1.a.2', '1.a.3', '1.b.1', '2.3.1', '2.3.2', '2.4.1', '2.5.1', '2.a.1', '2.a.2', '3.1.2', '3.5.1', '3.7.1', '3.8.1', '3.b.1', '3.b.2', '3.b.3', '3.c.1', '3.d.1', '4.1.1', '4.2.1', '4.2.2', '4.3.1', '4.4.1', '4.6.1', '4.7.1', '4.a.1', '4.b.1', '4.c.1', '5.1.1', '5.5.1', '5.5.2', '5.6.1', '5.6.2', '5.a.1', '5.a.2', '5.b.1', '5.c.1', '6.1.1', '6.2.1', '6.3.1', '6.3.2', '6.4.1', '6.5.1', '6.5.2', '6.6.1', '6.a.1', '6.b.1', '7.1.1', '7.1.2', '7.2.1', '7.3.1', '7.a.1', '7.b.1', '8.1.1', '8.2.1', '8.3.1', '8.5.1', '8.8.2', '8.9.1', '8.9.2', '8.10.1', '8.10.2', '8.a.1', '8.b.1', '9.1.1', '9.1.2', '9.2.1', '9.2.2', '9.3.1', '9.3.2', '9.5.1', '9.5.2', '9.a.1', '9.b.1', '9.c.1', '10.1.1', '10.4.1', '10.5.1', '10.6.1', '10.7.2', '10.a.1', '10.b.1', '11.2.1', '11.3.2', '11.4.1', '11.6.1', '11.7.1', '11.a.1', '11.b.1', '11.b.2', '11.c.1', '12.1.1', '12.4.1', '12.5.1', '12.6.1', '12.7.1', '12.8.1', '12.a.1', '12.b.1', '13.1.2', '13.1.3', '13.2.1', '13.3.1', '13.3.2', '13.a.1', '13.b.1', '14.2.1', '14.3.1', '14.4.1', '14.5.1', '14.6.1', '14.7.1', '14.a.1', '14.b.1', '14.c.1', '15.1.1', '15.1.2', '15.2.1', '15.4.1', '15.4.2', '15.6.1', '15.8.1', '15.9.1', '15.a.1', '15.b.1', '16.1.4', '16.6.2', '16.7.1', '16.7.2', '16.8.1', '16.9.1', '16.10.2', '16.a.1', '17.1.1', '17.1.2', '17.2.1', '17.3.1', '17.3.2', '17.4.1', '17.5.1', '17.6.1', '17.6.2', '17.7.1', '17.8.1', '17.9.1', '17.11.1', '17.13.1', '17.14.1', '17.15.1', '17.16.1', '17.17.1', '17.18.1', '17.18.2', '17.18.3', '17.19.1', '17.19.2']\n",
    "decrease = ['1.1.1', '1.2.1', '1.2.2', '1.5.1', '1.5.2', '2.1.1', '2.1.2', '2.2.1', '2.2.2', '2.5.2','2.b.1', '2.c.1', '3.1.1', '3.2.1', '3.2.2', '3.3.1', '3.3.2', '3.3.3', '3.3.4', '3.3.5', '3.4.1', '3.4.2', '3.5.2', '3.6.1', '3.7.2', '3.8.2', '3.9.1', '3.9.2', '3.9.3', '3.a.1', '4.5.1', '5.2.1', '5.2.2', '5.3.1', '5.3.2', '5.4.1', '6.4.2', '8.4.1', '8.4.2', '8.5.2', '8.6.1', '8.7.1', '8.8.1', '9.4.1', '10.2.1', '10.3.1', '10.7.1', '10.c.1', '11.1.1', '11.3.1', '11.5.1', '11.5.2', '11.6.2', '11.7.2', '12.2.1', '12.2.2', '12.3.1', '12.4.2', '12.c.1', '13.1.1', '14.1.1', '15.3.1', '15.5.1', '15.7.1', '15.c.1', '16.1.1', '16.1.2', '16.1.3', '16.2.1', '16.2.2', '16.2.3', '16.3.1', '16.3.2', '16.4.1', '16.4.2', '16.5.1', '16.5.2', '16.6.1', '16.10.1', '16.b.1', '17.10.1', '17.12.1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making all time-series \"pointing\" upwards when they are meant to increase\n",
    "\n",
    "indicators_values_i_up = {}\n",
    "\n",
    "for country in countries:\n",
    "    indicators_values_i_up[country] = pd.DataFrame(index=list(dict_indicators.keys()), columns=period)\n",
    "    \n",
    "    for indicator in dict_indicators.keys():\n",
    "        if indicator in decrease:\n",
    "            #indicators_values_up[country].loc[indicator] = indicators_values[country].loc[indicator]*(-1)\n",
    "            #indicators_values_std_up[country].loc[indicator] = indicators_values_std[country].loc[indicator]*(-1)\n",
    "            indicators_values_i_up[country].loc[indicator] = list(np.multiply(list(indicators_values_i[country].loc[indicator]), -1))\n",
    "        else:\n",
    "            #indicators_values_up[country].loc[indicator] = indicators_values[country].loc[indicator]\n",
    "            #indicators_values_std_up[country].loc[indicator] = indicators_values_std[country].loc[indicator]\n",
    "            indicators_values_i_up[country].loc[indicator] = indicators_values_i[country].loc[indicator]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# check\n",
    "indicators_values_i_up['Germany']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "#ind_val = open('utils/data/indicators_values_up.pkl', 'wb')\n",
    "#ind_val_std = open('utils/data/indicators_values_std_up.pkl', 'wb')\n",
    "ind_val_i = open('utils/data/indicators_values_i_up.pkl', 'wb')\n",
    "#pickle.dump(indicators_values_up, ind_val)\n",
    "#pickle.dump(indicators_values_std_up, ind_val_std)\n",
    "pickle.dump(indicators_values_i_up, ind_val_i)\n",
    "#ind_val.close()\n",
    "#ind_val_std.close()\n",
    "ind_val_i.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining dictionaries for targets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = list(info['Target'].unique())\n",
    "\n",
    "dict_targets = {}\n",
    "\n",
    "for target in targets:\n",
    "    t = info['Indicator'].where(info['Target'] == target)\n",
    "\n",
    "    dict_targets[target] = [i for i in t if str(i) != 'nan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check\n",
    "list(indicators_values_i_up['Germany'].loc['1.1.1', '2000'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can simply average or concatenate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets_values_i = {}\n",
    "#targets_values_up = {}\n",
    "#targets_values_std_up = {}\n",
    "targets_values_i_up = {}    # for Granger-causality\n",
    "\n",
    "for country in countries:\n",
    "    \n",
    "    #targets_values_up[country] = pd.DataFrame(columns=period, index=targets)\n",
    "    #targets_values_std_up[country] = pd.DataFrame(columns=period, index=targets)\n",
    "    targets_values_i[country] = pd.DataFrame(columns=period, index=list(dict_targets.keys()))\n",
    "    targets_values_i_up[country] = pd.DataFrame(columns=period, index=list(dict_targets.keys()))\n",
    "    \n",
    "    for year in period:\n",
    "        \n",
    "        for target in list(dict_targets.keys()):\n",
    "            #list_indicators_values = []\n",
    "            #list_indicators_values_std = []\n",
    "            list_indicators_values_i = []\n",
    "            list_indicators_values_i_up = []\n",
    "    \n",
    "            for indicator in list(dict_targets[target]):\n",
    "                #list_indicators_values.append(indicators_values[country].loc[indicator, year])\n",
    "                #list_indicators_values_std.append(indicators_values_std[country].loc[indicator, year])\n",
    "                list_indicators_values_i.extend(indicators_values_i[country].loc[indicator, year])\n",
    "                list_indicators_values_i_up.extend(indicators_values_i_up[country].loc[indicator, year])\n",
    "    \n",
    "            #print(list_indicators_values_i)\n",
    "            \n",
    "            # 1. averaging\n",
    "            #targets_values_up[country].loc[target, year] = np.mean(list_indicators_values)\n",
    "            #targets_values_std_up[country].loc[target, year] = np.mean(list_indicators_values_std)\n",
    "            #targets_values_i_up[country].loc[target, year] = np.mean(list_indicators_values_i)\n",
    "            \n",
    "            # 2. concatenating\n",
    "            targets_values_i[country].loc[target, year] = list_indicators_values_i\n",
    "            targets_values_i_up[country].loc[target, year] = list_indicators_values_i_up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check (each goal should have list in cells with values for sub-indicators)\n",
    "targets_values_i_up['Germany'].loc['1.1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first inner parentheses in each cell contains the values from the first indicator; the second inner parentheses in each cell contains the values from the second indicator; etc. Each indicator represents a couple of sub-indicators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "#tar_val = open('utils/data/targets_values_up.pkl', 'wb')\n",
    "#tar_val_std = open('utils/data/targets_values_std_up.pkl', 'wb')\n",
    "tar_val_i = open('utils/data/targets_values_i.pkl', 'wb')\n",
    "tar_val_i_up = open('utils/data/targets_values_i_up.pkl', 'wb')\n",
    "#pickle.dump(targets_values_up, tar_val)\n",
    "#pickle.dump(targets_values_std_up, tar_val_std)\n",
    "pickle.dump(targets_values_i, tar_val_i)\n",
    "pickle.dump(targets_values_i_up, tar_val_i_up)\n",
    "#tar_val.close()\n",
    "#tar_val_std.close()\n",
    "tar_val_i.close()\n",
    "tar_val_i_up.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *(UN data set)* Averaging and concatenating target data to goal-level\n",
    "\n",
    "Defining dictionaries for goals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goals = list(info['Goal'].unique())\n",
    "\n",
    "dict_goals = {}\n",
    "\n",
    "for goal in goals:\n",
    "    g = info['Target'].where(info['Goal'] == goal)\n",
    "\n",
    "    dict_goals[goal] = [t for t in g if str(t) != 'nan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goals_values_i = {}\n",
    "#goals_values_up = {}\n",
    "#goals_values_std_up = {}\n",
    "goals_values_i_up = {}    # for Granger-causality\n",
    "\n",
    "for country in countries:\n",
    "    \n",
    "    #goals_values_up[country] = pd.DataFrame(columns=period, index=goals)\n",
    "    #goals_values_std_up[country] = pd.DataFrame(columns=period, index=goals)\n",
    "    goals_values_i[country] = pd.DataFrame(columns=period, index=list(dict_goals.keys()))\n",
    "    goals_values_i_up[country] = pd.DataFrame(columns=period, index=list(dict_goals.keys()))\n",
    "    \n",
    "    for year in period:\n",
    "        \n",
    "        for goal in goals:\n",
    "            #list_targets_values = []\n",
    "            #list_targets_values_std = []\n",
    "            list_targets_values_i = []\n",
    "            list_targets_values_i_up = []\n",
    "    \n",
    "            for target in list(dict_goals[goal]):\n",
    "                #list_targets_values.append(targets_values_up[country].loc[target, year])\n",
    "                #list_targets_values_std.append(targets_values_std_up[country].loc[target, year])\n",
    "                list_targets_values_i.extend(targets_values_i[country].loc[target, year])\n",
    "                list_targets_values_i_up.extend(targets_values_i_up[country].loc[target, year])\n",
    "    \n",
    "            #print(list_targets_values_i)\n",
    "            \n",
    "            # 1. averaging\n",
    "            #goals_values_up[country].loc[goal, year] = np.mean(list_targets_values)\n",
    "            #goals_values_std_up[country].loc[goal, year] = np.mean(list_targets_values_std)\n",
    "            #goals_values_i_up[country].loc[goal, year] = np.mean(list_targets_values_i)\n",
    "            \n",
    "            # 2. concatenating\n",
    "            goals_values_i[country].loc[goal, year] = list_targets_values_i\n",
    "            goals_values_i_up[country].loc[goal, year] = list_targets_values_i_up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check (each goal should have list in cells with values for sub-indicators)\n",
    "goals_values_i_up['Germany'].loc['13']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "#goa_val = open('utils/data/goals_values_up.pkl', 'wb')\n",
    "#goa_val_std = open('utils/data/goals_values_std_up.pkl', 'wb')\n",
    "goa_val_i = open('utils/data/goals_values_i.pkl', 'wb')\n",
    "goa_val_i_up = open('utils/data/goals_values_i_up.pkl', 'wb')\n",
    "#pickle.dump(goals_values_up, goa_val)\n",
    "#pickle.dump(goals_values_std_up, goa_val_std)\n",
    "pickle.dump(goals_values_i, goa_val_i)\n",
    "pickle.dump(goals_values_i_up, goa_val_i_up)\n",
    "#goa_val.close()\n",
    "#goa_val_std.close()\n",
    "goa_val_i.close()\n",
    "goa_val_i_up.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *(WorldBank data set)* Concatenating indicator data to target-level\n",
    "As we have done with the UN data set, we concatenate indicator data to target-level. We jump over the indicators, because sub-indicators are not mapped to specific indicators in the WorldBank data set, only to targets directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1901</th>\n",
       "      <th>1902</th>\n",
       "      <th>1903</th>\n",
       "      <th>1904</th>\n",
       "      <th>1905</th>\n",
       "      <th>1906</th>\n",
       "      <th>1907</th>\n",
       "      <th>1908</th>\n",
       "      <th>1909</th>\n",
       "      <th>1910</th>\n",
       "      <th>...</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "      <th>2017</th>\n",
       "      <th>2018</th>\n",
       "      <th>2019</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EG.CFT.ACCS.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.279600</td>\n",
       "      <td>0.506205</td>\n",
       "      <td>0.746543</td>\n",
       "      <td>1.033575</td>\n",
       "      <td>1.283527</td>\n",
       "      <td>1.573306</td>\n",
       "      <td>1.894672</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.695603</td>\n",
       "      <td>-0.675679</td>\n",
       "      <td>0.312007</td>\n",
       "      <td>0.305643</td>\n",
       "      <td>1.090614</td>\n",
       "      <td>0.403608</td>\n",
       "      <td>1.403584</td>\n",
       "      <td>1.403584</td>\n",
       "      <td>1.442255</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.RU.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.686047</td>\n",
       "      <td>-0.706401</td>\n",
       "      <td>0.279194</td>\n",
       "      <td>0.292840</td>\n",
       "      <td>1.087533</td>\n",
       "      <td>0.396553</td>\n",
       "      <td>1.421530</td>\n",
       "      <td>1.421297</td>\n",
       "      <td>1.458510</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EG.ELC.ACCS.UR.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.896915</td>\n",
       "      <td>-0.423962</td>\n",
       "      <td>0.634500</td>\n",
       "      <td>0.342497</td>\n",
       "      <td>1.098945</td>\n",
       "      <td>0.320685</td>\n",
       "      <td>1.199366</td>\n",
       "      <td>1.199366</td>\n",
       "      <td>1.262129</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FX.OWN.TOTL.ZS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.884230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.513711</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.397941</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_MTMP</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDAN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.737766</td>\n",
       "      <td>-0.675998</td>\n",
       "      <td>1.413764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDLN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VC_DSR_PDYN</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.535038</td>\n",
       "      <td>-0.866192</td>\n",
       "      <td>1.401230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Temperature</th>\n",
       "      <td>0.172491</td>\n",
       "      <td>0.407291</td>\n",
       "      <td>-1.322813</td>\n",
       "      <td>-0.655487</td>\n",
       "      <td>-0.976793</td>\n",
       "      <td>-0.235319</td>\n",
       "      <td>-1.199235</td>\n",
       "      <td>-0.606056</td>\n",
       "      <td>0.12306</td>\n",
       "      <td>-0.704919</td>\n",
       "      <td>...</td>\n",
       "      <td>1.939670</td>\n",
       "      <td>1.259986</td>\n",
       "      <td>0.320786</td>\n",
       "      <td>1.717227</td>\n",
       "      <td>0.802744</td>\n",
       "      <td>1.383564</td>\n",
       "      <td>2.186827</td>\n",
       "      <td>1.630722</td>\n",
       "      <td>2.273333</td>\n",
       "      <td>1.247628</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>400 rows × 119 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       1901      1902      1903      1904      1905      1906  \\\n",
       "EG.CFT.ACCS.ZS          NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.ZS          NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.RU.ZS       NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "EG.ELC.ACCS.UR.ZS       NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "FX.OWN.TOTL.ZS          NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "...                     ...       ...       ...       ...       ...       ...   \n",
       "VC_DSR_MTMP             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDAN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDLN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDYN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "Temperature        0.172491  0.407291 -1.322813 -0.655487 -0.976793 -0.235319   \n",
       "\n",
       "                       1907      1908     1909      1910  ...      2010  \\\n",
       "EG.CFT.ACCS.ZS          NaN       NaN      NaN       NaN  ...  0.279600   \n",
       "EG.ELC.ACCS.ZS          NaN       NaN      NaN       NaN  ... -0.695603   \n",
       "EG.ELC.ACCS.RU.ZS       NaN       NaN      NaN       NaN  ... -0.686047   \n",
       "EG.ELC.ACCS.UR.ZS       NaN       NaN      NaN       NaN  ... -0.896915   \n",
       "FX.OWN.TOTL.ZS          NaN       NaN      NaN       NaN  ...       NaN   \n",
       "...                     ...       ...      ...       ...  ...       ...   \n",
       "VC_DSR_MTMP             NaN       NaN      NaN       NaN  ...       NaN   \n",
       "VC_DSR_PDAN             NaN       NaN      NaN       NaN  ...       NaN   \n",
       "VC_DSR_PDLN             NaN       NaN      NaN       NaN  ...       NaN   \n",
       "VC_DSR_PDYN             NaN       NaN      NaN       NaN  ...       NaN   \n",
       "Temperature       -1.199235 -0.606056  0.12306 -0.704919  ...  1.939670   \n",
       "\n",
       "                       2011      2012      2013      2014      2015      2016  \\\n",
       "EG.CFT.ACCS.ZS     0.506205  0.746543  1.033575  1.283527  1.573306  1.894672   \n",
       "EG.ELC.ACCS.ZS    -0.675679  0.312007  0.305643  1.090614  0.403608  1.403584   \n",
       "EG.ELC.ACCS.RU.ZS -0.706401  0.279194  0.292840  1.087533  0.396553  1.421530   \n",
       "EG.ELC.ACCS.UR.ZS -0.423962  0.634500  0.342497  1.098945  0.320685  1.199366   \n",
       "FX.OWN.TOTL.ZS    -0.884230       NaN       NaN -0.513711       NaN       NaN   \n",
       "...                     ...       ...       ...       ...       ...       ...   \n",
       "VC_DSR_MTMP             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDAN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDLN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "VC_DSR_PDYN             NaN       NaN       NaN       NaN       NaN       NaN   \n",
       "Temperature        1.259986  0.320786  1.717227  0.802744  1.383564  2.186827   \n",
       "\n",
       "                       2017      2018      2019  \n",
       "EG.CFT.ACCS.ZS          NaN       NaN       NaN  \n",
       "EG.ELC.ACCS.ZS     1.403584  1.442255       NaN  \n",
       "EG.ELC.ACCS.RU.ZS  1.421297  1.458510       NaN  \n",
       "EG.ELC.ACCS.UR.ZS  1.199366  1.262129       NaN  \n",
       "FX.OWN.TOTL.ZS     1.397941       NaN       NaN  \n",
       "...                     ...       ...       ...  \n",
       "VC_DSR_MTMP       -1.000000       NaN  1.000000  \n",
       "VC_DSR_PDAN       -0.737766 -0.675998  1.413764  \n",
       "VC_DSR_PDLN       -1.000000  1.000000       NaN  \n",
       "VC_DSR_PDYN       -0.535038 -0.866192  1.401230  \n",
       "Temperature        1.630722  2.273333  1.247628  \n",
       "\n",
       "[400 rows x 119 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "dict_all_std['Afghanistan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHECKPOINT\n",
    "indicators_values_i_up = pickle.load(open('utils/data/indicators_values_i_up_wb.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making all time-series \"pointing\" upwards when they are meant to increase\n",
    "# not necessary when computing non-linear dependence\n",
    "\n",
    "indicators_values_i_up = {}\n",
    "indicators_values_std_up = {}\n",
    "\n",
    "for country in countries:\n",
    "    indicators_values_i_up[country] = pd.DataFrame(index=list(dict_all_i[country].index), columns=period)\n",
    "    indicators_values_std_up[country] = pd.DataFrame(index=list(dict_all_std[country].index), columns=period)\n",
    "    \n",
    "    for seriescode in list(dict_all_std['France'].index):\n",
    "        #indicators_values_i_up[country].at[seriescode] = list(np.multiply(list(dict_all_i[country].loc[seriescode]), int(info.loc[info[0]==seriescode][5])))\n",
    "        # leaving them as they are to investigate dependencies between indicators (direction does not matter then)\n",
    "        indicators_values_i_up[country].at[seriescode] = list(dict_all_i[country].loc[seriescode])\n",
    "        indicators_values_std_up[country].at[seriescode] = list(dict_all_std[country].loc[seriescode][period])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000          NaN\n",
       "2001          NaN\n",
       "2002    0.0832496\n",
       "2003          NaN\n",
       "2004          NaN\n",
       "2005          NaN\n",
       "2006          NaN\n",
       "2007    -0.331485\n",
       "2008          NaN\n",
       "2009          NaN\n",
       "2010          NaN\n",
       "2011          NaN\n",
       "2012    -0.688701\n",
       "2013          NaN\n",
       "2014    -0.688701\n",
       "2015          NaN\n",
       "2016          NaN\n",
       "2017          NaN\n",
       "2018          NaN\n",
       "2019          NaN\n",
       "Name: ER.H2O.FWTL.ZS, dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "indicators_values_std_up['France'].loc['ER.H2O.FWTL.ZS']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious data\n",
    "ind_val_i = open('utils/data/indicators_values_i_up_wb.pkl', 'wb')\n",
    "pickle.dump(indicators_values_i_up, ind_val_i)\n",
    "ind_val_i.close()\n",
    "\n",
    "ind_val_std = open('utils/data/indicators_values_std_up_wb.pkl', 'wb')\n",
    "pickle.dump(indicators_values_std_up, ind_val_std)\n",
    "ind_val_std.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining dictionaries for targets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = list(info[4].unique())\n",
    "\n",
    "dict_targets = {}\n",
    "\n",
    "for target in targets:\n",
    "    t = info[0].where(info[4] == target)\n",
    "\n",
    "    dict_targets[target] = [i for i in t if str(i) != 'nan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EN.CLC.MDAT.ZS',\n",
       " 'SG_DSR_SILN',\n",
       " 'SG_DSR_SILS',\n",
       " 'SG_GOV_LOGV',\n",
       " 'VC_DSR_AFFCT',\n",
       " 'VC_DSR_DAFF',\n",
       " 'VC_DSR_IJILN',\n",
       " 'VC_DSR_MISS',\n",
       " 'VC_DSR_MORT',\n",
       " 'VC_DSR_MTMN',\n",
       " 'VC_DSR_MTMP',\n",
       " 'VC_DSR_PDAN',\n",
       " 'VC_DSR_PDLN',\n",
       " 'VC_DSR_PDYN']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "dict_targets['13.1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Target values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0c07156bd8b473b8bc5c2314b4521cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=180.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# with imputed values\n",
    "\n",
    "targets_values_i_up = {}\n",
    "targets_values_i_up_arr = {}\n",
    "targets_values_i_up_avg = {}\n",
    "\n",
    "for country in tqdm(countries):\n",
    "    targets_values_i_up_arr[country] = []\n",
    "    targets_values_i_up_avg[country] = pd.DataFrame(index=targets, columns=period)\n",
    "    \n",
    "    for t, target in enumerate(list(dict_targets.keys())):\n",
    "        list_indicators_values_i_up_arr = []\n",
    "\n",
    "        for y, year in enumerate(period):\n",
    "            list_indicators_values_i_up_avg = []\n",
    "            for i, indicator in enumerate(list(dict_targets[target])):\n",
    "                # do not append NaNs                  \n",
    "                if np.isnan(indicators_values_i_up[country].loc[indicator, year])==False:\n",
    "                    list_indicators_values_i_up_arr.append(indicators_values_i_up[country].loc[indicator, year])\n",
    "                    list_indicators_values_i_up_avg.append(indicators_values_i_up[country].loc[indicator, year])\n",
    "            \n",
    "            # 1. averaging\n",
    "            targets_values_i_up_avg[country].loc[target, year] = np.mean(list_indicators_values_i_up_avg)\n",
    "\n",
    "        # 2. concatenating\n",
    "        targets_values_i_up_arr[country].append(list_indicators_values_i_up_arr)\n",
    "    \n",
    "    targets_values_i_up[country] = pd.DataFrame(data=targets_values_i_up_arr[country], index=list(dict_targets.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1319d82f28e45adb52b83bd26c72d4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=180.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# with standardised values\n",
    "\n",
    "targets_values_std_up = {}\n",
    "targets_values_std_up_arr = {}\n",
    "targets_values_std_up_avg = {}\n",
    "\n",
    "for country in tqdm(countries):\n",
    "    targets_values_std_up_arr[country] = []\n",
    "    targets_values_std_up_avg[country] = pd.DataFrame(index=targets, columns=period)\n",
    "    \n",
    "    for t, target in enumerate(list(dict_targets.keys())):\n",
    "        list_indicators_values_std_up_arr = []\n",
    "\n",
    "        for y, year in enumerate(period):\n",
    "            list_indicators_values_std_up_avg = []\n",
    "            for i, indicator in enumerate(list(dict_targets[target])):\n",
    "                # do not append NaNs                  \n",
    "                if np.isnan(indicators_values_std_up[country].loc[indicator, year])==False:\n",
    "                    list_indicators_values_std_up_arr.append(indicators_values_std_up[country].loc[indicator, year])\n",
    "                    list_indicators_values_std_up_avg.append(indicators_values_std_up[country].loc[indicator, year])\n",
    "            \n",
    "            # 1. averaging\n",
    "            targets_values_std_up_avg[country].loc[target, year] = np.mean(list_indicators_values_std_up_avg)\n",
    "\n",
    "        # 2. concatenating\n",
    "        targets_values_std_up_arr[country].append(list_indicators_values_std_up_arr)\n",
    "    \n",
    "    targets_values_std_up[country] = pd.DataFrame(data=targets_values_std_up_arr[country], index=list(dict_targets.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0870694138599098\n",
      "0.0870694138599098\n"
     ]
    }
   ],
   "source": [
    "# check whether averages were correctly computed\n",
    "print(targets_values_i_up_avg['France'].loc['2.1', '2017'])\n",
    "print(np.mean(indicators_values_i_up['France'].loc[info[info[4]=='2.1'][0]]['2017']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.3 18\n",
      "1.1 5\n",
      "1.2 7\n",
      "2.3 18\n",
      "2.2 71\n",
      "2.1 29\n",
      "3.5 6\n",
      "3.3 136\n",
      "3.2 80\n",
      "3.4 30\n",
      "3.7 21\n",
      "3.8 48\n",
      "3.c 30\n",
      "3.a 18\n",
      "3.9 19\n",
      "3.1 35\n",
      "3.6 2\n",
      "4.6 70\n",
      "4.1 371\n",
      "4.5 34\n",
      "4.2 76\n",
      "4.c 163\n",
      "4.4 167\n",
      "4.3 39\n",
      "5.5 24\n",
      "5.6 0\n",
      "5.1 22\n",
      "5.4 42\n",
      "5.2 1\n",
      "5.3 4\n",
      "6.4 26\n",
      "6.1 72\n",
      "6.2 130\n",
      "7.1 74\n",
      "7.3 16\n",
      "7.2 32\n",
      "8.1 79\n",
      "8.3 11\n",
      "8.2 260\n",
      "8.5 282\n",
      "8.7 6\n",
      "8.6 0\n",
      "9.4 68\n",
      "9.5 19\n",
      "9.1 74\n",
      "9.2 40\n",
      "9.b 19\n",
      "10.b 38\n",
      "10.2 0\n",
      "10.c 7\n",
      "10.1 0\n",
      "11.6 50\n",
      "11.1 60\n",
      "12.2 133\n",
      "13.2 0\n",
      "13.1 1\n",
      "14.4 51\n",
      "14.5 3\n",
      "15.1 40\n",
      "15.5 4\n",
      "16.6 13\n",
      "16.5 6\n",
      "16.9 5\n",
      "16.1 40\n",
      "17.3 60\n",
      "17.2 19\n",
      "17.4 19\n",
      "17.1 22\n",
      "17.17 2\n",
      "17.6 32\n",
      "17.19 48\n",
      "17.18 16\n",
      "17.8 17\n",
      "17.11 20\n",
      "17.13 447\n",
      "17.12 60\n",
      "T 20\n"
     ]
    }
   ],
   "source": [
    "# check for std\n",
    "for t, target in enumerate(targets):\n",
    "    print(target, len(targets_values_std_up_arr['Azerbaijan'][t]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.3 324\n",
      "1.1 19\n",
      "1.2 92\n",
      "2.3 18\n",
      "2.2 317\n",
      "2.1 45\n",
      "3.5 6\n",
      "3.3 138\n",
      "3.2 80\n",
      "3.4 30\n",
      "3.7 38\n",
      "3.8 78\n",
      "3.c 38\n",
      "3.a 18\n",
      "3.9 19\n",
      "3.1 37\n",
      "3.6 2\n",
      "4.6 133\n",
      "4.1 420\n",
      "4.5 80\n",
      "4.2 80\n",
      "4.c 420\n",
      "4.4 376\n",
      "4.3 60\n",
      "5.5 69\n",
      "5.6 13\n",
      "5.1 25\n",
      "5.4 76\n",
      "5.2 14\n",
      "5.3 57\n",
      "6.4 97\n",
      "6.1 108\n",
      "6.2 216\n",
      "7.1 74\n",
      "7.3 16\n",
      "7.2 32\n",
      "8.1 82\n",
      "8.3 73\n",
      "8.2 260\n",
      "8.5 300\n",
      "8.7 51\n",
      "8.6 60\n",
      "9.4 68\n",
      "9.5 38\n",
      "9.1 76\n",
      "9.2 40\n",
      "9.b 19\n",
      "10.b 38\n",
      "10.2 1\n",
      "10.c 14\n",
      "10.1 8\n",
      "11.6 50\n",
      "11.1 65\n",
      "12.2 133\n",
      "13.2 1\n",
      "13.1 166\n",
      "14.4 51\n",
      "14.5 3\n",
      "15.1 40\n",
      "15.5 4\n",
      "16.6 16\n",
      "16.5 32\n",
      "16.9 89\n",
      "16.1 76\n",
      "17.3 60\n",
      "17.2 19\n",
      "17.4 19\n",
      "17.1 40\n",
      "17.17 60\n",
      "17.6 38\n",
      "17.19 48\n",
      "17.18 16\n",
      "17.8 20\n",
      "17.11 20\n",
      "17.13 520\n",
      "17.12 114\n",
      "T 20\n"
     ]
    }
   ],
   "source": [
    "# check for i\n",
    "for t, target in enumerate(targets):\n",
    "    print(target, len(targets_values_i_up_arr['Azerbaijan'][t]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious imputed data\n",
    "tar_val_i_up = open('utils/data/targets_values_i_up_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_i_up, tar_val_i_up)\n",
    "tar_val_i_up.close()\n",
    "\n",
    "tar_val_i_up_arr = open('utils/data/targets_values_i_up_arr_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_i_up_arr, tar_val_i_up_arr)\n",
    "tar_val_i_up_arr.close()\n",
    "\n",
    "tar_val_i_up_avg = open('utils/data/targets_values_i_up_avg_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_i_up_avg, tar_val_i_up_avg)\n",
    "tar_val_i_up_avg.close()\n",
    "\n",
    "# better save these precious standardised data\n",
    "tar_val_std_up = open('utils/data/targets_values_std_up_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_std_up, tar_val_std_up)\n",
    "tar_val_std_up.close()\n",
    "\n",
    "tar_val_std_up_arr = open('utils/data/targets_values_std_up_arr_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_std_up_arr, tar_val_std_up_arr)\n",
    "tar_val_std_up_arr.close()\n",
    "\n",
    "tar_val_std_up_avg = open('utils/data/targets_values_std_up_avg_wb.pkl', 'wb')\n",
    "pickle.dump(targets_values_std_up_avg, tar_val_std_up_avg)\n",
    "tar_val_std_up_avg.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *(WorldBank data set)* Concatenating target data to goal-level\n",
    "\n",
    "Defining dictionaries for goals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "goals = list(info[3].unique())\n",
    "\n",
    "dict_goals = {}\n",
    "\n",
    "for goal in goals:\n",
    "    g = info[4].where(info[3] == goal)\n",
    "\n",
    "    dict_goals[goal] = [t for t in g if str(t) != 'nan']\n",
    "    dict_goals[goal] = list(set(dict_goals[goal]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1.1', '1.3', '1.2']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check\n",
    "dict_goals['1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenating:\n",
    "\n",
    "Recall that we define our dimensionality as $d \\times T$, and countries are independent samples. We do this in the next cell exemplarily by appending the years for all indicators of SDG 1 to a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "435"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example with SDG 1\n",
    "len([x for x in list(targets_values_i_up[country].loc['1.1']) if str(x) != 'nan']) + len([x for x in list(targets_values_i_up[country].loc['1.2']) if str(x) != 'nan']) + len([x for x in list(targets_values_i_up[country].loc['1.3']) if str(x) != 'nan'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab328968f58d44b5b76e13c2ce4cad86",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=180.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "goals_values_i_up = {}\n",
    "goals_values_i_up_arr = {}\n",
    "goals_values_i_up_avg = {}\n",
    "\n",
    "for country in tqdm(countries):\n",
    "    goals_values_i_up_arr[country] = []\n",
    "    goals_values_i_up_avg[country] = []   # define this list with target values being averages over indicators\n",
    "    \n",
    "    for g, goal in enumerate(list(dict_goals.keys())):\n",
    "        list_targets_values_i_up = []\n",
    "        list_targets_values_i_up_avg = []\n",
    "\n",
    "        for t in dict_goals[goal]:   # do not append NaN's\n",
    "            list_targets_values_i_up.extend([x for x in list(targets_values_i_up[country].loc[t]) if np.isnan(x)==False])\n",
    "            list_targets_values_i_up_avg.extend([x for x in list(targets_values_i_up_avg[country].loc[t]) if np.isnan(x)==False])\n",
    "            \n",
    "        # 1. append target averages\n",
    "        goals_values_i_up_avg[country].append(np.asarray(list_targets_values_i_up_avg))\n",
    "\n",
    "        # 2. concatenating\n",
    "        goals_values_i_up_arr[country].append(np.asarray(list_targets_values_i_up))\n",
    "    \n",
    "    goals_values_i_up_avg[country] = np.asarray(goals_values_i_up_avg[country])\n",
    "    goals_values_i_up[country] = pd.DataFrame(data=goals_values_i_up_arr[country], index=list(dict_goals.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "430b942c1aac42fcb02822c22daac3bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=180.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "goals_values_std_up = {}\n",
    "goals_values_std_up_arr = {}\n",
    "goals_values_std_up_avg = {}\n",
    "\n",
    "for country in tqdm(countries):\n",
    "    goals_values_std_up_arr[country] = []\n",
    "    goals_values_std_up_avg[country] = []   # define this list with target values being averages over indicators\n",
    "    \n",
    "    for g, goal in enumerate(list(dict_goals.keys())):\n",
    "        list_targets_values_std_up = []\n",
    "        list_targets_values_std_up_avg = []\n",
    "\n",
    "        for t in dict_goals[goal]:   # do not append NaN's\n",
    "            list_targets_values_std_up.extend([x for x in list(targets_values_std_up[country].loc[t]) if np.isnan(x)==False])\n",
    "            list_targets_values_std_up_avg.extend([x for x in list(targets_values_std_up_avg[country].loc[t]) if np.isnan(x)==False])\n",
    "            \n",
    "        # 1. append target averages\n",
    "        goals_values_std_up_avg[country].append(np.asarray(list_targets_values_std_up_avg))\n",
    "\n",
    "        # 2. concatenating\n",
    "        goals_values_std_up_arr[country].append(np.asarray(list_targets_values_std_up))\n",
    "    \n",
    "    goals_values_std_up_avg[country] = np.asarray(goals_values_std_up_avg[country])\n",
    "    goals_values_std_up[country] = pd.DataFrame(data=goals_values_std_up_arr[country], index=list(dict_goals.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18,)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# concatenating target averages\n",
    "goals_values_std_up_avg['France'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 39\n",
      "2 140\n",
      "3 414\n",
      "4 715\n",
      "5 96\n",
      "6 227\n",
      "7 120\n",
      "8 516\n",
      "9 195\n",
      "10 45\n",
      "11 115\n",
      "12 128\n",
      "13 114\n",
      "14 54\n",
      "15 44\n",
      "16 39\n",
      "17 855\n",
      "T 20\n"
     ]
    }
   ],
   "source": [
    "# check for std\n",
    "for g, goal in enumerate(list(dict_goals.keys())):\n",
    "    print(goal, len(goals_values_std_up_arr['Mozambique'][g]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 435\n",
      "2 380\n",
      "3 484\n",
      "4 1569\n",
      "5 254\n",
      "6 421\n",
      "7 122\n",
      "8 826\n",
      "9 241\n",
      "10 61\n",
      "11 115\n",
      "12 133\n",
      "13 167\n",
      "14 54\n",
      "15 44\n",
      "16 213\n",
      "17 974\n",
      "T 20\n"
     ]
    }
   ],
   "source": [
    "# check for i\n",
    "for g, goal in enumerate(list(dict_goals.keys())):\n",
    "    print(goal, len(goals_values_i_up_arr['France'][g]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage from saving these numbers in arrays and not dataframes should be clear now: in arrays we only save the actual numbers, but dataframes have a fixed number of columns, hence add NaN's to the goals which do not have data available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# better save these precious imputed data\n",
    "goa_val_i_up = open('utils/data/goals_values_i_up_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_i_up, goa_val_i_up)\n",
    "goa_val_i_up.close()\n",
    "\n",
    "goa_val_i_up_arr = open('utils/data/goals_values_i_up_arr_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_i_up_arr, goa_val_i_up_arr)\n",
    "goa_val_i_up_arr.close()\n",
    "\n",
    "goa_val_i_up_avg = open('utils/data/goals_values_i_up_avg_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_i_up_avg, goa_val_i_up_avg)\n",
    "goa_val_i_up_avg.close()\n",
    "\n",
    "# better save these precious standardised data\n",
    "goa_val_std_up = open('utils/data/goals_values_std_up_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_std_up, goa_val_std_up)\n",
    "goa_val_std_up.close()\n",
    "\n",
    "goa_val_std_up_arr = open('utils/data/goals_values_std_up_arr_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_std_up_arr, goa_val_std_up_arr)\n",
    "goa_val_std_up_arr.close()\n",
    "\n",
    "goa_val_std_up_avg = open('utils/data/goals_values_std_up_avg_wb.pkl', 'wb')\n",
    "pickle.dump(goals_values_std_up_avg, goa_val_std_up_avg)\n",
    "goa_val_std_up_avg.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 - Spark (local)",
   "language": "python",
   "name": "spark-3-python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}