understand-citoid-coverage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understand Citoid coverage for Web2Cit\n",
    "\n",
    "This notebook describes how to retrieve the source code and the [Citation templates](https://en.wikipedia.org/wiki/Wikipedia:Citation_templates) of the citations of a sample of Wikipedia articles to compare them with the results of the [Citoid](https://www.mediawiki.org/wiki/Citoid) API for the same citations.\n",
    "\n",
    "To follow along, we recommend running the script portions piecemeal, in order.\n",
    "\n",
    "__Author:__\n",
    "\n",
    "* Nidia Hernández, [nidiahernandez@conicet.gov.ar](mailto:nidiahy@gmail.com), [CAICYT-CONICET](http://www.caicyt-conicet.gov.ar/sitio/) (Argentina)\n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "0. Setting Up \n",
    "1. Retrieving data from Wikipedia articles\n",
    "    - 1.1. Fetching featured articles using Mediawiki's action API\n",
    "    - 1.2. Retrieving featured articles data\n",
    "2. Citation template metadata extraction\n",
    "    - 2.1 Parameter name mapping\n",
    "    - 2.2 Inspecting all featured articles\n",
    "    - 2.3 Citation template extraction summary\n",
    "3. Querying Citoid API\n",
    "    - 3.1 Validating URLs\n",
    "    - 3.2 Query function and parallel requests\n",
    "    - 3.3 Tidying up Citoid's response\n",
    "4. Data normalization\n",
    "    - 4.1 Manual data\n",
    "    - 4.2 Citoid data\n",
    "5. Evaluation\n",
    "    - 5.1 Source type\n",
    "    - 5.2 Publishing date\n",
    "    - 5.3 Title and publishing source\n",
    "    - 5.4 Author\n",
    "    - 5.5 Applying comparison methods\n",
    "6. Results\n",
    "    - 6.1. By URL\n",
    "    - 6.2. By domain\n",
    "    - 6.3 By fieldname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Setting Up\n",
    "\n",
    "Before we get started, let's install and import the libraries that we will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/yoki/anaconda3/lib/python3.7/site-packages/tqdm/std.py:668: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  from pandas import Panel\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /home/yoki/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import os\n",
    "import requests\n",
    "import urllib\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from operator import itemgetter\n",
    "import json\n",
    "import gzip\n",
    "from glob import glob\n",
    "from pprint import pprint\n",
    "from tqdm import tqdm\n",
    "tqdm.pandas()\n",
    "import mwparserfromhell\n",
    "import validators\n",
    "from datetime import datetime\n",
    "import concurrent.futures\n",
    "from time import sleep\n",
    "from more_itertools import chunked\n",
    "from unidecode import unidecode\n",
    "from ast import literal_eval\n",
    "import dateparser\n",
    "from nltk.metrics import edit_distance\n",
    "from itertools import permutations\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we set the global parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HEADER={'User-Agent': 'https://phabricator.wikimedia.org/tag/web2cit-research/; mailto:nidiahernandez@conicet.gov.ar'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1. Retrieving data from Wikipedia articles\n",
    "\n",
    "In this section, we will fetch all the featured articles from a selection of Wikipedias ir order to find:\n",
    "\n",
    "- the citation templates used on each article\n",
    "- the URLs of the citations\n",
    "\n",
    "The selected Wikipedias are the following:\n",
    "\n",
    "- [English Wikipedia](https://en.wikipedia.org/wiki/Category:Featured_articles): ~6k featured articles\n",
    "- [Spanish Wikipedia](https://es.wikipedia.org/wiki/Categor%C3%ADa:Wikipedia:Art%C3%ADculos_destacados): ~1.2k artículos destacados\n",
    "- [French Wikipedia](https://fr.wikipedia.org/wiki/Cat%C3%A9gorie:Article_de_qualit%C3%A9): ~2k articles de qualité\n",
    "- [Portuguese Wikipedia](https://pt.wikipedia.org/wiki/Categoria:!Artigos_destacados): ~1.3k artigos destacados\n",
    "\n",
    "Some general information about featured articles in all Wikipedias: https://meta.wikimedia.org/wiki/Wikipedia_featured_articles (out of date).\n",
    "\n",
    "#### Featured content endpoint in Mediawiki's API\n",
    "\n",
    "There is a specific endpoint for featured content in Mediawiki's API:\n",
    "\n",
    "https://api.wikimedia.org/feed/v1/wikipedia/{lang}/featured/\n",
    "\n",
    "For example, the request https://api.wikimedia.org/feed/v1/wikipedia/en/featured/2021/12/06 returns today's featured article (tfa) for December 6th 2021 in English Wikipedia.\n",
    "\n",
    "Unfortunatatly, this is not avalilable for all the Wikipedias of our interest because \"not all Wikipedias are integrated into the Feed endpoint\" ([Feed endpoint doc](https://api.wikimedia.org/wiki/API_reference/Feed/Featured_content)). This way, the request https://api.wikimedia.org/feed/v1/wikipedia/fr/featured/2021/12/06 does not return tfa info. Same for Spanish and Portuguese. For this reason, we do not use this specific endpoint.\n",
    "\n",
    "### 1.1 Fetching featured articles using Mediawiki's action API\n",
    "\n",
    "We will use each [Wikipedias' action API](https://www.mediawiki.org/wiki/API:Main_page) to retrive:\n",
    "\n",
    "1. the list of all featured articles for each language\n",
    "2. the wikicode and some metadata of each featured article\n",
    "\n",
    "The action API has the following URL:\n",
    "\n",
    "`https://{lang}.wikipedia.org/w/api.php?`\n",
    "\n",
    "The following request, for example, retrieves the first 15 featured articles for the category \"Articles de qualité\" in French Wikipedia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title = urllib.parse.unquote(\"Cat%C3%A9gorie:Article_de_qualit%C3%A9\") # category name for featured articles in French Wikipedia\n",
    "\n",
    "PARAMS = {\n",
    "    \"action\": \"query\",\n",
    "    \"list\": \"categorymembers\",\n",
    "    \"cmtitle\": title,\n",
    "    \"cmlimit\": 15,\n",
    "    \"format\": \"json\"\n",
    "}\n",
    "\n",
    "response = requests.get(\n",
    "    url='https://fr.wikipedia.org/w/api.php?', \n",
    "    params=PARAMS\n",
    ")\n",
    "\n",
    "response_data = response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a max of 500 per request. In order to retrieve all the members of the category, we should add the parameter `cmcontinue` to our request as follows: we get the value of `cmcontinue` from the response of the first request and pass it to the following request. This repeats until the end of the list. See API doc for more details on listing category members: https://www.mediawiki.org/wiki/API:Categorymembers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'page|2a2e38324232443a30324e01428844880901dc10|52519'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response_data['continue']['cmcontinue'] # here is the reference of the starting point for the next request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the response, we obtain a list showing the pageid and the title of each article:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'pageid': 507899, 'ns': 4, 'title': 'Wikipédia:Articles de qualité'},\n",
       " {'pageid': 9253139, 'ns': 0, 'title': '1 000 kilomètres de Spa 2009'},\n",
       " {'pageid': 10267261, 'ns': 0, 'title': '6 Heures du Castellet 2011'},\n",
       " {'pageid': 91996, 'ns': 0, 'title': '32X'},\n",
       " {'pageid': 605584, 'ns': 0, 'title': 'Les 101 Dalmatiens (film, 1961)'},\n",
       " {'pageid': 5008651, 'ns': 0, 'title': 'A Different Corner'},\n",
       " {'pageid': 639219, 'ns': 0, 'title': \"A Hard Day's Night (album)\"},\n",
       " {'pageid': 6672844, 'ns': 0, 'title': \"À l'Olympia (album d'Alan Stivell)\"},\n",
       " {'pageid': 124634, 'ns': 0, 'title': 'À la croisée des mondes'},\n",
       " {'pageid': 1486250, 'ns': 0, 'title': \"L'Abbaye de Northanger\"},\n",
       " {'pageid': 11085052, 'ns': 0, 'title': 'Abbaye Saint-Paul de Cormery'},\n",
       " {'pageid': 1210952, 'ns': 0, 'title': 'Abbaye Saint-Victor de Marseille'},\n",
       " {'pageid': 4293975, 'ns': 0, 'title': \"Parc national d'Abisko\"},\n",
       " {'pageid': 1415801, 'ns': 0, 'title': 'Acanthaster planci'},\n",
       " {'pageid': 3906668,\n",
       "  'ns': 0,\n",
       "  'title': 'Accident sur la base de Fairchild en 1994'}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response_data['query']['categorymembers']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the item is a featured article, `ns` (namespace) is `0`. So, to avoid retriving other items (meta-pages about other categories, for example) elements having other values for this key must be discarded (cf. Retrieve article data below).\n",
    "\n",
    "Let's request all the featured articles for each language and dump the response to a json:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the name of the category in each language\n",
    "catname_bylang = {\n",
    "    'en': 'Category:Featured_articles',\n",
    "    'es': 'Categor%C3%ADa:Wikipedia:Art%C3%ADculos_destacados',\n",
    "    'fr': 'Cat%C3%A9gorie:Article_de_qualit%C3%A9',\n",
    "    'pt': 'Categoria:!Artigos_destacados',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrieving featured articles: 100%|██████████| 4/4 [00:00<00:00, 176.40it/s]\n"
     ]
    }
   ],
   "source": [
    "for lang in tqdm(catname_bylang, desc='Retrieving featured articles'):\n",
    "    \n",
    "    MEDIAWIKI_API_URL = f'https://{lang}.wikipedia.org/w/api.php?'\n",
    "    \n",
    "    category_title = urllib.parse.unquote(catname_bylang[lang])\n",
    "    cmcont = \"start\"\n",
    "    i = 0\n",
    "    \n",
    "    while (cmcont == \"start\") or cmcont.startswith('page'):\n",
    "        i += 1\n",
    "        filename = f'data/featured-articles/{lang}-{i:02}.json'\n",
    "        \n",
    "        if os.path.isfile(filename):\n",
    "            with open(filename) as fi:\n",
    "                featured_art_json = json.load(fi)\n",
    "                if 'continue' in featured_art_json:\n",
    "                    cmcont = featured_art_json['continue']['cmcontinue']\n",
    "                else:\n",
    "                    cmcont = \"end\"\n",
    "\n",
    "            continue\n",
    "        \n",
    "        PARAMS = {\n",
    "            \"action\": \"query\",\n",
    "            \"list\": \"categorymembers\",\n",
    "            \"cmtitle\": category_title,\n",
    "            \"cmlimit\": \"max\", \n",
    "            \"format\": \"json\",\n",
    "            \"curtimestamp\": True\n",
    "        }\n",
    "            \n",
    "        if cmcont.startswith('page'):\n",
    "            PARAMS[\"cmcontinue\"]=cmcont\n",
    "\n",
    "        response = requests.get(\n",
    "            url=MEDIAWIKI_API_URL, \n",
    "            params=PARAMS,\n",
    "            headers=HEADER\n",
    "        )\n",
    "\n",
    "        data = response.json()\n",
    "\n",
    "        if response.status_code == 200:\n",
    "\n",
    "            if 'continue' in data:\n",
    "                cmcont = data['continue']['cmcontinue']\n",
    "            \n",
    "            else:\n",
    "                cmcont='categoryend'\n",
    "\n",
    "            with open(filename, 'w') as fo:\n",
    "                fo.write(json.dumps(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Retrieve featured articles data\n",
    "\n",
    "Now that we have the pageids of all the featured articles, we can use them to retrieve the URL, the content and the id of the last edition (`revid`). First, we load the information from the json and we build a dataframe where we will store the data for each featured article:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = []\n",
    "\n",
    "for fname in sorted(os.listdir('featured-articles')):\n",
    "    if fname.endswith('.json'):\n",
    "        \n",
    "        wikilang = fname.split('-')[0]\n",
    "        \n",
    "        with open(f'featured-articles/{fname}') as fi:\n",
    "            featured_arts = json.load(fi)\n",
    "            \n",
    "        df = pd.DataFrame(featured_arts['query']['categorymembers'])\n",
    "        df['wiki_lang'] = wikilang\n",
    "        df = df[df['ns'] == 0] # discard items that are not featured articles\n",
    "        df.drop('ns', axis='columns', inplace = True)\n",
    "\n",
    "        dfs.append(df)\n",
    "    \n",
    "articles_data = pd.concat(dfs, ignore_index= True)\n",
    "articles_data.rename(columns={'title':'article_title'}, inplace = True)\n",
    "articles_data.drop_duplicates(inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pageid</th>\n",
       "      <th>article_title</th>\n",
       "      <th>wiki_lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33653136</td>\n",
       "      <td>? (film)</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1849799</td>\n",
       "      <td>0.999...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9702578</td>\n",
       "      <td>1 − 2 + 3 − 4 + ⋯</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48723612</td>\n",
       "      <td>1st Cavalry Division (Kingdom of Yugoslavia)</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>64662351</td>\n",
       "      <td>1st Missouri Field Battery</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15065</th>\n",
       "      <td>132185</td>\n",
       "      <td>Yorkshire terrier</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15066</th>\n",
       "      <td>2269483</td>\n",
       "      <td>You Belong with Me</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15067</th>\n",
       "      <td>4779921</td>\n",
       "      <td>You Don't Know What to Do</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15068</th>\n",
       "      <td>1585284</td>\n",
       "      <td>You Know I'm No Good</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15069</th>\n",
       "      <td>2929652</td>\n",
       "      <td>Álvaro de Zúñiga y Guzmán</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10570 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         pageid                                 article_title wiki_lang\n",
       "0      33653136                                      ? (film)        en\n",
       "1       1849799                                      0.999...        en\n",
       "2       9702578                             1 − 2 + 3 − 4 + ⋯        en\n",
       "3      48723612  1st Cavalry Division (Kingdom of Yugoslavia)        en\n",
       "4      64662351                    1st Missouri Field Battery        en\n",
       "...         ...                                           ...       ...\n",
       "15065    132185                             Yorkshire terrier        pt\n",
       "15066   2269483                            You Belong with Me        pt\n",
       "15067   4779921                     You Don't Know What to Do        pt\n",
       "15068   1585284                          You Know I'm No Good        pt\n",
       "15069   2929652                     Álvaro de Zúñiga y Guzmán        pt\n",
       "\n",
       "[10570 rows x 3 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add a new column containing the URL of each article using the information from the article title and the language of the wikipedia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_data['article_url'] = \"https://\"+articles_data['wiki_lang']+\".wikipedia.org/wiki/\"+articles_data['article_title'].map(urllib.parse.quote)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pageid</th>\n",
       "      <th>article_title</th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>article_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33653136</td>\n",
       "      <td>? (film)</td>\n",
       "      <td>en</td>\n",
       "      <td>https://en.wikipedia.org/wiki/%3F%20%28film%29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1849799</td>\n",
       "      <td>0.999...</td>\n",
       "      <td>en</td>\n",
       "      <td>https://en.wikipedia.org/wiki/0.999...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9702578</td>\n",
       "      <td>1 − 2 + 3 − 4 + ⋯</td>\n",
       "      <td>en</td>\n",
       "      <td>https://en.wikipedia.org/wiki/1%20%E2%88%92%202%20%2B%203%20%E2%88%92%204%20%2B%20%E2%8B%AF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48723612</td>\n",
       "      <td>1st Cavalry Division (Kingdom of Yugoslavia)</td>\n",
       "      <td>en</td>\n",
       "      <td>https://en.wikipedia.org/wiki/1st%20Cavalry%20Division%20%28Kingdom%20of%20Yugoslavia%29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>64662351</td>\n",
       "      <td>1st Missouri Field Battery</td>\n",
       "      <td>en</td>\n",
       "      <td>https://en.wikipedia.org/wiki/1st%20Missouri%20Field%20Battery</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15065</th>\n",
       "      <td>132185</td>\n",
       "      <td>Yorkshire terrier</td>\n",
       "      <td>pt</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/Yorkshire%20terrier</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15066</th>\n",
       "      <td>2269483</td>\n",
       "      <td>You Belong with Me</td>\n",
       "      <td>pt</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/You%20Belong%20with%20Me</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15067</th>\n",
       "      <td>4779921</td>\n",
       "      <td>You Don't Know What to Do</td>\n",
       "      <td>pt</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/You%20Don%27t%20Know%20What%20to%20Do</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15068</th>\n",
       "      <td>1585284</td>\n",
       "      <td>You Know I'm No Good</td>\n",
       "      <td>pt</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/You%20Know%20I%27m%20No%20Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15069</th>\n",
       "      <td>2929652</td>\n",
       "      <td>Álvaro de Zúñiga y Guzmán</td>\n",
       "      <td>pt</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/%C3%81lvaro%20de%20Z%C3%BA%C3%B1iga%20y%20Guzm%C3%A1n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10570 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         pageid                                 article_title wiki_lang  \\\n",
       "0      33653136                                      ? (film)        en   \n",
       "1       1849799                                      0.999...        en   \n",
       "2       9702578                             1 − 2 + 3 − 4 + ⋯        en   \n",
       "3      48723612  1st Cavalry Division (Kingdom of Yugoslavia)        en   \n",
       "4      64662351                    1st Missouri Field Battery        en   \n",
       "...         ...                                           ...       ...   \n",
       "15065    132185                             Yorkshire terrier        pt   \n",
       "15066   2269483                            You Belong with Me        pt   \n",
       "15067   4779921                     You Don't Know What to Do        pt   \n",
       "15068   1585284                          You Know I'm No Good        pt   \n",
       "15069   2929652                     Álvaro de Zúñiga y Guzmán        pt   \n",
       "\n",
       "                                                                                       article_url  \n",
       "0                                                   https://en.wikipedia.org/wiki/%3F%20%28film%29  \n",
       "1                                                           https://en.wikipedia.org/wiki/0.999...  \n",
       "2      https://en.wikipedia.org/wiki/1%20%E2%88%92%202%20%2B%203%20%E2%88%92%204%20%2B%20%E2%8B%AF  \n",
       "3         https://en.wikipedia.org/wiki/1st%20Cavalry%20Division%20%28Kingdom%20of%20Yugoslavia%29  \n",
       "4                                   https://en.wikipedia.org/wiki/1st%20Missouri%20Field%20Battery  \n",
       "...                                                                                            ...  \n",
       "15065                                            https://pt.wikipedia.org/wiki/Yorkshire%20terrier  \n",
       "15066                                       https://pt.wikipedia.org/wiki/You%20Belong%20with%20Me  \n",
       "15067                          https://pt.wikipedia.org/wiki/You%20Don%27t%20Know%20What%20to%20Do  \n",
       "15068                                 https://pt.wikipedia.org/wiki/You%20Know%20I%27m%20No%20Good  \n",
       "15069          https://pt.wikipedia.org/wiki/%C3%81lvaro%20de%20Z%C3%BA%C3%B1iga%20y%20Guzm%C3%A1n  \n",
       "\n",
       "[10570 rows x 4 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save this information to a csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_data.to_csv('data/featured_articles.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to query Mediawiki's action API again to get the content of each featured article. \n",
    "\n",
    "For example, to retrieve the wikitext and the id of the last edition for the article #3906668 'Accident sur la base de Fairchild en 1994' from French Wikipedia, we can do the following request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pageid = 3906668\n",
    "\n",
    "PARAMS = {\n",
    "    \"action\": \"parse\",\n",
    "    \"pageid\": pageid,\n",
    "    \"prop\": \"wikitext|revid\",\n",
    "    \"format\": \"json\"\n",
    "}\n",
    "\n",
    "response = requests.get(\n",
    "    url='https://fr.wikipedia.org/w/api.php?', \n",
    "    params=PARAMS,\n",
    ")\n",
    "\n",
    "page_data = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'parse': {'title': 'Accident sur la base de Fairchild en 1994',\n",
       "  'pageid': 3906668,\n",
       "  'revid': 186075124,\n",
       "  'wikitext': {'*': '{{En-tête label|AdQ}}\\n{{Infobox Accident de transport\\n  | nom                  = Accident du B-52 sur la base de Fairchild en 1994\\n  | image                = FairchildB52Crash.jpg\\n  | légende              = Le B-52 sur la tranche une seconde avant de toucher le sol.\\n  | date                 = {{date|24|juin|1994|en aéronautique}}\\n  | phase                = Vol d\\'entraînement\\n  | type                 = Erreur de pilotage\\n  | site                 = [[Fairchild Air Force Base]], [[Washington (État)|Washington]], [[États-Unis]]\\n  | passagers            = \\n  | équipage             = 4 militaires\\n  | morts                = 4 militaires\\n  | blessés              = \\n  | survivants           = \\n  | appareil             = [[Boeing B-52 Stratofortress|Boeing B-52H \\'\\'Stratofortress\\'\\']]\\n  | compagnie            = [[United States Air Force]]\\n  | numéro_identification= 61-0026\\n  | latitude             = 47.615\\n  | longitude            = -117.655556\\n  | carte_pays           = États-Unis\\n}}\\n\\nLe {{date|24|juin|1994|en aéronautique}}, un [[Boeing B-52 Stratofortress|Boeing B-52 \\'\\'Stratofortress\\'\\']] baptisé \\'\\'Czar 52\\'\\'<ref name=\"Kern\">{{ouvrage| langue =en| prénom1 =Tony T.| nom1 =Kern| titre = Darker Shades of Blue| sous-titre =The Rogue Pilot}}</ref> s\\'écrase au cours d\\'un vol d\\'entraînement en vue d\\'une démonstration sur la \\'\\'\\'[[Fairchild Air Force Base|base de Fairchild]]\\'\\'\\', entraînant la mort des quatre membres d\\'équipage. L\\'accident s\\'est produit car le commandant de bord, Bud Holland, a piloté l\\'avion hors de ses limites opérationnelles et en a perdu le contrôle. Il en résulta un [[Décrochage (aéronautique)|décrochage]] et la destruction de l\\'appareil par écrasement au sol. L\\'accident fut filmé et les images furent largement diffusées lors des journaux télévisés à travers les  [[États-Unis]]<ref name=\"Big4\">{{en}} Diehl, \\'\\'Silent Knights\\'\\', {{p.|125}}, Thompson, \\'\\'Way, Way Off in the Wild Blue Yonder\\'\\', USAF, \\'\\'AFR 110-14\\'\\', {{p.|2–3}}, et Kern, \\'\\'Darker Shades of Blue\\'\\'</ref>.\\n\\nLe rapport de la commission d\\'enquête conclut que les principaux facteurs ayant favorisé l\\'accident étaient la [[personnalité]] et le [[comportement]] du pilote Bud Holland vis-à-vis des règles de sécurité et la réaction de sa hiérarchie par rapport à celui-ci, ainsi que l\\'enchaînement des événements. Cet accident est utilisé comme [[étude de cas|cas d\\'école]] dans l\\'aéronautique militaire et civile pour l\\'enseignement de la [[Gestion des ressources de l\\'équipage|gestion des équipages]]. En outre, les forces armées américaines utilisent souvent cet accident pendant la formation à la [[sécurité aérienne]] pour sensibiliser le personnel à l\\'importance du respect des règles de sécurité et à la conduite à tenir vis-à-vis de toute personne qui enfreint les procédures de sécurité.\\n\\n== L\\'accident ==\\n\\n[[Fichier:AFwing.png|thumb|left|alt=Schéma de l\\'organisation d\\'une escadre avec ses deux groupes et ses six escadrons|Structure normale d\\'une escadre de l\\'USAF avec ses groupes et ses escadrons.]]\\n\\nLe {{date|24|juin|1994}} à {{heure|7|30}}, heure locale (\\'\\'[[UTC-8|{{lang|en|Pacific Standard Time}}]]\\'\\'), sur la [[Fairchild Air Force Base|base aérienne de Fairchild]], l\\'équipage du [[bombardier (avion)|bombardier]] B-52H \\'\\'Czar52\\'\\' de l\\'USAF stationné à Fairchild se prépare pour un vol d\\'entraînement en vue d\\'une démonstration au cours d\\'un prochain [[meeting aérien]]. L\\'équipage est composé des [[Pilote (aviation)|pilotes]] [[Lieutenant-colonel|Lt.Col]] Arthur « Bud » Holland ({{unité|46|ans}}), Lt.Col Mark McGeehan ({{unité|36|ans}}), [[Colonel|Col.]] Robert Wolff ({{unité|46|ans}}) et de l\\'opérateur système d\\'arme/navigateur radar Lt.Col Ken Huston ({{unité|41|ans}}). Holland est le commandant de bord pour ce vol avec McGeehan comme copilote et Wolff comme observateur de sécurité. Holland est le commandant en second de la section « Évaluation et standardisation » du [[92nd Air Refueling Wing|{{lang|en|92nd Bomb Wing}}]], McGeehan est commandant du {{lang|en|325th Bomb Squadron}}, Wolff est commandant en second du {{lang|en|92nd Bomb Wing}}, et Huston est officier opérations du {{lang|en|325th Bomb Squadron}}<ref>{{en}} Thompson, \\'\\'Way, Way Off in the Wild Blue Yonder\\'\\'. Kern, \\'\\'Darker Shades of Blue\\'\\' et USAF, \\'\\'AFR 110-14\\'\\', {{p.|2–3}}.</ref>{{,}}<ref group=\"note\">Holland, en tant que chef de l\\'évaluation et de la standardisation, était responsable de la connaissance et du respect des normes de vol pour les opérations de l\\'escadre.</ref>.\\n\\nLe plan de vol enchaîne des manœuvres difficiles : passages à basse altitude, virages inclinés à 60°, montée sous forte pente et un [[touch-and-go|touché]] sur la piste 23 de Fairchild. Ce vol était aussi le vol d\\'adieu pour Wolff ; une tradition de l\\'USAF veut que lorsqu\\'un membre d\\'équipage prend sa retraite, il est accueilli à sa descente d\\'avion par ses collègues, ses amis et sa famille et arrosé d\\'eau. Donc ce jour-là, la femme de Wolff et plusieurs de ses proches amis sont présents sur l\\'[[aérodrome]] pour regarder le vol et participer à la cérémonie. La femme de McGeehan et ses deux plus jeunes fils regardent aussi le vol depuis leur maison située non loin<ref>{{en}} Piper, \\'\\'Chain of Events\\'\\', {{p.|136}}, Kern, \\'\\'Darker Shades of Blue\\'\\', et USAF, \\'\\'AFR 110-14\\'\\', {{p.|2–3}}.</ref>.\\n\\nLe B-52 décolle à {{heure|13|58}} et effectue la plupart des évolutions sans incident. Alors qu\\'il se prépare à exécuter le touché sur la piste 23 à la fin de sa présentation, l\\'appareil est obligé de [[Go-around|remettre les gaz]] car un [[Boeing KC-135 Stratotanker|{{nobr|KC-135}} Stratotanker]] vient juste de se poser et est encore sur la piste. Conservant une hauteur-sol de {{unité|75|m}} ({{unité|250|ft}}), Holland demande à la [[Contrôle du trafic aérien|tour de contrôle]] l\\'autorisation d\\'effectuer une boucle à 360° par la gauche, ce qui lui est accordé. Le B-52 commence alors son virage autour de la tour à partir du milieu de la piste. Une zone abritant un site de stockage d\\'[[arme nucléaire|armes nucléaires]]<ref>Diehl, \\'\\'Silent Knights\\'\\', {{p.|125}}.</ref>, dont le survol est interdit, se trouve juste derrière la tour. Il semble qu\\'Holland, pour éviter de traverser cet espace aérien, ait effectué un virage très serré et très incliné tout en maintenant sa faible hauteur ({{unité|75|m}}). Approximativement aux trois quarts de son virage, à {{heure|14|16}}, l\\'appareil s\\'incline à plus de 90°, chute rapidement et touche le sol, explosant et tuant tous les membres d\\'équipage. Le Lt.Col McGeehan était parvenu à s\\'[[Siège éjectable|éjecter]], mais il périt dans la boule de feu créée par l\\'écrasement de l\\'appareil. Il n\\'y a pas de blessé au sol<ref name=\"AFR\">USAF, \\'\\'AFR 110-14\\'\\', toutes les pages.</ref>.\\n\\n== L\\'enquête ==\\n[[Fichier:BGgodsey.jpg|thumb|left|alt=Portrait du général Godsey|Le brigadier-général Orin L. Godsey, qui dirigea l\\'enquête de sécurité après l\\'accident.]]\\n\\nL\\'USAF réunit immédiatement une commission d\\'enquête de sécurité sous la direction du chef de la sécurité de l\\'USAF, le [[Général de brigade|brigadier-général]] Orin L. Godsey. La commission remet un rapport préliminaire le {{date|10|août|1994}} et le rapport final d\\'enquête est publié le {{date|31|janvier|1995}}. Le rapport de sécurité de l\\'US Air Force ne fut diffusé qu\\'au personnel du [[Département de la Défense des États-Unis]] et ne fut pas rendu public. Un comité d\\'enquête sur l\\'accident, appelé « Enquête AFR 110-14 », a publié un rapport séparé en 1995. Contrairement à l\\'enquête de sécurité de l\\'USAF, le rapport AFR 110-14 a été rendu public<ref>Air Force Link, \\'\\'Brigadier General Orin L. Godsey\\'\\', et USAF, \\'\\'AFR 110-14\\'\\', {{p.|2–3}}.</ref>.\\n\\nL\\'enquête AFR 110-14 identifie plusieurs facteurs ayant contribué à l\\'accident, dont l\\'enchaînement réel de l\\'accident, la personnalité et le comportement antérieur de Bud Holland, l\\'absence de surveillance et de mesures correctives de l\\'USAF vis-à-vis de Holland, la préparation et l\\'exécution de la mission, et d\\'autres facteurs environnementaux et humains<ref name=\"AFR\"/>.\\n\\n=== Enchaînement des événements ===\\n[[Fichier:Accelerated stall fr.gif|thumb|upright=1.5|alt=Schéma montrant les force s\\'exerçant sur l\\'avion en virage|Illustration du décrochage en virage.<br />Plus le virage est serré, plus la [[portance (mécanique des fluides)|portance]] (et donc l\\'[[angle d\\'attaque]]) doit être élevée pour contrer le [[poids apparent]] (la somme de la force centrifuge et du poids). Le décrochage se produit donc à une vitesse plus élevée qu\\'à plat.]]\\n\\nL\\'enquête montre que lorsque le B-52 entame son dernier virage autour de la tour de contrôle, la vitesse indiquée était de {{unité|337|km/h}} ({{unité|182|[[Nœud (unité)|kt]]}}). Lorsque Holland amorce son virage, il augmente la poussée des moteurs, mais cette action est trop tardive car les [[Turboréacteur|réacteurs]] mettent {{nombre|8|secondes}} à réagir à l\\'action de la commande de gaz. Bien que l\\'[[anémomètre]] soit visible de l\\'ensemble des quatre membres d\\'équipage, ils laissent la vitesse décroître. Huit secondes avant l\\'impact, la vitesse indiquée de l\\'appareil était descendue à {{unité|269|km/h}} ({{unité|145|kt}}) et le roulis de l\\'appareil dépassait les 60°. C\\'est à ce moment-là que soit Holland soit McGeehan met le manche et le palonnier complètement à droite tout en redressant le nez de l\\'appareil et que l\\'avion entre dans une phase de [[décrochage (aéronautique)|décrochage]] en virage (ou décrochage sous [[facteur de charge (aérodynamique)|facteur de charge]]). Ce phénomène, le décrochage en virage, se produit à une vitesse plus élevée que la vitesse de décrochage normale calculée pour un vol en palier. À cause de son roulis dépassant les 60°, la vitesse de décrochage de l\\'avion était à ce moment de {{unité|272|km/h}} ({{unité|147|kt}}). Donc, en volant à {{unité|269|km/h}} ({{unité|145|kt}}), l\\'appareil ne pouvait être maintenu à son altitude ; une augmentation de l\\'incidence dans le but de maintenir l\\'altitude ne pouvait que conduire au décrochage et la hauteur était insuffisante pour sortir du décrochage avant de heurter le sol<ref name=\"AFR\"/>.\\n\\n=== Comportement antérieur de Holland et réactions de la hiérarchie de l\\'USAF ===\\nLa commission sur l\\'accident établit que la personnalité de Bud Holland constituait un élément  significatif relativement aux causes préalables à cet accident. Le personnel de l\\'USAF attesta que Holland avait acquis une réputation de pilote impulsif qui bafouait régulièrement les règles de sécurité. Il volait souvent en dessous de l\\'altitude minimale de sécurité, dépassait les limites d\\'inclinaison en virage et les taux de montée<ref name=\"Kern\"/>{{,}}<ref name=\"Big2\">USAF, \\'\\'AFR 110-14\\'\\', {{p.|3–4}}</ref>.\\n\\nUn incident eut lieu en [[1991 en aéronautique|1991]] quand un B-52 piloté par Holland effectua un tour au-dessus d\\'un stade de [[softball]] où sa fille jouait. Débutant à {{unité|760|m}} ({{unité|2500|[[pied (unité)|ft]]}}) de hauteur-sol, l\\'appareil de Holland effectua son tour avec une inclinaison de 65°, se retrouvant dans une situation de virage engagé. Un témoin décrivit la scène comme une « spirale mortelle », le nez de l\\'appareil ayant continué à descendre durant la manœuvre et l\\'inclinaison atteignant les 80°. Après avoir perdu {{unité|300|m}} ({{unité|1000|ft}}), Holland put reprendre le contrôle de l\\'appareil<ref>Diehl, \\'\\'Silent Knights\\'\\', {{p.|125}}, Thompson, \\'\\'Way, Way Off in the Wild Blue Yonder\\'\\'</ref>{{,}}<ref name=\"USAF\">USAF, \\'\\'AFR 110-14\\'\\', {{p.|3–4}}.</ref>.\\n\\n[[Fichier:B52.climbout.arp.jpg|thumb|alt=Un B-52H au décollage.|left|Un B-52H au décollage pour une présentation avec une pente de montée normale.]]\\n\\nLe {{date|19|mai|1991}} pour le meeting aérien de Fairchild, Holland était le commandant du B-52 effectuant la démonstration aérienne. Durant ce vol Holland viola plusieurs règles de sécurité, dont une inclinaison et une pente trop importante, l\\'interdiction du survol du public et peut-être aussi le non-respect de l\\'altitude minimale. Le commandant de la base et de l\\'[[escadre]], le colonel Arne Weinman, comme l\\'encadrement, observa la démonstration mais ne prit aucune sanction<ref name=\"Kern\"/>.\\n\\nLe {{date|12|juillet|1991}}, Holland était aux commandes d\\'un B-52 pour un survol de la cérémonie de passation de commandement du {{lang|en|325th Bomb Squadron}}. Pendant la répétition et le survol réel, l\\'appareil de Holland vola à une altitude de {{unité|30|m}} ({{unité|100|ft}}) — bien en dessous de l\\'altitude minimale autorisée —, effectua un virage incliné à plus de 45°, dépassa la pente maximum et réalisa un [[renversement (aviation)|renversement]]. Le renversement n\\'est pas spécifiquement interdit mais n\\'est pas recommandé car il peut endommager l\\'avion. Après avoir assisté au vol le colonel Weinman et son adjoint, le colonel Julich commandant des opérations, réprimandèrent verbalement Holland mais ne prirent pas de sanctions officielles<ref name=\"Big2\"/>.\\n\\nLe {{date|17|mai|1992}} pour le meeting aérien de Fairchild, Holland est de nouveau aux commandes du B-52 qui effectue la démonstration aérienne. Pendant ce vol, Holland viole sévèrement les règles de sécurité, virages serrés à faible altitude à plus de 45° d\\'inclinaison et grand angle de montée, estimé à plus de 60° que Holland termine par un renversement. Le nouveau commandant d\\'escadre, le colonel Mickael G. Ruotsala, ne prend apparemment aucune sanction. Une semaine plus tard, le nouveau commandant adjoint des opérations, le colonel Capotosti, de sa propre initiative avertit  Holland que s\\'il violait encore une fois les règles de sécurité, il l\\'interdirait de vol. Capotosti ne fait pas de compte-rendu de son avertissement et ne prend aucune autre sanction officielle<ref name=\"Kern\"/>.\\n\\nLes 14 et {{date-|15 avril 1993}}, Holland est chef de mission pour une mission d\\'entraînement de deux B-52 sur un champ de tir près de [[Guam]] dans l\\'[[océan Pacifique]]. Pendant la mission, il fait voler son avion plus près de l\\'autre appareil que la réglementation ne le permet. Holland demande aussi à son navigateur de filmer le largage des bombes à partir de la soute à bombe, ce qui est aussi contraire au règlement. Plus tard, le navigateur de Holland montrera la vidéo à trois cadres de la base de Fairchild. Le premier, le lieutenant-colonel Bullock, le commandant du {{lang|en|325th Bomb Squadron}}, ne fera rien à ce sujet et aurait même essayé d\\'utiliser la vidéo pour [[Chantage|faire pression]] sur le navigateur pour qu\\'il accepte un poste de planificateur de mission pour l\\'escadre. Le deuxième, le commandant adjoint aux opérations du groupe, le lieutenant-colonel Harper, demandera au membre d\\'équipage de dissimuler la preuve. Le troisième, le commandant adjoint aux opérations, aurait répondu à propos de la vidéo : « OK je ne veux rien savoir au sujet de cette vidéo. Je n\\'en ai rien à faire. »<ref>Diehl, \\'\\'Silent Knights\\'\\', {{p.|125}}, Kern, \\'\\'Darker Shades of Blue\\'\\'.</ref>{{,}}<ref group=\"note\">L\\'article ne dit pas s\\'il s\\'agit du colonel Capotosti ou du nouveau commandant, le colonel William E. Pellerin.</ref>\\n\\nLe {{date|8|août|1993}}, Holland est une fois de plus aux commandes du B-52 de démonstration pour le meeting aérien de Fairchild. Là encore le profil du vol inclut des virages avec des inclinaisons à plus de 45°, des passages à basse altitude et autres manœuvres en montée à forte pente atteignant cette fois plus de 80°. L\\'angle est tellement élevé que le carburant sort par les [[évent]]s des réservoirs d\\'aile de l\\'appareil. Le nouveau commandant d\\'escadre le brigadier-général James M. Richards, et le nouveau commandant adjoint aux opérations, le colonel William E. Pellerin, sont tous les deux témoins de la démonstration mais ne font rien<ref name=\"Kern\"/>{{,}}<ref>Air Force Link, \\'\\'Brigadier General James M. Richards USAF biography\\'\\'</ref>{{,}}<ref name=\"Thompson\">Thompson, \\'\\'Way, Way Off in the Wild Blue Yonder\\'\\'</ref>.\\n\\nLe {{date|10|mars|1994}}, Holland est le commandant de bord d\\'un B-52 lors une mission d\\'entraînement au bombardement sur le champ de tir de [[Comté de Yakima|Yakima]] ; il doit donner la possibilité à un photographe accrédité de prendre en photo l\\'appareil larguant des bombes d\\'entraînement. La hauteur minimum autorisée sur cette zone était de {{unité|150|m}} ({{unité|500|ft}}). Durant cette mission, Holland est filmé passant une ligne de crête à seulement {{unité|10|m}} ({{unité|30|ft}}) du sol. L\\'équipe photographique, craignant pour sa sécurité, cesse de filmer et se met à l\\'abri alors que l\\'avion de Holland les survole une seconde fois à une hauteur estimée à {{unité|1|m}}. Le copilote de Holland affirme avoir repris les commandes  pour éviter que Holland ne fasse s\\'écraser l\\'appareil sur la crête pendant que les deux autres membres d\\'équipage lui criaient « Monte ! Monte ! ». Holland répondit en rigolant et en traitant l\\'un d\\'eux de « gonzesse »<ref name=\"Big4\" />.\\n\\n[[Image:Mcgeehan.jpg|thumb|alt=Portrait du lieutenant-colonel McGeehan.|left|Mark McGeehan, le commandant d\\'escadron qui refusa de laisser ses équipages voler avec Holland s\\'il n\\'était pas présent dans l\\'appareil.]]\\n\\nAprès cette mission, l\\'équipage décide de ne plus jamais voler avec Holland et, rend compte de l\\'incident au commandant de l\\'escadron, le lieutenant-colonel Mark McGeehan. McGeehan rapporte l\\'incident à Pellerin et recommande que Holland soit interdit de vol. Pellerin convoque Holland, le réprimande verbalement et l\\'avertit de ne pas recommencer ce comportement, mais se refuse à lui interdire de voler. Il ne consigne pas non plus l\\'incident ni la réprimande et laisse son supérieur dans l\\'ignorance de ce qui s\\'est passé. McGeehan décide ensuite que, pour protéger ses équipages, il sera le copilote pour les futures missions où Holland serait le commandant de bord. Les faits semblent indiquer qu\\'après cet incident une « animosité considérable » devait exister entre Holland et McGeehan<ref name=\"USAF\"/>{{,}}<ref name=\"Thompson\"/>{{,}}<ref>Kern, \\'\\'Darker Shades of Blue\\'\\'</ref>{{,}}<ref>Lou Holtz/Upper Ohio Valley Hall of Fame, \\'\\'Lt. Col. Mark C. McGeehan\\'\\'.</ref>.\\n\\n[[Fichier:Fairchild B-52 crash overview fr.png|thumb|alt=Trajectoire de l\\'appareil autour de la tour.|Dernier virage de l\\'appareil :\\n{{Légende/Début}}\\n{{Ligne de légende|#268CFF solid 2px|Trajectoire de l\\'appareil}}\\n{{Légende|black|Piste 23 de Fairchild}}\\n{{Légende|#E5E5E5|Taxiways et parkings}}\\n{{Légende|#FFCEBF|Zone d\\'interdiction de survol servant de zone de stockage d\\'armes nucléaires}}\\n{{Légende/Fin}}\\n]]\\n\\nAu cours des préparatifs pour le meeting de Fairchild de 1994, Holland est encore choisi pour effectuer la présentation du B-52. Il expose sa présentation au nouveau commandant d\\'escadre, le colonel William Brooks, le {{date|15|juin|1994}} ; elle inclut la violation de nombreuses règles, dont des virages serrés à forte inclinaison, des passages bas et des montées à forte pente. Brooks ordonne à Holland de ne pas dépasser 45° d\\'inclinaison et 25° de pente pendant la démonstration. Pendant le premier entraînement, le {{date-|17 juin}}, Holland viole à plusieurs reprises ces ordres. Brooks en est témoin mais ne fait rien. Pellerin qui vole avec Holland lors de cette démonstration rend compte à Brooks que « le profil de la démonstration lui semble bon, sûr et bien à l\\'intérieur des paramètres ». L\\'entraînement suivant, le {{date-|24 juin}}, se termine par l\\'accident<ref name=\"Big2\" />.\\n\\n=== Autres facteurs ===\\nLe profil de démonstration conçu par Holland prévoyait un virage à 360° autour de la tour de contrôle de Fairchild. Holland n\\'avait jamais tenté cette manœuvre dans les démonstrations précédentes. Pendant le dernier vol, Holland effectua une série de virages à 60° d\\'inclinaison et une montée à 68° en violation des ordres de Brooks. Rien ne démontre que McGeehan ou Wolff tentèrent d\\'intervenir pendant qu\\'Holland effectuait ces manœuvres<ref name=\"AFR\"/>.\\n\\nInitialement, Pellerin devait  participer au vol, comme il l\\'avait fait pour celui du {{date-|17 juin}}. Pellerin, cependant, n\\'était pas disponible pour le vol du {{date-|24 juin}} et Wolff fut choisi comme remplaçant au sein de l\\'équipage. En raison du court préavis avant son affectation à la mission, Wolff ne participa pas à la préparation du vol et rejoignit l\\'appareil alors que les moteurs tournaient déjà. Wolff n\\'était donc pas au courant du profil de la mission et n\\'eut pas la possibilité de faire part d\\'éventuelles objections avant le décollage<ref name=\"AFR\"/>.\\n\\nTous les membres d\\'équipage impliqués dans l\\'accident n\\'avaient eu que peu d\\'heures de vol dans les mois précédant celui-ci. L\\'équipage du B-52 ne s\\'était apparemment pas aperçu que l\\'appareil avait décroché peu avant l\\'impact, ceci est mis en évidence par l\\'absence d\\'utilisation des techniques de récupération standard sur cet appareil une fois entré en phase de décrochage. Cependant, l\\'enquête montre que même avec les techniques appropriées, l\\'avion était probablement trop bas pour se rétablir avant de toucher le sol<ref name=\"AFR\"/>.\\n\\nQuatre jours avant l\\'accident, le {{date-|20 juin}}, Dean Mellberg, un ancien militaire de l\\'US Air Force émotionnellement perturbé, était entré dans l\\'hôpital de Fairchild et, tirant à plusieurs reprises, avait tué cinq personnes en laissant plusieurs autres, blessées, avant d\\'être abattu par des membres des [[US Air Force Security Forces|forces de sécurité de la force aérienne]]. Cette fusillade a profondément perturbé le personnel de la base de Fairchild et ceci jusque bien après l\\'accident<ref name=\"AFR\"/>.\\n\\n=== Conclusions ===\\n\\nLa commission d\\'enquête conclut que l\\'enchaînement d\\'événements qui conduisit à l\\'accident était principalement attribuable à la [[personnalité]] et au [[comportement]] de Holland, aux réactions (ou à leur absence) de la part de la hiérarchie de l\\'[[United States Air Force|US Air Force]] et, enfin, à la succession des incidents pendant le vol. Le non-respect par Holland des procédures garantissant la sûreté du B-52 qu\\'il commandait et l\\'absence de mesures fermes et cohérentes de la part de ses supérieurs amenèrent Holland à croire qu\\'il pouvait effectuer des manœuvres dangereuses durant son dernier vol, avec pour point d\\'orgue un virage à 360° lent et très incliné autour de la tour de contrôle<ref name=\"Big4\"/>.\\n\\nPendant le dernier vol de Holland, les autres facteurs environnementaux mis en cause incluent l\\'ajout de manœuvres nouvelles (virage à 360° autour de la tour), l\\'insuffisante implication avant-vol du colonel Wolff, et les perturbations causées par la fusillade du {{date-|20 juin}}. Alliés au comportement dangereux de Holland, ces facteurs produisirent les conditions favorables à l\\'accident. Le dernier facteur, relevé par le rapport d\\'enquête de l\\'USAF, est le vent soufflant à {{unité|19|km/h}} ({{unité|10|kt}}) et ses effets sur les manœuvres nécessaires pour réaliser la trajectoire de vol prévue, par rapport au sol<ref name=\"Big4\"/>.\\n\\n== Conséquences ==\\nLe {{date|19|mai|1995}}, Pellerin a plaidé coupable devant une [[cour martiale]] de l\\'USAF pour deux accusations de manquement au devoir pour ses actions, ou leur absence, qui ont contribué à l\\'accident. Il fut condamné à renoncer à {{unité|1500|[[dollar américain|dollars américains]]}} de salaire par mois pendant cinq mois et reçut une réprimande écrite. L\\'US Air Force n\\'a pas révélé si d\\'autres officiers impliqués dans la chaîne d\\'événements ayant conduit à l\\'accident se sont vu imposer des sanctions administratives ou disciplinaires. Les critiques du rapport de sécurité de l\\'USAF établissent que cet accident est un exemple d\\'un ensemble de problèmes liés à l\\'application des procédures de sécurité au sein de l\\'USAF<ref>Diehl, \\'\\'Silent Knights\\'\\', {{p.|126}}, Thompson, \\'\\'Way, Way Off in the Wild Blue Yonder\\'\\', Kern, \\'\\'Darker Shades of Blue\\'\\'</ref>.\\n\\nBien que l\\'enquête sur l\\'accident ait révélé que les procédures et les principes étaient déjà en place pour empêcher un tel accident de se produire, le fait que cet accident soit arrivé a montré que, dans au moins un cas, les politiques de sécurité existantes et leur application avaient été insuffisantes. Afin de réitérer l\\'importance de l\\'adhésion aux politiques de sécurité existantes et de corriger toute infraction à l\\'un de ces principes, l\\'USAF, comme pour chaque accident, a rapidement distribué les conclusions de l\\'enquête sur l\\'accident à toute l\\'armée. Actuellement, l\\'accident est utilisé dans l\\'aviation civile et militaire comme support pour l\\'enseignement de la [[Gestion des ressources de l\\'équipage|gestion des équipages]]. Il est aussi souvent utilisé dans les entraînements à la sécurité comme exemple de l\\'importance de faire respecter les règles de sécurité<ref name=\"Kern\"/>{{,}}<ref>Check-Six.com, \\'\\'The Crash of Czar 52\\'\\'.</ref>.\\n\\n== Voir aussi ==\\n\\n* [[Gestion des ressources de l\\'équipage]]\\n\\n== Notes et références ==\\n=== Notes ===\\n{{Références|groupe=note}}\\n=== Références ===\\n{{Références|colonnes=2}}\\n\\n=== Ouvrages de référence ===\\n* {{Traduction/Référence|en|1994 Fairchild Air Force Base B-52 crash|340163884}}\\n* {{ouvrage\\n | langue =en\\n | nom1 = Diehl\\n | prénom1 = Alan E.\\n | titre = Silent Knights: Blowing the Whistle on Military Accidents and Their Cover-Ups\\n | année = 2003\\n | éditeur = Potomac Books\\n | isbn = 1-57488-544-8\\n}}\\n* {{ouvrage\\n | langue =en\\n | prénom1 =Tony T.\\n | nom1 =Kern\\n | titre = Darker Shades of Blue\\n | sous-titre =The Rogue Pilot\\n | éditeur =McGraw-Hill Professional Publishing\\n | année =1999\\n | isbn =0-07-034927-4\\n}}\\n* {{ouvrage\\n | langue =en\\n | nom1 = Piper\\n | prénom1 = Joan L.\\n | titre = Chain of Events: The Government Cover-up of the Black Hawk Incident and the Friendly-fire Death of Lt. Laura Piper\\n | année = 2001\\n | éditeur = Brassey\\'s\\n | isbn = 1-57488-344-5\\n}}\\n* {{ouvrage\\n | langue =en\\n | nom1 = United States Air Force\\n | prénom1 =\\n | lien auteur = United States Air Force\\n | titre = Summary of AFR 110-14 USAF Accident Investigation Board Report\\n | année = 1994\\n}}\\n\\n=== Ressources internet ===\\n* {{en}} {{Lien web\\n|url=http://www.af.mil/information/bios/bio.asp?bioID=5566\\n|titre=Brigadier General Orin L. Godsey : Biographie du chef de la sécurité de l\\'USAF qui dirigea l\\'enquête initiale\\n|auteur=\\n|année=1996\\n|éditeur=Air Force Link\\n|consulté le=26-01-2010\\n}}\\n* {{en}} {{Lien web\\n|url=http://www.af.mil/information/bios/bio.asp?bioID=6898\\n|titre=Brigadier General James M. Richards : Bombraphie du commandant du 92nd Bomb Wing d\\'août 1992 à août 1993\\n|auteur=\\n|année=1996\\n|éditeur=Air Force Link\\n|consulté le=26-01-2010\\n}}\\n* {{en}} {{Lien web\\n|url=http://www.check-six.com/Crash_Sites/Czar52Crash.htm\\n|titre=The Crash of \\'Czar 52\\'\\n|auteur=\\n|année=2007\\n|éditeur=Check-Six.com\\n|consulté le=26-01-2010\\n}} - Contient des séquences vidéo de l\\'accident.\\n* {{en}} {{Lien web\\n|url=http://www.crm-devel.org/resources/paper/darkblue/darkblue.htm\\n|titre=Darker Shades of Blue: A Case Study of Failed Leadership\\n|auteur=Tony T. Kern\\n|année=1995\\n|éditeur=Neil Krey\\'s CRM Developers Forum\\n|consulté le=26-01-2010\\n}}\\n* {{en}} {{Lien web\\n|url=http://www.louholtzhalloffame.com/honorees/2000/index.html\\n|titre=Lt. Col. Mark C. McGeehan\\n|auteur=\\n|année=2000\\n|éditeur=Lou Holtz/Upper Ohio Valley Hall of Fame\\n|consulté le=26-01-2010\\n}}\\n* {{en}} {{Lien web\\n|url=http://www.time.com/time/magazine/article/0,9171,982358,00.html\\n|titre= Way, Way Off in the Wild Blue Yonder\\n|auteur=Mark Thompson\\n|année=29 mai 1995\\n|éditeur=Time\\n|consulté le=26-01-2010\\n}}\\n* {{YouTube|ReSm7r45_ds|The Crash of Czar 52 at Fairchild AFB, WA, in 1994|en}}\\n* {{en}} {{Lien web\\n|url=https://query.nytimes.com/gst/fullpage.html?res=990CE7DA1038F934A15757C0A963958260\\n|titre=Nuclear Storage Site Linked to Plane Crash\\n|auteur=\\n|année=1995\\n|éditeur=The New York Times\\n|consulté le=26-01-2010\\n}}\\n\\n== Liens externes ==\\n{{autres projets|commons=B-52 aircraft crash at Fairchild Air Force Base}}\\nVidéo amateur montrant l\\'accident :\\n* {{YouTube|FUEhNKBi4DY|B-52 Fairchild AFB Crash 1994|en}}\\n\\n{{Palette|Accident avion 1994|Accidents aériens aux États-Unis}}\\n{{Portail|aéronautique|années 1990|Washington (État)|Forces armées des États-Unis}}\\n\\n{{Article de qualité|oldid=50305926|date={{1er}} mars 2010}}\\n\\n[[Catégorie:Histoire militaire des États-Unis]]\\n[[Catégorie:Accident ou incident aérien en 1994]]\\n[[Catégorie:Accident ou incident aérien aux États-Unis|Fairchild AB]]\\n[[Catégorie:Accident ou incident aérien impliquant un Boeing]]\\n[[Catégorie:Comté de Spokane]]\\n[[Catégorie:1994 aux États-Unis]]\\n[[Catégorie:Juin 1994]]\\n[[Catégorie:Histoire de l\\'État de Washington]]'}}}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fetch the wikitext and the id of the last edition for each featured article and save the response to a compressed file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrieving data from articles: 10570it [00:06, 1640.30it/s]\n"
     ]
    }
   ],
   "source": [
    "for index, row in tqdm(articles_data.iterrows(), desc='Retrieving data from articles'):\n",
    "\n",
    "    pageid = articles_data.loc[index]['pageid']\n",
    "    lang = articles_data.loc[index]['wiki_lang']\n",
    "    filename = f'articles-content/{pageid}-{lang}.json.gz'\n",
    "    \n",
    "    MEDIAWIKI_API_URL = f'https://{lang}.wikipedia.org/w/api.php?'\n",
    "\n",
    "    if not os.path.isfile(filename):\n",
    "\n",
    "        PARAMS = {\n",
    "            \"action\": \"parse\",\n",
    "            \"pageid\": pageid,\n",
    "            \"prop\": \"wikitext|revid\",\n",
    "            \"format\": \"json\"\n",
    "        }\n",
    "\n",
    "        response = requests.get(\n",
    "            url=MEDIAWIKI_API_URL, \n",
    "            params=PARAMS,\n",
    "            headers=HEADER\n",
    "        )\n",
    "\n",
    "        page_data = response.json()\n",
    "\n",
    "        if response.status_code == 200:\n",
    "            with gzip.open(filename, 'w') as fo:\n",
    "                fo.write(json.dumps(page_data).encode('utf-8'))\n",
    "    \n",
    "    else:\n",
    "        continue\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Citation template metadata extraction\n",
    "\n",
    "Now that we already have the wikitext of the articles, we will parse it in order to retrieve the references that were introduced using a citation template. In other words, we are not interested in:\n",
    "\n",
    "1. Manually entered references (ie, which do not use citation templates)\n",
    "\n",
    "`<ref name=Briggs>Briggs, Asa & Burke, Peter (2002) ''A Social History of the Media: from Gutenberg to the Internet'', Cambridge: Polity, pp. 15–23, 61–73.</ref>`\n",
    "\n",
    "2. Unlinked references\n",
    "\n",
    "```<ref name=Neeham>{{cite book |title=Paper and Printing |author=[[Tsien Tsuen-Hsuin]] |author2=[[Joseph Needham]] |series=Science and Civilisation in China|volume=5 part 1|publisher=Cambridge University Press|pages=158, 201|year=1985}}</ref> ```\n",
    "\n",
    "An example of the type of references that we want to keep:\n",
    "\n",
    "```<ref name=\"VB1992\">{{cite journal|last1=Osmond|first1=Patricia J.|last2=Ulery|first2=Robert W.|date=2003|title=Sallustius|url=http://catalogustranslationum.org/PDFs/volume08/v08_sallustius.pdf#page=17|journal=[[Catalogus Translationum et Commentariorum]]|volume=8|page=199|access-date=27 August 2015}}</ref>```\n",
    "\n",
    "Therefore, we parse the page content looking only for the citation templates including a URL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Parameter name mapping\n",
    "\n",
    "From each reference, we are interested in extracting the following metadata: \n",
    "1. The source type (journal, book, website, etc.)\n",
    "1. The URL of the source\n",
    "2. The author(s)\n",
    "3. The title\n",
    "4. The publishing date\n",
    "5. The publishing source (publisher, location, etc.)\n",
    "\n",
    "The names of each data may vary between templates. For instance, the publishing source is under \"periodical\" for the news template and under \"publisher\" for the books template. Or the publishing date might be called \"date\" or \"year\" in the maps template. The [following spreadsheet](https://docs.google.com/spreadsheets/d/1xbc3FKE0m4JQHa6WCXtBbzeJ9in8P0EQ2NF_VNsaBaM/edit#gid=0) maps the name of each parameter in the citations templates to our fieldnames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>template</th>\n",
       "      <th>title</th>\n",
       "      <th>author_last</th>\n",
       "      <th>author_first</th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>en</td>\n",
       "      <td>Cite web</td>\n",
       "      <td>title</td>\n",
       "      <td>last\\d*, author\\d*</td>\n",
       "      <td>first\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>website, publisher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>es</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>título</td>\n",
       "      <td>apellidos?\\d*, autor\\d*</td>\n",
       "      <td>nombre\\d*</td>\n",
       "      <td>fecha</td>\n",
       "      <td>sitioweb, obra, publicación, editorial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pt</td>\n",
       "      <td>Citar web</td>\n",
       "      <td>titulo</td>\n",
       "      <td>ultimo\\d*, autor\\d*</td>\n",
       "      <td>primeiro\\d*</td>\n",
       "      <td>data</td>\n",
       "      <td>obra, publicado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fr</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>titre</td>\n",
       "      <td>nom\\d*, auteur\\d*</td>\n",
       "      <td>prénom\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>site</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>en</td>\n",
       "      <td>Cite book</td>\n",
       "      <td>title</td>\n",
       "      <td>last\\d*, author\\d*</td>\n",
       "      <td>first\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>publication-place, location, publisher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>fr</td>\n",
       "      <td>Lien arXiv</td>\n",
       "      <td>titre</td>\n",
       "      <td>nom\\d*</td>\n",
       "      <td>prénom\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>eprint</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>pt</td>\n",
       "      <td>Citar arXiv</td>\n",
       "      <td>titulo</td>\n",
       "      <td>ultimo\\d*, autor\\d*</td>\n",
       "      <td>primeiro\\d*</td>\n",
       "      <td>data</td>\n",
       "      <td>eprint</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>en</td>\n",
       "      <td>Citation</td>\n",
       "      <td>title</td>\n",
       "      <td>last\\d*</td>\n",
       "      <td>first\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>place, publisher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>es</td>\n",
       "      <td>Obra citada</td>\n",
       "      <td>título</td>\n",
       "      <td>apellidos?\\d*</td>\n",
       "      <td>nombre\\d*</td>\n",
       "      <td>fecha</td>\n",
       "      <td>editorial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>pt</td>\n",
       "      <td>Citation</td>\n",
       "      <td>title</td>\n",
       "      <td>last\\d*</td>\n",
       "      <td>first\\d*</td>\n",
       "      <td>date</td>\n",
       "      <td>publisher</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>83 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wiki_lang     template   title              author_last author_first  \\\n",
       "0         en     Cite web   title       last\\d*, author\\d*     first\\d*   \n",
       "1         es     Cita web  título  apellidos?\\d*, autor\\d*    nombre\\d*   \n",
       "2         pt    Citar web  titulo      ultimo\\d*, autor\\d*  primeiro\\d*   \n",
       "3         fr     Lien web   titre        nom\\d*, auteur\\d*    prénom\\d*   \n",
       "4         en    Cite book   title       last\\d*, author\\d*     first\\d*   \n",
       "..       ...          ...     ...                      ...          ...   \n",
       "78        fr   Lien arXiv   titre                   nom\\d*    prénom\\d*   \n",
       "79        pt  Citar arXiv  titulo      ultimo\\d*, autor\\d*  primeiro\\d*   \n",
       "80        en     Citation   title                  last\\d*     first\\d*   \n",
       "81        es  Obra citada  título            apellidos?\\d*    nombre\\d*   \n",
       "82        pt     Citation   title                  last\\d*     first\\d*   \n",
       "\n",
       "   pub_date                              pub_source  \n",
       "0      date                      website, publisher  \n",
       "1     fecha  sitioweb, obra, publicación, editorial  \n",
       "2      data                         obra, publicado  \n",
       "3      date                                    site  \n",
       "4      date  publication-place, location, publisher  \n",
       "..      ...                                     ...  \n",
       "78     date                                  eprint  \n",
       "79     data                                  eprint  \n",
       "80     date                        place, publisher  \n",
       "81    fecha                               editorial  \n",
       "82     date                               publisher  \n",
       "\n",
       "[83 rows x 7 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping_sheet = f'https://docs.google.com/spreadsheets/d/1xbc3FKE0m4JQHa6WCXtBbzeJ9in8P0EQ2NF_VNsaBaM/export?gid=0&format=csv'\n",
    "\n",
    "fieldnames_df = pd.read_csv(mapping_sheet)\n",
    "## rename columns\n",
    "fieldnames_df.rename(columns={\n",
    "    \"Template\": \"template\",\n",
    "    \"authorLast\": \"author_last\",\n",
    "    \"authorFirst\": \"author_first\",\n",
    "    \"pubDate\": \"pub_date\",\n",
    "    \"source (published In + published By)\": \"pub_source\"\n",
    "}, inplace=True)\n",
    "\n",
    "fieldnames_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The spreadsheet accepts several parameter names for the same field (`sitioweb, obra, publicación, editorial`) and it also accepts regular expressions (`apellidos?\\d*`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 26 templates in en\n",
      "We have 18 templates in es\n",
      "We have 24 templates in pt\n",
      "We have 12 templates in fr\n"
     ]
    }
   ],
   "source": [
    "for lang in fieldnames_df['wiki_lang'].unique():\n",
    "    print(f\"We have {len(fieldnames_df[fieldnames_df['wiki_lang'] == lang])} templates in {lang}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Inspecting all featured articles\n",
    "\n",
    "Taking this spreadsheet into account, we define some methods to extract the citation metadata from the articles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_article(filename):\n",
    "    with gzip.open(filename) as fi:\n",
    "        cont = json.loads(fi.read())\n",
    "    return cont\n",
    "    \n",
    "def get_article_data(filename, cont):\n",
    "    lang = os.path.basename(filename).split('-')[1].replace('.json.gz', '')\n",
    "    pageid = cont['parse']['pageid'] \n",
    "    revid = cont['parse']['revid']\n",
    "    article_title = cont['parse']['title']\n",
    "    article_url = f'https://{lang}.wikipedia.org/wiki/{urllib.parse.quote(article_title)}'\n",
    "    \n",
    "    article_data = {\n",
    "        \"article_title\": article_title,\n",
    "        \"article_url\": article_url,\n",
    "        \"page_id\": pageid,\n",
    "        \"revid\": revid,\n",
    "        \"wiki_lang\": lang,\n",
    "    }\n",
    "    return article_data\n",
    "\n",
    "def normalize(s):\n",
    "    return unidecode(s.lower())\n",
    "\n",
    "def get_citation_data(fieldname, template, article_data):\n",
    "    '''\n",
    "    Receives the wikicode of a citation template and a dataframe mapping w2c fieldnames to citation template fieldnames.\n",
    "    Matches w2c fieldnames with citation template parameter names and extracts the values for those parameters\n",
    "    Returns a dictionnary with the extracted data.\n",
    "    '''\n",
    "    ## Build result dict\n",
    "    citation_data = article_data.copy()\n",
    "    citation_data.update({\n",
    "        \"url\": None,\n",
    "        \"source_type\": template.name.strip(),\n",
    "        \"title\": [],\n",
    "        \"author_last\": [],\n",
    "        \"author_first\": [],\n",
    "        \"pub_date\": [],\n",
    "        \"pub_source\": [],\n",
    "    })\n",
    "\n",
    "    ## Find parameters in wikicode\n",
    "    wikicode_params = [param.name.strip() for param in template.params]\n",
    "\n",
    "    ## Match wikicode parameters with the fieldnames of our spreadsheet\n",
    "    for key in [\"title\", \"author_first\", \"author_last\", \"pub_date\", \"pub_source\"]:\n",
    "        params_found_in_wikicode = []\n",
    "        \n",
    "        if pd.isna(fieldname[key]):\n",
    "            continue\n",
    "        for param_re in fieldname[key].split(\", \"):\n",
    "            for wikicode_param in wikicode_params:\n",
    "                if re.fullmatch(normalize(param_re), normalize(wikicode_param)):\n",
    "                    params_found_in_wikicode.append(wikicode_param)\n",
    "\n",
    "        for param_found in params_found_in_wikicode:\n",
    "             citation_data[key].append(template.get(param_found).value.strip())\n",
    "                \n",
    "    url_fieldname = \"url\" if \"url\" in wikicode_params else \"URL\"\n",
    "    citation_data[\"url\"] = template.get(url_fieldname).value.strip()\n",
    "\n",
    "    return citation_data\n",
    "\n",
    "## NH mover remove_na_elements acá"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we apply these methods to the ~10.5k featured articles that we obtained in the previous step. In the next cell, we inspect the json having the content of each article, we parse the wikicode using [mwparserfromhell](https://github.com/earwig/mwparserfromhell) and for each template:\n",
    "\n",
    "1. we verify if it appears in our spreadsheet\n",
    "2. we verify if it contains a URL\n",
    "3. we extract the values for URL, title, author_last, author_first, pub_date, pub_source\n",
    "4. we store the information in a dataframe containing all the citations of the article which respond to 1 and 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Extract citation data for each citation in all featured articles\n",
    "os.makedirs('data/citations-metadata', exist_ok = True)\n",
    "\n",
    "for filename in tqdm(glob('data/articles-content/*json.gz'), desc=f\"Finding citations and extracting metadata\"):\n",
    "    new_filename = f\"data/citations-metadata/{os.path.basename(filename).replace('json.gz', 'csv.gz')}\"\n",
    "    \n",
    "    if os.path.isfile(new_filename):\n",
    "        continue\n",
    "\n",
    "    article_cont = read_article(filename)\n",
    "    wikitext = article_cont['parse']['wikitext']['*']\n",
    "    parsed_wiki = mwparserfromhell.parse(wikitext)\n",
    "    article_data = get_article_data(filename, article_cont)\n",
    "    lang = article_data['wiki_lang']\n",
    "\n",
    "    article_citations = []\n",
    "    for template in parsed_wiki.filter_templates():        \n",
    "        for index, fieldname in fieldnames_df[fieldnames_df['wiki_lang'] == lang].iterrows():\n",
    "            \n",
    "            ## Inspect templates listed in our spreadsheet and extract data for each citation    \n",
    "            if template.name.matches(fieldname['template']):\n",
    "                if template.has_param('url', ignore_empty=True) or template.has_param('URL', ignore_empty=True):\n",
    "\n",
    "                    citation_data = get_citation_data(fieldname, template, article_data)\n",
    "                    article_citations.append(citation_data)\n",
    "                    \n",
    "    article_citations_df = pd.DataFrame(article_citations)\n",
    "    if not article_citations_df.empty:\n",
    "        article_citations_df.to_csv(new_filename, index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Citation template extraction summary\n",
    "\n",
    "Let's take a look at the results. First, we build a dataframe of the citations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9978/9978 [01:00<00:00, 164.34it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_title</th>\n",
       "      <th>article_url</th>\n",
       "      <th>page_id</th>\n",
       "      <th>revid</th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>url</th>\n",
       "      <th>source_type</th>\n",
       "      <th>title</th>\n",
       "      <th>author_last</th>\n",
       "      <th>author_first</th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Argentina</td>\n",
       "      <td>https://es.wikipedia.org/wiki/Argentina</td>\n",
       "      <td>10</td>\n",
       "      <td>141259425</td>\n",
       "      <td>es</td>\n",
       "      <td>http://www.me.gov.ar/efeme/escudonacional/asam...</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>['ASAMBLEA GENERAL CONSTITUYENTE - Sesión del ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>['Ministerio de Economía de la Nación']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Argentina</td>\n",
       "      <td>https://es.wikipedia.org/wiki/Argentina</td>\n",
       "      <td>10</td>\n",
       "      <td>141259425</td>\n",
       "      <td>es</td>\n",
       "      <td>http://www.centrosanfrancisco.org.ar/noticias/...</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>['EN UNIÓN Y LIBERTAD']</td>\n",
       "      <td>['Pezzano, Luciano']</td>\n",
       "      <td>[]</td>\n",
       "      <td>['2005']</td>\n",
       "      <td>['Jornario de las XXIV Jornadas Nacionales de ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Argentina</td>\n",
       "      <td>https://es.wikipedia.org/wiki/Argentina</td>\n",
       "      <td>10</td>\n",
       "      <td>141259425</td>\n",
       "      <td>es</td>\n",
       "      <td>http://www.caei.com.ar/sites/default/files/cul...</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>['La Cultura como Factor de Poder de un Estado']</td>\n",
       "      <td>['Fernández, Sebastián Martín']</td>\n",
       "      <td>[]</td>\n",
       "      <td>['6 de junio de 2016']</td>\n",
       "      <td>['Centro Argentino de Estudios Internacionales...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Argentina</td>\n",
       "      <td>https://es.wikipedia.org/wiki/Argentina</td>\n",
       "      <td>10</td>\n",
       "      <td>141259425</td>\n",
       "      <td>es</td>\n",
       "      <td>https://www.argentina.gob.ar/pais</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>['El País']</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Argentina</td>\n",
       "      <td>https://es.wikipedia.org/wiki/Argentina</td>\n",
       "      <td>10</td>\n",
       "      <td>141259425</td>\n",
       "      <td>es</td>\n",
       "      <td>http://www.oratlas.com/libro-mundial/argentina...</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>['Argentina - Geografía - Libro Mundial de Hec...</td>\n",
       "      <td>['CIA']</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464738</th>\n",
       "      <td>Match AlphaGo - Lee Sedol</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Match%20AlphaGo%...</td>\n",
       "      <td>9998888</td>\n",
       "      <td>189803489</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://www.zdnet.fr/actualites/une-equipe-chin...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>[\"Une équipe chinoise spécialisée dans l'Intel...</td>\n",
       "      <td>['Guillaume Serries']</td>\n",
       "      <td>[]</td>\n",
       "      <td>['4 avril 2016']</td>\n",
       "      <td>['[[ZDNet|zdnet.fr]]']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464739</th>\n",
       "      <td>Match AlphaGo - Lee Sedol</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Match%20AlphaGo%...</td>\n",
       "      <td>9998888</td>\n",
       "      <td>189803489</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://www.nature.com/news/south-korea-trumpet...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>[\"South Korea trumpets $860-million AI fund af...</td>\n",
       "      <td>['Zastrow']</td>\n",
       "      <td>['Mark']</td>\n",
       "      <td>['18 mars 2016']</td>\n",
       "      <td>['Nature News']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464740</th>\n",
       "      <td>Match AlphaGo - Lee Sedol</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Match%20AlphaGo%...</td>\n",
       "      <td>9998888</td>\n",
       "      <td>189803489</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://mobile.lemonde.fr/idees/article/2016/03...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>['Défaite humaine au jeu de go : « Les ordinat...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>['10 mars 2016']</td>\n",
       "      <td>['[[Le Monde#Le Monde.fr|lemonde.fr]]']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464741</th>\n",
       "      <td>Match AlphaGo - Lee Sedol</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Match%20AlphaGo%...</td>\n",
       "      <td>9998888</td>\n",
       "      <td>189803489</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://www.slate.fr/story/115655/lee-sedol-per...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>[\"Cher Lee Sedol, moi aussi j'ai été humilié p...</td>\n",
       "      <td>['[[Ken Jennings]]']</td>\n",
       "      <td>[]</td>\n",
       "      <td>['23 mars 2016']</td>\n",
       "      <td>['[[Slate (magazine)|Slate.fr]]']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464742</th>\n",
       "      <td>Match AlphaGo - Lee Sedol</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Match%20AlphaGo%...</td>\n",
       "      <td>9998888</td>\n",
       "      <td>189803489</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://www.slate.com/articles/technology/techn...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>['The Go Champion, the Grandmaster, and Me']</td>\n",
       "      <td>['[[Ken Jennings]]']</td>\n",
       "      <td>[]</td>\n",
       "      <td>['15 mars 2016']</td>\n",
       "      <td>['[[Slate (magazine)|Slate.com]]']</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>464743 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    article_title  \\\n",
       "0                       Argentina   \n",
       "1                       Argentina   \n",
       "2                       Argentina   \n",
       "3                       Argentina   \n",
       "4                       Argentina   \n",
       "...                           ...   \n",
       "464738  Match AlphaGo - Lee Sedol   \n",
       "464739  Match AlphaGo - Lee Sedol   \n",
       "464740  Match AlphaGo - Lee Sedol   \n",
       "464741  Match AlphaGo - Lee Sedol   \n",
       "464742  Match AlphaGo - Lee Sedol   \n",
       "\n",
       "                                              article_url  page_id      revid  \\\n",
       "0                 https://es.wikipedia.org/wiki/Argentina       10  141259425   \n",
       "1                 https://es.wikipedia.org/wiki/Argentina       10  141259425   \n",
       "2                 https://es.wikipedia.org/wiki/Argentina       10  141259425   \n",
       "3                 https://es.wikipedia.org/wiki/Argentina       10  141259425   \n",
       "4                 https://es.wikipedia.org/wiki/Argentina       10  141259425   \n",
       "...                                                   ...      ...        ...   \n",
       "464738  https://fr.wikipedia.org/wiki/Match%20AlphaGo%...  9998888  189803489   \n",
       "464739  https://fr.wikipedia.org/wiki/Match%20AlphaGo%...  9998888  189803489   \n",
       "464740  https://fr.wikipedia.org/wiki/Match%20AlphaGo%...  9998888  189803489   \n",
       "464741  https://fr.wikipedia.org/wiki/Match%20AlphaGo%...  9998888  189803489   \n",
       "464742  https://fr.wikipedia.org/wiki/Match%20AlphaGo%...  9998888  189803489   \n",
       "\n",
       "       wiki_lang                                                url  \\\n",
       "0             es  http://www.me.gov.ar/efeme/escudonacional/asam...   \n",
       "1             es  http://www.centrosanfrancisco.org.ar/noticias/...   \n",
       "2             es  http://www.caei.com.ar/sites/default/files/cul...   \n",
       "3             es                  https://www.argentina.gob.ar/pais   \n",
       "4             es  http://www.oratlas.com/libro-mundial/argentina...   \n",
       "...          ...                                                ...   \n",
       "464738        fr  http://www.zdnet.fr/actualites/une-equipe-chin...   \n",
       "464739        fr  http://www.nature.com/news/south-korea-trumpet...   \n",
       "464740        fr  http://mobile.lemonde.fr/idees/article/2016/03...   \n",
       "464741        fr  http://www.slate.fr/story/115655/lee-sedol-per...   \n",
       "464742        fr  http://www.slate.com/articles/technology/techn...   \n",
       "\n",
       "       source_type                                              title  \\\n",
       "0         Cita web  ['ASAMBLEA GENERAL CONSTITUYENTE - Sesión del ...   \n",
       "1         Cita web                            ['EN UNIÓN Y LIBERTAD']   \n",
       "2         Cita web   ['La Cultura como Factor de Poder de un Estado']   \n",
       "3         Cita web                                        ['El País']   \n",
       "4         Cita web  ['Argentina - Geografía - Libro Mundial de Hec...   \n",
       "...            ...                                                ...   \n",
       "464738    Lien web  [\"Une équipe chinoise spécialisée dans l'Intel...   \n",
       "464739    Lien web  [\"South Korea trumpets $860-million AI fund af...   \n",
       "464740    Lien web  ['Défaite humaine au jeu de go : « Les ordinat...   \n",
       "464741    Lien web  [\"Cher Lee Sedol, moi aussi j'ai été humilié p...   \n",
       "464742    Lien web       ['The Go Champion, the Grandmaster, and Me']   \n",
       "\n",
       "                            author_last author_first                pub_date  \\\n",
       "0                                    []           []                      []   \n",
       "1                  ['Pezzano, Luciano']           []                ['2005']   \n",
       "2       ['Fernández, Sebastián Martín']           []  ['6 de junio de 2016']   \n",
       "3                                    []           []                      []   \n",
       "4                               ['CIA']           []                      []   \n",
       "...                                 ...          ...                     ...   \n",
       "464738            ['Guillaume Serries']           []        ['4 avril 2016']   \n",
       "464739                      ['Zastrow']     ['Mark']        ['18 mars 2016']   \n",
       "464740                               []           []        ['10 mars 2016']   \n",
       "464741             ['[[Ken Jennings]]']           []        ['23 mars 2016']   \n",
       "464742             ['[[Ken Jennings]]']           []        ['15 mars 2016']   \n",
       "\n",
       "                                               pub_source  \n",
       "0                 ['Ministerio de Economía de la Nación']  \n",
       "1       ['Jornario de las XXIV Jornadas Nacionales de ...  \n",
       "2       ['Centro Argentino de Estudios Internacionales...  \n",
       "3                                                      []  \n",
       "4                                                      []  \n",
       "...                                                   ...  \n",
       "464738                             ['[[ZDNet|zdnet.fr]]']  \n",
       "464739                                    ['Nature News']  \n",
       "464740            ['[[Le Monde#Le Monde.fr|lemonde.fr]]']  \n",
       "464741                  ['[[Slate (magazine)|Slate.fr]]']  \n",
       "464742                 ['[[Slate (magazine)|Slate.com]]']  \n",
       "\n",
       "[464743 rows x 12 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfs = []\n",
    "\n",
    "for filename in tqdm(sorted(os.listdir('data/citations-metadata'))): \n",
    "\n",
    "    with gzip.open(f'data/citations-metadata/{filename}') as fi:\n",
    "        article_citations = pd.read_csv(fi)\n",
    "\n",
    "    dfs.append(article_citations)\n",
    "    \n",
    "all_citations = pd.concat(dfs, ignore_index= True)\n",
    "all_citations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found ~461k citations in ~9.9k articles. This means that there are 629 articles for which no reference was extracted. There are several possible reasons for this:\n",
    "\n",
    "1. the references in those articles do not use citation templates\n",
    "2. the references use citation templates but they do not include a URL\n",
    "3. the citation templates are not listed in our spreadsheet\n",
    "\n",
    "Let's summaryze the names of the citation templates that were identified, and how many instances of each were extracted. First, we normalize the template names (unify case, delete comments, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Normalize template names\n",
    "all_citations.source_type = all_citations.source_type.str.lower()\n",
    "all_citations.source_type = all_citations.source_type.str.replace(\"<!--[^>]+?-->\", \"\", regex=True)\n",
    "all_citations.source_type = all_citations.source_type.str.strip()\n",
    "all_citations.source_type = all_citations.source_type.map(unidecode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>source_type</th>\n",
       "      <th>freq</th>\n",
       "      <th>freq_rel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>en</td>\n",
       "      <td>cite web</td>\n",
       "      <td>122649</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>en</td>\n",
       "      <td>cite news</td>\n",
       "      <td>52053</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>en</td>\n",
       "      <td>cite book</td>\n",
       "      <td>28383</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>en</td>\n",
       "      <td>cite journal</td>\n",
       "      <td>15026</td>\n",
       "      <td>0.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>en</td>\n",
       "      <td>citation</td>\n",
       "      <td>6250</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>en</td>\n",
       "      <td>cite magazine</td>\n",
       "      <td>6155</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>en</td>\n",
       "      <td>cite report</td>\n",
       "      <td>1939</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>en</td>\n",
       "      <td>cite encyclopedia</td>\n",
       "      <td>1060</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>en</td>\n",
       "      <td>cite press release</td>\n",
       "      <td>974</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>en</td>\n",
       "      <td>cite map</td>\n",
       "      <td>471</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>en</td>\n",
       "      <td>cite av media</td>\n",
       "      <td>322</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>en</td>\n",
       "      <td>cite conference</td>\n",
       "      <td>300</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>en</td>\n",
       "      <td>cite thesis</td>\n",
       "      <td>274</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>en</td>\n",
       "      <td>cite interview</td>\n",
       "      <td>189</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>en</td>\n",
       "      <td>cite episode</td>\n",
       "      <td>95</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>en</td>\n",
       "      <td>cite court</td>\n",
       "      <td>57</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>en</td>\n",
       "      <td>cite podcast</td>\n",
       "      <td>27</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>en</td>\n",
       "      <td>cite techreport</td>\n",
       "      <td>26</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>en</td>\n",
       "      <td>cite sign</td>\n",
       "      <td>19</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>en</td>\n",
       "      <td>cite speech</td>\n",
       "      <td>16</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>en</td>\n",
       "      <td>cite patent</td>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>en</td>\n",
       "      <td>cite mailing list</td>\n",
       "      <td>4</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>en</td>\n",
       "      <td>cite newsgroup</td>\n",
       "      <td>4</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>en</td>\n",
       "      <td>cite video game</td>\n",
       "      <td>4</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>en</td>\n",
       "      <td>cite comic</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>es</td>\n",
       "      <td>cita web</td>\n",
       "      <td>49462</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>es</td>\n",
       "      <td>cita publicacion</td>\n",
       "      <td>5820</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>es</td>\n",
       "      <td>cita libro</td>\n",
       "      <td>5317</td>\n",
       "      <td>0.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>es</td>\n",
       "      <td>cita noticia</td>\n",
       "      <td>3322</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>es</td>\n",
       "      <td>obra citada</td>\n",
       "      <td>379</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>es</td>\n",
       "      <td>cita enciclopedia</td>\n",
       "      <td>99</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>es</td>\n",
       "      <td>cita video</td>\n",
       "      <td>73</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>es</td>\n",
       "      <td>cita episodio</td>\n",
       "      <td>33</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>es</td>\n",
       "      <td>cita conferencia</td>\n",
       "      <td>21</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>es</td>\n",
       "      <td>cita tesis</td>\n",
       "      <td>12</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>es</td>\n",
       "      <td>cita entrevista</td>\n",
       "      <td>8</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>es</td>\n",
       "      <td>cita mapa</td>\n",
       "      <td>6</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>fr</td>\n",
       "      <td>lien web</td>\n",
       "      <td>67913</td>\n",
       "      <td>0.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>fr</td>\n",
       "      <td>article</td>\n",
       "      <td>3065</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>fr</td>\n",
       "      <td>ouvrage</td>\n",
       "      <td>979</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>fr</td>\n",
       "      <td>lien video</td>\n",
       "      <td>49</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>fr</td>\n",
       "      <td>lien conference</td>\n",
       "      <td>18</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>fr</td>\n",
       "      <td>lien_web</td>\n",
       "      <td>12</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>fr</td>\n",
       "      <td>interview</td>\n",
       "      <td>12</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>fr</td>\n",
       "      <td>article encyclopedique</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar web</td>\n",
       "      <td>77281</td>\n",
       "      <td>0.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar periodico</td>\n",
       "      <td>5655</td>\n",
       "      <td>0.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar livro</td>\n",
       "      <td>4249</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar jornal</td>\n",
       "      <td>3389</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>pt</td>\n",
       "      <td>citation</td>\n",
       "      <td>764</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar enciclopedia</td>\n",
       "      <td>112</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar video</td>\n",
       "      <td>93</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar tese</td>\n",
       "      <td>86</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar comunicado de imprensa</td>\n",
       "      <td>85</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar relatorio</td>\n",
       "      <td>48</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar conferencia</td>\n",
       "      <td>31</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar episodio</td>\n",
       "      <td>22</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar entrevista</td>\n",
       "      <td>22</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar lista de discussao</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>pt</td>\n",
       "      <td>citar podcast</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   wiki_lang                   source_type    freq  freq_rel\n",
       "0         en                      cite web  122649      0.52\n",
       "1         en                     cite news   52053      0.22\n",
       "2         en                     cite book   28383      0.12\n",
       "3         en                  cite journal   15026      0.06\n",
       "4         en                      citation    6250      0.03\n",
       "5         en                 cite magazine    6155      0.03\n",
       "6         en                   cite report    1939      0.01\n",
       "7         en             cite encyclopedia    1060      0.00\n",
       "8         en            cite press release     974      0.00\n",
       "9         en                      cite map     471      0.00\n",
       "10        en                 cite av media     322      0.00\n",
       "11        en               cite conference     300      0.00\n",
       "12        en                   cite thesis     274      0.00\n",
       "13        en                cite interview     189      0.00\n",
       "14        en                  cite episode      95      0.00\n",
       "15        en                    cite court      57      0.00\n",
       "16        en                  cite podcast      27      0.00\n",
       "17        en               cite techreport      26      0.00\n",
       "18        en                     cite sign      19      0.00\n",
       "19        en                   cite speech      16      0.00\n",
       "20        en                   cite patent       5      0.00\n",
       "21        en             cite mailing list       4      0.00\n",
       "22        en                cite newsgroup       4      0.00\n",
       "23        en               cite video game       4      0.00\n",
       "24        en                    cite comic       1      0.00\n",
       "25        es                      cita web   49462      0.77\n",
       "26        es              cita publicacion    5820      0.09\n",
       "27        es                    cita libro    5317      0.08\n",
       "28        es                  cita noticia    3322      0.05\n",
       "29        es                   obra citada     379      0.01\n",
       "30        es             cita enciclopedia      99      0.00\n",
       "31        es                    cita video      73      0.00\n",
       "32        es                 cita episodio      33      0.00\n",
       "33        es              cita conferencia      21      0.00\n",
       "34        es                    cita tesis      12      0.00\n",
       "35        es               cita entrevista       8      0.00\n",
       "36        es                     cita mapa       6      0.00\n",
       "37        fr                      lien web   67913      0.94\n",
       "38        fr                       article    3065      0.04\n",
       "39        fr                       ouvrage     979      0.01\n",
       "40        fr                    lien video      49      0.00\n",
       "41        fr               lien conference      18      0.00\n",
       "42        fr                      lien_web      12      0.00\n",
       "43        fr                     interview      12      0.00\n",
       "44        fr        article encyclopedique       1      0.00\n",
       "45        pt                     citar web   77281      0.84\n",
       "46        pt               citar periodico    5655      0.06\n",
       "47        pt                   citar livro    4249      0.05\n",
       "48        pt                  citar jornal    3389      0.04\n",
       "49        pt                      citation     764      0.01\n",
       "50        pt            citar enciclopedia     112      0.00\n",
       "51        pt                   citar video      93      0.00\n",
       "52        pt                    citar tese      86      0.00\n",
       "53        pt  citar comunicado de imprensa      85      0.00\n",
       "54        pt               citar relatorio      48      0.00\n",
       "55        pt             citar conferencia      31      0.00\n",
       "56        pt                citar episodio      22      0.00\n",
       "57        pt              citar entrevista      22      0.00\n",
       "58        pt      citar lista de discussao       1      0.00\n",
       "59        pt                 citar podcast       1      0.00"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "template_counts = all_citations[['wiki_lang', 'source_type']].value_counts()\n",
    "template_counts.name = 'freq'\n",
    "template_counts = template_counts.reset_index()\n",
    "template_counts = template_counts.sort_values(['wiki_lang', 'freq'], ascending=[True, False])\n",
    "template_counts = template_counts.reset_index(drop = True)\n",
    "\n",
    "freq_total = template_counts.groupby('wiki_lang')['freq'].sum().to_dict()\n",
    "template_nr = template_counts['wiki_lang'].map(freq_total)\n",
    "template_counts['freq_rel'] = round(template_counts['freq']/template_nr, 2)\n",
    "template_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "template_counts.to_csv('data/citation_templates_freq.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The templates for web sources are the most frequent for all languages (52% cite web, 77% cita web, 94% lien web and  84% citar web). English Wikipedia also shows a considerable number of citations using cite news (22%) while the values for the other languages fall to less than 1% (0.5% cita noticia, 0.4% for article and citar jornal). \n",
    "\n",
    "We can also summaryze the top web domains by language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_citations_domains = all_citations[[\"source_type\", \"wiki_lang\", \"url\"]]\n",
    "all_citations_domains = all_citations_domains[~all_citations_domains['url'].isna()]\n",
    "\n",
    "def find_domain(s):\n",
    "    if s.startswith(\"http\"):\n",
    "        domain = s.split(\"/\")[2].replace(\"www.\", \"\")\n",
    "    else:\n",
    "        domain = False\n",
    "    return domain\n",
    "\n",
    "all_citations_domains[\"domain\"] = all_citations_domains[\"url\"].map(find_domain)\n",
    "domains = all_citations_domains[all_citations_domains[\"domain\"].map(bool)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>domain</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>en</td>\n",
       "      <td>books.google.com</td>\n",
       "      <td>13663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>en</td>\n",
       "      <td>archive.org</td>\n",
       "      <td>12600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>en</td>\n",
       "      <td>nytimes.com</td>\n",
       "      <td>3868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>en</td>\n",
       "      <td>theguardian.com</td>\n",
       "      <td>3013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>en</td>\n",
       "      <td>news.bbc.co.uk</td>\n",
       "      <td>2632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>en</td>\n",
       "      <td>bbc.co.uk</td>\n",
       "      <td>2527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>es</td>\n",
       "      <td>archive.org</td>\n",
       "      <td>2572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>es</td>\n",
       "      <td>books.google.es</td>\n",
       "      <td>925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>es</td>\n",
       "      <td>books.google.com</td>\n",
       "      <td>879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>es</td>\n",
       "      <td>allmusic.com</td>\n",
       "      <td>699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>es</td>\n",
       "      <td>elpais.com</td>\n",
       "      <td>683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>es</td>\n",
       "      <td>billboard.com</td>\n",
       "      <td>536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>fr</td>\n",
       "      <td>statsf1.com</td>\n",
       "      <td>1918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>fr</td>\n",
       "      <td>web.archive.org</td>\n",
       "      <td>1010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>fr</td>\n",
       "      <td>insee.fr</td>\n",
       "      <td>664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>fr</td>\n",
       "      <td>ign.com</td>\n",
       "      <td>654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>fr</td>\n",
       "      <td>grandprix.com</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>fr</td>\n",
       "      <td>jeuxvideo.com</td>\n",
       "      <td>541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>pt</td>\n",
       "      <td>billboard.com</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>pt</td>\n",
       "      <td>books.google.com</td>\n",
       "      <td>1901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>pt</td>\n",
       "      <td>web.archive.org</td>\n",
       "      <td>1580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>pt</td>\n",
       "      <td>nytimes.com</td>\n",
       "      <td>995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>pt</td>\n",
       "      <td>g1.globo.com</td>\n",
       "      <td>986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>pt</td>\n",
       "      <td>mtv.com</td>\n",
       "      <td>878</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   wiki_lang            domain  count\n",
       "0         en  books.google.com  13663\n",
       "1         en       archive.org  12600\n",
       "2         en       nytimes.com   3868\n",
       "3         en   theguardian.com   3013\n",
       "4         en    news.bbc.co.uk   2632\n",
       "6         en         bbc.co.uk   2527\n",
       "5         es       archive.org   2572\n",
       "28        es   books.google.es    925\n",
       "29        es  books.google.com    879\n",
       "41        es      allmusic.com    699\n",
       "43        es        elpais.com    683\n",
       "57        es     billboard.com    536\n",
       "10        fr       statsf1.com   1918\n",
       "23        fr   web.archive.org   1010\n",
       "45        fr          insee.fr    664\n",
       "46        fr           ign.com    654\n",
       "47        fr     grandprix.com    644\n",
       "54        fr     jeuxvideo.com    541\n",
       "8         pt     billboard.com   2020\n",
       "11        pt  books.google.com   1901\n",
       "14        pt   web.archive.org   1580\n",
       "24        pt       nytimes.com    995\n",
       "26        pt      g1.globo.com    986\n",
       "30        pt           mtv.com    878"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain_count_per_lang = domains[[\"wiki_lang\", \"domain\"]].value_counts()\n",
    "domain_count_per_lang.name = \"count\"\n",
    "domain_count_per_lang = domain_count_per_lang.reset_index()\n",
    "domain_count_per_lang = domain_count_per_lang.sort_values(by=[\"wiki_lang\", \"count\"], ascending=[True, False])\n",
    "domain_count_per_lang = domain_count_per_lang.groupby(\"wiki_lang\").head(6)\n",
    "\n",
    "domain_count_per_lang"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The domains corresponding to Google books and Archive.org rank high in English and Spanish. In French and Portuguese we find high values for web.archive.org. Domains from news sites (nytimes.com, theguardian.com, elpais.com, g1.globo.com) are present in all languages except for French. French wikipedia shows high counts for websites related to car racing (statsf1.com, grandprix.com)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "domain_count_per_lang.to_csv('data/top_domains.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3. Querying Citoid API\n",
    "\n",
    "In this section, we request the [Citoid API](https://en.wikipedia.org/api/rest_v1/#/Citation/getCitation) to find the citation metadata for the list of URLs found on the previous step.\n",
    "\n",
    "### 3.1 Validate URLs before calling Citoid\n",
    "\n",
    "To optimize this step and to avoid to unnecessarily load the Citoid service, we will check if the URLs meet the following requierements:\n",
    "\n",
    "- to be well formed\n",
    "- to be public\n",
    "- to have http or https URL scheme\n",
    "- do not point to a pdf (Citoid does not support pdfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_url(url):\n",
    "    is_pdf = url.endswith(\".pdf\") or url.endswith(\".PDF\")\n",
    "    is_ftp = url.startswith(\"ftp\") # validators excludes schemes other than http, https, ftp\n",
    "    is_valid = validators.url(url, public=True)\n",
    "\n",
    "    if is_pdf or is_ftp or not is_valid:\n",
    "        return False\n",
    "    else:\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Validating URLs: 100%|██████████| 9978/9978 [01:01<00:00, 163.08it/s]\n"
     ]
    }
   ],
   "source": [
    "dfs = []\n",
    "\n",
    "for filename in tqdm(sorted(os.listdir('data/citations-metadata')), desc='Validating URLs'):\n",
    "    with gzip.open(f'data/citations-metadata/{filename}') as fi:\n",
    "        article_citations = pd.read_csv(fi)\n",
    "        article_citations = article_citations.dropna(subset=['url'])\n",
    "\n",
    "    valid_urls = article_citations[article_citations['url'].map(validate_url)]\n",
    "    dfs.append(valid_urls)\n",
    "    \n",
    "all_valid_urls = pd.concat(dfs, ignore_index= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_valid_urls.to_csv('citations_metadata_valid_urls.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also eliminate duplicated urls to avoid to request for the same information twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_urls = all_valid_urls.drop_duplicates(subset=['url'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid urls: 431929\n",
      "Valid urls without duplicates: 382984\n"
     ]
    }
   ],
   "source": [
    "print(f'Valid urls: {len(all_valid_urls)}')\n",
    "print(f'Valid urls without duplicates: {len(valid_urls)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Query function and parallel requests\n",
    "\n",
    "We set up the result directory and the function for requesting Citoid's data for one URL. In case of execution interruption, a cache verification is performed before request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "citoid_cache_dir = 'data/citoid-cache/'\n",
    "os.makedirs(citoid_cache_dir, exist_ok = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_and_cache_response(reference_url, outfname, HEADER):\n",
    "    \n",
    "    if os.path.isfile(outfname):\n",
    "        return f\"Citoid response for {reference_url} already in cache\"\n",
    "    \n",
    "    parsed_url = urllib.parse.quote(reference_url).replace('/', '%2F')\n",
    "    response = requests.get(\n",
    "        url = 'https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/'+parsed_url,\n",
    "        headers = HEADER,\n",
    "    )\n",
    "\n",
    "    citoid_data = response.json()\n",
    "\n",
    "    # Add request timestamp in Zulu format\n",
    "    tstamp = f\"{datetime.now().isoformat(timespec='seconds')}Z\"\n",
    "    if isinstance(citoid_data, dict): \n",
    "        citoid_data[\"requestTimestamp\"] = tstamp\n",
    "        # Add requested URL to response\n",
    "        citoid_data[\"requestedUrl\"] = reference_url\n",
    "    else:\n",
    "        citoid_data[0][\"requestTimestamp\"] = tstamp\n",
    "        citoid_data[0][\"requestedUrl\"] = reference_url\n",
    "    \n",
    "    with gzip.open(outfname, 'w') as fo:\n",
    "        fo.write(json.dumps(citoid_data).encode('utf-8'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Citoid's response for each url is saved as a json file named using the pageid of the featured article and the index of the citation on the manual citations dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_fnames = []\n",
    "\n",
    "for index, row in valid_urls.iterrows():\n",
    "\n",
    "    pageid = row['page_id']\n",
    "    outfname = f\"{citoid_cache_dir}page_{pageid:08}-valid_url_{index:06}.json.gz\" ## NH\n",
    "    url = row['url']\n",
    "    \n",
    "    urls_fnames.append((url, outfname))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we send chunks of 500 the parallel requests to Citoid's API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/765 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://www.telegraph.co.uk/news/worldnews/australiaandthepacific/newzealand/7846625/Most-expensive-feather-ever-fetches-4000-at-auction.html', 'citoid-cache/page_00100018-valid_url_000068.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.lillemetropole.fr/mel/institution/competences/dechets-menagers.html', 'citoid-cache/page_00100154-valid_url_000395.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('https://recordsearch.naa.gov.au/SearchNRetrieve/NAAMedia/ShowImage.aspx?B=1492204&T=PDF', 'citoid-cache/page_01001558-valid_url_000427.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 1/765 [02:17<29:10:44, 137.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://www.jstage.jst.go.jp/article/kikaic1979/72/714/72_714_471/_pdf', 'citoid-cache/page_00100340-valid_url_000666.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 2/765 [03:37<21:55:42, 103.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://www.timeanddate.com/worldclock/city.html?n=265', 'citoid-cache/page_00100730-valid_url_001415.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://mdotnetpublic.state.mi.us/tmispublic/', 'citoid-cache/page_01006150-valid_url_001182.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 3/765 [05:51<24:54:48, 117.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://myride.winnipegtransit.com/en/inside-transit/interestingtransitfacts/', 'citoid-cache/page_00100730-valid_url_001579.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('https://www.amazon.fr/Je-suis-n%C3%A9-jour-bleu/dp/2290011436', 'citoid-cache/page_10101628-valid_url_002004.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 4/765 [07:24<22:49:47, 108.00s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://news.bbc.co.uk/1/hi/england/tyne/7579943.stm', 'citoid-cache/page_01011219-valid_url_002143.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('https://acervo.folha.com.br/leitor.do?numero=17133&keyword=SANDY&anchor=5625013&origem=busca&pd=d66444d2be912980e04eef9fe075feee', 'citoid-cache/page_00101409-valid_url_002526.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 5/765 [08:56<21:30:58, 101.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://f5.folha.uol.com.br/celebridades/carnaval/2019/02/fa-de-sandy-lorena-queiroz-febre-de-carinha-de-anjo-tera-bloquinho-proprio-em-sp.shtml', 'citoid-cache/page_00101409-valid_url_002773.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www1.folha.uol.com.br/folha/ilustrada/ult90u7934.shtml', 'citoid-cache/page_00101409-valid_url_002851.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 7/765 [12:27<22:23:08, 106.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://seer.ufrgs.br/index.php/bgg/article/download/40018/25538', 'citoid-cache/page_00010193-valid_url_004145.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.lume.ufrgs.br/bitstream/handle/10183/33270/000114367.pdf?sequence=1', 'citoid-cache/page_00010193-valid_url_004139.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 8/765 [14:04<21:40:56, 103.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://periodicos.furg.br/momento/article/download/4408/2766', 'citoid-cache/page_00010193-valid_url_004251.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.les-lettres-francaises.fr/2010/09/jack-kerouac-et-le-jazz/', 'citoid-cache/page_01019581-valid_url_004311.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 9/765 [15:24<20:10:25, 96.07s/it] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://www.amazon.com/Lives-Pillars-Orthodoxy-Dormition-Skete/dp/0944359043', 'citoid-cache/page_01022186-valid_url_004683.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|▏         | 10/765 [16:51<19:35:01, 93.38s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('http://www.olympic.org/fr/saint-moritz-1948-olympiques-hiver', 'citoid-cache/page_00102878-valid_url_005499.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.cliffordmeth.com/alanmooretalkstocliff-pt2.htm', 'citoid-cache/page_00102915-valid_url_005550.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.portlandonline.com/bes/watershedapp/index.cfm?action=DisplayContent&SubWaterShedID=26&SubjectID=3&TopicID=26&SectionID=1', 'citoid-cache/page_10298609-valid_url_005566.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.cbc.ca/beta/news/politics/sir-john-a-macdonald-toonie-to-celebrate-1st-pm-s-200th-birthday-1.2879467', 'citoid-cache/page_00103110-valid_url_005629.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n",
      "('http://www.cbc.ca/news/canada/ottawa/ottawa-river-parkway-renamed-after-sir-john-a-macdonald-1.1131806', 'citoid-cache/page_00103110-valid_url_005630.json.gz') generated an exception: [Errno Expecting value] upstream request timeout: 0\n"
     ]
    }
   ],
   "source": [
    "for batch_of_ufs in tqdm(chunked(urls_fnames, 500), total=len(urls_fnames)//500):\n",
    "    \n",
    "    with concurrent.futures.ThreadPoolExecutor(max_workers=1000) as executor:\n",
    "        \n",
    "        future_to_url = {}\n",
    "        \n",
    "        for uf in batch_of_ufs:\n",
    "            url, outfname = uf\n",
    "            future = executor.submit(get_and_cache_response, url, outfname, HEADER)\n",
    "            future_to_url[future] = uf\n",
    "            \n",
    "        for future in concurrent.futures.as_completed(future_to_url):\n",
    "            uf = future_to_url[future]\n",
    "            try:\n",
    "                data = future.result()\n",
    "            except Exception as exc:\n",
    "                print( f'{uf} generated an exception: {exc}')\n",
    "    \n",
    "    sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 382126 responses from Citoid\n"
     ]
    }
   ],
   "source": [
    "print(f\"There are {len(os.listdir(citoid_cache_dir))} responses from Citoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, some of the responses correspond to not found URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'method': 'get',\n",
      " 'requestTimestamp': '2022-04-12T01:50:43Z',\n",
      " 'title': 'Not found.',\n",
      " 'type': 'https://mediawiki.org/wiki/HyperSwitch/errors/not_found',\n",
      " 'uri': '/en.wikipedia.org/v1/data/citation/mediawiki/http%3A%2F%2Fwww.history.navy.mil%2Ffaqs%2Ffaq109-1.htm'}\n"
     ]
    }
   ],
   "source": [
    "with gzip.open(f'{citoid_cache_dir}page_01000152-valid_url_000058.json.gz') as fi:\n",
    "    cont = json.loads(fi.read())\n",
    "\n",
    "pprint(cont)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how a correct answer looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'abstractNote': 'Seventy years ago, thousands of Allied troops landed on the '\n",
      "                  \"French coast. Here is the BBC's account of the day.\",\n",
      "  'accessDate': '2022-04-12',\n",
      "  'date': '2014-06-05',\n",
      "  'itemType': 'newspaperArticle',\n",
      "  'key': '8Y9WNVAZ',\n",
      "  'language': 'en-GB',\n",
      "  'libraryCatalog': 'www.bbc.com',\n",
      "  'publicationTitle': 'BBC News',\n",
      "  'requestTimestamp': '2022-04-12T01:50:44Z',\n",
      "  'section': 'Magazine',\n",
      "  'shortTitle': 'D-Day',\n",
      "  'source': ['Zotero'],\n",
      "  'tags': [],\n",
      "  'title': 'D-Day: In the words of the BBC journalists (text version)',\n",
      "  'url': 'https://www.bbc.com/news/magazine-27715250',\n",
      "  'version': 0}]\n"
     ]
    }
   ],
   "source": [
    "with gzip.open(f'{citoid_cache_dir}page_01000152-valid_url_000056.json.gz') as fi:\n",
    "    cont = json.loads(fi.read())\n",
    "\n",
    "pprint(cont)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Tidying up Citoid's response\n",
    "\n",
    "We are only interested in the information corresponding to title, author, publication date and publication source. The following spreadsheet maps Zotero (= Citoid) field names to Web2Cit field names. This mapping will allow us to compare the Citoid response vs the citation metadata collected from Featured articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>field</th>\n",
       "      <th>baseField</th>\n",
       "      <th>type</th>\n",
       "      <th>cslField</th>\n",
       "      <th>w2c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>title</td>\n",
       "      <td>title</td>\n",
       "      <td>regular</td>\n",
       "      <td>title</td>\n",
       "      <td>title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>date</td>\n",
       "      <td>date</td>\n",
       "      <td>regular</td>\n",
       "      <td>issued</td>\n",
       "      <td>date</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>label</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>code</td>\n",
       "      <td>code</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>blogTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>publisher</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>bookTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>caseName</td>\n",
       "      <td>title</td>\n",
       "      <td>regular</td>\n",
       "      <td>title</td>\n",
       "      <td>title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>dateDecided</td>\n",
       "      <td>date</td>\n",
       "      <td>regular</td>\n",
       "      <td>issued</td>\n",
       "      <td>date</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>reporter</td>\n",
       "      <td>reporter</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>company</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>proceedingsTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>dictionaryTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>subject</td>\n",
       "      <td>title</td>\n",
       "      <td>regular</td>\n",
       "      <td>title</td>\n",
       "      <td>title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>encyclopediaTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>distributor</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>forumTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>issueDate</td>\n",
       "      <td>date</td>\n",
       "      <td>regular</td>\n",
       "      <td>issued</td>\n",
       "      <td>date</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>programTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>network</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>institution</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>nameOfAct</td>\n",
       "      <td>title</td>\n",
       "      <td>regular</td>\n",
       "      <td>title</td>\n",
       "      <td>title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>dateEnacted</td>\n",
       "      <td>date</td>\n",
       "      <td>regular</td>\n",
       "      <td>issued</td>\n",
       "      <td>date</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>university</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>studio</td>\n",
       "      <td>publisher</td>\n",
       "      <td>regular</td>\n",
       "      <td>publisher</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>websiteTitle</td>\n",
       "      <td>publicationTitle</td>\n",
       "      <td>regular</td>\n",
       "      <td>container-title</td>\n",
       "      <td>source</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>contributor</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>interviewee</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>director</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>artist</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>sponsor</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>inventor</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>cartographer</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>performer</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>presenter</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>podcaster</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>programmer</td>\n",
       "      <td>author</td>\n",
       "      <td>creator</td>\n",
       "      <td>author</td>\n",
       "      <td>author</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 field         baseField     type         cslField     w2c\n",
       "0                title             title  regular            title   title\n",
       "4                 date              date  regular           issued    date\n",
       "20               label         publisher  regular        publisher  source\n",
       "24                code              code  regular  container-title  source\n",
       "31           blogTitle  publicationTitle  regular  container-title  source\n",
       "36           publisher         publisher  regular        publisher  source\n",
       "38           bookTitle  publicationTitle  regular  container-title  source\n",
       "40            caseName             title  regular            title   title\n",
       "42         dateDecided              date  regular           issued    date\n",
       "44            reporter          reporter  regular  container-title  source\n",
       "49             company         publisher  regular        publisher  source\n",
       "51    proceedingsTitle  publicationTitle  regular  container-title  source\n",
       "54     dictionaryTitle  publicationTitle  regular  container-title  source\n",
       "55             subject             title  regular            title   title\n",
       "56   encyclopediaTitle  publicationTitle  regular  container-title  source\n",
       "57         distributor         publisher  regular        publisher  source\n",
       "60          forumTitle  publicationTitle  regular  container-title  source\n",
       "65    publicationTitle  publicationTitle  regular  container-title  source\n",
       "81           issueDate              date  regular           issued    date\n",
       "88        programTitle  publicationTitle  regular  container-title  source\n",
       "89             network         publisher  regular        publisher  source\n",
       "92         institution         publisher  regular        publisher  source\n",
       "93           nameOfAct             title  regular            title   title\n",
       "96         dateEnacted              date  regular           issued    date\n",
       "98          university         publisher  regular        publisher  source\n",
       "99              studio         publisher  regular        publisher  source\n",
       "100       websiteTitle  publicationTitle  regular  container-title  source\n",
       "104             author            author  creator           author  author\n",
       "105        contributor            author  creator           author  author\n",
       "112        interviewee            author  creator           author  author\n",
       "114           director            author  creator           author  author\n",
       "117             artist            author  creator           author  author\n",
       "118            sponsor            author  creator           author  author\n",
       "121           inventor            author  creator           author  author\n",
       "123       cartographer            author  creator           author  author\n",
       "125          performer            author  creator           author  author\n",
       "128          presenter            author  creator           author  author\n",
       "131          podcaster            author  creator           author  author\n",
       "132         programmer            author  creator           author  author"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping_sheet = \"https://docs.google.com/spreadsheets/d/1YcscX9krRtmSflOLKSa1FeVAfVzwZYo72Mev6h8yWOw/export?gid=505421956&format=csv\"\n",
    "z2wtc = pd.read_csv(mapping_sheet)\n",
    "z2wtc = z2wtc.dropna(subset=[\"w2c\"])\n",
    "z2wtc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this mapping, we inspect the responses from Citoid and extract the metadata for the fields of our interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Mapping Citoid's response:   0%|          | 1589/382125 [00:39<2:40:51, 39.43it/s]"
     ]
    }
   ],
   "source": [
    "rows = []\n",
    "\n",
    "for fname in tqdm(glob('data/citoid-cache-corrected/*json.gz'), desc=\"Mapping Citoid's response\"):\n",
    "    \n",
    "    citation_data = {\n",
    "        \"url\": None,\n",
    "        \"url_citoid\": None,\n",
    "        \"source_type_citoid\": None,\n",
    "        \"title_citoid\": None,\n",
    "        \"author_citoid\": None,\n",
    "        \"pub_date_citoid\": None,\n",
    "        \"pub_source_citoid\": None,\n",
    "    }\n",
    "        \n",
    "    with gzip.open(fname) as fi:\n",
    "        cont = json.load(fi)\n",
    "\n",
    "        if \"requestedUrl\" in cont: # correct answers\n",
    "            citoid_resp = cont\n",
    "\n",
    "            citation_data[\"url\"] = citoid_resp[\"requestedUrl\"]\n",
    "            citation_data[\"url_citoid\"] = citoid_resp[\"url\"]\n",
    "            citation_data[\"source_type_citoid\"] = citoid_resp[\"itemType\"]\n",
    "            \n",
    "            for key in citoid_resp:\n",
    "                for index, row in z2wtc.iterrows():\n",
    "                    if key == row[\"field\"]:\n",
    "                        if row[\"w2c\"] == \"title\":\n",
    "                            citation_data[\"title_citoid\"] = citoid_resp[key]\n",
    "                        elif row[\"w2c\"] == \"author\":\n",
    "                            citation_data[\"author_citoid\"] = citoid_resp[key]\n",
    "                        elif row[\"w2c\"] == \"date\":\n",
    "                            citation_data[\"pub_date_citoid\"] = citoid_resp[key]\n",
    "                        elif row[\"w2c\"] == \"source\":\n",
    "                            citation_data[\"pub_source_citoid\"] = citoid_resp[key]\n",
    "        \n",
    "            rows.append(citation_data)\n",
    "            \n",
    "citoid_citations = pd.DataFrame(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "citoid_citations.to_csv('data/citoid_citations.csv.gz', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 287838 citations generated by Citoid\n"
     ]
    }
   ],
   "source": [
    "print(f\"We have {len(citoid_citations)} citations generated by Citoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Data normalization\n",
    "\n",
    "### 4.1 Manual data\n",
    "\n",
    "Before comparing Citoid's response to the data from the manual citations, we will perform some normalization.\n",
    "\n",
    "We load the metadata extracted from the manual citations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [],
   "source": [
    "manual_citations = pd.read_csv('data/citations_metadata_valid_urls.csv.gz')\n",
    "\n",
    "cols = ['title', \n",
    "        'author_last',\n",
    "        'author_first',\n",
    "        'pub_date',\n",
    "        'pub_source']\n",
    "\n",
    "for col in cols:\n",
    "    manual_citations[col] = manual_citations[col].fillna('None').map(eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to compare the information about the source type (website, book, etc.) given by Citoid with the information of the source type defined in the manual citations. Source types in manual citations are more general than in Citoid: for example, Citoid distinguishes between \"blogPost\" and \"webpage\" while citation templates label both cases as \"Cite web\". To address this problem, a new column is added with the equivalent in  Citoid's itemType for each citation template name.\n",
    "\n",
    "First, we fetch the mapping file defined in Wikipedia for each language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "aux_dir = 'aux/'\n",
    "os.makedirs(aux_dir, exist_ok = True)\n",
    "\n",
    "for lang in manual_citations.wiki_lang.drop_duplicates():\n",
    "    fname = f'{aux_dir}citoid-template-type-map-{lang}.json'\n",
    "    if not os.path.isfile(fname):\n",
    "        r = requests.get(f\"https://{lang}.wikipedia.org/w/api.php?action=parse&page=MediaWiki:Citoid-template-type-map.json&prop=wikitext&format=json\")\n",
    "        data = r.json()\n",
    "        mapping_citoid_template = json.loads(data[\"parse\"][\"wikitext\"][\"*\"])\n",
    "        with open(fname, 'w') as fo:\n",
    "            json.dump(mapping_citoid_template, fo, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that information, we define the mapping function and apply it to the dataframe of manual citations metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [],
   "source": [
    "aux_dir = 'aux/'\n",
    "def map_template_name_citoid_type(lang, template_name):\n",
    "    with open(f'{aux_dir}citoid-template-type-map-{lang}.json') as fi:\n",
    "        mapping = json.load(fi)\n",
    "        \n",
    "    citoid_types = []\n",
    "    for key in mapping:\n",
    "        if normalize(mapping[key]) == normalize(template_name):\n",
    "            citoid_types.append(key)\n",
    "            \n",
    "    return citoid_types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 431928/431928 [01:34<00:00, 4555.57it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_type</th>\n",
       "      <th>source_type_map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>277669</th>\n",
       "      <td>lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350355</th>\n",
       "      <td>citar web</td>\n",
       "      <td>[blogPost, email, forumPost, webpage]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140549</th>\n",
       "      <td>cite book</td>\n",
       "      <td>[book, manuscript]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300297</th>\n",
       "      <td>Lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274931</th>\n",
       "      <td>citar web</td>\n",
       "      <td>[blogPost, email, forumPost, webpage]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89823</th>\n",
       "      <td>lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       source_type                                    source_type_map\n",
       "277669    lien web  [blogPost, email, forumPost, podcast, videoRec...\n",
       "350355   citar web              [blogPost, email, forumPost, webpage]\n",
       "140549   cite book                                 [book, manuscript]\n",
       "300297    Lien web  [blogPost, email, forumPost, podcast, videoRec...\n",
       "274931   citar web              [blogPost, email, forumPost, webpage]\n",
       "89823     lien web  [blogPost, email, forumPost, podcast, videoRec..."
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manual_citations['source_type_map'] = manual_citations.progress_apply(\n",
    "    lambda row: map_template_name_citoid_type(row['wiki_lang'], row['source_type']), axis=1)\n",
    "manual_citations[['source_type', 'source_type_map']].sample(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the cleaning functions for removing empty strings and wikilinks from all the extracted metadata (title, author_first, author_last, pub_date, pub_source). We also set strings to lowercase and translitterate accents and other non ASCII characters to their ASCII equivalent.\n",
    "\n",
    "Dates are also normalized from human format to YYYY-MM-DD using [dateparser](https://dateparser.readthedocs.io/en/latest/index.html). This procedure introduces some noise when the original date has only year information because dateparser assigns the current month and day to the output. This problem will be addressed later. Finally, date values without year information are discarded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_na_elements(li): \n",
    "    return list(filter(None, li))\n",
    "\n",
    "def normalize(s):\n",
    "    return unidecode(s.lower())\n",
    "\n",
    "def normalize_list(li):\n",
    "    return [normalize(s) for s in li] \n",
    "\n",
    "def remove_wikilinks(li): \n",
    "    parsed = []\n",
    "    for el in li:\n",
    "        if '[[' in el and '|' not in el: ## internal link \n",
    "            parsed.append(el.replace('[', '').replace(']', '') )\n",
    "        elif '[[' in el and '|' in el: ## internal link, overwritten title\n",
    "            el = re.sub(r'\\[\\[[^\\|]+\\|', r'', el) \n",
    "            parsed.append(el.replace(']', '') )\n",
    "        elif '[' in el and 'http' in el: ## external link\n",
    "            el = re.sub(r'\\[[^\\ ]+\\ ', r'', el)\n",
    "            parsed.append(el.replace(']', '') )\n",
    "        else: ## plain text\n",
    "            parsed.append(el)\n",
    "    return parsed\n",
    "\n",
    "def parse_dates(li):\n",
    "    parsed_dates = []\n",
    "    for date in li:\n",
    "        ## NH: 1 de enero de 0001\n",
    "        parsed_date = dateparser.parse(date, settings={'REQUIRE_PARTS': ['year']})\n",
    "        if parsed_date:\n",
    "            parsed_dates.append(parsed_date)\n",
    "    return parsed_dates\n",
    "\n",
    "def date2str(dates):\n",
    "    return [date.strftime(\"%Y-%m-%d\") for date in dates]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we apply the cleaning functions to the manual citations metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 431928/431928 [00:00<00:00, 453192.92it/s]\n",
      "100%|██████████| 431928/431928 [00:01<00:00, 306421.58it/s]\n",
      "100%|██████████| 431928/431928 [00:00<00:00, 514720.00it/s]\n",
      "100%|██████████| 431928/431928 [00:00<00:00, 506342.52it/s]\n",
      "100%|██████████| 431928/431928 [00:01<00:00, 416120.15it/s]\n"
     ]
    }
   ],
   "source": [
    "for colname in ['title', 'author_first', 'author_last', 'pub_date', 'pub_source']:\n",
    "    manual_citations[f\"{colname}_clean\"] = manual_citations[colname].apply(remove_na_elements).apply(normalize_list).progress_apply(remove_wikilinks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we set the publishing date to the same format as the response from Citoid to simplify the comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "manual_citations[\"pub_date_clean\"] = manual_citations[\"pub_date_clean\"].progress_apply(parse_dates)\n",
    "manual_citations[\"pub_date_clean\"] = manual_citations[\"pub_date_clean\"].progress_apply(date2str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we remove the noise introduced on date parsing. We rerun date parsing **on a different date** and add new column (pub_date_control). This way the day and month information changes (see items 313649 and 183664) in the example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_date_clean</th>\n",
       "      <th>pub_date_control</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>269312</th>\n",
       "      <td>[22 de noviembre de 2010]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304352</th>\n",
       "      <td>[5 février 1997]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313649</th>\n",
       "      <td>[2001]</td>\n",
       "      <td>[2001-07-25]</td>\n",
       "      <td>[2001-09-06]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183664</th>\n",
       "      <td>[1998]</td>\n",
       "      <td>[1998-07-25]</td>\n",
       "      <td>[1998-09-06]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100976</th>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         pub_date pub_date_clean pub_date_control\n",
       "269312  [22 de noviembre de 2010]   [2010-11-22]     [2010-11-22]\n",
       "304352           [5 février 1997]   [1997-02-05]     [1997-02-05]\n",
       "313649                     [2001]   [2001-07-25]     [2001-09-06]\n",
       "183664                     [1998]   [1998-07-25]     [1998-09-06]\n",
       "100976                         []             []               []"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manual_citations[\"pub_date_control\"] = manual_citations[\"pub_date\"].progress_apply(parse_dates)\n",
    "manual_citations[\"pub_date_control\"] = manual_citations[\"pub_date_control\"].progress_apply(date2str)\n",
    "year_examples = [269312, 304352, 313649, 183664, 100976]\n",
    "manual_citations[[\"pub_date\", \"pub_date_clean\", \"pub_date_control\"]].loc[year_examples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_orig_year_only(l1, l2):\n",
    "    '''\n",
    "    input: l1=list of dates in YYYY-MM-DD format, l2= list of dates same format\n",
    "    output: list of publishing dates YYYY-MM-DD or YYYY\n",
    "    '''\n",
    "    dates = []\n",
    "    for parsed_date in l1:\n",
    "        for diff_parsed_date in l2:\n",
    "            if parsed_date.endswith(\"07-25\") and diff_parsed_date.endswith(\"09-06\"):\n",
    "                dates.append(parsed_date.split(\"-\")[0])\n",
    "            else:\n",
    "                dates.append(parsed_date)\n",
    "    return dates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 431928/431928 [00:11<00:00, 36309.26it/s]\n"
     ]
    }
   ],
   "source": [
    "manual_citations[\"pub_date_clean_new\"] = manual_citations.progress_apply(\n",
    "    lambda row: find_orig_year_only(row[\"pub_date_clean\"], row[\"pub_date_control\"]),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_date_clean</th>\n",
       "      <th>pub_date_control</th>\n",
       "      <th>pub_date_clean_new</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>269312</th>\n",
       "      <td>[22 de noviembre de 2010]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304352</th>\n",
       "      <td>[5 février 1997]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313649</th>\n",
       "      <td>[2001]</td>\n",
       "      <td>[2001-07-25]</td>\n",
       "      <td>[2001-09-06]</td>\n",
       "      <td>[2001]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183664</th>\n",
       "      <td>[1998]</td>\n",
       "      <td>[1998-07-25]</td>\n",
       "      <td>[1998-09-06]</td>\n",
       "      <td>[1998]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100976</th>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         pub_date pub_date_clean pub_date_control  \\\n",
       "269312  [22 de noviembre de 2010]   [2010-11-22]     [2010-11-22]   \n",
       "304352           [5 février 1997]   [1997-02-05]     [1997-02-05]   \n",
       "313649                     [2001]   [2001-07-25]     [2001-09-06]   \n",
       "183664                     [1998]   [1998-07-25]     [1998-09-06]   \n",
       "100976                         []             []               []   \n",
       "\n",
       "       pub_date_clean_new  \n",
       "269312       [2010-11-22]  \n",
       "304352       [1997-02-05]  \n",
       "313649             [2001]  \n",
       "183664             [1998]  \n",
       "100976                 []  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manual_citations[[\"pub_date\", \"pub_date_clean\", \"pub_date_control\", \"pub_date_clean_new\"]].loc[year_examples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_date_clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>269312</th>\n",
       "      <td>[22 de noviembre de 2010]</td>\n",
       "      <td>[2010-11-22]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304352</th>\n",
       "      <td>[5 février 1997]</td>\n",
       "      <td>[1997-02-05]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313649</th>\n",
       "      <td>[2001]</td>\n",
       "      <td>[2001]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183664</th>\n",
       "      <td>[1998]</td>\n",
       "      <td>[1998]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100976</th>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         pub_date pub_date_clean\n",
       "269312  [22 de noviembre de 2010]   [2010-11-22]\n",
       "304352           [5 février 1997]   [1997-02-05]\n",
       "313649                     [2001]         [2001]\n",
       "183664                     [1998]         [1998]\n",
       "100976                         []             []"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manual_citations[\"pub_date_clean\"] = manual_citations[\"pub_date_clean_new\"]\n",
    "manual_citations = manual_citations.drop(columns=[\"pub_date_control\", \"pub_date_clean_new\"])\n",
    "manual_citations[[\"pub_date\", \"pub_date_clean\"]].loc[year_examples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reorder the columns and save:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_title</th>\n",
       "      <th>article_url</th>\n",
       "      <th>page_id</th>\n",
       "      <th>revid</th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>url</th>\n",
       "      <th>source_type</th>\n",
       "      <th>source_type_map</th>\n",
       "      <th>title</th>\n",
       "      <th>title_clean</th>\n",
       "      <th>author_last</th>\n",
       "      <th>author_last_clean</th>\n",
       "      <th>author_first</th>\n",
       "      <th>author_first_clean</th>\n",
       "      <th>pub_date</th>\n",
       "      <th>pub_date_clean</th>\n",
       "      <th>pub_source</th>\n",
       "      <th>pub_source_clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>316464</th>\n",
       "      <td>Somerset</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Somerset</td>\n",
       "      <td>51763</td>\n",
       "      <td>1069538648</td>\n",
       "      <td>en</td>\n",
       "      <td>http://www.naturalengland.org.uk/ourwork/conse...</td>\n",
       "      <td>cite web</td>\n",
       "      <td>[blogPost, email, forumPost, webpage]</td>\n",
       "      <td>['Somerset']</td>\n",
       "      <td>[somerset]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>398453</th>\n",
       "      <td>Theoren Fleury</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Theoren%20Fleury</td>\n",
       "      <td>797197</td>\n",
       "      <td>1069493562</td>\n",
       "      <td>en</td>\n",
       "      <td>http://slam.canoe.ca/Slam/Columnists/Friesen/2...</td>\n",
       "      <td>citation</td>\n",
       "      <td>[artwork, audioRecording, bill, bookSection, c...</td>\n",
       "      <td>['Fleury far from done']</td>\n",
       "      <td>[fleury far from done]</td>\n",
       "      <td>['Friesen']</td>\n",
       "      <td>[friesen]</td>\n",
       "      <td>['Paul']</td>\n",
       "      <td>[paul]</td>\n",
       "      <td>[2008-11-21]</td>\n",
       "      <td>[2008-11-21]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160960</th>\n",
       "      <td>Rise of Neville Chamberlain</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Rise%20of%20Nevi...</td>\n",
       "      <td>25088935</td>\n",
       "      <td>1043732646</td>\n",
       "      <td>en</td>\n",
       "      <td>http://www.measuringworth.com/ppoweruk/</td>\n",
       "      <td>cite web</td>\n",
       "      <td>[blogPost, email, forumPost, webpage]</td>\n",
       "      <td>['Purchasing Power of British Pounds 1264–2008']</td>\n",
       "      <td>[purchasing power of british pounds 1264-2008]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>['MeasuringWorth']</td>\n",
       "      <td>[measuringworth]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248266</th>\n",
       "      <td>Uru: Ages Beyond Myst</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Uru%3A%20Ages%20...</td>\n",
       "      <td>383712</td>\n",
       "      <td>1064104358</td>\n",
       "      <td>en</td>\n",
       "      <td>http://wiki.openuru.org/index.php?title=CyanWo...</td>\n",
       "      <td>cite news</td>\n",
       "      <td>[magazineArticle, newspaperArticle]</td>\n",
       "      <td>['CyanWorlds.com Engine']</td>\n",
       "      <td>[cyanworlds.com engine]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>['OpenUru.org']</td>\n",
       "      <td>[openuru.org]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281791</th>\n",
       "      <td>Parc national de Padjelanta</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Parc%20national%...</td>\n",
       "      <td>4395542</td>\n",
       "      <td>177505717</td>\n",
       "      <td>fr</td>\n",
       "      <td>http://laponia.nu/en/about-us/laponiatjuottjudus/</td>\n",
       "      <td>lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "      <td>['Organisation']</td>\n",
       "      <td>[organisation]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>['Laponia']</td>\n",
       "      <td>[laponia]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149990</th>\n",
       "      <td>Michigan State Trunkline Highway System</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Michigan%20State...</td>\n",
       "      <td>23554903</td>\n",
       "      <td>1067569652</td>\n",
       "      <td>en</td>\n",
       "      <td>https://news.google.com/newspapers?id=ydJOAAAA...</td>\n",
       "      <td>cite news</td>\n",
       "      <td>[magazineArticle, newspaperArticle]</td>\n",
       "      <td>['Gasoline Tax Wins Senate Approval']</td>\n",
       "      <td>[gasoline tax wins senate approval]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[December 13, 1972]</td>\n",
       "      <td>[1972-12-13]</td>\n",
       "      <td>['Ludington Daily News', 'United Press Interna...</td>\n",
       "      <td>[ludington daily news, united press internatio...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  article_title  \\\n",
       "316464                                 Somerset   \n",
       "398453                           Theoren Fleury   \n",
       "160960              Rise of Neville Chamberlain   \n",
       "248266                    Uru: Ages Beyond Myst   \n",
       "281791              Parc national de Padjelanta   \n",
       "149990  Michigan State Trunkline Highway System   \n",
       "\n",
       "                                              article_url   page_id  \\\n",
       "316464             https://en.wikipedia.org/wiki/Somerset     51763   \n",
       "398453     https://en.wikipedia.org/wiki/Theoren%20Fleury    797197   \n",
       "160960  https://en.wikipedia.org/wiki/Rise%20of%20Nevi...  25088935   \n",
       "248266  https://en.wikipedia.org/wiki/Uru%3A%20Ages%20...    383712   \n",
       "281791  https://fr.wikipedia.org/wiki/Parc%20national%...   4395542   \n",
       "149990  https://en.wikipedia.org/wiki/Michigan%20State...  23554903   \n",
       "\n",
       "             revid wiki_lang  \\\n",
       "316464  1069538648        en   \n",
       "398453  1069493562        en   \n",
       "160960  1043732646        en   \n",
       "248266  1064104358        en   \n",
       "281791   177505717        fr   \n",
       "149990  1067569652        en   \n",
       "\n",
       "                                                      url source_type  \\\n",
       "316464  http://www.naturalengland.org.uk/ourwork/conse...    cite web   \n",
       "398453  http://slam.canoe.ca/Slam/Columnists/Friesen/2...    citation   \n",
       "160960            http://www.measuringworth.com/ppoweruk/    cite web   \n",
       "248266  http://wiki.openuru.org/index.php?title=CyanWo...   cite news   \n",
       "281791  http://laponia.nu/en/about-us/laponiatjuottjudus/    lien web   \n",
       "149990  https://news.google.com/newspapers?id=ydJOAAAA...   cite news   \n",
       "\n",
       "                                          source_type_map  \\\n",
       "316464              [blogPost, email, forumPost, webpage]   \n",
       "398453  [artwork, audioRecording, bill, bookSection, c...   \n",
       "160960              [blogPost, email, forumPost, webpage]   \n",
       "248266                [magazineArticle, newspaperArticle]   \n",
       "281791  [blogPost, email, forumPost, podcast, videoRec...   \n",
       "149990                [magazineArticle, newspaperArticle]   \n",
       "\n",
       "                                                   title  \\\n",
       "316464                                      ['Somerset']   \n",
       "398453                          ['Fleury far from done']   \n",
       "160960  ['Purchasing Power of British Pounds 1264–2008']   \n",
       "248266                         ['CyanWorlds.com Engine']   \n",
       "281791                                  ['Organisation']   \n",
       "149990             ['Gasoline Tax Wins Senate Approval']   \n",
       "\n",
       "                                           title_clean  author_last  \\\n",
       "316464                                      [somerset]           []   \n",
       "398453                          [fleury far from done]  ['Friesen']   \n",
       "160960  [purchasing power of british pounds 1264-2008]           []   \n",
       "248266                         [cyanworlds.com engine]           []   \n",
       "281791                                  [organisation]           []   \n",
       "149990             [gasoline tax wins senate approval]           []   \n",
       "\n",
       "       author_last_clean author_first author_first_clean             pub_date  \\\n",
       "316464                []           []                 []                   []   \n",
       "398453         [friesen]     ['Paul']             [paul]         [2008-11-21]   \n",
       "160960                []           []                 []                   []   \n",
       "248266                []           []                 []                   []   \n",
       "281791                []           []                 []                   []   \n",
       "149990                []           []                 []  [December 13, 1972]   \n",
       "\n",
       "       pub_date_clean                                         pub_source  \\\n",
       "316464             []                                                 []   \n",
       "398453   [2008-11-21]                                                 []   \n",
       "160960             []                                 ['MeasuringWorth']   \n",
       "248266             []                                    ['OpenUru.org']   \n",
       "281791             []                                        ['Laponia']   \n",
       "149990   [1972-12-13]  ['Ludington Daily News', 'United Press Interna...   \n",
       "\n",
       "                                         pub_source_clean  \n",
       "316464                                                 []  \n",
       "398453                                                 []  \n",
       "160960                                   [measuringworth]  \n",
       "248266                                      [openuru.org]  \n",
       "281791                                          [laponia]  \n",
       "149990  [ludington daily news, united press internatio...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manual_citations = manual_citations[['article_title', 'article_url', 'page_id', 'revid', 'wiki_lang', 'url','source_type','source_type_map',\n",
    "                                     'title', 'title_clean', 'author_last', 'author_last_clean', 'author_first', 'author_first_clean', \n",
    "                                     'pub_date', 'pub_date_clean', 'pub_source', 'pub_source_clean']]\n",
    "manual_citations.sample(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 299,
   "metadata": {},
   "outputs": [],
   "source": [
    "manual_citations.to_csv('data/citations_metadata_normalized.csv.gz', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Citoid data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the comparison with the manual data, we split the author information from Citoid in two columns (author_first and author_last). We also apply translitteration of non-ASCII to ASCII and lowercase the values for title, author_first, author_last and pub_source, as we did with the corresponding fields in the manual data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_author_component(author_citoid, idx):\n",
    "    if author_citoid:\n",
    "        author_el = [normalize(author[idx]) for author in author_citoid if author_citoid]\n",
    "        return author_el\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cols = ['author_citoid']\n",
    "\n",
    "citoid_citations = pd.read_csv('data/citoid_citations.csv.gz')\n",
    "for col in cols:\n",
    "    citoid_citations[col] = citoid_citations[col].fillna('None').map(eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "citoid_citations['title_citoid'] = citoid_citations['title_citoid'].map(normalize, na_action='ignore')\n",
    "citoid_citations['pub_source_citoid'] = citoid_citations['pub_source_citoid'].map(normalize, na_action='ignore')\n",
    "\n",
    "\n",
    "citoid_citations['author_first_citoid'] = citoid_citations['author_citoid'].map(lambda x : get_author_component(x, idx=0), na_action='ignore')\n",
    "citoid_citations['author_last_citoid'] = citoid_citations['author_citoid'].map(lambda x: get_author_component(x, idx=1), na_action='ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reorder columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>url_citoid</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>title_citoid</th>\n",
       "      <th>author_citoid</th>\n",
       "      <th>author_first_citoid</th>\n",
       "      <th>author_last_citoid</th>\n",
       "      <th>pub_date_citoid</th>\n",
       "      <th>pub_source_citoid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>195853</th>\n",
       "      <td>http://www.rockpapershotgun.com/2009/04/06/int...</td>\n",
       "      <td>https://www.rockpapershotgun.com/best-rpgs</td>\n",
       "      <td>magazineArticle</td>\n",
       "      <td>the best rpgs to play on pc in 2022</td>\n",
       "      <td>[[Graham, Smith]]</td>\n",
       "      <td>['graham']</td>\n",
       "      <td>['smith']</td>\n",
       "      <td>2020-11-11</td>\n",
       "      <td>rock, paper, shotgun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>250718</th>\n",
       "      <td>https://archive.org/details/sugarspiceeveryt00...</td>\n",
       "      <td>https://www.reelviews.net/reelviews/boys-don-t...</td>\n",
       "      <td>webpage</td>\n",
       "      <td>boys don't cry</td>\n",
       "      <td>[[James, Berardinelli]]</td>\n",
       "      <td>['james']</td>\n",
       "      <td>['berardinelli']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>reelviews movie reviews</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10601</th>\n",
       "      <td>https://insidethemagic.net/2021/09/disney-tink...</td>\n",
       "      <td>https://deadline.com/2021/09/gary-marsh-exit-d...</td>\n",
       "      <td>blogPost</td>\n",
       "      <td>gary marsh exiting as head of disney branded t...</td>\n",
       "      <td>[[Nellie, Andreeva], [Nellie, Andreeva]]</td>\n",
       "      <td>['nellie', 'nellie']</td>\n",
       "      <td>['andreeva', 'andreeva']</td>\n",
       "      <td>2021-09-21</td>\n",
       "      <td>deadline</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233857</th>\n",
       "      <td>https://www.nytimes.com/1993/05/15/us/newark-s...</td>\n",
       "      <td>https://www.nytimes.com/1993/05/15/us/newark-s...</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>newark students, both good and bad, make do</td>\n",
       "      <td>[[Felicia R., Lee]]</td>\n",
       "      <td>['felicia r.']</td>\n",
       "      <td>['lee']</td>\n",
       "      <td>1993-05-15</td>\n",
       "      <td>the new york times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142220</th>\n",
       "      <td>http://www.ft.com/cms/s/2/6478998a-fe85-11e2-b...</td>\n",
       "      <td>https://www.ft.com/content/6478998a-fe85-11e2-...</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>bumpy ride</td>\n",
       "      <td>[[Tom, Robbins]]</td>\n",
       "      <td>['tom']</td>\n",
       "      <td>['robbins']</td>\n",
       "      <td>2013-08-09</td>\n",
       "      <td>financial times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178579</th>\n",
       "      <td>https://www.thestar.com/entertainment/movies/a...</td>\n",
       "      <td>https://www.thestar.com/entertainment/movies/2...</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>portia doubleday: michael cera's transformer</td>\n",
       "      <td>[[Richard, Ouzounian]]</td>\n",
       "      <td>['richard']</td>\n",
       "      <td>['ouzounian']</td>\n",
       "      <td>2010-01-05</td>\n",
       "      <td>the toronto star</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      url  \\\n",
       "195853  http://www.rockpapershotgun.com/2009/04/06/int...   \n",
       "250718  https://archive.org/details/sugarspiceeveryt00...   \n",
       "10601   https://insidethemagic.net/2021/09/disney-tink...   \n",
       "233857  https://www.nytimes.com/1993/05/15/us/newark-s...   \n",
       "142220  http://www.ft.com/cms/s/2/6478998a-fe85-11e2-b...   \n",
       "178579  https://www.thestar.com/entertainment/movies/a...   \n",
       "\n",
       "                                               url_citoid source_type_citoid  \\\n",
       "195853         https://www.rockpapershotgun.com/best-rpgs    magazineArticle   \n",
       "250718  https://www.reelviews.net/reelviews/boys-don-t...            webpage   \n",
       "10601   https://deadline.com/2021/09/gary-marsh-exit-d...           blogPost   \n",
       "233857  https://www.nytimes.com/1993/05/15/us/newark-s...   newspaperArticle   \n",
       "142220  https://www.ft.com/content/6478998a-fe85-11e2-...   newspaperArticle   \n",
       "178579  https://www.thestar.com/entertainment/movies/2...   newspaperArticle   \n",
       "\n",
       "                                             title_citoid  \\\n",
       "195853                the best rpgs to play on pc in 2022   \n",
       "250718                                     boys don't cry   \n",
       "10601   gary marsh exiting as head of disney branded t...   \n",
       "233857        newark students, both good and bad, make do   \n",
       "142220                                         bumpy ride   \n",
       "178579       portia doubleday: michael cera's transformer   \n",
       "\n",
       "                                   author_citoid   author_first_citoid  \\\n",
       "195853                         [[Graham, Smith]]            ['graham']   \n",
       "250718                   [[James, Berardinelli]]             ['james']   \n",
       "10601   [[Nellie, Andreeva], [Nellie, Andreeva]]  ['nellie', 'nellie']   \n",
       "233857                       [[Felicia R., Lee]]        ['felicia r.']   \n",
       "142220                          [[Tom, Robbins]]               ['tom']   \n",
       "178579                    [[Richard, Ouzounian]]           ['richard']   \n",
       "\n",
       "              author_last_citoid pub_date_citoid        pub_source_citoid  \n",
       "195853                 ['smith']      2020-11-11     rock, paper, shotgun  \n",
       "250718          ['berardinelli']             NaN  reelviews movie reviews  \n",
       "10601   ['andreeva', 'andreeva']      2021-09-21                 deadline  \n",
       "233857                   ['lee']      1993-05-15       the new york times  \n",
       "142220               ['robbins']      2013-08-09          financial times  \n",
       "178579             ['ouzounian']      2010-01-05         the toronto star  "
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "citoid_citations = citoid_citations[['url', 'url_citoid', 'source_type_citoid', 'title_citoid', 'author_citoid', \n",
    "                                    'author_first_citoid', 'author_last_citoid', 'pub_date_citoid', 'pub_source_citoid']]\n",
    "citoid_citations.sample(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [],
   "source": [
    "citoid_citations.to_csv('data/citoid_citations_normalized.csv.gz', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "In this section, we compare and evaluate the response from Citoid with the values obtained from the manual citations for each URL.\n",
    "\n",
    "First, we load the manual citations and the citations from Citoid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "manual_citations = pd.read_csv('data/citations_metadata_normalized.csv.gz')\n",
    "cols = ['title_clean', \n",
    "        'source_type_map',\n",
    "        'author_last_clean',\n",
    "        'author_first_clean',\n",
    "        'pub_date_clean',\n",
    "        'pub_source_clean']\n",
    "\n",
    "for col in cols:\n",
    "    manual_citations[col] = manual_citations[col].fillna('None').map(eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we verify the information obtained for automatic citations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cols = ['author_last_citoid','author_first_citoid']\n",
    "\n",
    "citoid_citations = pd.read_csv('data/citoid_citations_normalized.csv.gz')\n",
    "for col in cols:\n",
    "    citoid_citations[col] = citoid_citations[col].fillna('None').map(eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.insee.fr/fr/information/4802589</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.observatoire-des-territoires.gouv....</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://archive.org/details/beatlesbiography00...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http://www.leperon.fr/</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://archive.org/details/transformationsv00...</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4724</th>\n",
       "      <td>https://www.rollingstone.com/music/music-lists...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4725</th>\n",
       "      <td>https://archive.org/details/citytroyanditsv00w...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4726</th>\n",
       "      <td>http://www.hispaniaracing.com/news_es.php?nid=400</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4727</th>\n",
       "      <td>https://archive.org/details/nixoninwhitehous00...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4728</th>\n",
       "      <td>https://archive.org/details/ravenuntoldstory00...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4729 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  index  count\n",
       "0           https://www.insee.fr/fr/information/4802589     12\n",
       "1     https://www.observatoire-des-territoires.gouv....     12\n",
       "2     https://archive.org/details/beatlesbiography00...     10\n",
       "3                                http://www.leperon.fr/      8\n",
       "4     https://archive.org/details/transformationsv00...      8\n",
       "...                                                 ...    ...\n",
       "4724  https://www.rollingstone.com/music/music-lists...      2\n",
       "4725  https://archive.org/details/citytroyanditsv00w...      2\n",
       "4726  http://www.hispaniaracing.com/news_es.php?nid=400      2\n",
       "4727  https://archive.org/details/nixoninwhitehous00...      2\n",
       "4728  https://archive.org/details/ravenuntoldstory00...      2\n",
       "\n",
       "[4729 rows x 2 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "citoid_citations_url_counts = citoid_citations[\"url\"].value_counts()\n",
    "citoid_citations_url_counts.name = \"count\"\n",
    "citoid_citations_url_counts =  citoid_citations_url_counts.reset_index()\n",
    "citoid_citations_url_counts[citoid_citations_url_counts[\"count\"] > 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There should not be duplicated URLs. Citoid's response does not return the requested URL but the URL used to obtain the citation metadata (sometimes a redirection, sometimes the canonical URL). The requested URL was not saved. We tried to recover the requestedUrl from the manual data but this shows that there was a problem with our procedure. The requests should be done again to correct this problem.\n",
    "\n",
    "This is why, for the evaluation we will only consider the cases were the columns \"url\" and \"citoid_url\" match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "citoid_citations = citoid_citations[citoid_citations[\"url\"] == citoid_citations[\"url_citoid\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "80228"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(citoid_citations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We merge the data from the automatic citations with the data from the manual citations and keep only the URL and the columns that are useful for the evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = citoid_citations.merge(manual_citations, on=\"url\", how=\"inner\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 91338 citations to evaluate\n"
     ]
    }
   ],
   "source": [
    "print(f\"We have {len(comparison)} citations to evaluate\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>url_citoid</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>title_citoid</th>\n",
       "      <th>author_first_citoid</th>\n",
       "      <th>author_last_citoid</th>\n",
       "      <th>pub_date_citoid</th>\n",
       "      <th>pub_source_citoid</th>\n",
       "      <th>citation_template</th>\n",
       "      <th>source_type_manual</th>\n",
       "      <th>title_manual</th>\n",
       "      <th>author_last_manual</th>\n",
       "      <th>author_first_manual</th>\n",
       "      <th>pub_date_manual</th>\n",
       "      <th>pub_source_manual</th>\n",
       "      <th>article_url</th>\n",
       "      <th>page_id</th>\n",
       "      <th>revid</th>\n",
       "      <th>wiki_lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>59389</th>\n",
       "      <td>https://en.wikisource.org/wiki/Letters_to_his_...</td>\n",
       "      <td>https://en.wikisource.org/wiki/Letters_to_his_...</td>\n",
       "      <td>manuscript</td>\n",
       "      <td>letters to his brother quintus</td>\n",
       "      <td>[marcus tullius]</td>\n",
       "      <td>[cicero]</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Citar livro</td>\n",
       "      <td>[book, manuscript]</td>\n",
       "      <td>[cartas para quinto]</td>\n",
       "      <td>[cicero]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>https://pt.wikipedia.org/wiki/J%C3%BAlio%20C%C...</td>\n",
       "      <td>13467</td>\n",
       "      <td>62983290</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58410</th>\n",
       "      <td>https://www.larep.fr/chapelle-saint-mesmin-453...</td>\n",
       "      <td>https://www.larep.fr/chapelle-saint-mesmin-453...</td>\n",
       "      <td>webpage</td>\n",
       "      <td>14 grands cormorans de moins</td>\n",
       "      <td>[centre]</td>\n",
       "      <td>[france]</td>\n",
       "      <td>2014-01-23</td>\n",
       "      <td>www.larep.fr</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "      <td>[14 grands cormorans de moins]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[2014-01-23]</td>\n",
       "      <td>[]</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/La%20Chapelle-Sa...</td>\n",
       "      <td>300500</td>\n",
       "      <td>189885002</td>\n",
       "      <td>fr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28633</th>\n",
       "      <td>https://books.google.com/books?id=tGfmounsYWMC...</td>\n",
       "      <td>https://books.google.com/books?id=tGfmounsYWMC...</td>\n",
       "      <td>book</td>\n",
       "      <td>baseball in the garden of eden: the secret his...</td>\n",
       "      <td>[john]</td>\n",
       "      <td>[thorn]</td>\n",
       "      <td>2011-03-15</td>\n",
       "      <td>simon and schuster</td>\n",
       "      <td>cite book</td>\n",
       "      <td>[book, manuscript]</td>\n",
       "      <td>[baseball in the garden of eden: the secret hi...</td>\n",
       "      <td>[thorn]</td>\n",
       "      <td>[john]</td>\n",
       "      <td>[2011]</td>\n",
       "      <td>[new york city, simon &amp; schuster]</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Doc%20Adams</td>\n",
       "      <td>33939820</td>\n",
       "      <td>1068110506</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32637</th>\n",
       "      <td>https://www.lemonde.fr/economie/article/2019/0...</td>\n",
       "      <td>https://www.lemonde.fr/economie/article/2019/0...</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>comment la russie a pris la main sur l'opep</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>2019-07-02</td>\n",
       "      <td>le monde.fr</td>\n",
       "      <td>article</td>\n",
       "      <td>[conferencePaper, interview, journalArticle, m...</td>\n",
       "      <td>[comment la russie a pris la main sur l'opep]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[2019-07-02]</td>\n",
       "      <td>[le monde]</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Politique%20%C3%...</td>\n",
       "      <td>12994098</td>\n",
       "      <td>190395263</td>\n",
       "      <td>fr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48576</th>\n",
       "      <td>https://data.canadensys.net/vascan/taxon/26949/</td>\n",
       "      <td>https://data.canadensys.net/vascan/taxon/26949/</td>\n",
       "      <td>webpage</td>\n",
       "      <td>symphyotrichum laeve var. laeve x symphyotrich...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>data.canadensys.net</td>\n",
       "      <td>Cite web &lt;!-- Citation bot bypass--&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>[''symphyotrichum laeve'' var. ''laeve'' x ''s...</td>\n",
       "      <td>[brouillet, desmet, coursol, meades, favreau, ...</td>\n",
       "      <td>[l., p.    &lt;!-- peter desmet --&gt;, f.    &lt;!-- f...</td>\n",
       "      <td>[2020-09-04]</td>\n",
       "      <td>[data.canadensys.net, database of vascular pla...</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Symphyotrichum%2...</td>\n",
       "      <td>14433281</td>\n",
       "      <td>1064059399</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52934</th>\n",
       "      <td>https://www.gamespot.com/reviews/banjo-kazooie...</td>\n",
       "      <td>https://www.gamespot.com/reviews/banjo-kazooie...</td>\n",
       "      <td>blogPost</td>\n",
       "      <td>banjo-kazooie: grunty's revenge review</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>gamespot</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>[blogPost, email, forumPost, podcast, videoRec...</td>\n",
       "      <td>[''banjo-kazooie: grunty's revenge review'']</td>\n",
       "      <td>[frank provo]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[2003]</td>\n",
       "      <td>[{{langue|en|[[gamespot}}]</td>\n",
       "      <td>https://fr.wikipedia.org/wiki/Banjo-Kazooie%20...</td>\n",
       "      <td>339910</td>\n",
       "      <td>190045739</td>\n",
       "      <td>fr</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     url  \\\n",
       "59389  https://en.wikisource.org/wiki/Letters_to_his_...   \n",
       "58410  https://www.larep.fr/chapelle-saint-mesmin-453...   \n",
       "28633  https://books.google.com/books?id=tGfmounsYWMC...   \n",
       "32637  https://www.lemonde.fr/economie/article/2019/0...   \n",
       "48576    https://data.canadensys.net/vascan/taxon/26949/   \n",
       "52934  https://www.gamespot.com/reviews/banjo-kazooie...   \n",
       "\n",
       "                                              url_citoid source_type_citoid  \\\n",
       "59389  https://en.wikisource.org/wiki/Letters_to_his_...         manuscript   \n",
       "58410  https://www.larep.fr/chapelle-saint-mesmin-453...            webpage   \n",
       "28633  https://books.google.com/books?id=tGfmounsYWMC...               book   \n",
       "32637  https://www.lemonde.fr/economie/article/2019/0...   newspaperArticle   \n",
       "48576    https://data.canadensys.net/vascan/taxon/26949/            webpage   \n",
       "52934  https://www.gamespot.com/reviews/banjo-kazooie...           blogPost   \n",
       "\n",
       "                                            title_citoid author_first_citoid  \\\n",
       "59389                     letters to his brother quintus    [marcus tullius]   \n",
       "58410                       14 grands cormorans de moins            [centre]   \n",
       "28633  baseball in the garden of eden: the secret his...              [john]   \n",
       "32637        comment la russie a pris la main sur l'opep                None   \n",
       "48576  symphyotrichum laeve var. laeve x symphyotrich...                None   \n",
       "52934             banjo-kazooie: grunty's revenge review                None   \n",
       "\n",
       "      author_last_citoid pub_date_citoid    pub_source_citoid  \\\n",
       "59389           [cicero]            None                 None   \n",
       "58410           [france]      2014-01-23         www.larep.fr   \n",
       "28633            [thorn]      2011-03-15   simon and schuster   \n",
       "32637               None      2019-07-02          le monde.fr   \n",
       "48576               None            None  data.canadensys.net   \n",
       "52934               None            None             gamespot   \n",
       "\n",
       "                          citation_template  \\\n",
       "59389                           Citar livro   \n",
       "58410                              Lien web   \n",
       "28633                             cite book   \n",
       "32637                               article   \n",
       "48576  Cite web <!-- Citation bot bypass-->   \n",
       "52934                              Lien web   \n",
       "\n",
       "                                      source_type_manual  \\\n",
       "59389                                 [book, manuscript]   \n",
       "58410  [blogPost, email, forumPost, podcast, videoRec...   \n",
       "28633                                 [book, manuscript]   \n",
       "32637  [conferencePaper, interview, journalArticle, m...   \n",
       "48576                                                 []   \n",
       "52934  [blogPost, email, forumPost, podcast, videoRec...   \n",
       "\n",
       "                                            title_manual  \\\n",
       "59389                               [cartas para quinto]   \n",
       "58410                     [14 grands cormorans de moins]   \n",
       "28633  [baseball in the garden of eden: the secret hi...   \n",
       "32637      [comment la russie a pris la main sur l'opep]   \n",
       "48576  [''symphyotrichum laeve'' var. ''laeve'' x ''s...   \n",
       "52934       [''banjo-kazooie: grunty's revenge review'']   \n",
       "\n",
       "                                      author_last_manual  \\\n",
       "59389                                           [cicero]   \n",
       "58410                                                 []   \n",
       "28633                                            [thorn]   \n",
       "32637                                                 []   \n",
       "48576  [brouillet, desmet, coursol, meades, favreau, ...   \n",
       "52934                                      [frank provo]   \n",
       "\n",
       "                                     author_first_manual pub_date_manual  \\\n",
       "59389                                                 []              []   \n",
       "58410                                                 []    [2014-01-23]   \n",
       "28633                                             [john]          [2011]   \n",
       "32637                                                 []    [2019-07-02]   \n",
       "48576  [l., p.    <!-- peter desmet -->, f.    <!-- f...    [2020-09-04]   \n",
       "52934                                                 []          [2003]   \n",
       "\n",
       "                                       pub_source_manual  \\\n",
       "59389                                                 []   \n",
       "58410                                                 []   \n",
       "28633                  [new york city, simon & schuster]   \n",
       "32637                                         [le monde]   \n",
       "48576  [data.canadensys.net, database of vascular pla...   \n",
       "52934                         [{{langue|en|[[gamespot}}]   \n",
       "\n",
       "                                             article_url   page_id  \\\n",
       "59389  https://pt.wikipedia.org/wiki/J%C3%BAlio%20C%C...     13467   \n",
       "58410  https://fr.wikipedia.org/wiki/La%20Chapelle-Sa...    300500   \n",
       "28633          https://en.wikipedia.org/wiki/Doc%20Adams  33939820   \n",
       "32637  https://fr.wikipedia.org/wiki/Politique%20%C3%...  12994098   \n",
       "48576  https://en.wikipedia.org/wiki/Symphyotrichum%2...  14433281   \n",
       "52934  https://fr.wikipedia.org/wiki/Banjo-Kazooie%20...    339910   \n",
       "\n",
       "            revid wiki_lang  \n",
       "59389    62983290        pt  \n",
       "58410   189885002        fr  \n",
       "28633  1068110506        en  \n",
       "32637   190395263        fr  \n",
       "48576  1064059399        en  \n",
       "52934   190045739        fr  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select columns \n",
    "comparison = comparison[['url', 'url_citoid', 'source_type_citoid', 'title_citoid',\n",
    "       'author_first_citoid', 'author_last_citoid','pub_date_citoid', 'pub_source_citoid', 'source_type',\n",
    "       'source_type_map', 'title_clean', 'author_last_clean', 'author_first_clean',  \n",
    "        'pub_date_clean', 'pub_source_clean', 'article_url', 'page_id', 'revid', 'wiki_lang']]\n",
    "\n",
    "# Rename columns\n",
    "comparison = comparison.rename(columns=\n",
    "        {'source_type':'citation_template',\n",
    "        'source_type_map': 'source_type_manual', \n",
    "        'title_clean': 'title_manual',\n",
    "        'author_first_clean': 'author_first_manual',\n",
    "         'author_last_clean': 'author_last_manual',\n",
    "        'pub_date_clean': 'pub_date_manual',\n",
    "        'pub_source_clean': 'pub_source_manual'\n",
    "        })\n",
    "\n",
    "# Replace NaN with None\n",
    "comparison = comparison.replace({np.nan:None})\n",
    "\n",
    "comparison.sample(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison.to_csv(\"data/merge_citoid_manual.csv.gz\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define the general comparison method that calls the particular functions according to the type of data to compare. Scores go from 1 (Citoid's response is a complete match of the data from the manual citation) to 0 (no match).  There are two main approaches:\n",
    "\n",
    "1. in the first one, we check if there is a coincidence between the data from Citoid and the manual data. This is the case for fields having more structured information (source type and publishing date)\n",
    "2. in the second approach, we measure the similarity between the response from Citoid and the manual data by calculating the edit distance. This is the case for fields having less structured data (title, publishing source, author first and author last) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, the general comparison function first evaluates the cases of absent data (perfect score if missing information in both sides, 0 if data is missing in one side only). Then, it considers the field type in order to call specific comparison functions adapted to each data type: \"compare_bool\", \"compare_dates\", \"compare_string_list\" and \"compare_lists\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def is_absent(el):\n",
    "    return (not el) or type(el) == 'NoneType'\n",
    "\n",
    "def compare(citoid_data, manual_data, fieldname):\n",
    "    if is_absent(citoid_data) and is_absent(manual_data):\n",
    "        score = 1\n",
    "    \n",
    "    elif is_absent(citoid_data) or is_absent(manual_data):\n",
    "        score = 0\n",
    "    \n",
    "    else:\n",
    "        if fieldname == 'source_type':\n",
    "            score = compare_bool(citoid_data, manual_data)\n",
    "        elif fieldname == 'pub_date':\n",
    "            score = compare_dates(citoid_data, manual_data)\n",
    "        elif fieldname == 'title' or fieldname == 'pub_source':\n",
    "            score = compare_string_list(citoid_data, manual_data)\n",
    "        else:\n",
    "            score = compare_lists(citoid_data, manual_data)\n",
    "            \n",
    "    return score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5.1 Exact match comparison\n",
    "\n",
    "#### Source type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The evaluation of the source type is done via the **compare_bool** function. Let's see a sample of the source type data returned by Citoid and the source type data identified by Wikipedia's editors to better understand the evaluation principle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>source_type_manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>141471</th>\n",
       "      <td>book</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92526</th>\n",
       "      <td>webpage</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221738</th>\n",
       "      <td>webpage</td>\n",
       "      <td>['conferencePaper', 'dictionaryEntry', 'encycl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174345</th>\n",
       "      <td>webpage</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85958</th>\n",
       "      <td>webpage</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'podcast', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200700</th>\n",
       "      <td>webpage</td>\n",
       "      <td>['book', 'manuscript']</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       source_type_citoid                                 source_type_manual\n",
       "141471               book                                                 []\n",
       "92526             webpage      ['blogPost', 'email', 'forumPost', 'webpage']\n",
       "221738            webpage  ['conferencePaper', 'dictionaryEntry', 'encycl...\n",
       "174345            webpage      ['blogPost', 'email', 'forumPost', 'webpage']\n",
       "85958             webpage  ['blogPost', 'email', 'forumPost', 'podcast', ...\n",
       "200700            webpage                             ['book', 'manuscript']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparison[[\"source_type_citoid\", \"source_type_manual\"]].sample(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a single value on Citoid's side and a list of a limited set of values on the manual citations side (see section 4. Data normalization, for an explanation of this mapping). Consequently, the comparison of the source type is a simple boolean verification: if the response from Citoid is contained in the manual data for source type, the score is 1; if it is absent, the score is 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_bool(source_type_citoid, source_type_manual):\n",
    "    '''\n",
    "    source_type_citoid: string\n",
    "    source_type_manual: list. Limited set of elements\n",
    "    '''\n",
    "    return int(source_type_citoid in source_type_manual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Publishing date\n",
    "\n",
    "In the case of the date information, there is a unique date in Citoid's side and a list of possible dates in the manual citations side (date information can be under different parameters of the citation template, see section 2. Citation template metadata extraction). If different date values appear in the manual data, we keep the score of the best match.\n",
    "\n",
    "For the evaluation, **compare_dates** considers if the full date from Citoid matches any of the manual dates (score = 1) otherwise, it sets the score to 0. Hence, the result of this comparison gives discrete values, as for the source type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_dates(pub_date_citoid, pub_date_manual):\n",
    "    '''\n",
    "    pub_date_citoid: string YY-MM-DD|YY-MM|YY\n",
    "    pub_date_manual: list of strings YY-MM-DD\n",
    "    '''\n",
    "    score = 0\n",
    "    if pub_date_citoid in pub_date_manual:\n",
    "        score = 1\n",
    "    else:\n",
    "        scores = []\n",
    "        year_citoid = get_year(pub_date_citoid)\n",
    "        for element in pub_date_manual:\n",
    "            year_manual = get_year(element)\n",
    "            if year_citoid == year_manual:\n",
    "                scores.append(only_year_score)\n",
    "                break\n",
    "            else:\n",
    "                scores.append(0)\n",
    "        score = max(scores)\n",
    "\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Edit distance comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate Citoid's responses for non structured fields (title, publishing source and author first and author last), **compare_strings** calculates the edit distance (a measure of the number of edits to do to arrive from one string to another) between Citoid's data and the manual data. The resulting score is a continuous range of values going from 1 (full match) to 0 (complete difference)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 299,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_strings(s1, s2):\n",
    "    '''Calculates Levenstein distance between two strings'''\n",
    "    if s1 and s2:\n",
    "        denominator = max(len(s1), len(s2))\n",
    "        score = 1 - edit_distance(s1, s2, transpositions=True)/denominator\n",
    "        return round(score, 2)\n",
    "    elif s1 or s2:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Title and publishing source\n",
    "\n",
    "For these fields, the data in Citoid's side is a string and the data in the manual side is a list of one or more elements. In the case of having different values in the manual data, we keep the score of the best match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def compare_string_list(s, li):\n",
    "    '''\n",
    "    Calculates Levenstein distance between a string and each element of a list of strings\n",
    "    '''\n",
    "    scores = []\n",
    "    for element in li:\n",
    "        scores.append(compare_strings(s, element))\n",
    "        \n",
    "    return max(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Author\n",
    "\n",
    "Author first and author last are the only fields where we have lists on both sides of our comparison because a citation can have several authors. Citoid's data and the manual data can be in different order or one of the lists can have missing information.\n",
    "\n",
    "The comparison method for the author fields faces these difficulties using a permutation strategy: **compare_lists** creates permutations of the manual data, previously adding empty elements to the shortest list in the case of having lists of different length. Then, it compares each element in Citoid's data to the corresponding element in the permutted list measuring the edit distance. For each element, we have several scores resulting from the comparison of each permutation, so we first calculate the average score for the element and finally we keep the best score of the permutations.\n",
    "\n",
    "The number of permutations grows fast depending on the length of the list. For example, the permutations for a list of 3 elements give 6 possibilities but the permutations for a list of 8 elements give 40320 possibilities. Managing the comparison for this number of permutations becomes too demanding for the memory, this is why, for citations having more than 7 elements in author first or author last, **compare_lists** compares the elements without permutting them. Citations of more than 7 elements in author_first or author_last represent 0.5% of the evaluation corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4751582035954367"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Citations having more than 7 elements in the author metadata\n",
    "len(comparison[\n",
    "    (comparison[\"author_first_citoid\"].map(len, na_action=\"ignore\") > 7) | \n",
    "    (comparison[\"author_first_manual\"].map(len) > 7) |\n",
    "     (comparison[\"author_last_citoid\"].map(len, na_action=\"ignore\") > 7) |\n",
    "    (comparison[\"author_last_manual\"].map(len) > 7)\n",
    "])/len(comparison)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_possible_score = 1\n",
    "\n",
    "def compare_lists(au_citoid, au_manual):\n",
    "    '''\n",
    "    au_citoid: list of strings\n",
    "    au_manual: list of strings\n",
    "    Calculates Levenstein distance between i element of au_citoid and i element of each permutation of au_manual.\n",
    "    '''\n",
    "    # Create lists having the same len\n",
    "    shortest = min( [au_manual, au_citoid], key=len)\n",
    "    missing_el_nr = abs( len(au_citoid) - len(au_manual) )\n",
    "    shortest.extend([''] * missing_el_nr)\n",
    "    \n",
    "    if len(au_manual) > 7:\n",
    "        # NOTE: Permutations of more than 7 elements are too demanding in terms of processing\n",
    "        # In those cases, we compare without doing permutations\n",
    "        scores = []\n",
    "        for manual_element, citoid_element in zip(au_manual, au_citoid):\n",
    "            scores.append(compare_strings(manual_element, citoid_element))\n",
    "        return sum(scores)/len(scores)\n",
    "    else:\n",
    "        \n",
    "        best_score = 0\n",
    "\n",
    "        for permutation in permutations(au_manual):\n",
    "            # permutation is a list of N strings\n",
    "            # au_citoid is a list of N strings\n",
    "            element_scores = []\n",
    "            for permu_element, citoid_element in zip(permutation, au_citoid):\n",
    "                element_scores.append(compare_strings(permu_element, citoid_element))\n",
    "            permu_score = sum(element_scores)/len(element_scores)\n",
    "            \n",
    "            best_score = max(best_score, permu_score)\n",
    "            \n",
    "            if best_score == max_possible_score:\n",
    "                break\n",
    "            \n",
    "        return round(best_score, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Applying comparison methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we apply the comparison methods to each field, starting by comparing the title data. Title comparison demands some processing time because of the Levenshtein calculations on strings of a certain length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [08:01<00:00, 189.86it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_title\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"title_citoid\"], row[\"title_manual\"], fieldname=\"title\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell applies the comparison method to the source type data. This is a fast processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [00:02<00:00, 35575.06it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_source_type\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"source_type_citoid\"], row[\"source_type_manual\"], fieldname=\"source_type\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we compare the publishing source data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [00:51<00:00, 1783.30it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_pub_source\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"pub_source_citoid\"], row[\"pub_source_manual\"], fieldname=\"pub_source\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the application of the publishing date comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [00:02<00:00, 39092.56it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_pub_date\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"pub_date_citoid\"], row[\"pub_date_manual\"], fieldname=\"pub_date\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we perform the evaluation of the author information by comparing the author_first and author_last fields from the manual and the Citoid data. This processing can take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [00:51<00:00, 1784.51it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_author_first\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"author_first_citoid\"], row[\"author_first_manual\"], fieldname=\"author_first\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 91338/91338 [01:04<00:00, 1423.53it/s]\n"
     ]
    }
   ],
   "source": [
    "comparison[\"comp_author_last\"] = comparison.progress_apply(\n",
    "    lambda row: compare(row[\"author_last_citoid\"], row[\"author_last_manual\"], fieldname=\"author_last\"),\n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison.to_csv(\"eval/eval_all_citations_v2.csv.gz\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 6. Results\n",
    "\n",
    "In the following section, we present some basic plots and a brief analysis of the results.\n",
    "\n",
    "In our corpus of manual citations gathered from featured articles, some URLs appear several times, cited in different Wikipedias or in different articles of a same Wikipedia. This is why we will examine the scores by URL in the first place. In second place, since many of our URLs belong to a same website, an analysis grouping by domain will be presented. Finally, the scores will be examined by fieldname."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 By URL\n",
    "\n",
    "First, the comparison scores for each URL are selected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "comparison = pd.read_csv(\"eval/eval_all_citations.csv.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 393,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>citation_template</th>\n",
       "      <th>source_type_manual</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>wiki_lang</th>\n",
       "      <th>comp_author_first</th>\n",
       "      <th>comp_author_last</th>\n",
       "      <th>comp_title</th>\n",
       "      <th>comp_source_type</th>\n",
       "      <th>comp_pub_source</th>\n",
       "      <th>comp_pub_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.theage.com.au/entertainment/movies...</td>\n",
       "      <td>cite news</td>\n",
       "      <td>['magazineArticle', 'newspaperArticle']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>en</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://books.google.com/books?id=QSwEAAAAMBAJ...</td>\n",
       "      <td>cite magazine</td>\n",
       "      <td>[]</td>\n",
       "      <td>book</td>\n",
       "      <td>en</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.ecaho.org/about-ecaho/about-us</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'podcast', ...</td>\n",
       "      <td>webpage</td>\n",
       "      <td>fr</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.58</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://www.gq.com/story/pacific-rim-uprising-...</td>\n",
       "      <td>cite web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>en</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.22</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http://bgtfans.com/2013/fifth-falcons-fury-tow...</td>\n",
       "      <td>cite web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>en</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91333</th>\n",
       "      <td>https://www.lequipe.fr/Football/Actualites/Dec...</td>\n",
       "      <td>lien web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'podcast', ...</td>\n",
       "      <td>webpage</td>\n",
       "      <td>fr</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91334</th>\n",
       "      <td>http://www.univale.com/linhas_horarios.asp</td>\n",
       "      <td>citar web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>pt</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91335</th>\n",
       "      <td>https://www.walesonline.co.uk/news/wales-news/...</td>\n",
       "      <td>cite web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>en</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91336</th>\n",
       "      <td>https://web.archive.org/web/20170313102855/htt...</td>\n",
       "      <td>Citar web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'webpage']</td>\n",
       "      <td>webpage</td>\n",
       "      <td>pt</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91337</th>\n",
       "      <td>https://www.jeuneafrique.com/131131/archives-t...</td>\n",
       "      <td>Lien web</td>\n",
       "      <td>['blogPost', 'email', 'forumPost', 'podcast', ...</td>\n",
       "      <td>webpage</td>\n",
       "      <td>fr</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.81</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91338 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     url citation_template  \\\n",
       "0      https://www.theage.com.au/entertainment/movies...         cite news   \n",
       "1      https://books.google.com/books?id=QSwEAAAAMBAJ...     cite magazine   \n",
       "2             https://www.ecaho.org/about-ecaho/about-us          Lien web   \n",
       "3      https://www.gq.com/story/pacific-rim-uprising-...          cite web   \n",
       "4      http://bgtfans.com/2013/fifth-falcons-fury-tow...          cite web   \n",
       "...                                                  ...               ...   \n",
       "91333  https://www.lequipe.fr/Football/Actualites/Dec...          lien web   \n",
       "91334         http://www.univale.com/linhas_horarios.asp         citar web   \n",
       "91335  https://www.walesonline.co.uk/news/wales-news/...          cite web   \n",
       "91336  https://web.archive.org/web/20170313102855/htt...         Citar web   \n",
       "91337  https://www.jeuneafrique.com/131131/archives-t...          Lien web   \n",
       "\n",
       "                                      source_type_manual source_type_citoid  \\\n",
       "0                ['magazineArticle', 'newspaperArticle']            webpage   \n",
       "1                                                     []               book   \n",
       "2      ['blogPost', 'email', 'forumPost', 'podcast', ...            webpage   \n",
       "3          ['blogPost', 'email', 'forumPost', 'webpage']            webpage   \n",
       "4          ['blogPost', 'email', 'forumPost', 'webpage']            webpage   \n",
       "...                                                  ...                ...   \n",
       "91333  ['blogPost', 'email', 'forumPost', 'podcast', ...            webpage   \n",
       "91334      ['blogPost', 'email', 'forumPost', 'webpage']            webpage   \n",
       "91335      ['blogPost', 'email', 'forumPost', 'webpage']            webpage   \n",
       "91336      ['blogPost', 'email', 'forumPost', 'webpage']            webpage   \n",
       "91337  ['blogPost', 'email', 'forumPost', 'podcast', ...            webpage   \n",
       "\n",
       "      wiki_lang  comp_author_first  comp_author_last  comp_title  \\\n",
       "0            en                1.0              0.00        1.00   \n",
       "1            en                0.2              0.12        0.05   \n",
       "2            fr                1.0              1.00        0.58   \n",
       "3            en                0.2              0.17        0.22   \n",
       "4            en                1.0              1.00        0.00   \n",
       "...         ...                ...               ...         ...   \n",
       "91333        fr                1.0              1.00        1.00   \n",
       "91334        pt                1.0              0.00        0.18   \n",
       "91335        en                1.0              1.00        1.00   \n",
       "91336        pt                0.0              0.00        1.00   \n",
       "91337        fr                1.0              1.00        0.81   \n",
       "\n",
       "       comp_source_type  comp_pub_source  comp_pub_date  \n",
       "0                     0             0.00            1.0  \n",
       "1                     0             1.00            0.0  \n",
       "2                     1             1.00            1.0  \n",
       "3                     1             1.00            1.0  \n",
       "4                     1             0.55            0.0  \n",
       "...                 ...              ...            ...  \n",
       "91333                 1             0.60            0.0  \n",
       "91334                 1             0.00            1.0  \n",
       "91335                 1             1.00            1.0  \n",
       "91336                 1             0.15            0.0  \n",
       "91337                 1             1.00            0.0  \n",
       "\n",
       "[91338 rows x 11 columns]"
      ]
     },
     "execution_count": 393,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = comparison[['url','citation_template','source_type_manual','source_type_citoid','wiki_lang', 'comp_author_first', 'comp_author_last', 'comp_title',\n",
    "       'comp_source_type', 'comp_pub_source', 'comp_pub_date']].copy()\n",
    "\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before calculating Citoid's success, let's visualize the most used URLs. As we can see in the plot, following a Zipf's curve, a reduced number of URLs are used many times and a long tail of URLs are used only one time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe4AAAEWCAYAAACg1nQiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAinklEQVR4nO3deZgdVbnv8e8vcwgkIRBiGMMkGkECBhRBZJRBJr3KgYOIiCKKXJBzlEGQyaMi1wGEI0QGg8xEGeSgggEUkAOEeQhIIAECgQQMBMKY5L1/1GqodLo7uztde/fq/fs8z366au0a3tXZnXevWlVrKSIwMzOzPPRpdABmZmZWOyduMzOzjDhxm5mZZcSJ28zMLCNO3GZmZhlx4jYzM8uIE7dZDyXpdUnrdOPxzpZ0fHcdr3RcSbpA0lxJd3X38c1scU7cZq1ICknrtSo7UdJFaXkbSYtSYn1N0uOSDlzaMTorIpaPiKfS8X4r6YedqMNXJN3W6niHRMQpyxJTO7YCdgRWj4jNKzi+mZU4cZt1zfMRsTwwFPgO8BtJGzQ4pkZZC5gREfPbelNSvzrHY9arOXGbLYMoXA/8C/hoZ/eX1FfSsZKeTK33eyStkd4LSetJOhjYD/heauX/Mb1/dGm/RyV9LpV/GDgb2CJt/0oqX6zVLunrkqZJ+pekayWtWnovJB0i6QlJr0g6S5LaiP8g4NzSuU5KVyRmSjpK0gvABZL6lOJ9WdIVkkaUjrO/pKfTe9+XNEPSDu3EvY2kmaX1VSX9XtIcSdMl/d/Seyemc12Yfk+PSBpfen8NSX9I+74s6UxJA9LvZKPSdqtIekPSyM7+G5t1Nydus2WQEtIewMrAtC4c4khgX2BXitb7V4E3yhtExATgYuCn6fL57umtJ4FPAcOAk4CLJI2OiKnAIcAdafvhbcS9HfBjYG9gNPA0cFmrzXYDNqP4QrI3sFPr40TEea3OdUJ66wPACIrW+MHAYcBewKeBVYG5wFkplrHAr4H903srAat38Dsr16MP8EfgAWA1YHvgCEnlWPdIdRsOXAucmfbtC1yX6j4m7X9ZRLyTtv9S6Rj7ApMjYk4tcZlVyYnbrGtWTS3ZN4GrgCMj4r4uHOdrwHER8XhqvT8QES/XsmNEXBkRz0fEooi4HHgCqLWPeT/g/Ii4NyLeBo6haDWPKW3zk4h4JSKeAW4GxtVaKWARcEJEvB0Rb1Ik9+9HxMx0vhOBL6TL6F8ArouIv6f3jk/712IzYGREnBwR76R7An4D7FPa5raIuD4iFgK/AzZO5ZtTfFH4bkTMj4i3IqLlvoCJwL6lqwz7p33NGs6J22xJC4H+rcr6A++W1p9PLdmhwBnAdl081xoULedOk/RlSfenS9mvABtStPxrsSpFSxOAiHgdeJmi1dnihdLyG8DynQhvTkS8VVpfC7iqFOtUit/zqBTLs6VY5qdYarEW6UtU6djHpuO2V49B6QvDGsDTEbGg9UEj4s607TaSPgSsR9FaN2s43zRitqRnKC6dTi2VrQ38s/WGEfG2pKOAxyXtFRFXd/JczwLrAg8vZbvFpvGTtBZFy3J7isvUCyXdD6it7dvwPEXSazneEIpL1M/VHHkn4qWo51cj4vbWG0qaBXy4tL5ciqXFfGC50voHWh13ekSs34UYnwXWlNSvreRN0er+EkXin9Tqi4hZw7jFbbaky4HjJK2e+rB3AHYHJrW1ceoT/Rnwg1ZvDZA0qPTq28bu5wKnSFpfhY9KWqmN7V4Eys90D6FIjnMAVDyOtmGr7VeXNKCdOl4KHChpnKSBwI+AOyNiRjvbL6uzgf9KXziQNFLSnum9ScBukrZK8Z7M4v833Q/sKmmEpA8AR5Teuwt4Ld0IN1jFzX4bStqshpjuAmYBP5E0JP0bbVl6/yLgcxTJ+8JO19isIk7cZks6GfgHcBvFTVQ/BfaLiI5axedTtN52L5U9QtEH3vI6sI39fg5cAdwAzAPOAwa3sd15wNh0OfjqiHiU4svCHRRJeiOg3Jq9KZ3/BUkvtT5YRPyVoi/59xTJa10W7xfubqdTXGq+QdJrwP8CH0+xPAIcClySYpkLzCzt+zuKm89mUPyeLi/VYyHFTXTjgOnASxRfhoYtLaC07+4Ul8GfSef8t9L7zwL3UnxBurWzFTariiKWdkXNzKy+JM0Avpa+YDQyjvMp7mc4rpFxmJW5j9vMrA3pDvvPA5s0OBSzxfhSuZlZK5JOobhh8LSImN7oeMzKfKnczMwsI25xm5mZZSSLPu6VV145xowZ0+gwzMzM6uKee+55KSLaHBs/i8Q9ZswYpkyZ0ugwzMzM6kLS0+2950vlZmZmGXHiNjMzy4gTt5mZWUacuM3MzDLixG1mZpYRJ24zM7OMOHGbmZllpOkS958emsW5tz7V6DDMzMy6pOkS9w2PvsiFd7T7XLuZmVmP1nSJ28zMLGdO3GZmZhlpysQdeCpTMzPLU9MlbjU6ADMzs2VQ6exgkmYArwELgQURMV7SCOByYAwwA9g7IuZWGYeZmVlvUY8W97YRMS4ixqf1o4HJEbE+MDmtm5mZWQ0acal8T2BiWp4I7FXvAMJd3GZmlqmqE3cAN0i6R9LBqWxURMxKyy8Ao9raUdLBkqZImjJnzpzui8id3GZmlrFK+7iBrSLiOUmrADdKeqz8ZkSEpDbbvxExAZgAMH78eLeRzczMqLjFHRHPpZ+zgauAzYEXJY0GSD9nVxmDmZlZb1JZ4pY0RNIKLcvAZ4CHgWuBA9JmBwDXVBVDe9zHbWZmuaryUvko4CpJLee5JCL+LOlu4ApJBwFPA3tXGMMS5E5uMzPLWGWJOyKeAjZuo/xlYPuqzmtmZtabNd3IaWZmZjlz4jYzM8tI0yVuuYvbzMwy1nSJ28zMLGdO3GZmZhlpysQdfpDbzMwy1XSJ213cZmaWs6ZL3GZmZjlz4jYzM8tIUyZu93CbmVmumi5x+zluMzPLWdMlbjMzs5w5cZuZmWWkKRO3H+M2M7NcNV3i9nzcZmaWs6ZL3GZmZjlz4jYzM8tIUybu8JPcZmaWqaZL3H6O28zMctZ0idvMzCxnTtxmZmYZacrE7ee4zcwsV02XuN3HbWZmOWu6xG1mZpYzJ24zM7OMNGXidhe3mZnlqgkTtzu5zcwsX02YuM3MzPLlxG1mZpaRpkzcfo7bzMxyVXniltRX0n2Srkvra0u6U9I0SZdLGlB1DIvHU8+zmZmZda96tLgPB6aW1k8FfhER6wFzgYPqEIOZmVmvUGnilrQ68Fng3LQuYDtgUtpkIrBXlTGYmZn1JlW3uH8JfA9YlNZXAl6JiAVpfSawWls7SjpY0hRJU+bMmdPNYbmT28zM8lRZ4pa0GzA7Iu7pyv4RMSEixkfE+JEjR3ZfXN12JDMzs/rrV+GxtwT2kLQrMAgYCpwODJfUL7W6VweeqzAGMzOzXqWyFndEHBMRq0fEGGAf4KaI2A+4GfhC2uwA4JqqYjAzM+ttOpW4JfWRNHQZz3kUcKSkaRR93uct4/E6zc9xm5lZrpaauCVdImmopCHAw8Cjkr7bmZNExC0RsVtafioiNo+I9SLiixHxdtdC7xo/x21mZjmrpcU9NiLmUTy29SdgbWD/KoMyMzOzttWSuPtL6k+RuK+NiHfx81RmZmYNUUviPgeYAQwB/i5pLWBelUFVzd86zMwsV0t9HCwizgDOKBU9LWnb6kKqlvwkt5mZZWypiVvSQOD/AGNabX9yRTGZmZlZO2oZgOUa4FXgHqCud4CbmZnZ4mpJ3KtHxM6VR1JH4Qe5zcwsU7XcnPYPSRtVHkmd+DluMzPLWS0t7q2Ar0iaTnGpXEBExEcrjczMzMyWUEvi3qXyKMzMzKwmS71UHhFPA8OB3dNreCrLlnu4zcwsV7WMVX44cDGwSnpdJOmwqgOriru4zcwsZ7VcKj8I+HhEzAeQdCpwB/CrKgMzMzOzJdVyV7mAhaX1hbjhamZm1hC1tLgvAO6UdFVa34sGzKHdnfwYt5mZ5aqWscp/LukWisfCAA6MiPsqjapC8oPcZmaWsXYTt6ShETFP0giK2cFmlN4bERH/qj48MzMzK+uoxX0JsBvFGOXli8tK6+tUGJeZmZm1od3EHRG7pZ9r1y+c+vBY5WZmlqtanuOeXEuZmZmZVa+jPu5BwHLAypJW5P1HwIYCq9UhNjMzM2uloz7ubwBHAKtS9HO3JO55wJnVhmVmZmZt6aiP+3TgdEmHRUSvGiXNPdxmZparWp7j/pWkDYGxwKBS+YVVBlYVP8ZtZmY5W2rilnQCsA1F4r6eYprP24AsE7eZmVnOahmr/AvA9sALEXEgsDEwrNKozMzMrE21JO43I2IRsEDSUGA2sEa1YVXMndxmZpapWiYZmSJpOPAbirvLX6eY1jNL8sRmZmaWsVpuTvtWWjxb0p+BoRHxYLVhmZmZWVs6NXJaRMyIiAdrGTlN0iBJd0l6QNIjkk5K5WtLulPSNEmXSxqwbFUwMzNrHu0m7pR4R5BGTpM0Ir3GUNvIaW8D20XExsA4YGdJnwBOBX4REesBc4GDlrUSneUubjMzy1VHLe5vUPRpfwi4Ny3fA1xDDSOnReH1tNo/vQLYDpiUyicCe3Ul8K7yc9xmZpazSkdOk9SXItmvB5wFPAm8EhEL0iYz8bjnZmZmNetokpHtIuIm4DlJn2/9fkT8YWkHj4iFwLh0V/pVFK33mkg6GDgYYM0116x1NzMzs16to7vKPw3cBOzexnsBLDVxv7dxxCuSbga2AIZL6pda3asDz7WzzwRgAsD48eO7tVva83GbmVmuOrpUfkL6eWBXDixpJPBuStqDgR0pbky7mWI0tsuAAyj6zOvGXdxmZpazWh4H+1G61N2yvqKkH9Zw7NHAzZIeBO4GboyI64CjgCMlTQNWAs7rUuRmZmZNqJaR03aJiGNbViJirqRdgeM62ikN0rJJG+VPAZt3NlAzMzOrbazyvpIGtqyky94DO9i+x3MPt5mZ5aqWFvfFwGRJF6T1Aymev86Sn+M2M7Oc1TJW+amSHgB2SEWnRMRfqg3LzMzM2lJLi5uI+DPw54pjMTMzs6WopY+71/Fj3GZmlqumS9xyJ7eZmWWso9nBJqefp9YvHDMzM+tIR33coyV9EthD0mW0GnQsIu6tNDIzMzNbQkeJ+wfA8RTjif+81Xst03NmKfwkt5mZZaqjsconAZMkHR8Rp9Qxpkq5h9vMzHJWy3Pcp0jaA9g6Fd2Sxhw3MzOzOqtlkpEfA4cDj6bX4ZJ+VHVgZmZmtqRaBmD5LDAuIhYBSJoI3Acc2+FePZif4zYzs1zV+hz38NLysAriqB93cpuZWcZqaXH/GLhP0s0UaW9r4OhKozIzM7M21XJz2qWSbgE2S0VHRcQLlUZlZmZmbap1kpFZwLUVx1I37uI2M7NcNd9Y5e7kNjOzjDVd4jYzM8tZh4lbUl9Jj9UrGDMzM+tYh4k7IhYCj0tas07x1Ic7uc3MLFO13Jy2IvCIpLuA+S2FEbFHZVFVyNNxm5lZzmpJ3MdXHoWZmZnVpJbnuP8maS1g/Yj4q6TlgL7Vh2ZmZmat1TLJyNeBScA5qWg14OoKY6qc5+M2M7Nc1fI42KHAlsA8gIh4AlilyqCq5C5uMzPLWS2J++2IeKdlRVI/fF+2mZlZQ9SSuP8m6VhgsKQdgSuBP1YblpmZmbWllsR9NDAHeAj4BnA9cFyVQVXN83GbmVmuarmrfJGkicCdFJfIH4/IN/X5OW4zM8tZLXeVfxZ4EjgDOBOYJmmXGvZbQ9LNkh6V9Iikw1P5CEk3Snoi/VxxWSthZmbWLGq5VP4zYNuI2CYiPg1sC/yihv0WAP8REWOBTwCHShpLcel9ckSsD0xO62ZmZlaDWhL3axExrbT+FPDa0naKiFkRcW9afg2YSvEM+J7AxLTZRGCvzgTcHbK9zm9mZk2v3T5uSZ9Pi1MkXQ9cQZHzvgjc3ZmTSBoDbELRTz4qImalt14ARrWzz8HAwQBrrtl9c5x4Pm4zM8tZRzen7V5afhH4dFqeAwyu9QSSlgd+DxwREfNUujssIkJSmw3giJgATAAYP368G8lmZmZ0kLgj4sBlPbik/hRJ++KI+EMqflHS6IiYJWk0MHtZz2NmZtYslvo4mKS1gcOAMeXtlzatp4qm9XnA1Ij4eemta4EDgJ+kn9d0OupllPHTbGZm1uRqmdbzaooE/EdgUSeOvSWwP/CQpPtT2bEUCfsKSQcBTwN7d+KYy8zPcZuZWc5qSdxvRcQZnT1wRNxG+3N6bN/Z45mZmVltift0SScANwBvtxS2POplZmZm9VNL4t6I4pL3drx/qTzSepbcw21mZrmqJXF/EVinPLVnztzFbWZmOatl5LSHgeEVx2FmZmY1qKXFPRx4TNLdLN7H3eHjYGZmZtb9akncJ1QeRZ35MW4zM8tVLfNx/60egdSNH+Q2M7OM1TJy2mu8fyP2AKA/MD8ihlYZmJmZmS2plhb3Ci3LaRjTPSnm1zYzM7M6q+Wu8vdE4Wpgp2rCMTMzs47Ucqn886XVPsB44K3KIqqYe7jNzCxntdxVXp6XewEwg+JyuZmZmdVZLX3cyzwvt5mZmXWPdhO3pB90sF9ExCkVxFM3EYH8aJiZmWWmoxb3/DbKhgAHASsBWSZu52ozM8tZu4k7In7WsixpBeBw4EDgMuBn7e1nZmZm1emwj1vSCOBIYD9gIrBpRMytR2BmZma2pI76uE8DPg9MADaKiNfrFlUdRPiyuZmZ5aejAVj+A1gVOA54XtK89HpN0rz6hNf95Ce5zcwsYx31cXdqVDUzMzOrnpOzmZlZRpoucb88/20AFizypNxmZpafpkvco4YOAmBROHGbmVl+mi5x9+tT3JzmxG1mZjlqusTdJz0DttCXys3MLEPNl7hbWtyLGhyImZlZFzRd4u6bHuP2pXIzM8tR0yXulhb3QiduMzPLUPMlbrVcKnfiNjOz/DRd4u773l3lDQ7EzMysCypL3JLOlzRb0sOlshGSbpT0RPq5YlXnb0/K275UbmZmWaqyxf1bYOdWZUcDkyNifWByWq8rXyo3M7OcVZa4I+LvwL9aFe9JMa836edeVZ2/PS2Xyt98d2G9T21mZrbM6t3HPSoiZqXlF4BR7W0o6WBJUyRNmTNnTrcF8M6C4gHut9/1g9xmZpafht2cFhEBtHu9OiImRMT4iBg/cuTIbjtvy1jl7uM2M7Mc1TtxvyhpNED6ObvO53/vUvlCD51mZmYZqnfivhY4IC0fAFxT5/O/l7gXLHSL28zM8lPl42CXAncAG0iaKekg4CfAjpKeAHZI63XV1yOnmZlZxvpVdeCI2Ledt7av6py16NfHs4OZmVm+mnbktAVO3GZmlqGmS9wtA7BMnzO/wZGYmZl1XtMl7tHDisfB+rXM72lmZpaRpkvcgwb0Bd4fiMXMzCwnTZe4B/Yrqvy2E7eZmWWo6RL3gL5FlR+dNa/BkZiZmXVe0yVupZvThg3u3+BIzMzMOq/pEjfAqsMGuY/bzMyy1JSJu3+/Psx+7e1Gh2FmZtZpTZm4312wiAeefaXRYZiZmXVaUybudVdZnuXSY2FmZmY5acrEvcaI5Xh3ofu4zcwsP02ZuAf07cNLr7/Dq2++2+hQzMzMOqUpE/cqQwcCcN6tTzU4EjMzs85pysR9yNbrAjDvrQUNjsTMzKxzmjJx9+kjVl5+IO+4n9vMzDLTlIkbijHLH5r5Ktfc/1yjQzEzM6tZ0ybudUYO4aHnXuXwy+5n7vx3Gh2OmZlZTZo2cU88cHNO2fMjALz57sIGR2NmZlabpk3cffqIIQP7AZ6b28zM8tGv0QE00sB+xehpP7vxnwwb3G+x8m9vux4rDhnQqNDMzMza1NSJ+4Ojlme14YP5x7SX3itbsCh49c13GbfGcHbfeNUGRmdmZrakpk7c649agduP3m6xsmdefoOtT7uZt3353MzMeqCm7eNuz8D+xa/E/d5mZtYTNXWLuy0D+haJ+4Lbp3PDoy/U7bz9+ogjd9yAsasOrds5zcwsP07crQwb3J/PbjSamXPfqNvz3QE8OPNVNllzRSduMzPrkBN3K336iLP227Su51y0KFjn2Ovdr25mZkvlPu4eoE8fMaBvH/erm5nZUrnF3UMM6NeHy+5+hslTX2x0KGZm1klbrrcyJ+7xkbqcy4m7h/j2duvx4MxXGh2GmZl1wehhg+p2roYkbkk7A6cDfYFzI+InjYijJznk0+s2OgQzM8tA3fu4JfUFzgJ2AcYC+0oaW+84zMzMctSIm9M2B6ZFxFMR8Q5wGbBnA+IwMzPLTiMS92rAs6X1malsMZIOljRF0pQ5c+bULTgzM7OerMc+DhYREyJifESMHzlyZKPDMTMz6xEakbifA9Yora+eyszMzGwpGpG47wbWl7S2pAHAPsC1DYjDzMwsO3V/HCwiFkj6NvAXisfBzo+IR+odh5mZWY4a8hx3RFwPXN+Ic5uZmeVMEdHoGJZK0hzg6W485MrAS914vEbqLXXpLfWA3lOX3lIPcF16ot5SD6imLmtFRJt3ZmeRuLubpCkRMb7RcXSH3lKX3lIP6D116S31ANelJ+ot9YD616XHPg5mZmZmS3LiNjMzy0izJu4JjQ6gG/WWuvSWekDvqUtvqQe4Lj1Rb6kH1LkuTdnHbWZmlqtmbXGbmZllyYnbzMwsI02XuCXtLOlxSdMkHd3oeAAknS9ptqSHS2UjJN0o6Yn0c8VULklnpPgflLRpaZ8D0vZPSDqgVP4xSQ+lfc6QpIrqsYakmyU9KukRSYdnXJdBku6S9ECqy0mpfG1Jd6bzX56G7UXSwLQ+Lb0/pnSsY1L545J2KpXX7bMoqa+k+yRdl3k9ZqR///slTUll2X2+0rmGS5ok6TFJUyVtkVtdJG2Q/i1aXvMkHZFbPUrn+o6Kv/eHJV2q4v+Bnve3EhFN86IYYvVJYB1gAPAAMLYHxLU1sCnwcKnsp8DRaflo4NS0vCvwJ0DAJ4A7U/kI4Kn0c8W0vGJ67660rdK+u1RUj9HApml5BeCfwNhM6yJg+bTcH7gznfcKYJ9UfjbwzbT8LeDstLwPcHlaHps+ZwOBtdPnr2+9P4vAkcAlwHVpPdd6zABWblWW3ecrnWsi8LW0PAAYnmtd0vn6Ai8Aa+VYD4rppacDg0t/I1/piX8rlf0j9sQXsAXwl9L6McAxjY4rxTKGxRP348DotDwaeDwtnwPs23o7YF/gnFL5OalsNPBYqXyx7Squ0zXAjrnXBVgOuBf4OMXoSP1af54oxt7fIi33S9up9WesZbt6fhYpZuCbDGwHXJfiyq4e6fgzWDJxZ/f5AoZRJAnlXpfSOT4D3J5rPSgS97MUXx76pb+VnXri30qzXSpv+YdpMTOV9USjImJWWn4BGJWW26tDR+Uz2yivVLpstAlFSzXLuqi4vHw/MBu4keLb8isRsaCN878Xc3r/VWAlOl/HKvwS+B6wKK2vRJ71AAjgBkn3SDo4leX4+VobmANcoKIL41xJQ8izLi32AS5Ny9nVIyKeA/4f8Awwi+Kzfw898G+l2RJ3lqL4epbNc3uSlgd+DxwREfPK7+VUl4hYGBHjKFqsmwMfamxEnSdpN2B2RNzT6Fi6yVYRsSmwC3CopK3Lb2b0+epH0T3264jYBJhPcUn5PRnVhdTvuwdwZev3cqlH6offk+JL1arAEGDnhgbVjmZL3M8Ba5TWV09lPdGLkkYDpJ+zU3l7deiofPU2yishqT9F0r44Iv6QirOsS4uIeAW4meJS13BJLbPqlc//Xszp/WHAy3S+jt1tS2APSTOAyygul5+eYT2A91pFRMRs4CqKL1Q5fr5mAjMj4s60PokikedYFyi+SN0bES+m9RzrsQMwPSLmRMS7wB8o/n563t9KlX0ePe1F8S33KYpvVC03B3yk0XGl2MaweB/3aSx+c8dP0/JnWfzmjrtS+QiKPrMV02s6MCK91/rmjl0rqoOAC4FftirPsS4jgeFpeTBwK7AbRYuifKPKt9LyoSx+o8oVafkjLH6jylMUN6nU/bMIbMP7N6dlVw+KFtAKpeV/ULSIsvt8pXPdCmyQlk9M9ci1LpcBB5bWs6sHxT0sj1Dc0yKKmwcP64l/K5X9J9FTXxR3Nf6Tor/y+42OJ8V0KUWfyrsU38QPougrmQw8Afy19CEWcFaK/yFgfOk4XwWmpVf5j2g88HDa50xa3RDTjfXYiuKS2IPA/em1a6Z1+ShwX6rLw8APUvk66T+SaekPemAqH5TWp6X31ykd6/sp3scp3RFb788iiyfu7OqRYn4gvR5pOVeOn690rnHAlPQZu5oiYWVXF4ovUS8Dw0pl2dUjnesk4LF0vt9RJN8e97fiIU/NzMwy0mx93GZmZllz4jYzM8uIE7eZmVlGnLjNzMwy4sRtZmaWESdusy6SNEalGd1S2YmS/jMt/1bS9DRr0gOSti9td4uk8R0ce1VJk9p5r91902xT63StRosdZy9JY7uw3+ttlC3xe+rkMT+VZmy6X9Lgrh6ndLzLJK2/rMcxaxQnbrNqfTeKYVOPoBi8oSYR8XxEfKEzJ5L0EaBvRDzVqQiXPE4/YC+KWY56gv2AH0fEuIh4s6WwNJpVZ/2aYux2syw5cZvVxx10YkKBcitV0uDUSpwq6SqKkdzash/FjGwtE6T8Ns0r/JCk76TycZL+N82FfFVpnuRbJP1SxRzXR1GMO31aauWum15/TpN73CrpQ2m/tSXdkc7xww6q1E/SxakOkyQtJ2k7SVeX6rxjql/59/A1YG/glLT/Nun81wKPpnqeJunuVKdvpP0k6cw09/FfJV0vqeWL0K3ADsuQ+M0ayonbrD52phgdqyu+CbwRER8GTgA+1s52W1LMZgTFqFyrRcSGEbERcEEqvxA4KiI+SjFy1Qml/QdExPiI+C/gWtLVgoh4EpgAHBYRHwP+E/jvtM/pFBNlbEQx+l97NgD+O9VhHsVcxjcDH5I0Mm1zIHB+eaeIOLcUy36peFPg8Ij4IMUog69GxGbAZsDXJa0NfC6dcyzwZeCTpWMuohjtauMO4jXrsZy4zbquvWEHy+WnSfoncAlwahfPszVwEUBEPEgxRGZbRlNMFQnFmMjrSPqVpJ2BeZKGUYy//re0zcR07BaXt3XQNNvbJ4Er0zSn56RzQfFloWUqx991UIdnI+L2tHwRxSxfkfb5kqThFJO4/KmDY7S4KyKmp+XPAF9Ocd1JMdTm+qlel0Yxw9vzwE2tjjGbYgYos+z4UpFZ171MMb50WctkCS2+GxGTJB1G0Zpsr7XcHd6kGD+ZiJgraWNgJ+AQisvN31nK/vPbKe9DMSfxuHber2Xc5NbbtKxfAPwReAu4Mt6f97gj5ThFcSXgL+UNJO26lGMMovh9mWXHLW6zLoqI14FZkrYDkDSC4pL4bW1sfibQR9JOXTjV34F/T+fYkGIClLZMBdZL260M9ImI3wPHAZtGxKvAXEmfStvvD/ytzSPBa8AKAFHMqT5d0hfTsZW+FADcTjEzEhR97O1ZU9IWafnfSb+j1Bp+PsV4QTv7duQvwDdVTCeLpA9KGkLxO/u31Ac+Gti21X4fpJhIwiw7Ttxmy+bLwPHpUu1NwEmpT3gx6bLwD1n8bub/kTQzva7s4By/BpaXNBU4mff7sVv7H4oZwKC4Ee6WFNdFwDGp/ACKy/cPUvSDn9zOsS4DvivpPknrUiTlgyS1zMy1Z9rucOBQSQ/R8c13j6ftplJcpfh16b2LKS6lT+1g//acCzwK3Jtu5juH4kriVRQzUz1K0a9/R8sOkkYBb0bEC104n1nDeXYws14iPeN8M7BlRCxsdDy1knQmcF9EnFfhOX5LMaXppHSH/bwqz2dWJbe4zXqJ9IzzCXTisbNGk3QPxaX/i+p42lcobswzy5Jb3GZmZhlxi9vMzCwjTtxmZmYZceI2MzPLiBO3mZlZRpy4zczMMvL/AQGjf9tWSTKbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "url_counts = scores[\"url\"].value_counts()\n",
    "plot_data = url_counts.reset_index()\n",
    "\n",
    "ax = plot_data.plot(title=\"URL citation frequency\", legend=False, figsize=[8,4])\n",
    "ax.set_ylabel(\"Number of citations\")\n",
    "ax.set_xlabel(\"URL id (sorted by freq)\")\n",
    "\n",
    "plt.savefig(\"plots/url_freq_zipf.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7433 URLs were used in more than one citation\n"
     ]
    }
   ],
   "source": [
    "url_counts_several = url_counts[url_counts > 1]\n",
    "print(f\"{len(url_counts_several)} URLs were used in more than one citation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72795 URLs were used in only one citation\n"
     ]
    }
   ],
   "source": [
    "url_counts_unique = url_counts[url_counts == 1]\n",
    "print(f\"{len(url_counts_unique)} URLs were used in only one citation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.4k URLs were used in more than one citation, that is 8% of our corpus, while 73k URLs were used in only one citation (80% of the URLs).\n",
    "\n",
    "Below, we show the 10 most cited URLs of our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAEYCAYAAADPrtzUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAB14UlEQVR4nO3deZgdVbn2/+9NwjxFBn0hAQOIRhAI0AQwBMM8KaCCoMwqviiH8YfHePAgoB4jHEU5MiMigoIgKEcggBggBAh0ZmYRgkwvCJJImJTw/P5Yz05X7+zdQ9LQnfT9ua6+UrVq1apVq2p3+qm1Vm1FBGZmZmZmZmbWu5bq7QqYmZmZmZmZmQN0MzMzMzMzsz7BAbqZmZmZmZlZH+AA3czMzMzMzKwPcIBuZmZmZmZm1gc4QDczMzMzMzPrAxygm5mZmVm/IGmUpEd7sLxTJV3eU+WZmTlANzMzM7MOSZpb+XlH0huV9YN66Bifk3S3pNcl3d5g+3BJk3P7ZEnDu3uMiJgQER+plDlL0s6LVvPGJB0u6a4G6fOPKelSSf/Mdvy7pFslDeusDDNbcjlANzMzM7MORcRKtR/gr8CnKmlX9NBh/g78GBhbv0HSMsDvgcuB9wG/AH6f6Yu7M7JdBwPPAj/r5fqYWS9ygG5mZmZmC0XSspJ+LOm5/PmxpGVz22hJz0j6D0kvZc9x0972iPhjRPwGeK7B5tHAQODHEfFWRJwNCNixSb1Wk/TzrNMrkn5XrVMu/xJYF/jf7MH+90zfJnvyZ0uaLml0pdz1JN0h6VVJtwJrdLPJmoqIN4DfAMO7kj9715/IujzZUyMZzKx3OUA3MzMzs4V1MrANJajcDBgBfKuy/f9QgtjBwGHAhZI+QvdtDMyIiKikzcj0Rn4JrJDb3w+cVZ8hIg6h/WiAMyQNBm4AvgusBpwE/FbSmrnbr4DJeU7fyXPqEZJWBD4PPN7FvGcDe0TEysDHgWk9VRcz6z0O0M3MzMxsYR0EnB4RL0bE34DTgEPq8vxn9nrfQQl+P7cQx1kJmFOXNgdYuT6jpLWAPYCjIuKViPhXHrsrDgZujIgbI+KdiLgVaAX2lLQusFXlfO4E/nchzqXeSZJmA68C27Fg+zXzDvAxSctHxPMR8WAP1MXMepkDdDMzMzNbWGsDT1XWn8q0mlci4rUOtnfVXGCVurRVKEFtvXWAv0fEKwtxnA8C++fw9tkZOG8HrEWpd6PzaeZtYOkG6UsD/6qs/3dEDAKGAm8AnY4wyDocABwFPC/phurL5cxs8eUA3czMzMwW1nOUoLZmXdrPIX9fDsdutr2rHgQ2laRK2qaZXu9pYDVJg7pQbtStPw38MiIGVX5WjIixwPM0Pp9m/gqsW62zpBUoQ+4XCOwj4q/AccBPJC3facUjbo6IXSgPDx4BLupsHzPr+xygm5mZmdnC+jXwLUlrSloDOIXypvWq0yQtI2kU8Eng6kYFSRogaTnKy+CWkrScpFoP9O3APODYfDHdv2X6n+rLiYjngZuAcyW9T9LSkrZvUv8XgPUr65cDn5K0W60++WK5IRHxFGW4e+18tgM+1UHbTALeBMZkOStS3lDfSpOe9xxS/xzwlfZNo+Xqfj4gaZ8s8y3KCIN3OqiLmS0mHKCbmZmZ2cL6LiXgnAHMBKZkWs3/A16hBJ1XUOaFP9KkrEMoQ7zPA0bl8kUAEfFPYF/gUGA28EVg30xvVta/KD3LLwLHN8n3fcoDhtmSToqIp4F9gP8A/kbpUf86bX8zfwHYmvKVcN8GLmtSLhHxFrAX5Q30zwBPUIbJf67uZXf1zgT+vfY2fMoL4N6o+1kGOJHSrn8HPgF8tYMyzWwxoY5/P5iZmZmZdV9+PdnlETGkl6tiZrbYcA+6mZmZmZmZWR/gAN3MzMzMzMysD/AQdzMzMzMzM7M+wD3oZmZmZmZmZn3AwN6ugJmZ9V9rrLFGDB06tLerYWZmZvaemjx58ksRsWZ9ugN0MzPrNUOHDqW1tbW3q2FmZmb2npL0VKN0D3E3MzMzMzMz6wMcoJuZmZmZmZn1AQ7QzczMzMzMzPoAB+hmZmZmZmZmfYBfEmdmZr1m5rNzGDrmht6uhpmZmfVzs8bu1dtVANyDbmZmZmZmZtYnOEC3fknSUEkPNEg/XNLalfXjJa3Qg8e9SdKQXD5Q0sk9VXYXj3+6pJ3fy2PmcYdJmiZpqqQNmuQZI+kgSWtKmpR5RzXJ27Tt8tp+oQt1apdPUouks3sq/7tB0mRJy1bWr6/ex5KGS7o327pV0ohMl6SzJT0uaYakLSr7rCvpFkkPS3pI0tDKPt+T9FhuOzbT3yfpuiznPkkfq6vjgLx2f3iXm8PMzMxsieMA3ay9w4G1K+vHAz0SoEtaHlg9Ip7JpD2AcT1RdldFxCkR8cf38phpX+CaiNg8Iv5SS8wgsPZ7aDfgFmAnYGbmnVAtRNKAXOyo7YYCnQbo9fkiojUiju3B/D1K0nrAsxHxVq5/Bphbl+0M4LSIGA6ckutQ2mvD/PkKcF5ln8uAMyPio8AI4MVMPxxYBxiW267M9P8ApkXEpsChwE/q6nAc8PBCn6iZmZlZP+YA3fqzAZIukvRg9iAeArQAV2QP5HGUYH28pPEAkuZKOiv3uU3Smpl+bPY+zpB0ZZPjjQZuz/wChgNTJM2UNCiD1ZclHZp5LpO0i6QbJG2aaVMlnZLLp0s6UtI5kvbOtOskXZLLX5T0vWoFJF0qab9cniXpNEm1OgzL9E/k+dd6vFfO9K9Luj/P8bRKmQdnT+o0SRdUguja9j0pDzq+Kml89kQ/Kuky4AFgHUmrAMsAgylB5T5Z3vLZ5j+UNB3Ytq7tGtV1LDAq007I403I85wi6eNZtfp8o2u9vl0st5p/9byHHpR0saSnJK2hupEakk6SdGoubyBpnEqv+IRK++8v6QFJ0yXdWWnK3cmHEpJWAk4Evlt3jwWwSi6vCjyXy/sAl0VxLzBI0lqSNgIGRsStABExNyJez32+CpweEe/ktlrgvhHwp0x7BBgq6QNZryHAXsDFmJmZmVm3OUC3/mxD4JyI2BiYTQluWoGDImJ4RPyEEuDsEBE75D4rAq25zx3AtzN9DLB59ioeBfOHQFcDlWqv7+bA9IgIYCIwEtgYeAKoDeveFrgbmEAJDFcF3s68ZL47a9szbTAlgKpu78hLEbEFpUf1pEw7CTg6e2FHAW9I2jXbawQlON5S0vaSPgocAIzM/POAg6oHiIgbgfOBsyrtuCFwbkRsHBFPATsDt0XENErP71V5Dd6gtPmkiNgsIu6qa7sF6kq5FhNy/7MoPcK75HkeANSGpdfnq+pKuVXfBu7K++I6YN2Omx2AC4FjImLLPN65mX4KsFtEbAbsXck/P0AHvgP8EHid9o4HzpT0NPDfwDczfTDwdCXfM5n2YWC2pGvzQcSZlQcsGwAHqAyVv0nShpk+HfgMgMoQ+g8CQ3Lbj4F/B97p6MQlfSXLbZ33+pyOspqZmZn1Kw7QrT97MgNCgMmUIcydeQe4KpcvB7bL5RmUnveDKUF0bQj0lyv7jgTuyuXdgZtyeQKwff6cB2wiaTDwSkS8Vtk+ErgBWEllXvx6EfEobQH8RsBDwAuS1qItwO/ItQ3OfyLwI5U5x4Mi4m1g1/yZCkwBhlGC7J2ALYH7JU3L9fU7OSbAU9mTW1Ntj3rzgN82yduorvWWBi6SNBO4mrYHGB3pSrlV21PuByLiBuCVjjJnD/jHgauz3S4A1qoc+1JJRwIDMv8ywJCIeELScGCDiLiuQdFfBU6IiHWAE4CfdVLvgZQHECcBW1Gu3eG5bVngzYhoAS4CLsn0sZQe+GnAMZR7Yp6kTwIvRsTkTo5JRFwYES0R0TJghVU7y25mZmbWbzhAt/7srcryPBbuawcj/90LOAfYghKstitL0vrA0xHxz0zalTLfGkov96j8uR34G7AfJfAGuJ8y9L7WIz4VOJISVBMRzwKDKIFrrUf9c8DciHi1k/rX2mD++UfEWODLwPLAxBx6LeD72Xs8PCI+FBE/y/RfVNI/EhGndnJMgNfq1kcA9zXJ+2ZEzKusz2+7JnWtdwLwArAZpR2X6axyXSy3K96m/e/Z5fLfpYDZlXYbnvO8iYijgG9R5n9PlrQ65drXHu5sC7RImpVpH5Z0e247jLaHLldT2hXg2SyvZkimPUOZT/5EPoT4HeUeJrfVyroO2DTr94+IOCJHFxwKrEkZ+TES2DvrdSWwo6TLu9xSZmZmZuYA3azOq8DKHawvRQmeobww7C6Vl5ytExHjgW9Q5v6uVFfu/OHtOVR9YES8DBARTwNrABtGxBOUoOskcnh6BvVPA/sD91AC8Pnb072U4c13Vra3e8FaV0naICJmRsQPKA8HhgE3A1/Mnl8kDZb0fuA2YL9cRtJqkj7YzeNtDDxSF4Q3y9uu7ZrUtf6arQo8n3OpDyF7pRvk66wNmuantPsXct89gPdl+gvA+3OO+rLAJ6EEucCTkvbPfSRps8qxJ0XEKZSHNetQGTUQEedFxNoRMZQyguOxiBidx3sO+EQu7wj8OZevBw7N42wDzImI5/PcBinfpZD7PJTLvwNqUxI+ATyW9RuUPfpQHmLcmUH7NyNiSNbrQOBPEXFwk/YyMzMzswYcoJu1dylwfr4IbHnKPOFxypfEUXp+R+SLv3YETqcEfJfnEOqpwNkRMbtuDnp1/vAuQP2b1CeRARAlsB5MW49pLe3FnJM9gdIDOqFu+8CIeJwyBH212vZ8aVlLN9rg+HxJ2QzgX8BNEXEL8CvgnjzPa4CVI+IhSm/vLZn/VnKodjeO25232de33QJ1pUw3mJcvWTuBMrf7MJWXzA2jrfe+Pl+HbdBJ/tOA7SU9SJmf/VeAiPgX5R65j9I2j1T2OQj4UtbrQcqL3KDMIZ+Z99jdlDnfoynvPOjMkUDthXr/RXljO8CNlF7uxynD1b+W9ZtHeZhzW15X5XYoQ9k/m+nfpwTjAB8FHpD0KOXaHdeFepmZmZlZF6i8Z8nMukLS3Iio7x3vbJ9lgYk5l5cM2i+um4Pdb0m6FTg0e3Q7y7tYtF0O826JiJd6oKwhwEURscciV6wPWnatDWOtw37c29UwMzOzfm7W2L3e0+NJmlyLD9qlO0A367qFCdCt/+nJAH1J19LSEq2trb1dDTMzM7P3VLMAfWFeimXWbzk4t67IedhmZmZmZt3iOehmZmZmZmZmfYADdDMzMzMzM7M+wAG6mZmZmZmZWR/gAN3MzMzMzMysD3CAbmZmZmZmZtYHOEA3MzMzMzMz6wMcoJuZmZmZmZn1AQ7QzczMzMzMzPoAB+hmZmZmZmZmfcDA3q6AmZn1XzOfncPQMTf0djXMzMysG2aN3au3q7DEcg+6mZmZmZmZWR/wngfokoZKeqBB+uGS1q6sHy9phR487k2ShuTygZJO7qmyF6FOoyV9vLv5JB0l6dCeyv9ekXS7pJa+XuZ7LT8TX+hG3gU+P4tw7Ibtl5/Hn/bUcXqapGMlPSzpig7yTJa07CIep0XS2YtSRqWs0ZL+0ENl9cnrI+mbkh6X9Kik3Xq7PmZmZmaLm740xP1w4AHguVw/HrgceH1RC5a0PLB6RDyTSXsAPfJH9yIaDcwF7u5Ovog4v4fzW+8aCnwB+NW7UbikAREx790ouxd9Ddi58pkGQNLAiHhb0nrAsxHx1sIeIMtqBVoXsa5LpFpbV9Y3Ag4ENgbWBv4o6cNL4L1nZmZm9q7prSHuAyRdJOlBSbdIOgRoAa6QNE3ScZQ/8MZLGg8gaa6ks3Kf2yStmenHSnpI0gxJVzY53mjg9swvYDgwRdJMSYNUvFzrZZZ0maRdJN0gadNMmyrplFw+XdKRks6RtHemXSfpklz+oqTvVStQX09JQ4GjgBPynEdJ+pSkSXmsP0r6QJN8p0o6qRvlVvNvKWl6/pxZ642t75GT9AdJo3N5V0n3SJoi6WpJK2X62Mqx/7u+0SUtn3V6WNJ1wPKVbQuUKWl3SVdX8szvcWxWh7rjfT6v6QOSflBJb3bv3C7pJ9lOD0gakekrSrpE0n15LfaptNHvJN0qaZakf5N0Yua5V9JqmW8DSeNUenAnSBqW6ZdKOlvS3ZKekLRfVnEsMCrrcUKD85p/zYCjK+kdXbO5kn6Y+2wr6RRJ9+d5XihJlUMcUt8GdcdfU9Jvc//7JY1skGcpSedKeiTb58ba+UnaKdtoZrbrspk+S9IaudyS12OpTB9UKfvPkj5QWT8fWB+4SdIJKvf3LyVNBH6Z2XYHxmX+3fO+mS7ptkxbLa/ljLx2tc95u7Lq7sFTs/635/U7tlKng/N+mSbpAkkD6tuorr1G5P08Ne+Hj1Su6bV5//xZ0hmVfY6Q9Jik+4AFrkElX5fOt6O2bnbNm7R1zT7AlRHxVkQ8CTwOLHA/mZmZmVlzvRWgbwicExEbA7OBoPRSHRQRwyPiJ5Se9B0iYofcZ0WgNfe5A/h2po8BNo+ITSmBae2P/Ysrx9uD/GMd2ByYHhEBTKT8obsx8AQwKvNsS+l9nkAJnFYF3qbtj+JRwJ217Zk2GNiobntVu3pGxCzgfOCsPOcJwF3ANhGxOXAl8O9N8nW33KqfA8dExGZ0QQZQ36L0Vm5BuU4nSlod+DSwcR77uw12/yrwekR8lHK9tuyoTOCPwNaSVsz9DwCu7CB/tZ5rAz8AdqQ8gNlK0r65udm9A7BCRAyn9MhekmknA3+KiBHADsCZlTp9DPgMsBXwvTy/zYF7gNo0ggspbbwlcBJwbuV4awHbAZ+kBOZQruGEvF5nNWjHbl2zyjlPiojNIuIu4KcRsVVEfIzyoOSTnbRB1U8o99NWwGeBixvk+QxlJMBGwCGUzxCSlgMuBQ6IiE0oo3a+2qzSEfEO8HvKvYWkrYGnIuKFSp6jaPv9UGuvjSj3x+dzfXdgnMrDmIuAz2b77Z/bTwOm5r37H8BllWrUl1U1DNiNEnh+W9LSkj5KuVdHZjvOAw5qdo7pEWBU3junAP9V2TY8y9sEOEDSOpLWyjqPpNw/G9FAd863k7bu6Jo3a5/BwNOV9WcyrVE9vyKpVVLrvNfnNMpiZmZm1i/1VoD+ZERMy+XJlD/sO/MOcFUuX075IxVgBqXn/WBKEE1EtEbElyv7jqQEv1D+cL8plycA2+fPecAmkgYDr0TEa5XtI4EbgJVU5sWvFxGP0hbAbwQ8BLyQf0jXAvyqBerZwBDgZkkzga9THhx0pivlApA9ZYMiovbwoL4HrJFtKH+QT5Q0DTgM+CAwB3gT+Jmkz9B4KsL2lGtFRMzIujYtM4fLjgM+JWkgsBclgGhWh6qtgNsj4m9ZzhV5fGh+7wD8Out3J7BKttGuwJg81u3AcsC6mX98RLwaEX/LNvjfTJ8JDFXp2f84cHXufwElKK/5XUS8ExEPAR+gEwt5zaAEib+trO+gMjpjJuUhRvXeatQGVTsDP83zuT7z1I9g2A64Os/t/wHjM/0jlM/7Y7n+C9quSzNXUQJUKEOmr+ogb831EfEGgKRlgCER8QTl3rkze3SJiL9X6vvLTPsTsLqkVerLauCG7CF+CXiRcg13ojx8uj/baCdKD39HVqXcIw8AZ9H+etwWEXMi4k3K75UPAlvTdn//k+Zt0t3zbdbWHV3zjtqnSyLiwohoiYiWASusuihFmZmZmS1RemsOenVe6DwqQ5+7IfLfvSh/8H8KOFnSJnXzItcHns4/aqEEX5/N5TspQ4bXpfSafhrYjxJ4A9xPGXr/BHArsAZwJOWhAhHxbAYzu2dZqwGfA+ZGxKt19V2gng3O6X+AH0XE9SpDlU/tQjt0pdyueJv2D2yWy38F3NqoN1FlOPROlDb7N0rg1xVNy6SMHPg34O+UXu9XJXWUf2FEk+Xauig9kI+2q3TpYazeu+9U1t+hfJ6WAmZnT2oj1f3VKIOkn1NGejxHmZveTLNrBvBmbe5v9mKfC7RExNOSTq3L26gNqpaijOx4s4O6LIxq/av1uQf4UPYG70vj0Rn1Xqssj6LtgdzCeK2DbfW/uwZSruMvIuKb1YySPk3baI3qA0OA71Ae9nxaZVrK7Z0co6EcSj85V6+n/M7qjmZt3fCal49iaZ8G5/cssE4l+5BMMzMzM7Mu6ktfs/YqsHIH60tRAkEoQctdkpYC1omI8cA3KL1S9T1784e351D1gRHxMkBEPE0JujfM3ra7KEOS78zt/6QM2dyf8ofshOr2dC/lhXZ3Vra3G1beQT3rz3FV2v6gPayDtuhuueT5zAZmS6r1IFeH4c4Chue81HVomzt6LzBS0ofymCtK+nD2pq0aETcCJwCNhl/fSQaYkj4GbNpRmbntDmALyoOQK7uQv+Y+4BOS1sig5fNZFjS4dyr7HZBlbgfMiYg5wM3AMflgAEmbNzi3hiLiH8CTkvbPfSWps6Hp7a5XRByRw933XMhrVq8W/L6U122/uu2N2qDqFuCY2oqk4Q2OMRH4bNblA5T3PgA8ShlZ8KFcP4S26zKLnPZA20MzIiKA64AfAQ/XPq/dUB0lcy+wvcpL41C+J4DyGT0o00YDL+W1Wxi3AftJen/tGJI+GBHX5XUcHuVlc1XVz/rhXTjGJMr9vbqkpcmh6xExr3KMU7p7vh20dafXvMH5XQ8cKGnZPP6GlM+lmZmZmXVRXwrQLwXOV3nJ0vKUebzjlC+Jo/TajMghoTsCpwMDgMtz2O5U4OyImK32c9DnvywK2IUyz7lqElAbfjuBMmeyGsBNAF7MIZ0TKL1CE+q2D4yIx4EplF70CQCSLlb5CquG9aQMj/50nvMoSo/51ZImAy9VjlGfr6ar5VYdAZyTQ1erPbgTgScpQ2rPznMhh3IfDvxa0gzKg4phlIDyD5l2FzknXNLekk7PMs+jTAt4mHK9JndSJtnr+wfKg5U/dJa/JiKep8zlHg9MByZHxO9zc6N7p+ZNSVMp8/a/lGnfAZYGZkh6MNe74yDgSyovaHuQ8vKsjswA5qm81GuBl8TRzWtWL++JiyjfknAzC/ayNmqDqmOBFpUXjD1E43c9/JYy5/ghyjSCKZRg/82s/9V5n76Tx4EyL/onklopPcVVVwEH579IWlvSjY3Or4HR5EOAvHe+Alyb16M2hPtUYMu8n8bS/oFYt+R0hW8Bt2R5t9J+WkMjZwDfz3bvdCRT3t+nUu79icDDTfItzPm2a+vU8Jp3UscHgd9Q7oFxwNHhN7ibmZmZdYtKB0rfJ2luRCzw5u5O9lkWmBgRLbl+MXBxRNz7btRxcZNDa/8Q5cVhS6xm946k24GTGvRu2kKQtFJEzFV5geB9lJem/b/3uA5DgIsiYo/38ri28JZda8NY67Af93Y1zMzMrBtmjd2rt6uw2JM0uRanVvWl70HvcVG+A7mlsl4/D9TMes4fVN7JsAzwnfc6OAeI8r3oDs4XI5sMXpVW/ydvZmZmBixGAXp3e8+tc1G+km2J7j2H5vdORIx+j6uyRHN7mpmZmZktmr40B93MzMzMzMys33KAbmZmZmZmZtYHOEA3MzMzMzMz6wMcoJuZmZmZmZn1AQ7QzczMzMzMzPoAB+hmZmZmZmZmfYADdDMzMzMzM7M+wAG6mZmZmZmZWR/gAN3MzMzMzMysDxjY2xUwM7P+a+azcxg65oberoaZmZl1w6yxe/V2FZZY7kE3MzMzMzMz6wP6ZYAuaaikBxqkHy5p7cr68ZJW6MHj3iRpSC4fKOnknip7UUi6XVJLH6jHf3Qx3909fNy5PVDGUElf6KH63N2dMiW1SDq7J47dWySNkvSgpGmSlm+S53xJI+s/pz1Yh0sl7beQ++4taUxP1ynLPlzST7u5z42SBr0b9elGHUZL+kNv1sHMzMxscdMvA/QOHA5U//A/HuiRAD2DjtUj4plM2gMY1xNl9zUqFube6lKAHhEfb3DM3p6uMRToVoBeX+faeuX8ulRmRLRGxLGdld/HHQR8PyKGR8QbtcS6c9gGuJcFP6e9LiKuj4ixPV3uwl7DiNgzImYv5DG7/PldzO4xMzMzsz6vPwfoAyRdlL12t0g6BGgBrshevOMoQcB4SeOh9LRKOiv3uU3Smpl+rKSHJM2QdGWT440Gbs/8AoYDUyTNlDQo/yh+WdKhmecySbtIukHSppk2VdIpuXy6pCMlnSNp70y7TtIlufxFSd+rVkDScEn3Zj2vk/S+yuZD8rwfkDQi838i06blsVfO9K9Luj/LOS3Thkp6VNJlwAPAf0o6s3Ls+b2Akn4naXK241cybSywfB7rikw7MevzgKTjK2XNzX9HS5og6XrgIUkDJJ1Zqdv/bXQhJK0n6Z5s++/WbWt0bivmdZiedTmgQbFjgVFZ/xOa1aVBndutV8+vQZnLSfp51nuqpB0qZf4hl0+V9EtJE4FfSlpT0m+zHvdLGtnRta1riy9JekzSfSqfldr1GyrpT3let0laN9Pb9UBXrtOVkvaqpNfn+zLwOeA7kq5o0iYfBR4DPk37z+lekn5XKWsXSdfVjq/Gn9eOPge1cnbKdpkp6RJJy2b6npIeyfv37Eq7V+/vhm3e4BizJK2Ryy2Sbm90DTP7OiojXf4s6duVMhb4LFXLzmv1sNr/rltghIIW/PyuU7kPkbSfpEsr1+98SZOAMySNUPk8TZV0t6SPNDpfMzMzM+tcfw7QNwTOiYiNgdlAAK3AQdmL9xPgOWCHiNgh91kRaM197gBqfyiPATaPiE2Bo2D+H9wXV45X7THfHJgeEQFMBEYCGwNPAKMyz7bA3cAESpC2KvB25iXz3VnbnmmDgY3qtlddBnwj6zmzUn+AFSJiOPA14JJMOwk4OtNHAW9I2jXbbgTlIcOWkravtOm52T7nUoKpmgOA2sOLL0bElpRA61hJq0fEGOCNbPuDJG0JHAFsTek5PVLS5ixoC+C4iPgw8CVgTkRsBWyV+6zXYJ+fAOdFxCbA87XEDs5td+C5iNgsIj5G45EPY4AJWf+zOqlLtc6N1puVeTQQWe/PA7+QtFyDumwE7BwRn89zPSvr8Vmgdk8ucG2rBagMIf9PStuPBIZVNv8P8Iu8j64AOhtefxUlAEfSMsBOwPy3gkXExcD1wNcj4qAmbbIHMC4irqHyOQVuBIbVgm/KPVO7f5t9Xjv6HJBteilwQLb1QOCrmX4BsEfev2vSWLM2747qNYRyT34W2BTYX21TUhb4LDUoq/533WebHHP+5zcinuqkfkOAj0fEicAjwKiI2Bw4Bfivzk5O0lcktUpqnff6nM6ym5mZmfUb/TlAfzIipuXyZMpw4s68Qwk2AC4HtsvlGZQevYMpQXRt2PGXK/uOBO7K5d2Bm3J5ArB9/pwHbCJpMPBKRLxW2T6SEtSspDIvfr2IeJS2AH4jSm/jC5LWoi3AByAD/EERcUcm/SLLrfl11vtOYBWV+asTgR9JOjb3fRvYNX+mAlMogduGWcZTEXFvlvM34AlJ22TQMCzLgxJITKcMV16nsn/VdsB1EfFaRMwFrqXtQUTVfRHxZC7vChwqaRowCVi9Sdkja+dLWw9lbf9G5zYT2EXSDySNioiuRBQd1aVa50brzWxHue+IiEeAp4D6oB7g+sow8Z2Bn2Y9rqdc25VofG2rRgB3RMTfI+JfwNWVbdsCv8rlX9L2OWjmJmCH7IXeA7izOoy9ifo22Y0GD0byIdcvgYPznt2Wts/WAp/XLnwOAD5C+f3wWF2eYcATlXr9msaatXl3XF/XRrdGxMuZdi1tbd6Vz1JXf9fN//x2wdURMS+XVwWuVnmvx1mUh40diogLI6IlIloGrLBqFw9pZmZmtuTrz/MH36oszwMavpiqE5H/7kX5A/5TwMmSNqkGPJLWB56OiH9m0q609WLdSekZXRc4mdLrvB8l8Aa4n9I79gRwK7AGcCTlD20i4tkMTHbPslaj9FbOjYhXF+Jc5q9HxFhJNwB7AhMl7QaIMlf4gmpmSUOB1+rKuDLr8ggl2A5JoykBzLYR8XoO623UC9xV1WMKOCYibq6r2/co14jsdYUFz7e2/wLnlmVsQWmH70q6DbiZ0psKpdfwHw3KalSX0SzYTvXri6pa3lLANhHxZl2eBa5tBv2L4u08HipzmJcBiIg38zrvRvuRFB2Zfw75QGpQRDzXJO/Pgf8F3qQEjvUPG2oaXfN3Q8M2l3Qz8AFKr/6XqbQXC34G6u+JBT6f3fgsLfC7TtI6lDYDOJ/y8KOjY3ZUv+8A4yPi0/l74PYGdTAzMzOzLujPPeiNvAqs3MH6UpTgGcrLu+7KQGSdiBgPfIPSm1TfWzZ/eHv24A2MiJcBIuJpStC9YUQ8QellP4kcnp5B/dPA/sA9lMB9/vZ0L+WFdndWtk+obCd7fV+RVOuFPoQy7LfmgKzfdpSh2XMkbRARMyPiB5QHBcMogekXaz2CkgZLev8CLVlcB+xDGY5dC8pWpYwOeF3SMMoQ6pp/SVo6lycA+0paQdKKlAcX7c6pgZspQ5GXzrp9WNKKEXFyDhMfnvkmAgfm8kF1+y9wbjnc+/WIuBw4E9giIibVyoyI61nwXmlYl07qX6++zAm1+kr6MOWhzqOdlHELcExtRdLw/LfRta26H/iEpPepvAisOiz6btq3X+26zAK2zOW9gaUr+1xFGX4+iu6/HHEHYHxlvV27ZOD+HPAtSrBes8DntQufAyhtOlTSh+ryPAqsn0Eo5GemgYZtHhG75f1SG1kzi7b2ajbsvGYXSavl/PF9KfdwR5+lDkXE05X79/wm2V6Q9NH8HffpJnnIejyby4d3tQ5mZmZmtiAH6O1dCpyvtq96uhAYp3xJHKXXaEQO5dwROB0YAFwuaSZlaPTZETG7bg767rQFJbsAf6w77iTKC7CgBDuDaRsOX0t7MYe3TqDM/5xQt31gRDxOGZq9Wm27pIsr81UPA86UNIMyx/r0ShlvSppK6U37UqYdr/JStBnAv4CbIuIWyvDme/Kcr6F9EDlfRLwCPAx8MCLuy+RxwEBJD1NeglYdUnshMEPSFRExhXI97sv2uTgipjY6TsXFlGH+U/IaXUDjUSLHAUdn/QdX6tvs3DYB7sshy98GvrtAiWWawzyVF8md0I26dKS+zHOBpbJuVwGHR8RbHZYAxwItKi9Ee4h8RwINri1AniMR8SxlLvF9lGBwFlAb2n8McETuewilPQEuogT10ylDzau9rLcAnwD+WBtJovJVYF15G3v9Nx5cSvvPKZS58E9HxMOVfI0+r9Dx54Ds+T6CMmx7JmWo/Pn5+fsa5XfCZMqDgkbTHZq1eb3TgJ9IaqX0bHfkPuC3lHvitxHRSsefpZ4wBvgD5YHM8x3kOwP4fv7+6M+jsszMzMwWmcoUTusKSXMjoltzSXPe7cSIaMn1iynBZk//MW3WoyStFBFzswf9OuCSiLiuF+oxBdg658I3y/NTYGpE/KyS1u3PaxfqUmsTAecAf47yAj9bSMuutWGsddiPe7saZmZm1g2zxu7VeSbrkKTJtRixXboD9K57N/7gN+urJP03ZY7zcpQe8OOiD/7CyN7s14BdqiMK3qUA/QRKD/wylBEzR0bE6z15jP6mpaUlWltbe7saZmZmZu8pB+hmZtbnOEA3MzOz/qhZgO456GZmZmZmZmZ9gAN0MzMzMzMzsz7AAbqZmZmZmZlZH+AA3czMzMzMzKwPcIBuZmZmZmZm1gc4QDczMzMzMzPrAxygm5mZmZmZmfUBDtDNzMzMzMzM+gAH6GZmZmZmZmZ9wMDeroCZmfVfM5+dw9AxN/R2NczMbAk3a+xevV0Fsy5xD7qZmZmZmZlZH+AA3fotSUMlPdAg/XBJa1fWj5e0Qg8e9yZJQ3L5QEkn91TZXTz+6ZJ2fi+PmccdJmmapKmSNmiSZ4ykgyStKWlS5h3VJO+Bkk6WtLekMV04/pmSHpR05qKeS1fV3zuSbpQ0aBHK20bSRZX1dSXNlXRSJe2EPM8HJP1a0nKZvl626eOSrpK0TKafKOkhSTMk3Sbpg5Wy5uU1mybp+kr6TpKmZPpdkj6U6R/MMmZIur12n5uZmZlZ1zhAN1vQ4cDalfXjgR4J0CUtD6weEc9k0h7AuJ4ou6si4pSI+ON7ecy0L3BNRGweEX+pJaqo/S7aDbgF2AmYmXknVAuRNCAX9wDGRcT1ETG2C8f/CrBpRHy9K5WV1BNTgI6ncu9ExJ4RMXsRyqu/X34E3FRbkTQYOBZoiYiPAQOAA3PzD4CzIuJDwCvAlzJ9aubfFLgGOKNS/hsRMTx/9q6knwccFBHDgV8B38r0/wYuy7JOB76/COdqZmZm1u84QLf+boCki7LH8RZJhwAtwBXZO3gcJVgfL2k8QPZYnpX73CZpzUw/ttITeWWT440Gbs/8AoYDUyTNlDQog9WXJR2aeS6TtIukGyRtmmlTJZ2Sy6dLOlLSOZL2zrTrJF2Sy1+U9L1qBSRdKmm/XJ4l6bTsDZ0paVimf6LSczpV0sqZ/nVJ9+c5nlYp82BJ92X+CypBdG37npRg9auSxquMXnhU0mXAA8A6klYBlgEGU4LEfbK85bPNfyhpOrBtXdsdLumnlXM7W9Ldkp6onOf1wErAZEkH5PH/VOk1Xrey//mSJgFn5Pp5ku7N8kZLukTSw5IurZzfeZJa8544rXY/NLh3ZklaI5dPVOnlfkDS8Zk2NMuu3pPLV5pyJ+CPmXdf4Engwbp7bCCwfD5gWAF4LttrR0oADvALygMTImJ8RLye6fcCXen1DmCVXF4VeC6XNwL+lMvjgX26UJaZmZmZJQfo1t9tCJwTERsDsymBRyvZOxgRP6EEHztExA65z4pAa+5zB/DtTB8DbJ69h0cBSGqRdHHleNUe0M2B6RERwERgJLAx8ARQG9a9LXA3MAEYJWlV4O3MS+a7s7Y90wZTAqXq9o68FBFbUHpFa0OlTwKOzh7SUcAbknbN9hpBCY63lLS9pI8CBwAjM/884KDqASLiRuB8Sg9urR03BM6NiI0j4ilgZ+C2iJgGnAJcldfgDUqbT4qIzSLirrq2q7cWsB3wSWBsHn9v2nqDrwL+B/hFXqsrgLMr+w8BPh4RJ+b6+yjX4QTgeuAsynXaRNLwzHNyRLQAmwKfkLRpRJzNgvcOAJK2BI4Atga2AY6UtHmlXar35GdznzWAf0XEHEkrAd8ATquWGxHPUnqx/wo8D8yJiFuA1YHZEfF2Zn2Gcp/U+xKVHnlguXzwcG8+EKj5MnCjpGeAQ2rtDEwHPpPLnwZWlrR6/UEkfSXLbZ33+pwG1TAzMzPrnxygW3/3ZAaEAJOBoV3Y5x3gqly+nBIMAsyg9LwfTAmiiYjWiPhyZd+RwF25vDttwdAEYPv8OY8S/A0GXomI1yrbRwI3ACupzG1eLyIepS2A3wh4CHhB0lq0BfgdubbB+U8EfpS9wIMysNs1f6YCU4BhlGByJ2BL4H5J03J9/U6OCfBURNxbWa+2R715wG+7mPd3EfFORDwEfKBJnm0pQ7MBfknbNQS4OiLmVdb/Nx8EzAReiIiZEfEOped6aOb5nKQplLbZmLYHJM1sB1wXEa9FxFzKNag9YGl2T+5KGf4PcCrlYcfcaqGS3kfptV6P0nu/Yt6Pncp8LUB1jv4H88HDF4Afq+3dAScAe0bEEODnlKH2UB7sfELSVOATwLOUa9dORFwYES0R0TJghVW7Uj0zMzOzfsFfs2b93VuV5XnA8s0ydqDWi7sXJYj+FHCypE0qPZZIWh94OiL+mUm7kr2jlF7uo4F1gZMpvY/7UQJvgPspwdMTwK3AGsCRlACOiHhW5eVju2dZqwGfA+ZGxKud1L/WBvPI3wkRMVbSDcCewERJuwECvh8RF1R3lnQMpTf6m50cp95rdesjgK82yftmXdBcbbt61WuqbtapUb1q5b1TV/Y7wEBJ61EC060i4pUc+r7cQhy3/njQ/p7cg7ZAeGtgP0lnAIOAdyS9CbxACfD/BiDpWuDjlFECgyQNzHtyCCV4JvPtTLnvPhER84+fPfJExBOSbgc2l/QPYLOImJTZriJHhUTEc2QPevbyf3YR59ybmZmZ9SvuQTdb0KvAyh2sL0UJnqH0LN6l8pKzdSJiPGXo8aqUOc9V84e351D1gRHxMkBEPE0JujeMiCcovewnkcPTM6h/GtgfuIcSuM/fnu6lzPO+s7K93QvWukrSBtlT/APKw4FhwM3AFzPwQtJgSe8HbqMEi+/P9NVUeRN4F4+3MfBIXRDeLG+7tltId9P28rSDWMh2SqtQgvo5kj5Auc419fdOzQRgX0krSFqR8kCmaR1yDvmmwDSAiBgVEUMjYijwY+C/IuKnlKHt22S5ooxmeDhHAIyn7b49DPh9lr05cAGwd0S8WDnm+yQtm8trUEZvPER5wdyqkj6cWXcBHq7lU9sL/74JXNLsnMzMzMxsQQ7QzRZ0KXC+8gVlwIXAuNqLvijB2AiVr2jbkfK26gHA5ZJmUoY5nx0Rs+vmoO9O2/zzXciXfVVMAh7L5QmUOcJ3VbZPAF7MOdkTKL2gE+q2D4yIxylD0FerbZd0saSWbrTB8fnyshnAv4Cbci7zr4B78jyvAVbOoeTfAm7J/LdS5oF357jdeZt9o7brrmOAI7K+hwDHLWxBETGdcs0fobTPxMrm+nunts8Uyn12H+W6XxwRUzs4zJbA1CZz7qvlTqJclymUIflLZR2gPDg6UdLjlDnpP8v0MykPk65W+69T+yjQqvJivvHA2Ih4KHvgjwR+m9sOAWpvxh8NPCrpMcr0gnYvKDQzMzOzjqmTv/fMrI6kuRFR3zve2T7LAhNzPi8ZtF9cNwe735J0K3BoRDzfhbz9ru0kfQt4PCKafTvAYmvZtTaMtQ77cW9Xw8zMlnCzxu7V21Uwa0fS5Fps0C7dAbpZ9yxMgG5mjbW0tERra2tvV8PMzMzsPdUsQPcQd7NucnBuZmZmZmbvBgfoZmZmZmZmZn2AA3QzMzMzMzOzPsABupmZmZmZmVkf4ADdzMzMzMzMrA9wgG5mZmZmZmbWBzhANzMzMzMzM+sDHKCbmZmZmZmZ9QEO0M3MzMzMzMz6AAfoZmZmZmZmZn3AwN6ugJmZ9V8zn53D0DE39HY1zMxsCTdr7F69XQWzLnEPupmZmZmZmVkf4ADd+iVJQyU90CD9cElrV9aPl7RCDx73JklDcvlASSf3VNldPP7pknZ+L4+Zxx0maZqkqZI2aJJnjKSDJK0paVLmHdUkb9O2y2v7hS7UqV0+SS2Szu6p/O8GSZMlLVtZv756H0saLunebOtWSSMyXZLOlvS4pBmStqjss66kWyQ9LOkhSUMr+3xP0mO57dhMX1XS/0qaLulBSUfU1XEVSc9I+um73BxmZmZmSxwH6GbtHQ6sXVk/HuiRAF3S8sDqEfFMJu0BjOuJsrsqIk6JiD++l8dM+wLXRMTmEfGXWmIGgbXfQ7sBtwA7ATMz74RqIZIG5GJHbTcU6DRAr88XEa0RcWwP5u9RktYDno2It3L9M8DcumxnAKdFxHDglFyH0l4b5s9XgPMq+1wGnBkRHwVGAC9m+uHAOsCw3HZlph8NPBQRmwGjgR9KWqZS3neAOxflXM3MzMz6Kwfo1p8NkHRR9gLeIukQoAW4Insgj6ME6+MljQeQNFfSWbnPbZLWzPRjs/dxhqQrmxxvNHB75hcwHJgiaaakQRmsvizp0MxzmaRdJN0gadNMmyrplFw+XdKRks6RtHemXSfpklz+oqTvVSsg6VJJ++XyLEmnSarVYVimfyLPv9bjvXKmf13S/XmOp1XKPFjSfZn/gkoQXdu+J+VBx1cljc+e6EclXQY8AKwjaRVgGWAwJajcJ8tbPtv8h5KmA9vWtV2juo4FRmXaCXm8CXmeUyR9PKtWn2+0pD900AYd5V8976EHJV0s6SlJa6hupIakkySdmssbSBqn0is+odL++0t6QKWHuhro7k4+lJC0EnAi8N26eyyAVXJ5VeC5XN4HuCyKe4FBktaStBEwMCJuBYiIuRHxeu7zVeD0iHgnt71YOcbKeR1WAv4OvJ312hL4AOVBi5mZmZl1kwN06882BM6JiI2B2ZTAoxU4KCKGR8RPKAHODhGxQ+6zItCa+9wBfDvTxwCbR8SmwFEwfwj0xZXjVXt9NwemR0QAE4GRwMbAE0BtWPe2wN3ABEpguColEBqZ20dReionVPYZDGxUt70jL0XEFpQe1ZMy7STg6OyFHQW8IWnXbK8RlOB4S0nbS/oocAAwMvPPAw6qHiAibgTOB86qtOOGwLkRsXFEPAXsDNwWEdMoPb9X5TV4g9LmkyJis4i4q67tFqgr5VpMyP3PovQI75LneQBQG5Zen6+qK+VWfRu4K++L64B1O252AC4EjomILfN452b6KcBu2UO9dyX//ACd0kv9Q+B12jseOFPS08B/A9/M9MHA05V8z2Tah4HZkq7NBxFnVh6wbAAcoDJU/iZJG2b6T4GPUj4bM4HjIuIdlZEQP6TtPmpK0ley3NZ5r8/pLLuZmZlZv+EA3fqzJzMgBJhMGcLcmXeAq3L5cmC7XJ5B6Xk/mOxNzCHQX67sOxK4K5d3B27K5QnA9vlzHrCJpMHAKxHxWmX7SOAGYCWVefHrRcSjtAXwGwEPAS9IWou2AL8j1zY4/4nAj1TmHA+KiLeBXfNnKjAFGEYJsncCtgTulzQt19fv5JgAT2VPbk21PerNA37bJG+jutZbGrhI0kzgatoeYHSkK+VWbU+5H4iIG4BXOsqcPeAfB67OdrsAWKty7EslHQkMyPzLAEMi4glJw4ENIuK6BkV/FTghItYBTgB+1km9B1IeQJwEbEW5dofntmWBNyOiBbgIuCTTdwOmUUaXDAd+miMgvgbcWJnC0VREXBgRLRHRMmCFVTvLbmZmZtZvOEC3/uytyvI8Fu5rByP/3Qs4B9iCEqy2K0vS+sDTEfHPTNqVtmHAd1KCpFGUIfB/A/ajBN4A91OG3td6xKcCR1KCaiLiWWAQJXCt9ah/DpgbEa92Uv9aG8w//4gYC3wZWB6YmEOvBXw/e4+HR8SHIuJnmf6LSvpHIuLUTo4J8Frd+gjgviZ534yIeZX1+W3XpK71TgBeADajtOMyDfK008Vyu+Jt2v+eXS7/XQqYXWm34TnPm4g4CvgWZf73ZEmrU6597eHOtkCLpFmZ9mFJt+e2w2h76HI1pV0Bns3yaoZk2jPAtIh4Ih9C/I5yD5PbamVdB2yay0cA1+Zw+ceBJykPbLYF/i3r9d/AoZLGdrGdzMzMzAwH6Gb1XgVW7mB9KUrwDOWFYXfl0N51ImI88A3K3N+V6sqdP7w9h6oPjIiXASLiaWANYMOIeIISdJ1EDk/PoP5pYH/gHkoAPn97upcyvPnOyvZ2L1jrKkkbRMTMiPgB5eHAMOBm4IvZ84ukwZLeD9wG7JfLSFpN0ge7ebyNgUfqgvBmedu1XZO61l+zVYHncy71IWSvdIN8nbVB0/yUdv9C7rsH8L5MfwF4f85RXxb4JEBE/AN4UtL+uY8kbVY59qSIOIXysGYdKqMGIuK8iFg7IoZSRnA8FhGj83jPAZ/I5R2BP+fy9ZSAWZK2AeZExPN5boOU71LIfR7K5d8BtSkJnwAey+W/UkZKIOkDwEeAJyLioIhYN+t1EmXO+5gm7WVmZmZmDSx0gC7pqs5zmS12LgXOzxeBLU+ZJzxO+ZI4Ss/viHzx147A6ZSA7/IcQj0VODsiZtfNQa/OH94FqH+T+iTaAqAJlPnBd1W2TwBezDnZEyg9oBPqtg/MHs0pwGq17fnSspZutMHx+ZKyGcC/gJsi4hbgV8A9eZ7XACtHxEOU3t5bMv+t5FDtbhy3O2+zr2+7BepKmW4wL1+ydgJlbvdhKi+ZG0Zb7319vg7boJP8pwHbS3oQ+AwliCUi/kW5R+6jtM0jlX0OAr6U9XqQ8iI3KHPIZ+Y9djcwnfKCwTu60D5HUt6qPh34L8ob2wFupLzf4HHKcPWvZf3mUYLp2/K6KrdDeSneZzP9+5QRBVDmv388028DvhERL3WhbmZmZmbWCZX3LC3EjtJfI6IrL0IyW2JImhsR9b3jne2zLDAx5/KSQfvFdXOw+y1JtwKHZo9uZ3kXi7bLYd4tPRG4ShoCXBQReyxyxfqgZdfaMNY67Me9XQ0zM1vCzRq7V29XwawdSZNr8UG7dAfoZl23MAG69T89GaAv6VpaWqK1tbW3q2FmZmb2nmoWoHf4UixJWzTbRHkzslm/4uDcuiLnYZuZmZmZdUtnb63+YQfbHulgm5mZmZmZmZl1Q4cBekTs0NF2MzMzMzMzM+sZC/UWd0m75IudzMzMzMzMzKwHdBigS9pR0mOS5kq6XNImklopX79z3ntTRTMzMzMzM7MlX2c96D+kfI/u6pTvPb4HuDQitoyIa9/typmZmZmZmZn1F529JC4i4vZc/p2kZyPip+9ynczMzMzMzMz6nc4C9EGSPlNZX7q67l50MzMzMzMzs57RWYB+B/DJuvVP5XIADtDNzMzMzMzMekBnAfoDdevvAC8Bd0XEk+9OlczMzMzMzMz6n84C9JUapA0FTpZ0akRc2fNVMjOz/mLms3MYOuaG3q6GmZn1IbPG7tXbVTDrNR0G6BFxWqN0SasBfwQcoJuZmZmZmZn1gM6+Zq2hiPg7oB6uiy1GJA2VVD8FAkmHS1q7sn68pBV68Lg3SRqSywdKOrmnyu7i8U+XtPN7ecw87jBJ0yRNlbRBkzxjJB0kaU1JkzLvqCZ5e6Tt6q93F/dpeO/0JZLOlzSyLq1L9ZZ0pqQH89+jJB3aSf4WSWfn8mhJH69s63T/LtRnsqRlm2wbJukeSW9JOqnB9gF5H/2hkjYh78Vpkp6T9LtMP0jSDEkzJd0tabNFqbeZmZlZf9TZEPeGJO0AvNLDdbElw+GUdxc8l+vHA5cDry9qwZKWB1aPiGcyaQ/g7EUttzsi4pT38ngV+wLXRMR3q4mSBCgi3gF2Az4H7ATMjIgv1xciaUBEzKPn2u5w2l/vJcU2wNELue9XgNWynTsVEa1Aa66OBuYCd+e28xeyDgBIWg94NiLeapLl78CxlPurkeOAh4FVKvWd/9BH0m+B3+fqk8AnIuIVSXsAFwJbL0r9zczMzPqbDnvQsydkRt3PM8APgK+9N1W0PmyApIuyt/AWSYcALcAV2bt2HLA2MF7SeABJcyWdlfvcJmnNTD9W0kN5jzWbOjEauD3zCxgOTMn7dJCKl2s9jpIuk7SLpBskbZppUyWdksunSzpS0jmS9s606yRdkstflPS9agUkXSppv1yeJek0SbU6DMv0T1R6GKdKWjnTvy7p/jzH0yplHizpvsx/gaQBdcfck/Kg46uSxmdP7qOSLqMEx+tIWgVYBhgMnAHsk+Utn23+Q0nTgW3r2u5USb/IXtGnJH1G0hl5PuMkLZ112FLSHdkbe7OktbIdqtd7eUmn5Dk+IOnCPFZt/+lZh6Mr57acpJ/n8abmw79az/y1WYc/Szqjss9cSd/L8u6V9IFMX1PSb/P49yt7wCWtKOmSbOOpkvbJ9I0r7T5D0oaZ/lHgsYiY10G9B6j0kNeu5//N9Osp7+6YLOmAbN+Tctvtkn6Qx3xMObpBpdf8D5KGAkcBJ2SdRtXtPzzPd4bKffq+jspNuwPjMt/uKvfqdEm3AUTEixFxP/Av6qiMVNkLuLh+W25fBdgR+F2WdXdE1B7c3gsMabSfmZmZmTXX2RD3T1K+Vq3280ngIxExIiIeebcrZ33ehsA5EbExMJvy1XutwEERMTwifkLpWd0hInbIfVYEWnOfO4BvZ/oYYPOI2JQSpNSG/laDgz3IYAPYHJgeEQFMBEYCGwNPALUAZVtKT+QEYJSkVYG3My+Z787a9kwbDGxUt70jL0XEFsB5QG2I8EnA0RExPMt4Q9Ku2V4jKMHxlpK2z2DwAGBk5p8HHFQ9QETcCJwPnFVpxw2BcyNi44h4CtgZuC0ipgGnAFflNXiD0uaTImKziLirru0ANqAEWntTRjuMj4hNgDeAvTJI/x9gv4jYErgE+F5EXEP76/0G8NOI2CoiPgYsT9vXNP4cOCYi6oc9H11OMTYBPg/8QtJyuW14ts0mwAGS1sn0FYF7s6w7gSMz/SfZRlsBn6UtsDwZ+FNEjAB2AM6UtCLlPvtJtnsLUB2ZUbvPmtX7S8CcPNZWwJGS1ouIvYE3sj2uYkEDsx7H03bvQ2mEWbRd5+ERMaFu38uAb+RnZGbd/s3K3R0Yp/Ig7CLgs3ku+zeoW70fA/9O+faORval3HP/aLDtS8BNzQqW9BVJrZJa570+pwtVMTMzM+sfOgzQI+Kpup+/RsRr71XlrM97MgNCgMmUN/x35h2gFrhcDmyXyzMoPbEHU4JoIqK1bpj2SOCuXN6dtgBgArB9/pwHbCJpMPBK3q+17SOBG4CVVObFrxcRj9IWwG8EPAS8IGkt2gL8jlzb4PwnAj+SdCwwKCLeBnbNn6nAFGAYJcjeCdgSuF/StFxfv5NjAjwVEfdW1qvtUW8e8NsO8t4UEf+iBH0DaAtOZ+Y5fQT4GHBr1vFbNO8d3UFl/vtMStC/saRBlHaoPez4ZSX/dpT7gHzo9xTw4dx2W0TMiYg3Kdflg5n+T6A2J7ra7jsDP806Xg+sImklSruPyfTbgeWAdYF7gP+Q9A3gg/mAAcpUgXGd1HtX4NAscxKwOuV6dqbR/dKpfLg0KCLuyKRfUO7ppuVKWgYYEhFPUIbs31n7esx8j0hHx/sk8GJETO4g2+eBXzfYdwdKgP6NZjtGxIUR0RIRLQNWWLWjqpiZmZn1Kws1B90sVee1zqP0mHZXrRd3L0rA8SnK1/htkoEtAJLWB56OiH9m0q6UXlIovahHU4Kuk4FPA/tRAm+A+yk9pE8AtwJrUHpdJwNExLMZjO2eZa1Gmcs9NyJe7aT+tTaYR36eImKspBuAPYGJknajvFTx+xFxQXVnSccAv4iIb3ZynHr1D8pGAF9tkvfNuvnQ1babfw4R8Y6kf1V61t/JcxLwYERs21GFsuf7XKAlIp6WdColGF5Y9fdX7fdVtY7V9KWAbTKgr9ZLlJ7jR+vKf1jSJMq9d2MOU7+XEgg/l/dEM6L0rN+8kOdUrXdPaFTuKNoeaHXXSGBvlekVy1EedlweEQcDSFqDcs99urqTylSSi4E9IuLlhTy2mZmZWb+1UG9xN+vAq8DKHawvRQmeAb4A3CVpKWCdiBhP6XVblTKPt2r+sOPsTRxYCwAi4mlK0L1h9hbeRRlmfmdu/yfwNGVY7z2UwH3+9nQvZXjwnZXt9UOMu0TSBhExMyJ+QHk4MAy4Gfhi9ugiabCk9wO3AfvlMpJWk/TBZmU3Od7GwCNdeSlZfdt10aPAmpK2zTKWzmNC++tbC8ZfyvPcDyAiZgOzJdVGS1SH8E+orUv6MOUhS30g3VW3AMfUViQNz8WbgWMyUEfS5vnv+sATEXE25UVnm1KGwI/vQr1vprwToDZH/8M5bH5R1X9eyLrMAV6pzC8/hDJFpCPVkRL3AturvDSu9lWZTUXENyNiSEQMBQ6kTBE4uJJlP+AP1Ychktal9OQfEhGPdVI3MzMzM2vAAbr1tEuB8/MlV8tT3uQ8TvmSOErP7wiVr6vaETidMqz68hwWPRU4OyJm181Bn/+yK2AX4I91x50E1IKCCZS55NXewwmUIbtv5PIQ2gfgEyiB6+OUIeir1bZLulhSSzfa4HiVl6TNoLx866aIuAX4FXBPnuc1wMoR8RBlyPgtmf9WYK1uHrc6Z7ozjdquQ/mAYz/gByovS5sG1L4K7FLyelN6cS+ivLjuZsrDiZojgHMyX/UrGs8Flso2uQo4PJq/cbwzxwIt+RK1h8h3GQDfAZYGZkh6MNehjJJ4IOv0Mcoc7/q2bFbviynD7qfkvXwBPdMj/r/Ap/PzU/8VeYdR5s/PoMzPP72TskaTQXxE/I3ydvlr8xpeBSDp/6i8+PNE4FuSnsmXv3XmQBYc3n4KZaj/uVn/1gV3MzMzM7OOqG2kqNm7T9LciKjvHe9sn2WBiRHRkusXAxfXzcHutyTdChwaEc93Ia/brgOSpgBb55z8xZbKG9gviog9ersunVl2rQ1jrcN+3NvVMDOzPmTW2L16uwpm7zpJk2vxTbt0B+j2XlqYAN3MllwtLS3R2urOdjMzM+tfmgXoHuJu7ykH52ZmZmZmZo05QDczMzMzMzPrAxygm5mZmZmZmfUBDtDNzMzMzMzM+gAH6GZmZmZmZmZ9gAN0MzMzMzMzsz7AAbqZmZmZmZlZH+AA3czMzMzMzKwPcIBuZmZmZmZm1gc4QDczMzMzMzPrAwb2dgXMzKz/mvnsHIaOuaG3q2FmttiaNXav3q6CmfUg96CbmZmZmZmZ9QEO0M26SdJQSQ80SD9c0tqV9eMlrdCDx71J0pBcPlDSyR3kvVHSoPz5Wk/VoSdUz6OHyx0j6aDFpdxFIWmWpDVy+e7ePL6ZmZmZ9RwH6GY953Bg7cr68UCPBOiSlgdWj4hnMmkPYFyz/BGxZ0TMBgYB71mALqnDaTMNzqMn7QbcshiV2yMi4uO9XQczMzMz6xkO0M0WzgBJF0l6UNItkg4BWoArJE2TdBwlWB8vaTyApLmSzsp9bpO0ZqYfK+khSTMkXdnkeKOB2zO/gOHAFEkrSfq5pJm5/2czT62HcyywQdbpzNz2dUn3Z/7TMm1FSTdImi7pAUkH1FcgRw78Kfe7TdK6mX6ppPMlTQLOkLSBpHuzTt+VNLfJeWwp6Q5JkyXdLGmtTL9d0g8k3SfpMUmjMn0FSb/JtrpO0iRJLbltFWCZiPibpE/ltqmS/ijpA5nnVEm/lHSPpD9LOjLTR0u6M8//0TyXpd7lcnfN/FMkXS1ppcp1Oy3TZ0oalumr5332oKSLAVWuy9zK8jdyv+mSxmbaBpLGZTtPqJS5f17r6ZLubHK9H5F0haSHJV2j9iNCjmlQz4ZtYWZmZmZd4wDdbOFsCJwTERsDs4EAWoGDImJ4RPwEeA7YISJ2yH1WBFpznzuAb2f6GGDziNgUOApAUksGYjXVHvPNgekREcB/AnMiYpPc/0919RwD/CXr9HVJu2bdR1CC/C0lbQ/sDjwXEZtFxMdo3Dv/P8Av8jhXAGdXtg0BPh4RJwI/AX4SEZsA9T3lewDjJC2d5e0XEVsClwDfq+QbGBEjKKMQau30NeCViNgoz3vLSv6dgdty+S5gm4jYHLgS+PdKvk2BHYFtgVPUNiVhBHAMsBGwAfCZd6tclQcn3wJ2jogtKPfNiZWyXsr084CTMu3bwF1571wHrEsdSXsA+wBbR8RmwBm56ULgmGznk4BzM/0UYLfMu3d9eekjwLkR8VHgH7QfjdGonh21RbWuX5HUKql13utzmhzazMzMrP9xgG62cJ6MiGm5PBkY2oV93gGuyuXLge1yeQal5/1g4G2AiGiNiC9X9h1JCRChBNM35fLOwDm1TBHxSid12DV/pgJTgGGUgH0msEv2XI+KiEZR07bAr3L5l5X6A1wdEfMq+a7O5V/RXu08PgJ8DLhV0jRKwFqdl35t/ltt2+0ogTER8QCl3WqqbTIEuFnSTODrwMaVfL+PiDci4iVgPCWABrgvIp7Ic/h15dzejXK3oQTsE/PcDwM+2Mm5b0+5Z4iIG4BG13ln4OcR8Xrm+3v2zH8cuDqPdQGwVuafCFyavdwDGpQH8HRETMzl6j3brJ4dtcV8EXFhRLRERMuAFVZtcmgzMzOz/sdfs2a2cN6qLM8Dll+IMiL/3YsSgH0KOFnSJhHxdi2TpPUpgdI/M2lX4LMLcTwoQ6O/HxEXLLBB2gLYE/iupNsi4vRulPtapweunIckAQ9GxLZNstfadx5d+z01AvhqLv8P8KOIuF7SaODUSr5ov9v89Wbp70a5Am6NiM83PJPun3tHlgJmR8Tw+g0RcZSkrSn332RJW0bEyw3q22y9WT072sfMzMzMOuAedLOe8yqwcgfrSwH75fIXgLtyTvI6ETEe+AawKrBSXbnzh7dLWpUy/LsWSN0KHF3LKOl9ndTpZuCLlTnPgyW9P4chvx4RlwNnAls0OL+7gQNz+SBgQoM8APfS9gDhwEp6dZj+o8CakrbNeiwtqdoj3chE4HOZfyNgk1zeGHik0oO/KvBsLh9WV8Y+kpaTtDplPvz9mT5C0np5PQ6gXJt3pVxK+4yU9KGs/4qSPtzJud9JuWdqQ9nrrzOUe+GI2jxxSatFxD+AJyXtn2mStFkubxARkyLiFOBvwDoNyly3do3y+Hc1yFOvWVuYmZmZWSccoJv1nEuB81VeyLY8Ze7vOOVL4ii9zCNUvqJtR+B0ytDiy3PY9FTg7IiYXTcHfXfaAttdgD9Wjvld4H21l30BO1S2kYH8xNx+ZkTcQhl2fk8e8xpKAL8JcF8Og/52louk0yXV5icfQwkAZwCHAMc1aYfjgRMz34eA2nD5+eeRowH2A36Q9Z5GGYrdkXMpQf1DWb8Hs+z6N9qfShnSPRl4qa6MGZRh1/cC34mI5zL9fuCnwMPAk5R53u9KuRHxN8ob/3+dbXQPZapBR04Dtpf0IGV+/F/rM0TEOOB6oDWvY21e+EHAl7KdH6TMUwc4M1/w9gDl4ct0SWtLurFS7KPA0ZIepjwUOK+TenbUFmZmZmbWCZX3TJnZu03S3Iio7x3vbJ9lgYkRUXtb+cXAxRFx77tRx56QPbhvRERIOhD4PKXne/55LGS5A4ClI+JNSRtQHlR8BLgBODQinu9k/1OBuRHx33Xpo4GTIuKTdem3vhvlLi4kDQX+kC8N7Oo+p9KgLTrS0tISra2t3a+gmZmZ2WJM0uRGfxt7DrpZHxYRb1G+vq22/uUOsvcVWwI/zXnms4Ev1p/HQlqB8rV1S1PmcX8te+J3WcRyG4qId6VcMzMzM7Nm3INuZma9xj3oZmZm1h8160H3HHQzMzMzMzOzPsABupmZmZmZmVkf4ADdzMzMzMzMrA9wgG5mZmZmZmbWBzhANzMzMzMzM+sDHKCbmZmZmZmZ9QEO0M3MzMzMzMz6AAfoZmZmZmZmZn3AwN6ugJmZ9V8zn53D0DE39HY1zMwW2qyxe/V2FcxsCeIedDMzMzMzM7M+wAG6mZmZmZmZWR/gAN1sMSBpqKQHGqQfLmntyvrxklbowePeJGlILh8o6eSeKruLx79dUks38q8t6Zp3s06VYzW8JpXtkyUtK2luF8s7StKhudzuuvYkSaMlfbzJcYdJmiZpqqQNJN3dzbJPlXRSZX0bSRf1XO3NzMzMlmwO0M0Wb4cD1UDueKBHAnRJywOrR8QzmbQHMK4nyn63RMRzEbFfb9dD0nrAsxHxVhfzD4yI8yPiskw6nPbXtafqNRAYDcwP0OuOuy9wTURsHhF/iYiPL1hKt/T5e8bMzMysL3GAbrb4GCDpIkkPSrpF0iFAC3BF9noeRwnqxksaDyBprqSzcp/bJK2Z6cdKekjSDElXNjneaOD2zC9gODBF0kxJg1S8XOl9vUzSLpJukLRppk2VdEouny7pSEnnSNo7066TdEkuf1HS9xrU45A8vwckjci8p0r6paR7JP1Z0pGZ3rRXW9Kheb7TJf2ykv9PmX6bpHUb7FffK/yApKF1edbPc90qk3anEpg2uQa3S/qxpFbguNpxJO1H++u6fIM6fSOvw3RJYzNtA0njsud+gqRhmX6ppPMlTQJ+AxwFnJBlj6ocd0/KA56vVu+fyjG/Lun+bKvTKuknS3pM0l3AR+qquhPwx0bXw8zMzMwW5ADdbPGxIXBORGwMzAYCaAUOiojhEfET4Dlgh4jYIfdZEWjNfe4Avp3pY4DNI2JTSsCGpBZJF1eOV+393ByYHhEBTARGAhsDTwCjMs+2wN3ABGCUpFWBtzMvme/O2vZMGwxsVLe93goRMRz4GnBJJX1TYMc87ikdDQmXtDHwLWDHiNgMOC43/Q/wi2yHK4Czm5XRQdkfAX4LHB4R92dyNUBvdg0AlomIloj4YS0hIq6h/XV9o+54ewD7AFvnuZyRmy4EjomILYGTgHMruw0BPh4RnwHOB87KsidUjntjZdsOlX2RtCvl/htBeVCzpaTtJW0JHJhpewJbVfZZA/hXRMxp0GZfkdQqqXXe6wtsNjMzM+u3HKCbLT6ejIhpuTwZGNqFfd4Brsrly4HtcnkGpYf2YEoQTUS0RsSXK/uOBO7K5d2Bm3J5ArB9/pwHbCJpMPBKRLxW2T4SuAFYSWVe/HoR8ShtAfxGwEPAC5LWoi3Ar/frrN+dwCqSBmX67yPijYh4CRhPCR6b2RG4OvMSEX/P9G2BX+XyLyvt01VrAr+nBNPTASQtAwyJiCcyT7NrQCW9O3YGfh4Rr0M5F0krUYatXy1pGnABsFZln6sjYt5CHKtm1/yZCkwBhlEC9lHAdRHxekT8A7i+bp9bGhUWERfmg4mWASusugjVMjMzM1uyOEA3W3xU5zPPAwYuRBmR/+4FnANsAdyvMjd5PknrA09HxD8zqRps3UkJzEZRhsD/DdiPEngD3E8Zol3rEZ8KHEl5qEBEPAsMogT9tR71zwFzI+LVDupcv94svae9TfvflctVlucAf6V90D2KtgcbjVTr+VpnB5e0dQ5Hn1abGtDAUsDs7BWv/Xy0O8fprBrA9ytlfygiftbJPp5/bmZmZtZNDtDNFm+vAit3sL4UJXgG+AJwl6SlgHUiYjzwDWBVYKW6cucHVzlUfWBEvAwQEU8DawAbZi/xXZQh1Xfm9n8CTwP7A/dQAvD529O9lPnOd1a2T6CxA7Ie2wFzKkOm95G0nKTVKfPl72+yP8CfgP0zL5JWy/S7KUO0AQ5qUodZlAcZSNoCWK+y7Z/Ap4FDJX0h06qjDaDBNeignjXzr2NETKoExtcDtwJH5KgEJK2WvddPSto/0yRps87K7oabgS9mTz2SBkt6P+X67StpeUkrA5+qHZ8yBWFaN49jZmZm1q85QDdbvF0KnF95mdiFwLjaS74oPacj8sVpOwKnAwOAyyXNpPRunx0Rs+vmoFfnUO/Cgi/6mgQ8lssTKHPJq4HnBODFnD89gTIHekLd9oER8ThlyPRqte2SLlb7r1Z7U9JUyvzoL1XSZ1CGtt8LfCcinqtWUOUr124EiIgHge8Bd0iaDvwosx1DCXZnAIeQc9NVvnrsqMzzW2A1SQ8C/1Y5b7Ls14BPUl68tjflYcEdlSyNrkFnLqX9da0ebxxlKHlrDmevvcDuIOBLeX4PUuapN/K/wKdrL4nrQl2IiFsoUwHuyfvmGmDliJhCGaY/nfJQovaQZEtgar6zwMzMzMy6SP77yWzJJWluRNT3jne2z7LAxIhoyfWLgYsj4t53o44LQ9KplCHx/93bdalS+c74iyJij96uS2+S9C3g8Yho9g0B87W0tERra+t7UCszMzOzvkPS5Nrf21ULM4fVzJZg+d3dLZX1L3eQ3SryO+P7dXAOEBHf7e06mJmZmS2OHKCbLcG623u+uIiIU3u7DmZmZmZmPc1z0M3MzMzMzMz6AAfoZmZmZmZmZn2AA3QzMzMzMzOzPsABupmZmZmZmVkf4ADdzMzMzMzMrA9wgG5mZmZmZmbWBzhANzMzMzMzM+sDHKCbmZmZmZmZ9QEDe7sCZmbWf818dg5Dx9zQ29UwM2tn1ti9ersKZtZPuQfdzMzMzMzMrA9wgG5mZmZmZmbWBzhAt35L0lBJDzRIP1zS2pX14yWt0IPHvUnSkFw+UNLJPVV2F49/uqSd38tj5nGHSZomaaqkDZrkGSPpIElrSpqUeUc1yXugpJMl7S1pTBeOf6akByWduajn0lX1946kGyUNWoTytpF0UWV9XUlzJZ1USTshz/MBSb+WtFymr5dt+rikqyQtk+lHSZqZ1+YuSRtl+jKSfp7bpksaXTnG7ZIezX2mSXp/XT0/KykktSzsuZqZmZn1Rw7QzRZ0OLB2Zf14oEcCdEnLA6tHxDOZtAcwrifK7qqIOCUi/vheHjPtC1wTEZtHxF9qiSpqv4t2A24BdgJmZt4J1UIkDcjFPYBxEXF9RIztwvG/AmwaEV/vSmUl9cQ7Oo6ncu9ExJ4RMXsRyqu/X34E3FRbkTQYOBZoiYiPAQOAA3PzD4CzIuJDwCvAlzL9VxGxSUQMB87IMgGOzDpvAuwC/LBynQAOiojh+fNipQ4rA8cBkxbhPM3MzMz6JQfo1t8NkHRR9jjeIukQoAW4InsGj6ME6+MljQfIHsuzcp/bJK2Z6cdKekjSDElXNjneaOD2zC9gODAleykHZbD6sqRDM89lknaRdIOkTTNtqqRTcvl0SUdKOkfS3pl2naRLcvmLkr5XrYCkSyXtl8uzJJ0mqVaHYZn+iUrv6NQMupD0dUn35zmeVinzYEn3Zf4LKkF0bfuelGD1q5LGq4xeeFTSZcADwDqSVgGWAQZTAsV9srzls81/KGk6sG1d2x0u6aeVcztb0t2Snqic5/XASsBkSQfk8f+U53GbpHUr+58vaRJwRq6fJ+neLG+0pEskPSzp0sr5nSepNe+J02r3Q4N7Z5akNXL5RJVe7gckHZ9pQ7Ps6j25fKUpdwL+mHn3BZ4EHqy7xwYCy+cDhhWA57K9dgSuyTy/oDwwISL+Udl3RSByeSPgT5nnRWA25bPRme9QHga82YW8ZmZmZlbhAN36uw2BcyJiY0oAEkArbb2DPwGeA3aIiB1ynxWB1tznDuDbmT4G2DwiNgWOApDUIuniyvGqPaCbA9MjIoCJwEhgY+AJoDase1vgbmACMErSqsDbmZfMd2dte6YNpgRX1e0deSkitgDOA2pDpU8Cjs5e1VHAG5J2zfYaQQmOt5S0vaSPAgcAIzP/POCg6gEi4kbgfEoPbq0dNwTOjYiNI+IpYGfgtoiYBpwCXJXX4A1Km0+KiM0i4q66tqu3FrAd8ElgbB5/b+CNLO8q4H+AX+S1ugI4u7L/EODjEXFirr+Pch1OAK4HzqJcp00kDc88J0dEC7Ap8AlJm0bE2Sx47wAgaUvgCGBrYBvgSEmbV9qlek9+NvdZA/hXRMyRtBLwDeC0arkR8Szw38BfgeeBORFxC7A6MDsi3s6sz1Duk1p9jpb0F8qDkWMzeTqwt6SBktYDtgTWqRzu5/kA5T/zAQCStgDWiYgOX8su6Sv5QKN13utzOspqZmZm1q84QLf+7skMCAEmA0O7sM87wFW5fDklGASYQel5P5gSRBMRrRHx5cq+I4G7cnl32oYnTwC2z5/zKMHfYOCViHitsn0kcAOwksrc5vUi4lHaAviNgIeAFyStRVuA35FrG5z/ROBH2Qs8KAO7XfNnKjAFGEYJJneiBG/3S5qW6+t3ckyApyLi3sp6tT3qzQN+28W8v4uIdyLiIeADTfJsC/wql39J2zUEuDoi5lXW/zcfBMwEXoiImRHxDqXnemjm+ZykKZS22Zi2ByTNbAdcFxGvRcRcyjWoPWBpdk/uShn+D3Aq5WHH3Gqhkt4H7AOsR+m9XzHvxw5FxDkRsQEl6P9WJl9CCeRbgR9T7qNauxyUQ99H5c8hKsPffwT8f1043oUR0RIRLQNWWLWz7GZmZmb9hr8H3fq7tyrL84Dlm2XsQK0Xdy9KEP0p4GRJm1R6LJG0PvB0RPwzk3Yle0cpvdxHA+sCJwOfBvajBN4A91OGFz8B3AqsQZkjPBlKz6nKy8d2z7JWAz4HzI2IVzupf60N5pG/EyJirKQbgD2BiZJ2AwR8PyIuqO4s6RhKb/Q3OzlOvdfq1kcAX22S9826oLnadvWq11TdrFOjetXKe6eu7HeAWu/yScBWEfFKDn1fbiGOW388aH9P7kHb/PCtgf0knQEMAt6R9CbwAiXA/xuApGuBj1NGCQySNDDvySHAsw2OfSXlARGZ74TaBkl3A4/ltmfz31cl/Ypy7X4PfAy4PTvU/w9wvaS9I6J1oVvDzMzMrB9xD7rZgl4FVu5gfSlK8AzwBeCu7D1cJyLGU3ohV6XMea6aP7w9h6oPjIiXASLiaUrQvWFEPEHpZT+JHJ6eQf3TwP7APZTAff72dC9lnvedle3tXrDWVZI2yJ7iH1AeDgwDbga+mMOrkTRY5e3dt1GCxfdn+mqSPtjN420MPFIXhDfL267tFtLdtL087SAWsp3SKpSgfo6kD1Cuc039vVMzAdhX0gqSVqQ8kGlahxxCvikwDSAiRkXE0IgYSund/q+I+CllaPs2Wa4ooxkezhEA42m7bw+jBNRI2rByqL2AP2d6rW5I2gV4OyIeyiHvtXn0S1OmEjwQEXMiYo1Kve4FHJybmZmZdYMDdLMFXQqcn/NrlwcuBMbVXvRFCcZGqHxF247A6ZS3ZV8uaSZlmPPZETG7bg767rTNP9+FfNlXxSSyh5ISrA2mbTh8Le3FnJM9gdILOqFu+8CIeJwyBH212nZJF6t7X3l1fL68bAbwL+CmnMv8K+CePM9rgJVzKPm3gFsy/62UeeDdOW533mbfqO266xjgiKzvIZS3ji+UiJhOueaPUNpnYmVz/b1T22cK5T67j3LdL46IqR0cZktgapM599VyJ1GuyxTKkPylsg5QHhydKOlxypz0n2X6v+UL6aYBJ1KCd4D3U17C93Due0imLwvcnG03jdITP/+r38zMzMxs4amTv/fMrI6kuRFR3zve2T7LAhPzRWJk0H5x3RzsfkvSrcChEfF8F/L2u7aT9C3g8Yho9u0Ai62WlpZobXUnu5mZmfUvkibXYoN26Q7QzbpnYQJ0M2vMAbqZmZn1R80CdA9xN+smB+dmZmZmZvZucIBuZmZmZmZm1gc4QDczMzMzMzPrAxygm5mZmZmZmfUBDtDNzMzMzMzM+gAH6GZmZmZmZmZ9gAN0MzMzMzMzsz7AAbqZmZmZmZlZH+AA3czMzMzMzKwPGNjbFTAzs/5r5rNzGDrmht6uhpn1QbPG7tXbVTAze8+5B93MzMzMzMysD3CAbmZmZmZmZtYHvGsBuqShkh5okH64pLUr68dLWmEhyl9a0pTK+vmSRi58jZdskv7jXS6/3fXowXInS1p2IfddS9ItPV2nJsfqcvtKuknSEEm3S2p5N+vV5Pg3Shq0iGVMlrSspLlNth8l6dAO9h8t6eOLUocGZa4l6ZZmv3t68DijJf2hybaLJW3Uwb7tfv+9C3X7pqTHJT0qabdK+iWSXuyoXSSdKumkBumDJH1tIerSsDwzMzMza643etAPB6p/oB4PdDtAB7YDJlbWtwHuXehaLfkWKUCXNKCTLPXXY5FJWg94NiLeWsgidgdu7sEqdaRL7StpeWD1iHjmXa5PUxGxZ0TMXtj9u3JdIuL8iLisg2JGA90K0CV19s6M9/J6NxQRX46IhzrIcjjtf/91qgufvVq+jYADgY0pbXFuZd9LM21hDAK6HaCbmZmZWfe92wH6AEkXSXowe7YOAVqAKyRNk3Qc5Y/V8ZLGA0iaK+ms3Oc2SWs2KXt34Kbc56PAY8DqkiZn2maSQtK6uf4XSStKelLFIEnzJG2f2++UtKGkmblNkl6u9QJKukzSLtUKVHsPJe0n6dJc3l/SA5KmS7oz01aQ9BtJD0m6TtKkWu+ppF0l3SNpiqSrJa2U6bMknZbpMyUNy/RTs0fsdklPSDq2Uo+DJd2X7XuBpAGSxgLLZ9oV9Q0paSdJU/MYlyh7rPP4P1DpGd9f0p6SHsne07PrehF3B26q772UdJKkU3P52Dz/GZKuzLQRee5TJd0t6SN1ZY7r6L6QdKSk+7Otf6v2ozGq98g38vymZ3sgabike7M+10l6X6bP79mWtIakWbl8uKRrJY2T9GdJZ2R6u/aVdLqk4ytt8D2Vex1KYHp7Xft/UdKPK+tHSjorlw/N+k2X9MtMGyrpT5l+m9ru8UslnZfn9IRKT+8lkh6u3ZuV67pGlvOw2n9Gl888W2X50ySdqfY9r/OvS+X8pudxP5Bp83tP66+7pKHAUcAJWf6oTs7pfEmTgDOy3WvXfimV3uI1K/W6qa5t75Q0vLJ+l8rvhpUk/TzviRmSPpvbP59pD0j6Ac2tJOkalc/DFZKU+98uqUXlc3dpljNT0gmS9qP977/l1bXP3hi1Hy20oRqPVtkHuDIi3oqIJ4HHgREAEXEn8PcOzqdmIy34e2UssEHlXhgt6Q5Jv898YyUdpPJ7Z6akDbpwHDMzMzNr4N0O0DcEzomIjYHZQACtwEERMTwifgI8B+wQETvkPisCrbnPHcC3Yf6Q2aMqZe9AW6CzBzAuIl4ElpO0CjAqjzVK0geBFyPiNeBRYCNKj++U3L4ssE5E/JnSCzyS0gv1RJYDsC1wdxfP+xRgt4jYDNg7074GvBIRGwH/CWyZ57UG8C1g54jYIut8YqWslzL9PKA6XHQYsBvlD/Bvqwwx/yhwADAyIoYD8yhtPQZ4I9v8oGpFJS1H6V07ICI2obzZ/6uVLC/n8X8HXADsERFbAvUPTqrXo5kxwOYRsSklQAN4BBgVEZtnu/1XJX81EGx4XwDXRsRW2dYPA1/K8xoAfCQiHpK0ByV42TrznZH7XgZ8I+szs1JmR4ZT2ngT4ABJ6zRo30uA2oOdpSi9mpfn/ntUzqnmN8CnJC2d60cAl0jamHJv7Jj1rgX5/wP8Iut9BXB2paz3Ue7VE4DrgbMo9/Im1UC1ov4z+tlM/znwfyv3UVX9dbk363cncGSDY7S77hExCzgfOCvbbEIn5zQE+HhEnEhpx9o9vDMwPSL+Vr3edcf+GaXXGkkfBpaLiOmUz+CciNgkj/knlaHnPwB2pFznrSTt2+B8ADanjP7ZCFif8jujajgwOCI+lp+rn0fENVR+/1F+H15KJ5+9iPgeMKdy/Y6gXJ96g4GnK+vPZFp3LPB7hXL9/pLX6uuZbzPKZ/ijwCHAhyNiBHAxcExnB5H0FUmtklrnvT6nm1U0MzMzW3K92wH6kxExLZcnA0O7sM87wFW5fDklkK4NmT0fQNJg4O8R8Xrm2422gOFuyh/L21OCve0pQfaE3D4h07YHvp/lbwXc32D7eZTAZjAluH6ti+c9EbhU0pFAbYjpdsCVeS4PADMyfRvKH/kTJU0DDgM+WCnr2vy3vv1uyJ6yl4AXgQ8AO1EC//uzrJ0owUNHPkK5To/l+i8o515TuxbDgCeyZw7g17UMDa5HMzMovYcHA29n2qrA1dlDWwsmkbQMMCQinsh8De8L4GOSJkiaSQncNs70rYFJubwzJUB6HSAi/i5pVWBQRNzR5LybuS0i5kTEm8BDtL9WZPmzgJclbQ7sCkyNiJdz80jgrrr8c4E/AZ9UGSWxdETMpASKV+c1JiJqPaDbAr/K5V9W2gLgfyMiKA8cXoiImRHxDvAgjT9/C3xGVeanrxwR92R67ViNrss/gT9U929wjEbXvV5H53R1RNQeEsx/+AF8kbZAtXq9q66mtOvSmf/STN8ZOKeWKSJeofweuD0i/hYRb1MeFDS7J+6LiGeybaex4Hk/Aawv6X8k7Q78o0EZXf3sQQl8j8gHEQdQuSY9rNHvlUbuj4jnc5rDX4Daux5m0oXf8xFxYUS0RETLgBVW7Yl6m5mZmS0R3u0AvTpHdR4L973r0SBt/lxTlSHNgyLiudx2JyUg/yDwe0pPz3a0Bei17SOAGynzK0c32D6K0iP8N2C/yvZmdVtufmLEUZSez3WAyZJW7+D8BNyavVPDI2KjiPhSZXutDevbr1HbitILWSvrIxFxagfH7oquPJSozv19m/b31XKV5b0oQdEWlIcIA4HvAOMj4mPApyr5R1EXyNaptf2lwL9lD+Rplf0b9VR3VfUclqvb1tV7+mJKz+0RlKASSesDT0fEPzvJ36h3tKtq9Xunrq7vNKlrdz+j9dflX/lAoKP9G1337ph/D0bE08ALknakfIZrQ9obXu98KHMrZQTF5yhBd7dI2jqHd0+TVBsR02G7ZcC/GeV3yFGU69td1c/ebynn+ElgckS8LOnTlXq1AM9SfufUDMm07ujq/VB/b1Xvu4X5PW9mZmZm9M5L4l4FVu5gfSlKQAzwBRoHadW5pjsA4yvbJgAHA3/O3q2/A3tWyrmP8nKqd7IXdBrwfymBeS0AWAPYMHsJ76IMLb+zQT1ekPTRHMb86VqipA0iYlJEnEIJ8Neh9Kp/LrdvRBkiDeXFdiMlfSi3rZhDcRfGbcB+kt6fZa2Ww/sB/lUZQl31KKXX9EO5fghlCHmjfOurzB+G0otXU70eLwDvl7R6Th34ZNZlKco0gvHANyg95yvlv7Ug4vAmZULz+2Jl4Pk8t+rw/Z2AP+byrZTexxWyLqtFxBzgFUm1KQzV855FTkGoHLMz9e17XZ7DVrQ9vGj60CAiJlHuky/QNjrhT5S5/6vX6p3pd1OGzUM550YPjxZalBfIvSpp60w6sLJ5gXneHengutd/7rtzThdTRlFUe9ar17tR/rMpvb6vZNqtwNGVer6P8rvhEyrz8wcAnwfuyM9y7aHX9V087zWApSLit5SHdVvkpup5d/WzR/6uupkyqufnmXZdpV6tlCkNB6q8XX89yvSF+7pS307UXyszMzMze5f0RoB+KXB+7SVJwIXAOOVL4ii9RiNyyPOOwOnQNgc9/3D+UEQ8kvnbBT05vFi0BdR3AbNrf5jnkMynaXvj+wTKH58zK3WcRHnpXG374CyHHDpeM4YyvPdu4PlK+pn5sqQHctt04FxgTUkPAd+lDDmeExF/owSmv5Y0A7iHMpy823L+7beAW7KsW4G1cvOFwAzlS+JUvmpr7fzD/wjKMPOZlB6w8xuU/QZlHv04lRfxvUqZF9vuekTEvyjX7L48fu06DQAuz2NMBc7OQPAM4PuSptK+52007YOVhvcFZS7xJMoDkEfy3NYE3oyIV7NO4yjBS2tev9pc/sMo12oGZc5wrcz/Br6adVqjYWMvqF37Zi/5eOA3lSCy3cvVGvgNMLFyrz4IfA+4Q9J04EeZ7xjKA4cZlKDuuEaFLaIvARdle60I1CYKj6ZJENlEs+v+v0CtB3gU3Tun6ylB/s9hwetdLyImU4aYV0cmfBd4n/JljpT3YDxP+UyPp3xmJ0fE77txrlWDgduz/S4Hvpnpl5K//yi/pzr97FVckXkafnVg3i+/oUy9GAccXbv3JP2a8rvlI5KekVR7V0P9uz0alfsyZQrOA5LO7OS8zczMzGwRqG10at8gaW5ErNTB9u2Ag3MYOSpvM946A8M+KwPZpSPiTZW3HP+R8lKrRsOd+yRJK0XEXEmiDFn+M2Xu/vzr0UPHGQJcFBF7VNI6vC/q9j+YMk96bE/Vqbuy53gKsH9E/DlHE0yMiKbfe67yVvyzIuK296qeHdRlpZwbj6QxlAc9Z1J3XXqpbi2UdhqV6x1eb5WXv90ODMtRNYsllbfirxoR/9nbdelJLS0t0dra2tvVMDMzM3tPSZrcKDZY7OYKRsRdVIa9R3nD+OJgBcrXyS1N6Tn72uIUnKcjJR0GLEPpDb0g5/h2NFe826J8R/hCB4ERcXnnud49OYXhD8B1Ub4ZoDZyo2FwrvJStvsobyTv9eA87SXpm5TfEU8Bh+doj94OzsdQ3nQ+fzpDR9db5WsSvwecuJgH59cBG1BGj5iZmZnZEqrP9aCbmVn/4R50MzMz64+a9aD3xhx0MzMzMzMzM6vjAN3MzMzMzMysD3CAbmZmZmZmZtYHeA66mZn1GkmvUr4T3vquNYCXersS1ilfp8WDr1Pf52u0eFgSrtMHI2LN+sTF7i3uZma2RHm0o68ftN4nqdXXqO/zdVo8+Dr1fb5Gi4cl+Tp5iLuZmZmZmZlZH+AA3czMzMzMzKwPcIBuZma96cLeroB1ytdo8eDrtHjwder7fI0WD0vsdfJL4szMzMzMzMz6APegm5mZmZmZmfUBDtDNzMzMzMzM+gAH6GZm9p6TtLukRyU9LmlMb9fHCkmXSHpR0gOVtNUk3Srpz/nv+3qzjv2dpHUkjZf0kKQHJR2X6b5OfYik5STdJ2l6XqfTMn09SZPyd99Vkpbp7br2d5IGSJoq6Q+57mvUx0iaJWmmpGmSWjNtif2d5wDdzMzeU5IGAOcAewAbAZ+XtFHv1srSpcDudWljgNsiYkPgtly33vM28P9FxEbANsDR+fnxdepb3gJ2jIjNgOHA7pK2AX4AnBURHwJeAb7Ue1W0dBzwcGXd16hv2iEihle++3yJ/Z3nAN3MzN5rI4DHI+KJiPgncCWwTy/XyYCIuBP4e13yPsAvcvkXwL7vZZ2svYh4PiKm5PKrlMBiML5OfUoUc3N16fwJYEfgmkz3deplkoYAewEX57rwNVpcLLG/8xygm5nZe20w8HRl/ZlMs77pAxHxfC7/P+ADvVkZayNpKLA5MAlfpz4nh05PA14EbgX+AsyOiLczi3/39b4fA/8OvJPrq+Nr1BcFcIukyZK+kmlL7O+8gb1dATMzM1s8RERI8vez9gGSVgJ+CxwfEf8oHX+Fr1PfEBHzgOGSBgHXAcN6t0ZWJemTwIsRMVnS6F6ujnVsu4h4VtL7gVslPVLduKT9znMPupmZvdeeBdaprA/JNOubXpC0FkD++2Iv16ffk7Q0JTi/IiKuzWRfpz4qImYD44FtgUGSah1k/t3Xu0YCe0uaRZlqtSPwE3yN+pyIeDb/fZHysGsES/DvPAfoZmb2Xrsf2DDflLsMcCBwfS/XyZq7Hjgslw8Dft+Lden3co7sz4CHI+JHlU2+Tn2IpDWz5xxJywO7UN4XMB7YL7P5OvWiiPhmRAyJiKGU/4f+FBEH4WvUp0haUdLKtWVgV+ABluDfeYpYYkYDmJnZYkLSnpS5fwOASyLie71bIwOQ9GtgNLAG8ALwbeB3wG+AdYGngM9FRP2L5Ow9Imk7YAIwk7Z5s/9BmYfu69RHSNqU8uKqAZQOsd9ExOmS1qf01q4GTAUOjoi3eq+mBpBD3E+KiE/6GvUteT2uy9WBwK8i4nuSVmcJ/Z3nAN3MzMzMzMysD/AQdzMzMzMzM7M+wAG6mZmZmZmZWR/gAN3MzMzMzMysD3CAbmZmZmZmZtYHOEA3MzMzMzMz6wMcoJuZmZlZt0kKST+srJ8k6dQeKvtSSft1nnORj7O/pIclje9i/tMl7ZzLx0taoQv7tMsn6cbad6SbmdVzgG5mZmZmC+Mt4DOS1ujtilRJGtiN7F8CjoyIHbqSOSJOiYg/5urxQKcBen2+iNgzImZ3o45m1o84QDczMzOzhfE2cCFwQv2G+h5wSXPz39GS7pD0e0lPSBor6SBJ90maKWmDSjE7S2qV9JikT+b+AySdKel+STMk/d9KuRMkXQ881KA+n8/yH5D0g0w7BdgO+JmkMxvs843cZ7qksdXzknQssDYwvtb7Lum8rO+Dkk7LtEb5ZtUeakg6Mev0gKTjM21o9upflGXdImn5WnmSHspzv7LLV8rMFhvdecJoZmZmZlZ1DjBD0hnd2Gcz4KPA34EngIsjYoSk44BjKD3OAEOBEcAGlAD3Q8ChwJyI2ErSssBESbdk/i2Aj0XEk9WDSVob+AGwJfAKcIukfSPidEk7AidFRGvdPnsA+wBbR8Trklarbo+IsyWdCOwQES9l8skR8XdJA4DbJG3aJF/tGFsCRwBbAwImSboj67gh8PmIOFLSb4DPApcDY4D1IuItD5M3WzK5B93MzMzMFkpE/AO4DDi2G7vdHxHPR8RbwF+AWoA9kxKU1/wmIt6JiD9TAvlhwK7AoZKmAZOA1SnBLMB99cF52gq4PSL+FhFvA1cA23dSx52Bn0fE63mef+/CeX1O0hRgKrAxsFEn+bcDrouI1yJiLnAtMCq3PRkR03J5Mm3tMgO4QtLBlBEMZraEcYBuZmZmZovix5S53CtW0t4m/86UtBSwTGXbW5Xldyrr79B+dGfUHScoPc3HRMTw/FkvImoB/muLchKLQtJ6wEnAThGxKXADsNwiFFlto3m0tctelFELWwD3d3O+vZktBhygm5mZmdlCy97l31CC9JpZlCHlAHsDSy9E0ftLWirnpa8PPArcDHxV0tIAkj4sacWOCgHuAz4haY0cfv554I5O9rkVOKL29vX6Ie7pVWDlXF6F8oBgjqQPAHs0yVc1AdhX0gp5Dp/OtIbyQcc6ETEe+AawKrBSJ+dhZosZP3UzMzMzs0X1Q+DfKusXAb+XNB0Yx8L1bv+VElyvAhwVEW9Kupgy3HuKJAF/A/btqJCIeF7SGGA8pQf+hoj4fSf7jJM0HGiV9E/gRuA/6rJdCIyT9FxE7CBpKvAI8DQwsVm+yjGmSLo0zxHKXPypkoY2qdYA4HJJq+Z5nO23wZsteRRRP3rIzMzMzMzMzN5rHuJuZmZmZmZm1gc4QDczMzMzMzPrAxygm5mZmZmZmfUBDtDNzMzMzMzM+gAH6GZmZmZmZmZ9gAN0MzMzMzMzsz7AAbqZmZmZmZlZH/D/A/8qhSb5iIb9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "top_urls = url_counts_several.head(10).sort_values()\n",
    "\n",
    "ax = top_urls.plot(figsize=[14,4], title=\"Top 10 cited URLs\", legend=False, kind=\"barh\")\n",
    "ax.set_ylabel(\"URL\")\n",
    "ax.set_xlabel(\"Number of citations\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/top_cited_urls.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following step, for each citation, Citoid's success is calculated as the sum of correct answers from the 6 evaluated metadata ('comp_author_first', 'comp_author_last', 'comp_title', 'comp_source_type', 'comp_pub_source', 'comp_pub_date'). For instance, if only the source type and the publishing date are correct, Citoid's success is 2. A response from Citoid is considered correct if the comparison with the manual data matches 75% or more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "comp_fields = ['comp_author_first', 'comp_author_last', 'comp_title',\n",
    "          'comp_source_type', 'comp_pub_source', 'comp_pub_date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "scores[\"citoid_success\"] = (scores[comp_fields] >= 0.75).sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's group the citations having the same URL, calculate the average success score for the repeated URLs and plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlNUlEQVR4nO3de7gdZX328e9tQEQJx0QaEjCIAQtUESLEekJRCIgFrdXwVoiIRitYT60CtUVBFGtBX1vFRogcFCKCSOobhYgieAgQkFMAJQSQhEBCAoZTUfB+/5hn67DZa++VSfZae5H7c13r2jPPzDzzm1lrr98888yakW0iIiKaeFa3A4iIiN6VJBIREY0liURERGNJIhER0ViSSERENJYkEhERjSWJjACSvibpX9dRXdtJeljSqDJ+maT3rIu6S30/kDR9XdW3Buv9jKT7Jd3bcPmFkvZuuOyxkk4bZPqdkt7QpO4YeSTtLWlJt+PoFUkiw6x8wTwm6SFJD0r6haT3S/rTvrf9ftsntFnXoF9Wtn9rexPbT66D2D8l6Zv96t/f9plrW/caxrEd8DFgZ9t/0WKeTSV9SdJvSxK9vYyPKXHvYvuyMu/Ttmswtj9ru61EXOr+VLt1j2RJjmtvoIO4/klKkiU9Uj63SyWd0ncQ2KqOkSRJpDPebHs08ALgJOATwOnreiWSNljXdY4Q2wErbS8faKKkZwOXArsAU4FNgVcAK4E9OxVktw30/j+DPxNrpAf2w0ttbwK8FngH8O4ux9O2JJEOsv0723OoPiTTJe0KIOkMSZ8pw2Mkfb+0WlZJukLSsySdTfVl+j/liOXjkiaWo5gjJP0W+HGtrP5Ps4OkqyStlnSRpC3Lup7WbO87+pQ0FTgWeEdZ3/Vl+p+Oikpcn5R0l6Tlks6StFmZ1hfH9NI6uF/Sv7TaN5I2K8uvKPV9stT/BmAesE2J44wBFj+s7Ju32L7Z9h9tL7d9gu25bW7XNpLmlH2+SNJ7a7E9peUi6dAS48ohtmnA97LFvLtImlfmu0/SsaV8o9Kiuqe8viRpo/r7J+kTqk7zfaPEer6kb0paDbyr7NvTJS0rR7qf6Xek+15Jt6hqLd8safeBPm8DxLxF2b4Vkh4owxNq0y+TdIKkn5e6L1FpGUp6TolxZdk/V0vaWtLrJN1Yq2OepKtr41dIOrj2nl1Q1n+HpH/s95713w97Slqg6v/gPkmntHrvSh3Hls/tnZL+vpS9vCxb339v7fscrS3bi4CfA7uti/o6wnZew/gC7gTeMED5b4F/KMNnAJ8pw58DvgZsWF6vBjRQXcBEwMBZwPOAjWtlG5R5LgOWAruWeS4Avlmm7Q0saRUv8Km+eWvTLwPeU4bfDSwCXghsAnwXOLtfbF8vcb0UeBz4yxb76SzgImB0WfY3wBGt4uy37GzgzHbfhxbbdTnwVeA5VP/AK4DX958f2Bl4GHgNsBFwCvBEi/e45XvZb77RwDKqU3bPKeN7lWnHA/OB5wNjgV8AJ9T2yxPA50ssG5dY/wAcTHWQuDFwIfDf5f1/PnAV8L5Sx9+Vz8fLAQEvAl4w2Ge3FvdWwN8Czy0xfwf4Xr/Pyu3AjiWOy4CTyrT3Af9Tlh0F7EHVgtwY+F9gTNln95X4Rpdpj5X1Pgu4Bvg34NlUn8HFwH6196z/fvglcGiZvgkwpcV29e3XU8p+fS3wCLBTmX4zsH9t/guBj7Wo6zLK/0u/+pfUxg28qAy/uHwWPjJYHSPplZZI99wDbDlA+R+AcVT/yH+wfYXLJ2kQn7L9iO3HWkw/2/ZNth8B/hV4e/1Iai38PXCK7cW2HwaOAabpqa2gT9t+zPb1wPVUyeQpSizTgGNsP2T7TuBk4NA249iK6h+vEUnbAq8EPmH7f21fB5xG1cLp723A921fbvtxqv35xxZVt/teHgjca/vksv6HbF9Zpv09cLyrltUK4NM8db/8ETjO9uO19/+Xtr9n+49UX8wHAB8un5HlwBep9jfAe4B/t321K4ts3zX4HqvYXmn7AtuP2n4IOJHqC7fuG7Z/U2I7jz8fYf+B6n17ke0nbV9je3WZ72qqJL0H1Wfm51TvzxTgNtsrqZLeWNvH2/697cVUByzTauv+034o9f4BeJGkMbYftj1/iE3817Jffwr8P+DtpfxM4J0Aqlr1+wHntLPPBnGtpEeAW6iSxlfXsr6OSRLpnvHAqgHKv0B1dH+JpMWSjm6jrrvXYPpdVEd4Y9qKcnDblPrqdW8AbF0rq19N9SjVEWB/fUed/esa32YcK6m+rJvaBlhVvgiHWv821PZnScwrW9Tb7nu5LdURe6vY+u+XbWrjK2z/b79l6u/3C6j27bJy2uhBqlbJ89tY96AkPVfSf5dTe6upWnOb9ztAafX+nw1cDMwup+n+XdKGZdpPqY7WX1OGL6NKTq8t433btU3fNpXtOpanfvb6/18cQdUqurWcPjtwkM17oLy3fer7/ZvAmyU9jyqxXGG71UHME1T7v25DqoRWtzvVvnkHsBdVq7EnJIl0gaSXU31B/az/tHIU+jHbLwT+BviopH36JreocqiWyra14e2oPsD3UzXRn1uLaxTVKZN2672H6p+5XvcTVKcg1sT9Jab+dS1tc/kfAfuVf+p29N+ue4AtJY1uY/3LqO1PSc+lOqJ++koGfy/r7qY6HTOQgfbxPYNsS/+yu6lOI46xvXl5bWp7l9r0HVqse6j3/2PATlSn3jal+tKH6rTYoErL7NO2dwb+mqo11tfy659EfsrTk8jdwB21bdrc9mjbB7SK3/Zttg+hSqCfB84f5DOzRb9pf9rvtpdSnRp7K1Wr8OxBNvW3VKdn67bnqQcGffHZ9nml7n8bpM4RJUmkg1Rdhnog1Tn8b9q+cYB5DpT0IkkCfgc8yZ9Pl9xH6y+bwbxT0s7lC+944HxXlwD/BniOpDeVo8BPUp0D7nMfMFEtOoOBc4GPSNpe0ibAZ4Fv235iTYIrsZwHnChptKQXAB+lOuJrx9lUXyoXSHqxqg75rUrH6AEDzP+U7bJ9N1Vfw+dKh+9LqI5aB1r/+cCBkl6l6qqw42nxfzTEe1n3fWCcpA+r6kgfLWmvMu1c4JOSxpZO6X9rEdeAyhHyJcDJ5fP3LEk7SOo77XQa8E+S9lDlRWX/9+2nwT5vo6n6KB4sp3WOazeu0oH+V+XAZTXVQUTfvvkFVXLaE7jK9kKqRLoXVWsHqn6dh1RdVLCxpFGSdi0HaK3W+U5JY8tpvgdLcatTkQCflvRsSa+mSnLfqU07C/g48FdUfYGtfBs4vHTqS9KOwEeovgNaOQl4r6T65ewblM9m36t/66ZrkkQ6438kPUT1RfcvVB12h7eYdxLVkfXDVEckX7X9kzLtc1RfKA9K+qc1WP/ZVJ3391J13P4jVFeLAR+g+iJZStUyqV+t1fdPs1LStQPUO6vUfTlwB1WH6AfXIK66D5b1L6ZqoZ1T6h9S6Zt4A3Ar1ZVcq6m+ZMYAVw6wyEDbdQjVEeM9VB2lx9n+0QDrWggcWeJbBjzAU/dZ3WDvZb3Oh4A3Am+meo9uA15XJn8GWADcANwIXFvK1sRhVJ3PN5d4z6ec/rP9Haq+jHOAh4Dv8ee+uqE+b1+i6rC+n6rz/4drENNflDhWU/UD/JRyRF9OI10LLLT9+zL/L4G7Sp9O34HHgVR9LHeUGE4DNhtknVOBhZIeBv4vMG2QfsR7qfbVPcC3gPfbvrU2/UKqxHah7UdbrdD2xcDRwDeoDiTmUvWpzBxkmRup/qf+uVZ8KlXC7nt9o/VmdlbfVT8REbEGJN1OdZXb0w421idpiURErCFJf0vV5/LjbsfSbSP9V5wRESOKpMuofi90aOlfWa/ldFZERDSW01kREdHYenc6a8yYMZ44cWK3w4iI6CnXXHPN/bbH9i9f75LIxIkTWbBgQbfDiIjoKZIGvB1OTmdFRERjSSIREdFYkkhERDSWJBIREY0liURERGNJIhER0ViSSERENJYkEhERjSWJREREY+vdL9Yj1jfzF69k2sz53Q5jSLNnTGHKCwd80nCMYMPWEpG0raSfSLpZ0kJJHyrlW0qaJ+m28neLUi5JX5a0SNINknav1TW9zH+bpOm18j0k3ViW+XJ5DGlERHTIcJ7OegL4mO2dgSnAkZJ2pnpU5KW2JwGXlnGA/akeJzoJmEH1OEhqz27ei+qZy8f1JZ4yz3try00dxu2JiIh+hi2J2F5m+9oy/BDVc5THAwdRPWOY8vfgMnwQcJYr84HNJY0D9gPm2V5l+wGqZ2hPLdM2tT3f1UNRzqrVFRERHdCRjnVJE4GXAVcCW9teVibdC2xdhscDd9cWW1LKBitfMkB5RER0yLAnEUmbABcAH7a9uj6ttCCG/dGKkmZIWiBpwYoVK4Z7dRER641hTSKSNqRKIN+y/d1SfF85FUX5u7yULwW2rS0+oZQNVj5hgPKnsT3T9mTbk8eOfdozVSIioqHhvDpLwOnALbZPqU2aA/RdYTUduKhWfli5SmsK8Lty2utiYF9JW5QO9X2Bi8u01ZKmlHUdVqsrIiI6YDh/J/JK4FDgRknXlbJjgZOA8yQdAdwFvL1MmwscACwCHgUOB7C9StIJwNVlvuNtryrDHwDOADYGflBeERHRIcOWRGz/DGj1u419BpjfwJEt6poFzBqgfAGw61qEGRERayG3PYmIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorHhfJ5IxDPS/MUrmTZzfrfDaMvsGVO6HUI8w6UlEhERjSWJREREY8P5jPVZkpZLuqlW9m1J15XXnX2PzZU0UdJjtWlfqy2zh6QbJS2S9OXyPHUkbSlpnqTbyt8thmtbIiJiYMPZEjkDmFovsP0O27vZ3g24APhubfLtfdNsv79WfirwXmBSefXVeTRwqe1JwKVlPCIiOmjYkojty4FVA00rrYm3A+cOVoekccCmtueXZ7CfBRxcJh8EnFmGz6yVR0REh3SrT+TVwH22b6uVbS/pV5J+KunVpWw8sKQ2z5JSBrC17WVl+F5g61YrkzRD0gJJC1asWLGONiEiIrqVRA7hqa2QZcB2tl8GfBQ4R9Km7VZWWikeZPpM25NtTx47dmzTmCMiop+O/05E0gbAW4E9+spsPw48XoavkXQ7sCOwFJhQW3xCKQO4T9I428vKaa/lnYg/IiL+rBstkTcAt9r+02kqSWMljSrDL6TqQF9cTletljSl9KMcBlxUFpsDTC/D02vlERHRIcN5ie+5wC+BnSQtkXREmTSNp3eovwa4oVzyez7wftt9nfIfAE4DFgG3Az8o5ScBb5R0G1ViOmm4tiUiIgY2bKezbB/SovxdA5RdQHXJ70DzLwB2HaB8JbDP2kUZERFrI79Yj4iIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaG84nG86StFzSTbWyT0laKum68jqgNu0YSYsk/VrSfrXyqaVskaSja+XbS7qylH9b0rOHa1siImJgw9kSOQOYOkD5F23vVl5zASTtTPXY3F3KMl+VNKo8d/0rwP7AzsAhZV6Az5e6XgQ8ABzRf0URETG8hi2J2L4cWDXkjJWDgNm2H7d9B9Xz1Pcsr0W2F9v+PTAbOEiSgNdTPY8d4Ezg4HUZf0REDK0bfSJHSbqhnO7aopSNB+6uzbOklLUq3wp40PYT/coHJGmGpAWSFqxYsWJdbUdExHqv00nkVGAHYDdgGXByJ1Zqe6btybYnjx07thOrjIhYL2zQyZXZvq9vWNLXge+X0aXAtrVZJ5QyWpSvBDaXtEFpjdTnj4iIDuloS0TSuNroW4C+K7fmANMkbSRpe2AScBVwNTCpXIn1bKrO9zm2DfwEeFtZfjpwUSe2ISIi/mzYWiKSzgX2BsZIWgIcB+wtaTfAwJ3A+wBsL5R0HnAz8ARwpO0nSz1HARcDo4BZtheWVXwCmC3pM8CvgNOHa1siImJgw5ZEbB8yQHHLL3rbJwInDlA+F5g7QPliqqu3IiKiS/KL9YiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIa6+gv1iNamb94JdNmzu92GEOaPWNKt0OIGFHSEomIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSGTiKSz2ymLiIj1TzstkV3qI5JGAXsMTzgREdFLWiYRScdIegh4iaTV5fUQsJw8RTAiIhgkidj+nO3RwBdsb1peo21vZfuYoSqWNEvSckk31cq+IOlWSTdIulDS5qV8oqTHJF1XXl+rLbOHpBslLZL0ZUkq5VtKmifptvJ3i7XZERERseaGvO2J7WMkjQdeUJ/f9uVDLHoG8F/AWbWyecAxtp+Q9HngGKrH3ALcbnu3Aeo5FXgvcCXVEw6nAj8AjgYutX2SpKPL+CcGWD4iekSv3P4GqlvgTHnhVt0Oo+uGTCKSTgKmUT3//MlSbGDQJGL7ckkT+5VdUhudD7xtiHWPAza1Pb+MnwUcTJVEDqJ6hjvAmcBlJIlERHRUOzdgfAuwk+3H1/G63w18uza+vaRfAauBT9q+AhgPLKnNs6SUAWxte1kZvhfYutWKJM0AZgBst9126yb6iIho6+qsxcCG63Klkv4FeAL4VilaBmxn+2XAR4FzJG3abn22TdU6ajV9pu3JtiePHTt2LSKPiIi6dloijwLXSboU+FNrxPY/NlmhpHcBBwL7lC9/Sivn8TJ8jaTbgR2BpcCE2uITShnAfZLG2V5WTnstbxJPREQ0104SmVNea03SVODjwGttP1orHwussv2kpBcCk4DFtleVS4unUHWsHwb8Zy2u6cBJ5W8uO46I6LB2rs46s0nFks6l6vgeI2kJcBzV1VgbAfPKlbrzbb8feA1wvKQ/AH8E3m97VanqA1RXem1M1aH+g1J+EnCepCOAu4C3N4kzIiKaa+fqrDsYoL/B9gsHW872IQMUn95i3guAC1pMWwDsOkD5SmCfwWKIiIjh1c7prMm14ecAfwdsOTzhRERELxny6izbK2uvpba/BLxp+EOLiIiRrp3TWbvXRp9F1TJppwUTERHPcO0kg5Nrw08Ad5JO7IiIoL2rs17XiUAiIqL3tPNQqs0knSJpQXmdLGmzTgQXEREjWzu3PZkFPER1CuvtVPe2+sZwBhUREb2hnT6RHWz/bW3805KuG6Z4IiKih7TTEnlM0qv6RiS9Enhs+EKKiIhe0U5L5B+AM2v9IA8A7xq2iCIiome0c3XWdcBL+27Nbnv1cAcVERG9oZ2rsz4raXPbq22vlrSFpM90IriIiBjZ2ukT2d/2g30jth8ADhi2iCIiome0k0RGSdqob0TSxlS3c4+IiPVcOx3r3wIuldT325DDgUbPGImIiGeWdjrWPy/peuANpegE2xcPb1gREdEL2robr+0fAj8c5lgiIqLHtNMn0pikWZKWS7qpVralpHmSbit/tyjlkvRlSYsk3VC/Bb2k6WX+2yRNr5XvIenGssyXVZ65GxERnTGsSYTq2ehT+5UdDVxqexJwaRkH2B+YVF4zgFOhSjpUz2ffC9gTOK4v8ZR53ltbrv+6IiJiGDVKIpK+3c58ti8HVvUrPog/d8yfCRxcKz/LlfnA5pLGAfsB82yvKpcXzwOmlmmb2p5v28BZtboiIqIDmrZEXrEW69za9rIyfC+wdRkeD9xdm29JKRusfMkA5U8jaUbfrexXrFixFqFHRETdcJ/OGlRpQbgD65lpe7LtyWPHjh3u1UVErDdaXp3V79nqT5kEbLgW67xP0jjby8opqeWlfCmwbW2+CaVsKbB3v/LLSvmEAeaPiIgOGewS35MHmXbrWqxzDjAdOKn8vahWfpSk2VSd6L8rieZi4LO1zvR9gWNsr5K0WtIU4ErgMOA/1yKuiIhYQy2TyLp4trqkc6laEWMkLaG6yuok4DxJRwB3UT0tEWAu1T25FgGPUv0ynpIsTgCuLvMdb7uvs/4DVFeAbQz8oLwiIqJD2vqxYZ2kNwIft/3Goea1fUiLSfsMMK+BI1vUM4vqMb39yxcAuw4VR0REDI+WHeuSXi/pN5IelvRNSX8laQFVS+LUzoUYEREj1WBXZ51M9aO/rYDzgV8CZ9jew/Z3OxFcRESMbIOdzrLty8rw9yQttf1fHYgpIiJ6xGBJZHNJb63PWx9PayQiIgZLIj8F3lwbv7w2biBJJCJiPTfYJb6HdzKQiIjoPYP9Yv2j/YoM3A/8zPYdwxpVRET0hMGuzhrd77UpMBn4gaRpHYgtIiJGuMFOZ316oPLyfI8fAbOHK6iIiOgNa3wX33LLkTxBMCIi1jyJSHod8MAwxBIRET1msI71G3n6sz62BO6humNuRESs5wb7nciB/cYNrLT9yDDGExERPWSwjvW7OhlIRET0nq4+HjciInpbkkhERDTW8SQiaSdJ19VeqyV9WNKnJC2tlR9QW+YYSYsk/VrSfrXyqaVskaSjO70tERHruzV+suHasv1rYDcASaOApcCFVI/D/aLt/6jPL2lnYBqwC7AN8CNJO5bJXwHeCCwBrpY0x/bNndiOiIjoQhLpZx/gdtt3SS1/v3gQMNv248AdkhYBe5Zpi2wvBpA0u8ybJBIR0SHd7hOZBpxbGz9K0g2SZknaopSNB+6uzbOklLUqfxpJMyQtkLRgxYoV6y76iIj1XNeSiKRnA38DfKcUnQrsQHWqaxnV43nXCdszbU+2PXns2LHrqtqIiPVeN09n7Q9ca/s+gL6/AJK+Dny/jC4Ftq0tN6GUMUh5RER0QDdPZx1C7VSWpHG1aW8BbirDc4BpkjaStD0wCbgKuBqYJGn70qqZVuaNiIgO6UpLRNLzqK6qel+t+N8l7UZ1e5U7+6bZXijpPKoO8yeAI20/Weo5CrgYGAXMsr2wU9sQERFdSiLl/ltb9Ss7dJD5TwROHKB8LjB3nQcYERFt6fbVWRER0cOSRCIiorFu/9gwIqJnzV+8kmkz53c7jLbcedKbhqXetEQiIqKxJJGIiGgsSSQiIhpLEomIiMbSsf4M1iudfrNnTOl2CBHRUFoiERHRWJJIREQ0liQSERGNJYlERERjSSIREdFYkkhERDSWJBIREY0liURERGNdSyKS7pR0o6TrJC0oZVtKmifptvJ3i1IuSV+WtEjSDZJ2r9Uzvcx/m6Tp3dqeiIj1UbdbIq+zvZvtyWX8aOBS25OAS8s4wP5Uz1afBMwAToUq6QDHAXsBewLH9SWeiIgYft1OIv0dBJxZhs8EDq6Vn+XKfGBzSeOA/YB5tlfZfgCYB0ztcMwREeutbiYRA5dIukbSjFK2te1lZfheYOsyPB64u7bsklLWqvwpJM2QtEDSghUrVqzLbYiIWK918waMr7K9VNLzgXmSbq1PtG1JXhcrsj0TmAkwefLkdVJnRER0sSVie2n5uxy4kKpP475ymoryd3mZfSmwbW3xCaWsVXlERHRAV5KIpOdJGt03DOwL3ATMAfqusJoOXFSG5wCHlau0pgC/K6e9Lgb2lbRF6VDft5RFREQHdOt01tbAhZL6YjjH9g8lXQ2cJ+kI4C7g7WX+ucABwCLgUeBwANurJJ0AXF3mO972qs5tRkTE+q0rScT2YuClA5SvBPYZoNzAkS3qmgXMWtcxRkTE0EbaJb4REdFDkkQiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGIiGgsSSQiIhrreBKRtK2kn0i6WdJCSR8q5Z+StFTSdeV1QG2ZYyQtkvRrSfvVyqeWskWSju70tkRErO+68WTDJ4CP2b62PGf9GknzyrQv2v6P+sySdgamAbsA2wA/krRjmfwV4I3AEuBqSXNs39yRrYiIiM4nEdvLgGVl+CFJtwDjB1nkIGC27ceBOyQtAvYs0xaVR+0iaXaZN0kkIqJDutonImki8DLgylJ0lKQbJM2StEUpGw/cXVtsSSlrVR4RER3StSQiaRPgAuDDtlcDpwI7ALtRtVROXofrmiFpgaQFK1asWFfVRkSs97qSRCRtSJVAvmX7uwC277P9pO0/Al/nz6eslgLb1hafUMpalT+N7Zm2J9uePHbs2HW7MRER67GO94lIEnA6cIvtU2rl40p/CcBbgJvK8BzgHEmnUHWsTwKuAgRMkrQ9VfKYBvyf4Yx9/uKVTJs5fzhXsc7MnjGl2yFExHqgG1dnvRI4FLhR0nWl7FjgEEm7AQbuBN4HYHuhpPOoOsyfAI60/SSApKOAi4FRwCzbCzu3GRER0Y2rs35G1Yrob+4gy5wInDhA+dzBlouIiOGVX6xHRERjSSIREdFYkkhERDSWJBIREY0liURERGNJIhER0ViSSERENJYkEhERjSWJREREY0kiERHRWJJIREQ0liQSERGNJYlERERjSSIREdFYkkhERDSWJBIREY0liURERGM9n0QkTZX0a0mLJB3d7XgiItYnPZ1EJI0CvgLsD+xM9Zz2nbsbVUTE+qOnkwiwJ7DI9mLbvwdmAwd1OaaIiPWGbHc7hsYkvQ2Yavs9ZfxQYC/bR/WbbwYwo4zuBPy6o4EObQxwf7eDaFMvxQq9FW8vxQq9FW8vxQojM94X2B7bv3CDbkTSabZnAjO7HUcrkhbYntztONrRS7FCb8XbS7FCb8XbS7FCb8Xb66ezlgLb1sYnlLKIiOiAXk8iVwOTJG0v6dnANGBOl2OKiFhv9PTpLNtPSDoKuBgYBcyyvbDLYTUxYk+1DaCXYoXeireXYoXeireXYoUeirenO9YjIqK7ev10VkREdFGSSERENJYk0kW9dMsWSbMkLZd0U7djGYqkbSX9RNLNkhZK+lC3YxqMpOdIukrS9SXeT3c7pqFIGiXpV5K+3+1YhiLpTkk3SrpO0oJuxzMYSZtLOl/SrZJukfSKbsc0lPSJdEm5ZctvgDcCS6iuNDvE9s1dDawFSa8BHgbOsr1rt+MZjKRxwDjb10oaDVwDHDyC962A59l+WNKGwM+AD9me3+XQWpL0UWAysKntA7sdz2Ak3QlMtj3Sfrz3NJLOBK6wfVq54vS5th/scliDSkuke3rqli22LwdWdTuOdtheZvvaMvwQcAswvrtRtebKw2V0w/IasUd3kiYAbwJO63YszySSNgNeA5wOYPv3Iz2BQJJIN40H7q6NL2EEf9H1KkkTgZcBV3Y5lEGV00PXAcuBebZHcrxfAj4O/LHLcbTLwCWSrim3QBqptgdWAN8opwpPk/S8bgc1lCSReMaStAlwAfBh26u7Hc9gbD9pezequy7sKWlEnjKUdCCw3PY13Y5lDbzK9u5Ud/s+spyaHYk2AHYHTrX9MuARYET3lUKSSDflli3DqPQtXAB8y/Z3ux1Pu8rpi58AU7scSiuvBP6m9DPMBl4v6ZvdDWlwtpeWv8uBC6lOJY9ES4AltVbo+VRJZURLEume3LJlmJSO6tOBW2yf0u14hiJprKTNy/DGVBdb3NrVoFqwfYztCbYnUn1mf2z7nV0OqyVJzysXV1BODe0LjMgrDG3fC9wtaadStA8wIi8Gqevp2570sl67ZYukc4G9gTGSlgDH2T69u1G19ErgUODG0s8AcKztud0LaVDjgDPLFXvPAs6zPeIvne0RWwMXVscVbACcY/uH3Q1pUB8EvlUOLBcDh3c5niHlEt+IiGgsp7MiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorEkkYhC0l9Imi3p9nKLjLmSduzg+t8laZsW015c7kL7K0k7SPpFG/U93KL8DElvW9t4IyBJJAL40w8ULwQus72D7T2AY6h+Z9DO8hsMNt6mdwEDJhHgYOB82y+zfbvtv25Qf8Q6lyQSUXkd8AfbX+srsH297StU+YKkm8pzKd4BIGlvSVdImgPcPMD4qLLc1ZJukPS+vrolfaLUdb2kk0rLYDLVD82uK79c75v3AODDwD9I+kkpe7g2/Z9r63jas0hK/P+l6tk1PwKeX5t2kqrnrtwg6T/W2d6M9UZ+sR5R2ZXquSMDeSuwG/BSYAxwtaTLy7TdgV1t3yFp737jM4Df2X65pI2An0u6BHgx1W3/97L9qKQtba8qdzD4J9tPeXCS7bmSvgY8bPspX/SS9gUmUd0PSsAcSa8pt+7v8xZgJ2BnqpbVzcAsSVuVaS+27b5br0SsiSSRiKG9CjjX9pPAfZJ+CrwcWA1cZfuO2rz18X2Bl9T6Hzaj+sJ/A/AN248C2F6b57TsW16/KuOblHXUk8hravHfI+nHpfx3wP8Cp6t6QmFutRJrLEkkorIQaNLZ/Mgg4wI+aPvi+gyS9muwnlYEfM72f6/pguX+bXtS3ejvbcBRwOvXYWyxHkifSETlx8BG9YcWSXqJpFcDVwDvKH0cY6mO7K9qo86LqfoxNiz17VjuJDsPOFzSc0v5lmX+h4DRaxj3xcC7y7NTkDRe0vP7zXN5Lf5xVP0/fc9b2azcmPIjVKfrItZIWiIRVI+olfQW4EuSPkF1mudOqg7tnwGvAK6nekrex23fK+nFQ1R7GjARuLZc/bWC6lnvP5S0G7BA0u+BucCxwBnA1yQ9BrzC9mNtxH2JpL8EflnuVPsw8E6qJyT2uZCqhXEz8Fvgl6V8NHCRpOdQtWg+OtT6IvrLXXwjIqKxnM6KiIjGkkQiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorEkkYiIaOz/A1bO4UUxLEl6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "success_by_url = scores.groupby(\"url\")[\"citoid_success\"].mean().sort_values().reset_index()\n",
    "\n",
    "bins = np.arange(8)\n",
    "ax = success_by_url[\"citoid_success\"].plot.hist(bins=bins)\n",
    "for x in np.arange(7):\n",
    "    ax.axvline(x, color=\"White\")\n",
    "\n",
    "xticks = (bins + 0.5)[:-1]\n",
    "\n",
    "ax.set_xticks(xticks)\n",
    "ax.set_xticklabels(bins[:-1])\n",
    "ax.set_title(\"Distribution of Citoid's correct answers by URL\")\n",
    "ax.set_ylabel(\"URL count\")\n",
    "ax.set_xlabel(\"Correct fields\")\n",
    "\n",
    "plt.savefig(\"plots/citoid_success_by_url.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The majority of the citations received ~4 correct answers of 6, followed by ~3 correct answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 By domain\n",
    "\n",
    "In this section, we present the results of Citoid's evaluation by domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores['domain'] = scores['url'].map(lambda x : urllib.parse.urlparse(x).netloc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 15501 different domains in our corpus\n"
     ]
    }
   ],
   "source": [
    "domain_counts = scores[\"domain\"].value_counts()\n",
    "print(f\"There are {len(domain_counts)} different domains in our corpus\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we group the citations having the same domain, calculate the average success score and plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgSUlEQVR4nO3debgcVbnv8e+PJBCGQMgAQhgCMp3AUZEAUZQDiIxRUBHDEQQE4/HiCF4RRAmT4PGA+DznKuYSIAQlckAkclFAZpQACYNIEA0hmARCQkJIwhx47x9rNXQ23V21N+ndvbN/n+fZz+5aVbXq7arqervWqqpWRGBmZtbIGq0OwMzM2p+ThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJ4t3SdJFkr6/iuraQtJySX3y8O2Sjl8Vdef6fi/p6FVVXyeWe7ak5yTN7+L8j0raq4vznirp4gbjZ0vatyt1W/uRtJekuU2q+xhJdzej7oLlNtyHu0vfVgfQziTNBjYGVgBvADOAy4HxEfEmQET8RyfqOj4i/lhvmoj4J7Deu4v6reWNA7aJiCOr6j9wVdTdyTi2AE4CtoyIBXWmWR84E/g0MAh4FvgdcHZEPBcRO1ZNO44O76uRiPhhJ2Idl+cZV3aedlVmf7OeoTP7cDP5zKLYJyJiALAlcB5wMjBhVS9E0uqauLcAFjVIFGsCtwA7AgcA6wMfAhYBu3VXkK1Wa/uvxvtEp3g9tImI8F+dP2A2sG+Hst2AN4Gd8vBlpG/AAEOA64ElwGLgLlJCnpTneRlYDnwHGA4EcBzwT+DOqrK+ub7bgXOB+4ClwHXAoDxuL2BurXhJB93XgNfz8h6uqu/4/HoN4DTgKWAB6YxpgzyuEsfRObbngO81WE8b5PkX5vpOy/Xvm9/zmzmOy2rMezzpTGK9ou3Q4H1tCkzJ63wm8KWqeccBV1QNH5VjXAR8r3ob52nHNdqWdeLbEbg5T/cscGouXwu4EHg6/10IrFW9/UhfPuaT9pFxwNXAFXl7H5/X7QTgGWAecDbQp2rZXwIeA5aRznw/SI39rUbMG+b3txB4Pr/erGr87cBZwJ9y3TcBQ/K4/jnGRXn93E86A98beKSqjpuB+6uG7wIOrdpm1+TlPwl8vcM267gedgOm5eFngQvqbIvKej2VtN/OBj6fx+2a561ef58m70c16hpM2q+Wkj6DZwF3V43/cH7vL+T/H+6w/s4G/py3we9yfb/M9d0PDK+a/qfAnDxuOvDRWvswnfxsrtLjYXcspKf+USNZ5PJ/Al/Jry/j7WRxLnAR0C//fRRQrbqqNvrlwLrA2tROFvOAnfI011TtNHtRJ1l03ME67MCVZPFF0oF1a1LT12+ASR1i+785rvcDrwL/Umc9XU5KZAPyvH8HjqsXZ4d5JwMTy26HOu/rTuBnpIPYB0gHoH06Tg+MIH1w9yQdyC8gNTHW2sZ1t2WH6QaQDuQn5eUPAHbP484EpgIbAUNJB46zqtbLCuBHOZa1c6yvA4eSku3awLXAL/L234h00PpyruOzef/YFRCwDam5b6V1VmedDgY+A6yTY/4f4Lcd9pUngO1yHLcD5+VxXyYd/NYB+gC7kM4I1wZeISXafqQD87xc/9qk5DU4v7fpwA+ANUn74Cxg/6pt1nE93AMclcevB4yq874q6/WCvF7/DXgR2D6PnwEcWDX9tcBJDfbNq/K63ym/l7vzuEGkJHsUqTn/iDw8uGr9zQTeS0r4M0ifi33z9JcDl1Yt68i8bvqS9qX5QP8a+/BwOvHZXJV/bobqmqdJO0tHrwObkD6wr0fEXZG3cAPjIuLFiHi5zvhJEfHXiHgR+D5weKUD/F36POnb2ayIWA6cAozpcMp/RkS8HBEPAw+TdsyV5FjGAKdExLKImA2cT/oQlTGYdLDtEkmbA3sAJ0fEKxHxEHAx8IUakx8GXB8Rd0bEq6T1+Wadqstuy9HA/Ig4Py9/WUTcm8d9HjgzIhZExELgDFZeL28Cp0fEq1Xb/56I+G2kPrH1gYOAb+Z9ZAHwE9L6hvSN+z8j4v5IZkbEU43XWBIRiyLimoh4KSKWAeeQDqzVLo2Iv+fYriIl4sq6GUzqO3ojIqZHxNI83f2kZLwLaZ/5E2n7jAL+ERGLSMltaEScGRGvRcQs0sFvTNWy31oPud7XgW0kDYmI5RExteAtfj+v1zuA/wccnssnkg7MSBoE7A/8quPMeb/+DPCDvO7/muetODi/n0kRsSIirgT+Bnyiw/p7IiJeAH4PPBERf4yIFaTkvHNlwoi4Im+TFRFxPinRbd/g/RV+Nlc1J4uuGUZqcujox6RvEzdJmiXpuyXqmtOJ8U+RvrENKRVlY5vm+qrr7ktqTqiovnrpJWp3vle+RXasa1jJOBaRDspdtSmwOB/wipa/KVXrMyfgRXXqLbstNyd9A68XW8f1smnV8MKIeKXDPNXbe0vSun1G0hJJS0hnGRuVWHZDktaR9AtJT0laSjo7G9jhi0i97T8JuBGYLOlpSf8pqV8edwfp2/2e+fXtpCT0b3m48r42rbyn/L5OZeV9r+Pn4jjSWc7fJN0vaXSDt/d83rYV1ev9CuATktYlJZC7IqLWl5WhpM9Dx89fRcdtWxlfvd89W/X65RrDb32eJH1b0mOSXsjrYwMaf87LfDZXKSeLTpK0K2mHeMcldPlb5UkRsTXwSeBESR+rjK5TZdGZx+ZVr7cgfcN6jnRqvU5VXH1IO3jZep8mfWir617Byjt0Gc/lmDrWNa/k/H8E9s8f3jI6vq+ngUGSBpRY/jNUrU9J65C+Ib9zIY23ZbU5pGaUWmqt46cbvJeOZXNITQxDImJg/ls/3r46bA6pmaPmW6hTXnES6Zvr7hGxPungDqk5q6F8pnVGRIwgtduP5u0zuY7J4g7emSzmAE9WvaeBETEgIg6qF39E/CMijiAlyh8BVzfYZzbsMO6t9R4R80hNWp8mneVNqlPHQtLnoePnr6Ljtq2ML7vfv0XSR0n9mIcDG0bEQFI/SOG26E5OFiVJWj9/m5lMaj98pMY0oyVtI0mkjf0GbzdzPEv9g0ojR0oakQ9sZwJXR8QbpPbP/pIOzt/qTiOdulY8CwyXVG8bXwl8S9JWktYDfgj8Op8il5ZjuQo4R9IASVsCJ5K+wZUxiXTwuEbSDpLWkDQ4X1t+UI3pV3pfETGH1BdwrqT+kt5H+hZaa/lXA6MlfSRfhXUmdT4DBduy2vXAJpK+KWmtvA52z+OuBE6TNFTSEFIbfdn1Qv7GexNwft7/1pD0XkmV5qKLgW9L2kXJNnn9V9ZTo/1tAOnb7ZLcHHN62bgk7S3pX/MXlKWkLwuVdfNnUhLaDbgvIh4lHVR3J529QOp3WSbpZElrS+ojaaf8RazeMo+UNDQ3zy3JxfWaEAHOkLRmPhCPJjX7VFxOOjj/K6mv7h3yfv0bYFw+CxtB6lSuuAHYTtK/S+or6XOkPrHrG8RUzwBSYloI9JX0A1ITZFtxsij2O0nLSAe075E6zo6tM+22pG/Ky0nfXn4WEbflceeSDhxLJH27E8ufROpEn0/qQP06QG4H/V+kA8Y80plG9c1IlQ/HIkkP1Kj3klz3naSrUV4BvtaJuKp9LS9/FumM61e5/kK572BfUnvvzbx95ckQ4N4as9R6X0eQOv6eJnVYnh417i/IB64TcnzPkDok693A1WhbVte5DPg4qa16PvAP0lVBkK6GmQb8BXgEeCCXdcYXSJ3AM3K8V5Ob7SLif0h9Db8iXbH0W97uSyva3y4kdZA+R+qE/0MnYnpPjmMp6UqsO8jf0HPzzwPAoxHxWp7+HuCp3OdSORCPJvWBPJljuJjU9FLPAcCjkpaTrhwa06Cfbz5pXT1NuvroPyLib1XjryUlsGsj4qUGy/wqqXlnPukzeGllRO57GU06Q1tESj6jI+K5BvXVcyNp/f+d1JT1CsXN092ucqWOmVmvIekJ0lVlvmmxJJ9ZmFmvIukzpD6RW1sdS0/iOyPNrNeQdDupb+Go3P9hJbkZyszMCrkZyszMCq2WzVBDhgyJ4cOHtzoMM7MeZfr06c9FxNBa41bLZDF8+HCmTZvW6jDMzHoUSXUfF+NmKDMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK7Ra3sFt1ttMnbWIMeOntjqMUiaPHcWorWv+mq21MZ9ZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVqjpyUJSH0kPSro+D28l6V5JMyX9WtKauXytPDwzjx9eVccpufxxSfs3O2YzM1tZd5xZfAN4rGr4R8BPImIb4HnguFx+HPB8Lv9Jng5JI4AxwI7AAcDPJPXphrjNzCxrarKQtBlwMHBxHhawD3B1nmQicGh+fUgeJo//WJ7+EGByRLwaEU8CM4Hdmhm3mZmtrNlnFhcC3wHezMODgSURsSIPzwWG5dfDgDkAefwLefq3ymvM8xZJYyVNkzRt4cKFq/htmJn1bk1LFpJGAwsiYnqzllEtIsZHxMiIGDl06NDuWKSZWa/Rt4l17wF8UtJBQH9gfeCnwEBJffPZw2bAvDz9PGBzYK6kvsAGwKKq8orqeczMrBs07cwiIk6JiM0iYjipg/rWiPg8cBtwWJ7saOC6/HpKHiaPvzUiIpePyVdLbQVsC9zXrLjNzOydmnlmUc/JwGRJZwMPAhNy+QRgkqSZwGJSgiEiHpV0FTADWAGcEBFvdH/YZma9V7cki4i4Hbg9v55FjauZIuIV4LN15j8HOKd5EZqZWSO+g9vMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAq14nEfZj3C1FmLGDN+aqvDKDR57KhWh2C9gM8szMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWaHCZCFpUpkyMzNbfZU5s9ixekBSH2CX5oRjZmbtqG6ykHSKpGXA+yQtzX/LgAXAdd0WoZmZtVzdZBER50bEAODHEbF+/hsQEYMj4pRujNHMzFqssBkqIk6RNEzShyXtWfkrmk9Sf0n3SXpY0qOSzsjlW0m6V9JMSb+WtGYuXysPz8zjh1fVdUouf1zS/u/i/ZqZWRf0LZpA0nnAGGAG8EYuDuDOgllfBfaJiOWS+gF3S/o9cCLwk4iYLOki4Djg5/n/8xGxjaQxwI+Az0kakZe/I7Ap8EdJ20XEG7UWamZmq15hsgA+BWwfEa92puKICGB5HuyX/wLYB/j3XD4RGEdKFofk1wBXA/8tSbl8cl7+k5JmArsB93QmHjMz67oyV0PNIh3oO01SH0kPkTrFbwaeAJZExIo8yVxgWH49DJgDkMe/AAyuLq8xT/WyxkqaJmnawoULuxKumZnVUebM4iXgIUm3kJqWAIiIrxfNmJuKPiBpIHAtsEMX4ywUEeOB8QAjR46MZi3HzKw3KpMspuS/LouIJZJuAz4EDJTUN589bAbMy5PNAzYH5krqC2wALKoqr6iex8zMukFhsoiIiV2pWNJQ4PWcKNYGPk7qtL4NOAyYDBzN2/dsTMnD9+Txt0ZESJoC/ErSBaQO7m2B+7oSk5mZdU2Zq6GeJHVMryQiti6YdRNgYr7jew3gqoi4XtIMYLKks4EHgQl5+gnApNyBvZh0BRQR8aikq0hXY60ATvCVUGZm3atMM9TIqtf9gc8Cg4pmioi/ADvXKJ9FupqpY/krue5adZ0DnFMiVjMza4IyN+UtqvqbFxEXAgc3PzQzM2sXZZqhPlg1uAbpTKPMGYmZma0myhz0z696vQKYDRzelGjMzKwtlbkaau/uCMTMzNpXmR8/2kDSBZW7oyWdL2mD7gjOzMzaQ5nHfVwCLCM1PR0OLAUubWZQZmbWXsr0Wbw3Ij5TNXxGft6TmZn1EmXOLF6W9JHKgKQ9gJebF5KZmbWbMmcWXyHdiV3pp3geOKZpEZmZWdspczXUQ8D7Ja2fh5c2OygzM2svZa6G+qGkgRGxNCKWStowP9fJzMx6iTJ9FgdGxJLKQEQ8DxzUtIjMzKztlEkWfSStVRnIjxtfq8H0Zma2minTwf1L4BZJlXsrjiX9draZmfUSZTq4fyTpYWDfXHRWRNzY3LDMzKydlHp6bET8AfhDk2MxM7M2VabPwszMejknCzMzK+RkYWZmhcr8Ut4ewDhgyzy9gIiIrZsbmpmZtYsyHdwTgG8B04E3mhuOmZm1ozLJ4oWI+H3TI7FeYeqsRYwZP7XVYRSaPHZUq0MwaytlksVtkn4M/AZ4tVIYEQ80LSozM2srZZLF7vn/yKqyAPZZ9eGYmVk7KnMH997dEYiZmbWvuslC0pERcYWkE2uNj4gLmheWmZm1k0ZnFuvm/wO6IxAzM2tfdZNFRPwi/z+j+8IxM7N2VOamvP7AccCOQP9KeUR8sYlxmdlqrCddQj1q68GtDqMtlHncxyTgPcD+wB3AZsCyZgZlZmbtpUyy2CYivg+8GBETgYN5+3JaMzPrBcoki9fz/yWSdgI2ADZqXkhmZtZuytyUN17ShsD3gSnAesAPmhqVmZm1lTI35V2cX94B+EmzZma9UJmroQYCXwCGV08fEV9vWlRmZtZWyjRD3QBMBR4B3mxuOGZm1o7KJIv+EVHzkR+NSNocuBzYmPTgwfER8VNJg4Bfk85UZgOHR8TzkgT8FDgIeAk4pvJkW0lHA6flqs/OV2WZmVk3KXWfhaQvSdpE0qDKX4n5VgAnRcQIYBRwgqQRwHeBWyJiW+CWPAxwILBt/hsL/BwgL+t00uW6uwGn5w53MzPrJmWSxWvAj4F7SL+WNx2YVjRTRDxTOTOIiGXAY8Aw4BCgcmYwETg0vz4EuDySqcBASZuQbga8OSIWR8TzwM3AAeXenpmZrQplmqFOIt2Y91xXFyJpOLAzcC+wcUQ8k0fNJzVTQUokc6pmm5vL6pV3XMZY0hkJW2yxRVdDNTOzGsqcWcwk9SF0iaT1gGuAb0bE0upxERGk/ox3LSLGR8TIiBg5dOjQVVGlmZllZc4sXgQeknQbK/+sauGls5L6kRLFLyPiN7n4WUmbRMQzuZlpQS6fB2xeNftmuWwesFeH8ttLxG1mZqtImTOL3wLnAH/m7T6L6UUz5aubJgCPdfihpCnA0fn10cB1VeVfUDIKeCE3V90I7Cdpw9yxvV8uMzOzblLmDu6JktYEtstFj0fE643myfYAjgIekfRQLjsVOA+4StJxwFPA4XncDaTLZivNXsfm5S+WdBZwf57uzIhYXGL5Zma2ipS5g3sv0lVLswEBm0s6OiLubDRfRNydp6/lYzWmD+CEOnVdAlxSFKuZmTVHmT6L84H9IuJxAEnbAVcCuzQzMDMzax9l+iz6VRIFQET8HejXvJDMzKzdlDmzmCbpYuCKPPx5StyUZ2Zmq48yyeIrpL6EyqWydwE/a1pEZmbWdspcDfWqpEnApIhY2A0xmZlZm6nbZ5Hvdxgn6TngceBxSQsl+VfyzMx6mUYd3N8i3Suxa0QMiohBpCe/7iHpW90SnZmZtYVGyeIo4IiIeLJSEBGzgCNJv5xnZma9RKNk0a/Wk2Zzv4UvnTUz60UaJYvXujjOzMxWM42uhnq/pKU1ygX0b1I8ZmbWhuomi4jo052BmJlZ+yrzuA8zM+vlnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr1OiX8szMer2psxYxZvzUVodR2uzzDm5KvT6zMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrFDT7rOQdAkwGlgQETvlskHAr4HhwGzg8Ih4XpKAnwIHAS8Bx0TEA3meo4HTcrVnR8TEZsXcE/Wka8Anjx3V6hDMrIuaeWZxGXBAh7LvArdExLbALXkY4EBg2/w3Fvg5vJVcTgd2B3YDTpe0YRNjNjOzGpqWLCLiTmBxh+JDgMqZwUTg0KryyyOZCgyUtAmwP3BzRCyOiOeBm3lnAjIzsybr7j6LjSPimfx6PrBxfj0MmFM13dxcVq/8HSSNlTRN0rSFCxeu2qjNzHq5lnVwR0QAsQrrGx8RIyNi5NChQ1dVtWZmRvcni2dz8xL5/4JcPg/YvGq6zXJZvXIzM+tG3Z0spgBH59dHA9dVlX9BySjghdxcdSOwn6QNc8f2frnMzMy6UTMvnb0S2AsYImku6aqm84CrJB0HPAUcnie/gXTZ7EzSpbPHAkTEYklnAffn6c6MiI6d5mZm1mRNSxYRcUSdUR+rMW0AJ9Sp5xLgklUYmpmZdZLv4DYzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVqhvqwNoR1NnLWLM+KmtDqPQ5LGjWh2CmfUSPrMwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCPSZZSDpA0uOSZkr6bqvjMTPrTXpEspDUB/g/wIHACOAISSNaG5WZWe/RI5IFsBswMyJmRcRrwGTgkBbHZGbWaygiWh1DIUmHAQdExPF5+Chg94j4atU0Y4GxeXB74PFuD7SxIcBzrQ6iE3pSvD0pVuhZ8fakWKFnxduOsW4ZEUNrjVhtnjobEeOB8a2Oox5J0yJiZKvjKKsnxduTYoWeFW9PihV6Vrw9KVboOc1Q84DNq4Y3y2VmZtYNekqyuB/YVtJWktYExgBTWhyTmVmv0SOaoSJihaSvAjcCfYBLIuLRFofVWW3bRFZHT4q3J8UKPSvenhQr9Kx4e1KsPaOD28zMWqunNEOZmVkLOVmYmVkhJ4tu0JMeVSLpEkkLJP211bEUkbS5pNskzZD0qKRvtDqmeiT1l3SfpIdzrGe0OqYyJPWR9KCk61sdSyOSZkt6RNJDkqa1Op4ikgZKulrS3yQ9JulDrY6piPssmiw/quTvwMeBuaQru46IiBktDawOSXsCy4HLI2KnVsfTiKRNgE0i4gFJA4DpwKHtuG4lCVg3IpZL6gfcDXwjIqa2OLSGJJ0IjATWj4jRrY6nHkmzgZER0W43udUkaSJwV0RcnK/wXCcilrQ4rIZ8ZtF8PepRJRFxJ7C41XGUERHPRMQD+fUy4DFgWGujqi2S5XmwX/5r629qkjYDDgYubnUsqxNJGwB7AhMAIuK1dk8U4GTRHYYBc6qG59KmB7SeTNJwYGfg3haHUldu0nkIWADcHBFtG2t2IfAd4M0Wx1FGADdJmp4f/dPOtgIWApfmJr6LJa3b6qCKOFlYjydpPeAa4JsRsbTV8dQTEW9ExAdITyDYTVLbNvNJGg0siIjprY6lpI9ExAdJT6Y+ITentqu+wAeBn0fEzsCLQFv3ZYKTRXfwo0qaKLf/XwP8MiJ+0+p4yshNDrcBB7Q4lEb2AD6Z+wImA/tIuqK1IdUXEfPy/wXAtaTm33Y1F5hbdWZ5NSl5tDUni+bzo0qaJHcaTwAei4gLWh1PI5KGShqYX69NuuDhby0NqoGIOCUiNouI4aR99taIOLLFYdUkad18gQO5OWc/oG2v5ouI+cAcSdvnoo8BbXdRRkc94nEfPVlPe1SJpCuBvYAhkuYCp0fEhNZGVdcewFHAI7kvAODUiLihdSHVtQkwMV8dtwZwVUS09eWoPcjGwLXpuwN9gV9FxB9aG1KhrwG/zF8gZwHHtjieQr501szMCrkZyszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4X1KpLeI2mypCfyoyFukLRdNy7/GEmb1hm3Q35q6oOS3ivpzyXqW16n/DJJh73beM0qnCys18g38V0L3B4R742IXYBTSNfpl5m/b6Phko4BaiYL4FDg6ojYOSKeiIgPd6F+s6ZwsrDeZG/g9Yi4qFIQEQ9HxF1Kfizpr/l3ET4HIGkvSXdJmgLMqDHcJ893v6S/SPpypW5JJ+e6HpZ0Xv6mP5J0M9ZD+U7uyrQHAd8EviLptly2vGr8/65axjt+CyPH/99Kv5vyR2CjqnHnKf3mx18k/dcqW5vWq/gObutNdiL95kUtnwY+ALwfGALcL+nOPO6DwE4R8aSkvToMjwVeiIhdJa0F/EnSTcAOpEfR7x4RL0kaFBGL8938346IlX6gJyJukHQRsDwiVjqgS9oP2Jb0vCMBUyTtmR8nX/EpYHtgBOlMaQZwiaTBedwOERGVR46YdZaThVnyEeDKiHgDeFbSHcCuwFLgvoh4smra6uH9gPdV9Q9sQDqw7wtcGhEvAUTEu/mNkP3y34N5eL28jOpksWdV/E9LujWXvwC8AkxQ+rU7P2LEusTJwnqTR4GudPq+2GBYwNci4sbqCSTt34Xl1CPg3Ij4RWdnzM8m2430sLrDgK8C+6zC2KyXcJ+F9Sa3AmtV/ziOpPdJ+ihwF/C53AcxlPRN/b4Sdd5I6mfol+vbLj/59GbgWEnr5PJBefplwIBOxn0j8MX8ux1IGiZpow7T3FkV/yak/pnKb31skB+u+C1SM5tZp/nMwnqN3Gb/KeBCSSeTmmdmkzqW7wY+BDxM+tW170TEfEk7FFR7MTAceCBfbbWQ9Dvgf5D0AWCapNeAG4BTgcuAiyS9DHwoIl4uEfdNkv4FuCc/WXU5cCTpF/cqriWdMcwA/gnck8sHANdJ6k86QzmxaHlmtfips2ZmVsjNUGZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRX6/wVErZhGaetWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "success_by_domain = scores.groupby(\"domain\")[\"citoid_success\"].mean().sort_values().reset_index()\n",
    "\n",
    "bins = np.arange(8)\n",
    "ax = success_by_domain[\"citoid_success\"].plot.hist(bins=bins)\n",
    "for x in np.arange(7):\n",
    "    ax.axvline(x, color=\"White\")\n",
    "\n",
    "xticks = (bins + 0.5)[:-1]\n",
    "\n",
    "ax.set_xticks(xticks)\n",
    "ax.set_xticklabels(bins[:-1])\n",
    "ax.set_title(\"Distribution of Citoid's correct answers by domain\")\n",
    "ax.set_ylabel(\"Domain count\")\n",
    "ax.set_xlabel(\"Correct fields\")\n",
    "\n",
    "plt.savefig(\"plots/citoid_success_by_domain.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As for the scoring by URL, on average, domains receive ~4 correct answers of 6, followed by ~3 correct answers.\n",
    "\n",
    "We found 15.5kk different domains in our corpus. Here is the top 10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7ZklEQVR4nO3debheVXn38e+PAGGUQaiNiEYmFUUiBBQFBLRWiwWqIioqIBWnMkjRl7ZK0doWoQJOqEgVFRxAoVJxAAUhjQokhAxMDiSKiIgCYY4Q7vePvQ48HM5JziFPcpLw/VzXc52911p7rXuv5CS5s9beJ1WFJEmSJGnJrTLWAUiSJEnSysIES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSpGUkyS5Jru9jf8cmOWMU7SvJFv0af4RjPj3J3UnGLctxJWmsmGBJkpZ77R/oA5+HktzXc75/n8Z4fZKfJLk3yY+HqJ+UZHqrn55k0mjHqKopVfWsnj7nJXn5kkW+fKuq31TVOlW1cKxjkaRlwQRLkrTca/9AX6eq1gF+A/xtT9mZfRrmNuBk4LjBFUlWB74NnAFsAHwJ+HYrlyTpYSZYkqQVVpLxSU5O8rv2OTnJ+Fa3W5LfJvnnJH9sq0XDrnZV1Q+r6izgd0NU7wasCpxcVQuq6hNAgD2GiWvDJF9sMd2e5H96Y2rHXwGeDvxvW4l7fyt/UVtJuyPJzCS79fT7zCSXJLkryYXARouZn/clubnF8bZBdesl+XKSW5P8OskHkqzS6g5MMjXJSS2OG5K8uJXfmOQPSQ7o6WvPJDOS3Nnqj+2pm9i2Jq7azn+c5N9a/3cluSDJRq1ujSRnJPlTG/eKJE9Z1D1K0vLGBEuStCL7F+BFwCRgW2BH4AM99X9Jl4RsAhwAnJrkWYzec4FZVVU9ZbNa+VC+AqzV6v8COGlwg6p6C49ejTs+ySbA+cBHgA2Bo4BvJdm4XfZVYHq7p39r9zSkJK9s1/8VsCUweCviJ4H1gM2AlwJvBQ7qqX9hu8cnt3G/DuwAbAG8GfhUknVa23va9esDewLvSrLPcLEBb2pj/QWweouTdj/rAZu2cd8J3LeIfiRpuWOCJUlake0PfLiq/lBVtwIfAt4yqM0H26rTJXTJy+sfxzjrAPMHlc0H1h3cMMkE4FXAO6vq9qp6oI09Em8GvltV362qh6rqQmAa8DdJnk6X4Azcz6XA/y6ir9cDX6yqOVV1D3BsT4zjgDcA/1RVd1XVPOBjPHru5lbVF9uzU9+gS3o+3Ma+APgzXbJFVf24qma3mGcBX6NL2obzxar6eVXdB5xFlyADPECXWG1RVQuranpV3bm4SZOk5YkJliRpRfZU4Nc9579uZQNub8nFcPUjdTfwpEFlTwLuGqLtpsBtVXX74xjnGcC+bXvcHUnuAHYGJtDFPdT9DOepwI3DtN0IWI3Hzt0mPee39BzfB1BVg8vWAUjywiQXt+2G8+lWnha1ffH3Pcf3DvRDt/L3A+DrbVvj8UlWW0Q/krTcMcGSJK3IfkeXlAx4Oo9+hmqDJGsvon6krgaenyQ9Zc9v5YPdCGyYZP0R9FuDzm8EvlJV6/d81q6q44CbGfp+hnMzXbI3VNs/0q0WDZ67m0YQ81C+CpwHbFpV6wGfpXtGbVTaat+Hqmpr4MXAq+m2HkrSCsMES5K0Ivsa8IEkG7cXJRxD96a/Xh9KsnqSXej+wX72UB0lGZdkDbqXWazSXrgwsHryY2AhcFh7scY/tPKLBvdTVTcD3wNOSbJBktWS7DpM/LfQPQM14Azgb5P89UA87cUYT6uqX9NtFxy4n52Bv13E3JwFHJhk6yRrAf/aE+PCVv/vSdZN8gzgSB47dyO1Lt2q3f1JdqR7xmrUkuyeZJu2hfFOuiTwoccZkySNCRMsSdKK7CN0SccsYDZwZSsb8HvgdrpVqzPpnou6bpi+3kK37e0zwC7t+PMAVfVnYB+61ZQ7gLcB+7Ty4fp6ALgO+ANwxDDt/pMuQbwjyVFVdSOwN/DPwK10K1rv45G/r99E9/KJ2+gSpi8P0y9V9T26185fBPySxyaDh9K9nOIG4P/oVqG+MFx/i/Fu4MNJ7qJLcs96nP38JfBNuuTqWuASum2DkrTCyKNfiCRJ0sqhvd78jKp62hiHIkl6AnEFS5IkSZL6xARLkiRJkvrELYKSJEmS1CeuYEmSJElSn6w61gFo+bPRRhvVxIkTxzoMSZIkabk1ffr0P1bVxoPLTbD0GBMnTmTatGljHYYkSZK03Ery66HK3SIoSZIkSX1igiVJkiRJfWKCJUmSJEl9YoIlSZIkSX3iSy70GLNvms/Eo88f6zAkSZKkR5l33J5jHcJiuYIlSZIkSX2y3CVYSSYmmdOHfuYl2agfMfVbkh8nmTzWcUiSJEnqr+UuwZIkSZKkFdXymmCtmuTMJNcm+WaStZK8LMmMJLOTfCHJeIDhygckWTPJ95K8PcnaSc5PMjPJnCT7DR44yeZJftb6+0iSu1t5kpzQrps9cO0iyldJckqS65JcmOS7SV43xHivSPLTJFcmOTvJOkO0eUqSc1vcM5O8uJUf2cadk+SIVjaxjXl6kp+3eXx5kqlJfpFkxyX+1ZEkSZI0pOU1wXoWcEpVPQe4EzgSOB3Yr6q2oXs5x7uSrDFUeU8/6wD/C3ytqj4PvBL4XVVtW1XPA74/xNgfBz7e+vttT/lrgEnAtsDLgROSTFhM+URga+AtwE6DB2pbGD8AvLyqtgOmtXsd7BPAJVW1LbAdcHWS7YGDgBcCLwLenuQFrf0WwMeAZ7fPm4CdgaOAfx6if5IckmRakmkL750/VBNJkiRJi7G8Jlg3VtXUdnwG8DJgblX9vJV9CdiVLhEbqnzAt4EvVtWX2/ls4K+SfDTJLlU1VCaxE3B2O/5qT/nOdInawqq6BbgE2GEx5WdX1UNV9Xvg4iHGehFdAjY1yVXAAcAzhmi3B/AZgDbO/Nb/uVV1T1XdDZwD7NLaz62q2VX1EHA18KOqqnb/E4fon6o6taomV9XkcWutN1QTSZIkSYuxvCZYNej8jsfZz1TglUkC0BKx7egSjY8kOeZxR9gfAS6sqknts3VVHdyHfhf0HD/Uc/4QvppfkiRJWmqW1wTr6UkGttS9iW7r3MQkW7Syt9CtFF0/TPmAY4DbgU8DJHkqcG9VnQGcQJdsDfYz4LXt+A095VOA/ZKMS7Ix3UrZ5Ysonwq8tj2L9RRgt2HGeslA/O0Zsa2GaPcj2tbHNs56bdx92vNpawN/18okSZIkjZHlNcG6HnhPkmuBDYCT6J43OjvJbLqVmM9W1f1DlQ/q63BgzSTHA9sAl7fteP8KfAQgyYeT7NXaHwEcmWQW3bNMA9sIzwVmATOBi4D3t61/w5V/i+4Zrmvotjle2dMXAFV1K3Ag8LU23k/pnpkaHNPhwO7tHqcDW1fVlXTPn10OXAacVlUzRji/kiRJkpaCdI/maECStYD7qqqSvAF4Y1Xt/Tj7Wqeq7k7yZLpE6CUt+VqujZ+wZU044OSxDkOSJEl6lHnH7TnWITwsyfSqeszPtvV5nMfaHvhUe27rDuBtS9DXd5KsD6wO/NuKkFwBbLPJekxbjn7zSpIkSSsKE6xBqmoK3SvX+9HXbv3oR5IkSdKKYXl9BkuSJEmSVjgmWJIkSZLUJyZYkiRJktQnJliSJEmS1CcmWJIkSZLUJyZYkiRJktQnJliSJEmS1CcmWJIkSZLUJyZYkiRJktQnq451AFr+zL5pPhOPPn+sw5BWSvOO23OsQ5AkSUuRK1iSJEmS1CcmWCOQZP0k7+45f2qSb45lTJIkSZKWPyZYI7M+8HCCVVW/q6rXjV04kiRJkpZHy1WCleR9SQ5rxycluagd75HkniQntvPDk9zQjjdLMjXJDknOaWV7J7kvyepJ1hho2zPOgUnOSfL9JL9Icnwrf1uSk3vavT3JScBxwOZJrkpyQpKJSeb09PU/SS5MMi/JPyQ5MsmMJD9LsmFrt3kbb3qSKUme3cr3TTInycwklw4zL29NMqu1+Uorm5jkolb+oyRPb+WnJ/lMG/uGJLsl+UKSa5Oc3p9fKUmSJElDWa4SLGAKsEs7ngysk2S1VnZ8T90uwJ+SbNKOLwVmAJN66ucAOwAvBC4bYqxJwH7ANsB+STYFzgL+to0JcBDwBeBo4FdVNamq3jdEX88DXtPG+3fg3qp6AfBT4K2tzanAoVW1PXAUcEorPwb466raFthrcMdJngt8ANijtTm8VX0S+FJVPR84E/hEz2UbADsB7wXOA04Cngtsk2QSQ0hySJJpSaYtvHf+UE0kSZIkLcbylmBNB7ZP8iRgAV2CMpkuYTqbLuFaF9gU+Cqwa6ubUlUPAr9K8hxgR+DE3vohxvpRVc2vqvuBa4BnVNXdwEXAq9sK02pVNXsEcV9cVXdV1a3AfOB/W/lsYGKSdYAXA2cnuQr4HDChtZkKnJ7k7cC4IfreAzi7qv4IUFW3tfKd2hwAfAXYueea/62qauPfUlWzq+oh4Gpg4lA3UFWnVtXkqpo8bq31RnDLkiRJkgZbrl7TXlUPJJkLHAj8BJgF7A5sAVzbyg4CrqdLmt5Gl2j8Y+viUuBVwAPAD4HT6ZKWoVadFvQcL+SRuTgN+GfgOuCLIwy9t6+Hes4fav2uAtxRVZMGX1hV70zyQmBPYHqS7avqTyMcd3Hx9MbSG48kSZKkpWB5W8GCLnE6ii5ZmgK8E5jRVmR662bQJV8Lqmp+z7VHAD9tq0lPBp5Ft11wRKrqMroVsjcBX2vFdwHrPt4bqqo7gblJ9gVIZ9t2vHlVXVZVxwC3trF7XQTsm+TJrf2GrfwnwBva8f4MvUonSZIkaRlaXhOsCXRJ0i3A/TySPEyhS0AuraqFwI3A//VcexnwFLoEDLoVsNlVVUn2SvLhEcZwFjC1qm4HaCtKU9vLKE54nPe1P3Bwkpl0W/X2buUnJJndXprxE2Bmew38d9vYV9M913VJu/bEdt2hwEFJZgFv4ZFnsyRJkiSNkXQLQ+qV5DvASVX1o7GOZSyMn7BlTTjg5LEOQ1opzTtuz7EOQZIk9UGS6VU1eXC5z+P0SLI+cDkw84maXAFss8l6TPMfgZIkSdKomWD1qKo7gK3GOg5JkiRJK6bl8RksSZIkSVohmWBJkiRJUp+YYEmSJElSn5hgSZIkSVKfmGBJkiRJUp+YYEmSJElSn5hgSZIkSVKfmGBJkiRJUp+YYEmSJElSn6w61gFo+TP7pvlMPPr8sQ5DK4F5x+051iFIkiQtU65gSZIkSVKfmGAtgSR3D1P+4ySTl3U8w1ne4pEkSZJWViZYkiRJktQnK02ClWRikmuTfD7J1UkuSLJmks2TfD/J9CRTkjw7ybgkc9NZP8nCJLu2fi5NsmWSlya5qn1mJFl3mHFPauP9KMnGPVVvadfOSbJja7tOki8mmZ1kVpLXDtHfFkl+mGRmkitb/ElyQutrdpL9hrn/OT3nRyU5dlCbVZKcnuQjj2+WJUmSJC3KSpNgNVsCn66q5wJ3AK8FTgUOrartgaOAU6pqIXA9sDWwM3AlsEuS8cCmVfWL1vY9VTUJ2AW4b4jx1gamtfEuAf61p26tdu27gS+0sg8C86tqm6p6PnDREH2e2e5hW+DFwM3Aa4BJwLbAy4ETkkwY3dSwauv7F1X1gcGVSQ5JMi3JtIX3zh9l15IkSZJg5Uuw5lbVVe14OjCRLkk5O8lVwOeAgcRkCrBr+/wnXaK1A3BFq58KnJjkMGD9qnpwiPEeAr7Rjs9ofQz4GkBVXQo8Kcn6dMnRpwcaVNXtvZ21VbJNqurcVn9/Vd3b+v1aVS2sqlvokrkdRjQjj/gcMKeq/n2oyqo6taomV9XkcWutN8quJUmSJMHKl2At6DleCGwI3FFVk3o+z2n1l9KtTO0IfBdYH9iNLvGiqo4D/h5YE5ia5NkjGL+GOR7qvN8e5NG/nmsMqv8JsHuSweWSJEmS+mRlS7AGuxOYm2RfgPYs07at7nK61a2Hqup+4CrgHXSJF0k2r6rZVfVRulWtoRKsVYDXteM3Af/XU7df62dnum2B84ELgfcMNEiyQW9nVXUX8Nsk+7T68UnWokv69mvPjm1Mt+p2+aBYbgH+IsmT21bHVw+q/2+6RPKsJP78M0mSJGkpWNkTLID9gYOTzASuBvYGqKoFwI3Az1q7KcC6wOx2fkR7qcQs4AHgewBtq+GAe4Ad28sl9gA+3FN3f5IZwGeBg1vZR4ANWr8zgd1bn6f1vEb9LcBhbdyfAH8JnAvMAmbSPbf1/qr6fZKnJvluu58H2viX0yVy1w2eiKo6EZgBfCXJE+HXXpIkSVqmUrW0d65pRTN+wpY14YCTxzoMrQTmHbfnWIcgSZK0VCSZXlWP+VmzbhXTY2yzyXpM8x/GkiRJ0qi5TUySJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSJEmS+mTVsQ5Ay5/ZN81n4tHnj3UYWgnMO27PsQ5BkiRpmXIFS5IkSZL65AmXYCWZl2SjsRgjyV5Jjl6aY0uSJEkaO24RfJySrFpVD47mmqo6DzhvKYX0KI8nPkmSJElLZoVdwUryviSHteOTklzUjvdIcmaSVyT5aZIrk5ydZJ2ey9+fZHaSy5NsMUTfE5NMaddemeTFrXy3Vn4ecE2ScUn+K8mcJLOSHNrTzaHt2tlJnt2uPzDJp5Ksl+TXSVZp5WsnuTHJakk2T/L9JNPbWM8eIr41knyx9T0jye49/Z/X5uJHSdZKclaSa5Kcm+SyJJP78ysgSZIkabAVNsECpgC7tOPJwDpJVmtls4APAC+vqu2AacCRPdfOr6ptgE8BJw/R9x+Av2rX7gd8oqduO+DwqtoKOASYCEyqqucDZ/a0+2O7/jPAUb2dV9V84Crgpa3o1cAPquoB4FTg0Kravl13yhDxvafrprYB3gh8KckaPfG9rqpeCrwbuL2qtgY+CGw/RF8AJDkkybQk0xbeO3+4ZpIkSZIWYUXeIjgd2D7Jk4AFwJV0idYudNvwtgamJgFYHfhpz7Vf6/l60hB9rwZ8KskkYCGwVU/d5VU1tx2/HPjswFa8qrqtp905PXG+ZogxvkGXvF0MvAE4pa2yvRg4u8UNMH6Ia3cGPtnGvC7Jr3tivLAnjp2Bj7d2c5LMGqIvWv2pdMkd4ydsWcO1kyRJkjS8FTbBqqoHkswFDgR+QrdqtTuwBTCXLtF443CXD3M84L3ALcC2dKt89/fU3TPCEBe0rwsZep7PA/4jyYZ0K0sXAWsDd1TVpBGOMZSRxidJkiSpz1bkLYLQbRM8Cri0Hb8TmAH8DHjJwPNV7Rmn3lWo/Xq+9q5sDVgPuLmqHgLeAowbZvwLgXckWbWNs+FIA6+qu4Er6FaYvlNVC6vqTmBukn1bf0my7RCXTwH2b222Ap4OXD9Eu6nA61u7rYFtRhqfJEmSpNFbGRKsCcBPq+oWupWmKVV1K93K1tfatrifAr0vi9iglR9Ot1o18Ar1D7f6U4ADksxs1w23KnQa8BtgVmv7plHG/w3gze3rgP2Bg1t/VwN7DxPfKklmt2sPrKoFPNYpwMZJrgE+0vrzAStJkiRpKUmVj9usrJKMA1arqvuTbA78EHhWVf15UdeNn7BlTTjg5GURolZy847bc6xDkCRJWiqSTK+qx7yhe4V9BksjshZwcXu7YoB3Ly65Athmk/WY5j+MJUmSpFEzwVqJVdVddG9WlCRJkrQMrOjPYEmSJEnScsMES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvpk1ZE0SvIS4FjgGe2aAFVVmy290DRWZt80n4lHnz/WYaz05h2351iHIEmSpD4bUYIF/DfwXmA6sHDphSNJkiRJK66RbhGcX1Xfq6o/VNWfBj5LNbIRSjIpyd+MoN1uSb6zLGIaZvxjkxw1RmPPS7LRWIwtSZIkPZGMdAXr4iQnAOcACwYKq+rKpRLV6EwCJgPfXVYDJhlXVa7kSZIkSXqUka5gvZAuifkP4GPt81+LuiDJ+5Ic1o5PSnJRO94jyT1JTmznhye5oR1vlmRqkh2SnNPK9k5yX5LVk6wx0LbVrQ58GNgvyVVJ9kuydpIvJLk8yYwkew8R25BtkqyV5Kwk1yQ5N8llSSa3uruTfCzJTGCnJMckuSLJnCSnJklr9+MkH2/xzEmyY8/QW7f6GwbmZojYdkjykyQzW3zrtvv+YpLZLd7dh7juUSt0ST6V5MBBbdZM8r0kb1/Ur50kSZKkx2dEK1hV9Zh/0I/AFOAfgU/QJWfjk6wG7AIcD7y6tdsF+FOSTdrxpcAMupWpgfo5wA4t3st64vpzkmOAyVX1DwBJ/gO4qKrelmR94PIkPxwU278M0+ZdwO1VtXWS5wFX9VyzNnBZVf1jG+eaqvpwO/5Ku5//bW3XqqpJSXYFvgA8r5U/G9gdWBe4PslnquqBgQFawvgNYL+quiLJk4D7gMO7261tkjwbuCDJVlV1/6J+AQZZB/g68OWq+vLgyiSHAIcAjHvSxqPoVpIkSdKARSZYSd5cVWckOXKo+qo6cRGXTwe2b0nCAuBKukRrF+Aw4A1J1gU2Bb4K7NrqzqmqB5P8KslzgB2BE1v9OLrEbVFeAezV87zTGsDTR9hmZ+Dj7d7mJJnVc81C4Fs957sneT+wFrAhcDWPJFhfa31cmuRJLYkDOL+qFgALkvwBeArw254+nwXcXFVXtOvvBEiyM/DJVnZdkl8DWwG98S3Ot4Hjq+rMoSqr6lTgVIDxE7asUfQrSZIkqVncCtba7eu6o+24qh5IMhc4EPgJXTKwO7AFcG0rOwi4ni5pehuwE92qF3QrWa8CHgB+CJxOl2C9bzFDB3htVV3/qMLkKSNos6h+7x947irJGsApdCtnNyY5li5Je/j2B107cL6gp2whI38GbiQe5NFbPtcYVD8VeGWSr1aVCZQkSZK0FCzyGayq+lz7+qGhPiPofwpwFF2yNAV4JzCj/QO/t24GXfK1oKrm91x7BPDTqroVeDLdCs+cQWPcxaMTwB8Ah/Y8E/WCIeIars1U4PWtbGtgm2HuayB5+WOSdYDXDarfr/WxM90bGOczMtcDE5Ls0K5fN8mqdHOxfyvbim617fpB1/6a7hmv8W3F7GWD6o8Bbgc+PcJYJEmSJI3SiF5y0V6y8J4kp7SXQ3whyRdGcOkUYAJdknQLcD+PbPGbQrc98NK2MnQj8H89115Gt4Xu0nY+C5hdVZVkryQfbuUX0yUWVyXZD/g3YDVgVpKr2/lgw7U5Bdg4yTXAR+i2/T0mOaqqO4DP0yV7PwCuGNTk/iQzgM8CBy9ifgBI8t0kT62qP9MlZ59sL9O4kC6ZOwVYJclsume0DqyqBUkmJzmtxXQjcFaL6Sy6pHWww4E1kxy/uJgkSZIkjV5GslssydnAdcCb6N7atz9wbVUdvnTDW7aSjANWq6r7k2xOtzXxWS3xGWkfPwaOqqppSynMpW78hC1rwgEnj3UYK715x+051iFIkiTpcUoyvaomDy4f6TNAW1TVvkn2rqovJfkqi3/ZxIpoLbqf+bUa3XNa7x5NcrWy2GaT9ZjmP/4lSZKkURtpgjXwKvE72uvLfw/8xdIJaexU1V10bzpckj526080kiRJklY0I02wTk2yAfBB4Dy6n6l0zFKLSpIkSZJWQCP9QcOntcNLgM2WXjiSJEmStOIaUYLVXvv9VmBi7zVVddhSiUqSJEmSVkAj3SL4XeBnwGzgoaUXjiRJkiStuEaaYK1RVUcu1UgkSZIkaQU3oh80DHwlyduTTEiy4cBnqUYmSZIkSSuYka5g/Rk4AfgXYOAnExe+8EKSJEmSHjbSBOsf6X7Y8B+XZjCSJEmStCIb6RbBXwL3Ls1AJEmSJGlFN9IVrHuAq5JcDCwYKPQ17Sun2TfNZ+LR5491GCu0ecftOdYhSJIkaQyMNMH6n/aRJEmSJA1jRFsEq+pLwNeA6e3z1Va2TCTZK8nRi6ifmGTOKPrbJ8nWfYpttyTfGabutH6NI0mSJGn5N6IVrCS7AV8C5gEBNk1yQFVdutQie2TsVavqPOC8Pna7D/Ad4JpRxvHgaAapqr8fZVySJEmSVmAj3SL4MeAVVXU9QJKt6Fa0tu9HEEneChxF9+r3WcBC4H7gBcDUJLOAyVX1D0meAnyWR14R/y7gdz19bQZ8CzgEuA34NLAx3Us63g5sCOwFvDTJB4DXAnu09qvTvdDjLVV1b5LTB8VxSht74xbjvm3YdZJ8E3ge3Qrfm6uqkvy43ddkYPOqel+L8cCe+3kzcFgb+zLg3VW1cND8bDHEuDcAxwOvavP2kar6RkuGPwTcAWwDnAXMBg4H1gT2qapfLf5XRZIkSdJojfQtgqsNJFcAVfVzYLV+BJDkucAHgD2qalu6RADgacCLq+rIQZd8Ariktd0OuLqnr2fRJVcHVtUVwKnAoVW1PV2ic0pV/YRuNex9VTWpJRvnVNUOrc9rgYN7xuuN40zg063di4GbW5sXAEcAW9Mlfi8ZFPO3gL/rOd8P+HqS57Tjl1TVJLrkaf8hpmmocV8DTAK2BV4OnJBkQmu/LfBO4DnAW4CtqmpH4DTg0CH6J8khSaYlmbbw3vlDNZEkSZK0GCNdwZqW5DTgjHa+PzCtTzHsAZw98DO2quq2JLSyhcO0f2truxCYn2QDutWdbwOvqaprkqxDl4yc3foDGD9MDM9L8hFgfWAd4Ac9dWdX1cIk6wKbVNW5bez7AVrfl1fVb9v5VcBE4P8GOqiqW5PckORFwC+AZwNTgffQrQJe0fpZE/hDb2CLGHdn4GttDm5JcgmwA3AncEVV3dza/Qq4oHU3G9h9qAmoqlPpElLGT9iyhmojSZIkadFGmmC9iy4ZGHgt+xTglKUS0SPuGWX7+cBvgJ3pnq1aBbijrQwtzul0W+dmtu17u40yjgU9xwsZel6/DrweuA44t20hDPClqvqnEYwxGr3xPNRz/tAwsUmSJEnqg5G+RXAB8BXgHVX1mqo6qZX1w0XAvkmeDJBkw8W0/xFdwkeScUnWa+V/ptuG99Ykb6qqO4G5SfZtbZNk29b2LmDdnj7XBW5OshpDb9Gjqu4Cfptkn9bf+CRrjeI+zwX2Bt5Il2wN3MvrkvxF63PDJM8Y4bhTgP3aHGwM7ApcPop4JEmSJPXZIhOslpQcm+SPwPXA9UluTXJMvwKoqquBfwcuSTITOHExlxwO7J5kNt0LJR5+DXpV3QO8Gnhvkr3okqWDW79X0yU40CU470syI8nmwAfpXjAxlW6FaThvAQ5rL934CfCXo7jP2+me73pGVV3eyq6he/7sgtbnhcAEePgV75MXMe65dC8EmUmXpL6/qn4/0ngkSZIk9V+qhn/cJsmRdG+pO6Sq5rayzYDPAN+vqpOWSZRapsZP2LImHHDyWIexQpt33J5jHYIkSZKWoiTTq2ryY8oXk2DNAP5q4AUUPeUbAxdU1Qv6HqnG3OTJk2vatH69w0SSJEla+QyXYC3uGazVBidX0L0Vjz69pl2SJEmSVhaLS7D+/DjrJEmSJOkJZ3Gv7N42yZ1DlAdYYynEI0mSJEkrrEUmWFU1blkFIkmSJEkruhH9HCxJkiRJ0uKZYEmSJElSn5hgSZIkSVKfmGBJkiRJUp+YYEmSJElSn5hgSZIkSVKfLO7nYOkJaPZN85l49PljHcZybd5xe451CJIkSVoOuYIlSZIkSX1igjUGkkxMMmes45AkSZLUXyZYkiRJktQny32C1VZ7rk3y+SRXJ7kgyZpJNk/y/STTk0xJ8uwk45LMTWf9JAuT7Nr6uTTJlklemuSq9pmRZN0hxjw4yc+TXN7G/VRPLBclmZXkR0mevpjyzZP8LMnsJB9JcvcQY41LckKSK9r17xhmHl6Z5MokM5P8qJVtmOR/2nU/S/L8Vn5ski+1efl1ktckOb7F8f0kq/Xr10eSJEnSI5b7BKvZEvh0VT0XuAN4LXAqcGhVbQ8cBZxSVQuB64GtgZ2BK4FdkowHNq2qX7S276mqScAuwH29AyV5KvBB4EXAS4Bn91R/EvhSVT0fOBP4xGLKPw58vKq2AX47zL0dDMyvqh2AHYC3J3nmoJg2Bj4PvLaqtgX2bVUfAma0cf8Z+HLPZZsDewB7AWcAF7c47gMe84aGJIckmZZk2sJ75w8TqiRJkqRFWVESrLlVdVU7ng5MBF4MnJ3kKuBzwIRWPwXYtX3+ky7R2gG4otVPBU5MchiwflU9OGisHYFLquq2qnoAOLunbifgq+34K63vxZUPXD9QP9grgLe2+7gMeDJdQtnrRcClVTUXoKpua+U7t/GoqouAJyd5Uqv7Xot/NjAO+H4rn003f49SVadW1eSqmjxurfWGCVWSJEnSoqwoCdaCnuOFwIbAHVU1qefznFZ/Kd3K1I7Ad4H1gd3oEi+q6jjg74E1galJeleoxkLoVuIG7uOZVXVBH/pdAFBVDwEPVFW18ofw9fySJEnSUrGiJFiD3QnMTbIvQHvmattWdznd6tZDVXU/cBXwDrrEiySbV9Xsqvoo3arW4ATrCuClSTZIsirddsQBPwHe0I73pyVtiyj/Wc/1A/WD/QB418BzUUm2SrL2oDY/A3Yd2DqYZMNWPqWNR5LdgD9W1Z3DjCNJkiRpKVtREyzoEouDk8wErgb2BqiqBcCNdEkJdEnIunRb4wCOSDInySzgAeB7AG2LHlV1E/AfdInaVGAeMPBQ0qHAQe3atwCHL6b8CODIVr5FTz+9TgOuAa5sr27/HG2FqSemW4FDgHPa/X6jXXsssH3r/zjggBHMmyRJkqSlJI/sHNOAJOtU1d1tBetc4AtVde7j6Gct4L6qqiRvAN5YVXv3O95+mzx5ck2bNm2sw5AkSZKWW0mmV9XkweU+izO0Y5O8HFgDuAD4n8fZz/bAp5KE7u2Hb+tLdJIkSZKWSyZYQ6iqo/rUzxRg28U2lCRJkrRSWJGfwZIkSZKk5YoJliRJkiT1iQmWJEmSJPWJCZYkSZIk9YkJliRJkiT1iQmWJEmSJPWJCZYkSZIk9YkJliRJkiT1iT9oWI8x+6b5TDz6/LEOY7ky77g9xzoESZIkrQBcwZIkSZKkPjHBkiRJkqQ+McFajiQZn+SHSa5Kst9YxyNJkiRpdHwGq0+SrFpVDy5hNy8AqKpJSx6RJEmSpGVtpVzBSvK+JIe145OSXNSO90hyT5IT2/nhSW5ox5slmZpkhyTntLK9k9yXZPUkawy07RnnwCTntf5/lGRCkkvbCtScJLskGZfk9HY+O8l727WHJbkmyawkX0/yF8AZwA7t+s17xhmX5L9aH7OSHNrKX5ZkRuv3C0nGt/J5Sf6z9TMtyXZJfpDkV0neuZSnX5IkSXrCWllXsKYA/wh8ApgMjE+yGrALcDzw6tZuF+BPSTZpx5cCM4BJPfVzgB3o5uqyIcbaDnh+Vd2W5B+BH1TVvycZB6zV+tqkqp4HkGT9dt3RwDOrakGS9avqjiR/DxxVVa8eNMYhwERgUlU9mGTDJGsApwMvq6qfJ/ky8C7g5HbNb6pqUpKTWruXAGu0+/ns4JtIckgbh3FP2nioOZUkSZK0GCvlChYwHdg+yZOABcBP6RKtXYCzgXWSrAtsCnwV2LXVTWnb/H6V5DnAjsCJvfVDjHVhVd3Wjq8ADkpyLLBNVd0F3ABsluSTSV4J3NnazgLOTPJmYHFbC18OfG5gC2Ib71nA3Kr6eWvzpRbngPPa19nAZVV1V1XdCizoSfIeVlWnVtXkqpo8bq31FhOOJEmSpKGslAlWVT0AzAUOBH5ClxjtDmwBXNvKDgKub3W7ADsBU1sXlwKvAh4Afgjs3D5DJVj39Ix7KV2ScxNwepK3VtXtwLbAj4F3Aqe15nsCn6ZbAbsiSb9XExe0rw/1HA+cr6wrl5IkSdKYWikTrGYKcBRdsjSFLrmZUVU1qG4GXfK1oKrm91x7BPDTturzZLoVozmLGjDJM4BbqurzdInUdkk2Alapqm8BH2hlqwCbVtXFwP8D1gPWWUTXFwLvGEjCkmxIlxxOTLJFa/MW4JKRTIwkSZKkpWNlXsmYAvwLXZJ0T5L7eWQFagrd9sBLq2phkhuB63quvQx4Cl0CBt12vr+sqkqyFzC5qo4ZYszdgPcleQC4G3grsAnwxZZUAfwTMA44I8l6QIBPtGewHu5o0DinAVsBs1rfn6+qTyU5CDi7JV5XMMSzVZIkSZKWnXQLOtIjJk+eXNOmTRvrMCRJkqTlVpLpVTV5cPnKvEVQkiRJkpYpEyxJkiRJ6hMTLEmSJEnqExMsSZIkSeoTEyxJkiRJ6hMTLEmSJEnqExMsSZIkSeoTEyxJkiRJ6hMTLEmSJEnqExMsSZIkSeqTVcc6AC1/Zt80n4lHnz/WYSwX5h2351iHIEmSpBWIK1iSJEmS1CcmWJIkSZLUJyZYfZDk9CSvG+s4JEmSJI0tEyxJkiRJ6hMTrFFIMjHJtUk+n+TqJBckWXNQm2OSXJFkTpJTk2SIfjZOcmHr47Qkv06yUas7sl07J8kRrey4JO/puf7YJEcN6vPtbdyZSb6VZK1W/qjVtSR393VSJEmSJD3MBGv0tgQ+XVXPBe4AXjuo/lNVtUNVPQ9YE3j1EH38K3BR6+ObwNMBkmwPHAS8EHgR8PYkLwC+Aby+5/rXt7Je57RxtwWuBQ4ezU0lOSTJtCTTFt47fzSXSpIkSWpMsEZvblVd1Y6nAxMH1e+e5LIks4E9gOcO0cfOwNcBqur7wO095edW1T1VdTdwDrBLVc0A/iLJU5NsC9xeVTcO6vN5Saa0cfcfZtxhVdWpVTW5qiaPW2u90VwqSZIkqfHnYI3egp7jhXSrVAAkWQM4BZhcVTcmORZYo0/jng28DvhLHrt6BXA6sE9VzUxyILBbK3+QlkgnWQVYvU/xSJIkSRrEFaz+Gkim/phkHbqEaChTaVv+krwC2KCVTwH2SbJWkrWBv2tl0CVVb2h9nj1En+sCNydZjW4Fa8A8YPt2vBew2ijvSZIkSdIImWD1UVXdAXwemAP8ALhioC7JO5O8s51+CHhFkjnAvsDvgbuq6kq6lajLgcuA09r2QKrqarok6qaqurn1eVqSya3PD7ZrpgLX9YT1eeClSWYCOwH39Pm2JUmSJDWpqrGO4QknyXhgYVU9mGQn4DNVNWmMw3rY5MmTa9q0aWMdhiRJkrTcSjK9qiYPLvcZrLHxdOCs9kzUn4G3j3E8kiRJkvrABGsMVNUvgBeMdRySJEmS+stnsCRJkiSpT0ywJEmSJKlPTLAkSZIkqU9MsCRJkiSpT0ywJEmSJKlPTLAkSZIkqU9MsCRJkiSpT0ywJEmSJKlP/EHDeozZN81n4tHnj3UYY2LecXuOdQiSJElagbmCJUmSJEl9YoIlSZIkSX1igvUEkeTAJE8d6zgkSZKklZkJ1iBJxo11DENJsqTPyx0ImGBJkiRJS9EyS7CSvC/JYe34pCQXteM9ktyT5MR2fniSG9rxZkmmJtkhyTmtbO8k9yVZPckaA217xhmX5IQkVySZleQdrXy3JN/pafepJAe243lJPprkSmDfJG9MMjvJnCQf7bnm4CQ/T3J5ks8n+VQr/9sklyWZkeSHSZ7Syo9N8qUkU5L8Oslrkhzf+v5+ktVau+2TXJJkepIfJJnQyn+c5OQk04DDe+LYrdV9M8l1Sc5MkuH6SvI6YDJwZpKrkqzZt19YSZIkSQ9blitYU4Bd2vFkYJ2WYOwCHN9TtwvwpySbtONLgRnApJ76OcAOwAuBywaNczAwv6p2aG3enuSZI4jvT1W1XRvvo8AebcwdkuzTttd9EHgR8BLg2T3X/h/woqp6AfB14P09dZu3vvYCzgAurqptgPuAPdscfBJ4XVVtD3wB+Pee61evqslV9bFB8b4AOALYGtgMeMlwfVXVN4FpwP5VNamq7ht880kOSTItybSF984fwXRJkiRJGmxZvqZ9OrB9kicBC4Ar6RKtXYDDgDckWRfYFPgqsGurO6eqHkzyqyTPAXYETmz14+gSt16vAJ7fVm0A1gO2BP68mPi+0b7uAPy4qm4FSHJmGwvgkqq6rZWfDWzVyp8GfKOtPK0OzO3p93tV9UCS2S3e77fy2cBE4FnA84AL2yLUOODmIeIa7PKq+m2L5arW1x2L6WtYVXUqcCrA+Alb1kiukSRJkvRoyyzBaknGXLpngX4CzAJ2B7YArm1lBwHX0yVNbwN2Av6xdXEp8CrgAeCHwOl0CcT7Bg0V4NCq+sGjCpOdefSK3RqDrrvncd9ct2p0YlWdl2Q34NieugUAVfVQkgeqaiB5eYhu/gNcXVU7DdP3cHEt6DleOMK+JEmSJC1Fy/olF1OAo+iSpSnAO4EZLenorZtBl3wtqKr5PdceAfy0rS49mW71Z86gMX4AvKvn+aatkqwN/BrYOsn4JOsDLxsmxsuBlybZqL3w4o3AJcAVrXyD9sKJ1/Zcsx5wUzs+YHRTwvXAxkl2avGuluS5o+xjJH3dBaz7OPuVJEmSNAJjkWBNoEuSbgHu55EtflPotgdeWlULgRvpnm0acBnwFLoEDLoVsNlVVUn2SvLhVn4acA1wZZI5wOeAVavqRuAsuoTsLLok7jGq6mbgaOBiYCYwvaq+XVU3Af9Bl4BNBeYBA8nfscDZSaYDfxzNhFTVn4HXAR9NMhO4Cnjx4HZJJic5bQn6Oh34rC+5kCRJkpaePLJjTYuTZJ2qurutYJ0LfKGqzh3ruPpt8uTJNW3atLEOQ5IkSVpuJZleVZMHl/tzsEbn2PZCiTl0L7L4nzGNRpIkSdJyZVm+RXCFV1VHjXUMkiRJkpZfrmBJkiRJUp+YYEmSJElSn5hgSZIkSVKf+BZBPUaSu+h+ppYev40Y5Sv79SjO35Jx/pacc7hknL8l5xwuGedvyTh/I/OMqtp4cKEvudBQrh/qlZMauSTTnMPHz/lbMs7fknMOl4zzt+ScwyXj/C0Z52/JuEVQkiRJkvrEBEuSJEmS+sQES0M5dawDWAk4h0vG+Vsyzt+Scw6XjPO35JzDJeP8LRnnbwn4kgtJkiRJ6hNXsCRJkiSpT0ywJEmSJKlPTLD0KElemeT6JL9McvRYx7O8SPKFJH9IMqenbMMkFyb5Rfu6QStPkk+0OZyVZLueaw5o7X+R5ICxuJexkGTTJBcnuSbJ1UkOb+XO4QglWSPJ5Ulmtjn8UCt/ZpLL2lx9I8nqrXx8O/9lq5/Y09c/tfLrk/z1GN3SmEgyLsmMJN9p587fCCWZl2R2kquSTGtlfg+PQpL1k3wzyXVJrk2yk3M4Mkme1X7vDXzuTHKE8zc6Sd7b/g6Zk+Rr7e8W/xzst6ry44eqAhgH/ArYDFgdmAlsPdZxLQ8fYFdgO2BOT9nxwNHt+Gjgo+34b4DvAQFeBFzWyjcEbmhfN2jHG4z1vS2j+ZsAbNeO1wV+DmztHI5qDgOs045XAy5rc3MW8IZW/lngXe343cBn2/EbgG+0463b9/Z44Jnte37cWN/fMpzHI4GvAt9p587fyOduHrDRoDK/h0c3h18C/r4drw6s7xw+rnkcB/weeIbzN6p52wSYC6zZzs8CDvTPwf5/XMFSrx2BX1bVDVX1Z+DrwN5jHNNyoaouBW4bVLw33V+WtK/79JR/uTo/A9ZPMgH4a+DCqrqtqm4HLgReudSDXw5U1c1VdWU7vgu4lu4PeudwhNpc3N1OV2ufAvYAvtnKB8/hwNx+E3hZkrTyr1fVgqqaC/yS7nt/pZfkacCewGntPDh/S8rv4RFKsh7df9b9N0BV/bmq7sA5fDxeBvyqqn6N8zdaqwJrJlkVWAu4Gf8c7DsTLPXaBLix5/y3rUxDe0pV3dyOfw88pR0PN4/OL9C2GLyAbgXGORyFtr3tKuAPdP8o+BVwR1U92Jr0zsfDc9Xq5wNP5ok9hycD7wceaudPxvkbjQIuSDI9ySGtzO/hkXsmcCvwxbZN9bQka+McPh5vAL7Wjp2/Eaqqm4D/An5Dl1jNB6bjn4N9Z4Il9UFVFd0/PrQISdYBvgUcUVV39tY5h4tXVQurahLwNLr/LXz22Ea04kjyauAPVTV9rGNZge1cVdsBrwLek2TX3kq/hxdrVbqt5p+pqhcA99BtaXuYc7h47fmgvYCzB9c5f4vWnk/bmy7ZfyqwNk+s1btlxgRLvW4CNu05f1or09BuadsNaF//0MqHm8cn9PwmWY0uuTqzqs5pxc7h49C2FV0M7ES37WXVVtU7Hw/PVatfD/gTT9w5fAmwV5J5dNuf9wA+jvM3Yu1/v6mqPwDn0iX5fg+P3G+B31bVZe38m3QJl3M4Oq8CrqyqW9q58zdyLwfmVtWtVfUAcA7dn43+OdhnJljqdQWwZXubzOp0S/DnjXFMy7PzgIG3Dx0AfLun/K3tDUYvAua37Qs/AF6RZIP2v0ivaGUrvbZn+7+Ba6vqxJ4q53CEkmycZP12vCbwV3TPsl0MvK41GzyHA3P7OuCi9r+75wFvaG+HeiawJXD5MrmJMVRV/1RVT6uqiXR/tl1UVfvj/I1IkrWTrDtwTPe9Nwe/h0esqn4P3JjkWa3oZcA1OIej9UYe2R4Izt9o/AZ4UZK12t/LA78H/XOw3/r5xgw/K/6H7q07P6d7tuNfxjqe5eVD94f5zcADdP8LeTDdPuQfAb8Afghs2NoG+HSbw9nA5J5+3kb3MOgvgYPG+r6W4fztTLdtYxZwVfv8jXM4qjl8PjCjzeEc4JhWvhndX2y/pNsyM76Vr9HOf9nqN+vp61/a3F4PvGqs720M5nI3HnmLoPM3sjnbjO6tYTOBqwf+fvB7eNTzOAmY1r6P/4fuLXbO4cjnb226FZT1esqcv9HN4YeA69rfI1+hexOgfw72+ZM2SZIkSZKkJeQWQUmSJEnqExMsSZIkSeoTEyxJkiRJ6hMTLEmSJEnqExMsSZIkSeoTEyxJ0kovSSX5WM/5UUmO7VPfpyd53eJbLvE4+ya5NsnFI2z/4SQvb8dHJFlrBNc8ql2S7w78/DVJ0siYYEmSnggWAK9JstFYB9IryaqjaH4w8Paq2n0kjavqmKr6YTs9AlhsgjW4XVX9TVXdMYoYJekJzwRLkvRE8CBwKvDewRWDV6CS3N2+7pbkkiTfTnJDkuOS7J/k8iSzk2ze083Lk0xL8vMkr27Xj0tyQpIrksxK8o6efqckOQ+4Zoh43tj6n5Pko63sGLof2P3fSU4Y4pr/166ZmeS43vtKchjwVODigdWvJJ9p8V6d5EOtbKh28waS0iRHtpjmJDmilU1sq2qfb31dkGTNgf6SXNPu/esj/pWSpBXcaP7nTJKkFdmngVlJjh/FNdsCzwFuA24ATquqHZMcDhxKt+IDMBHYEdicLkHZAngrML+qdkgyHpia5ILWfjvgeVU1t3ewJE8FPgpsD9wOXJBkn6r6cJI9gKOqatqga14F7A28sKruTbJhb31VfSLJkcDuVfXHVvwvVXVbknHAj5I8f5h2A2NsDxwEvBAIcFmSS1qMWwJvrKq3JzkLeC1wBnA08MyqWuA2Q0lPJK5gSZKeEKrqTuDLwGGjuOyKqrq5qhYAvwIGEqTZdEnVgLOq6qGq+gVdIvZs4BXAW5NcBVwGPJkuGQG4fHBy1ewA/Liqbq2qB4EzgV0XE+PLgS9W1b3tPm8bwX29PsmVwAzgucDWi2m/M3BuVd1TVXcD5wC7tLq5VXVVO57OI/MyCzgzyZvpVhAl6QnBBEuS9ERyMt2zTGv3lD1I+/swySrA6j11C3qOH+o5f4hH7wKpQeMU3UrPoVU1qX2eWVUDCdo9S3ITSyLJM4GjgJdV1fOB84E1lqDL3jlayCPzsifdquF2wBWjfN5MklZYJliSpCeMtrpzFl2SNWAe3ZY8gL2A1R5H1/smWaU9l7UZcD3wA+BdSVYDSLJVkrUX1QlwOfDSJBu17XtvBC5ZzDUXAgcNvP1v8BbB5i5g3Xb8JLoEb36SpwCvGqZdrynAPknWavfwd61sSC1R3bSqLgb+H7AesM5i7kOSVgr+b5Ik6YnmY8A/9Jx/Hvh2kpnA93l8q0u/oUuOngS8s6ruT3Ia3Xa5K5MEuBXYZ1GdVNXNSY4GLqZbATu/qr69mGu+n2QSMC3Jn4HvAv88qNmpwPeT/K6qdk8yA7gOuBGYOly7njGuTHJ6u0fonkWbkWTiMGGNA85Isl67j0/4NkJJTxSpGryrQZIkSZL0eLhFUJIkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL6xARLkiRJkvrEBEuSJEmS+sQES5IkSZL65P8D20qtohQZKzYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "top_domains = domain_counts.head(10).sort_values()\n",
    "\n",
    "ax = top_domains.plot(figsize=[12,4], title=\"Top 10 cited domains\", legend=False, kind=\"barh\")\n",
    "ax.set_ylabel(\"Domain\")\n",
    "ax.set_xlabel(\"Number of citations\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/top_cited_domains.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By far, the most cited domains is books.google.com, having more than twice the number of citations than the second most cited domain, www.nytimes.com. The rest the most cited domains belong mainly to news sites (news.bbc.co.uk, www.telegraph.co.uk, news.google.com) and some specialized sites (cricket.archive.com, www.rsssf.org, www.eurogamer.net). A special mention must be made to web.archive.org, a site for filing copies of websites, a good citation practice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's visualize Citoid's performance for these domains:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEYCAYAAADmugmLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3HUlEQVR4nO3de5xVVf3/8ddbRBBB8JahaZNXNBWUwbyAqfn1a19NrTQ0MzHT7OIl07Lvt8z6WZn2VSszb1+l1LxQUpbmJW9MqMBwHUCtFAzNTFNQVAjh/ftjraPbwxnmgDPMnuHzfDzOY/ZZe+21PnsPnM+svffZS7YJIYQQymitzg4ghBBCaE0kqRBCCKUVSSqEEEJpRZIKIYRQWpGkQgghlFYkqRBCCKUVSSqs0ST9t6SrV3HbLSUtlNSjlfXnSrr+nUXYNUj6vKTn8vHYqLPjWRmSGiRZ0tqrud8Rkh5fnX12RZGkQrcn6ZOSmvMH6LOS/iBpOIDt79n+bK63Uh9Wtv9mu6/tpR0Zf9lJ6glcBByYj8e/2qHNuZIOeOfRlZftJtvbd3YcZRdJKnRrks4ALgG+B2wKbAlcBhzWiWF1N5sCvYFZK7uhkvgcCq2Kfxyh25LUH/gO8EXbt9p+1fYS27+zfVauUzwlNy7/nJ9HXXtKWkvSNyQ9Jemfkn6R211u5CXpfZIelPSKpHuAjVcQ28aSfi9pvqQXJTVVPqxzm9sU6o6WdF7h/WGSpkl6WdITkg7K5RtKulbS3yW9JOk3hW0OydvMl/SQpF0K674m6Zkc9+OSPpTLd88j0JfzqbyLauzHdkDllNV8Sffl8r0kTZK0IP/cq7DNA5K+K2k88BqwVVWb15H+mPhd/j18NZcfKmlW3ocHJO1Q2GaupK9Lmp33/VpJvVs59j0k/VDSC5KeBA6uWr+ZpNvy7+Wvkk4srDtX0hhJ1+fj1SJpu9z3PyXNk3Rgof7xkh7NdZ+U9LnCun0lPV21D2dKmpGP282VfVjRv5duz3a84tUtX8BBwBvA2iuocy5wfV5uAFysD3wG+Cvpg7QvcCtwXa36wMOk0169gH2AVypt1+j3+8DlQM/8GgEorzOwTaHuaOC8vLw7sAD4D9IfmZsDg/K624GbgQ1ymx/M5bsC/wQ+APQAjgPm5ji3B+YBmxX2aevC/hybl/sCe7SyL9XHYUPgJeBYYG3g6Px+o7z+AeBvwPvz+p412pwLHFB4vx3wat7vnsBX8+9lnUL9mcAWuf/xlWNWo+2TgccKde+vin8cabTdGxgCPA/sX/j3sgj4zxz7L4A5wP/kuE4E5hT6OhjYGhDwQVJS3i2v2xd4umqfJwKb5bgeBU5u699Ld3+tGZk4rKk2Al6w/cY7aOMY4CLbT9peCHwdOEpV160kbQkMA75pe7HtccDvVtDuEmAg8F6n0V2T86dRG04ArrF9j+1ltp+x/ZikgcCHSR9qL+U2H8zbnARcYXuC7aW2fw4sBvYAlpKS1Y6Setqea/uJQozbSNrY9kLbj9QRH6QP5r/Yvs72G7ZvJCWFjxTqjLY9K69fUkebI4Hb834vAX4IrAvsVahzqe15tl8EvktKjrV8ArikUPf7lRWStgD2Br5me5HtacDVwKcL2zfZviv/uxoDbAKcn+O6CWiQNADA9u22n3DyIHA3KcG05se2/57j+h0pScKq/3vp8iJJhe7sX8DG1QllJW0GPFV4/xTpL+hNa9R7yfarVXVbcyFpJHB3Pg10dp3xbAE80Ur5i7ZfqrHuvcBX8qmi+ZLm5/qb2f4rcDpphPBPSTdJ2ixvdwJpBPNYPmV3SJ0xVh8z8vvNC+/n1dlWzTZtL8tttNbmU3mb1tqqrltc96LtV6rWF/t5rrD8OukPoaWF95BGnkj6sKRH8im6+cB/sYLTwMA/CsuvVdph1f+9dHmRpEJ39jBpxHB4nfVr/WX6d9KHfMWWpFOIz1XVexbYQNJ6VXVrd2S/YvsrtrcCDgXOqFwLIn049SlUf3dheR7p9FG1ecCGlb/ga6z7ru0BhVefPMLB9i9tD8/7aeAHufwvto8G3pXLflW1f62pPmaQjsUzhfdtjQKq17+tTUkiJdpim1tU9ff3Vtp+tkbdYj8bSupXtb7YT10k9QJ+TRr1bWp7AHAH6dTfSmnj30u3FkkqdFu2FwDnAD+VdLikPpJ65r9uL6ixyfPAMt5+If9G4MtKN0X0Jd0leHP1KUTbTwHNwLclraN0i3vx9Nbb5BsZtskftgtIp92W5dXTgE/mC/wHka5lVPwfcLykDynd1LG5pEG2nwX+AFwmaYO8n/vkba4CTpb0ASXrSTpYUj9J20vaP3+gLiKNBJblGD8laZM8apmf21pG2+4AtlO69X9tSSOBHYHf17FtxXO8/fdwC3Bw3u+ewFdIf4A8VKjzRUnvkbQh6RrRza20fQtwaq67AfDmqMT2vNzm9yX1VrrB5ARgVb7vtg7pVOrzwBuSPgwcuOJNamvj30u3FkkqdGu2/xc4A/gG6cNiHvAl4Dc16r5GupYxPp8W2wO4BriOdDF9DumD/JRWuvsk6eaEF4FvkS6qt2Zb4I/AQtKI7zLb9+d1p5ES3HzSNbE3Y7U9ETgeuJj0YfUgb40wjiVdu3iMdKPE6XmbZtIF/UtJNzD8FRiVt+kFnA+8QDrV9C7SdTdIN57MkrQQ+BFwlO3K6axWOX1P6hBSIvkX6SaHQ2y/0Na2Bd8HvpF/D2fafhz4FPCTHOtHgI/Y/ndhm1+Srvk8SToleh61XQXcBUwHppBuhik6mnQzyN+BscC3bP9xJWIH0ugHOJWUFF8i/fu4bWXbyVb076Vbq9xNFEIIXZakucBnVyWZhHKLkVQIIYTSiiQVQgihtOJ0XwghhNKKkVQIIYTSWq2Ppg9dw8Ybb+yGhobODiOEsIaYPHnyC7Y3qbUuklRYTkNDA83NzZ0dRghhDSGp1aezxOm+EEIIpRVJKoQQQmlFkgohhFBakaRCCCGUVtw4EZbT8swCGs6+vbPDCCF0MXPPP7jtSispRlIhhBBKq0skKUmHrmiSL0kNkmauRHuHS9qxnWLbV1LNKQgkXd1e/YQQwpqo9Kf7JK1t+zZW/RH3tRxOmttm9krGsVLTkNv+7ErGFUIIoaAUIylJn5Y0Q9J0SddJGi3pckkTgAskjZJ0aa67qaSxue50SXtVtbWVpKmShknaWtKdkiZLapI0KNc/FLhQ0rRc58Q8PfZ0Sb+W1Ce3VR3HNpL+mOtNkVSZIbWvpF9JekzSDXliMiQ9IKlR0smSLizEWNyfT0mamGO5QlKPGsdnuX7z5HUXSpopqSVPLFcZ2T0o6bdK00yfL+mY3EdLIeYQQii9Tk9Skt5PmpBuf9uDSRO+AbwH2Mv2GVWb/Bh4MNfdDZhVaGt70nTNo2xPAq4ETrE9FDiTNFHYQ6RR2Vm2h9h+ArjV9rDc5qOkmTgrinHcAPw019uLNA01wK6kCeZ2JM0mundVzL8GPlp4PxK4SdIOeXlv20NIs20eU+Mw1er3Y8AQYDBwACnpDsz1BwMnAzuQJsLbzvbuwNW0MmGfpJMkNUtqXvraglpVQghhtSvD6b79gTGVWTttv5gHImNsL22l/qdz3aXAgjwF9CbAb4GP2Z6tNNX3XsCY3B6kWUhr2UnSecAAoC9p1s6KMbaXSuoHbG57bO57EUBue6Ltp/P7aaRZPf9UacD283lUswfwF2AQMB74IjAUmJTbWZc0o+qbVtDvcODGfAyek/QgMAx4GZiUpxNH0hOk2UoBWoD9ah0A21eSkjq9Bm4bj8YPIZRCGZJUa15dyfoLgL8Bw0nXmtYC5ucRSltGA4fbni5pFLDvSsaxuLC8lNrH9SbgE6Spvcfadj4t+HPbX69R/50oxrOs8H5ZK7GFEEIpdfrpPuA+4EhJGwFI2rCN+vcCn891e0jqn8v/TTql9mlJn7T9MjBH0pG5riQNznVfAfoV2uwHPCupJ7VPt2H7FeBpSYfn9npVrl3VaSxwGHA0KWFV9uUISe/KbW4o6b119tsEjMzHYBNgH2DiSsQTQgil1+lJyvYs4LvAg5KmAxe1sclpwH6SWoDJpOtAlbZeBQ4BvizpUFLCOSG3O4uUJCAlibPyDRZbA98EJpBOwT22gr6PBU6VNAN4CHj3SuznS6TrXe+1PTGXzSZdj7s7t3kPMBDevH29cQX9jgVmANNJif6rtv9RbzwhhNAVxMy8YTm9Bm7rgcdd0tlhhBC6mFV94oSkybYba62L6xNhOTtv3p/mDni8SQghrKxOP90XQgghtCaSVAghhNKKJBVCCKG0IkmFEEIorUhSIYQQSiuSVAghhNKKJBVCCKG0IkmFEEIorUhSIYQQSiuSVAghhNKKxyKF5bQ8s4CGs2/v7DBCCO1gVZ+nVxYxkgohhFBakaTqIGmApC8U3m8m6VedGVMIIawJIknVZwDwZpKy/XfbR3ReOCGEsGYoVZKSdJakU/PyxZLuy8v7S3pV0kX5/WmSnszLW0kaL2mYpFtz2WGSXpe0jqTelbqFfkZJulXSnZL+IumCXP4ZSZcU6p0o6WLgfGBrSdMkXSipQdLMQlu/kXSPpLmSviTpjDyh4iOVmYYlbZ37myypSdKgXH6kpJmSpksa18px+bSkGbnOdbmsQdJ9ufxeSVvm8tGSfpb7flLSvpKukfSopNHt85sKIYTVo1RJijQl+oi83Aj0zVO6jwAuKKwbAfxL0uZ5eRwwFRhSWD8TGAZ8gDTrbrUhwEhgZ9I07FsAtwAfyX0CHA9cA5wNPGF7iO2zarS1E/Cx3N93gdds7wo8DHw617kSOMX2UOBM4LJcfg7wn7YHA4dWNyzp/aTZe/fPdU7Lq34C/Nz2LsANwI8Lm20A7Al8GbgNuBh4P7CzpCHUIOkkSc2Smpe+tqBWlRBCWO3KlqQmA0MlrQ8sJn3IN5KSzhhS0uoHbAH8Etgnr2uy/QbwhKQdgN1J09C/ub5GX/faXmB7ETCbNK37QtJU7IfkkU5P2y11xH2/7VdsPw8sAH6Xy1uABkl9gb2AMZKmAVeQp4knTVk/WtKJQI8abe8PjLH9AoDtF3P5nvkYAFwHDC9s8zunKZdbgOdst9heBswCGmrtgO0rbTfabuzRp38duxxCCB2vVLeg214iaQ4wCngImAHsB2wDPJrLjgceJyWez5A+rL+SmxgHfBhYAvwRGE364K81+llcWF7KW8fiauC/gceAa+sMvdjWssL7ZbndtYD5todUb2j7ZEkfAA4GJksaavtfdfbbVjzFWIrxhBBCl1C2kRSk5HMmKeE0AScDU/PIoLhuKimBLba9oLDt6cDDeVSzEbA96dRfXWxPII3UPgncmItfAfqt6g7ZfhmYI+lIACWD8/LWtifYPgd4PvdddB9wpKSNcv0Nc/lDwFF5+RhqjxZDCKFLK2uSGkhKNM8Bi3jrA7iJ9CE+zvZSYB7wp8K2E4BNSUkM0kisxbYlHSrpO3XGcAsw3vZLAHlkMz7f4HDhKu7XMcAJkqaTTrsdlssvlNSSb8R4CJieb3G/I/c9i3Sd68G87UV5u1OA4yXNAI7lrWtVIYTQbSgNUEKRpN8DF9u+t7Nj6Qy9Bm7rgcdd0tlhhBDaQVd44oSkybYba62L6xMFkgYAE4Hpa2qCAth58/40d4F/2CGE7i+SVIHt+cB2nR1HCCGEpIzXpEIIIQQgklQIIYQSiyQVQgihtCJJhRBCKK1IUiGEEEorklQIIYTSiiQVQgihtCJJhRBCKK1IUiGEEEornjgRltPyzAIazr69s8MIoVvqCs/SK5MYSYUQQiitSFLvgKSFrZQ/IKnmE307Q9niCSGEekWSCiGEUFrdJklJapD0qKSrJM2SdLekdSVtLelOSZMlNUkaJKmHpDl5htwBkpZK2ie3M07StpI+KGlafk2VVHNmXkkX5/7ulbRJYdWxeduZknbPdftKujZPcjhD0sdrtLeNpD9Kmi5pSo5fki7MbbVIGtnK/s8svD9T0rlVddaSNFrSeat2lEMIYfXqNkkq2xb4qe33A/OBjwNXAqfYHkqaev6yPKvv48COwHBgCjBCUi9gC9t/yXW/aHsIMAJ4vUZ/6wHNub8HgW8V1vXJ234BuCaXfRNYYHtn27uQpoavdkPeh8HAXsCzwMeAIcBg4ADSbL4DV+7QsHZu+y+2v1G9UtJJkpolNS99bcFKNh1CCB2juyWpOban5eXJQAPpg36MpGnAFaSp6SFNRb9Pfn2flKyGAZPy+vHARZJOBQbYfqNGf8uAm/Py9bmNihsBbI8D1s8TKh4A/LRSoTI9fUUerW1ue2xev8j2a7ndG20vtf0cKSEOq+uIvOUKYKbt79ZaaftK2422G3v06b+STYcQQsfobklqcWF5KbAhMN/2kMJrh7x+HGmEtDtwBzAA2JeUvLB9PvBZYF1gvKRBdfTvVpZrvW9vb/D232fvqvUPAftJqi4PIYTS6m5JqtrLwBxJRwLkazuD87qJpFHWMtuLgGnA50jJC0lb226x/QPS6KpWkloLOCIvfxL4U2HdyNzOcNIpvgXAPcAXKxUkbVBszPYrwNOSDs/re0nqQ0qcI/O1tE1Io7+JVbE8B7xL0kb5tOUhVev/j5SMb5EU348LIXQJ3T1JARwDnCBpOjALOAzA9mJgHvBIrtcE9ANa8vvT840KM4AlwB8A8mnDileB3fMNC/sD3ymsWyRpKnA5cEIuOw/YILc7Hdgvt3l14RbxY4FTc78PAe8GxgIzgOmk61hftf0PSZtJuiPvz5Lc/0RSMnys+kDYvgiYClwnaU343YcQujjZHX0WKnQ1vQZu64HHXdLZYYTQLcUTJ5YnabLtmt/ljNM+YTk7b96f5viPFEIogTjlE0IIobQiSYUQQiitSFIhhBBKK5JUCCGE0ookFUIIobQiSYUQQiitSFIhhBBKK5JUCCGE0ookFUIIobQiSYUQQiiteCxSWE7LMwtoOPv2zg4jhC4hnsXXsWIkFUIIobRKl6QkNeSpL95pO3MlbdweMbU3SQ8UpuYIIYTQitIlqRBCCKGirElqbUk3SHpU0q8k9ZH0IUlTJbVIuibPPktr5RWS1pX0B0knSlpP0u2SpueJB0dWdyxpa0mP5PbOk7Qwl0vShXm7lsq2KyhfS9Jlkh6TdI+kOyQdUaO/AyU9LGmKpDGS+taos6mksTnu6ZL2yuVn5H5nSjo9lzXkPkdL+nM+jgdIGi/pL5J2f8e/nRBCWE3KmqS2By6zvQNpCvgzgNHASNs7k274+Lyk3rXKC+30BX4H3Gj7KuAg4O+2B9veCbizRt8/An6U23u6UP4xYAgwGDgAuFDSwDbKG4AdSbPt7lndUT4d+Q3gANu7Ac15X6v9GHjQ9mBgN2CWpKHA8cAHgD2AEyXtmutvA/wvacr7QaSp7YcDZwL/XaN9JJ0kqVlS89LXFtSqEkIIq11Zk9Q82+Pz8vXAh4A5tv+cy34O7ENKZrXKK34LXGv7F/l9C/Afkn4gaYTtWp/GewJj8vIvC+XDScluqe3ngAeBYW2Uj7G9zPY/gPtr9LUHKYmNz9PSHwe8t0a9/YGfAeR+FuT2x9p+1fZC4FZgRK4/x3aL7WXALOBepymYW0iJczm2r7TdaLuxR5/+taqEEMJqV9YkVT2n/fxVbGc8cJAkAeRkthvpw/o8SeescoTtQ8A9tofk1462T2iHdhcXlpcV3i8jvnYQQuhCypqktpRUOT32SdJpsAZJ2+SyY0kjlsdbKa84B3gJ+CmApM2A12xfD1xISljVHgE+npePKpQ3ASMl9ZC0CWnENnEF5eOBj+drU5sC+7bS196V+PM1s+1q1LuXfBoz99M/93t4vl63HvDRXBZCCN1Gm0lK0pGS+uXlb0i6VVKtD/f29DjwRUmPAhsAF5Ouv4yR1EIaEVxue1Gt8qq2TgPWlXQBsDMwMZ9a+xZwXt6v70g6NNc/HThD0gzStZ3KKcGxwAxgOnAf8NV8Gq+18l+TrmnNJp2ynFJoCwDbzwOjgBtzfw+TriFVx3QasF/ex8nAjrankK7HTQQmAFfbnlrn8Q0hhC5B6VLFCipIM2zvImk46UP9QuAc2x9YHQGubpL6AK/btqSjgKNtH7aKbfW1vVDSRqRksndOYKXWa+C2HnjcJZ0dRghdQjxx4p2TNNl2ze+O1nN9Ymn+eTBwpe3bJZ3XbtGVz1Dg0nwdaz7wmXfQ1u8lDQDWAf5fV0hQADtv3p/m+I8XQiiBepLUM5KuAP4D+EH+HlJZr2W9Y7abSLeTt0db+7ZHOyGEsKaqJ9l8ArgL+E/b84ENgbM6MqgQQggB6khStl8jfd/oVUlbAj2Bxzo6sBBCCKHN032STiHdCfcc6e45SN9j2qUD4wohhBDquiZ1GrC97X91dDAhhBBCUT3XpOZR9f2eEEIIYXWoZyT1JPCApNspPG7H9kUdFlUIIYRAfUnqb/m1Tn6FEEIIq0WbScr2t1dHICGEEEK1VpOUpEtsny7pdyz/VHJsH1pjsxBCCKHdrGgkdV3++cPVEUgoj5ZnFtBw9u2dHUYInSaex1cerSYp25PzzwdbqxNCCCF0pHqm6thW0q8kzZb0ZOW1OoLrKiSNlnREZ8cRQgjdTT3fk7qWNHX5G8B+wC9I8yOFEEIIHaqeJLWu7XtJc089Zftc0rQdaxxJDZIelXSVpFmS7pa0blWdcyRNkjRT0pWVqeur6mwi6Z7cxtWSnpK0cV53Rt52pqTTc9n5kr5Y2P5cSWdWtXli7ne6pF/nebGWG+VJWtiuByWEEDpQPUlqsaS1gL9I+pKkjwJ9OziuMtsW+Knt95Pmm/p41fpLbQ+zvROwLnBIjTa+BdyX2/gVsCWApKGkmYY/AOwBnChpV+Bm0tPoKz6Ry4puzf0OBh4FTliZnZJ0kqRmSc1LX4sHjIQQyqGeJHUa0Ac4lTQh4LHAcR0ZVMnNsT0tL08GGqrW7ydpQp7qfX/g/TXaGA7cBGD7TuClQvlY26/aXgjcCozI08K/S9JmkgYDL9meV9XmTpKacr/HtNJvq2xfabvRdmOPPv1XZtMQQugw9XyZd1JeXEj6K39Nt7iwvJQ0WgJAUm/gMqDR9jxJ5wK926nfMcARwLtZfhQFMBo43PZ0SaOAfXP5G+Q/RvKIOJ4aEkLoMuq5u69R0lhJUyTNqLxWR3BdUCUhvSCpLymp1DKefPpO0oHABrm8CThcUh9J6wEfzWWQEtNRuc0xNdrsBzwrqSdpJFUxlzQCBjiUNB9YCCF0CfU8u+8G0ky8Lbw1n1SowfZ8SVcBM4F/AJVRKJJOznUuB74N3CjpWODhXPcV21MkjQYm5s2uzqf6sD1LUj/gGdvP5javBi633Qx8E5gAPJ9/9sttXAX8VtJ04E7g1Y7a/xBCaG+yl3vi0dsrSH+yPXw1xbNGkNQLWGr7DUl7Aj+zPaSTw3pTr4HbeuBxl3R2GCF0mnjixOolabLtxlrr6hlJfSv/xX4vb5+q49Z2im9NtCVwS75G9G/gxE6O52123rw/zfGfNIRQAvUkqeOBQaRrGcXp4yNJrSLbfwF27ew4Qgih7OpJUsNsb9/hkYQQQghV6vme1EOSduzwSEIIIYQq9Yyk9gCmSZpDuiYlwLZ36dDIQgghrPHqSVIHdXgUIYQQQg1tnu6z/RQwAPhIfg3IZSGEEEKHqueJE6eRvtD7rvy6XtIpHR1YCCGEUM/pvhOAD9h+FUDSD0hPSfhJRwYWQggh1HN3n0gPUq1YmstCCCGEDlXPSOpaYIKksfn94cD/dVhEIYQQQtbms/sAJO1GmusIoKny0NPQPcWz+0J3EM/f6zpW6dl9kjYsvJ2bX2+us/1iewUYQggh1LKi032TSc/oE+mBqC/l5QHA34D3dXRwa5r8dPTbgY2B79uuNblhCCGsMVpNUrbfB5DnRxpr+478/sOk61KhQNLatt94h83sClCmaTtCCKEz1XN33x6VBAVg+w/AXh0X0jsn6SxJp+bliyXdl5f3l/SqpIvy+9MkPZmXt5I0XtIwSbfmssMkvS5pHUm9K3UL/YySdFtu/15JAyWNkzRN0kxJIyT1kDQ6v2+R9OW87amSZueZjm+S9C7gemBY3n7rQj89JP0wtzGj8j01SR+SNDW3e00eiSFprqTv53aaJe0m6S5JT1QmXwwhhK6gnrv7/i7pG6QPUEhTk/+940JqF03AV4AfA41Arzyt+gjgAuCQXG8E8C9Jm+flccBUYEhh/UxgGOlYTajR127ALrZflPQV4C7b35XUA+iT29rc9k4Akgbk7c4G3md7saQBeVbfzwJn2j6kqo+TgAZgSJ4ocUNJvYHRwIds/1nSL4DPA5fkbf5me4iki3O9vUnT288ELq/eCUkn5X7osf4mtY5pCCGsdvWMpI4GNgHGkuaQ2iSXldlkYKik9UkPxX2YlKxGAGOAvnkq9i2AXwL75HVN+ZTdE5J2AHYHLiqur9HXPYWbSCYBx0s6F9jZ9ivAk8BWkn4i6SDg5Vx3BnCDpE8BbZ0mPAC4onI6Mfe3PTDH9p9znZ/nOCtuyz9bgAm2X7H9PLC4kCjfZPtK2422G3v06d9GOCGEsHrU8+y+F22fZntX27vZPr3sd/bZXgLMAUYBD5GSy37ANsCjuex44PG8bgSwJzA+NzEO+DCwBPgj6fb74dROUq8W+h1HShTPAKMlfdr2S8Bg4AHgZODqXP1g4KekkdgkSfWMaldGZRblZYXlyvv27iuEEDpEPSOprqoJOJOUcJpICWKq0xfDiuumkhLYYtsLCtueDjycRx8bkUYuM1fUoaT3As/ZvoqUjHaTtDGwlu1fA9/IZWsBW9i+H/ga0B/ou4Km7wE+V0lk+esBjwMNkrbJdY4FHqznwIQQQlfRnf+ibgL+h5RoXpW0iLdGQk2kU33jbC+VNA94rLDtBGBTUhKDdGru3bYt6VCg0fY5NfrcFzhL0hJgIfBpYHPg2pyYAL4O9CA9qLc/6bb+H+drUm82VNXP1cB2wIzc9lW2L5V0PDAmJ69J1LjWFEIIXVldT5wIa5Z44kToDuKJE13HKj1xorDxVsCPSNdslpFuQviy7SdXuGHosnbevD/N8R88hFAC9VyT+iVwC/BuYDPS3XE3dmRQIYQQAtSXpPrYvs72G/l1Pen7NiGEEEKHqufGiT9IOhu4ifQsv5HAHZUH0Jb9dvQQQghdVz1J6hP55+eqyo8iJa2t2jWiEEIIIWszSVUeNBtCCCGsbvXc3deT9Ey4yiN3HiA9omdJB8YVQggh1HW672dAT+Cy/P7YXPbZjgoqhBBCgBXPzFuZH2mY7cGFVfdJmt7xoYUQQljTregW9In559KquY22ApZ2aFQhhBACKz7dV3mQ3JnA/YUJ/xpITxAPIYQQOtSKktQmks7Iy1eQHooKaRS1K3B/RwYWOk/LMwtoOPv2zg4jhJUSz+rrnlaUpHqQpo9QVfnaQL8OiyiEEELIVpSknrX9ndUWyWoiaS5pCowXVncfefqNHW2f31F9hxBCd1LPNalQQ+Hux7rZvo23pnXvUKsSXwghlM2K7u770GqLYhVIOkvSqXn5Ykn35eX9Jd0g6UBJD0uaImmMpOLMt1+V1CJpYmFm22LbDZKa8rZTJO2Vy/fN5bcBsyX1kPRDSTMlzZB0SqGZU/K2LZIG5e1HSbpUUn9JT1UmQpS0nqR5knpK2lrSnZIm574G1Yivt6Rrc9tTJe1XaP+2fCzuldRH0i2SZksaK2mCpJpztoQQQhm1mqS6wINjm4ARebkR6JufjjGCNJPuN4ADbO8GNANnFLZdYHtn4FLgkhpt/xP4j7ztSODHhXW7AafZ3g44iXS34xDbuwA3FOq9kLf/GekOyTflaeqnAR/MRYcAd+WneFwJnGJ7aN7uMpb3xdSMdwaOBn4uqfJk+t2AI2x/EPgC8JLtHYFvAkNrtAWApJMkNUtqXvragtaqhRDCatWVp4+fDAyVtD6wGJhCSlYjSKfUdgTG5ynZ1yFN1lhxY+HnxTXa7glcKmkI6W7G7QrrJtqek5cPAC6vnFarSuy3FuL8WI0+biYlwPtJD+u9LI/29iJNCV+p16vGtsOBn+Q+H5P0VCHGewpxDCdNWIntmZJm1GiLvP5KUoKk18BtY7rmEEIpdNkkZXuJpDnAKOAh0uhpP2AbYA7pw/ro1jZvZbniy8BzwGDSaHNRYd2rdYa4OP9cSu3jfBvwvTzlyVDgPmA9YL7tIXX2UUu98YUQQunVM+lhmTWRTomNy8snA1OBR4C9K9eb8jWf4mhoZOFncYRV0Z90d+My0rMKe9SoA3AP8DlJa+d+Nqw3cNsLgUmkkc7vbS+1/TIwR9KRuT1JGlxj8ybgmFxnO2BL4PEa9caTp1qRtCOwc73xhRBCGXSHJDUQeNj2c6QRT5Pt50kjrBvzKa6HgeINCBvk8tNIoyYkHSqpcsv9ZcBx+RmFg2h9dHI18DdgRq77yZWM/2bgU/lnxTHACbm9WcBhrcS3lqSWvO0o24tZ3mWkL2XPBs7L7cUFpxBClyE7Lj90V5J6AD1tL8rPX/wjsL3tf69ou8bGRjc3N6+WGEMIQdJk2zXvPO6y16RCXfqQnrvYk/S9ty+0laBCCKFMIkl1Y7ZfId3xGEIIXVJXvyYVQgihG4skFUIIobQiSYUQQiitSFIhhBBKK5JUCCGE0ookFUIIobQiSYUQQiitSFIhhBBKK77MG5bT8swCGs6+vbPDCKvZ3PMP7uwQQlhOjKRCCCGUViSpEEIIpdXlk5SkIZL+q456+0r6/eqIqZX+z5V0Zts1O6TvuZI27oy+QwjhnejySQoYArSZpNpTngIjhBBCB+uwJCXpLEmn5uWLJd2Xl/eX9Kqki/L70yQ9mZe3kjRe0jBJt+aywyS9LmkdSb0rdfO6dYDvACMlTZM0Ms/Ce42kiZKmSjqsRmw160jqI+kWSbMljZU0QVJjXrdQ0v/myQj3lHSOpEmSZkq6UpJyvQck/SjHM1PS7oWud8zrn6wcmxqxDZP0kKTpOb5+eb+vldSS492vxnZvGylKulTSqKo660r6g6QT2/wFhhBCCXTk3X1NwFeAH5Omi+iV5zUaAVwAHJLrjQD+JWnzvDyONAX8kML6mcCwHO+ESge2/y3pHKDR9pcAJH0PuM/2ZyQNACZK+mNVbP/TSp3PAy/Z3lHSTsC0wjbrARNsfyX3M9v2d/LydXl/fpfr9rE9RNI+wDXATrl8ELAf0A94XNLPbC+pdJCT7s3ASNuTJK0PvE6aQdi2d5Y0CLhb0na2F63oF1ClL3AT8Avbv6heKekk4CSAHutvshLNhhBCx+nI032TgaH5g3YxaQr3RlLSGQP0ldQP2AL4JbBPXtdk+w3gCUk7ALsDFxXXt9HvgcDZkqYBDwC9gS3rrDOc9EGO7ZnAjMI2S4FfF97vl0daLcD+wPsL627MbYwD1s+JEOB224ttvwD8E9i0Kq7tgWdtT8rbv5yPxXDg+lz2GPAUsF0bx6Hab4FrayWo3O6VthttN/bo038lmw4hhI7RYSMp20skzQFGAQ+RPvD3A7YBHs1lxwOPkxLPZ4A9SaMvSCOqDwNLSNOejwZ6AGe10bWAj9t+/G2F0qZ11FlRu4tsL831egOXkUZw8ySdS0p0b+5+1baV94sLZUtp3+P/Bm//o6N31frxwEGSfmm7Or4QQiiljr5xogk4k5RwmoCTgan5Q7K4biopgS22vaCw7enAw7afBzYijTRmVvXxCun0WcVdwCmFa0S71oirtTrjgU/ksh2BnVvZr0oCeEFSX+CIqvUjcxvDgQWFfWrL48BAScPy9v0krU06Fsfksu1Io77Hq7Z9inTNq1ceuX2oav05wEvAT+uMJYQQOt3qSFIDSYnmOWARb52uayKd6huXRyjzgD8Vtp1AOh02Lr+fAbTYtqRDJX0nl99P+nCeJmkk8P+AnsAMSbPy+2qt1bkM2ETSbOA8YBawXIKxPR+4ipQw7wImVVVZJGkqcDlwwgqODwCS7pC0me1/kxLcT/INGveQEuJlwFr51OLNwCjbiyU1Sro6xzQPuCXHdAsp8Vc7DVhX0gVtxRRCCGWgOPPzFqVby3vaXiRpa9Jpxu1z8qi3jQeAM203d1CYHa6xsdHNzV02/BBCFyNpsu3GWuvi2X1v1we4P9+FKOALK5OgQgghtK9IUgW2XyHdgfhO2ti3faIJIYTQHZ44EUIIoZuKJBVCCKG0IkmFEEIorUhSIYQQSiuSVAghhNKKJBVCCKG0IkmFEEIorUhSIYQQSiu+zBuW0/LMAhrOvr2zwwgdbO75B3d2CCG0KUZSIYQQSiuSVAghhNKKJLWGkDRK0madHUcIIayMSFJV8nQdpZMnP3wnRgGRpEIIXcpqS1KSzpJ0al6+WNJ9eXl/Sa9Kuii/P03Sk3l5K0njJQ2TdGsuO0zS65LWkdS7UrfQTw9JF0qaJGmGpM/l8n0l/b5Q71JJo/LyXEk/kDQFOFLS0ZJaJM2U9IPCNidI+rOkiZKuknRpLv+IpAmSpkr6Y2WqeknnSvq5pCZJT0n6mKQLctt35ilBkDRU0oOSJku6S9LAXP6ApEskNZMmLKSwLw9I+pWkxyTdUJhleLm2JB1Berr7DXlyyHXb7RcbQggdaHWOpJqAEXm5EeibP6RHABcU1o0A/iVp87xcmV5+SGH9TGAY8AHSDL5FJ5CmbB+W65wo6X11xPcv27vl/n4A7J/7HCbp8Hyq7JvAHsDewKDCtn8C9rC9K3AT8NXCuq1zW4cC1wP3294ZeB04OB+DnwBH2B4KXAN8t7D9OrYbbf9vVby7AqcDOwJbAXu31pbtXwHNwDG2h9h+vXrnJZ0kqVlS89LX6p3tPoQQOtbqvAV9MjBU0vrAYmAKKVmNAE4FjpLUjzSl/C+BffK6W22/IekJSTsAuwMX5fU9eGs6+ooDgV3y6AGgP7At0NbkhTfnn8OAB2w/DyDphtwXwIO2X8zlY4Dtcvl7gJvzCGgdYE6h3T/YXpKnfu8B3JnLW4AGYHtgJ+CePBjqATxbI65qE20/nWOZltua30ZbrbJ9JXAlQK+B28Z0zSGEUlhtSSp/UM8hXRt5CJgB7AdsAzyay44HHiclns8AewJfyU2MAz4MLCFN6z6a9CF8VlVXAk6xfdfbCqXhvH3k2Ltqu1dXeefS6OUi27dJ2hc4t7BuMYDtZZKW2K4kgGWk4y9glu09W2m7tbgWF5aX1tlWCCF0Kav7xokm4ExSwmkCTgam5g/u4rqppAS22PaCwranAw/nUc5GpFHIzKo+7gI+X7jes52k9YCngB0l9ZI0APhQKzFOBD4oaeN8E8XRwIPApFy+Qb6J4eOFbfoDz+Tl41bukPA4sImkPXO8PSW9fyXbqKetV4B+q9huCCF0is5IUgNJieY5YBFvna5rIp3qG2d7KTCPdK2nYgKwKSmJQRqJtdi2pEMlfSeXXw3MBqZImglcAaxtex5wCymp3UJKhMux/SxwNnA/MB2YbPu3tp8BvkdKYuOBuUAlgZ4LjJE0GXhhZQ6I7X8DRwA/kDQdmAbsVV1PUqOkq99BW6OBy+PGiRBCV6K3zj6Ftkjqa3thHkmNBa6xPbaz42pvjY2Nbm5u7uwwQghrCEmTbTfWWhffk1o55+abFGaSbo74TadGE0II3Vw8YHYl2D6zs2MIIYQ1SYykQgghlFYkqRBCCKUVSSqEEEJpRZIKIYRQWpGkQgghlFYkqRBCCKUVSSqEEEJpRZIKIYRQWvFl3rCclmcW0HD27Z0dRrc09/yDOzuEELqUGEmFEEIorUhSIYQQSiuSVCeQ1JCnEQkhhLACkaRCCCGUVumTVB51PCrpKkmzJN0taV1JW0u6U9JkSU2SBknqIWmOkgGSlkraJ7czTtK2kj6YJ/6bJmmqpOVmq5V0gqQ/S5qY+720EMt9kmZIulfSlm2Uby3pEUktks6TtLBGXz0kXShpUt7+c60ch4MkTZE0XdK9uWxDSb/J2z0iaZdcfq6kn+fj8pSkj0m6IMdxZ2XW4hBCKLvSJ6lsW+Cntt8PzCdN3X4lcIrtoaRp5y/LM/o+DuwIDAemACMk9QK2sP2XXPeLtocAI4DXix1J2gz4JrAHsDcwqLD6J8DPbe8C3AD8uI3yHwE/sr0z8HQr+3YCsMD2MGAYcKKk91XFtAlwFfBx24OBI/OqbwNTc7//DfyisNnWwP7AocD1wP05jteB5W4xk3SSpGZJzUtfW1C9OoQQOkVXSVJzbE/Ly5OBBtK06GPyJIRXkKalhzQN/T759X1SshoGTMrrxwMXSToVGGD7jaq+dgcetP2i7SXAmMK6PYFf5uXrctttlVe2r6yvdiDw6bwfE4CNSEm5aA9gnO05ALZfzOXDc3/Yvg/YSNL6ed0fcvwtQA/gzlzeQjp+b2P7StuNtht79OnfSqghhLB6dZUktbiwvBTYEJhve0jhtUNeP440QtoduAMYAOxLSl7YPh/4LLAuMF5ScaTUGUQaEVb24322726HdhcD2F4GLLHtXL6M+H5cCKGL6CpJqtrLwBxJRwLka1CD87qJpFHWMtuLgGnA50jJC0lb226x/QPS6Ko6SU0CPihpA0lrk04tVjwEHJWXjyEnvhWUP1LYvrK+2l3A5yvXiSRtJ2m9qjqPAPtUTgNK2jCXN+X+kLQv8ILtl1vpJ4QQupyumqQgfTifIGk6MAs4DMD2YmAe6YMd0gd5P9JpLoDTJc2UNANYAvwBIJ9uw/YzwPdIyW48MBeoXKQ5BTg+b3sscFob5acDZ+TybQrtFF0NzAam5NvSryCPdAoxPQ+cBNya9/fmvO25wNDc/vnAcXUctxBC6DL01lmgUCGpr+2FeSQ1FrjG9thVaKcP8LptSzoKONr2Ye0db3trbGx0c3NzZ4cRQlhDSJpsu7HWurg2Udu5kg4AegN3A79ZxXaGApdKEumuxM+0S3QhhLCGiCRVg+0z26mdJmBwmxVDCCHU1JWvSYUQQujmIkmFEEIorUhSIYQQSivu7gvLkfQK6fFS3dnGwAudHUQHiv3r+rr7Phb37722N6lVKW6cCLU83trtoN2FpObuvI+xf11fd9/HevcvTveFEEIorUhSIYQQSiuSVKjlys4OYDXo7vsY+9f1dfd9rGv/4saJEEIIpRUjqRBCCKUVSSqEEEJpRZIKbyPpIEmPS/qrpLM7O572JOkaSf/MU6J0O5K2kHS/pNmSZkk6re2tuhZJvSVNlDQ97+O3OzumjiCph6Spkn7f2bF0BElzJbVImiZphVMuxDWp8CZJPYA/A/8BPE2aAPJo27M7NbB2ImkfYCHwC9s7dXY87U3SQGCg7SmS+gGTgcO7y+8P0gSnwHp5Kp2ewJ+A02w/0samXYqkM4BGYH3bh3R2PO1N0lyg0XabX1aOkVQo2h34q+0nbf8buIk8mWR3YHsc8GJnx9FRbD9re0pefgV4FNi8c6NqX04W5rc986tb/aUt6T3AwaQJUdd4kaRC0eakWY0rnqabfcitKSQ1ALsCEzo5lHaXT4VNA/4J3GO7u+3jJcBXgWWdHEdHMnC3pMmSTlpRxUhSIXQzkvoCvwZOt/1yZ8fT3mwvtT0EeA+wu6Ruc+pW0iHAP21P7uxYOthw27sBHwa+mE/F1xRJKhQ9A2xReP+eXBa6iHyd5tfADbZv7ex4OpLt+cD9wEGdHEp72hs4NF+zuQnYX9L1nRtS+7P9TP75T2As6VJDTZGkQtEkYFtJ75O0DnAUcFsnxxTqlG8q+D/gUdsXdXY8HUHSJpIG5OV1STf5PNapQbUj21+3/R7bDaT/f/fZ/lQnh9WuJK2Xb+xB0nrAgUCrd9xGkgpvsv0G8CXgLtJF91tsz+rcqNqPpBuBh4HtJT0t6YTOjqmd7Q0cS/rre1p+/VdnB9XOBgL3S5pB+qPqHtvd8jbtbmxT4E+SpgMTgdtt39la5bgFPYQQQmnFSCqEEEJpRZIKIYRQWpGkQgghlFYkqRBCCKUVSSqEEEJpRZIKoZuQ9G5JN0l6Ij9u5g5J263G/kdJ2qyVdYPyLfFTJW0t6aE62lvYSvloSUe803hD1xBJKoRuIH+RdyzwgO2tbQ8Fvk76Tko926+9ovd1GgXUTFLA4cCvbO9q+wnbe61C+2ENFEkqhO5hP2CJ7csrBban225ScqGkmXkOn5EAkvaV1CTpNmB2jfc98naTJM2Q9LlK25K+ltuaLun8PLJpBG7II6Z1C3X/Czgd+Lyk+3PZwsL6swp9LDc/VI7/0jzP2R+BdxXWnZ/nz5oh6YftdjRDaazKX0shhPLZiTR/VC0fA4YAg4GNgUmSxuV1uwE72Z4jad+q9ycBC2wPk9QLGC/pbmAQaQqXD9h+TdKGtl+U9CXgTNtvm8TO9h2SLgcW2n5bIpF0ILAt6dltAm6TtE+eVqXio8D2wI6kkeFs4BpJG+V1g2y78rik0L1Ekgqh+xsO3Gh7KfCcpAeBYcDLwETbcwp1i+8PBHYpXP/pT0ooBwDX2n4NwPY7maPrwPyamt/3zX0Uk9Q+hfj/Lum+XL4AWAT8X57BNh6P1A1Fkgqhe5gFrMrNBK+u4L2AU2zfVawg6T9XoZ/WCPi+7StWdkPbb0jaHfgQad+/BOzfjrGFEohrUiF0D/cBvYoTyEnaRdIIoAkYma8xbUIamUyso827SNeReub2tstPrb4HOF5Sn1y+Ya7/CtBvJeO+C/hMngMLSZtLeldVnXGF+AeSrr9V5s3qb/sO4Muk05mhm4mRVAjdQL4m81HgEklfI50Gm0u6YeFPwJ7AdNKMqF+1/Q9Jg9po9mqgAZiS7x58Hjjc9p2ShgDNkv4N3AH8NzAauFzS68Cetl+vI+67Je0APJy6YCHwKdKsuxVjSSOk2cDfSE+yh5QQfyupN2lEdkZb/YWuJ56CHkIIobTidF8IIYTSiiQVQgihtCJJhRBCKK1IUiGEEEorklQIIYTSiiQVQgihtCJJhRBCKK3/D9MgMKsUoenfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "success_by_domain = scores.groupby(\"domain\")[\"citoid_success\"].mean().sort_values().reset_index()\n",
    "\n",
    "plot_data = success_by_domain.set_index(\"domain\").loc[top_domains.index]\n",
    "plot_data = plot_data.sort_values(by=\"citoid_success\")\n",
    "ax = plot_data.plot.barh(legend=False)\n",
    "\n",
    "ax.set_title(\"Citoid success for top domains\")\n",
    "ax.set_ylabel(\"Top domains\")\n",
    "ax.set_xlabel(\"Correct fields\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/citoid_success_top_domains.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best performance is for cricket.archive.com, www.nytimes.com and news.bbc.co.uk. The performance for books.google.com, the most cited domain, shows ~4 correct fields (60% success)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Google books"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>domain</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>success%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7068</th>\n",
       "      <td>books.google.com</td>\n",
       "      <td>3.613952</td>\n",
       "      <td>60.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                domain  citoid_success  success%\n",
       "7068  books.google.com        3.613952      60.0"
      ]
     },
     "execution_count": 374,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain = \"books.google.com\"\n",
    "success_for_domain = success_by_domain[success_by_domain[\"domain\"] == domain].copy()\n",
    "success_for_domain[\"success%\"] = round(success_for_domain[\"citoid_success\"]/6*100)\n",
    "success_for_domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see some examples of manual metadata vs. Citoid's metadata for citations from books.google.com:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [],
   "source": [
    "books_google_sample = scores[scores.domain == \"books.google.com\"].sample(10).index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_url</th>\n",
       "      <th>url</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>citation_template</th>\n",
       "      <th>title_citoid</th>\n",
       "      <th>title_manual</th>\n",
       "      <th>author_first_citoid</th>\n",
       "      <th>author_first_manual</th>\n",
       "      <th>author_last_citoid</th>\n",
       "      <th>author_last_manual</th>\n",
       "      <th>pub_date_citoid</th>\n",
       "      <th>pub_date_manual</th>\n",
       "      <th>pub_source_citoid</th>\n",
       "      <th>pub_source_manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14004</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Michael%20Jackson</td>\n",
       "      <td>https://books.google.com/books?id=L70DAAAAMBAJ...</td>\n",
       "      <td>2</td>\n",
       "      <td>book</td>\n",
       "      <td>cite magazine</td>\n",
       "      <td>jet</td>\n",
       "      <td>[michael jackson turns 30!]</td>\n",
       "      <td>[johnson publishing]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[company]</td>\n",
       "      <td>[]</td>\n",
       "      <td>1988-08-29</td>\n",
       "      <td>[1988-08-29]</td>\n",
       "      <td>johnson publishing company</td>\n",
       "      <td>[jet, johnson publishing company]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46601</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Marshall%20Apple...</td>\n",
       "      <td>https://books.google.com/books?id=M2QEAAAAMBAJ...</td>\n",
       "      <td>1</td>\n",
       "      <td>book</td>\n",
       "      <td>citar periódico</td>\n",
       "      <td>the advocate</td>\n",
       "      <td>[heaven's scapegoat]</td>\n",
       "      <td>[here]</td>\n",
       "      <td>[mubarak]</td>\n",
       "      <td>[publishing]</td>\n",
       "      <td>[dahir]</td>\n",
       "      <td>1997-05-13</td>\n",
       "      <td>[1997-05-13]</td>\n",
       "      <td>here publishing</td>\n",
       "      <td>[the advocate]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10416</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Operation%20Cros...</td>\n",
       "      <td>https://books.google.com/books?id=V04EAAAAMBAJ...</td>\n",
       "      <td>0</td>\n",
       "      <td>book</td>\n",
       "      <td>citation</td>\n",
       "      <td>life</td>\n",
       "      <td>[what science learned at bikini]</td>\n",
       "      <td>[time]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[inc]</td>\n",
       "      <td>[]</td>\n",
       "      <td>1947-08-11</td>\n",
       "      <td>[]</td>\n",
       "      <td>time inc</td>\n",
       "      <td>[life]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68239</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Drowning%20Girl</td>\n",
       "      <td>https://books.google.com/books?id=sPGdBxzaWj0C...</td>\n",
       "      <td>4</td>\n",
       "      <td>book</td>\n",
       "      <td>cite book</td>\n",
       "      <td>the grove encyclopedia of american art</td>\n",
       "      <td>[the grove encyclopedia of american art]</td>\n",
       "      <td>[joan m.]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[marter]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2011</td>\n",
       "      <td>[2011]</td>\n",
       "      <td>oxford university press</td>\n",
       "      <td>[oxford university press]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90315</th>\n",
       "      <td>https://en.wikipedia.org/wiki/George%20S.%20Pa...</td>\n",
       "      <td>https://books.google.com/books?id=6_KcBQAAQBAJ...</td>\n",
       "      <td>3</td>\n",
       "      <td>book</td>\n",
       "      <td>cite book</td>\n",
       "      <td>great leaders: george patton</td>\n",
       "      <td>[great leaders: george patton]</td>\n",
       "      <td>[willard sterne, nancy]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[randall, nahra]</td>\n",
       "      <td>[willard sterne randall, nancy nahra]</td>\n",
       "      <td>2014-11-28</td>\n",
       "      <td>[2014]</td>\n",
       "      <td>new word city</td>\n",
       "      <td>[new word city]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27793</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Paul%20McCartney</td>\n",
       "      <td>https://books.google.com/books?id=ZhDx0XKwzPkC...</td>\n",
       "      <td>4</td>\n",
       "      <td>book</td>\n",
       "      <td>cite book</td>\n",
       "      <td>the complete how to kazoo</td>\n",
       "      <td>[the complete how to kazoo]</td>\n",
       "      <td>[barbara]</td>\n",
       "      <td>[barbara]</td>\n",
       "      <td>[stewart]</td>\n",
       "      <td>[stewart]</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>[2006]</td>\n",
       "      <td>workman publishing</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46075</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Hurricane%20Juan...</td>\n",
       "      <td>http://www.wpc.ncep.noaa.gov/tropical/rain/tcm...</td>\n",
       "      <td>2</td>\n",
       "      <td>webpage</td>\n",
       "      <td>cite report</td>\n",
       "      <td>tropical cyclone rainfall for the midwest</td>\n",
       "      <td>[tropical cyclone rainfall in the midwest]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[david m. roth]</td>\n",
       "      <td>None</td>\n",
       "      <td>[2013-03-06]</td>\n",
       "      <td>www.wpc.ncep.noaa.gov</td>\n",
       "      <td>[weather prediction center]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23432</th>\n",
       "      <td>https://fr.wikipedia.org/wiki/Histoire%20de%20...</td>\n",
       "      <td>https://books.google.com/books?id=4Ait9MeAg8UC...</td>\n",
       "      <td>0</td>\n",
       "      <td>book</td>\n",
       "      <td>lien web</td>\n",
       "      <td>'pamela' in the marketplace: literary controve...</td>\n",
       "      <td>[''pamela in the marketplace'']</td>\n",
       "      <td>[thomas, elmore fellow and tutor in english la...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[keymer, keymer, sabor, sabor]</td>\n",
       "      <td>[keymer, sabor, , ]</td>\n",
       "      <td>2005-12-15</td>\n",
       "      <td>[]</td>\n",
       "      <td>cambridge university press</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51954</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Black%20stork</td>\n",
       "      <td>https://books.google.com/books?id=Sf7UBAAAQBAJ...</td>\n",
       "      <td>5</td>\n",
       "      <td>book</td>\n",
       "      <td>cite book</td>\n",
       "      <td>birds in scotland</td>\n",
       "      <td>[birds in scotland]</td>\n",
       "      <td>[valerie m.]</td>\n",
       "      <td>[valerie m.]</td>\n",
       "      <td>[thom]</td>\n",
       "      <td>[thom]</td>\n",
       "      <td>2010-11-30</td>\n",
       "      <td>[2010]</td>\n",
       "      <td>bloomsbury publishing</td>\n",
       "      <td>[london, bloomsbury publishing]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58144</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Japan</td>\n",
       "      <td>https://books.google.com/books?id=njnRAgAAQBAJ...</td>\n",
       "      <td>5</td>\n",
       "      <td>book</td>\n",
       "      <td>cite book</td>\n",
       "      <td>traditional japanese architecture: an explorat...</td>\n",
       "      <td>[traditional japanese architecture: an explora...</td>\n",
       "      <td>[mira]</td>\n",
       "      <td>[mira]</td>\n",
       "      <td>[locher]</td>\n",
       "      <td>[locher]</td>\n",
       "      <td>2012-04-17</td>\n",
       "      <td>[2012]</td>\n",
       "      <td>tuttle publishing</td>\n",
       "      <td>[tuttle publishing]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             article_url  \\\n",
       "14004    https://en.wikipedia.org/wiki/Michael%20Jackson   \n",
       "46601  https://pt.wikipedia.org/wiki/Marshall%20Apple...   \n",
       "10416  https://en.wikipedia.org/wiki/Operation%20Cros...   \n",
       "68239      https://en.wikipedia.org/wiki/Drowning%20Girl   \n",
       "90315  https://en.wikipedia.org/wiki/George%20S.%20Pa...   \n",
       "27793     https://en.wikipedia.org/wiki/Paul%20McCartney   \n",
       "46075  https://en.wikipedia.org/wiki/Hurricane%20Juan...   \n",
       "23432  https://fr.wikipedia.org/wiki/Histoire%20de%20...   \n",
       "51954        https://en.wikipedia.org/wiki/Black%20stork   \n",
       "58144                https://en.wikipedia.org/wiki/Japan   \n",
       "\n",
       "                                                     url  citoid_success  \\\n",
       "14004  https://books.google.com/books?id=L70DAAAAMBAJ...               2   \n",
       "46601  https://books.google.com/books?id=M2QEAAAAMBAJ...               1   \n",
       "10416  https://books.google.com/books?id=V04EAAAAMBAJ...               0   \n",
       "68239  https://books.google.com/books?id=sPGdBxzaWj0C...               4   \n",
       "90315  https://books.google.com/books?id=6_KcBQAAQBAJ...               3   \n",
       "27793  https://books.google.com/books?id=ZhDx0XKwzPkC...               4   \n",
       "46075  http://www.wpc.ncep.noaa.gov/tropical/rain/tcm...               2   \n",
       "23432  https://books.google.com/books?id=4Ait9MeAg8UC...               0   \n",
       "51954  https://books.google.com/books?id=Sf7UBAAAQBAJ...               5   \n",
       "58144  https://books.google.com/books?id=njnRAgAAQBAJ...               5   \n",
       "\n",
       "      source_type_citoid citation_template  \\\n",
       "14004               book     cite magazine   \n",
       "46601               book   citar periódico   \n",
       "10416               book          citation   \n",
       "68239               book         cite book   \n",
       "90315               book         cite book   \n",
       "27793               book         cite book   \n",
       "46075            webpage       cite report   \n",
       "23432               book          lien web   \n",
       "51954               book         cite book   \n",
       "58144               book         cite book   \n",
       "\n",
       "                                            title_citoid  \\\n",
       "14004                                                jet   \n",
       "46601                                       the advocate   \n",
       "10416                                               life   \n",
       "68239             the grove encyclopedia of american art   \n",
       "90315                       great leaders: george patton   \n",
       "27793                          the complete how to kazoo   \n",
       "46075          tropical cyclone rainfall for the midwest   \n",
       "23432  'pamela' in the marketplace: literary controve...   \n",
       "51954                                  birds in scotland   \n",
       "58144  traditional japanese architecture: an explorat...   \n",
       "\n",
       "                                            title_manual  \\\n",
       "14004                        [michael jackson turns 30!]   \n",
       "46601                               [heaven's scapegoat]   \n",
       "10416                   [what science learned at bikini]   \n",
       "68239           [the grove encyclopedia of american art]   \n",
       "90315                     [great leaders: george patton]   \n",
       "27793                        [the complete how to kazoo]   \n",
       "46075         [tropical cyclone rainfall in the midwest]   \n",
       "23432                    [''pamela in the marketplace'']   \n",
       "51954                                [birds in scotland]   \n",
       "58144  [traditional japanese architecture: an explora...   \n",
       "\n",
       "                                     author_first_citoid author_first_manual  \\\n",
       "14004                               [johnson publishing]                  []   \n",
       "46601                                             [here]           [mubarak]   \n",
       "10416                                             [time]                  []   \n",
       "68239                                          [joan m.]                  []   \n",
       "90315                            [willard sterne, nancy]                  []   \n",
       "27793                                          [barbara]           [barbara]   \n",
       "46075                                               None                  []   \n",
       "23432  [thomas, elmore fellow and tutor in english la...                  []   \n",
       "51954                                       [valerie m.]        [valerie m.]   \n",
       "58144                                             [mira]              [mira]   \n",
       "\n",
       "                   author_last_citoid                     author_last_manual  \\\n",
       "14004                       [company]                                     []   \n",
       "46601                    [publishing]                                [dahir]   \n",
       "10416                           [inc]                                     []   \n",
       "68239                        [marter]                                     []   \n",
       "90315                [randall, nahra]  [willard sterne randall, nancy nahra]   \n",
       "27793                       [stewart]                              [stewart]   \n",
       "46075                            None                        [david m. roth]   \n",
       "23432  [keymer, keymer, sabor, sabor]                    [keymer, sabor, , ]   \n",
       "51954                          [thom]                                 [thom]   \n",
       "58144                        [locher]                               [locher]   \n",
       "\n",
       "      pub_date_citoid pub_date_manual           pub_source_citoid  \\\n",
       "14004      1988-08-29    [1988-08-29]  johnson publishing company   \n",
       "46601      1997-05-13    [1997-05-13]             here publishing   \n",
       "10416      1947-08-11              []                    time inc   \n",
       "68239            2011          [2011]     oxford university press   \n",
       "90315      2014-11-28          [2014]               new word city   \n",
       "27793      2006-01-01          [2006]          workman publishing   \n",
       "46075            None    [2013-03-06]       www.wpc.ncep.noaa.gov   \n",
       "23432      2005-12-15              []  cambridge university press   \n",
       "51954      2010-11-30          [2010]       bloomsbury publishing   \n",
       "58144      2012-04-17          [2012]           tuttle publishing   \n",
       "\n",
       "                       pub_source_manual  \n",
       "14004  [jet, johnson publishing company]  \n",
       "46601                     [the advocate]  \n",
       "10416                             [life]  \n",
       "68239          [oxford university press]  \n",
       "90315                    [new word city]  \n",
       "27793                                 []  \n",
       "46075        [weather prediction center]  \n",
       "23432                                 []  \n",
       "51954    [london, bloomsbury publishing]  \n",
       "58144                [tuttle publishing]  "
      ]
     },
     "execution_count": 338,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "books_google_sample_scores = comparison.loc[books_google_sample]\n",
    "\n",
    "books_google_sample_scores[\"citoid_success\"] = (books_google_sample_scores[comp_fields] >= 0.75).sum(axis=1)\n",
    "\n",
    "cols = ['article_url','url',\"citoid_success\", 'source_type_citoid', 'citation_template', \n",
    "        'title_citoid', 'title_manual', \n",
    "        'author_first_citoid', 'author_first_manual',\n",
    "        'author_last_citoid', 'author_last_manual', \n",
    "        'pub_date_citoid','pub_date_manual', \n",
    "        'pub_source_citoid', 'pub_source_manual']\n",
    "\n",
    "books_google_sample_scores[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://fr.wikipedia.org/wiki/Histoire%20de%20Tom%20Jones%2C%20enfant%20trouv%C3%A9'"
      ]
     },
     "execution_count": 431,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "books_google_sample_scores.loc[23432].article_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [],
   "source": [
    "books_google_sample_scores[cols].to_csv(\"eval/books_google_sample.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick inspection of the pages referred by the URLs shows that in many cases (for instance, items 14004, 46601, 10416), the errors are a consequence of part/whole identification of the source type: while manual citations focus on an a specific article of a periodical publication, Citoid returns the metadata for the whole magazine/newspaper and, for this reason, the values for the rest of the fields can also be different (title of the article vs. title of the magazine, publishing source as the magazine name vs. publishing source as the publisher, etc.). \n",
    "\n",
    "In the case of the publishing source for items 27793 and 23432, the automatic metadata for the publishing source is correct, but the manual metadata shows no information, hence Citod's answer is penalized. In some cases, it might be possible that the missing manual metadata is due to extraction problems. This could be solved by adding more parameter names to the [Citation templates spreadsheet](https://docs.google.com/spreadsheets/d/1xbc3FKE0m4JQHa6WCXtBbzeJ9in8P0EQ2NF_VNsaBaM/edit#gid=0) (see 2. Citation template metadata extraction)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### The New York times\n",
    "\n",
    "Now we show some examples for www.nytimes.com:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>domain</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>success%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11386</th>\n",
       "      <td>www.nytimes.com</td>\n",
       "      <td>4.32513</td>\n",
       "      <td>72.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                domain  citoid_success  success%\n",
       "11386  www.nytimes.com         4.32513      72.0"
      ]
     },
     "execution_count": 378,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain = \"www.nytimes.com\"\n",
    "success_for_domain = success_by_domain[success_by_domain[\"domain\"] == domain].copy()\n",
    "success_for_domain[\"success%\"] = round(success_for_domain[\"citoid_success\"]/6*100)\n",
    "success_for_domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated before, the performance for this domain is among the best, with a score of more than 70% correct answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [],
   "source": [
    "nytimes_sample = scores[scores.domain == \"www.nytimes.com\"].sample(10).index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_url</th>\n",
       "      <th>url</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>citation_template</th>\n",
       "      <th>title_citoid</th>\n",
       "      <th>title_manual</th>\n",
       "      <th>author_first_citoid</th>\n",
       "      <th>author_first_manual</th>\n",
       "      <th>author_last_citoid</th>\n",
       "      <th>author_last_manual</th>\n",
       "      <th>pub_date_citoid</th>\n",
       "      <th>pub_date_manual</th>\n",
       "      <th>pub_source_citoid</th>\n",
       "      <th>pub_source_manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>482</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Kim%20Clijsters</td>\n",
       "      <td>https://www.nytimes.com/2001/03/18/sports/tenn...</td>\n",
       "      <td>5</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite web</td>\n",
       "      <td>tennis; serena williams wins as the boos pour ...</td>\n",
       "      <td>[tennis; serena williams wins as the boos pour...</td>\n",
       "      <td>[selena]</td>\n",
       "      <td>[selena]</td>\n",
       "      <td>[roberts]</td>\n",
       "      <td>[roberts]</td>\n",
       "      <td>2001-03-18</td>\n",
       "      <td>[2001-03-18]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26287</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Jake%20Gyllenhaal</td>\n",
       "      <td>https://www.nytimes.com/2016/07/15/theater/sun...</td>\n",
       "      <td>6</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite news</td>\n",
       "      <td>'sunday in the park with george,' with jake gy...</td>\n",
       "      <td>[''sunday in the park with george'', with jake...</td>\n",
       "      <td>[michael]</td>\n",
       "      <td>[michael]</td>\n",
       "      <td>[paulson]</td>\n",
       "      <td>[paulson]</td>\n",
       "      <td>2016-07-14</td>\n",
       "      <td>[2016-07-14]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56926</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Antonin%20Scalia</td>\n",
       "      <td>https://www.nytimes.com/2015/12/03/opinion/jus...</td>\n",
       "      <td>2</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>citation</td>\n",
       "      <td>opinion | justice scalia's majoritarian theocracy</td>\n",
       "      <td>[justice scalia's majoritarian theocracy]</td>\n",
       "      <td>[richard a., eric j.]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[posner, segall]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2015-12-02</td>\n",
       "      <td>[2015-12-02]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71410</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Richard%20Cordray</td>\n",
       "      <td>https://www.nytimes.com/2012/01/05/us/politics...</td>\n",
       "      <td>3</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite web</td>\n",
       "      <td>bucking senate, obama appoints consumer chief</td>\n",
       "      <td>[bucking senate, obama appoints consumer chief]</td>\n",
       "      <td>[helene, jennifer]</td>\n",
       "      <td>[helene, ]</td>\n",
       "      <td>[cooper, steinhauer]</td>\n",
       "      <td>[cooper, steinhauer, jennifer]</td>\n",
       "      <td>2012-01-04</td>\n",
       "      <td>[2012-01-04]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12828</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Orel%20Hershiser...</td>\n",
       "      <td>https://www.nytimes.com/1988/10/07/sports/the-...</td>\n",
       "      <td>2</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite web</td>\n",
       "      <td>the playoffs; troubled cone stops the press</td>\n",
       "      <td>[the playoffs; troubled cone stops the press]</td>\n",
       "      <td>[joseph]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[durso]</td>\n",
       "      <td>[durso, joseph]</td>\n",
       "      <td>1988-10-07</td>\n",
       "      <td>[1988-10-07]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3436</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Huey%20Long</td>\n",
       "      <td>https://www.nytimes.com/1982/07/11/books/ameri...</td>\n",
       "      <td>6</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite news</td>\n",
       "      <td>american demagogues</td>\n",
       "      <td>[american demagogues]</td>\n",
       "      <td>[robert]</td>\n",
       "      <td>[robert]</td>\n",
       "      <td>[sherrill]</td>\n",
       "      <td>[sherrill]</td>\n",
       "      <td>1982-07-11</td>\n",
       "      <td>[1982-07-11]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71421</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Oklahoma%20City%...</td>\n",
       "      <td>https://www.nytimes.com/1995/04/20/us/terror-o...</td>\n",
       "      <td>3</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite news</td>\n",
       "      <td>terror in oklahoma city: the investigation; at...</td>\n",
       "      <td>[at least 31 are dead, scores are missing afte...</td>\n",
       "      <td>[david]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[johnston]</td>\n",
       "      <td>[johnson, david]</td>\n",
       "      <td>1995-04-20</td>\n",
       "      <td>[1995-04-20]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15337</th>\n",
       "      <td>https://es.wikipedia.org/wiki/The%20Powerpuff%...</td>\n",
       "      <td>https://www.nytimes.com/2002/07/03/movies/film...</td>\n",
       "      <td>5</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>Cita web</td>\n",
       "      <td>film review; they have a tantrum, then save th...</td>\n",
       "      <td>[film review; they have a tantrum, then save t...</td>\n",
       "      <td>[stephen]</td>\n",
       "      <td>[stephen]</td>\n",
       "      <td>[holden]</td>\n",
       "      <td>[holden]</td>\n",
       "      <td>2002-07-03</td>\n",
       "      <td>[2002-07-03]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3769</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Moon</td>\n",
       "      <td>https://www.nytimes.com/2014/09/09/science/rev...</td>\n",
       "      <td>5</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite news</td>\n",
       "      <td>the moon comes around again</td>\n",
       "      <td>[the moon comes around again]</td>\n",
       "      <td>[natalie]</td>\n",
       "      <td>[natalie]</td>\n",
       "      <td>[angier]</td>\n",
       "      <td>[angier]</td>\n",
       "      <td>2014-09-08</td>\n",
       "      <td>[2014-09-07]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47030</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Mars</td>\n",
       "      <td>https://www.nytimes.com/2021/05/14/science/chi...</td>\n",
       "      <td>6</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>cite news</td>\n",
       "      <td>china's mars rover mission lands on the red pl...</td>\n",
       "      <td>[china's mars rover mission lands on the red p...</td>\n",
       "      <td>[steven lee, kenneth]</td>\n",
       "      <td>[steven lee, kenneth]</td>\n",
       "      <td>[myers, chang]</td>\n",
       "      <td>[myers, chang]</td>\n",
       "      <td>2021-05-14</td>\n",
       "      <td>[2021-05-14]</td>\n",
       "      <td>the new york times</td>\n",
       "      <td>[the new york times]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             article_url  \\\n",
       "482        https://en.wikipedia.org/wiki/Kim%20Clijsters   \n",
       "26287    https://en.wikipedia.org/wiki/Jake%20Gyllenhaal   \n",
       "56926     https://en.wikipedia.org/wiki/Antonin%20Scalia   \n",
       "71410    https://en.wikipedia.org/wiki/Richard%20Cordray   \n",
       "12828  https://en.wikipedia.org/wiki/Orel%20Hershiser...   \n",
       "3436           https://en.wikipedia.org/wiki/Huey%20Long   \n",
       "71421  https://en.wikipedia.org/wiki/Oklahoma%20City%...   \n",
       "15337  https://es.wikipedia.org/wiki/The%20Powerpuff%...   \n",
       "3769                  https://en.wikipedia.org/wiki/Moon   \n",
       "47030                 https://en.wikipedia.org/wiki/Mars   \n",
       "\n",
       "                                                     url  citoid_success  \\\n",
       "482    https://www.nytimes.com/2001/03/18/sports/tenn...               5   \n",
       "26287  https://www.nytimes.com/2016/07/15/theater/sun...               6   \n",
       "56926  https://www.nytimes.com/2015/12/03/opinion/jus...               2   \n",
       "71410  https://www.nytimes.com/2012/01/05/us/politics...               3   \n",
       "12828  https://www.nytimes.com/1988/10/07/sports/the-...               2   \n",
       "3436   https://www.nytimes.com/1982/07/11/books/ameri...               6   \n",
       "71421  https://www.nytimes.com/1995/04/20/us/terror-o...               3   \n",
       "15337  https://www.nytimes.com/2002/07/03/movies/film...               5   \n",
       "3769   https://www.nytimes.com/2014/09/09/science/rev...               5   \n",
       "47030  https://www.nytimes.com/2021/05/14/science/chi...               6   \n",
       "\n",
       "      source_type_citoid citation_template  \\\n",
       "482     newspaperArticle          cite web   \n",
       "26287   newspaperArticle         cite news   \n",
       "56926   newspaperArticle          citation   \n",
       "71410   newspaperArticle          cite web   \n",
       "12828   newspaperArticle          cite web   \n",
       "3436    newspaperArticle         cite news   \n",
       "71421   newspaperArticle         cite news   \n",
       "15337   newspaperArticle          Cita web   \n",
       "3769    newspaperArticle         cite news   \n",
       "47030   newspaperArticle         cite news   \n",
       "\n",
       "                                            title_citoid  \\\n",
       "482    tennis; serena williams wins as the boos pour ...   \n",
       "26287  'sunday in the park with george,' with jake gy...   \n",
       "56926  opinion | justice scalia's majoritarian theocracy   \n",
       "71410      bucking senate, obama appoints consumer chief   \n",
       "12828        the playoffs; troubled cone stops the press   \n",
       "3436                                 american demagogues   \n",
       "71421  terror in oklahoma city: the investigation; at...   \n",
       "15337  film review; they have a tantrum, then save th...   \n",
       "3769                         the moon comes around again   \n",
       "47030  china's mars rover mission lands on the red pl...   \n",
       "\n",
       "                                            title_manual  \\\n",
       "482    [tennis; serena williams wins as the boos pour...   \n",
       "26287  [''sunday in the park with george'', with jake...   \n",
       "56926          [justice scalia's majoritarian theocracy]   \n",
       "71410    [bucking senate, obama appoints consumer chief]   \n",
       "12828      [the playoffs; troubled cone stops the press]   \n",
       "3436                               [american demagogues]   \n",
       "71421  [at least 31 are dead, scores are missing afte...   \n",
       "15337  [film review; they have a tantrum, then save t...   \n",
       "3769                       [the moon comes around again]   \n",
       "47030  [china's mars rover mission lands on the red p...   \n",
       "\n",
       "         author_first_citoid    author_first_manual    author_last_citoid  \\\n",
       "482                 [selena]               [selena]             [roberts]   \n",
       "26287              [michael]              [michael]             [paulson]   \n",
       "56926  [richard a., eric j.]                     []      [posner, segall]   \n",
       "71410     [helene, jennifer]             [helene, ]  [cooper, steinhauer]   \n",
       "12828               [joseph]                     []               [durso]   \n",
       "3436                [robert]               [robert]            [sherrill]   \n",
       "71421                [david]                     []            [johnston]   \n",
       "15337              [stephen]              [stephen]              [holden]   \n",
       "3769               [natalie]              [natalie]              [angier]   \n",
       "47030  [steven lee, kenneth]  [steven lee, kenneth]        [myers, chang]   \n",
       "\n",
       "                   author_last_manual pub_date_citoid pub_date_manual  \\\n",
       "482                         [roberts]      2001-03-18    [2001-03-18]   \n",
       "26287                       [paulson]      2016-07-14    [2016-07-14]   \n",
       "56926                              []      2015-12-02    [2015-12-02]   \n",
       "71410  [cooper, steinhauer, jennifer]      2012-01-04    [2012-01-04]   \n",
       "12828                 [durso, joseph]      1988-10-07    [1988-10-07]   \n",
       "3436                       [sherrill]      1982-07-11    [1982-07-11]   \n",
       "71421                [johnson, david]      1995-04-20    [1995-04-20]   \n",
       "15337                        [holden]      2002-07-03    [2002-07-03]   \n",
       "3769                         [angier]      2014-09-08    [2014-09-07]   \n",
       "47030                  [myers, chang]      2021-05-14    [2021-05-14]   \n",
       "\n",
       "        pub_source_citoid     pub_source_manual  \n",
       "482    the new york times  [the new york times]  \n",
       "26287  the new york times  [the new york times]  \n",
       "56926  the new york times                    []  \n",
       "71410  the new york times                    []  \n",
       "12828  the new york times                    []  \n",
       "3436   the new york times  [the new york times]  \n",
       "71421  the new york times  [the new york times]  \n",
       "15337  the new york times  [the new york times]  \n",
       "3769   the new york times  [the new york times]  \n",
       "47030  the new york times  [the new york times]  "
      ]
     },
     "execution_count": 342,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nytimes_sample_scores = comparison.loc[nytimes_sample]\n",
    "\n",
    "nytimes_sample_scores[\"citoid_success\"] = (nytimes_sample_scores[comp_fields] >= 0.75).sum(axis=1)\n",
    "\n",
    "cols = ['article_url','url',\"citoid_success\", 'source_type_citoid', 'citation_template', \n",
    "        'title_citoid', 'title_manual', \n",
    "        'author_first_citoid', 'author_first_manual',\n",
    "        'author_last_citoid', 'author_last_manual', \n",
    "        'pub_date_citoid','pub_date_manual', \n",
    "        'pub_source_citoid', 'pub_source_manual']\n",
    "\n",
    "nytimes_sample_scores[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://en.wikipedia.org/wiki/Antonin%20Scalia'"
      ]
     },
     "execution_count": 380,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nytimes_sample_scores.loc[56926].article_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {},
   "outputs": [],
   "source": [
    "nytimes_sample_scores[cols].to_csv(\"eval/nytimes_sample.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Guardian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>domain</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>success%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7328</th>\n",
       "      <td>www.theguardian.com</td>\n",
       "      <td>3.829132</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   domain  citoid_success  success%\n",
       "7328  www.theguardian.com        3.829132      64.0"
      ]
     },
     "execution_count": 381,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain = \"www.theguardian.com\"\n",
    "success_for_domain = success_by_domain[success_by_domain[\"domain\"] == domain].copy()\n",
    "success_for_domain[\"success%\"] = round(success_for_domain[\"citoid_success\"]/6*100)\n",
    "success_for_domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_url</th>\n",
       "      <th>url</th>\n",
       "      <th>citoid_success</th>\n",
       "      <th>source_type_citoid</th>\n",
       "      <th>citation_template</th>\n",
       "      <th>title_citoid</th>\n",
       "      <th>title_manual</th>\n",
       "      <th>author_first_citoid</th>\n",
       "      <th>author_first_manual</th>\n",
       "      <th>author_last_citoid</th>\n",
       "      <th>author_last_manual</th>\n",
       "      <th>pub_date_citoid</th>\n",
       "      <th>pub_date_manual</th>\n",
       "      <th>pub_source_citoid</th>\n",
       "      <th>pub_source_manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40406</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Nintendo</td>\n",
       "      <td>https://www.theguardian.com/technology/gallery...</td>\n",
       "      <td>3</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>citar web</td>\n",
       "      <td>the 10 most influential video games of all tim...</td>\n",
       "      <td>[the 10 most influential video games of all ti...</td>\n",
       "      <td>[keith]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[stuart]</td>\n",
       "      <td>[stuart, keith]</td>\n",
       "      <td>2017-05-16</td>\n",
       "      <td>[2017-05-16]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68408</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Radiohead</td>\n",
       "      <td>http://www.theguardian.com/music/2021/may/23/l...</td>\n",
       "      <td>4</td>\n",
       "      <td>webpage</td>\n",
       "      <td>Cite web</td>\n",
       "      <td>live at worthy farm review - beautiful music m...</td>\n",
       "      <td>[live at worthy farm review - beautiful music ...</td>\n",
       "      <td>None</td>\n",
       "      <td>[alexis]</td>\n",
       "      <td>None</td>\n",
       "      <td>[petridis]</td>\n",
       "      <td>2021-05-23</td>\n",
       "      <td>[2021-05-23]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65494</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Marilyn%20Monroe</td>\n",
       "      <td>https://www.theguardian.com/film/features/feat...</td>\n",
       "      <td>3</td>\n",
       "      <td>webpage</td>\n",
       "      <td>cite web</td>\n",
       "      <td>marilyn monroe: feminist icon? | features | gu...</td>\n",
       "      <td>[happy birthday, marilyn]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[2001-05-29]</td>\n",
       "      <td>www.theguardian.com</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2114</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Bradley%20Cooper</td>\n",
       "      <td>http://www.theguardian.com/film/2013/jan/14/go...</td>\n",
       "      <td>4</td>\n",
       "      <td>webpage</td>\n",
       "      <td>Citar web</td>\n",
       "      <td>golden globes 2013: full list of winners</td>\n",
       "      <td>[golden globes 2013: full list of winners]</td>\n",
       "      <td>[guardian]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[staff]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2013-01-14</td>\n",
       "      <td>[2013-01-14]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80344</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Russell%20T%20Da...</td>\n",
       "      <td>http://www.theguardian.com/media/2006/jul/17/m...</td>\n",
       "      <td>3</td>\n",
       "      <td>webpage</td>\n",
       "      <td>citar jornal</td>\n",
       "      <td>28. russell t davies</td>\n",
       "      <td>[28. russell t davies]</td>\n",
       "      <td>[guardian]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[staff]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2006-07-17</td>\n",
       "      <td>[2006-07-17]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75882</th>\n",
       "      <td>https://en.wikipedia.org/wiki/2002%20World%20S...</td>\n",
       "      <td>http://www.theguardian.com/sport/2006/jan/19/s...</td>\n",
       "      <td>3</td>\n",
       "      <td>webpage</td>\n",
       "      <td>Cite news</td>\n",
       "      <td>snooker: world championship finds new sponsors</td>\n",
       "      <td>[snooker: world championship finds new sponsors]</td>\n",
       "      <td>[mike]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[anstead]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2006-01-19</td>\n",
       "      <td>[2006-01-19]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52526</th>\n",
       "      <td>https://es.wikipedia.org/wiki/Radiohead</td>\n",
       "      <td>http://www.theguardian.com/music/2009/nov/29/r...</td>\n",
       "      <td>3</td>\n",
       "      <td>webpage</td>\n",
       "      <td>cita web</td>\n",
       "      <td>radiohead: band of the decade</td>\n",
       "      <td>[radiohead: band of the decade]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[gareth grundy]</td>\n",
       "      <td>2009-11-29</td>\n",
       "      <td>[]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[[http://www.theguardian.com/uk]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82433</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Daniel%20Radcliffe</td>\n",
       "      <td>https://www.theguardian.com/stage/gallery/2017...</td>\n",
       "      <td>4</td>\n",
       "      <td>newspaperArticle</td>\n",
       "      <td>citar web</td>\n",
       "      <td>daniel radcliffe in rosencrantz and guildenste...</td>\n",
       "      <td>[daniel radcliffe in rosencrantz and guildenst...</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>2017-03-06</td>\n",
       "      <td>[2017-03-06]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76007</th>\n",
       "      <td>https://pt.wikipedia.org/wiki/Russell%20T%20Da...</td>\n",
       "      <td>http://www.theguardian.com/media/2008/jul/14/m...</td>\n",
       "      <td>2</td>\n",
       "      <td>webpage</td>\n",
       "      <td>citar jornal</td>\n",
       "      <td>31. russell t davies</td>\n",
       "      <td>[31. russell t davies]</td>\n",
       "      <td>[guardian]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[staff]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2008-07-13</td>\n",
       "      <td>[2008-07-14]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[the guardian]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66215</th>\n",
       "      <td>https://en.wikipedia.org/wiki/Taylor%20Swift</td>\n",
       "      <td>http://www.theguardian.com/music/2014/dec/03/t...</td>\n",
       "      <td>4</td>\n",
       "      <td>webpage</td>\n",
       "      <td>Cite news</td>\n",
       "      <td>taylor swift leads poptimism's rebirth</td>\n",
       "      <td>[taylor swift leads poptimism's rebirth]</td>\n",
       "      <td>[ian]</td>\n",
       "      <td>[ian]</td>\n",
       "      <td>[gormely]</td>\n",
       "      <td>[gormely]</td>\n",
       "      <td>2014-12-03</td>\n",
       "      <td>[2014-12-03]</td>\n",
       "      <td>the guardian</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             article_url  \\\n",
       "40406             https://pt.wikipedia.org/wiki/Nintendo   \n",
       "68408            https://en.wikipedia.org/wiki/Radiohead   \n",
       "65494     https://en.wikipedia.org/wiki/Marilyn%20Monroe   \n",
       "2114      https://pt.wikipedia.org/wiki/Bradley%20Cooper   \n",
       "80344  https://pt.wikipedia.org/wiki/Russell%20T%20Da...   \n",
       "75882  https://en.wikipedia.org/wiki/2002%20World%20S...   \n",
       "52526            https://es.wikipedia.org/wiki/Radiohead   \n",
       "82433   https://pt.wikipedia.org/wiki/Daniel%20Radcliffe   \n",
       "76007  https://pt.wikipedia.org/wiki/Russell%20T%20Da...   \n",
       "66215       https://en.wikipedia.org/wiki/Taylor%20Swift   \n",
       "\n",
       "                                                     url  citoid_success  \\\n",
       "40406  https://www.theguardian.com/technology/gallery...               3   \n",
       "68408  http://www.theguardian.com/music/2021/may/23/l...               4   \n",
       "65494  https://www.theguardian.com/film/features/feat...               3   \n",
       "2114   http://www.theguardian.com/film/2013/jan/14/go...               4   \n",
       "80344  http://www.theguardian.com/media/2006/jul/17/m...               3   \n",
       "75882  http://www.theguardian.com/sport/2006/jan/19/s...               3   \n",
       "52526  http://www.theguardian.com/music/2009/nov/29/r...               3   \n",
       "82433  https://www.theguardian.com/stage/gallery/2017...               4   \n",
       "76007  http://www.theguardian.com/media/2008/jul/14/m...               2   \n",
       "66215  http://www.theguardian.com/music/2014/dec/03/t...               4   \n",
       "\n",
       "      source_type_citoid citation_template  \\\n",
       "40406   newspaperArticle         citar web   \n",
       "68408            webpage          Cite web   \n",
       "65494            webpage          cite web   \n",
       "2114             webpage         Citar web   \n",
       "80344            webpage      citar jornal   \n",
       "75882            webpage         Cite news   \n",
       "52526            webpage          cita web   \n",
       "82433   newspaperArticle         citar web   \n",
       "76007            webpage      citar jornal   \n",
       "66215            webpage         Cite news   \n",
       "\n",
       "                                            title_citoid  \\\n",
       "40406  the 10 most influential video games of all tim...   \n",
       "68408  live at worthy farm review - beautiful music m...   \n",
       "65494  marilyn monroe: feminist icon? | features | gu...   \n",
       "2114            golden globes 2013: full list of winners   \n",
       "80344                               28. russell t davies   \n",
       "75882     snooker: world championship finds new sponsors   \n",
       "52526                      radiohead: band of the decade   \n",
       "82433  daniel radcliffe in rosencrantz and guildenste...   \n",
       "76007                               31. russell t davies   \n",
       "66215             taylor swift leads poptimism's rebirth   \n",
       "\n",
       "                                            title_manual author_first_citoid  \\\n",
       "40406  [the 10 most influential video games of all ti...             [keith]   \n",
       "68408  [live at worthy farm review - beautiful music ...                None   \n",
       "65494                          [happy birthday, marilyn]                None   \n",
       "2114          [golden globes 2013: full list of winners]          [guardian]   \n",
       "80344                             [28. russell t davies]          [guardian]   \n",
       "75882   [snooker: world championship finds new sponsors]              [mike]   \n",
       "52526                    [radiohead: band of the decade]                None   \n",
       "82433  [daniel radcliffe in rosencrantz and guildenst...                None   \n",
       "76007                             [31. russell t davies]          [guardian]   \n",
       "66215           [taylor swift leads poptimism's rebirth]               [ian]   \n",
       "\n",
       "      author_first_manual author_last_citoid author_last_manual  \\\n",
       "40406                  []           [stuart]    [stuart, keith]   \n",
       "68408            [alexis]               None         [petridis]   \n",
       "65494                  []               None                 []   \n",
       "2114                   []            [staff]                 []   \n",
       "80344                  []            [staff]                 []   \n",
       "75882                  []          [anstead]                 []   \n",
       "52526                  []               None    [gareth grundy]   \n",
       "82433                  []               None                 []   \n",
       "76007                  []            [staff]                 []   \n",
       "66215               [ian]          [gormely]          [gormely]   \n",
       "\n",
       "      pub_date_citoid pub_date_manual    pub_source_citoid  \\\n",
       "40406      2017-05-16    [2017-05-16]         the guardian   \n",
       "68408      2021-05-23    [2021-05-23]         the guardian   \n",
       "65494            None    [2001-05-29]  www.theguardian.com   \n",
       "2114       2013-01-14    [2013-01-14]         the guardian   \n",
       "80344      2006-07-17    [2006-07-17]         the guardian   \n",
       "75882      2006-01-19    [2006-01-19]         the guardian   \n",
       "52526      2009-11-29              []         the guardian   \n",
       "82433      2017-03-06    [2017-03-06]         the guardian   \n",
       "76007      2008-07-13    [2008-07-14]         the guardian   \n",
       "66215      2014-12-03    [2014-12-03]         the guardian   \n",
       "\n",
       "                      pub_source_manual  \n",
       "40406                    [the guardian]  \n",
       "68408                    [the guardian]  \n",
       "65494                                []  \n",
       "2114                     [the guardian]  \n",
       "80344                    [the guardian]  \n",
       "75882                    [the guardian]  \n",
       "52526  [[http://www.theguardian.com/uk]  \n",
       "82433                                []  \n",
       "76007                    [the guardian]  \n",
       "66215                                []  "
      ]
     },
     "execution_count": 376,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guardian_sample = scores[scores.domain == \"www.theguardian.com\"].sample(10).index\n",
    "guardian_sample_scores = comparison.loc[guardian_sample]\n",
    "\n",
    "guardian_sample_scores[\"citoid_success\"] = (guardian_sample_scores[comp_fields] >= 0.75).sum(axis=1)\n",
    "\n",
    "cols = ['article_url','url',\"citoid_success\", 'source_type_citoid', 'citation_template', \n",
    "        'title_citoid', 'title_manual', \n",
    "        'author_first_citoid', 'author_first_manual',\n",
    "        'author_last_citoid', 'author_last_manual', \n",
    "        'pub_date_citoid','pub_date_manual', \n",
    "        'pub_source_citoid', 'pub_source_manual']\n",
    "\n",
    "guardian_sample_scores[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [],
   "source": [
    "guardian_sample_scores[cols].to_csv(\"eval/guardian_sample.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 By language\n",
    "\n",
    "In this section, we show the results for each Wikipedia.\n",
    "\n",
    "First, we visualize the number of evaluated citations by language of the Wikipedia. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "citations_by_lang = scores[['wiki_lang']].value_counts()\n",
    "citations_by_lang.name = \"count\"\n",
    "citations_by_lang =  citations_by_lang.reset_index().sort_values(by=[\"count\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcn0lEQVR4nO3deZRlZX3u8e9DNzPY2IBExgZjCETUC6hw5aoIKtpRuAkaEdRljARnxAmjS4EYRbN0Gaco4lUUjBBwIKAIKkg0YWikGRxQ1BZEIgLSTIrQ/O4f+y09lk33qe596nR1fT9r7VV7v2cPv1P96uGp9937pKqQJEmSJK2+dcZdgCRJkiStLQxYkiRJktQTA5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJAmAJP+Q5MQez7d9kjuTzOnrnENc85NJ3t7TuSrJn/ZxLknS7GHAkqRZJMnzkixqwefGJF9Osg9AVb2jqv6u7begBYy5Uzj3kiT7T2xX1XVVtUlVLev/nUiStGYyYEnSLJHkKOB9wDuArYDtgQ8DB46xLEmS1ioGLEmaBZLMA44DXl5Vn6uqu6rq3qr6j6p6fdvnmCQnt0MubD9va6Ndeyd5WJKvJ7klyc1JTkmyWTv203SB7T/a/m+YPAqWZOskZya5Ncm1SV4yUN8xSU5L8qkkdyT5TpI9B15/Y5Ib2mvXJNlvBW93iyTntX2/kWSHdo4PJXnPpN/LmUleM8Tvb2GSy5PcnuT6JMcMvDbxPl+Y5Lr2u3nzwOsbJjkpya+SfK/9bn428PofTEUcnOaY5MFJzkryy3b8WUm2Hdh3xyQXtvf61fYeTx54fa8k/5XktiRXJHnSyt6rJGn1GLAkaXbYG9gA+PyQ+z+h/dysTfP7byDAO4GtgV2A7YBjAKrq+cB1wDPb/u9ezjk/C/ysHX8w8I4kTx54/Vltn82AM4EPAiTZGXgF8Jiq2hR4GrBkBbUfCvwjsAWwGDiltZ8EHJJknXbeLYD9gc+s7JcB3AW8oNW2EHhpkoMm7bMPsDOwH/DWJLu09rcBC4CdgKcAhw1xvQnrAJ8AdqALsL+m/V6azwCXAJvT/Vs8f+KFJNsAZwNvB+YDrwPOSLLlFK4vSZoiA5YkzQ6bAzdX1X2reoKquraqzquqe6rql8B7gScOc2yS7YDHA2+sqt9U1WLgRLrQMuGbVfWlds/Wp4FHtfZlwPrArknWraolVfWjFVzu7Kq6sKruAd4M7J1ku6q6BFhKF4AAngtcUFW/GOK9X1BVV1XV/VV1JfBvy3nvx1bVr6vqCuCKgfqfA7yjqn5VVT8D3r+y6w1c95aqOqOq7q6qO4B/mrhuku2BxwBvrarfVtU36YLphMOAL7Xf6f1VdR6wCHjGsNeXJE2dAUuSZodb6KbODf3QismSbJXks22q3u3AyXSjRMPYGri1hYQJPwW2Gdj+n4H1u4ENksytqmuBI+lGaG5qNWy9gmtdP7FSVXcCt7brQzeKNTGCdBhdkFupJI9Lcn6bqrcUOII/fu+T69+krW89WNOk9ZVdd6MkH03y0/Y7vxDYLN2TGSd+p3c/wLl3AJ7dpgfeluQ2ulG2hw57fUnS1BmwJGl2+G/gHuCgIfev5bS9o7XvVlUPogsoWckxE34OzE+y6UDb9sANQxVT9Zmq2ocuNBTwrhXsvt3ESpJN6KbH/bw1nQwcmORRdNMcvzDM9emm4p0JbFdV84CP8IfvfUVuBLYd2N5u0ut3AxsNbP/JwPpr6aYdPq79ziembqadd36SwWMHz3098Omq2mxg2biqjh+ybknSKjBgSdIsUFVLgbcCH0pyUBsZWTfJ05Ms736pXwL30903NGFT4E5gabu/5/WTjvnFpP0Hr3898F/AO5NskOSRwIvpAs8KJdk5yZOTrA/8hu4+pPtXcMgzkuyTZD26e7EuatenTdG7lG7k6oyq+vXKrt9sSjda9JskjwWeN+RxAKcBb2oPrNiG7n6yQYuB5yWZk+QA/nDq4aZ07/e2JPPp7ueivZef0k35OybJekn2Bp45cOzJwDOTPK2de4MkTxp8SIYkqX8GLEmaJarqPcBRwFvoAtT1dP+x/4Xl7Hs33f0+32rTy/YCjgV2p7uP6Wzgc5MOeyfwlrb/65ZTwiF0D3v4Od3DNt5WVV8dovT1geOBm+mm4T0EeNMK9v8MXRC5FdiDP36oxEnAbgw5PbB5GXBckjvoguppUzj2OLqHe/wE+CpwOt1o4oRX0wWj2+ge0PGFgdfeB2xI994vAs6ZdO5D6R5gcgvdwyxOnTh3C5UHAv/A7/+9X4+f/ZI0Uqla0YwOSZLWLkmeQDe6s0ON4UMwyUuB51bVUA8ImeK5TwW+X1VvW+nOkqSR8K9YkqRZI8m6dCNGJ05XuEry0CSPT7JOe+T8axn+cfkrO/dj0n0/2TpteuGBDH9fmSRpBFb5aVKSJM0k7XupFtE9Qv1F03jp9YCPAjvSTQP8LPDhns79J3RTNTenm4b40qq6vKdzS5JWgVMEJUmSJKknThGUJEmSpJ7MiCmCW2yxRS1YsGDcZUiSJEkSAJdddtnNVbXl5PYZEbAWLFjAokWLxl2GJEmSJAGQ5KfLa3eKoCRJkiT1xIAlSZIkST0xYEmSJElSTwxYkiRJktQTA5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJEmSJPXEgCVJkiRJPTFgSZIkSVJPDFiSJEmS1JO54y5gGFfdsJQFR5897jIkSZIkTaMlxy8cdwlT5giWJEmSJPXEgCVJkiRJPTFgSZIkSVJPDFiSJEmS1BMDliRJkiT1xIAlSZIkST0xYEmSJElSTwxYkiRJktQTA5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUk5EGrCSHJbkkyeIkH00yJ8mdSf4pyRVJLkqy1ShrkCRJkqTpMrKAlWQX4G+Ax1fVo4FlwKHAxsBFVfUo4ELgJQ9w/OFJFiVZtOzupaMqU5IkSZJ6M3eE594P2AO4NAnAhsBNwG+Bs9o+lwFPWd7BVXUCcALA+g99eI2wTkmSJEnqxSgDVoCTqupNf9CYvK6qJgLTshHXIEmSJEnTZpT3YH0NODjJQwCSzE+ywwivJ0mSJEljNbLRo6r6bpK3AOcmWQe4F3j5qK4nSZIkSeM20ul5VXUqcOqk5k0GXj8dOH2UNUiSJEnSdPF7sCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqScGLEmSJEnqiQFLkiRJknpiwJIkSZKknoz0i4b7sts281h0/MJxlyFJkiRJK+QIliRJkiT1xIAlSZIkST0xYEmSJElSTwxYkiRJktQTA5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJEmSJPXEgCVJkiRJPTFgSZIkSVJPDFiSJEmS1BMDliRJkiT1xIAlSZIkST0xYEmSJElSTwxYkiRJktQTA5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJEmSJPXEgCVJkiRJPTFgSZIkSVJPDFiSJEmS1BMDliRJkiT1xIAlSZIkST2ZO+4ChnHVDUtZcPTZ4y5D0gosOX7huEuQJEkaO0ewJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJEmSpJ4YsCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqSdjC1hJjkyy0biuL0mSJEl9G+cI1pGAAUuSJEnSWmPuqC+QZAFwDnAZsDvwHeBCYGvg/CQ3V9W+o65DkiRJkkZtukawdgY+XFW7ALcD6wE/B/Y1XEmSJElaW0xXwLq+qr7V1k8G9lnZAUkOT7IoyaJldy8dbXWSJEmS1IPpCli1ku0/PqDqhKras6r2nLPRvBGVJUmSJEn9ma6AtX2Svdv684BvAncAm07T9SVJkiRp5KYrYF0DvDzJ94AHA/8KnACck+T8aapBkiRJkkZq5E8RbO6rqsMmtX2gLZIkSZK0Vhjn92BJkiRJ0lplqICVZKskH0/y5ba9a5IXD3NsVS2pqkesTpGSJEmSNBMMO4L1SeArdF8ODPAD4MgR1CNJkiRJM9awAWuLqjoNuB+gqu4Dlo2sKkmSJEmagYYNWHcl2Zz2/VVJ9gL89l9JkiRJGjDsUwSPAs4EHpbkW8CWwMEjq0qSJEmSZqChAlZVfTvJE4GdgQDXVNW9I61MkiRJkmaYoQJWkr+a1PRnSZYCV1XVTf2XJUmSJEkzz7BTBF8M7A2c37afBFwG7JjkuKr69Ahq+53dtpnHouMXjvISkiRJkrTahg1Yc4FdquoX0H0vFvAp4HHAhcBIA5YkSZIkzQTDPkVwu4lw1dzU2m4FvBdLkiRJkhh+BOuCJGcB/962/7q1bQzcNorCJEmSJGmmGTZgvZwuVD2+bX8KOKOqCth3FIVJkiRJ0kwz7GPaCzi9LZIkSZKk5RjqHqwkeyW5NMmdSX6bZFmS20ddnCRJkiTNJMM+5OKDwCHAD4ENgb8DPjSqoiRJkiRpJho2YFFV1wJzqmpZVX0COGB0ZUmSJEnSzDPsQy7uTrIesDjJu4EbmUI4kyRJkqTZYNiQ9HxgDvAK4C5gO7qnCkqSJEmSmmGfIvjTtvpr4NjRlSNJkiRJM9dQASvJT4Ca3F5VO/VekSRJkiTNUMPeg7XnwPoGwLOB+f2XI0mSJEkz11D3YFXVLQPLDVX1PmDhaEuTJEmSpJll2CmCuw9srkM3ojXs6JckSZIkzQrDhqT3DKzfBywBntN7NZIkSZI0gw37FMF9R12IJEmSJM10w04RPGo5zUuBy6pqca8VSZIkSdIMNewXDe8JHAFs05a/Bw4APpbkDSOqTZIkSZJmlGHvwdoW2L2q7gRI8jbgbOAJwGXAu0dTniRJkiTNHMOOYD0EuGdg+15gq6r69aR2SZIkSZq1hh3BOgW4OMkX2/Yzgc8k2Rj47kgqkyRJkqQZZtinCP5jknOA/92ajqiqRW390JFUJkmSJEkzzFS+LPjbwA0TxyTZvqquG0lVkiRJkjQDDfuY9lcCbwN+ASwDAhTwyNGVJkmSJEkzy7AjWK8Gdq6qW0ZZjCRJkiTNZMM+RfB6ui8WliRJkiQ9gGFHsH4MXJDkbAYey15V7x1JVZNcdcNSFhx99nRcStIqWHL8wnGXIEmStEYYNmBd15b12iJJkiRJmmTYx7QfO+pCJEmSJGmmG/YpglsCbwD+Athgor2qnjyiuiRJkiRpxhn2IRenAN8HdgSOBZYAl46oJkmSJEmakYYNWJtX1ceBe6vqG1X1t4CjV5IkSZI0YNiHXNzbft6YZCHwc2D+aEqSJEmSpJlp2ID19iTzgNcCHwAeBBw5qqIkSZIkaSYa9imCZ7XVpcC+AEmOHFFNkiRJkjQjDXsP1vIc1VsVkiRJkrQWWJ2AlSkfkLwqyfeSnLIa15UkSZKkNdKw92AtT63CMS8D9q+qn000JJlbVfetRh2SJEmStEZYYcBKcgfLD1IBNpzKhZJ8BNgJ+HKS7YEz2/Z1wCFTOZckSZIkrYlWGLCqatO+LlRVRyQ5gO4hGa8AngnsU1W/7usakiRJkjROq3MP1uo6c0XhKsnhSRYlWbTs7qXTWZckSZIkrZJxBqy7VvRiVZ1QVXtW1Z5zNpo3XTVJkiRJ0iobZ8CSJEmSpLWKAUuSJEmSerI6j2mfsqpa0FaPmc7rSpIkSdJ0cARLkiRJknpiwJIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJEmSpJ4YsCRJkiSpJwYsSZIkSerJtH7R8KrabZt5LDp+4bjLkCRJkqQVcgRLkiRJknpiwJIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJEmSpJ4YsCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqScGLEmSJEnqiQFLkiRJknpiwJIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJEmSpJ4YsCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqScGLEmSJEnqiQFLkiRJknpiwJIkSZKknhiwJEmSJKknc8ddwDCuumEpC44+e9xlaC2z5PiF4y5BkiRJaxlHsCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqScGLEmSJEnqiQFLkiRJknpiwJIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6MvKAleSwJJckWZzko0nmJPlkkquTXJXkNaOuQZIkSZKmw9xRnjzJLsDfAI+vqnuTfBh4C7BNVT2i7bPZAxx7OHA4wJwHbTnKMiVJkiSpF6MewdoP2AO4NMnitj0f2CnJB5IcANy+vAOr6oSq2rOq9pyz0bwRlylJkiRJq2/UASvASVX16LbsXFWvBh4FXAAcAZw44hokSZIkaVqMOmB9DTg4yUMAksxPsgOwTlWdQTddcPcR1yBJkiRJ02Kk92BV1XeTvAU4N8k6wL3AUcDn2zbAm0ZZgyRJkiRNl5EGLICqOhU4dVKzo1aSJEmS1jp+D5YkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJEmSJPXEgCVJkiRJPTFgSZIkSVJPDFiSJEmS1JORf9FwH3bbZh6Ljl847jIkSZIkaYUcwZIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJEmSpJ4YsCRJkiSpJwYsSZIkSeqJAUuSJEmSemLAkiRJkqSeGLAkSZIkqScGLEmSJEnqiQFLkiRJknpiwJIkSZKknqSqxl3DSiW5A7hm3HVorbUFcPO4i9Bay/6lUbJ/aZTsXxqltaF/7VBVW05unDuOSlbBNVW157iL0NopySL7l0bF/qVRsn9plOxfGqW1uX85RVCSJEmSemLAkiRJkqSezJSAdcK4C9Bazf6lUbJ/aZTsXxol+5dGaa3tXzPiIReSJEmSNBPMlBEsSZIkSVrjGbAkSZIkqSdrdMBKckCSa5Jcm+TocdejNVeS/5fkpiRXD7TNT3Jekh+2nw9u7Uny/tavrkyy+8AxL2z7/zDJCwfa90hyVTvm/Ukyve9Q45RkuyTnJ/luku8keXVrt49ptSXZIMklSa5o/evY1r5jkotbnzg1yXqtff22fW17fcHAud7U2q9J8rSBdj9PZ7kkc5JcnuSstm3/Ui+SLGmfX4uTLGpts/vzsarWyAWYA/wI2AlYD7gC2HXcdbmsmQvwBGB34OqBtncDR7f1o4F3tfVnAF8GAuwFXNza5wM/bj8f3NYf3F67pO2bduzTx/2eXaa1fz0U2L2tbwr8ANjVPubSU/8KsElbXxe4uPWF04DntvaPAC9t6y8DPtLWnwuc2tZ3bZ+V6wM7ts/QOX6eurT+cRTwGeCstm3/cumrby0BtpjUNqs/H9fkEazHAtdW1Y+r6rfAZ4EDx1yT1lBVdSFw66TmA4GT2vpJwEED7Z+qzkXAZkkeCjwNOK+qbq2qXwHnAQe01x5UVRdV97/0Tw2cS7NAVd1YVd9u63cA3wO2wT6mHrR+cmfbXLctBTwZOL21T+5fE/3udGC/9hfdA4HPVtU9VfUT4Fq6z1I/T2e5JNsCC4ET23awf2m0ZvXn45ocsLYBrh/Y/llrk4a1VVXd2Nb/B9iqrT9Q31pR+8+W065ZqE2X+V90owz2MfWiTd9aDNxE9x8WPwJuq6r72i6DfeJ3/ai9vhTYnKn3O80e7wPeANzftjfH/qX+FHBuksuSHN7aZvXn49xxFyBNh6qqJH4ngVZLkk2AM4Ajq+r2wWng9jGtjqpaBjw6yWbA54E/H29FWlsk+Uvgpqq6LMmTxlyO1k77VNUNSR4CnJfk+4MvzsbPxzV5BOsGYLuB7W1bmzSsX7ShZdrPm1r7A/WtFbVvu5x2zSJJ1qULV6dU1edas31Mvaqq24Dzgb3pps5M/CF0sE/8rh+11+cBtzD1fqfZ4fHAs5IsoZu+92TgX7B/qSdVdUP7eRPdH4geyyz/fFyTA9alwMPbU27Wo7vR8swx16SZ5Uxg4ik0LwS+OND+gvYkm72ApW0Y+yvAU5M8uD3t5qnAV9prtyfZq81Df8HAuTQLtH/3jwPfq6r3DrxkH9NqS7JlG7kiyYbAU+ju8zsfOLjtNrl/TfS7g4Gvt3sTzgSe254CtyPwcLqbw/08ncWq6k1VtW1VLaD7t/96VR2K/Us9SLJxkk0n1uk+165mtn8+jvspGyta6J408gO6uehvHnc9LmvuAvwbcCNwL9383BfTzRn/GvBD4KvA/LZvgA+1fnUVsOfAef6W7sbda4EXDbTvSfd/GD8CPghk3O/ZZVr71z50c8yvBBa35Rn2MZee+tcjgctb/7oaeGtr34nuP2CvBf4dWL+1b9C2r22v7zRwrje3PnQNA0/a8vPUpfWDJ/H7pwjav1z66FM70T058grgOxP//rP98zGtcEmSJEnSalqTpwhKkiRJ0oxiwJIkSZKknhiwJEmSJKknBixJkiRJ6okBS5IkSZJ6YsCSJA0tSSV5z8D265Ic09O5P5nk4JXvudrXeXaS7yU5f8j9j0uyf1s/MslGQxzzB/sl+dLEd11JktZuBixJ0lTcA/xVki3GXcigJHOnsPuLgZdU1b7D7FxVb62qr7bNI4GVBqzJ+1XVM6rqtinUKEmaoQxYkqSpuA84AXjN5Bcmj0AlubP9fFKSbyT5YpIfJzk+yaFJLklyVZKHDZxm/ySLkvwgyV+24+ck+ecklya5MsnfD5z3P5OcCXx3OfUc0s5/dZJ3tba30n1x9MeT/PNyjnljO+aKJMcPvq8krwK2Bs6fGP1K8q+t3u8kOba1LW+/JROhNMlRraarkxzZ2ha0UbWPtXOdm2TDifMl+W57758d+l9KkjQWU/mLnyRJAB8Crkzy7ikc8yhgF+BW4MfAiVX12CSvBl5JN+IDsAB4LPAwuoDyp8ALgKVV9Zgk6wPfSnJu23934BFV9ZPBiyXZGngXsAfwK+DcJAdV1XFJngy8rqoWTTrm6cCBwOOq6u4k8wdfr6r3JzkK2Leqbm7Nb66qW5PMAb6W5JEPsN/ENfYAXgQ8DghwcZJvtBofDhxSVS9Jchrw18DJwNHAjlV1j9MMJWnN5wiWJGlKqup24FPAq6Zw2KVVdWNV3QP8CJgISFfRhaoJp1XV/VX1Q7og9ufAU4EXJFkMXAxsThdGAC6ZHK6axwAXVNUvq+o+4BTgCSupcX/gE1V1d3uftw7xvp6T5NvA5cBfALuuZP99gM9X1V1VdSfwOeD/tNd+UlWL2/pl/P73ciVwSpLD6EYQJUlrMAOWJGlVvI/uXqaNB9ruo32uJFkHWG/gtXsG1u8f2L6fP5xNUZOuU3QjPa+sqke3Zceqmghod63Om1gdSXYEXgfsV1WPBM4GNliNUw7+jpbx+9/LQrpRw92BS6d4v5kkaZoZsCRJU9ZGd06jC1kTltBNyQN4FrDuKpz62UnWafdl7QRcA3wFeGmSdQGS/FmSjVd0EuAS4IlJtmjT9w4BvrGSY84DXjTx9L/JUwSbO4BN2/qD6ALe0iRbAU9/gP0G/SdwUJKN2nv4v61tuVpQ3a6qzgfeCMwDNlnJ+5AkjZF/BZMkrar3AK8Y2P4Y8MUkVwDnsGqjS9fRhaMHAUdU1W+SnEg3Xe7bSQL8EjhoRSepqhuTHA2cTzcCdnZVfXElx5yT5NHAoiS/Bb4E/MOk3U4Azkny86raN8nlwPeB64FvPdB+A9f4dpJPtvcI3b1olydZ8ABlzQFOTjKvvY/3+zRCSVqzpWrybAxJkiRJ0qpwiqAkSZIk9cSAJUmSJEk9MWBJkiRJUk8MWJIkSZLUEwOWJEmSJPXEgCVJkiRJPTFgSZIkSVJP/j93FDGfY5LlNgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = citations_by_lang.set_index(\"wiki_lang\").plot(figsize=[12,4], title=\"Citations by language\", legend=False, kind=\"barh\")\n",
    "ax.set_ylabel(\"Language\")\n",
    "ax.set_xlabel(\"Number of citations\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/citations_by_lang.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in section 1 and 2 (Retrieving data from Wikipedia articles and Citation template metadata extraction), the featured articles and the citation templates registered in our spreadsheet present an overwhelming majority in English. It is not surprising to find this proportion of English citations in the evaluation corpus.\n",
    "\n",
    "Here, we group the citations by language, calculate the average success and plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 317,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZqElEQVR4nO3deZRkVZmu8eelGGQsxEJFEEocQEVFRBSnq2LbCCpqs5RWVLhto9dZnNClbdvd9qUHbfvqRUBEcUZxuIgT2g44IRQIooAKijIpIlBQQivDd/84JzUqycyKGiJiZ+XzWytWRcSZvjiZdd7c++w4J1WFJEmt2WDSBUiSNBMDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0ojleSNSY5bw2V3TLIiyaJZpv99kg+vXYXtS/LYJJeto3V9IMk/rYt1SaNmQGmtJXl2kmV9mFyZ5ItJHgVQVf9cVS/o51uapJJsOMx6q+pXVbVFVd06yvoltcmA0lpJcjjwTuCfgbsAOwJHAQdMsCxJ6wEDSmssyWLgH4CXVNWnq+r3VXVzVX2uql7bzzPYDXda/+91fWtr7yQbJHlTkl8muSrJB/v13q7FleQeSb6Z5IYkXwGWzFHbkiSnJLkuyTVJvpVkg35aJbnXwLwrdXslOSDJOUmuT3Jxkn3797dJ8v4kVyS5NslnB5Z5cr/MdUm+m+SBA9Nen+Tyvu6fJNmnf3+vvuV5fZLfJHnHKvb3G5NcneSSJM/p33tov+yigfmekeTcudbVz3fHfh/9tv88pyTZYWD6N5L8Y5Lv9LWfmmTJwPTn9T+33yV5c1/XE2bZpyt1UyY5ot+3NyQ5P8nTB6YtSvL2/rP+IslLp/0eLE7yvr61fnmSf5qtG1jzmwGltbE3cAfgM0PO/5j+3637rrvvAYf0j8cBOwNbAO+eZfmPAmfRBdM/As+fY1uvBi4DtqVr2b0RWOV1vZLsBXwQeC2wdV/zJf3kDwGbAfcH7gz8R7/Mg4HjgRcCdwKOAU5OskmSXYCXAg+tqi2BvxxY338C/1lVWwH3BD4xR2l37T/39v3nPjbJLlV1JvA74IkD8z63/wyrsgHwfmAnupbvTdx+3z8bOLT/vBsDr+k/8/3oWsrPAbYDFve1Deti4NH9cm8FPpxku37a3wJPAnYH9gCeNm3ZDwC3APcCHkz32V+wGtvWPGFAaW3cCbi6qm5Zi3U8B3hHVf28qlYAbwAOmn6eKsmOwEOBN1fVH6rqNOBzc6z3ZroD5059q+5bNdyFJ/8GOL6qvlJVt1XV5VV1YX/wfBLwoqq6tl/nN/tlDgOOqarvV9WtVXUC8Afg4cCtwCbA/ZJsVFWXVNXFAzXeK8mSqlpRVaevorapz/5N4PPAM/v3TwAO7vfTNnQh+NFVfdCq+l1VfaqqbqyqG4C3Af9j2mzvr6qfVtVNdAG6e//+gcDnqurbVfVH4O8Y4g+AgW1/sqqu6PfxicDPgL36yc+kC+7Lqupa4Mip5ZLcBdgPeGXfYr+K7g+Fg4bdtuYPA0pr43fAkmEHPczibsAvB17/EtiQrtUzfb5rq+r30+adzb8BFwGnJvl5kiOGrOfudH/dz/T+Nf0Bc7qdgFf33XvXJbmun/9uVXUR8Erg74Grknw8yd365f4GuA9wYZIzkzx5jrpm+uxT6/kw8JQkm9Md3L9VVVeu6oMm2SzJMX033fV0XbBbT+su+/XA8xvpWrj02750akJV3Uj3+zCUvnvwnIH9tRt/7rJdad3Tnu8EbARcObDsMXQtPK1nDCitje/RtRSeNuT8M/2FfQXdQWfKjnTdN7+ZNt+VwB37g/DgvDNvqOqGqnp1Ve0MPBU4fOrcD92BdrOB2e868PxSuu626S4Ftkmy9SzT3lZVWw88Nquqj/W1fLSqHtV/zgL+pX//Z1X113QH138BTpr2+QbN9Nmv6NdzOd3P4hl03XsfmmUd070a2AV4WN/NONUFmyGWvRIYPF+1KV2LesrvmWUfJ9kJeC9d1+edqmpr4EcD211p3XRhP+VSut+5JQP7eququv8QNWueMaC0xqpqOV3Xzv9N8rT+L/KNkjwpyb/OsMhvgdvozjVN+RjwqnQDILagGw144vRuw6r6JbAMeGuSjdMNY3/KbLX1gxbulSTAcrquttv6yecAz+5Pxu/Lyt1a7wMOTbJPugEc2yfZtW+RfBE4qh9csFGSqQP6e4EXJXlYOpsn2T/Jlkl2SfL4JJsA/013nue2vsaDk2xbVbcB1/Xruo3ZTX32RwNPBj45MO2DwOuABwCfnmMdg7bs67mu7xp8y5DLAZxE12p7RJKN6VqIg8F2DrBfuoEld6VrRU7ZnC6ofwuQ5FC6FtSUTwCv6Pf91sDrpyb0P4dTgbcn2ar/Gd0zyfSuSa0HDCitlap6O3A48Ca6A86ldH8Zf3aGeW+kO8/xnb575uF0gws+RNe99Au6g/jLZtncs4GHAdfQHUznGghwb+CrwAq61sVRVfX1ftor6MLtOrpzYH+qtarOoBsU8B90wfZN/tzCey7deaMLgavoD7pVtYzuxP67gWvpuhYP6ZfZhO4cytV03WV3pjvPBrAv8OMkK+gGTBzUn+uZya/7dV8BfITuXNiFA9M/09f5mX4/D+OdwKZ9bacDXxpyOarqx3Q/p4/TtXhW0O2TP/SzfAg4l25AyKnAiQPLng+8ne7n8hu6UP3OwOrf2y/zQ+AHwBfoWtVT34d7Ht2AjfPp9slJdOcbtZ6JNyyU1g9JLgZeWFVfncC2t6AL/HtX1S/W8bqfBBxdVTutcmatV2xBSeuBJH9F1232tTFu8yl9t+7mwL8D5/HnIfRrs95Nk+yXZMMk29O1lof9KoPWIwaUNM8l+QbwHrovTM91DmtdO4Cuy/EKui7Vg4Ycyr8qoftu1LV0XXwX0J3r1AJjF58kqUm2oCRJTVqbL1iOzZIlS2rp0qWTLkOStAbOOuusq6tq29Vdbl4E1NKlS1m2bNmky5AkrYEkc131ZVZ28UmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkpo0Ly4We97ly1l6xOcnXYYkLTiXHLn/xLZtC0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1KSxBlSSlye5IMlHxrldSdL8M+5r8b0YeEJVXTb1RpINq+qWMdchSWrc2AIqydHAzsAXk+wInNy//hXw1+OqQ5I0P4wtoKrqRUn2BR4HvBR4CvCoqrpppvmTHAYcBrBoq23HVaYkqRGTHCRx8mzhBFBVx1bVnlW156LNFo+zLklSAyYZUL+f4LYlSY1zmLkkqUkGlCSpSWMdZl5VS/unfz/O7UqS5h9bUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmjft+UGvkAdsvZtmR+0+6DEnSGNmCkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDVpw0kXMIzzLl/O0iM+P+kyJGmtXXLk/pMuYd6wBSVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlq0sgDKsnBSc5Ick6SY5IsSvKBJD9Kcl6SV426BknS/DPSi8UmuS/wLOCRVXVzkqOANwHbV9Vu/Txbj7IGSdL8NOoW1D7AQ4Azk5zTv94G2DnJu5LsC1w/04JJDkuyLMmyW29cPuIyJUmtGXVABTihqnbvH7tU1SuABwHfAF4EHDfTglV1bFXtWVV7Ltps8YjLlCS1ZtQB9V/AgUnuDJBkmyQ7ARtU1afouvv2GHENkqR5aKTnoKrq/CRvAk5NsgFwM3A48Jn+NcAbRlmDJGl+GvkddavqRODEaW/bapIkzcnvQUmSmrTKFlSSbWZ4+4aqunkE9UiSBAzXgjob+C3wU+Bn/fNLkpyd5CGjLE6StHANE1BfAfarqiVVdSfgScApwIuBo0ZZnCRp4RomoB5eVV+eelFVpwJ7V9XpwCYjq0yStKANM4rvyiSvBz7ev34W8Jski4DbRlaZJGlBG6YF9WxgB+Cz/WPH/r1FwDNHVZgkaWFbZQuqqq4GXjbL5IvWbTmSJHWGGWZ+H+A1wNLB+avq8aMra2UP2H4xy47cf1ybkyQ1YJhzUJ8Ejqa7qOutoy1HkqTOMAF1S1W9Z+SVSJI0YJhBEp9L8uIk2/VXI99mlqtLSJK0zgzTgnp+/+9rB94rYOd1X44kSZ1hRvHdYxyFSJI0aKjbbSTZDbgfcIep96rqg6MqSpKkYYaZvwV4LF1AfYHuWnzfBgwoSdLIDDNI4kBgH+DXVXUo8CBg8UirkiQteMME1E1VdRtwS5KtgKuAu4+2LEnSQjfMOahlSbYG3gucBawAvjfKoiRJGmYU34v7p0cn+RKwVVX9cLRlSZIWulkDKskec02rqrNHU5IkSXO3oN4+x7QCxnaxWEnSwjNrQFXV48ZZiCRJg4YZxSdJ0tgZUJKkJhlQkqQmzTWKb9equnC20XyO4pMkjdJco/gOBw5j5tF8juKTJI3UXKP4Duuf7tNf6uhPktxhhkUkSVpnhjkHddzgiySbA58fTTmSJHWGCajLkxwFkOSOwFeAD4+0KknSgrfKgKqqNwMrkhwNnAq8vareP/LKJEkL2lyj+J4x8PL7wJuBM4BK8oyq+vSoi5MkLVxzjeJ7yrTXPwA26t8vwICSJI3MXKP4Dh1nIZIkDZqri+91VfWvSd5F12JaSVW9fKSVSZIWtLm6+C7o/102jkLmct7ly1l6hCPbJc1vlxy5/6RLmFfmCqh7JtkL+EhV3TKugiRJgrkDagfgncCuSc4DvgN8F/huVV0zhtokSQvYXIMkXgOQZGNgT+ARwKHAsUmuq6r7jadESdJCNFcLasqmwFbA4v5xBXDeKIuSJGmuUXzHAvcHbqD7ou53gXdU1bVjqk2StIDNdamjHYFNgF8DlwOXAdeNoSZJkuY8B7VvktC1oh4BvBrYLck1wPeq6i1jqlGStADNeQ6qqgr4UZLrgOX948nAXoABJUkambnOQb2cruX0COBm+iHmwPE4SEKSNGJztaCWAp8EXlVVV46nHEmSOnOdgzp8nIVIkjRomDvqrrEkByc5I8k5SY5JsijJiiRvS3JuktOT3GWUNUiS5qeRBVSS+wLPAh5ZVbsDtwLPATYHTq+qBwGnAX87qhokSfPXMFeSWFP7AA8BzuxGq7MpcBXwR+CUfp6zgL+YaeEkhwGHASzaatsRlilJatEoAyrACVX1hpXeTF7TD1+HrlU1Yw1VdSxwLMAm2937dvejkiSt30Z5Duq/gAOT3BkgyTZJdhrh9iRJ65GRtaCq6vwkbwJOTbIB3XepXjKq7UmS1i+j7OKjqk4ETpz29hYD008CThplDZKk+Wmkw8wlSVpTBpQkqUkGlCSpSQaUJKlJBpQkqUkGlCSpSQaUJKlJBpQkqUkj/aLuuvKA7Rez7Mj9J12GJGmMbEFJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKatOGkCxjGeZcvZ+kRn590GZK0xi45cv9JlzDv2IKSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1aWIBleSVSTab1PYlSW2bZAvqlYABJUma0civxZdkKfAl4CxgD+DHwGnA3YCvJ7m6qh436jokSfPLuFpQuwBHVdV9geuBjYErgMfNFk5JDkuyLMmyW29cPqYyJUmtGFdAXVpV3+mffxh41KoWqKpjq2rPqtpz0WaLR1udJKk54wqoWsVrSZJWMq6A2jHJ3v3zZwPfBm4AthzT9iVJ88y4AuonwEuSXADcEXgPcCzwpSRfH1MNkqR5ZFx31L2lqg6e9t67+ockSbfjlSQkSU0aeQuqqi4Bdhv1diRJ6xdbUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCaN64u6a+UB2y9m2ZH7T7oMSdIY2YKSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDUpVTXpGlYpyQ3ATyZdxxpYAlw96SLWgHWPl3WP13ytG+Zv7btU1Zaru9C8uN0G8JOq2nPSRayuJMuse3yse7yse/zma+1Jlq3JcnbxSZKaZEBJkpo0XwLq2EkXsIase7yse7yse/zma+1rVPe8GCQhSVp45ksLSpK0wBhQkqQmNRNQSfZN8pMkFyU5YobpmyQ5sZ/+/SRLJ1DmjIao/ZAkv01yTv94wSTqnFbT8UmuSvKjWaYnyf/pP9MPk+wx7hpnMkTdj02yfGBf/924a5xJkrsn+XqS85P8OMkrZpinuX0+ZN3N7fMkd0hyRpJz+7rfOsM8zR1Thqy7uePJlCSLkvwgySkzTFv9/V1VE38Ai4CLgZ2BjYFzgftNm+fFwNH984OAEydd92rUfgjw7knXOq2mxwB7AD+aZfp+wBeBAA8Hvj/pmoes+7HAKZOuc4a6tgP26J9vCfx0ht+T5vb5kHU3t8/7fbhF/3wj4PvAw6fN09wxZci6mzueDNR2OPDRmX4f1mR/t9KC2gu4qKp+XlV/BD4OHDBtngOAE/rnJwH7JMkYa5zNMLU3p6pOA66ZY5YDgA9W53Rg6yTbjae62Q1Rd5Oq6sqqOrt/fgNwAbD9tNma2+dD1t2cfh+u6F9u1D+mjwhr7pgyZN1NSrIDsD9w3CyzrPb+biWgtgcuHXh9Gbf/T/CnearqFmA5cKexVDe3YWoH+Ku+2+akJHcfT2lrZdjP1aK9+y6SLya5/6SLma7v2ngw3V/Hg5re53PUDQ3u87676RzgKuArVTXr/m7pmDJE3dDm8eSdwOuA22aZvtr7u5WAWt99DlhaVQ8EvsKf/4rQunc2sFNVPQh4F/DZyZazsiRbAJ8CXllV10+6nmGtou4m93lV3VpVuwM7AHsl2W3CJQ1liLqbO54keTJwVVWdtS7X20pAXQ4M/hWwQ//ejPMk2RBYDPxuLNXNbZW1V9XvquoP/cvjgIeMqba1MczPpDlVdf1UF0lVfQHYKMmSCZcFQJKN6A7yH6mqT88wS5P7fFV1t7zPAarqOuDrwL7TJrV6TAFmr7vR48kjgacmuYTuNMfjk3x42jyrvb9bCagzgXsnuUeSjelOoJ08bZ6Tgef3zw8Evlb92bYJW2Xt084jPJWuH791JwPP60eWPRxYXlVXTrqoVUly16l+7SR70f2OT/yg09f0PuCCqnrHLLM1t8+HqbvFfZ5k2yRb9883Bf4CuHDabM0dU4apu8XjSVW9oap2qKqldMfAr1XVwdNmW+393cTVzKvqliQvBb5MNyru+Kr6cZJ/AJZV1cl0/0k+lOQiupPkB02u4j8bsvaXJ3kqcAtd7YdMrOBeko/Rjb5akuQy4C10J2SpqqOBL9CNKrsIuBE4dDKVrmyIug8E/leSW4CbgIMmfdDpPRJ4LnBef34B4I3AjtD0Ph+m7hb3+XbACUkW0QXmJ6rqlHlwTBmm7uaOJ7NZ2/3tpY4kSU1qpYtPkqSVGFCSpCYZUJKkJhlQkqQmGVCSpCYZUFrw+u/xfDzJxUnOSvKFJPcZ4/YPSXK3Wabt2l+x+gdJ7pnku0Osb8Us738gyYFrW680LgaUFrT+C6afAb5RVfesqocAbwDuMuTyG871ekiHADMGFPA04KSqenBVXVxVj1iD9UvzkgGlhe5xwM39F04BqKpzq+pb/RUd/i3Jj5Kcl+RZ8Kf7H30rycnA+TO8XtQvd2Z/Qc8XTq07yev7dZ2b5Mi+RbMn8JG+pbTpwLz7Aa+k+xLs1/v3VgxMf+3ANma6b1CSvDvdvcq+Ctx5YNqR6e7x9MMk/77O9qa0DjVxJQlpgnYDZrvA5TOA3YEHAUuAM5Oc1k/bA9itqn6R5LHTXh9Gd5mihybZBPhOklOBXeluOfCwqroxyTZVdU1/JZLXVNWywY1X1ReSHA2sqKqVQiTJE4F7093uJcDJSR7T345kytOBXYD70bUIzweOT3KnftquVVVTl9aRWmNASbN7FPCxqroV+E2SbwIPBa4HzqiqXwzMO/j6icADB873LKYLkycA76+qGwGqam3ua/XE/vGD/vUW/TYGA+oxA/VfkeRr/fvLgf8G3pfuzqe3u/up1AIDSgvdj+muJbe6fj/H6wAvq6ovD86Q5C/XYDuzCfC/q+qY1V2wv37kXsA+dJ/9pcDj12Ft0jrhOSgtdF8DNum75QBI8sAkjwa+BTyrP6e0LV2L5Iwh1vlluvNGG/Xru0+Szenu3XNoks3697fp57+B7nbqq+PLwP9Md58mkmyf5M7T5jltoP7t6M63Td3baXF/a4xX0XVhSs2xBaUFrT8H83TgnUleT9f1dQnd4IRvA3sD59Lddvt1VfXrJLuuYrXHAUuBs/tRgr8FnlZVX0qyO7AsyR/prl7+RuADwNFJbgL2rqqbhqj71CT3Bb7XbYIVwMF0d2Gd8hm6ltH5wK+A7/Xvbwn8vyR3oGuJHb6q7UmT4NXMJUlNsotPktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktSk/w8e1g9vTz//GQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "success_by_lang = scores.groupby(\"wiki_lang\")[\"citoid_success\"].mean().sort_values().reset_index()\n",
    "\n",
    "plot_data = success_by_lang.set_index(\"wiki_lang\")\n",
    "plot_data = plot_data.sort_values(by=\"citoid_success\")\n",
    "ax = plot_data.plot.barh(legend=False)\n",
    "\n",
    "ax.set_title(\"Citoid success by language\")\n",
    "ax.set_ylabel(\"Wiki lang\")\n",
    "ax.set_xlabel(\"Correct fields\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/citoid_by_lang.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Citoid's performance by language shows the best results for French Wikipedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4 By sourcetype\n",
    "\n",
    "This section shows Citoid's performance by source type.\n",
    "\n",
    "In the following graphic we can observe the 10 most used source types. The source types related to web sources (mapped as \"Webpage, forumPost, email...\" and \"webpage, forumPost, email...\") count the overwhelming majority of the citations (almost 70% of the corpus). This is due to the URL presence in the citation as a mandatory criterion to be included in the corpus (see section 2. Citation template metadata extraction). The unmmaped source types (see section 4. Data normalization) are only 3% of the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 394,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_sources = scores[\"source_type_manual\"].value_counts().head(10).sort_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 395,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_type_show = {\n",
    "    \"['conferencePaper', 'dictionaryEntry', 'encyclopediaArticle', 'journalArticle', 'presentation', 'report', 'thesis']\": \"dictionaryEntry, encyclopediaArticle...\",\n",
    "    \"['bookSection', 'conferencePaper', 'dictionaryEntry', 'encyclopediaArticle', 'journalArticle', 'magazineArticle', 'newspaperArticle', 'presentation', 'report', 'thesis']\": \"bookSection, conferencePaper...\",\n",
    "    \"['artwork', 'audioRecording', 'bill', 'bookSection', 'case', 'computerProgram', 'dictionaryEntry', 'document', 'encyclopediaArticle', 'film', 'hearing', 'instantMessage', 'interview', 'letter', 'map', 'podcast', 'radioBroadcast', 'statute', 'tvBroadcast', 'videoRecording']\": \"podcast, radioBroadcast, interview...\",\n",
    "    \"['book', 'bookSection', 'manuscript']\": \"book, bookSection, manuscript\",\n",
    "    \"['conferencePaper', 'journalArticle', 'presentation', 'report']\": \"conferencePaper, journalArticle...\",\n",
    "    \"[]\":\"unmapped\",\n",
    "    \"['book', 'manuscript']\": \"book, manuscript\",\n",
    "    \"['magazineArticle', 'newspaperArticle']\": \"magazineArticle, newspaperArticle\",\n",
    "    \"['blogPost', 'email', 'forumPost', 'podcast', 'videoRecording', 'webpage']\": \"webpage, videoRecording, podcast...\",\n",
    "    \"['blogPost', 'email', 'forumPost', 'webpage']\":\"webpage, forumPost, email...\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 396,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_sources_for_plot = top_sources.rename(index=source_type_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABOzUlEQVR4nO3debwcVZ3+8c9DQAIEAggyAZXLJsgaIEHZFBAZFVRQMCKyqSAqm/zQYUZFwGWiOAMiCgJi2ER2RVARWcMasicsAYUo24AIhh0heX5/1GmotHfpJH1zk3uf9+t1X1196tQ536pqTX85p07LNhEREREREbHglujrACIiIiIiIvqLJFgRERERERFtkgQrIiIiIiKiTZJgRUREREREtEkSrIiIiIiIiDZJghUREREREdEmSbAiIiIiIiLaJAlWREREL5L0fO1vjqSXau/3aVMfn5B0m6QXJd3Yyf7hkiaU/RMkDW9Hv4sCSZa0bl/HERHRkAQrIiKiF9ke0vgD/gp8uFZ2QZu6eRo4GRjdvEPSm4BfA+cDKwHnAL8u5X1C0pJ91XdERG9LghUREdEHJC0t6WRJj5W/kyUtXfbtIOkRSf8l6SlJM7sb7bL9R9sXA491snsHYEngZNuv2D4FELBTF3EdIOlBSc9JeqjRr6QlJH1d0l8kPSnpXElD6/E2tTNT0s5l+zhJl0o6X9KzwAGSVpb083Luz0j6Ve3Y3SRNlvSPMjK3aRex3lw2p5QRwVGSpkv6cK3OUuUabi6po4x4HVz6fVzS0bW6S0g6RtKfJf1d0sWSVi77Bpf4/17iukvSal3dk4gYuJJgRURE9I2vAe8GhgObAVsBX6/t/zdgFWANYH/gDEnrz0c/GwFTbbtWNrWUz0XScsApwAdtLw9sA0wuuw8ofzsCawNDgFPnIY6PApcCKwIXAOcBy5Y43gKcVGLYHDgb+DzwZuCnwJWN5LPO9nvK5mZlRPAi4Fzg07VqHwIetz2pVrYjsB6wC/AfjUQQOAzYHXgvsDrwDPDjsm9/YCjwthLXIcBL83D+ETFAJMGKiIjoG/sAJ9h+0vbfgOOBfZvqfKOMOt0EXA18Yj76GQLMaiqbBSzfRf05wMaSlrH9uO27a/H+r+0HbT8P/CfwyXmY7ne77V/ZnkOVZH0QOMT2M7ZfLecIcDDwU9t32p5t+xzgFapktBXnAx+StEJ5vy9VMld3vO0XbE8Dfg7sXcoPAb5m+xHbrwDHAXuWc3yVKrFat8Q1wfazLcYUEQNIEqyIiIi+sTrwl9r7v5Syhmdsv9DN/lY9D6zQVLYC8FxzxdLfKKpE43FJV0vaoJt4lwRanSb3cG37bcDTtp/ppN6awP8r0/D+IekfpX5L5277MeBW4OOSVqRK5JqfdavHUr+uawJX1Pq9F5hNdY7nAdcAvyzTC78vaalWYoqIgSUJVkRERN94jOoLfcPbmfsZqpXKlL2u9rfqbmBTSaqVbVrK/4Xta2y/HxgG3Aec2U28rwFPAC9QTfcDQNIgYNXmpmvbDwMrlwSo2cPAd2yvWPtb1vaF3Z/mXM6hmia4F9XI2aNN+9/WdB6N6/ow1fTIet+DbT9aRtmOt70h1dTJ3YD95iGmiBggkmBFRET0jQuBr0taVdIqwLFU09vqjpf0JknbU32hv6SzhiQNkjSYakRpibIgQ2N05UaqUZjDy8Iah5by6ztpZzVJHy2J3StUo19zavF+WdJakoYA3wUusv0acD8wWNKupd+vA//yzFSD7ceB3wE/kbRSWYii8TzVmcAhkt6lynKl3a6mND5B9UxY3a+ALYAjqJ7JavYNSctK2gg4ELiolJ8OfEfSmuV6rCrpo2V7R0mblOTxWaopg3M6aTsiBrgkWBEREX3j28B4qgUnpgETS1nD/1EtsvAY1RS3Q2zf10Vb+1ItuHAasH3ZPhPA9j+pFm7YD/gH8Blg91LebAngqNLn01SLPXyh7DubaprczcBDwMtUi0JgexbwReAs4FGqEa25VhXsIuZXqUbJngSOLG2NBw6iWkDjGeBPVItrdOU44Jwyre8TpY2XgMuAtYDLOznmptLudcAPbP+hlP8QuBL4g6TngDuAd5V9/0a1SMezVFMHb+Jfn+2KiEBzLyoUERERfU3SDsD5tt/ax6EstiQdC7zD9qdrZR1UyeFSZeQtIqLt8kN/ERER0a+U3676LP+6KmNERK/LFMGIiIjoNyQdRLVYxe9s39xT/YiIdssUwYiIiIiIiDbJCFZERERERESb5BmsiIVklVVWcUdHR1+HERERERFtMGHChKdsN//mXxKsiIWlo6OD8ePH93UYEREREdEGkv7SWXmmCEZERERERLRJEqyIiIiIiIg2SYIVERERERHRJkmwIiIiIiIi2iSLXEQsJNMenUXHMVf3dRgRERER/crM0bv2dQhzyQhWREREREREmyTBagNJMyWt0tdx1ElaWtIfJU2WNGoh9jtG0kOl34mStp6PNo6UtGxvxNdi/2dJ2rBsL3L3NiIiIiIWXZki2H9tDmB7eKsHSBpke3Yb+v6K7Usl7QL8FNh0Ho8/EjgfeLENscwz25/ri34jIiIiYvGXEaxC0lckHV62T5J0fdneSdIFZXsXSbeXkZlLJA2pNfFVSdMkjZO0bqk/RtLpksZLul/SbqW8Q9LY0s5ESduU8iUk/UTSfZKulfRbSXuWfVtKuknSBEnXSBrWzbm8hSpBGVlGktaR9D5Jk0qMZ0tautSdKel7kiYCe0m6UdKIsm8VSTPL9gGSflXiminpUElHlTbvkLRyJ6HcDDSuxVGSppe/I0vZcpKuljSllI8q92B14AZJN/Rwz9aR9PtyTcZK2qB23U8rcT0oaYdyzvdKGlM7/rRyb+6WdHyt/PVrEBERERExL5JgvWEssH3ZHgEMkbRUKbu5TBP7OrCz7S2A8cBRteNn2d4EOBU4uVbeAWwF7AqcLmkw8CTw/tLOKOCUUvdjpf6GwL7A1gAljh8Be9reEjgb+E5XJ2L7SeBzwNgygvUoMAYYVWJcEvhC7ZC/297C9i97uEYblxhHlv5ftL05cDuwXyf1PwxMk7QlcCDwLuDdwEGSNgc+ADxmezPbGwO/t30K8Biwo+0de4jnDOCwck2OBn5S27cS1fX7MnAlcBKwEbCJpOGlztdsj6AaYXuvpHkdaeuRpINLEjd+9ouz2t18RERERCxikmC9YQKwpaQVgFeokoYRVAnWWKrEYEPgVkmTgf2BNWvHX1h7rT93dLHtObYfAB4ENgCWAs6UNA24pLQLsB1wSan/f0BjBGd9quTm2tL314G3zsO5rQ88ZPv+8v4c4D21/Re12M4Ntp+z/TdgFvCbUj6NKjFsOLHEeTDw2XJeV9h+wfbzwOVU13Ua8P4ygra97ZYzkDJ6uA1wSenrp0B9VO83tl36eML2NNtzgLtrsX6ijNxNokq+NqTNbJ9he4TtEYOWHdru5iMiIiJiEZNnsArbr0p6CDgAuA2YCuxINcXtXmAd4Frbe3fVRAvbjfdfBp4ANqNKcl/uITwBd9ue5wUjWvRCbfs13ki8BzfVe6W2Paf2fg5zf5a+YvvSxhtJ7+usU9v3S9oC+BDwbUnX2T6hxZiXAP7RzTNm9dia415S0lpUo14jbT9Tpg42n29ERERExDzJCNbcxlJ96b65bB8CTCojIXcA29aer1pO0jtqx46qvd5eK9+rPFu1DrA2MAMYCjxeRlT2BQaVurcCHy/1VwN2KOUzgFVVVuSTtJSkjcr2oZIO7eG8ZgAdjdhLnzd1UXcmsGXZ3rOHdls1Fthd0rKSlgP2AMZKWp1qmuH5wInAFqX+c8DyjYMlnStpq3qDtp8FHpK0V6kjSZvNQ0wrUCWWs8q1/uB8nltERERExOuSYM1tLNU0s9ttP0E1sjQWoEyLOwC4UNJUqiRqg9qxK5XyI6hGqBr+CowDfgccYvtlqmeF9pc0pbTRGEG6DHgEuIdqkYqJVM92/ZMq2fleOWYy1fQ4yvF/7+6kSp8HUk2nm0Y1inN6F9V/AHxB0iSgLcuT255I9QzYOOBO4Czbk4BNgHFlit83gW+XQ84Afl9b5GJTqueymu0DfLZck7uBj85DTFOopgbeB/yCKrntVll0ZPWyfYKkj5Ttj0hqdeQtIiIiIvoxVYMz0RvKtLOr6tPlWjhmiO3nJb2ZKiHZtjyP1VX9q4CPlSSs3ynPxP3M9l59HcuCWnrYeh62/8l9HUZEREREvzJz9K590q+kCWXBtLnkGaxFz1WSVgTeBHyru+QKwPZuCyWqPlKmAi72yRXAJmsMZXwf/R9ARERERCwcSbB6ke0D5uOYHdofSURERERELAx5BisiIiIiIqJNkmBFRERERES0SRKsiIiIiIiINkmCFRERERER0SZJsCIiIiIiItokCVZERERERESbJMGKiIiIiIhokyRYERERERERbZIEKyIiIiIiok2W7OsAIgaKaY/OouOYq/s6jIiFZuboXfs6hIiIiIUuI1gRERERERFtslglWJJmSlqlr+NohaRDJO3XSXmHpOnz2WaHpJckTZZ0j6RzJS214NG23P8YSXuW7bMkbbiw+m7FglzbLtrbQdI27WovIiIiIvq/TBHsJbZP76Wm/2x7uKRBwLXAJ4AL2t2JpCVtv9bVftufa3efi6AdgOeB2/o4joiIiIhYTCz0ESxJX5F0eNk+SdL1ZXsnSReU7V0k3S5poqRLJA2pNfFVSdMkjZO0bqk/RtLpksZLul/SbqW8Q9LY0s7ExmiEpCUk/UTSfZKulfTb2sjMlpJukjRB0jWShnVzLkuUUbUVa2UPSFpN0nGSjq61OUXSFOBLtbqDJJ0o6S5JUyV9vpSrlE8v5zqquW/bs4FxwBrdxS1pXUl/LP1PlLROV+2XEZuxkq4E7in1TpU0Q9IfgbfUYr9R0oiy/byk75Q+7pC0Wilfp7yfJunbkp5v4fPxfPlc3C3pOkmrlvLhpa2pkq6QtFIL1/YH5RynSjqslB9brvd0SWdIUik/XNWo4FRJv5TUARwCfFnViOH2PcUeEREREdEXUwTHAo0vqyOAIaqmuW0P3KxqCuDXgZ1tbwGMB46qHT/L9ibAqcDJtfIOYCtgV+B0SYOBJ4H3l3ZGAaeUuh8r9TcE9gW2Bihx/AjY0/aWwNnAd7o6EdtzgF8De5Tj3wX8xfYTTVV/Dhxme7Om8s+W8xkJjAQOkrRWiW84sBmwM3Bic6JXzu9dwO97iPsC4Mel722Ax3tofwvgCNvvKOe1frlO+5XjO7MccEfp42bgoFL+Q+CH5X490sWxnbU13vZGwE3AN0v5ucB/2N4UmFYr7+raHkx1j4eXYxqjfKfaHml7Y2AZYLdSfgyweal7iO2ZwOnASbaH2x7bYvxzkXSwqsR//OwXZ81PExERERGxGOmLBGsCsKWkFYBXgNupEq3tqZKvd1N9ob9V0mRgf2DN2vEX1l63rpVfbHuO7QeAB4ENgKWAMyVNAy4p7QJsB1xS6v8fcEMpXx/YGLi29P114K09nM9FVMkbwCfL+9eV0a0Vbd9cis6r7d4F2K/0dSfwZmC9Et+FtmeXZO0mqgQMYJ1S/wngcdtTu4pb0vLAGravALD9su0Xe2h/nO2HyvZ7avUeA67v4hr8E7iqbE+gSmyguj+XlO1fdHFsszm8cQ3PB7aTNJTqGt5Uys8B3tPDtd0Z+GljmqPtp0v5jpLuLJ+JnYCNSvlU4AJJnwa6nBo5r2yfYXuE7RGDlh3armYjIiIiYhG10J/Bsv2qpIeAA6iebZkK7AisC9wLrANca3vvrppoYbvx/stUichmVMnkyz2EJ+Bu21v3UK/udmDdMpVtd+Db83CsqEZfrpmrUPpgN8c0nsFahSoJ/QjwUGdxlwRrXr0wH8e8artx/WfT3s9V832db2XU7yfACNsPSzoOGFx270qVUH4Y+JqkTdrVb0REREQMHH21iuBY4Giq6WRjqZ51mVS+pN8BbKs3nq9aTtI7aseOqr3eXivfS9UzUesAawMzgKFUozxzqKYCDip1bwU+XuqvRrWYAeWYVSW9PmVQ0kZl+1BJhzafSIn5CuB/gXtt/71p/z+Af0jarhTtU9t9DfCFMsUPSe+QtFy5JqPKc0SrUn3xH9fU7lNU09r+s6u4bT8HPCJp91K+tKRlW2m/uLlWbxhVIjwv7gA+XrY/Wd8h6b4ujlkC2LNsfwq4xfYs4Jnac1D7Ajf1cG2vBT4vacnS38q8kUw9peq5vsZzd0sAb7N9A/AfVJ+bIcBzwPwkqRERERExQPVlgjUMuL1MUXu5lGH7b1SjWxdKmkqVRG1QO3alUn4E1QhVw1+pkoTfUT1D8zLVaMX+ZQGEDXhjdOYyqmeC7qGahjaR6lmof1J96f5eOWYybzx3tAEwV/JUcxHwaZqmB9YcCPy4TN9TrfysEsNEVcuL/5Rq9OcKqpG9KVTT8r5apjI2+xWwLNWzWF3FvS9weLlmtwH/Ng/tXwE8UGI8l7kT2lYcCRxV+l4XmAVQRt/UxTEvAFuV67ETcEIp35/qWbGpVM+PNcq7u7Z/BaaWa/KpkpCdCUynSm7vKnUHAeeXaYOTgFNK3d8AezQWuZD0EUknlHNYXdJvG52pWihl9Xm7PBERERHR3+iNmV2LL0ljgKtsXzoPxwyx/bykN1MlZtt2kWQ06l8FfKwkYdGCMlr2km1L+iSwt+2PqlrlcW3bp3RyzPO2h/xLY/3A0sPW87D9T+7rMCIWmpmjd+3rECIiInqNpAm2RzSXD+TfwbqqLJLwJuBb3SVXALZ3625/dGpL4NSyFPo/gM8A2L6qu4P6q03WGMr4fOGMiIiI6Nf6RYJl+4D5OGaH9kcSdWVp8+bl03s6pl+OXkVERETEwNBXz2BFRERERET0O0mwIiIiIiIi2iQJVkRERERERJskwYqIiIiIiGiTJFgRERERERFtkgQrIiIiIiKiTZJgRUREREREtEkSrIiIiIiIiDZJghUREREREdEmS/Z1ABEDxbRHZ9FxzNV9HUYEM0fv2tchRERE9FsZwYqIiIiIiGiTJFg9kHSCpJ0XsI2TJT0qqcvrLWm4pA/V3n9E0jE9tPv8gsTVn0naXZIlbdBNnRUlfbH2fnVJl/bQ7o2SRrQz1oiIiIjoP5Jg9cD2sbb/OL/Hl6RqD+Bh4L1d1FkSGA68nmDZvtL26PntdyAq17Fhb+CW8tpV3RWB1xMs24/Z3rM3Y4yIiIiI/m2RTbAkdUi6T9IYSfdLukDSzpJulfSApK1Kva0k3S5pkqTbJK1fypeVdLGkeyRdIenOxsiDpNMkjZd0t6TjS9kISZPL3zRJLuVjJO1ZtmdKOl7SxFJng1K+nKSzJY0rcXy0dio7AHcDp1H7si/pOEnnSboVOA84ARhV+h8l6QBJp5a6q5VzmFL+tunken1F0l2SpjbOqYfrO0/nIulqSZuW7UmSji3bJ0g6SNIwSTeX+KdL2r7sf17SSeVaXydp1VJ+UIl3iqTLJC1bu96nl/tzv6TdSvkgSSfWzvHzpXwHSWMlXQncU8qGANsBnwU+WTvn5rqjgXVKzCeWz9z0Wn8/KOcyVdJhnVzDXcpnb6KkS0q/ERERETGALbIJVrEu8D/ABuXvU1RfnI8G/qvUuQ/Y3vbmwLHAd0v5F4FnbG8IfAPYstbu12yPADYF3itpU9vjbQ+3PRz4PfCDLmJ6yvYWVAnT0Y32gOttbwXsCJwoabmyb2/gQuAKYFdJS9Xa2hDY2fbeJfaLSgwXNfV5CnCT7c2ALagSttdJ2gVYD9iKaiRsS0nv6SL++T2XscD2koYCrwHblvrbAzdT3ZtryvXbDJhc9i8HjLe9EXAT8M1SfrntkeWc7qVKhho6yrnsCpwuaXDZP8v2SGAkcJCktUr9LYAjbL+jvP8o8Hvb9wN/l1S/9/W6xwB/Ltf8K03X5uASx3DbmwIX1HdKWgX4OtX92wIYDxzVfIElHVySxfGzX5zVvDsiIiIi+plFfRXBh2xPA5B0N3CdbUuaRvXlF2AocI6k9QADjQRmO+CHALanS5paa/cTkg6mOv9hVInO1NLPKKov4bt0EdPl5XUC8LGyvQvwEUmNJGUw8HZJf6aa9neU7eck3Qn8O3BVqXel7ZdauA47AfuVc5kNNH9T36X8TSrvh1AlXDf30G7L50KVYB0OPARcDby/jDqtZXuGpNWAs0sC+Svbk8vxc4BGwnh+rc+NJX2bapreEOCaWlwX254DPCDpQarkehdg08ZoItV9Xw/4JzDO9kO14/em3Hvgl+X9hPK+uW5XdgZOt/0agO2nm/a/m+pzc6skgDcBtzc3YvsM4AyApYet5xb6jYiIiIjF2KKeYL1S255Tez+HN2L/FnCD7T0kdQA3dtdgGfU4Ghhp+xlJY6iSCCRtDBwHvKckMt3FNLsWg4CP257R1NeHqRKIaeVL+LLAS7yRYL3QXazzQMB/2/7pPB43L+fyJmAE8CBwLbAKcBAlcbF9cxk12xUYI+l/bZ/bSZ+NJGMMsLvtKZIOoJpK2Vyn/l7AYbbriRiSdqB2HSWtTJWQbqJqmucgwJIaI1TtvObXltHHiIiIiAhg0Z8i2IqhwKNl+4Ba+a3AJwAkbQhsUspXoPqSPauMunyw1FmRairffrb/No8xXAMcppJFSdq8lO8NfM52h+0OYC3eGPlp9hywfBftXwd8obQ9qEzTa+7/M41ngCStIektZfs6SWss6LnY/ifVQh17UY3UjKVKVG8u9dYEnrB9JnAW1SggVJ+xxqjTp6gWnqCc6+NlxGufphj2krSEpHWAtYEZJa4vNKZYSnpHbRpm3Z7AebbXLNf9bVSjbtt3Ure7a34t8HmVhTNK4lZ3B7CtpHXL/uUkvYOIiIiIGND6Q4L1feC/JU1i7hG5nwCrSroH+DbVc0uzbE+hmkp3H/ALqkQMqud21gTOLIseTJ6HGL5FNTVxapnK+K2SRH2AajodALZfoEowPtxJGzcAG5a+RzXtOwLYsUyNnEA1Ne11tv9QzuX2UudSYHlVKxiuCzRPb5unc6ntGws8WaY1jgXeWl6hGoGaUu7DKN6YovcCsFVZPGInqsU8oHou7k6q639fUwx/BcYBvwMOsf0yVdJ2DzCxtPVTOh+B3Zvqebe6y+hkNUHbf6ea4jdd0olNu88qcUyVNIUqOawf+zeqhP7CMv30dqqpjBERERExgMnun4+FSBoELGX75TIS8kdg/TISMyCUKY+fsf0viy8sxBiet93y6nplyuZVtrv9ParF0dLD1vOw/U/u6zAimDl6174OISIiYrEnaUJZOG8ui/ozWAtiWeCGMqVMwBcHUnIF1eIedLKyXfSNTdYYyvh8sY2IiIjo1/ptgmX7OapFGaIPzcvoVal/QC+FEhERERHR6/rDM1gRERERERGLhCRYERERERERbZIEKyIiIiIiok2SYEVERERERLRJEqyIiIiIiIg2SYIVERERERHRJkmwIiIiIiIi2iQJVkRERERERJskwYqIiIiIiGiTJfs6gIiBYtqjs+g45uq+DiM6MXP0rn0dQkRERPQTGcGKiIiIiIhokyRYA5CkDknT29TWTEmrtKOtRY2kj0g6poc6wyV9aGHFFBERERGLtiRYEZ2QtKTtK22P7qHqcCAJVkREREQASbAGsiUlXSDpXkmXSloWQNL7JE2SNE3S2ZKW7q68QdIykn4n6aDuOi0jXv8tabKk8ZK2kHSNpD9LOqTUGSLpOkkTS38fLeUdJd4zJd0t6Q+Slin7bpQ0omyvImlm2d5I0rjS31RJ65Xy/cr7KZLOK2VjJJ0u6U7g+5IOkHRq077xku6XtJukNwEnAKNK+6Pac2siIiIiYnGVBGvgWh/4ie13As8CX5Q0GBgDjLK9CdUiKF/oqrzW1hDgN8CFts9soe+/2h4OjC3t7gm8Gzi+7H8Z2MP2FsCOwP9IUtm3HvBj2xsB/wA+3kNfhwA/LP2NAB6RtBHwdWAn25sBR9TqvxXYxvZRnbTVAWwF7AqcTvW/n2OBi2wPt31R8wGSDi5J2fjZL87qIdSIiIiIWNwlwRq4HrZ9a9k+H9iOKul6yPb9pfwc4D3dlDf8Gvi57XNb7PvK8joNuNP2c7b/BrwiaUVAwHclTQX+CKwBrFaOecj25LI9gSrp6c7twH9J+g9gTdsvATsBl9h+CsD207X6l9ie3UVbF9ueY/sB4EFgg55O1PYZtkfYHjFo2aE9VY+IiIiIxVwSrIHLPbyfF7cCH6iNMvXklfI6p7bdeL8ksA+wKrBlGXl6AhjcdCzAbN74qYHXeOPz3KiL7V8AHwFeAn4raaceYnuhm33tvGYRERER0Q8lwRq43i5p67L9KeAWYAbQIWndUr4vcFM35Q3HAs8AP24UlGeo1pjP2IYCT9p+VdKOwJotHDMT2LJs71mLY23gQdunUI20bQpcD+wl6c2lzsotxrWXpCUkrQOsTXVdngOWb/H4iIiIiOjnkmANXDOAL0m6F1gJOM32y8CBwCWSplGNKJ3eVXlTe0cAy0j6vqQlgHWBp5k/FwAjSl/7Afe1cMwPqJ4XmwTUl43/BDBd0mRgY+Bc23cD3wFukjQF+N8W4/orMA74HXBIuS43ABtmkYuIiIiIAJCdWU7RXpI2Bj7TxUIRiyVJY4CrbF86v20sPWw9D9v/5LbFFO0zc/SufR1CRERELGYkTbA9orl8yc4qNx24GvBdYHXbH5S0IbC17Z/1QpzRD9ieDvSb5KpdNlljKOPzRT4iIiKiX2tliuAY4Bpg9fL+fuDIXoonYpFk+4AFGb2KiIiIiIGhlQRrFdsXUz13g+3XqFZvi4iIiIiIiJpWEqwXymprBpD0biC/mBoREREREdGkx2ewqJ6luRJYR9KtVL9PtGf3h0RERERERAw8PSZYtidKei+wPiBghu1Xez2yiIiIiIiIxUwrqwgOBr4IbEc1TXCspMZvI0VERERERETRyhTBc4HngB+V958CzgP26q2gIiIiIiIiFketJFgb296w9v4GSff0VkARERERERGLq1ZWEZxYVg4EQNK7gPG9F1JERERERMTiqZURrC2B2yT9tbx/OzBD0jTAtjfttegiIiIiIiIWI60kWB/o9SgiBoBpj86i45ir+zqM180cvWtfhxARERHR77SSYB0O/Mx2nruKiIiIiIjoRivPYN0LnCnpTkmHSBra20FFLE4kHSDp1L6OIyIiIiL6Xo8Jlu2zbG8L7Ad0AFMl/ULSjr0dXERERERExOKklREsJA0CNih/TwFTgKMk/bIXY4t+TlKHpOm190dLOk7SjZK+J2mcpPslbV/2HyDpV5KulTRT0qGSjpI0SdIdklYu9Q6SdJekKZIuk7RsKR8j6XRJ40u7u9Xa/XXp9wFJ36zF9OkSx2RJPy3/W0DSgaWNccC2C/GyRURERMQirMcES9JJwH3Ah4Dv2t7S9vdsfxjYvLcDjAFrSdtbAUcC36yVbwx8DBgJfAd40fbmwO1Uo6wAl9seaXszqimun60d3wFsBewKnC5pcCnfCvg4sCmwl6QRkt4JjAK2tT0cmA3sI2kYcDxVYrUdUP+duLlIOrgkdONnvzhrvi5ERERERCw+WlnkYirwddsvdLJvqzbHE9FweXmdQJUUNdxg+zngOUmzgN+U8mlUyRHAxpK+DawIDAGuqR1/se05wAOSHqQalQW41vbfASRdTpU4vUb1MwV3SQJYBngSeBdwo+2/lfoXAe/o7CRsnwGcAbD0sPU8b5cgIiIiIhY3rUwR/HRzciXpOgDb+U/ysSBeY+7P4ODa9ivldTZz/4eAV2rbc2rv59TqjQEOtb0J1UhTvd3mJMfdlAs4x/bw8re+7eO6O6GIiIiIGNi6TLAkDS7PtKwiaSVJK5e/DmCNhRZh9GdPAG+R9GZJSwO7tand5YHHJS0F7NO0by9JS0haB1gbmFHK318+38sAuwO3AtcBe0p6C0DZvyZwJ/DeEvdSwF5tijsiIiIiFnPdTRH8PNXzL6tTTdNSKX8WyJLUscBsvyrpBGAc8CjVs37t8A2qJOhv5XX52r6/lv5WAA6x/XKZ/jcOuAx4K3C+7fEAkr4O/EHSEsCrwJds3yHpOKrnvv4BTG5T3BERERGxmJPd/WMhkg6z/aOFFE9Er5E0BrjK9qVN5QcAI2wf2pv9Lz1sPQ/b/+Te7GKezBy9a1+HEBEREbHYkjTB9ojm8h4XuUhyFdEem6wxlPFJaiIiIiL6tVZWEYzoF2wf0EX5GKqFMSIiIiIiFkhLPzQcERERERERPWvlh4Yl6dOSji3v3y4pv38VERERERHRpJURrJ8AWwN7l/fPAT/utYgiIiIiIiIWU608g/Uu21tImgRg+xlJb+rluCIiIiIiIhY7rYxgvSppEGAASasCc3o1qoiIiIiIiMVQKwnWKcAVwFskfQe4Bfhur0YVERERERGxGGrld7AukDQBeB8gYHfb9/Z6ZBEREREREYuZHhMsSe8G7rb94/J+BUnvsn1nr0cXERERERGxGGlliuBpwPO198+XsoiIiIiIiKhpZRVB2Xbjje05klo5LiJqpj06i45jrl5o/c0cvetC6ysiIiIiKq2MYD0o6XBJS5W/I4AHezuwiIiIiIiIxU0rCdYhwDbAo8AjwLuAg3szqIVJ0omS7pZ0Yl/H0iDpAEl/kzRZ0j2SDurDWG7rw753kHRVU9mvJN3RwnHb1N4fImm/bup3SJq+4BFHRERExEDX7VS/8vtXJ9n+5EKKpy8cDKxse3YrlSUtafu1Xo4J4CLbh0p6C3C3pCttP9HuTno6H9vbdLVvHvsZ1Oo17qaNFYEtgeclrW37X0ZSy/TVHaieFbwNwPbpC9JvRERERESruh3BKl+I15T0poUUT8sk7SdpqqQpks4rZR2Sri/l10l6eykfI+kUSbdJelDSnqX8SmAIMEHSKEmrSrpM0l3lb9tS7zhJ50m6FTivh3pnS7qx9HN4D/F22k6d7SeBP1Pdh9MkjS8jbsfX2p4p6fuSpkkaJ2nd7tpvPp8ervPz5VVltG966WdUKZ9rlEnSqZIOqMX1PUkTgb3K++MlTSxtbFDqbSXpdkmTyj1av4twPgb8Bvgl8HrSX+7v6ZLuBC6mGnX9chkB3L6c79Gl7rqS/ljuw0RJ6zSd76BynneV+/X57q5PRERERERdK4tVPAjcWpKRFxqFtv+316LqgaSNgK8D29h+StLKZdePgHNsnyPpM1Q/krx72TcM2A7YALgSuNT2RyQ9b3t4afcXVCN2t5Tk7BrgneX4DYHtbL/UQ70NgB2B5YEZkk4D3tFFvD/spp3Gua4NrA38Cfia7adVjSxeJ2lT21NL1Vm2N1E1Fe5kYLce2n/9fFq87B8DhgObAasAd0m6uYXj/m57i3Iuo4GnbG8h6YvA0cDngPuA7W2/Jmlnqh+y/ngnbe0NnAA8AVzG3D94/Vaq6ztb0nHA87Z/UPp9X63eBcBo21dIGkz1HxneUtv/WaprOVLS0lSf/T/YfqiFc/0Xkg6mTKkdtMKq89NERERERCxGWkmw/lz+lqBKGhYFOwGX2H4KwPbTpXxrqkQAqpGZ79eO+ZXtOcA9klbrot2dgQ0lNd6vIGlI2b6ylox0V+9q268Ar0h6Elitm3i7a2eUpO2AV4DPl8TqkPKFfUmqhHFDoJFgXVh7PWkez6cV2wEXllHNJyTdBIwEnu3huIua3l9eXifwxr0aCpwjaT3AwFLNjZR7th5wi21LelXSxrYbz05d0tMUREnLA2vYvgLA9sulvF5tF2BTlVHOEtt6wHwlWLbPAM4AWHrYeu6hekREREQs5npMsGwf31OdxcQrtW11UWcJ4N2NL96vV66+gL/QYr16P7Pp/hp3185Ftg+tla1FNeIz0vYzksYAg2uHuZPtVs9nQbzG3FNNBzftb+6ncX3q1+ZbwA2295DUAdzYST+fAFYCHirxr0A1ovW1LvqZXwIOs31Nm9qLiIiIiAGkx1UEJd2g6rmmuf4WRnDduJ7qmZ43lxgbU+5u441nc/YBxs5ju38ADmu8kTR8Aev1FO+8tLMCVRIxq4zmfLBp/6ja6+3z0r6kNSRd18M5jKUaVRskaVXgPcA44C9Uo2RLq1qE4n3dtNGVoVSrVAIc0EWdvYEP2O6w3UG12EVXi688RyejrbafAx6RtDtAiXnZpmrXAF+QtFSp8w5Jy7V+KhERERExkLUyRfDo2vZgqmdjFsYqel2yfbek7wA3SZoNTKL6Yn4Y8HNJXwH+Bhw4j00fDvxY0lSqa3Mz1YIJ81uvp3hbbsf2FEmTqJ5Xehi4tanKSqWdV6iSkXmJcxhd39PGaNgVVFMwp5Syr9r+PwBJFwPTqabRTeqine58n2qK4NeBf/kl3jKqtSbw+vLsth+SNEvSuzpp7zfApZI+Si3BLPYFfirpBOBVYC9gTm3/WUAHMFHVUNnfKM/xSZpce17vLOB02+MlHVJiymqFEREREQOc7Hl/LETSONtb9UI8MR8kzQRGNJ7xmo/jDwX+avvKpvI3AxNtr7ngUcbSw9bzsP1PXmj9zRy960LrKyIiImKgkTTB9ojm8h5HsGrT2aCaUrgl1ZSu6Cdsn9pcJml1qmehfrDQA+qnNlljKOOT9ERERET0a61MEZxANSVMVNPIHqJayjoWEeWZpHa3+RjV8vIREREREdGiVlYRXGthBBIREREREbG4a2WK4FLAF6hWjYNq2thPbb/ai3FFREREREQsdlqZInga1Q+//qS837eUfa63goqIiIiIiFgctZJgjbS9We399ZKm9FZAERERERERi6sef2gYmC1pncYbSWsDs3svpIiIiIiIiMVTKyNYXwFukPQg1UqCazLvP+AbERERERHR77WyiuB1ktYD1i9FM2y/0rthRURERERELH66nCIoaaSkfwMoCdVw4FvAiU0/PhwRERERERF0/wzWT4F/Akh6DzAaOBeYBZzR+6FFREREREQsXrqbIjjI9tNlexRwhu3LgMskTe71yCL6mWmPzqLjmKsXuJ2Zo3dtQzQRERER0Ru6G8EaJKmRgL0PuL62r5XFMSIiIiIiIgaU7hKsC4GbJP0aeAkYCyBpXappgn1G0g6Srmpje7tL2rBd7XXS/hhJe5bts3rqS9JMSdMkTS6vH+2luI6TdHQb2ztS0rIt1DtB0s491NlB0jZtjO0jko5pV3sREREREZ3pciTK9nckXQcMA/5g22XXEsBhCyO4hWh34CrgnlYPkLSk7dfmtSPbn2ux6o62n5K0PvAH4NdN/QuQ7TnzGkMvOhI4H3ixu0q2j22hrR2A54HbWu28u3ti+0rgylbbioiIiIiYH93+0LDtO2xfYfuFWtn9ticuSKeSOiTdJ+kCSfdKurQx8iHpfZImlZGbsyUtXco/UI6ZCHys1tYQST8v9adK+ngpP03SeEl3Szq+Vn+0pHtK3R+UUZKPUK2OOLn+o8qdxH2cpPMk3QqcV85jrKSJ5W+bUk+STpU0Q9IfgbfU2rhR0oiyvXeJe7qk73XR7QrAM7XrNkPSucB04G2STizHT5M0qnZNrisxzTUCJulrku6XdAtvLL2PpHUl/VHSlHLcOl21I2k5SVeXutMljZJ0OLA61W+m3dDD/a+P6M2UdHytjw0kdQCHAF8u92R7SatKukzSXeVv2y7uyR2SNmq+3pIOkHRqKeuqrWmSViz37++S9ivl50p6f3fnFBEREREBffss1frAZ23fKuls4IvlC/AY4H227y+JxBcknQ6cCewE/Am4qNbON4BZtjcBkLRSKf+a7aclDQKuk7Qp8CiwB7CBbUta0fY/JF0JXGX70hbi3hDYzvZLqpLC99t+WdVvhV0IjCh9rF/qrkY1MnZ2vRFJqwPfA7akSqD+IGl3278qVW6QJGBt4BO1Q9cD9rd9R0kmhwObAasAd0m6GfgbsIftZyWtAtxRznEL4JPlmCWBicCE0u4FwGjbV0gaTJV8/7OLdj4APGZ713IuQ23PknQUZeSthetY95TtLSR9ETja9ufKPX/e9g9KH78ATrJ9i6S3A9cA7yzH1+/Jl8v1+qakYcAw2+MlbVzr74ddtHUrsC3wF+BBYHuqlTO3Br4wj+dEiftg4GCAQSusOj9NRERERMRipNsRrF72sO1by/b5wHZUSclDtu8v5ecA7wE2KOUPlKmK59fa2Rn4ceON7WfK5idUjXZNAjai+hI+C3gZ+Jmkj9HDVLYuXGn7pbK9FHCmpGnAJaUPSswX2p5t+zHmXiCkYSRwo+2/lWltF5TjGna0vTGwCXCqpCGl/C+27yjb29X6eQK4qbQr4LuSpgJ/BNagSvS2B66w/aLtZylT5iQtD6xh+woA2y/bfrGbdqYB75f0PUnb217QZ/IuL68TgI4u6uxcrsPkEvcKtWtSvycXA3uW7U8AnSXNXbU1luoevAc4DdhE0hrAM/VR3Hlh+wzbI2yPGLTs0PlpIiIiIiIWI305guUe3s83SWsBRwMjbT8jaQww2PZrkraiWhVxT+BQqlGxeVH/ov1l4AmqEaQlqJK3trL9Z0lPUCVvTzb135V9gFWBLW2/KmkmMHg+uu+0nTK6uAXwIeDbkq6zfcJ8tN/wSnmdTdefySWAd9ue6xpXg3zUp7A+Wqb3bUr18wKHzENbNwNfAt4OfI1qJHJPygIvERERERE96csRrLdL2rpsfwq4BZgBdKhaqRBgX6pRmftKeeP5qL1r7VxL9aUYeH2K4ApUX7pnSVoN+GDZNwQYavu3VMnRZuWw54Dla20cKunQFs5hKPB4WWhiX2BQKb8ZGCVpUJmmtmMnx44D3itplTKNce9yrnOR9BZgLappa83G1vpZlWrkZVyJ68mSFO0IrFmLa3dJy5RRqw8D2H4OeETS7qXPpcv0x07bKdMbX7R9PnAi1dRD+NfreG5JaOfHXG1RLfTx+uIqkoZ3c+xFwFep7vXUTvZ32pbth6mmWq5n+0Gqz+TRVNctIiIiIqJHfZlgzQC+JOleYCXgtDKicCBwSZl2Nwc4vZQfDFxdpv09WWvn28BKqhZbmEI1tW4K1dTA+4BfUD1bA9UX9qvKlLdbgKNK+S+Br6haXGMdqimJf2/hHH4C7F/63YA3RlKuAB6gevbqXOD25gNtPw4cA9wATAEm2K6vFHhDmcJ2A3BMmQLY7Apgajn+euCrtv+ParrhiHIN9yvXgbI4yUWl/u+Au2pt7QscXq7NbcC/ddUO1bTFcSW+b1LdA4AzgN/rjUUuNgUe6+zCteA3wB4qi1wAh5dYpkq6h85HphoupXrW7OIu9nfX1p1AY4rqWKppkbcAqPYD26qW228sVnKIpO7iiYiIiIgBQm+svr4QO61WibuqPGO0yFH1G1sfs/3Pvo5lcSVpBeBntvfq61gWFSNGjPD48eP7OoyIiIiIaANJE2yPaC7vy2ewFlm2d+vrGBZ3ZRGNJFcRERERMaD0SYJleyawSI5eRUREREREzK++fAYrIiIiIiKiX0mCFRERERER0SZJsCIiIiIiItokCVZERERERESbJMGKiIiIiIhokyRYERERERERbZIEKyIiIiIiok2SYEVERERERLRJn/zQcMRANO3RWXQcc3W3dWaO3nUhRRMRERERvSEjWBEREREREW2SBCsiIiIiIqJN+l2CJalD0vQ2tTVT0iqdlH9N0t2SpkqaLOld8xnnp2rvR0g6ZUFj7i2S9pJ0r6Qb+jqWhnINXyr34B5Jp0vqd5/piIiIiFh85BmseSRpa2A3YAvbr5QE7E3z0VQH8CngFwC2xwPj2xVnL/gscJDtW1qpLGlJ26/1ckwAf7Y9XNKSwPXA7sDl7e5kIZ5PRERERCzG+ut/7V9S0gVlxOVSScsCSHqfpEmSpkk6W9LS3ZU3SFpG0u8kHQQMA56y/QqA7adsP1bqbSnpJkkTJF0jaVgpX1fSHyVNkTRR0jrAaGD7MvryZUk7SLqq1F9Z0q/KCNkdkjYt5ceV+G6U9KCkw3u6EJJGSrqt9D1O0vKSBkv6eTnfSZJ2LHUPkHS5pN9LekDS90v5scB2wM8knShpUHm9q8T4+VJvB0ljJV0J3NNDvRvLvbmv3Ct1E2+n7dSV5Oc2YF1JB5W6UyRdVrv/Y8oo13hJ90varZS3dD6tfPAiIiIiYmDrrwnW+sBPbL8TeBb4oqTBwBhglO1NqEbvvtBVea2tIcBvgAttnwn8AXhb+YL+E0nvBZC0FPAjYE/bWwJnA98pbVwA/Nj2ZsA2wOPAMcBY28Ntn9QU//HAJNubAv8FnFvbtwHw78BWwDdLv52S9CbgIuCI0vfOwEvAlwCX890bOKdcB4DhwChgE2CUpLfZPoFqdG0f21+hGs2aZXskMBI4SNJa5fgtSn/v6KHe5sCRwIbA2sC23cTbXTuNc10WeB8wDbjc9sjSxr3l+IaOcu12BU4v593q+cwzSQeXhG787BdnzU8TEREREbEY6a9TBB+2fWvZPh84HLgWeMj2/aX8HKpE44Yuyk8u738NfN/2BQC2n5e0JbA9sCNwkaRjqBKQjYFry2DMIOBxScsDa9i+ohz/MkCp05XtgI+X+tdLerOkFcq+q8vo2SuSngRWAx7pop31gcdt31Xaerb0vR1VMojt+yT9BWgkENfZnlXq3QOsCTzc1O4uwKaS9izvhwLrAf8Extl+qMV6j5R+JlMlPrO6iLerdu4H1inHG/i17d9Jeq+kbwMrUiXI19Riv9j2HOABSQ9SJaytns88s30GcAbA0sPW8/y2ExERERGLh/6aYDV/kV2QL7a3Ah+Q9AvbBrA9G7gRuFHSNGB/YAJwt+2t6weXBKudXqltz6b997CV9gUcZvuauQqlHYAXWqw3L+fRVTsdlGewmuqPAXa3PUXSAcAOtX2dfTZaPZ+IiIiIiG711ymCb1e1GAVUC0ncAswAOiStW8r3BW7qprzhWOAZ4McAktaXtF5t/3DgL6WdVRv9SlpK0ka2nwMekbR7KV+6TGd7Dugq+RoL7FPq70D1zNez3Z2wpOskrdFUPAMYJmlkqbO8qsUg6u2/A3h7qduqa6imVy7VaEPScgtQr6d457Wd5alGD5dqnGfNXpKWUPUc3Nqlz3ltPyIiIiKiU/01wZoBfEnSvcBKwGllat6BwCVl1GkOcHpX5U3tHQEso2rRhyFUzyzdI2kq1TNEx9n+J7An8D1JU4DJVM9bQZW0HV7q3wb8GzAVmF0WYvhyU3/HAVuW+qOpRsi6pGpp8nWBp+vlJaZRwI9KTNcCg4GfAEuU870IOKCxaEeLzqJa9GGiqiXxf0rnI1Ct1usp3nlqB/gGcCfV6ON9Tfv+CowDfgccUu5/S+1LOkvSiLJ9iKRDyvYISWd1E09EREREDBAqs95iMSZpY+Azto/q61gWZZLGAFfZvrQv+h8xYoTHj1+UV+KPiIiIiFZJmmB7RHN5f30Ga0CxPR1IchURERER0ceSYMWAYfuAvo4hIiIiIvq3/voMVkRERERExEKXBCsiIiIiIqJNkmBFRERERES0SRKsiIiIiIiINkmCFRERERER0SZJsCIiIiIiItokCVZERERERESbJMGKiIiIiIhok/zQcMRCMu3RWXQcc3Wn+2aO3nUhRxMRERERvSEjWBEREREREW2SBCsiIiIiIqJNFkqCJek4SUeX7RMk7dxN3eGSPlR7/xFJxyyMOLuIZ7akybW/bmORtIOkbRZWfL1F0kxJq/Ri+x2SppftEZJOafG4kyU9KqnLz+78fIYkPd9q7BERERERXVnoz2DZPraHKsOBEcBvS/0rgSt7Kx5JAmR7ThdVXrI9fB6a3AF4Hritk76WtP3aPAfZz9keD4zvqV5JqvYAHgbeC9zQSZ0lWcifoYiIiIiIhl4bwZL0NUn3S7oFWL9WPkbSnmV7pKTbJE2RNE7SUOAEYFQZLRol6QBJp5b6HZKulzRV0nWS3l5r85TS1oO19oeUehMlTZP00Vo7MySdC0wHviHp5FqMB0k6qYfzmynp+FrbG0jqAA4Bvlzi377EdrqkO4HvS3pA0qqljSUk/anxvot+Bkk6UdJd5bw/X8p3kHSjpEsl3SfpgpIsdnZdl5d0s6ThtXZvkbRZuUY/L+cwVdLHO4nhKEnTy9+RtWvY6PfeEseyZd+Wkm6SNEHSNZKG1cqnSJoCfKnW/g6SrirbW0m6XdKkcg7r10LZAbgbOA3Yu3b8cZLOk3QrcB7df4ZWk3RFIw51Mtoo6Su16318V/cmIiIiIqJZryRYkrYEPkk1kvAhYGQndd4EXAQcYXszYGfgBeBY4CLbw21f1HTYj4BzbG8KXADUp5UNA7YDdgNGl7KXgT1sbwHsCPxPIwkB1gN+Ynsj4H+AD0taquw7EDi7bC+juacIjqr1+VRp+zTgaNszgdOBk0r8Y0u9twLb2D4KOB/Yp5TvDEyx/bfOryQAnwVm2R5JdR0PkrRW2bc5cCSwIbA2sG0X1/Ul4GfAAQCS3gEMtj0F+EZpf5NyXa+vd17u5YHAu4B3l/43L7vXL9fwncCzwBfLNfwRsKftLct1/E6p/3PgsBJXV+4Dtre9OdVn4bu1fXsDFwJXALvW7hflGuxse2+6/wydAtxUYtiCKmGrn+8uVJ+Nrag+v1tKek838XZL0sGSxksaP/vFWfPbTEREREQsJnprBGt74ArbL9p+ls6nZ60PPG77LgDbz7YwfW5r4Bdl+zyqhKrhV7bn2L4HWK2UCfiupKnAH4E1avv+YvuO0vfzVInFbpI2AJayPa3Ue6l8UR/eyRf2y8vrBKCjm7gvsT27bJ8N7Fe2P0OVdHRnF2A/SZOBO4E3UyUAAONsP1KmN04uMXR1XS8p57dU6XdMaWNn4MeNzmw/09T/dlT38oVynS6nur8AD9u+tWyfX+quD2wMXFti/jrwVkkrAivavrnUP6+L8x0KXKLq+ayTgI3g9YT8Q1T3+dlyLf69dtyVtl/qos26nagSYmzPtt2c9exS/iYBE4ENeON6zzPbZ9geYXvEoGWHzm8zEREREbGY6E+/g/VKbbsxSrUPsCqwpe1XJc0EBpd9LzQdfxbwX1QjKD0lPc19zqb7a/l6X7YflvSEpJ2oRkn26fowoDqXw2xfM1ehtANzn3O3Mdh+UdK1wEeBTwBb9tBvK9zJewF32966Kd4VW2zzW8ANtvcoUy5vLOX/DqwITCuDkMtSjcxdVfY338/5JeC/bf+0Te1FRERExADSWyNYNwO7S1pG0vLAhzupMwMYJmkkQHlOaEngOWD5Ltq9jWrqIVSJydgu6jUMBZ4sydWOwJpdVbR9J/A24FNU09DmV3fxN5xFNeLz+siWpD0k/Xcnda8BvtCYDifpHZKW66btrq5ro99TgLtqI1XXMvfzUCs1tTeW6l4uW/rdgzeu+9slNRKpTwG3lP5XbZRLWkrSRrb/AfxDUmPUsavEcijwaNk+oFa+N/A52x22O4C1gPc3nvtq0t09uA74QoltkKrn/uquAT4jaUips4akt3TRVkRERETEXHolwbI9keo5oCnA74C7OqnzT2AU8KOy6MG1VKNLNwAbdvK8E8BhwIFlyt++wBE9hHIBMELSNKppeff1UP9i4NamaXLNz2CN7urg4jfAHqXu9l3UuRIYwtwjZetQPcfU7CzgHmBimTb3U7ofqerqumJ7Qumj3u+3gZXKAhZTqJ5Vq7c3kWo64TiqaXln2Z5Uds8AviTpXmAl4LTS/57A90p7k4HGQhIHAj8uUwcbo4zNvg/8t6RJjfMsSdQHgKtrcb1AldB1lrx39xk6AtixfCYmUD27VT/fP1BNQ7291LmUkqxJ+q2k1cv2CZI+UrY/IumELs4nIiIiIgYQ2c2zvAauspLdSbav6+V+RpR+tq+VnQ98uYcFLxa039Wpptxt0M2y9K221QFcZXvjNoQ2IIwYMcLjx/e4Gn1ERERELAYkTbA9orl8ofzQ8KJO0oqS7qda0KK3k6tjgMuA/6yX2/50LydX+1GNQH1tQZOriIiIiIjoXEawIhaSjGBFRERE9B8ZwYqIiIiIiOhlSbAiIiIiIiLaJAlWREREREREmyTBioiIiIiIaJMkWBEREREREW2SBCsiIiIiIqJNkmBFRERERES0SRKsiIiIiIiINlmyrwOIGCimPTqLjmOu/pfymaN37YNoIiIiIqI3ZAQrIiIiIiKiTZJgRUREREREtMmASbAkdUia3qa2Zkpapbfqd9POGEl7dlK+m6RJkqZIukfS5+ez/f9qen/b/MbaH0haXdKlPdRZUdIXF1ZMEREREbFoGzAJVn8laSngDODDtjcDNgdunM/m5kqwbG+zYNEtviQtafsx2/+S0DZZEUiCFRERERHAwEuwlpR0gaR7JV0qaVkASe8rI0DTJJ0taenuyhskLSPpd5IOaqHvr5Z2xklatxzfIel6SVMlXSfp7d2VN/X9LUljgBWoFiv5O4DtV2zPKHVWlXSZpLvK37alfIikn5d4pkr6uKTRwDKSJku6oNR7vrxK0omSppdjRpXyHSTdWK7lfeXaqruLUEbhTpN0h6QHSxtnl3syplbvNEnjJd0t6fha+UxJx0uaWGLZoJQfJ+noWr3p5TouJ+nqMro3vRb7SEm3lfJxkpaXdICkKyVdD1xXH/Us+35dzvcBSd8sXY0G1inX7cQWPgcRERER0Y8NtARrfeAntt8JPAt8UdJgYAwwyvYmVMnKF7oqr7U1BPgNcKHtM1voe1Zp51Tg5FL2I+Ac25sCFwCn9FAOQPkivypwoO2/A1cCf5F0oaR9JDXu6w+Bk2yPBD4OnFXKv9GIp/Rxve1jgJdsD7e9T1PsHwOGA5sBOwMnShpW9m0OHAlsCKwNbNvCtVgJ2Br4con9JGAjYBNJw0udr9keAWwKvFfSprXjn7K9BXAacDTd+wDwmO3NbG8M/F7Sm4CLgCPKqN/OwEul/hbAnrbf20lbW1Fdx02BvSSNAI4B/lyu21eaD5B0cEkUx89+cVYPoUZERETE4m6gJVgP2761bJ8PbEeVdD1k+/5Sfg7wnm7KG34N/Nz2uS32fWHtdeuyvTXwi7J9Xomnu3KokqOhtg+xbQDbnwPeB4yjSjjOLnV3Bk6VNJkqkVlB0pBS/uNGg7af6SH27agSydm2nwBuAkaWfeNsP2J7DjAZ6OihLYDflNinAU/YnlaOv7t2/CckTQQmUSVfG9aOv7y8Tmihv2nA+yV9T9L2tmdR3dvHbd8FYPtZ26+V+tfafrqLtq61/XfbL5UYtuui3utsn2F7hO0Rg5Yd2lP1iIiIiFjMDbQEyz28nxe3Ah/oaUpcF30tSL93AVtKWnmuxqsk5STg/VSjLFDd33eX0ZXhttew/fwC9N2ZV2rbs2ntt9Uax8xpOn4O1TTOtagSxfeVEbargcGdHF/v7zXm/jwPBigJ8hZUida3JR3bQ2wvdLOvnZ+fiIiIiOiHBlqC9XZJjdGjTwG3ADOAjsZzUcC+VCM0XZU3HAs8Q20kqDwvtUYXfY+qvd5etm8DPlm29wHG9lAO8Huq536uLs8NDZG0Q23/cOAvZfsPwGG1+IaXzWuBL9XKVyqbr6paNKPZWGCUpEGSVqUayRvXxXk22vxvSXt0V6cbK1AlOrMkrQZ8sIVjZlIlUkjaAlirbK8OvGj7fODEUmcGMEzSyFJneUmtJIbvl7SypGWA3amS7OeA5Vs/tYiIiIjozwZagjUD+JKke6meAzrN9svAgcAlkqZRjaKc3lV5U3tHUC0M8f3y3NO6QFfTy1aSNLUc8+VSdhhwYCnft+zrrhwA25cAZ1JN+xPVAhozylTA44EDStXDgRGqFrK4BziklH+7xDNd0hRgx1J+BjBVZZGLmiuAqcAU4Hrgq7b/r4vzbNgE6KlOp2xPoZoaeB/VVMlbuz8CgMuAlSXdDRwKNKZ2bgKMK9fmm8C3bf+TKtH9UTn/a5l7hKwr40o/U4HLbI8vz8DdWq5lFrmIiIiIGOBUHuOJBSRpY+Azto/q61gWBZKusf3vfR1Hu0g6ABhh+9D5bWPEiBEeP358+4KKiIiIiD4jaUJZlG0urUyLihbYng4kuSr6U3IVEREREdGqJFgRLbA9hmrZ/oiIiIiILg20Z7AiIiIiIiJ6TRKsiIiIiIiINkmCFRERERER0SZZRTBiIZH0HNVPBcTAtArwVF8HEX0m939gy/2PfAb6pzVtr9pcmEUuIhaeGZ0t5RkDg6Txuf8DV+7/wJb7H/kMDCyZIhgREREREdEmSbAiIiIiIiLaJAlWxMJzRl8HEH0q939gy/0f2HL/I5+BASSLXERERERERLRJRrAiIiIiIiLaJAlWREREREREmyTBiuhlkj4gaYakP0k6pq/jiQUj6WxJT0qaXitbWdK1kh4oryuVckk6pdz7qZK2qB2zf6n/gKT9a+VbSppWjjlFkhbuGUZXJL1N0g2S7pF0t6QjSnnu/wAhabCkcZKmlM/A8aV8LUl3lvt2kaQ3lfKly/s/lf0dtbb+s5TPkPTvtfL8m7EIkzRI0iRJV5X3uffxL5JgRfQiSYOAHwMfBDYE9pa0Yd9GFQtoDPCBprJjgOtsrwdcV95Ddd/XK38HA6dB9YUc+CbwLmAr4JuNL+WlzkG145r7ir7zGvD/bG8IvBv4Uvnfc+7/wPEKsJPtzYDhwAckvRv4HnCS7XWBZ4DPlvqfBZ4p5SeVepTPzSeBjaju8U/KF/f8m7HoOwK4t/Y+9z7+RRKsiN61FfAn2w/a/ifwS+CjfRxTLADbNwNPNxV/FDinbJ8D7F4rP9eVO4AVJQ0D/h241vbTtp8BrqX6ojYMWMH2Ha5WIDq31lb0MduP255Ytp+j+pK1Brn/A0a5l8+Xt0uVPwM7AZeW8ubPQOOzcSnwvjIq+VHgl7Zfsf0Q8Ceqfy/yb8YiTNJbgV2Bs8p7kXsfnUiCFdG71gAerr1/pJRF/7Ka7cfL9v8Bq5Xtru5/d+WPdFIei5gy3Wdz4E5y/weUMtowGXiSKjn+M/AP26+VKvX79vq9LvtnAW9m3j8bsWg4GfgqMKe8fzO599GJJFgREW1URh7y+xf9mKQhwGXAkbafre/L/e//bM+2PRx4K9WowwZ9G1EsDJJ2A560PaGvY4lFXxKsiN71KPC22vu3lrLoX54o07sor0+W8q7uf3flb+2kPBYRkpaiSq4usH15Kc79H4Bs/wO4AdiaavrnkmVX/b69fq/L/qHA35n3z0b0vW2Bj0iaSTV9byfgh+TeRyeSYEX0rruA9coqQ2+ierD1yj6OKdrvSqCxEtz+wK9r5fuV1eTeDcwqU8muAXaRtFJZ3GAX4Jqy71lJ7y5z9fertRV9rNyTnwH32v7f2q7c/wFC0qqSVizbywDvp3oW7wZgz1Kt+TPQ+GzsCVxfRjmvBD5ZVppbi2pBk3Hk34xFlu3/tP1W2x1U9+V62/uQex+dWLLnKhExv2y/JulQqi9Ug4Czbd/dx2HFApB0IbADsIqkR6hWgxsNXCzps8BfgE+U6r8FPkT1EPOLwIEAtp+W9C2qf1ABTrDdWDjji1QrFS4D/K78xaJhW2BfYFp5Bgfgv8j9H0iGAeeUFd+WAC62fZWke4BfSvo2MIkqEae8nifpT1SL43wSwPbdki4G7qFanfJLtmcD5N+Mxc5/kHsfTVQl0xEREREREbGgMkUwIiIiIiKiTZJgRUREREREtEkSrIiIiIiIiDZJghUREREREdEmSbAiIiIiIiLaJAlWRETEIkSSJf1P7f3Rko5rU9tjJO3Zc80F7mcvSfdKuqHF+idI2rlsHylp2RaOmauepN82fqMqIqIvJcGKiIhYtLwCfEzSKn0dSJ2kefntzM8CB9nesZXKto+1/cfy9kigxwSruZ7tD9n+xzzEGBHRK5JgRURELFpeA84Avty8o3kEStLz5XUHSTdJ+rWkByWNlrSPpHGSpklap9bMzpLGS7pf0m7l+EGSTpR0l6Spkj5fa3espCupfhi1OZ69S/vTJX2vlB0LbAf8TNKJnRzzH+WYKZJG189L0uHA6sANjdEvSaeVeO+WdHwp66zezEZSKumoEtN0SUeWso4yqnZmaesPkpZptCfpnnLuv2z5TkVEdGJe/mtURERELBw/BqZK+v48HLMZ8E7gaeBB4CzbW0k6AjiMasQHoAPYCliHKkFZF9gPmGV7pKSlgVsl/aHU3wLY2PZD9c4krQ58D9gSeAb4g6TdbZ8gaSfgaNvjm475IPBR4F22X5S0cn2/7VMkHQXsaPupUvw1209LGgRcJ2nTLuo1+tgSOBB4FyDgTkk3lRjXA/a2fZCki4GPA+cDxwBr2X4l0wwjYkFlBCsiImIRY/tZ4Fzg8Hk47C7bj9t+Bfgz0EiQplElVQ0X255j+wGqRGwDYBdgP0mTgTuBN1MlIwDjmpOrYiRwo+2/2X4NuAB4Tw8x7gz83PaL5TyfbuG8PiFpIjAJ2AjYsIf62wFX2H7B9vPA5cD2Zd9DtieX7Qm8cV2mAhdI+jTVCGJExHxLghUREbFoOpnqWablamWvUf7tlrQE8Kbavldq23Nq7+cw94wVN/VjqpGew2wPL39r2W4kaC8syEksCElrAUcD77O9KXA1MHgBmqxfo9m8cV12pRo13AK4ax6fN4uImEsSrIiIiEVQGd25mCrJaphJNSUP4CPAUvPR9F6SlijPZa0NzACuAb4gaSkASe+QtFx3jQDjgPdKWqVM39sbuKmHY64FDmys/tc8RbB4Dli+bK9AleDNkrQa8MEu6tWNBXaXtGw5hz1KWadKovo22zcA/wEMBYb0cB4REV3Kf6GJiIhYdP0PcGjt/ZnAryVNAX7P/I0u/ZUqOVoBOMT2y5LOopouN1GSgL8Bu3fXiO3HJR0D3EA1Ana17V/3cMzvJQ0Hxkv6J/Bb4L+aqp0B/F7SY7Z3lDQJuA94GLi1q3q1PiZKGlPOEapn0SZJ6ugirEHA+ZKGlvM4JasRRsSCkN08UyAiIiIiIiLmR6YIRkREREREtEkSrIiIiIiIiDZJghUREREREdEmSbAiIiIiIiLaJAlWREREREREmyTBioiIiIiIaJMkWBEREREREW3y/wGZF8WOgIItOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = top_sources_for_plot.plot(figsize=[12,4], title=\"Top 10 source types\", legend=False, kind=\"barh\")\n",
    "ax.set_ylabel(\"Source type\")\n",
    "ax.set_xlabel(\"Number of citations\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/top_sourcetypes.png\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0458297751209793"
      ]
     },
     "execution_count": 400,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_sources_for_plot[\"unmapped\"]/len(scores) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will visualize Citoid's performance for these source types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [],
   "source": [
    "sourcetype_eval = scores.groupby(\"source_type_manual\")[\"comp_source_type\"].mean().sort_values().reset_index()\n",
    "\n",
    "plot_data = sourcetype_eval.set_index(\"source_type_manual\").loc[top_sources.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 403,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_data = plot_data.rename(index=source_type_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAEYCAYAAADrpHnMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABNr0lEQVR4nO2deZhdRbW33x8BQQiEmRsQaGZkDCRBUcAgyFVBBAEjohBUEFRAuejlE0VAVBSvIKKMYphEZkUQERlDGEIGMjAEFKJMMomBMBN+3x+1Dtk59Ok+nfR0kvU+z3l679pVq1bVPr3XXlV1ask2SZIkSdJKLNLXCiRJkiRJV0njlSRJkrQcabySJEmSliONV5IkSdJypPFKkiRJWo40XkmSJEnLkcYrSZI+Q9K2kqZ3cH20pON7qO5vSzq7cr67pEclzZK0haR7JY3ogXqvlbRfD8jtsb5qp67TJX13Hst2yz1P45Uk3YykbSTdLmmmpH9LGitpeF/r1RGSLGnd3q7X9hjbG/R0PZJGSHqsru4f2v5SJemnwNdsD7Q9yfbGtm+ez3qPkXRBXb0fs33u/MjtaVQ4VNI0SS9JekzSpZI2BbB9kO3vR9539G1HdNc9X3R+BSRJMgdJywBXAwcDlwDvArYFXuuBuha1/WZ3y12IWRO4t6+V6Cf8HNgZOAAYCwwAdo+0qX2o1xxs5yc/+emmDzAM+E8H1xcBvgP8A3gaOA8YFNdGAI/V5Z8B7BjHxwCXARcALwBfApYHfgM8ATwP/L5SdhfgHuA/wO3AZg10uhUw8BIwCxgJTAM+UcmzGPAssAXQFvkPjHqfBI6oa+ORwN+B5yhGfPkGdc/V5pA/EXgRuBj4HXB8M22KvjoCmALMjPJLAEsBrwBvRftmAatGf14ALB5ptT74ezt9PwD4drTpRWACsHpc+znwaNyTCcC2kf5R4HXgjZA/OdJvBr7UxPeh1s/7Af+M/j+qg+/WaOB04PrQ8RZgzbj2S+D/6vJfBXyjHTnrAbOBrTqp6/gO+nZx4OT4fjwRx4vPyz1v9MlhwyTpXh4EZks6V9LHJC1Xd31UfLYH1gYGAqd2Qf4nKQZsWeBC4HxgSWBjYGXgJABJWwDnAF8GVgDOAK6StHi9QNvbxeHmLkNmF1Meop+rZPs48KTtSZW07SkPup2A/5W0Y6QfAuwGfIjyIHue8vDsEEnvAn4fbVoeuBTYo3K9mTZ9mmI01gI2A0bZfgn4GPBEtG+g7Scq7X/N9sBKH6zTjnqHA3tHPywDfAF4Oa7dDQwJnX8LXCppCdt/Bn4IXBx1bt6O3FF0/n3YBtgA2AE4WtJ725FTYx/g+8CKFCN/YaSfC+wtaREASSsCO4a+9exAMS7jOqgHgA769ijg/ZR+2RzYimKk56Kze94RabySpBux/QLlYWPgLOAZSVdJWiWy7AP8zPbDtmcB/w/4jKRmh/DvsP17229RDNjHgINsP2/7Ddu3RL4DgTNs32V7tsscy2uUB0ozXAB8PIZBAT5PecBUOdb2S7anUry/vSP9IIqH8Jjt1ygezp5NtPH9FA/v5GjLZRTDUKOZNp1i+wnb/wb+SHl4dgdfAr5je7oLk20/B2D7AtvP2X7T9v9RvI5m53Sa+T4ca/sV25OByRRj0IhrbN8a/X4UsLWk1cMQzaQYJoDPADfbfqodGStQvOn5YR/gONtP234GOJbyHaqns3vekDReSdLN2L7f9ijb7wE2oXgfJ8flVSlDRDX+QZl7XoXmeLRyvDrwb9vPt5NvTeB/JP2n9on8qzbZhicocx17SFqWYiQvrMtW1eUfFdlrAldW6r2fMgzVWRtXBR53jCVV5HalTf+qHL9M8WS6g9UpQ4bvQNIRku6PBTr/AQZRPJ9maOb70JU2vX1Pwhj+mzn9cy5zvOnP8c6XkRrPAYM7U7wT2mtXe9+9zu55Q9J4JUkPYvsByvzAJpH0BOUhXGMN4E3gKcp8y5K1C5IGACvVi6wcPwosH8alnkeBH9hetvJZ0vZFXVC/9rDbi+LxPV53ffW6dtSG4h4FPlZX9xLtlK/nSWA1SaqT2x1tmt/wGY8C7xhOlLQt8C3KcOVytpeleDi1NnRWb0ffh3nh7XsiaSBlKK52Xy4APilpc+C9lOG69rgBeI+kYU3W2V4b22vXE+3k6+yeNySNV5J0I5I2lPQ/kt4T56tThtPujCwXAd+QtFY8XGpzIm9S5suWkLSzpMUocwTvmKOqYftJ4FrgV5KWk7SYpNr81VnAQZLeF8uelwq5SzcQ9xRlzqXK74EtgcMoc2D1fFfSkpI2BvanTLZDWTTwA0lrRh+sJOmTjdpR4Q7Kg/vQaMunKHMlNbrapvr2rSBpUBN52+Ns4PuS1ou6N5O0ArB06PwMsKikoylzYtV622pzTe3Q0fdhXvh4/FTjXZS5rzttPwpg+zHKkNz5wOW2X2lPgO2HgF8BF8Uy+HdJWkLSZyQd2U6R9vr2IuA7ce9XBI6mGM96OrvnDUnjlSTdy4vA+4C7JL1EMVrTgP+J6+dQHh63Ao8Ar1IWOGB7JvAVyoPycYon1tnvZz5PWc32AGW12tdD1njKMudTKQsm/kZZGNCIY4BzYzju0yHjFeByyuKHK9opc0vIvQH4qe2/RPrPKSvZ/iLpxeiD93XSDmy/Dnwq9Pw3ZdXjFZXrXW1TVfYDlAfqw9HGpoZPK/yMsmryL5RVhb8G3g1cB/yZ8uLxD8r9rA6nXhp/n5M0sR25Db8P88hvge9R+m8ocy+6geJNb0rjIcMah1L6+ZeUlZ1/pyyV/2N9xgZ9ezwwnrLycyplNeE7fnjc2T3vCM091JgkSTKH8CTWt/25Slob5UG72Hx4CEkfEJ75BZQl9C398M8fKSdJ0i6Slge+SPurxJIWI4aiDwPObnXDBTlsmCRJO0g6gDL8da3tW/tan2T+iN+G/YeyivDkPlWmm8hhwyRJkqTlSM8rSZIkaTlyzitJeokVV1zRbW1tfa1GkvQ7JkyY8Kzt+t80dkgaryTpJdra2hg/fnxfq5Ek/Q5JTe2qUSWHDZMkSZKWI41XkiRJ0nKk8UqSJElajjReSZIkScuRCzaSpJeY+vhM2o68pq/VSJJeY8YJO/eY7PS8kiRJkpajpYyXpBmxvX6/R9JBkvZtJ71N0rR5lNkm6RVJ90i6T9J5sV9ZryBptKQ94/hsSRv1Vt3NMD9920DeCEkf6C55SZJ0Hzls2EPYPr2HRP/d9pAIVHg9JQhefYTb+UbSoh3tGG77S91dZz9kBDALuL2P9UiSpI5e97wkfVPSoXF8kqQb4/jDki6M450k3SFpoqRLI0hbjW9JmippnKR1I/9oSadLGi/pQUm7RHqbpDEhZ2LtLVrSIpJ+JekBSddL+lPFoxgq6RZJEyRdJ6lhOOyQM0OVSLaSHpK0iqRjJB1RkTlZ0mTgq5W8AySdKOluSVMkfTnSFenToq0j6+u2PRsYB6zWkd6S1pX016h/oqR1GskPT2OMpKuA+yLfqZKmS/orsHJF95sVkVYlzZL0g6jjTkmrRPo6cT5V0vGSZjXx/ZgV34t7Jd0gaaVIHxKypki6UtJyTfTtT6ONUyQdEulHR39Pk3SmVCK4SjpUxZudIul3KmE/DqIECrxHJWJukiT9hL4YNhwD1B4Ew4CBKkNf2wK3qgwLfgfY0faWlIBmh1fKz7S9KSVQ2smV9DZKBM6dgdMlLUEJzveRkDMSOCXyfiryb0QJ97A1vB0y4BfAnraHUgLF/aBRQ2y/BfyBEqQNSe8D/mG7PoT3b4BDbG9el/7FaM9wYDhwgKS1Qr8hwObAjsCJ9UY02vc+4M+d6H0h8Muo+wOUsNsdyd8SOMz2+tGuDaKf9o3y7bEUJWLr5pSgegdE+s+Bn8f96iyoYlXWeNsbU4Idfi/SzwP+1/ZmlOB2tfRGfXsg5R4PiTI17/RU28Ntb0IJJrhLpB8JbBF5D7I9gxIR+CTbQ2yPaVL/uZB0oMpL1fjZL8+cFxFJkrRDXxivCcBQScsAr1HCQA+jGK8xwPspD8uxku4B9gPWrJS/qPJ360r6JbbfihDWDwMbAosBZ0maSoloWpuj2Qa4NPL/C7gp0jcANgGuj7q/A7ynk/ZcTDGMAJ9hTih0AMIrW7YSVqIawXQnYN+o6y5gBWC90O8i27PDEN5CMW4A60T+p4AnbU9ppLdKePTVbF8JYPtV2y93In+c7UfieLtKvieAGxv0wevA1XE8gWI0oNyfWiTZ3zYoW89bzOnDC4BtVMKLL2v7lkg/F9iuk77dETijNvRp+9+Rvr2ku+I78WFg40ifAlwo6XOUsOTdgu0zbQ+zPWzAkvMagT5Jknp6fc7L9huSHqGEfb6d8tDYHlgXuB9YB7je9t6NRDRxXDv/BuUhvznFUL/aiXoC7rW9dSf5qtwBrBvDW7vRTqjrTuo7xPZ1cyVKH+ugTG3Oa0WKgd+VEtX2HXqH8eoqL81DmTcqwe1m073fq26L2RPe6q+AYbYflXQMsERc3plirD8BHCVp0+6qN0mS7qevVhuOAY6gDDGNocwtTIoH4J3ABzVnPmspSetXyo6s/L2jkr6XyhzUOsDawHRgEMU7eYsyPDgg8o4F9oj8q1Am5okyK0l6exhR0sZx/DVJX6tvSOh8JfAz4H7bz9Vd/w/wH0nbRNI+lcvXAQfHsB+S1pe0VPTJyJi3WYnyUB1XJ/dZylDX/2ukt+0Xgcck7Rbpi0tashn5wa2VfIMpLxld4U5gjzj+TPWCpAcalFkE2DOOPwvcZnsm8Hxl3unzwC2d9O31wJclLRr1Lc8cQ/WsyjxqbZ5zEWB12zcB/0v53gwEXgTm5QUgSZIepi+N12Dgjhi2ejXSsP0MxSu7SNIUioHasFJ2uUg/jOJZ1fgn5QF8LWXO4lXKW/Z+MZm/IXO8isspczD3UYamJlLmnl6nPNB+HGXuYc48z4bAXIapwsXA56gbMqywP/DLGNJTJf3s0GGiyhLvMyhey5UUj3QyZajuWzG8Wc/vgSUpc1+N9P48cGj02e3Af3VB/pXAQ6Hjecz9stAMXwcOj7rXBWYChNeoBmVeAraK/vgwcFyk70eZm5tCma+rpXfUt/8EpkSffDaM3VnANMqLw92RdwBwQQwlTgJOibx/BHavLdiQtKuk46INq0r6U60ylUU/q3ate5IkmVcWiEjKkkYDV9u+rAtlBtqeJWkFitH7YIMHeC3/1cCnwsAlTRBe3iu2LekzwN62P6myGnRt26e0U2aW7YHvELYAsPjg9Tx4v5P7Wo0k6TWa3WFD0gTbw7oie2H+ndfVMeH/LuD7HRkuANu7dHQ9aZehwKmxHP0/wBcAbF/dUaEFlU1XG8T4HtwuJ0kWJhYI42V71DyUGdH9miRVYnl5/RL2zsoskF5XkiTdS0ttD5UkSZIkkMYrSZIkaUHSeCVJkiQtRxqvJEmSpOVI45UkSZK0HGm8kiRJkpYjjVeSJEnScqTxSpIkSVqONF5JkiRJy7FA7LCRJK3A1Mdn0nbkNX2tRpL0KM3uZzi/pOeVJEmStBxpvLoBSTMizEe/IWJ3/TXCeYzsvES31Tta0iNR78RajLEuyvh67EjfJ0g6W9JGcdzv7m2SJDlsuCCzBYDtIc0WkDTA9uxuqPubti+TtBMlRtlmXSz/dUqctZe7QZcuY/tLfVFvkiTNk55XIOmbkg6N45Mk3RjHH5Z0YRzvJOmO8CgujWi8Nb4laaqkcZUo0KMlnS5pvKQHI44VktokjQk5EyV9INIXkfQrSQ9Iuj4CHNai/Q6VdIukCZKui8jGjdqyMuXhPzw8oHUk7SBpUuh4jqTFI+8MST+WNJESjfpmScPi2oqSZsTxKEm/D71mqESWPjxk3hmRiuu5lRKEksg7LT5fj7SlJF0jaXKkj4x7sCpwk6SbOrln60j6c/TJGEkbVvr9tNDrYUkjos33R+y3WvnT4t7cK+nYSvrbfZAkSf8kjdccxgC1MPPDgIGSFou0W2Po6DvAjra3BMYDh1fKz7S9KXAqcHIlvQ3YCtgZOF3SEsDTwEdCzkigFpTxU5F/I0oE5K0BQo9fAHvaHgqcA/ygUUNsPw18CRgTntfjwGhgZOi4KHBwpchztre0/btO+miT0HF41P+y7S0oEZb3bSf/J4CpkoZSIh6/D3g/cICkLYCPAk/Y3tz2JsCfI0DlE8D2trfvRJ8zgUOiT46gRM6usRyl/74BXAWcBGwMbCppSOQ5KgLgbQZ8SFJXPcROkXRgGMjxs1+e2d3ik2ShJY3XHCYAQyUtA7xGeSAPoxivMZSH7kbAWJWQ8/sBa1bKX1T5W53nucT2W7YfAh4GNgQWA85SCTt/acgF2Aa4NPL/C6h5HhtQDMf1Ufd3gPd0oW0bAI/YfjDOzwW2q1y/uEk5N9l+0fYzwEzgj5E+lWJ0a5wYeh4IfDHadaXtl2zPAq6g9OtU4CPh+W1ru+mne3i9HwAujbrOAKre6B9dwoRPBZ6yPdX2W8C9FV0/HR7nJIph24huxvaZtofZHjZgyUHdLT5JFlpyziuw/YakR4BRwO3AFGB7yrDX/cA6wPW2924koonj2vk3gKcogRoXAV7tRD0B99ru8uKHJnmpcvwmc15qlqjL91rl+K3K+VvM/V36pu3LaieSdmivUtsPStoS+DhwvKQbbB/XpM6LAP/pYE6vqlu93otKWovirQ23/XwMJ9a3N0mSfkp6XnMzhvJAuzWODwImxRv8ncAHK/NZS0lav1J2ZOXvHZX0vWIuax1gbWA6MAh4MjyBzwMDIu9YYI/IvwowItKnAyspVu5JWkzSxnH8NUlf66Rd04G2mu5R5y0N8s4Ahsbxnp3IbZYxwG6SlpS0FLA7MEbSqpShxwuAE4EtI/+LwNK1wpLOk7RVVaDtF4BHJO0VeSSpK1Gbl6EY7ZnR1x+bx7YlSdIHpPGamzGUoac7bD9F8YjGAMRQ2SjgIklTKAZqw0rZ5SL9MIpnVeOfwDjgWuAg269S5mb2kzQ5ZNQ8n8uBx4D7KAsuJlLm0l6nGJIfR5l7KENmRPnnOmpU1Lk/ZYhtKsX7OL1B9p8CB0uaBHTLEnHbEylzbuOAu4CzbU8CNgXGxbDf94Djo8iZwJ8rCzY2o8yD1bMP8MXok3uBT3ZBp8mU4cIHgN9SXhw6JBbQrBrHx0naNY53ldSsx5gkSTeg4lQkPUEMRV1dHUJrosxA27MkrUB52H8w5r8a5b8a+FQYuAWOmIP8te29+lqX+WXxwet58H4n97UaSdKjzMsOG5ImxOKppsk5r/7H1ZKWBd4FfL8jwwVge5de0aqPiOHBljdcAJuuNojxvbR1TpIs6KTx6kFsj5qHMiO6X5MkSZIFi5zzSpIkSVqONF5JkiRJy5HGK0mSJGk50nglSZIkLUcaryRJkqTlSOOVJEmStBxpvJIkSZKWI41XkiRJ0nKk8UqSJElajtxhI0l6iamPz6TtyGv6Wo0k6RHmZU/D+SE9ryRJkqTlSOO1ECKpTdK0bpI1Q1K3hE7pb0SokyM7yTNE0sd7S6ckSQppvJKkHSQtavsq2yd0knUIJRJ0kiS9SBqvhZdFJV0o6X5Jl0laEkDSDpImSZoq6RxJi3eUXkPSuyVdK+mAjioNT+1Hku6RNF7SlpKuk/R3SQdFnoGSbpA0Mer7ZKS3hb5nSbpX0l8kvTuu3SxpWByvKGlGHG8saVzUN0XSepG+b5xPlnR+pI2WdLqku4CfSBol6dS6a+MlPShpF0nvAo4DRob8kSRJ0iuk8Vp42QD4le33Ai8AX5G0BCXi8Ujbm1IW9BzcKL0iayDwR+Ai22c1Ufc/bQ+hRKkeTYkS/X7g2Lj+KrC77S2B7YH/k6S4th7wS9sbA/8B9uikroOAn0d9w4DHJG0MfAf4sO3NKdGva7wH+IDtw9uR1QZsBexMiUS9CHA0cLHtIbYvri8g6cAweONnvzyzE1WTJGmWNF4LL4/aHhvHFwDbUAzaI7YfjPRzge06SK/xB+A3ts9rsu6r4u9U4C7bL9p+BngtAnEK+KGkKcBfgdWAVaLMI7bvieMJFIPSEXcA35b0v8Catl8BPgxcavtZANv/ruS/1PbsBrIusf2W7YeAh4ENO2uo7TNtD7M9bMCSgzrLniRJk6TxWnhxJ+ddYSzw0Yp31Bmvxd+3Kse180WBfYCVgKHhMT0FLFFXFmA2c37u8SZzvs+1vNj+LbAr8ArwJ0kf7kS3lzq41p19liTJfJDGa+FlDUlbx/FngduA6UCbpHUj/fPALR2k1zgaeB74ZS0h5qxWm0fdBgFP235D0vbAmk2UmQEMjeM9K3qsDTxs+xSKh7gZcCOwl6QVIs/yTeq1l6RFJK0DrE3plxeBpZssnyRJN5HGa+FlOvBVSfcDywGn2X4V2B+4VNJUiid0eqP0OnmHAe+W9BNJiwDrAv9m3rgQGBZ17Qs80ESZn1Lm5yYB1aX7nwamSboH2AQ4z/a9wA+AWyRNBn7WpF7/BMYB1wIHRb/cBGyUCzaSpHeRnSMfSfciaRPgCw0WPbQkkkYDV9u+bF5lLD54PQ/e7+Ru0ylJ+hPzs8OGpAm2h3WlTG4PlXQ7tqcBC4zh6i42XW0Q43t5C50kWVBJ45UkTWB7VF/rkCTJHHLOK0mSJGk50nglSZIkLUcaryRJkqTlSOOVJEmStBxpvJIkSZKWI41XkiRJ0nKk8UqSJElajjReSZIkScuRxitJkiRpOXKHjSTpJaY+PpO2I6/pazWShZT52XuwP5KeV5IkSdJyLDTGS1KbpGndJGuGpBU7zzlv+TuQM1rSnu2k7yJpkqTJku6T9OV5lP/tuvPb51XXBQFJq0rqcBd5SctK+kpv6ZQkSWGhMV4LKpIWA84EPmF7c2AL4OZ5FDeX8bL9gfnTrnWRtKjtJ2y/42WhjmWBNF5J0sssbMZrUUkXSrpf0mWSlgSQtEN4LlMlnSNp8Y7Sa0h6t6RrJR3QRN3fCjnjahGJwxu8UdKUiDy8RkfpdXV/P2JMLUOZu3wOwPZrtqdHnpUkXS7p7vh8MNIHSvpN6DNF0h6STqAEk7xH0oWRb1b8laQTJU2LMiMjfYSkm6MvH4i+VUedEN7jaZLulPRwyDgn7snoSr7TJI2XdK+kYyvpMyQdK2li6LJhpB8j6YhKvmnRj0tJuia80mkV3YdLuj3Sx0laWtIoSVdJuhG4oeqtx7U/RHsfkvS9qOoEYJ3otxOb+B4kSdINLGzGawPgV7bfC7wAfEXSEsBoYKTtTSmG4OBG6RVZA4E/AhfZPquJumeGnFOBkyPtF8C5tjejRA8+pZN0AOIhuRKwv+3ngKuAf0i6SNI+KpGMAX4OnGR7OLAHcHakf7emT9Rxo+0jgVdsD7G9T53unwKGAJsDOwInShoc17YAvg5sBKwNfLCJvlgO2Br4Ruh+ErAxsKmkIZHnqAhOtxnwIUmbVco/a3tL4DTgCDrmo8ATtje3vQnwZ0nvAi4GDgtvdUfglci/JbCn7Q+1I2srSj9uBuwlaRhwJPD36Ldv1heQdGAY4fGzX57ZiapJkjTLwma8HrU9No4vALahGLRHbD8Y6ecC23WQXuMPwG9sn9dk3RdV/m4dx1sDv43j80OfjtKhGJ5Btg9yhMG2/SVgB0qI+iOAcyLvjsCpku6hGIllJA2M9F/WBNp+vhPdt6EY6dm2nwJuAYbHtXG2H7P9FnAP0NaJLIA/hu5TgadsT43y91bKf1rSRGASxbBtVCl/Rfyd0ER9U4GPSPqxpG1tz6Tc2ydt3w1g+wXbb0b+623/u4Gs620/Z/uV0GGbBvnexvaZtofZHjZgyUGdZU+SpEm6ZLwkLSJpmZ5SphdwJ+ddYSzw0c6GyRrUNT/13g0MlbT8XMKLATgJ+AjFO4Byf98fXsEQ26vZnjUfdbfHa5Xj2TT384tambfqyr9FGdpdi2KEdwjP8BpgiXbKV+t7k7m/z0sAxMvHlhQjdrykozvR7aUOrnXn9ydJkvmgU+Ml6beSlpG0FDANuE/SO4ZHWoQ1JNW8ns8CtwHTgbbaPBTweYpn0Si9xtHA81Q8mJifWq1B3SMrf++I49uBz8TxPsCYTtIB/kyZZ7km5mkGShpRuT4E+Ecc/wU4pKLfkDi8HvhqJX25OHxDZQFIPWOAkZIGSFqJ4oGOa9DOmswfSdq9ozwdsAzFiMyUtArwsSbKzKAYKSRtCawVx6sCL9u+ADgx8kwHBksaHnmWltSM0f2IpOUlvRvYjfIC8yKwdPNNS5KkO2jG89rI9guUf9ZrKQ+Fz/ekUj3IdOCrku6nzLucZvtVYH/gUklTKW//pzdKr5N3GGWRw09inmldoNGQ03KSpkSZb0TaIcD+kf75uNZROgC2LwXOogwFirIYZHoMDx4LjIqshwLDVBZl3AccFOnHhz7TJE0Gto/0M4EpigUbFa4EpgCTgRuBb9n+V4N21tgU6CxPu9ieTBkufIAyfDq24xIAXA4sL+le4GtAbbh3U2Bc9M33gONtv055ifhFtP965vbsGjEu6pkCXG57fMw5jo2+zAUbSdJLKKZNGmcoD4MhlIfIqbZvkTQ5JrqTQNImwBdsH97XuvQHJF1n+7/7Wo/uQtIoYJjtr82rjMUHr+fB+53cbTolSVfozztsSJoQC7SappmhkjMoQzKTgVslrUlZqZdUsD0NSMMVLEiGq7vYdLVBjO/HD5AkaSU69bzaLVR+wPlm5zmTJKkxbNgwjx8/vq/VSJJ+x7x4Xs0s2FhF0q8lXRvnGwH7zaOOSZIkSTLfNLNgYzRwHbBqnD9I+VFqkiRJkvQJzRivFW1fQlltRwwXzu5RrZIkSZKkA5oxXi9JWoH4Qaak9wO5z02SJEnSZzSz2vBwyu+J1pE0lrKnXmc7bSdJkiRJj9Gp8bI9UdKHKPvBCZhu+40e1yxJkiRJGtCp8Yrd1b9C2YTUwBhJtR0okiRJkqTXaWbY8DzK/m2/iPPPUnY636unlEqSJEmSjmjGeG1iuxqO4qbYJy9JkiRJ+oRmjNdESe+3fSeApPcBuU1AknSRqY/PpO3Ia/pajaQB/Xnvv+SdNGO8hgK3S/pnnK8BTI+d1h3xlpIkSZKk12jmd14fpYRB+VB81oq0XYBP9JxqvYOkEyXd25/CWUgaJekZSfdIuk/SAX2oy+19WPcISVfXpf1e0p1NlPtA5fwgSft2kL9N0rT51zhJkt6iGc/rUODXthfUea4DgeVtN7VrSC9uSnyx7a9JWhm4V9JVtp/q7ko6a4/tDzS61sV6BjTbxx3IWJYyEjBL0tq2H24nz6LACGAWJagntuvjsCVJ0uI043ndD5wl6a54gx3U00o1g6R9I8jiZEnnR1qbpBsj/QZJa0T6aEmnSLpd0sOS9oz0q4CBwARJIyWtJOlySXfH54OR7xhJ58ePtM/vJN85km6Oeg7tRN925VSx/TTwd2BNSadJGh+e4rEV2TMiIOZUSeMU0Z+bbU8n/Twr/iq81GlRz8hIn8s7knRqxL6q6fVjSROBveL8WEkTQ8aGkW8rSXdImhT3aIMG6nwK+CPwO+ZEmq7d39Ml3QVcQgm6+Y3wXLeN9h4RedeV9Ne4DxMlrVPX3gHRzrvjfn25o/5JkqRvaOZHymcDZ8cDZX9KpN2xwFm2b+ppBdtD0sbAd4AP2H5W0vJx6RfAubbPlfQF4BRKBGiAwZTfqm1I2THkMtu7Spple0jI/S1wku3bwvBdB7w3ym8EbGP7lU7ybUiJTLw0ZW7wNGD9Bvr+vAM5tbauDawN/A04yva/JQ0AbpC0me0pkXWm7U1VhsdOpgzrdiT/7fY02e2fogQl3RxYEbhb0q1NlHvO9pbRlhOAZ21vKekrwBHAlygRk7e1/aakHYEfAnu0I2tv4DjgKUpE4x9Wrr2H0r+zJR0DzLL906h3h0q+C4ETbF+p8hvGRYCVK9e/SOnL4ZIWp0RJ/ovtR5po6zuQdCDFu2fAMivNi4gkSdqhmWFD4mG5YXyepQSmPFzSl21/psPCPcOHgUttPwtg+9+RvjXlIQvFo/hJpczvbb8F3CdplQZydwQ2klQ7X0bSwDi+qvKg7yjfNbZfA16T9DSwSgf6diRnpKRtgNeAL4fROigehotSjPFGlJD0ABdV/p7UxfY0wzbARTH095SkW4DhdB6Y9OK68yvi7wTm3KtBwLmS1qP8EH6xeiFxz9YDbrNtSW9I2iSCgELp3w6HJSUtDaxm+0qA2g/tK/0DsBOwmcI7D93WA+bJeNk+EzgTSiTleZGRJMk7aWaHjZMob/E3Aj+0PS4u/VjS9J5Urpt5rXKsBnkWAd5fv3tIPNxeajJftZ7ZdNzHHcm5uBpyXtJaFE9luO3nJY0GlqgUczvHzbZnfniTuYefl6i7Xl9PrX+qffN94Cbbu0tqA25up55PA8sBj4T+y1A8saMa1DOvCDjE9nXdJC9Jkh6gmTmvKcAQ21+uGK4aW/WATs1wI2UOZQWAyjDc7cyZC9kHGNNFuX8BDqmdSBoyn/k607crcpahPKBnhhfysbrrIyt/7+iKfEmrSbqhkzaMoXiDAyStBGwHjAP+QfHuFldZULFDBzIaMQh4PI5HNcizN/BR22222ygLNxp5/S9Shm3nwvaLwGOSdgMInZesy3YdcLCkxSLP+pKWar4pSZL0Bs0Yr8/Znuuttvags90noVFs3wv8ALhF0mTgZ3HpEGB/SVOAzwOHdVH0ocCwmKi/jzLxPz/5OtO3aTm2JwOTKPNDvwXG1mVZLtp9GPCNLsofTPGg2q06/l5JeZGZTDHG37L9L9uPUhZJTIu/kxq1oQN+AvxI0iTa8VTDG1sTeHuJfMxBzVT50Xw9fwR2ry3YqLv2eeDQ6Kvbgf+qu342cB/lx/nTgDNqOkm6p6LT2ZKGxfFBkjr8DiRJ0r3Ibn8YPiazlwRuoiw9rg21LQP82faGvaFg0jmSZgDDanNq81D+a8A/bV9Vl74CMNH2mvOvZbL44PU8eL+T+1qNpAG5w0bfIWmC7WFdKdPRfMyXga8Dq1Im12vG6wXg1HlRMOmf2H7H/ZS0KmXu6ae9rtACyqarDWJ8PiCTpFtoaLxs/xz4uaRDbP+iUb6k74k5oO6W+QRliX+SJEm/o9M5rzRcSZIkSX+jmQUbSZIkSdKvSOOVJEmStBydGi8VPifp6DhfQ1Jf/b4rSZIkSZryvH5F2XZp7zh/Efhlj2mUJEmSJJ3QzN6G74uNVCcBxNZE7+phvZIkSZKkIc14Xm/ExryGEmYDeKtHtUqSJEmSDmjGeJ1C2RpoZUk/AG5j7lAUSZIkSdKrNBPP60JJEygbrgrYzfb9Pa5ZkiRJkjSg4d6Gb2eQ3g/cGztyI2kZ4L227+oF/ZJkgSH3Nuwecg/CBY952duwmWHD04BZlfNZkZYkSZIkfUIzxkuuuGcRjbipCMx9gaS2CGXRHbJmSFqxnfSjJN0boUbuaRCWoxk9P1s5HybplPnVuaeQtJek+yXd1Ne61Ig+fCXuwX2STpeUP7xPkoWAZv7RH5Z0qKTF4nMY8HBPK9ZfkbQ1JbL0lrY3A3YEHp0HUW3A28bL9njbh3aLkj3DF4EDbG/fTGZJvfWC83fbQ4DNgI2A3Xqikl5sT5IkTdCM8ToI+AAl0u1jwPuAA3tSqW5gUUkXhqdwWS1arqQdJE2SNFXSOZIW7yi9hqR3S7pW0gGUwI3P2n4NwPazsQM7koZKukXSBEnXSRoc6etK+qukyZImSloHOAHYNryGb0gaIenqyL+8pN+HZ3enpM0i/ZjQ72ZJD0vq1NhJGi7p9qh7nKSlJS0h6TfR3kmSto+8oyRdIenPkh6S9JNIPxrYBvi1pBNVoimfKOnu0PHLkW+EpDGSrgLu6yTfzXFvHoh7pQ70bVdOFdtvUoJLrivpgMg7WdLllfs/Oryz8ZIelLRLpDfVnma+eEmS9A4dGi+V33edZPsztle2vYrtz9p+upf0m1c2AH5l+72U+GNfUQmuORoYaXtTytDnwY3SK7IGUiLzXmT7LOAvwOrx8PuVpA8BqISN/wWwp+2hwDmU6MkAFwK/tL055UXgSeBIYIztIbZPqtP/WGBSeHbfBs6rXNsQ+G9gK+B7UW+7qPyY/GLgsKh7R+AV4KuAo717A+dGPwAMAUYCmwIjJa1u+zhgPLCP7W9SvLCZtocDw4EDJK0V5beM+tbvJN8WlHhxGwFrAx/sQN+O5NTauiRlRexU4Arbw0PG/VG+Rlv03c7A6dHuZtvTZSQdGMZy/OyX+yTweJIskHQ4FGJ7tqQ1Jb3L9uu9pVQ38KjtsXF8AXAocD3wiO0HI/1cykP8pgbpJ8f5H4Cf2L4QwPYsSUOBbYHtgYslHUl5uG8CXB9OxADgSUlLA6vZvjLKvwoQeRqxDbBH5L9R0goqqzwBrgmv7zVJTwOrUDzi9tgAeNL23SHrhah7G4qhxfYDkv7BnNhdN9ieGfnuA9bkncOiOwGbSdozzgcB6wGvA+NsP9JkvseinnsoRmVmA30byXkQWCfKG/iD7WslfUjS8cCylJeP6yq6XxLztg9JepjyMtBse7qM7TOBM6GsNpxXOUmSzE0z4/gPA2Nj6OSlWqLtn/WYVvNP/UNifh4aY4GPSvptbeGK7dmUKMM3S5oK7EeJNn2v7a2rhcN4dSevVY5n0/2LZ5qRL+AQ29fNlSiNoPId6SRfV9rRSE4bc+a8qoym/B5xsqRRwIjKtfa+G822J0mSfkIzc15/B66OvEtXPv2ZNVQWVkBZFHEbMB1ok7RupH8euKWD9BpHA88TmxFL2kDSepXrQ4B/hJyVavWqLG7ZOH4f95ik3SJ98RjiepHG/TgG2Cfyj6DMsb3QUYMl3SBptbrk6cBgScMjz9IqCw+q8tcH1oi8zXIdZch1sZoMSUvNR77O9O2qnKUpXu9itXZW2EvSIirzjmtHnV2VnyRJH9PMDhvH9oYi3cx04KuSzqFMtJ9m+1VJ+wOXxgPxbuB026+1l14n7zDgHJUFDBcDv5C0LPAm8DfgQNuvx7DTKZIGUfr2ZOBeikE8Q9JxwBvAXsAUYLakyRRPYVKlvmOivinAyxTPriEqy8PXBf5dTQ+dRoa+76bMH+1IiRRwWniNbwKjoh867NQKZ1OG+SaqFHqG9lf5NZuvM327JAf4LnBX5LuLuV8S/gmMA5YBDorvRVPyI9/ptsdLOih0Pl3SsJD1pQ50SpKkG2lmh42baGfYzfaHe0qppGtI2gT4gu3D+1qX/oyk0cDVti/ri/pzh43uIXfYWPDQPOyw0cx8yRGV4yUoCwne7EolSc9iexqQhqufs+lqgxifD94k6RaaGTacUJc0VtK4HtInSXoM26P6WockSbqHTo2XpOUrp4sAQylLiZMkSZKkT2hm2HACc5YTvwk8wtw/+kySJEmSXqWZYcO1OsuTJEmSJL1JM8OGi1G2S9oukm4GzrD9Rg/qlSRJkiQNaWbY8DRgMcpvg6D8Zuk0IH/TkiRJkvQJzRiv2ganNW6MH9YmSZIkSZ/QzPZQs2MrHQAkrU3Ziy5JkiRJ+oRmPK9vAjfFDtyi7DK+f49qlSRJkiQd0MxqwxtiI9oNIml6LRBjkiRJkvQFDY1X7Oz9qO1/xaatQyhbQ/1D0jG2/92obJIk72Tq4zNpO/KavlajZcg9DJOO6GjO6wxKMD4kbUcJW38eJWDgmT2vWpIkSZK0T0fGa0DFuxoJnGn7ctvfpYTfWCiQdJykHedTxsmSHo/QJY3yDJH08cr5rhGhuSO5s+ZHrwUZSbtJsqQNO8izrKSvVM5XldThjvOSbo4QKEmS9CEdGq+IbwWwA3Bj5Vp3R+/tt9g+2vZf57V8GKzdgUeBDzXIsyglqOXbxsv2VbZPmNd6F0Yq31eAvSlBSPfuIO+ywNvGy/YTtvfsSR2TJOkeOjJeFwG3SPoDJSjgGICIODyzpxWT1CbpAUmjJT0o6UJJO0oaK+khSVtFvq0k3SFpkqTbJW0Q6UtKukTSfZKulHRX7Y1Z0mmSxku6V9KxkTZM0j3xmSrJkT46gkwiaYakYyVNjDwbRvpSks6RNC70+GSlKSMoASlPo/IglXSMpPMljQXOB44DRkb9IyWNknRq5F0l2jA5Ph9op7++KeluSVNqbeqkf7vUFknXSNosjidJOjqOj5N0gKTBkm4N/adJ2jauz5J0UvT1DZJWivQDQt/Jki5XiS5d6+/T4/48KGmXSB8g6cRKG78c6SMkjZF0FSXwKJIGAttQ9uD8TKXN9XlPANYJnU+M79y0Sn0/jbZMkXRIO324U3z3Jkq6NOpNkqQXaGi8bP8A+B9KlN9tPCdq5SLAO/6Re4h1gf8DNozPZykPpSOAb0eeB4BtbW8BHA38MNK/AjxveyNKZN2hFblHReCzzYAPSdrM9njbQ2wPAf4M/LSBTs/a3pJijGqxzo4CbrS9FbA9cKLmhJHfm/IicCWwsyLUfLARsKPtvUP3i0OHi+vqPAW4JX4sviXFGL6NpJ2A9YCtKB7cUJV5ys7oSlvGANuqRIl+E/hg5N8WuJVyb66L/tscuCeuLwWMt70xcAvwvUi/wnbtB/D3M/dmz23Rlp2B0yUtEddn2h4ODAcOkFTbd3NL4DDb68f5J4E/234QeE5S9d5X8x4J/D36/Jt1fXNg6DHE9mbAhdWLklYEvkO5f1sC42knppqkA8MQj5/9co+/8yXJQkOHw3+272wn7cGeU+cdPGJ7KoCke4EbbFslfH1b5BkEnKuynN+UraygGLmfh87TJE2pyP20pAMp7R9MMSJTop6RlAfcTg10uiL+TgA+Fcc7AbtKqhmAJYA1JP2dMhR4uO0XJd0F/DdwdeS7yvYrTfTDh4F9oy2zeafnu1N8JsX5QIoxu7UTuU23hWK8DqVEFbgG+Eh4S2vZni5pFeCcMM6/t31PlH8LqBnjCyp1biLpeMrQ3UDguopel9h+C3hI5feFG4Zem9W8YMp9X4+yqGic7Ucq5fcm7j3wuzivxaWrz9uIHYHTbb8J0M7q2vdTvjdjJQG8C7ijXojtM4kFTosPXq/jsOVJkjRNf5+7qv6e7K3K+VvM0f37wE22d5fURtk4uCHxtn4EZdur51VCwy8R1zYBjgG2CyPRkU6zKzoI2MP29Lq6PkF5OE+NB9ySlCHYmvF6qSNdu4CAH9k+o4vlutKWdwHDgIeB64EVgQMIo2D71vD2dgZGS/qZ7fPaqbP2AB8N7GZ7sqRRlOHV+jzVcwGH2K4aOSSNoNKPKvHnPgxsqjL0OwCwpJpn1Z19fn14zUmS9DLNbA/V3xkEPB7HoyrpY4FPA0jaCNg00pehPMBmhrfwscizLGV4b1/bz3RRh+uAQxQWStIWkb438CXbbbbbgLWY47HU8yKwdAP5N1B29q/NxdQHA70O+EJtzkXSapJWjuMbJK02v22x/Tpl0cleFA9jDOUl4NbItybwlO2zgLMp3iuU71jNW/osZREF0dYnw1Pbp06HvSQtorIt2drA9NDr4Nqwq6T1K0OzVfYEzre9ZvT76hRvcdt28nbU59cDX1YsAtHcQVkB7gQ+qDIHXJsrXJ8kSXqFBcF4/QT4kaRJzO1J/gpYSdJ9wPGUeaKZtidThtceAH5LMXJQ5knWBM6KCfx7uqDD9ynDlVNiePP7YaA+ShliA8D2S5SH9yfakXETsFHUPbLu2mHA9jFcOoEyXPU2tv8Sbbkj8lwGLK2y0nFdoCs/KH9HWyrXxgBPx1DnGOA98ReK5zQ57sNI5gzbvQRsFQshPkxZmAJlHvIuSv8/UKfDP4FxwLXAQbZfpRjE+4CJIesM2h852Jsyv1jlctpZdWj7Ocqw3zRJJ9ZdPjv0mKKyEfVn68o+Q3lZuiiGpO+gDG8mSdILaM46jAULSQOAxWy/Gm/wfwU2CA9ioSCGQb9g+x0LCXpRh1m2m16FF8O4V9vu8PdWrciwYcM8fvz4vlYjSfodkibEIrqm6e9zXvPDkpQNhRejzE98ZWEyXFAWqtDOCrgkSZJWZ4E1XrZfpCwwSPqQrnhdkX9UD6mSJMkCxIIw55UkSZIsZKTxSpIkSVqONF5JkiRJy5HGK0mSJGk50nglSZIkLUcaryRJkqTlSOOVJEmStBxpvJIkSZKWY4H9kXKS9DemPj6TtiOv6TxjP2XGCTv3tQpJ8jbpeSVJkiQtRxqvJEmSpOXoFeMl6ZhaZF5Jx0nasYO8QyR9vHK+q6Qje0PPBvrMroVIiU+HukgaIekDvaVfTyFpRoS67yn5bRHaBEnDJJ3SZLmTJT0e4V4a5enyd0jSrGZ1T5Kk7+n1OS/bR3eSZQhlQ90/Rf6rgKt6Sp8IuqgIO98er9ge0gWRI4BZwO3t1LVoLax8Mgfb44FOY4WEwdqdEhTzQ5QYaPV5FqWXv0NJkvQ+PeZ5STpK0oOSbgM2qKSPlrRnHA+XdLukyZLGRYTg44CRtaCMkkZJOjXyt0m6UdKUiBC8RkXmKSHr4Yr8gZFvoqSpkj5ZkTNd0nnANOC7kk6u6HiApJM6ad8MScdWZG8oqQ04CPhG6L9t6Ha6pLuAn0h6SNJKIWMRSX+rnTeoZ4CkEyXdHe3+cqSPkHSzpMskPSDpwjDE7fXr0pJulTSkIvc2SZtHH/0m2jBF0h7t6HB4BGycJunrlT6s1Xt/6LFkXBsq6RZJEyRdJ2lwJX2ySnDHr1bkj5B0dRxvJekOSZOiDRtUVBlBCSp6GpXgkiqe/fmSxgLn0/F3aBVJV9b0UDtesqRvVvr72Eb3JkmSvqNHjJekocBnKG/AHweGt5PnXcDFwGG2Nwd2pETdPRq42PYQ2xfXFfsFcK7tzYALgepQ02BgG2AX4IRIexXY3faWwPbA/9Ue8MB6wK9sbwz8H/AJRYh5YH/gnDh+t+YeNqxGOX42ZJ8GHGF7BnA6cFLoX4sy/B7gAxEU8gLmhL3fEZgcUXkb8UVKBOjhlH48QNJacW0L4OuUyMprU8LSt9evrwC/pkT+RSVc/RIRVfq7IX/T6Ncbq5XHvdwfeB/w/qh/i7i8QfThe4EXgK9EH/4C2NP20OjHH0T+3wCHhF6NeADY1vYWlO/CDyvX9gYuokRK3rlyv4g+2NH23nT8HToFuCV02JJiDKvt3Yny3diK8v0dKmm7DvTtEEkHShovafzsl2fOq5gkSeroKc9rW+BK2y/bfoH2h2w2AJ60fTeA7ReaGFLbmhLuHsob9jaVa7+3/Zbt+4BVIk3AD1XCtP8VWK1y7R+274y6Z1Ee2rtI2pASgXlq5HslHoJD2nkYXhF/JwBtHeh9qe3ZcXwOsG8cf4HyQO+InYB9Jd0D3AWsQHm4Aoyz/VgMed4TOjTq10ujfYtFvaNDxo7AL2uV2X6+rv5tKPfypeinKyj3F+BR22Pj+ILIuwGwCXB96Pwd4D2SlgWWtX1r5D+/QXsHAZeqzIedBGwMb7/sfJxyn1+IvvjvSrmrbL/SQGaVD1NeNrA923a9RdkpPpOAicCGzOnvLmP7TNvDbA8bsOSgeRWTJEkdC9LvvF6rHNe8q32AlYChtt+QNANYIq69VFf+bODblDf/zgxKfZ2z6bgv367L9qOSnpL0Ycrb/T6NiwGlLYfYvm6uRGkEc7e5Qx1svyzpeuCTwKeBoZ3U2wxu51zAvba3rtN32SZlfh+4yfbuMQx7c6T/N7AsMDWc5yUpHuXVcb3+fs4rAn5k+4xukpckSQ/QU57XrcBukt4taWngE+3kmQ4MljQcIOZlFgVeBJZuIPd2ynAklIf+mAb5agwCng7DtT2wZqOMtu8CVgc+Sxmamlc60r/G2RRP5W2PTNLukn7UTt7rgINrQ2SS1pe0VAeyG/Vrrd5TgLsrHtb1zD3/tFydvDGUe7lk1Ls7c/p9DUk1I/VZ4Laof6VauqTFJG1s+z/AfyTVvOVGRnsQ8Hgcj6qk7w18yXab7TZgLeAjtXm2Ojq6BzcAB4duA1TmWatcB3xB0sDIs5qklRvISpKkj+gR42V7ImXeZTJwLXB3O3leB0YCv4gJ/OspXtFNwEbtzC8BHALsH8OAnwcO60SVC4FhkqZShuoe6CT/JcDYuqGz+jmvExoVDv4I7B55t22Q5ypgIHN7eOtQ5o3qORu4D5gYQ2ln0LGH1ahfsT0h6qjWezywXCzGmEyZG6zKm0gZYhxHGao72/akuDwd+Kqk+4HlgNOi/j2BH4e8e4Daooj9gV/GcGLNO67nJ8CPJE2qtTMM1EeBt7ensP0SxVi292LU0XfoMGD7+E5MoMyVVdv7F8rQ9B2R5zLCEEr6k6RV4/g4SbvG8a6SjmvQniRJegDZ9SM/Cy+x4u0k2zf0cD3Dop5tK2kXAN/oZPHG/Na7KmUYbsMOfhrQrKw24Grbm3SDagsFw4YN8/jxnf4iIEkWOiRNsD2sK2Vyhw3KfIykBymLM3racB0JXA78v2q67c/1sOHal+I5HTW/hitJkqSvSc8rSXqJ9LySpH3S80qSJEkWCtJ4JUmSJC1HGq8kSZKk5UjjlSRJkrQcabySJEmSliONV5IkSdJypPFKkiRJWo40XkmSJEnLsSDtKp8k/Zqpj8+k7chrOs9Yx4wTdu4BbZKktUnPK0mSJGk50nglSZIkLUdLGi9JI2IH+O6St5ukjTrPOc/yR0vaM47P7qwuSTMkTY2QHlMlfbKH9DpG0hHdKO/rDeJr1ec7TtKOneQZIekDHeXpom67xqbISZIsALSk8eoBdqMurlNnVAI8dgnbX7J9XxNZt7c9hBIb65R26pek/nb/vk6JcNwhto+2/ddOso1gThywpujonti+ynZnsdiSJGkR+uThJ6lN0gOSLpR0v6TLam/sknaQNCk8jnMkLR7pH40yE4FPVWQNlPSbyD9F0h6Rfpqk8ZLulXRsJf8Jku6LvD+Nt/tdgRPD01mnA72PkXS+pLHA+dGOMZImxucDkU+STpU0XdJfgZUrMm6OeF5I2jv0nibpxw2qXQZ4vtJv0yWdB0wDVpd0YpSfWgu8GH1yQ+g0l+cm6ShJD0q6Ddigkr6upL9Kmhzl1mkkR9JSkq6JvNMkjZR0KLAqcJOkmzq5/1VPdIakYyt1bKgSK+wg4BtxT7aVtJKkyyXdHZ8PNrgnd0rauL6/JY2SdGqkNZI1VSU8jiQ9pxJGBknnSfpIR21KkqR36cvVhhsAX7Q9VtI5wFfi4TIa2MH2g/GQPljS6cBZwIeBv1GiNNf4LjDT9qYwVxj7o2z/W9IA4AZJm1HCy+9OCcZoScva/o+kqyiBFS9rQu+NgG1sv6JicD9i+1VJ6wEXAcOijg0i7yqUSMjnVIWoBIb8MTCUYpz+Imk327+PLDdJErA28OlK0fWA/WzfGYZ6CLA5sCJwt6RbgWeA3W2/IGlF4M5o45bAZ6LMosBESjRhKFGnT7B9paQlKC82rzeQ81HgCds7R1sG2Z4p6XCKx/hsE/1Y5VnbW0r6CnCE7S/FPZ9l+6dRx28pATxvk7QGcB3w3ihfvSffiP76nqTBwGDb4yVVg2b+vIGsscAHgX8ADwPbAucBWwMHd7FNhN4HAgcCDFhmpXkRkSRJO/TlsNOjtsfG8QXANpQH/iO2H4z0c4HtgA0j/SGXAGQXVOTsCPyydmL7+Tj8tIqXNgnYmPKAmwm8Cvxa0qeAl+dB76tsvxLHiwFnqYSLv5Q5Q4/bARfZnm37CeDGduQMB262/YztNynGY7vK9e0jSvGmwKmSBkb6P2zfGcfbVOp5Crgl5Ar4oaQpwF+B1ShGdFvgStsv234BuApA0tLAaravBLD9qu2XO5AzFfiIpB9L2tb2zHnoxypXxN8JQFuDPDtGP9wTei9T6ZPqPbmEMtQKxYi190LSSNYYyj3YDjgN2FTSasDztl+al4bZPtP2MNvDBiw5aF5EJEnSDn3pedVHwey2qJiS1gKOAIbbfl7SaGAJ229K2grYgfKA+xrFm+sK1YfYN4CnKJ7PIhTD2K3Y/rukpyiG8em6+huxD7ASMNT2G5JmAEvMQ/XtygmveEvg48Dxkm6wfdw8yK/xWvydTePv5CLA+23P1cfFOZ3TJ7YfjyG/zYCRlOHHZmXdCnwVWAM4iuJB70kxakmS9CP60vNaQ9LWcfxZ4DZgOtAmad1I/zzFm3gg0mvzUXtX5FxPeeAAbw8bLkN5oM2UtArwsbg2EBhk+08Uw7N5FHsRWLoi42uSvtZEGwYBT9p+K3QdEOm3AiMlDYihq+3bKTsO+JCkFWNoc+9o61xIWhlYizKUVc+YSj0rUTyGcaHX02FwtgfWrOi1m6R3h7f1CQDbLwKPSdot6lw8hkTblRNDni/bvgA4kTIcCe/sx/PiZWFemEsW8BfgkIrsIR2UvRj4FuVeT2nneruybD9KGX5dz/bDlO/kEZR+S5KkH9GXxms68FVJ9wPLAafFm/D+wKUxFPcWcHqkHwhcE0OBT1fkHA8sp7JwYDJluG0yZbjwAeC3lLkMKA/Dq2MY7Dbg8Ej/HfBNlYUi61CGKZ9rog2/AvaLejdkjgdwJfAQZa7rPOCO+oK2nwSOBG4CJgMTbP+hkuWmGNa6CTgyhgXruRKYEuVvBL5l+1+UIchh0Yf7Rj9geyLlwT4ZuBa4uyLr88Ch0Te3A//VSA5lKHNc6Pc9yj0AOBP4s+Ys2NgMeKK9jmuCPwK7KxZsAIeGLlMk3Uf7HlWNyyhze5c0uN6RrLuA2rD1GMpQ6W0A0V7i+GzNWXhzkKSO9EmSpJtRmULq5UrLarKrY06n36HyG7JP2X69r3VpVSQtA/za9l59rUt/YdiwYR4/fnxfq5Ek/Q5JE2wP60qZ3NuwHWzv0tc6tDqxICQNV5IkPUKfGC/bM4B+6XUlSZIk/Z/+tkNDkiRJknRKGq8kSZKk5UjjlSRJkrQcabySJEmSliONV5IkSdJypPFKkiRJWo40XkmSJEnLkcYrSZIkaTnSeCVJkiQtRxqvJEmSpOVI45UkSZK0HGm8kmQ+kTRK0ql9rUeSLEyk8UqSJElajjReSZ8hqU3StMr5EZKOkXSzpB9LGifpwQhGWfNwfi/pekkzIuL14RFE9E5Jy0e+AyTdLWmypMsjKjSSRks6XdL4kLtLRe4fot6HJH2votPnQo97JJ0RUa+RtH/IGAd8sBe7LUkS0ngl/ZdFbW8FfJ0SrbnGJsCngOHAD4CXbW9BiVa9b+S5wvZw25sD9wNfrJRvA7YCdgZOl7REpG8F7EGJ/ryXpGGS3guMBD5oewgwG9hH0mDgWIrR2gbYqFEjJB0YxnL8M888M08dkSTJO8lglEl/5Yr4O4FicGrcZPtF4EVJM4E/RvpUiuEB2ETS8cCywEDgukr5S2y/BTwk6WFgw0i/3vZzAJKuoBilN4GhwN2SAN4NPA28D7jZ9jOR/2Jg/fYaYftM4EwokZS71gVJkjQijVfSl7zJ3N7/EpXj1+LvbOb+nr5WOX6rcv5WJd9oYDfbkyWNAkZUytQbEHeQLuBc2/+vekHSbu9oSZIkvUoOGyZ9yVPAypJWkLQ4sEs3yV0aeFLSYsA+ddf2krSIpHWAtYHpkf4RSctLejewGzAWuAHYU9LKAHF9TeAu4EOh92LAXt2kd5IkTZKeV9Jn2H5D0nHAOOBx4IFuEv1dioF5Jv4uXbn2z6hvGeAg26/GkOA44HLgPcAFtscDSPoO8BdJiwBvAF+1faekYyjzbP8B7ukmvZMkaRLZOQyfLBxIGg1cbfuyuvRRwDDbX+vJ+ocNG+bx48f3ZBVJ0pJImmB7WFfK5LBhkiRJ0nLksGGy0GB7VIP00ZRFHkmStAjpeSVJkiQtRxqvJEmSpOVI45UkSZK0HLnaMEl6CUkvMud3Zf2VFYFn+1qJJmgFPVPH5lnT9kpdKZALNpKk95je1eXAvY2k8f1dR2gNPVPHniWHDZMkSZKWI41XkiRJ0nKk8UqS3uPMvlagCVpBR2gNPVPHHiQXbCRJkiQtR3peSZIkScuRxitJkiRpOdJ4JUk3I+mjkqZL+pukI9u5vriki+P6XZLa+qGOh0u6T9IUSTdEHLN+pWMl3x6SLKlPlnw3o6ekT0d/3ivpt/1NR0lrSLpJ0qS45x/vbR27jO385Cc/3fQBBgB/pwS6fBcwGdioLs9XgNPj+DPAxf1Qx+2BJeP44P6oY+RbGrgVuJMS1qY/3u/1gEnAcnG+cj/U8Uzg4DjeCJjR233Z1U96XknSvWwF/M32w7ZfB34HfLIuzyeBc+P4MmAHRUTM/qKj7Ztsvxynd1KCdPYmzfQjwPeBHwOv9qZyFZrR8wDgl7afB7D9dD/U0ZQArQCDgCd6Ub95Io1XknQvqwGPVs4fi7R289h+E5gJrNAr2tXVH7SnY5UvAtf2qEbvpFMdJW0JrG77mt5UrI5m+nJ9YH1JYyXdKemjvaZdoRkdjwE+J+kx4E/AIb2j2ryT20MlSdIQSZ8DhgEf6mtdqkhaBPgZMKqPVWmGRSlDhyMoHuytkja1/Z++VKqOvYHRtv9P0tbA+ZI2sf1WXyvWiPS8kqR7eRxYvXL+nkhrN4+kRSnDNM/1inZ19Qft6YikHYGjgF1tv9ZLutXoTMelgU2AmyXNAN4PXNUHizaa6cvHgKtsv2H7EeBBijHrLZrR8YvAJQC27wCWoGza229J45Uk3cvdwHqS1pL0LsqCjKvq8lwF7BfHewI3OmbK+4uOkrYAzqAYrt6eo+lUR9szba9ou812G2Veblfb4/uTnsHvKV4XklakDCM+3M90/CewQ+j4XorxeqYXdewyabySpBuJOayvAdcB9wOX2L5X0nGSdo1svwZWkPQ34HCg4TLwPtTxRGAgcKmkeyTVP+z6g459TpN6Xgc8J+k+4Cbgm7Z7zdNuUsf/AQ6QNBm4CBjVyy9UXSa3h0qSJElajvS8kiRJkpYjjVeSJEnScqTxSpIkSVqONF5JkiRJy5HGK0mSJGk50nglyUKOpP+S9DtJf5c0QdKfJK3fi/WPkrRqb9WXLBik8UqShZjYEPhK4Gbb69geCvw/YJUmyy/a0XmTjAL6vfGax7Z1R72K7bCSCtkhSbJwsz3whu3Tawm2J9seEw/NEyVNkzRV0kgASSMkjYkfLt/XzvmAKHd3xIb6ck22pP8NWZMlnSBpT8reiRfGj6HfXVVO0gEhZ7KkyyUtGemjJZ0i6XZJD4ccJA2WdGvImiZpW0l7SfpZXD9M0sNxvLaksXE8VNIt4XleJ2lwpN8s6WRJ44HDQta00OfW+s6UNFAl/tnEaOcnI/0ESV+t5DtG0hFx/M1KXx0baW0q8bfOA6YBq0s6TdJ4lZhgx1ZkfVzSA6H7KZKujvSlJJ0jaZxKnK72duVvXfo6Jkt+8pOfvvsAhwInNbi2B3A9JR7UKpQthAZTtjp6CVgr8tWfHwh8J44XB8YDawEfA25nTpyw5ePvzTSIxQWsUDk+HjgkjkcDl1JewDeihPyAslPEUXE8gLIH4n8Bd0faZZTtklajbNH1I2Cx0GulyDMSOKei268qOkwFVovjZdvRd1FgmTheEfgbIGAL4JZKvvso+w3uRImlpWjL1cB2QBvwFvD+SpnlK+26GdiMso3To5W+vwi4Oo5/CHyupitlT8Wl+vo7112f3FU+SZJGbANcZHs28JSkW4DhwAvAOJdNZmtUz3cCNqt5Q5SNh9cDdgR+44gTZvvfTeiwiaTjKQ/fgZQtjmr83mXX8/sk1YY57wbOkbRYXL8HeDE8oqUpBuO3FAOxLXAFsAFlk9/ryygqA4AnK/VcXDkeC4yWdEmUrUfADyVtRzE+qwGr2J4kaeWY21sJeN72o5IOi/6aFOUHRl/9E/iH7Tsrsj8t6UCKgRxMMdqLAA9X+v4iyssDIXfXmodHMXRrULaIannSeCXJws29lM2Bu8pLHZyL4iFVDQ2S/nse6hkN7GZ7sqRRxAa3QXWnewHYvjUMx84UI/Mz2+dRPKv9genAGOALwNYUT20N4F7bWzfQ4e222T5I0vtC/gRJQz33PoX7UIzTUNtvqOx4v0Rcu5TS1//FHIMo4Ee2z6hWKKmtWq+ktYAjgOG2n5c0uiK3EQL2sD29k3wtSc55JcnCzY3A4vFGD4CkzSRtS3nIj4w5rJUo3sq4JmReBxwc3g+S1pe0FGUIcv/KvNXykf9FyvBeeywNPBmy9umsYklrAk/ZPgs4G9gyLo2hPPxvpXg52wOv2Z5JMWgrqcSxQtJikjZuIH8d23fZPpqy6/rqdVkGAU+H4doeWLNy7WLKju57UgwZlL76gqSBIX81SSu3U/UyFGM2M7zMj0X6dGDtMHZQhjxrXAcconAnVSIFLDCk55UkCzG2LWl34GRJ/wu8CswAvg7cRvFOJlPCxH/L9r8kbdiJ2LMpczYT48H5DMV7+rOkIcB4Sa9TIvZ+m+JdnS7pFWBr269UZH0XuCtk3EVjI1djBPBNSW8As4B9I30MxdDcanu2pEeBB6IPXo8hzlMkDaI8F0+meKX1nChpPYpXc0P0TZULgT9KmkqZ63ugdsFlJ/elgcdtPxlpf1EJQXJH2JhZwOeA2VWh4XlOCnmPUoYvsf2KpK8Af5b0EmXYtMb3ox1TVFYrPgLs0nH3tQ65q3ySJEkLI2mg7VnxovBL4CHbJ/W1Xj1NDhsmSZK0NgdIuofiKQ6iBBFd4EnPK0mSJGk50vNKkiRJWo40XkmSJEnLkcYrSZIkaTnSeCVJkiQtRxqvJEmSpOX4/5z/TX3x5LsNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data = plot_data.sort_values(by=\"comp_source_type\")\n",
    "ax = plot_data.plot.barh(legend=False)\n",
    "\n",
    "ax.set_title(\"Source type identification by Citoid\")\n",
    "ax.set_ylabel(\"Source types\")\n",
    "ax.set_xlabel(\"Correct answers average\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/citoid_sourcetype_eval.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Citoid's answers for the most used source types present high values: 90% correct identification of webpages and forumPost and book. The performance descends to less than 50% for other source types.\n",
    "\n",
    "Two important observations must be noted here regarding the source type evaluation. On the one hand, we might be overestimating Citoid's success when identifying source types because modifying the citation template type when doing manual editions is not a simple change (it involves changing the whole template), hence Wikipedia editors usually leave them unmodified if the type is not correct. On the other hand, we might be underestimating Citoid's performance for source types because of our source type mapping (see section 4. Data normalization). Specific citation templates are not included in our mapping, leaving empty values in our manual data. In those cases, Citoid's response for the source type is interpreted as a missmatch with the manual data and penalized as incorrect.\n",
    "\n",
    "Now let's observe Citoid's performance for all the fields by source type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 408,
   "metadata": {},
   "outputs": [],
   "source": [
    "success_by_source = scores.groupby(\"source_type_manual\")[\"citoid_success\"].mean().sort_values().reset_index()\n",
    "plot_data = success_by_source.set_index(\"source_type_manual\").loc[top_sources.index]\n",
    "plot_data = plot_data.rename(index=source_type_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 409,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAawAAAEYCAYAAAAAk8LPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABMCUlEQVR4nO2deZhcRdX/P18CsgUCCPIGFIY9sgZIUCBgQEAURFAwIiJBBUFBhB8qrysgKoqvICKbiGETkU0RUAx7WEMWkrCjEERENiEQdpLv7486bW6G6Z6eySzp5Hyep5+5t+rUqVN1e+65tfQ9sk2SJEmSzO8s0t8GJEmSJEkzpMNKkiRJWoJ0WEmSJElLkA4rSZIkaQnSYSVJkiQtQTqsJEmSpCVIh5UkfYikb0o6q5tlV5M0U9KAOvlHSzp/3ixsDSQdLOmp6I939rc9Sd+QDitJehhJn5Y0IW6mT0r6s6QRALZ/aPsLIdcmyZIWbUav7X/YHmh7Vm/aP78jaTHgZ8BO0R/P9YDO6ZJ2mHfrWh9JYyQd1992dEQ6rCTpQSQdAZwE/BBYGVgNOBX4WD+ataCxMrAEcG9XC6qwwN33mn3oaXls5yc/+emBDzAImAns1UDmaOD8OP4H4CgzE9iS8hD5beAx4GngXGBQyLeF/KJxvgZwE/ASMBY4paa7g3pXBK4EXgD+A4wDFok8A2tXZMcAx1XOPwbcDbwI/B3YOdJXAH4D/At4HvhDpcyuUeYF4DZg40reN4Anwu4HgQ9G+hbAhKjnKeBnHbRjXeDlSr9dH+lbAXcBM+LvVpUyNwI/AG4FXq22NfLPA2ZH3kzg65G+G8UpvhA63lspMx34X+C+aPtvgCXq9P3acZ1mAM8CF1XyGtk9Hdihznen9l34POV7dHOkHwDcH317H7BZpK8CXAo8AzwKfKWOrQcCbwJvRF/8CfgacGk7uZOBn1f690fA+Lh2fwRWqMi+P74DLwBTgJGVvNHAI2Hvo8A+Df/H+vufPD/5WVA+wM7AW4RDqSPT0U1n0Ur+54C/AWsCA4HLgPM6kgdup0yNLQ5sG//09RzWj4DTgcXisw2gyKvrsChOZAawI8WZrgoMibyrgIuA5UPnByJ9U4qzfR8wANgvbr6LA+sBjwOrVNq0VqU9+8bxQOD9ddrSvh9WoDiNfYFFgb3j/J2RfyPlpr5B5C/Wgc7pzO0cao5xx2jb1+O6vKMifw/wnqj/VipOvp3uC4FvRf8tAYxo0u72Nh3N27875wJLA0sCe1EeBIYDojjK1aPeicB3gXdQvluPAB+qY+9/r3+cD46+WC7OF43ru3mlf58ANgxbLq3YuSrwHPCRsGPHOF8pZF8E1qvUs0Gj/7EFbmicJP3IO4Fnbb81Dzr2oYwsHrE9k/IU/6n2Uz6SVqPcmL5j+3XbN1OehuvxJuWGsLrtN22Pc9wlOuHzwNm2x9qebfsJ2w9IGgx8GDjI9vOh86YocyBwhu07bc+yfQ7wOuVJexbFca0vaTHb023/vWLj2pJWtD3T9h1N2AewC/Cw7fNsv2X7QuAB4KMVmTG27438N5vQOQq4Ktr9JvBTilPYqiJziu3Hbf+HMoLbu46uNymOYxXbr9m+pQt2d8bRtl+2/SrwBeAntu9y4W+2H6N8T1ayfaztN2w/AvwK+FQzFdh+EriZ4hChPJg9a3tiRew82/fYfhn4DvDJ2Bz0GeBq21fH92csZRT9kSg3G9hQ0pK2n7TdcJo3HVaS9BzPASvO43rCKpTpwBqPUZ5oV+5A7vm4QVRl63ECZYTwV0mPSDqqSXveQ5kG7Cj9P7af7yBvdeD/SXqh9gn5VWz/DfgqZbTwtKTfSVolyn2eMrJ5QNJdknZt0sb2fUacr1o5f7xJXR3qtD07dNTT+ViU6YivU0Y84yXdK+lzXbC7M6o21LtWqwOrtLse3+Tt36lGnENxPsTf8xrY8RhlVLpi1L1Xu7pHAIPjuzsKOAh4UtJVkoY0MiIdVpL0HLdTRhK7Nynf0QjnX5R/8hqrUaYZn2on9ySwvKSl28l2XJH9ku3/Z3tNytrMEZI+GNmvAEtVxP+ncvw4sFYHKh8HVpC0XJ28H9hervJZKkYQ2P6t7RHRTgM/jvSHbe8NvCvSLmnXvnq07zMoffFE5byz0WT7/Ll0ShLFIVR1vqddff/qULH9b9sH2F4F+CJwqqS1m7D7Zepfl47sbnStHm13PZax/ZEOZNvrrPEHYGNJG1LWJy9ol9++L96krNc9Thl9Vete2vbxALavsb0jZfT/AGXkV5d0WEnSQ9ieQVkn+KWk3SUtJWkxSR+W9JMOijxDmRJZs5J2IXC4pDUkDaTsNryo/TRjTPVMAI6R9I7YNl93KknSrpLWjhvvDMrU3OzIvhv4tKQBknYGPlAp+mtgf0kflLSIpFUlDYlpoj9Tbr7LRzu3jTK/Ag6S9L7Ylbe0pF0kLSNpPUnbS1oceI2y0WF22PgZSSvFaOaF0DWbzrkaWDd+TrCopFHA+pRNJs3yFHNfh98Du0S7FwP+H+Vh5LaKzJclvVvSCpQ1qos6UixpL0nvjtPnKQ5hdhN2302ZDl5M0jBgz07acBZwpKTNo9/XlrQ6ZTPES5K+IWnJuM4bShreZF9g+zXgEuC3wHjb/2hX5jOS1pe0FHAscInLzy/OBz4q6UNR7xKSRka/rSzpY/FQ8jplk0fj691ogSs/+clP1z+UdagJlCfkf1M2J2wVeUdT2RgR/9zPUG7Q76c8RH6X8mT6TPzDLx+ybcy92WBNym6/mXS+S/BwyiL+y8A/KWtftbxhlN1wL1Gmei5k7kX3PYCpkf83YrGesmngnLjBPQ9cVimzM2XX2wuU0eDFwDLAxsQNlLJb8UrmbMA4n7KYPzPs2b1OW+bqh0gbQdlYMCP+jqjk3Qh8oZNr9jHKxowXgCMr7b4vdN5EZUMAc+8SfCH6Yak6un9CGTXNpEzZHdik3WsCd0a5qyg78+pu2In0gyg7L2dSNoVsGumrxHX9d1yrO6hs6GinYx3m7PD8QztbDezfTv5G5t4l+CdgxUr++6L//kP5Tl9FGYUNZs7uyRdCz/qNrlNtl1CSJEnSJJKmU5zgtf1tS18RG30eAP7H9ouV9BspjrRbb3DpCjklmCRJkjQkfmx9BPC7qrPqaxaOX0cnSZIk3SLWmJ6i7P7buV9tySnBJEmSpBXIKcEkSZKkJcgpwSTpI1ZccUW3tbX1txlJ0u9MnDjxWdsrdbVcOqwk6SPa2tqYMGFCf5uRJP2OpEZvZalLTgkmSZIkLUE6rCRJkqQlSIeVJEmStATpsJIkSZKWIDddJEkfMe2JGbQddVV/m5Ekvcb043fpVf05wkqSJElagpZyWJKmS1qxv+1oBkkHSfpsB+ltku7pps42Sa9KulvSfZLOjdAHfYKkMZL2jOOzJK3fV3U3w7z0bR19IyVt1blkkiR9QU4J9hK2T+8l1X+3PVQl/PRY4JO8PZjaPCNpUTcI9W77Cz1d53zISEqYhts6kUuSpA/o8xGWpK9J+kocnyjp+jjeXtIFcbyTpNslTZJ0cQSyq/F1SdMkjY+onbUn/9MlTZD0kCK0djxxjws9k2pPyxGI7lRJD0gaK+nqyshhc0k3SZoo6RpJgxu0ZZEY9S1XSXs4ApMdLenIis4pkqYAX67IDpB0gko48KmSvhjpivR7oq2j2tftEhxtPBFOu57dEcTt2qh/kqS16umPEcU4SVcA94XcKZIelHQtJRJszfYbVYLKIWmmpB9EHXdIWjnS14rzaZKOkzSzie/HzPhe3CvpOkkrRfrQ0DVV0uWSlm+ib38abZwq6dBI/2709z2SzpSkSP+Kyqh1qkrY9jZKbKHDVUa023Rme5IkvUt/TAmOA2r//MOAgSrTWtsAN6tM+X2bElxsM0ogvCMq5WfY3ogSrO6kSnobsAWwC3C6pCUoweB2DD2jKAHQAD4e8usD+wJbAoQdvwD2tL05cDbwg3oNcYmM+kdKoDckvQ94zHb7cOa/AQ61vUm79M9He4YDw4EDJK0R9g0FNgF2AE5o7zijfe8D/tKJ3RcAv4y6t6IE02ukfzPgMNvrRrvWi376bJTviKWBO6KOm4EDIv3nwM/jev2zTtmOdE2wvQEluNv3Iv1c4Bu2NwamVdLr9e2BlGs8NMrURqGn2B5ue0NgSUq4b4CjKMHuNgYOsj0dOB040fZQ2+OatH8uJB2o8iA1YdYrM7qjIkmSoD8c1kRgc0nLUsIi305xXNtQnNn7KTfIWyXdDewHrF4pf2Hl75aV9N/bnm37YeARYAiwGPArSdMoEU9ray4jgItD/t/ADZG+HrAhMDbq/jbwbhpzEcUZAnyKdmGyY/S1nO2bI+m8SvZOwGejrjuBd1KifY4ALrQ9K5zfTRSHBrBWyD8FPGl7aj27JS0DrGr7cihhrm2/0on+8bYfjeNtK3L/Aq6v0wdvMCes90SKo4ByfS6O49/WKdue2czpw/OBEZIGUfrwpkg/B9i2k77dATijNq1p+z+Rvp2kO+M7sT2wQaRPBS6Q9Bmg7lRoV7F9pu1htocNWGpQT6lNkoWSPl/Dsv2mpEeB0ZS1ganAdsDawP3AWsBY23vXU9HEce38cMqNfROKc36tE/ME3Gt7y07kqtwOrB1TV7sDx3WhrCijg2vmSpQ+3KBMbQ1rRYpT3w14tCO7w2F1lZe7UeZNz4lTM4ue/V71WPybGJWeCgyz/biko4ElInsXioP+KPAtSRv1VL1JkvQM/bVLcBxwJGX6aBxlrWBy3PTuALbWnPWppSWtWyk7qvL39kr6XiprSmsBawIPAoMoo5DZlKm/ASF7K/CJkF+ZsrhOlFlJ0n+nCCVtEMeHSDqkfUPC5suBnwH3236uXf4LwAuSRkTSPpXsa4CDY0oPSeuqBEsbB4yKdZiVKDfS8e30PkuZxvrfenbbfgn4p6TdI31xSUs1oz+4uSI3mPJg0RXuAD4Rx5+qZkh6oE6ZRYA94/jTwC22ZwDPV9aR9gVu6qRvxwJflLRo1LcCc5zTsyrrorV1y0WA99i+AfgG5XszEHgJ6I7TT5KkF+hPhzUYuD2mpF6LNGw/Qxl9XShpKsUpDamUXT7SD6OMoGr8g3LT/TNlDeI1ytP0frEgP4Q5o4dLKWsq91GmnSZR1pLeoNzEfhxl7mbOus0QYC5nVOEi4DO0mw6ssD/wy5iuUyX9rLBhksp27DMoo5PLKSPPKZRpuK/H1GV7/gAsRVnLqmf3vsBXos9uA/6nC/ovBx4OG89l7geEZvgqcETUvTYwAyBGh6pT5mVgi+iP7YFjI30/ylrbVMr6Wy29Ud/+A5gaffLpcHC/Au6hPCzcFbIDgPNjmnAycHLI/gnYo7bpQtJuko6NNqwi6epaZSobd1bpWvckSdIVFoiIw5LGAFfavqQLZQbaninpnRRHt3Wdm3ZN/krg4+HUkiaI0dyrti3pU8Detj+msotzTdsnd1Bmpu2Bb1O2ALD44HU8eL+T+tuMJOk1mn3ThaSJtod1Vf/C/DusK2PR/h3A9xs5KwDbuzbKTzpkc+CU2Dr+AvA5ANtXNiq0oLLRqoOY0MuvrkmSBZkFwmHZHt2NMiN73pKkSmwFb7/dvLMyC+ToKkmSeaelXs2UJEmSLLykw0qSJElagnRYSZIkSUuQDitJkiRpCdJhJUmSJC1BOqwkSZKkJUiHlSRJkrQE6bCSJEmSliAdVpIkSdISLBBvukiSVmDaEzNoO+qq/jYjSeaJZt8X2BvkCCtJkiRpCdJhLYRIaovwHT2ha3qEC1ngiHAiR3UiM1TSR/rKpiRZmEmHlSQdIGlR21fYPr4T0aFAOqwk6QPSYS28LCrpAkn3S7okYlch6YOSJkuaJulsSYs3Sq8haUlJf5Z0QKNKY0T2owiKOEHSZpKukfR3SQeFzEBJ10maFPV9LNLbwt5fSbpX0l8lLRl5N0oaFscrSpoexxtIGh/1TZW0TqR/Ns6nSDov0sZIOl3SncBPJI2WdEq7vAmSHpK0q6R3UAJJjgr9o0iSpNdIh7Xwsh5wqu33Ai8CX5K0BDAGGGV7I8qmnIPrpVd0DaRE573Q9q+aqPsftodSokyPoURLfj9wTOS/BuxhezNgO+D/IqYWwDrAL21vQImx9YlO6joI+HnUNwz4p6QNgG8D29vehBK9usa7ga1sH9GBrjZgC2AX4HTK/893gYtsD7X9tojTkg4MJzdh1iszOjE1SZJGpMNaeHnc9q1xfD4wguLEHrX9UKSfA2zbIL3GH4Hf2D63ybqviL/TgDttv2T7GeD1CKop4IeSpgLXAqsCK0eZR23fHccTKU6kEbcD35T0DWB1268C2wMX234WwPZ/KvIX255VR9fvbc+2/TDwCDCks4baPtP2MNvDBiw1qDPxJEkakA5r4cWdnHeFW4GdK6Ogzng9/s6uHNfOFwX2AVYCNo+R0VPAEu3KAsxizk8z3mLO97kmi+3fArsBrwJXS9q+E9tebpDXk32WJEkXSYe18LKapC3j+NPALcCDQJuktSN9X+CmBuk1vgs8D/yylhBrUKt207ZBwNO235S0HbB6E2WmA5vH8Z4VO9YEHrF9MmUkuDFwPbCXpHeGzApN2rWXpEUkrQWsSemXl4BlmiyfJMk8kA5r4eVB4MuS7geWB06z/RqwP3CxpGmUEc/p9dLb6TsMWFLSTyQtAqwN/IfucQEwLOr6LPBAE2V+SllvmwxUt9l/ErhH0t3AhsC5tu8FfgDcJGkK8LMm7foHMB74M3BQ9MsNwPq56SJJeh/ZOauR9CySNgQ+V2fjQksiaQxwpe1Luqtj8cHrePB+J/WYTUnSH/TEmy4kTbQ9rKvl8tVMSY9j+x5ggXFWPcVGqw5iQj++1iZJWp10WEnSBLZH97cNSbKwk2tYSZIkSUuQDitJkiRpCdJhJUmSJC1BOqwkSZKkJUiHlSRJkrQE6bCSJEmSliAdVpIkSdISpMNKkiRJWoJ0WEmSJElLkG+6SJI+YtoTM2g76qr+NiNJ6tIT7wnsTXKElSRJkrQE6bB6AEnTJa3YuWTfIWlxSdf2ddgLSWMkPRr1TqrE3OqKjq9KWqo37Guy/rMkrR/H8921TZKFlZwSXHDZFCAi9jaFpAENwsN3ha/ZvkTSTsAZlKCJXeGrwPnAKz1gS5ex/YX+qDdJksbkCCuQ9DVJX4njEyVdH8fbS7ogjneSdHuMHC6WNLCi4uuSpkkaX4vMG6ON0yVNkPSQpF0jvU3SuNAzSdJWkb6IpFMlPSBprKSrJe0ZeZtLuknSREnXSBrcoC3votzwh8dIZy1JH5Q0OWw8W9LiITtd0o8lTaJE1L1R0rDIW1HS9DgeLekPYdd0SYdIOiJ03lEnau/NlECOhOw98flqpC0t6SpJUyJ9VFyDVYAbJN3QyTVbS9Jfok/GSRpS6ffTwq5HJI2MNt8fca1q5U+La3OvpGMq6f/tgyRJ5h/SYc1hHLBNHA8DBkpaLNJujmmhbwM72N4MmMDcMZ9m2N4IOAU4qZLeBmwB7AKcLmkJ4Glgx9AzCjg5ZD8e8utTwtBvCRB2/ALY0/bmwNmUiLkdYvtp4AvAuBhhPQGMAUaFjYsCB1eKPGd7M9u/66SPNgwbh0f9r9jeFLidEhm4PR8FpknanBKx+H3A+4EDJG0K7Az8y/YmtjcE/hKh7P8FbGd7u07sORM4NPrkSODUSt7ylP47HLgCOBHYANhI0tCQ+VYEkdsY+ICkro4EO0XSgeEUJ8x6ZUZPq0+ShYp0WHOYCGwuaVngdcpNeBjFYY2j3GjXB25VCbe+H7B6pfyFlb/VdZvf255t+2HgEWAIsBjwK5UQ8BeHXoARwMUh/29K+HWA9SjOYmzU/W3g3V1o23rAo7YfivNzgG0r+Rc1qecG2y/ZfgaYAfwp0qdRHG2NE8LOA4HPR7sut/2y7ZnAZZR+nQbsGCO8bWw3fUeP0e1WwMVR1xlAddT5J5dw2tOAp2xPsz0buLdi6ydjZDmZ4szWp4exfabtYbaHDVhqUE+rT5KFilzDCmy/KelRYDRwGzAV2I4ypXU/sBYw1vbe9VQ0cVw7Pxx4CtiE8tDwWifmCbjXdpc3MDTJy5Xjt5jzILNEO7nXK8ezK+ezmfu79LVqKHlJH+yoUtsPSdoM+AhwnKTrbB/bpM2LAC80WKOr2tbe7kUlrUEZlQ23/XxMFbZvb5Ik8xE5wpqbcZSb2M1xfBAwOZ7U7wC2rqxPLS1p3UrZUZW/t1fS94q1qbWANYEHgUHAk/HEvy8wIGRvBT4R8isDIyP9QWAlxY47SYtJ2iCOD5F0SCftehBoq9kedd5UR3Y6sHkc79mJ3mYZB+wuaSlJSwN7AOMkrUKZVjwfOAHYLORfApapFZZ0rqQtqgptvwg8KmmvkJGkTbpg07IURz0j+vrD3WxbkiR9RDqsuRlHmVa63fZTlJHPOICYBhsNXChpKsUpDamUXT7SD6OMoGr8AxgP/Bk4yPZrlLWW/SRNCR21Ec6lwD+B+yibJiZR1sbeoDiPH0eZuynTYUT55xo1KurcnzJ9No0yyji9jvhPgYMlTQZ6ZDu37UmUNbTxwJ3AWbYnAxsB42NK73vAcVHkTOAvlU0XG1PWtdqzD/D56JN7gY91waYplKnAB4DfUh4WGhKbYFaJ42Ml7RbHu0lqdmSYJEk3URk8JL1BTDNdWZ0ea6LMQNszJb2TcoPfOtaz6slfCXw8nNoCR6wp/tr2Xv1ty7yy+OB1PHi/k/rbjCSpS1+96ULSxNjw1CVyDWv+40pJywHvAL7fyFkB2N61T6zqJ2Lqr+WdFcBGqw5iwnz+6pskmZ9Jh9WL2B7djTIje96SJEmS1ifXsJIkSZKWIB1WkiRJ0hKkw0qSJElagnRYSZIkSUuQDitJkiRpCdJhJUmSJC1BOqwkSZKkJUiHlSRJkrQEnTosSXtJWiaOvy3psnjDdpIkSZL0Gc286eI7ti+WNALYgfJW7dMowfiSJGmSaU/MoO2oq/rbjGQBo6/e/zc/0MyU4Kz4uwtwpu2rKO+5S5IkSZI+oxmH9YSkMyhxnq6WtHiT5eYrJLVJuqeHdE2X1HToja7KN9AzRtLbYlRJ2lXSZElTJN0n6Yvd1P/Ndue3ddfWBQFJq0hq+KZ9SctJ+lJf2ZQkCzPNOJ5PAtcAH7L9ArAC8LXeNCppHkmLUeJHfdT2JsCmwI3dVDeXw7K9VT3BBR1Ji9r+l+3OglguB6TDSpI+oFOHZfsV4GlgRCS9BTzcm0b1IotKukDS/ZIukbQUlBDuMUKZJunsGEXWTa8haUlJf5Z0QBN1fz30jK9ELW6TdL2kqZKuk7Rao/R2dX8/4m0tS1mLfA7A9uu2HwyZlSRdKumu+Gwd6QMl/SbsmSrpE5KOB5aUdLekC0JuZvyVpBMk3RNlRkX6SEk3Rl8+EH2rRp0Qo8TTJN0h6ZHQcXZckzEVudMkTZB0r6RjKunTJR0jaVLYMiTSj5Z0ZEXunujHpSVdFaPPeyq2D5d0W6SPl7SMpNGSrpB0PXBddVQeeX+M9j4s6XtR1fHAWtFvJzTxPUiSpJs0s0vwe8A3gP+NpMUo0XBbkfWAU22/F3gR+JKkJSjRcEfZ3ohy8z+4XnpF10DgT8CFtn/VRN0zQs8pwEmR9gvgHNsbAxcAJ3eSDkDcGFcC9rf9HHAF8JikCyXtI6l2XX8OnGh7OPAJ4KxI/07NnqjjettHAa/aHmp7n3a2fxwYCmxCbLyRNDjyNgW+CqwPrAls3URfLA9sSYnMfAVwIrABsJGkoSHzrQjwtjHwAUkbV8o/a3szyuafI2nMzsC/bG9ie0NKJON3ABcBh8WodAfg1ZDfDNjT9gc60LUFpR83BvaSNAw4Cvh79NvbZh4kHRiOd8KsV2Z0YmqSJI1oZkpwD2A3Ioy77X8By/SmUb3I47ZrodDPp4wa1wMetf1QpJ8DbNsgvcYfgd/YPrfJui+s/N0yjrekhGcHOI85o9h66VCczSDbBznCRdv+AvBBSoTiI4GzQ3YH4BSVEPRXAMtKGhjpv6wptP18J7aPoDjmWbafAm4ChkfeeNv/tD0buBto60QXwJ/C9mnAU7anRfl7K+U/KWkSJYz9BhSHWOOy+DuxifqmATtK+rGkbWzPoFzbJ23fBSVIpO23Qn6s7f/U0TXW9nO2Xw0bRtSR+y+2z7Q9zPawAUsN6kw8SZIGNOOw3oibiwEkLd27JvUq7uS8K9wK7NzZFFiduual3ruAzSWtMJfyctM/EdiRMgqAcn3fH0//Q22vanvmPNTdEa9XjmfR3E8lamVmtys/mzJtuwbF8X4wRoBXAUt0UL5a31vM/X1eAiAeODajOK7jJH23E9tebpDXk9+fJEm6SDMO6/cquwSXi7Waa4FmpsDmR1aTVBvdfBq4BXgQaKutKwH7UkYQ9dJrfBd4nspIJdabVq1T96jK39vj+DbgU3G8DzCuk3SAv1DWTa6KdZeBkkZW8ocCj8XxX4FDK/YNjcOxwJcr6cvH4ZsqmzjaMw4YJWmApJUoI83xddpZ0/kjSXs0kmnAshTHMUPSysCHmygzneKYUPlh+xpxvArwiu3zKb8h3IxybQdLGh4yy0hqxtHuKGkFSUsCu1MeWl6idWcckqSlaGbTxU+BS4BLgXWB79r+RW8b1ks8CHxZ0v2UdZTTbL8G7A9cLGka5Sn/9Hrp7fQdRtmo8JNYN1obqDedtLykqVHm8Eg7FNg/0veNvEbpANi+mPLQcAUgyoaOB2Pq7xhgdIh+BRimsrHiPuCgSD8u7LlH0hRgu0g/E5iq2HRR4XJgKjAFuB74uu1/12lnjY2AzmQ6xPYUylTgA5Sp0VsblwDK93MFSfcChwC1qdyNgPHRN98DjrP9BuXB4RfR/rHMPYKrx/ioZypwqe0JsYZ4a/RlbrpIkl5EsQzSWEj6H8qCs4G7mrhZLXRI2hD4nO0j+tuW+QFJ19j+UH/b0VNIGg0Ms31Id3UsPngdD97vpB6zKUmgNd90IWlibKrqEp1Og0j6AmX663rK0/wvJB1r++zGJRcubN8DpLMKFiRn1VNstOogJrTgzSVJ5heambf/GrBpTH0g6Z2UNZZ0WMlCg+0xlJ85JEnSTzSz6eI5ysJyjZciLUmSJEn6jGZGWH8D7pT0R8oa1scoC/NHANj+WS/alyRJkiRAcw7r7/Gp8cf4m1t5kyRJkj6jGYd1me1pvW5JkiRJkjSgmTWsU+PloF+SlO+WSZIkSfqFZn44vA3wGeA9wERJv5W0U69bliRJkiQVmgrEGO9j+zblre0fAH6uEk7i471pXJIkSZLUaCa8yMaSTgTuB7anBAp8bxyf2Mv2JUmSJAnQ3KaLX1DiKH0zwioAJcyIpG/3mmVJkiRJUqEZh3W57fOqCZIOs/3z9ulJktRn2hMzaDvqqv42I2kxWvFdgb1FM2tYn+0gbXQP25EkSZIkDanrsCTtLelPwBqSrqh8bqB+CI0FDknHStphHnWcJOmJSuj6jmSGSvpI5Xw3SUd1orengzEuMEjaXZIlDWkgs5ykL1XOV5F0SSd6b5TU5bdMJ0ky7zSaErwNeBJYEfi/SvpLlHhACwW2O4tQ25BwUnsAj1N2WN7QgcyilMCLw4Cro94rKPGukiaRtGgl1P3elACde1PiYL1NFlgO+BJwKpR1WWDPPjE2SZIuU/eJ3/Zjtm+0vaXtmyqfSZWbQq8hqS22zo+R9JCkCyTtIOlWSQ9L2iLktpB0u6TJkm6TtF6kLyXp95Luk3S5pDtrT8aSTpM0QdK9ko6JtGGS7o7PNEmO9DGS9ozj6ZKOkTQpZIZE+tKSzo4fWE+W9LFKU0YC9wKnUW6etfYdLek8SbcC5wHHUqL63i1plKTRkk4J2ZWjDVPis1UH/fU1SXdFsMZjmujfLrVF0lWSNo7jyYpQ8zECPUDSYEk3h/33SNom8mdKOjH6+jqViMVEmbuiPZdKWqrS36fH9XlI0q6RPkDSCZU2fjHSR0oaJ+kK4L5IGwiMAD7PnMjNHckeD6wVNp8Q37l7KvX9NNoyVdJ/IzdX9O0U371Jki6OepMk6SWa+h1WP7I2ZXQ3JD6fptyIjgS+GTIPANvY3pQSt+uHkf4l4Hnb6wPfATav6P1WBA/bGPiApI0jeuxQ20MpYeh/WsemZ21vRnFAR9b0Adfb3oISvfcESUtH3t7AhZSovbto7hD06wM72N47bL8obLioXZ0nAzfZ3oQS4v3eaqbKD7nXoQTZHApsLmnbOvZ3ty3jgG1U3nbyFrB1yG8D3Ey5NtdE/20C3B35SwMTbG8A3MSc0c5ltodHm+6nOJcabdGWXYDTJS0R+TNsDweGAwdIWiPkNwMOs71unH8M+Ev8fvA5SdVrX5U9Cvh79PnX2vXNgWHHUNsbA3NFYZa0IuW3iTtEH06gg3hokg4M5zth1isz2mcnSdIFmtkl2J88WnuPoUro8+tsWyVkfVvIDALOkbQO5W3yNYcwAvg5lOCKKuHma3xS0oGU9g+mOI6pUc8oyk2t3ts8Lou/E4HaD6d3AnaTVLvpLwGsJunvwEeAI2y/JOlO4EPAlSF3RfWnAg3Yntj8YnsW0P7Ot1N8Jsf5QIoDu7kTvU23heKwvgI8ClwF7BijojVsPyhpZeDscMh/sH13lJ8N1Bzw+ZU6N5R0HGVabiBwTcWu39ueDTws6RHKw8pOwMa10S7luq8DvAGMt/1opfzexLUHfhfnE+O8vWw9dgBOr80m2G6/bvt+yvfmVkkA7wBub6/E9pnAmVAiDjdRb5IkdWjKYUlaEljN9oO9bE97Xq8cz66cz2aO7d8HbrC9h6Q24MZGCuOp/EhguO3nJY2h3JRrYe6PBrYNx9DIplkVGwR8on3/SPoo5YY8LW5qSwGvMsdhvdzI1i4g4Ee2z+hiua605R2UNbZHgLGUtc0DCEdg++YY1e0CjJH0M9vndlBn7aY9Btjd9hSV8PMjO5Cpngs41HbVsSFpJJV+lLQCxcFvpDKtOwCwpNoIqif7fGyMjpMk6QOaedPFRynTO3+J86GxBjC/MAh4Io5HV9JvBT4JIGl9YKNIX5Zy05oRo4IPh8xylKm7z9p+pos2XAMcqvBKkjaN9L2BL9hus90GrMGckUl7XqJ+yJbrgIND9wC9/SXE1wCfq62hSFpV0rvi+DpJq85rW2y/Qdk4shdlJDGO4vhvDrnVgads/4ryQ/PNQt8izNnI8GnKRgiirU/GiGyfdjbsJWkRSWsBawIPhl0H16ZUJa1bmXatsidwnu3Vo9/fQxkVbtOBbKM+Hwt8UWVzRs0RVrkD2FrS2pG/tKR1SZKk12hmDetoynrCCwAx1bNGffE+5yfAjyRNZu4R46nASpLuA46jrPvMsD2FMnX2APBbimODsu6xOvCrWIS/uws2fJ8yFTk1pi6/H05pZ8r0GQC2X6bcsD/agY4bgPWj7lHt8g4Dtoup0ImUqaj/Yvuv0ZbbQ+YSYBmVHYpr07WfIbytLZW8ccDTMY05Dnh3/IUyQpoS12EUc6bkXga2iM0M21M2l0BZV7yT0v8PtLPhH8B44M/AQbZfozjB+4BJoesMOp4h2JuyXljlUiobXmrYfo4ypXePpBPaZZ8VdkyVNIXibKtln6E8IF0Y0823U6YukyTpJWQ3nlaXdIft90uaHBsbkDQ1FqLnWyQNABaz/Vo8qV8LrBcjhYWCmOL8nO23bQboQxtm2m5691xM0V5pu+HvoVqRxQev48H7ndTfZiQtxoL4pgtJE2PjW5doZg3rXkmfBgbExoavUH6jNb+zFHBDTCEJ+NLC5KygbDahg51rSf+w0aqDmLAA3nySpK9oxmEdStnq/Dpl2ukayhTbfI3tlyibBJJ+pCujq5Af3UumJEnS4nTqsGy/QnFY3+p9c5IkSZKkY5rZJTg2dtDVzpeXdE2DIkmSJEnS4zSzS3BF2y/UTmw/D7yr1yxKkiRJkg5oxmHNlrRa7SR+b5O/2E+SJEn6lGY2XXwTuEXSTZTddttQ3rOWJEmSJH1GQ4cVPzwdRHlrwfsj+au2n+1tw5IkSZKkSkOHZXu2pK/b/j1z3n+XJEmSJH1OM2tY10o6UtJ7JK1Q+/S6ZUmSJElSoZk1rNp77b5cSTPlpaRJkiRJ0ic088Ph+elFt0nSskx7YgZtR13VuWCSsGC+Q3Be6dRhSfpsR+l1Yh0lSZIkSa/QzBrW8MpnG0q4kd160aZ5QlJbhJ/oCV3TIxR6+/RvSbpX0tQIB/K+btr56cr5MEknz6vNvYWkvSTdL+mG/ralRvThq3EN7pN0euxsTZJkAaSZKcFDq+fxmqbf9ZZB8zuStgR2BTaz/Xo4tHd0Q1UbJcbSbwFsTwAm9JSdvcDngQNs39KpJCBp0Vp4+V7m77aHRqDF64Hdgct6upI+bE+SJHXoztPoy8xfARw7YlFJF8SI4JJahF9JH5Q0WdI0SWdLWrxReg1JS0r6s6QDgMHAs7ZfB7D9rO1/hdzmkm6SNFHSNZIGR/rakq6VNEXSpIjPdTywTYwODpc0UtKVIb+CpD/ECO4OSRtH+tFh342SHpH0lc46QtJwSbdF3eMlLSNpCUm/ifZOlrRdyI6WdJmkv0h6WNJPIv27wAjg15JOUIl6fIKku8LGL4bcSEnjVCJS39eJ3I1xbR6Ia6UG9naop0o4k9uAtSUdELJTJF1auf5jYhQ2QdJDknaN9Kba08wXL0mS3qOZl9/+SdIV8bmKEq68fUTX+Y31gFNtvxd4EfiSpCWAMcAo2xtRRpcH10uv6BoI/Am4MMK//xV4T9zwTpX0AQCVuFu/APa0vTlwNvCD0HEB8EvbmwBbAU8CRwHjbA+1fWI7+48BJkeQzG8C1fXCIcCHKFGgvxf1doikdwAXAYdF3TsAr1J2fDrauzdwTvQDwFDKztCNgFGS3mP7WMrobx/bX6OMtmbYrk0VHyCp9hCzWdS3bidymwJfpURPXpMSbr6evY301Nq6FPBBYBpwme3hoeP+KF+jLfpuF+D0aHez7ekykg4MBzlh1iszuqMiSZKgmW3tP60cvwU8ZvufvWRPT/G47Vvj+HxK0MmxwKO2H4r0cyg37hvqpJ8U538EfmL7AgDbMyVtTlnP2w64SNJRlBv6hsDYGCwMAJ6UtAywqu3Lo/xrACFTjxHAJ0L+eknvlLRs5F0Vo7vXJT0NrAzUux7rAU/avit0vRh1j6A4V2w/IOkxoHZDvs72jJC7D1gdeLyd3p2AjSXtGeeDgHWAN4Dxth9tUu6fUc/dFEcyo4699fQ8BKwV5Q380fafJX1A0nHAcpQHjmp0gd/bng08LOkRygNAs+3pMrbPBM6EEnG4u3qSJGluDesmSStTnjwBHu5dk3qE9jeGeblR3ArsLOm3tg1gexZwI3CjpGnAfsBE4F7bW1YLh8PqSV6vHM+iuYeOntYv4FDbc4WZkTSSMmXcjFxX2lFPTxuxhtVOfgywu+0pkkYDIyt5HX03mm1PkiT9SDNTgp8ExgN7AZ8E7qw8ic6vrKayOQLKxoZbKFOZbZLWjvR9gZsapNf4LvA88EsASetJWqeSPxR4LPSsVKtX0mKSNojIx/+UtHukLx7TVy8B9ZzZOGCfkB9JWTN7sVGDJV0nadV2yQ8CgyUND5llVDYnVPWvC6wWss1yDWU6dbGaDklLz4NcZ/Z2Vc8ylNHtYrV2VthL0iIq64hrRp1d1Z8kST/QzNP5t4Dhtp8GkLQScC1wSW8aNo88CHxZ0tmUxfLTbL8maX/g4rgJ3gWcHjv93pbeTt9hwNkqmxAuAn6hslvyLeBvwIG23whHfrKkQZS+PQm4l+IEz5B0LPAmxflPBWZJmkIZEUyu1Hd01DcVeIUygquLylbutYH/VNPDplFh75KU9aAdgFOB02J0+BYwOvqhYadWOIsyhTdJpdAzlN153ZXrzN4u6QG+A9wZcncy94PBPygPYMsCB8X3oin9IXe67QmSDgqbT5c0LHR9oYFNSZLMI4pZrvoC0rRYnK+dLwJMqaYl/YukDYHP2T6iv22Zn5E0BrjSdr88bC0+eB0P3u+k/qg6aUEW5DddSJpoe1hXyzUzwvqLpGuAC+N8FHB1VytKeg/b9wDprOZzNlp1EBMW4JtQkvQ2zWy6+Jqkj1N2rgGcWdvxliSthO3R/W1DkiTdp5l3CS5N2S58maT1gPUkLWb7zd43L0mSJEkKzbzp4mZg8diB9hfKBoIxvWlUkiRJkrSnGYcl268AH6fsttsL2KB3zUqSJEmSuWnKYcVvi/YBasF8BvSeSUmSJEnydppxWIcB/wtcbvteSWtSXmeUJEmSJH1GM7sEb6asY9XOH6G8my9JkiRJ+owMdpckSZK0BOmwkiRJkpYgHVaSJEnSEjTzw+E1gZ8DWwKzgduBw2MtK0mSJpn2xAzajrqqc8FkoWJBfmdgT9PMCOu3wO+B/wFWAS5mznsFkyRJkqRPaMZhLWX7PNtvxed8YIlOS7UIkk6QdK+kE/rblhqSRkt6RtLdku6TdEA/2nJbP9Y9UtKV7dL+IOmOJsptVTk/SNJnG8i3Sbpn3i1OkqQ3aeZt7X9WCQH/O0p01lHA1ZJWALD9n0aFW4ADgRUiinCnSFrU9lu9bBPARbYPkfQu4F5JV9h+qqcr6aw9treql9fFegY028cNdCwHbA7MlLRmR9PSEdNsJDATuA1KzKp5qTdJkvmDZkZYnwS+SPmx8I3AwcCnKCHhJ/SaZZ0g6bOSpkqaIum8SGuTdH2kXydptUgfI+lkSbdJeqQWMVnSFcBAYKKkUZJWknSppLvis3XIHS3pPEm3Aud1Ine2pBujnq90Ym+HeqpE4My/A6tLOk3ShBgRHlPRPV3STyRNkzReET252fZ00s8z469iNHpP1DMq0ucaBUk6RSUsfc2uH0uaRIn0O13SMZImhY4hIbeFpNslTY5rtF4dcz4O/Iny8PSpSp1jJJ0u6U7K9PVBwOExQt0m2ntkyK4t6dq4DpNUIg9X2zsg2nlXXK8vNuqfJEn6jmZ+OLxGXxjSFSRtAHwb2Mr2s7XRHvAL4Bzb50j6HHAycyLHDqaESBkCXAFcYns3STNtDw29vwVOtH1LOLtrgPdG+fWBEbZf7URuCLAdJcrtg5JOA9atY+/PG+iptXVNSij3vwHfsv0fSQOA6yRtbHtqiM6wvZHK1NdJwK6d6P9ve5rs9o8DQ4FNgBWBuyTd3LBE4Tnbm0Vbjgeetb2ZpC8BRwJfAB4AtrH9lqQdgB8Cn+hA197AscBTwKUhV+PdlP6dJeloYKbtn0a9H6zIXQAcb/tySUtQHtreVcn/PKUvh0taHLhV0l9tP9pEW9+GpAMpo3gGLLtSd1QkSRI0s0twMcqoattIuhE4o5/Di2wPXGz7WZhrWnJLyo0VysjhJ5Uyf7A9G7hP0sp19O4ArK85oeKXlTQwjq+o3NwbyV1l+3XgdUlPAys3sLeRnlGSRgCvA18MR3VQ3AAXpTjg9YGaw7qw8vfELranGUYAF8a03lOSbgKGAy92Uu6idueXxd+JzLlWg4BzJK1DmXZerL2SuGbrALfYtqQ3JW0YwSuh9G/DKUdJywCr1uK52X4t0qtiOwEbK0bhYds6QLcclu0zgTOhRBzujo4kSQrNrGGdRrmBnBrn+0baF3rLqF7i9cqx6sgsAry/diP7r3C5ob3cpFy1nlk07uNGei6yfUglbQ3KiGS47edVwr1XN7+4g+Nm2zMvvMXcU8vtN+S0r6fWP9W++T5wg+09JLVRHora80lgeeDRsH9ZyojrW3Xq6S4CDrV9TQ/pS5Kkh6i7hqWyeA3lBrmf7evjsz/lybo/uZ6yJvJOgMoU223MWdvYBxjXRb1/BQ6tnUgaOo9yndnbFT3LUm7KM2K08eF2+aMqf2/vin5Jq0q6rpM2jKOM+gZIWoky4h4PPEYZxS2usinigw101GMQ8EQcj64jszews+02222UzRefqiP7EmVKdi5svwT8U9LuAGHzUu3ErgEOjpkFJK2rEsQ0SZJ+ptGmi/Hxd1Z1YTrWVOZpt9e8Yvte4AfATZKmAD+LrEOB/SVNpYwED+ui6q8Aw2Kx/T7K4v28yHVmb9N6bE8BJlPWe34L3NpOZPlo92HA4V3UP5gyUuqw6vh7OWX6cQrFAX/d9r9tP07Z6HBP/J1crw0N+AnwI0mT6WBEGqOu1YH/bmePNaUZkt7Xgb4/AXvUNl20y9sX+Er01W2U3xdWOQu4D5ikstX9jJpNku6u2HSWpGFxfJCkht+BJEnmHdkdT6tLmmx7U0nbUyIM17YQtwH7284QI/MJkqYDw2prZN0ofwjwD9tXtEt/JzDJ9urzbmUybNgwT5jQbxtrk2S+QdJE28O6Wq7R+spKko6I4zOYE7RxFrApGRNrgcH2Ke3TJK1CWUv6aZ8blCRJ0gGNHNYAym+U2m9QWJQO1geS/iPWdHpa578o2/GTJEnmCxo5rCdtH9tnliRJkiRJAxptuqi39TtJkiRJ+pxGDqs725OTJEmSpFeo67AWgJfaJkmSJAsQGXE4SZIkaQnSYSVJkiQtQTqsJEmSpCVIh5UkSZK0BM28rT1Jkh5g2hMzaDvqqv42I+kjph+/S3+bsMCRI6wkSZKkJUiHlSRJkrQELemwJI2UdGUP6ttd0vo9pa8D/WNqEWwjLEXDuiRNlzQtwmNMk/SxXrLraElH9qC+r3YQX6ojuWMl7dCJzEhJW/WgbbtJOqqn9CVJ0ve0pMPqBXanhJtvmkqAyy5h+wu272tCdDvbQ4E9gZM7qF+S5rfr91WgU4dl+7u2r+1EbCTQJYfV6JrYvsL28V3RlyTJ/EW/3PAktUl6QNIFku6XdEntyVzSByVNjpHF2ZIWj/Sdo8wk4OMVXQMl/Sbkp0r6RKSfJmmCpHslHVORP17SfSH703iK3w04IUY0a1GHGJGcJ+lW4LxoxzhJk+KzVchJ0imSHpR0LfCuio4bK4H/9g6775H04zrVLgs8X+m3ByWdSwmY+B5JJ0T5aZJGVfrkurBprhGapG9JekjSLcB6lfS1JV0raUqUW6ueHklLS7oqZO+RNErSV4BVgBskNQw9027EOV3SMZU6hqgEbDwIODyuyTaSVpJ0qaS74rN1nWtyh6QN2ve3pNGSTom0erqmSVourt9zkj4b6edK2rFRm5Ik6X36c5fgesDnbd8q6WzgS3FDGQN80PZDcWM+WNLpwK+A7YG/ARdV9HwHmGF7IwBJy0f6t2z/R9IA4DpJG1PCsO8BDLFtScvZfkHSFcCVti9pwu71gRG2X1Vxsjvafk3SOsCFwLCoY72QXZkSwfbsqhKVeFM/poR6fx74q6Tdbf8hRG6QJGBN4JOVousA+9m+I5zzUGATYEXgLkk3A88Ae9h+UdKKwB3Rxs0oYeWHUq79JGBi6L0AON725ZKWoDzMvFFHz87Av2zvEm0ZZHuGSvy07boRSPJZ25tJ+hJwpO0vxDWfafunUcdvgRNt3yJpNUoo+/dG+eo1OTz663uSBgODbU+QtGGlvp/X0XUrsDXwGCVg6TbAucCWwMFdbBNh94HAgQADll2pOyqSJAn6c0rpcdu1MO/nAyMoN/lHbT8U6ecA2wJDIv1hlxDJ51f07AD8snZi+/k4/KTKaGwysAHlpjYDeA34taSPA690w+4rbL8ax4sBv5I0DbiYOdOK2wIX2p4VcaWu70DPcOBG28/YfoviMLat5G9ne0NgI+AUSQMj/THbtVDxIyr1PAXcFHoF/FAlDPy1wKoUx7kNcLntV2y/CFwBIGkZYFXblwPYfs32Kw30TAN2lPRjSdvYntGNfqxyWfydSIlo3RE7RD/cHXYvW+mT6jX5PWUaFYrj6ughpJ6ucZRrsC1wGrCRpFWB522/3J2G2T7T9jDbwwYsNag7KpIkCfpzhOVOzruNpDWAI4Hhtp+XNAZYwvZbkragvIl+T+AQyqitK1RvXIcDT1FGOItQnGGPYvvvkp6iOMOn29Vfj32AlYDNbb8paTqwRDeq71BPjH43Az4CHCfpunmMnfZ6/J1F/e/kIsD7bc/Vx2UQOqdPbD8R03kbA6MoU4vN6roZ+DKwGvAtykh5T4ojS5Kkn+nPEdZqkraM408DtwAPAm2S1o70fSmjhgcivba+tHdFz1jKTQb475TgspSb2AxJKwMfjryBwCDbV1OczSZR7CUqUZQlHSLpkCbaMIgS6HJ22Dog0m8GRkkaENNS23VQdjzwAUkrxrTl3tHWuZD0LmANyjRVe8ZV6lmJMjIYH3Y9HU5mO2D1il27S1oyRlUfBbD9EvBPSbtHnYvHdGeHemI68xXb5wMnUKYa4e39eG48IHSHuXQBfwUOrege2qDsRcDXKdd6agf5Heqy/ThlanUd249QvpNHUvotSZJ+pj8d1oPAlyXdDywPnBZPvPsDF8c022zg9Eg/ELgqpvmerug5DlheZfF/CmUqbQplKvAB4LeUtQkoN8ArY4rrFuCISP8d8DWVzR5rUaYgn2uiDacC+0W9Q5jzpH858DBl7epc4Pb2BW0/CRwF3ABMASba/mNF5IaYsroBOCqm/NpzOTA1yl8PfN32vynTi8OiDz8b/YDtSZSb+RTgz8BdFV37Al+JvrkN+J96eijTlOPDvu9RrgHAmcBfNGfTxcbAvzrquCb4E7CHYtMF8JWwZaqk++h45FTjEspa3e/r5DfSdSdQm5IeR5kGvQUg2kscn6U5m2cOktTIniRJegCVJaE+rrTsArsy1mjmO1R+4/Vx22/0ty2tiqRlgV/b3qu/bZlfGDZsmCdMmNDfZiRJvyNpou1hXS2X7xLsANu79rcNrU5s6khnlSRJj9EvDsv2dGC+HF0lSZIk8yfz25sSkiRJkqRD0mElSZIkLUE6rCRJkqQlSIeVJEmStATpsJIkSZKWIB1WkiRJ0hKkw0qSJElagnRYSZIkSUuQb7pIkj5i2hMzaDvqqv42I+kBph+/S3+bsFCSI6wkSZKkJUiHlSRJkrQEfeKwJB0t6cg4PlbSDg1kh0r6SOV8N0lH9YWddeyZFSEuap+GtkgaKWmrvrKvt5A0XdKKvai/TdI9cTxM0slNljtJ0hOS6n53u/MdkjSzWduTJOkf+nwNy/Z3OxEZCgwDrg75K4hQ7r2BSshaRRDGjnjV9tAuqBwJzKTElGpf16K23+qykQs4ticAncbdCCe1B/A48AFKrLD2MovSx9+hJEn6hl4bYUn6lqSHJN0CrFdJHyNpzzgeLuk2SVMkjZc0CDiWEkX3bkmjJI2WdErIt0m6PgLvXSdptYrOk0PXIxX9A0NukqRpkj5W0fOgpHOBe4DvSDqpYuMBkk7spH3TJR1T0T0k4nwdBBxeCzwYtp0u6U7gJ5IeVokOjKRFJP2tdl6nngGSTpB0V7T7i5E+UtKNki6R9ICkC8L5dtSvy0i6WZUovZJukbRJ9NFvog1TJX2iAxuOUAmQeY+kr1b6sFbv/WHHUpG3uaSbJE2UdI1K1OVa+hSVgJfVKNEjVWKQIWkLSberBNO8TdJ6FVNGAvcCp1GJOq0ygj9P0q3AeTT+Dq0s6fKaHepgNCzpa5X+PqbetUmSpG/pFYclaXNKxNehwEeA4R3IvIMS/fYw25sAO1Ai9n4XuMj2UNsXtSv2C+Ac2xtTouFWp5EGAyOAXYHjI+01YA/bm1HC1P9f7aYOrAOcansD4P+Aj0paLPL2B86O4yU195TgqEqdz4bu04AjI2zK6cCJYf+4kHs3sJXtI4DzgX0ifQdgiu1nOu5JAD4PzLA9nNKPB0haI/I2Bb4KrA+sCWxdp19fBX4NjAaQtC6wRERm/k7o3yj69fpq5XEt9wfeB7w/6t80steLPnwv8CLwpejDXwB72t48+vEHIf8b4NCwqx4PANvY3pTyXfhhJW9v4EJKpOVdKteL6IMdbO9N4+/QycBNYcNmFAdYbe9OlO/GFpTv7+aStm1gb0MkHShpgqQJs16Z0V01SZLQeyOsbYDLbb8Sgfw6mo5ZD3jS9l1QAv41MV22JSXkPZQn6RGVvD/Ynm37PmDlSBPwQ5Ww79dSwp3X8h6zfUfUPZNyo95V0hBgMdvTQu7VuPEN7eAGeFn8nQi0NbD7Ytuz4vhsSrh5gM9RbuKN2An4rEp49juBd1JuqADjbf8zpjPvDhvq9evF0b7Fot4xoWMH4Je1ymw/367+EZRr+XL002WU6wvwuO1b4/j8kF2PEutsbNj8beDdkpYDlrN9c8ifV6e9g4CLVda3TgQ2gP8+4HyEcp1fjL74UKXcFbZfraOzyvaUBwxsz7Ld3ovsFJ/JwCRgCHP6u8vYPtP2MNvDBiw1qLtqkiRhwfod1uuV49ooah9gJWBz229Kmg4sEXkvtyt/FvBNyhN+Z06kfZ2zaNyX/63L9uOSnpK0PeUpfp/6xYDSlkNtXzNXojSSudvc0Abbr0gaC3wM+CSweSf1NoM7OBdwr+0t29m7XJM6vw/cYHuPmGK9MdI/BCwHTItB8lKUkeOVkd/+enYXAT+yfUYP6UuSpIforRHWzcDukpaUtAzw0Q5kHgQGSxoOEOssiwIvAcvU0XsbZaoRyo1+XB25GoOAp8NZbQesXk/Q9p3Ae4BPU6aduksj+2ucRRmR/HfkJWkPST/qQPYa4ODa9JekdSUt3UB3vX6t1XsycFdlJDWWudeTlm+nbxzlWi4V9e7BnH5fTVLNMX0auCXqX6mWLmkxSRvYfgF4QVJtVFzPUQ8Cnojj0ZX0vYEv2G6z3QasAexYWzdrR6NrcB1wcNg2QGXdtMo1wOckDQyZVSW9q46uJEn6kF5xWLYnUdZRpgB/Bu7qQOYNYBTwi1iEH0sZ/dwArN/BehHAocD+McW3L3BYJ6ZcAAyTNI0yDfdAJ/K/B25tNy3Wfg3r+HqFgz8Be4TsNnVkrgAGMvdIbi3KOlB7zgLuAybFNNkZNB5J1etXbE+MOqr1HgcsHxsqplDW+qr6JlGmD8dTpuHOsj05sh8EvizpfmB54LSof0/gx6HvbqC2sWF/4JcxVVgbBbfnJ8CPJE2utTOc0s7Af18TYftlioPs6GGo0XfoMGC7+E5MpKx9Vdv7V8q08+0hcwnh/CRdLWmVOD5W0m5xvJukY+u0J0mSHkJ2+1mdhZfYqXai7et6uZ5hUc82lbTzgcM72YAxr/WuQpliG9JgG3+zutqAK21v2AOmLRQMGzbMEyZ0uns/SRZ4JE20Payr5fJNF5T1FUkPUTZY9LazOgq4FPjfarrtz/Sys/osZYT0rXl1VkmSJP1BjrCSpI/IEVaSFHKElSRJkizQpMNKkiRJWoJ0WEmSJElLkA4rSZIkaQnSYSVJkiQtQTqsJEmSpCVIh5UkSZK0BOmwkiRJkpZgQXpbe5LM10x7YgZtR13VuWDSb0w/fpf+NiFpQI6wkiRJkpYgHVaSJEnSEqTDSpJ5RNJoSaf0tx1JsqCTDitJkiRpCdJhJf2GpLYISlk7P1LS0ZJulPRjSeMlPVQLhBkjmT9IGitpuqRDJB0habKkOyStEHIHSLpL0hRJl9aiEksaI+l0SRNC764VvX+Meh+W9L2KTZ8JO+6WdIakAZG+f+gYD2zdh92WJAst6bCS+ZVFbW8BfBX4XiV9Q+DjwHDgB8ArtjcFbqdElQa4zPZw25sA9wOfr5RvA7YAdgFOl7REpG8BfALYGNhL0jBJ76VEb97a9lBgFrCPpMHAMRRHNYJ2UYurSDowHOSEWa/M6FZHJElSyG3tyfzKZfF3IsXJ1LjB9kvAS5JmAH+K9GkUZwOwoaTjgOWAgcA1lfK/jwCWD0t6BBgS6WNtPwcg6TKKI3oL2By4SxLAksDTwPuAG2sBNyVdBKzbUSNsnwmcCbD44HUy+FySzAPpsJL+5C3mHuUvUTl+Pf7OYu7v6euV49mV89kVuTHA7ranSBoNjKyUae803CBdwDm254oOLWn3t7UkSZJeJ6cEk/7kKeBdkt4paXFg1x7SuwzwpKTFgH3a5e0laRFJawFrAg9G+o6SVpC0JLA7cCtwHbCnpHcBRP7qwJ3AB8LuxYC9esjuJEkakCOspN+w/aakY4HxwBPAAz2k+jsUp/JM/F2mkvePqG9Z4CDbr8V033jgUuDdwPm2JwBI+jbwV0mLAG8CX7Z9h6SjKetmLwB395DdSZI0QHZOqycLB5LGAFfavqRd+mhgmO1DerP+YcOGecKECb1ZRZK0BJIm2h7W1XI5JZgkSZK0BDklmCw02B5dJ30MZaNGkiTzMTnCSpIkSVqCdFhJkiRJS5AOK0mSJGkJcpdgkvQRkl5izu++WpEVgWf724h5IO3vX6r2r257pa4qyE0XSdJ3PNidrbzzC5ImpP39R9qfU4JJkiRJi5AOK0mSJGkJ0mElSd9xZn8bMI+k/f3LQm9/brpIkiRJWoIcYSVJkiQtQTqsJEmSpCVIh5UkvYyknSU9KOlvko7qb3u6iqSzJT0t6Z7+tqU7SHqPpBsk3SfpXkmH9bdNXUHSEpLGS5oS9h/T3zZ1B0kDJE2WdGV3daTDSpJeRNIA4JfAh4H1gb0lrd+/VnWZMcDO/W3EPPAW8P9srw+8H/hyi12D14HtbW8CDAV2lvT+/jWpWxwG3D8vCtJhJUnvsgXwN9uP2H4D+B3wsX62qUvYvhn4T3/b0V1sP2l7Uhy/RLlprtq/VjWPCzPjdLH4tNRuOUnvBnYBzpoXPemwkqR3WRV4vHL+T1roZrmgIakN2JQSibpliOm0u4GngbG2W8p+4CTg68DseVGSDitJkoUCSQOBS4Gv2n6xv+3pCrZn2R4KvBvYQtKG/WxS00jaFXja9sR51ZUOK0l6lyeA91TO3x1pSR8iaTGKs7rA9mX9bU93sf0CcAOttaa4NbCbpOmUKfHtJZ3fHUXpsJKkd7kLWEfSGpLeAXwKuKKfbVqokCTg18D9tn/W3/Z0FUkrSVoujpcEdgQe6FejuoDt/7X9btttlO//9bY/0x1d6bCSpBex/RZwCHANZbH/97bv7V+ruoakC4HbgfUk/VPS5/vbpi6yNbAv5cn+7vh8pL+N6gKDgRskTaU8AI213e2t4a1MvpopSZIkaQlyhJUkSZK0BOmwkiRJkpYgHVaSJEnSEqTDSpIkSVqCdFhJkiRJS5AOK0mSPkPS/0j6naS/S5oo6WpJ6/Zh/aMlrVInb0hseZ8saS1JtzWhb2ad9DGS9pxXe5O5SYeVJEmfED/gvRy40fZatjcH/hdYucnyizY6b5LRQIcOC9gduMT2prb/bnurbuhPepF0WEmS9BXbAW/aPr2WYHuK7XEqnCDpHknTJI0CkDRS0jhJVwD3dXA+IMrdJWmqpC/WdEv6RuiaIun4GPEMAy6IkdSSFdmPAF8FDpZ0Q6TNrOR/rVLH2+JRhf2nRNyza4F3VfKOj1hcUyX9tMd6cyGkO08oSZIk3WFDoN4LUD9OifW0CbAicJekmyNvM2BD249KGtnu/EBghu3hkhYHbpX0V2AIJYzL+2y/ImkF2/+RdAhwpO0J1cptXy3pdGCm7bmciqSdgHUooWIEXCFp2wi7UmMPYD1KzLOVgfuAsyW9M/KG2HbtFUtJ90iHlSTJ/MAI4ELbs4CnJN0EDAdeBMbbfrQiWz3fCdi4sl40iOJcdgB+Y/sVANvzEs9rp/hMjvOBUUfVYW1bsf9fkq6P9BnAa8CvI9LuQvlKpZ4iHVaSJH3FvUB3NiK83OBcwKG2r6kKSPpQN+qph4Af2T6jqwVtvyVpC+CDlLYfAmzfg7YtVOQaVpIkfcX1wOIxjQeApI0lbQOMA0bFmtRKlBHL+CZ0XkNZd1os9K0raWlgLLC/pKUifYWQfwlYpot2XwN8LuJpIWlVSe9qJ3Nzxf7BlPW6WgyuQbavBg6nTHkm3SRHWEmS9AmxhrMHcJKkb1CmyqZTNjvcAmwJTKGEf/+67X9LGtKJ2rOANmBS7EJ8Btjd9l8kDQUmSHoDuBr4JjAGOF3Sq8CWtl9twu6/SnovcHupgpnAZyjRf2tcThk53Qf8g/J2eyjO8Y+SlqCM1I7orL6kPvm29iRJkqQlyCnBJEmSpCVIh5UkSZK0BOmwkiRJkpYgHVaSJEnSEqTDSpIkSVqCdFhJkiRJS5AOK0mSJGkJ/j+vw1Hbi02SIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data = plot_data.sort_values(by=\"citoid_success\")\n",
    "ax = plot_data.plot.barh(legend=False)\n",
    "\n",
    "ax.set_title(\"Citoid success for top source types\")\n",
    "ax.set_ylabel(\"Top source types\")\n",
    "ax.set_xlabel(\"Correct fields\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"plots/citoid_success_sourcetypes.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Citoid's performance varies according to the source type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}