notebooks/Coronavirus_by_country.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison of the Coronavirus outbreak among countries\n",
    "\n",
    "Sunday, March 08, 2020\n",
    "\n",
    "This notebook is intended to check what is the oubreak evolution in each country, and to some extent to forecast what may be the next steps: some countries were exposed after some others, and **we have the opportunity to see the path they followed and to compare it to the countries that are now exposed.**\n",
    "\n",
    "The notebook is using **ECDC data** (https://www.ecdc.europa.eu). Unfortunately the data is **lagging roughly 2 days**, so you may have to add your own data points if you want to have realtime graphics.\n",
    "\n",
    "China, and now Italy are providing statistics that can be used for this purpose.\n",
    "This notebook shows that France is roughly following the same path 9 days later, and the USA too.\n",
    "**We are using the number of deaths to study the outbreak, as those numbers may prove more precise than the case numbers, which are subject to different policies among countries.**\n",
    "\n",
    "What we can note is that the **initial natural trend is doubling of the deaths every 2 days**. This is very fast, as some pandemics show that this doubling is usually observed every 6 days.\n",
    "**China deaths curve show a large slowing down of the outbreak after this initial phase, that may related to a very strong policy to contain it.**\n",
    "**Italy shows no slowdown since the beginning**, we are still in this early phase. The next days will show if the curve follows China curve, or if the exponential growth will go on for more days, especially that a full lockdown has been declared in the country.\n",
    "It's probably too early for France and USA to conclude, but the trend is very similar right now.\n",
    "\n",
    "**Following graphs are using logarithmic scale, because that's the right thing to do when processing data about an exponential process. That should not hide the fact that the growth is much more severe that what those graphs seems to convey.**\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'urllib' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-18ce6380e5dd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 68\u001b[0;31m \u001b[0mprepare_data_from_ecdc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-1-18ce6380e5dd>\u001b[0m in \u001b[0;36mprepare_data_from_ecdc\u001b[0;34m()\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0mdate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"2020-03-15\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m         \u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-geographic-disbtribution-worldwide-%s.xls\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mdate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0murllib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m         \u001b[0mfilename\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"data/ecdc/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mdate\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\".xls\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"wb\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'urllib' is not defined"
     ]
    }
   ],
   "source": [
    "# import urllib\n",
    "import datetime\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pathlib\n",
    "import math\n",
    "import json\n",
    "\n",
    "%matplotlib notebook\n",
    "\n",
    "country_data = None\n",
    "\n",
    "def prepare_data_from_ecdc():\n",
    "    global country_data\n",
    "    if country_data == None:\n",
    "        country_data = {}\n",
    "\n",
    "        from pandas import DataFrame, read_csv\n",
    "        import matplotlib.pyplot as plt\n",
    "        import pandas as pd \n",
    "\n",
    "        date = \"2020-03-15\"\n",
    "        url = \"https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-geographic-disbtribution-worldwide-%s.xls\" % date        \n",
    "        data = urllib.request.urlopen(url).read()    \n",
    "        filename = \"data/ecdc/\" + date + \".xls\"\n",
    "        f = open(filename, \"wb\")\n",
    "        f.write(data)\n",
    "        f.close()\n",
    "\n",
    "        df = pd.read_excel(filename)\n",
    "        for i in range(df.shape[0]):\n",
    "            country = str(df[\"CountryExp\"][i])\n",
    "            date = str(df[\"DateRep\"][i])\n",
    "            cases = int(df[\"NewConfCases\"][i])\n",
    "            try:\n",
    "                deaths = int(df[\"NewDeaths\"][i])\n",
    "            except Exception as e:\n",
    "                deaths = 0\n",
    "#                print(country, cases, date, df[\"NewDeaths\"][i])\n",
    "\n",
    "            if country not in country_data:\n",
    "                country_data[country] = []\n",
    "\n",
    "            entry = {\"date\":date, \"cases\":cases, \"deaths\":deaths}\n",
    "            country_data[country].append(entry)\n",
    "        \n",
    "        for country, data in country_data.items():\n",
    "            data.sort(key = lambda x : x[\"date\"])\n",
    "\n",
    "            cases = 0\n",
    "            deaths = 0\n",
    "            for d in data:\n",
    "                cases += d[\"cases\"]\n",
    "                deaths += d[\"deaths\"]\n",
    "                d[\"cases\"] = cases\n",
    "                d[\"deaths\"] = deaths\n",
    "            \n",
    "def get_data_from_ecdc(country):\n",
    "    global country_data\n",
    "    prepare_data_from_ecdc()\n",
    "    return country_data[country]\n",
    "\n",
    "def get_data_per_country(country, reset = False):\n",
    "    return get_data_from_ecdc(country)\n",
    "        \n",
    "    \n",
    "\n",
    "prepare_data_from_ecdc()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wikipedia = False\n",
    "if wikipedia:\n",
    "    countries = [\"China\", \"France\", \"South_Korea\", \"Italy\", \"United States\", \"Germany\", \"Iran\", \"Spain\"]\n",
    "    # Update Data from wikipedia:\n",
    "    update = False\n",
    "    if update:\n",
    "        for c in countries:\n",
    "            get_data_from_wikipedia(c, reset = True)\n",
    "\n",
    "\n",
    "china_offset= 37\n",
    "offset_by_country = dict(China=0,\n",
    "                         France=-9.5 - china_offset, \n",
    "                         Italy=0 - china_offset, \n",
    "                         Germany=-9 - china_offset, \n",
    "                         United_States_of_America=-8 - china_offset,\n",
    "                         South_Korea=2 - china_offset,\n",
    "                         Iran=2 - china_offset,\n",
    "                         Japan=30 - china_offset,\n",
    "                         Spain=-9 - china_offset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_date_values(country, values, offset, multiplier, log_axis, label, key):\n",
    "    start_time = datetime.datetime(2020, 1, 1)\n",
    "    dates = [(datetime.datetime.fromisoformat(v[\"date\"]) - start_time).days  + offset for v in values]\n",
    "    y = [v[key] * multiplier for v in values]\n",
    "    if log_axis:\n",
    "        y = [a for a in y]\n",
    "    if log_axis:\n",
    "        plt.yscale(\"log\")    \n",
    "    \n",
    "    plt.plot(dates, y, label = label + \" %d days\" % offset)\n",
    "    \n",
    "def finalize_plot_comparison(key, start, end, reference_shift):\n",
    "    time_base = 2.2\n",
    "    plt.plot([max(1, 2**((x - reference_shift) / time_base)) for x in range(0, 40)],\n",
    "             label=\"Doubling every %s days\" % time_base,\n",
    "            linestyle='dotted')\n",
    "\n",
    "    _ = plt.legend()\n",
    "    plt.title(\"%s by country over time.\\n Logarithmic scale.\\n X axis is custom shifted for each country to take into account different start time.\" % key.capitalize(),\n",
    "            fontdict = {'fontweight' : \"bold\", 'verticalalignment': 'baseline'})\n",
    "    plt.xlabel(\"Days since 01-01-2020 (for China, use offset for other countries)\")\n",
    "    plt.ylabel(key)\n",
    "    plt.xlim(start, end)\n",
    "\n",
    "def get_offset_by_country(country):\n",
    "    return offset_by_country[country.replace(\" \", \"_\")]\n",
    "    \n",
    "def plot_comparison(countries, key, start = 8, end = 78, reference_shift = 12.5):    \n",
    "    \"\"\"Key parameters is 'cases' or 'deaths'\"\"\"\n",
    "    log_axis = True\n",
    "    \n",
    "    # This dictionary can be used to correct for different countries testing policies\n",
    "    multiplier_by_country = dict()\n",
    "    default_multiplier = 1\n",
    "    \n",
    "    fig=plt.figure(figsize=(12, 8), dpi= 80, facecolor='w', edgecolor='k')\n",
    "\n",
    "    for i, country in enumerate(countries):\n",
    "\n",
    "        data_for_country = get_data_per_country(country)\n",
    "        plot_date_values(country,\n",
    "                         data_for_country,\n",
    "                         get_offset_by_country(country),\n",
    "                         multiplier_by_country.get(country, default_multiplier),\n",
    "                         log_axis, country, key)\n",
    "    finalize_plot_comparison(key, start, end, reference_shift)\n",
    "    \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = [\"China\", \"South Korea\", \"Italy\", \"France\", \"Spain\"] #, \"Germany\", \"Iran\", \"Spain\", ]\n",
    "plot_comparison(countries, \"deaths\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes on number of deaths\n",
    "\n",
    "## Time offsets\n",
    "China obviously came first.\n",
    "Italy and South Korea almost at the same time, 36 days later.\n",
    "Then France and USA are roughly 9 days behind Italy and South Korea.\n",
    "It's important to note this, as you can tell somehow what will happen to you by looking at the previous countries.\n",
    "\n",
    "## South Korea\n",
    "From the graphics, it looks like *South Korea* is doing quite well to limit the number of deaths.\n",
    "It looks like too that there was at least two steps in the growth, one stabilizing at 12 deaths, and a second growth stabilizing at 44.\n",
    "\n",
    "## Italy\n",
    "The curve for *Italy* is much more worrying : no inflection is visible right now, the growth is still exponential.\n",
    "\n",
    "## United States\n",
    "It's a bit early to be sure of anything, but the curve clearly follow an exponential growth right now\n",
    "\n",
    "## France\n",
    "A first cluster appeared early, explaining the early start, and a second wave has started. Right now it's difficult to judge of the trend.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_comparison(countries, \"cases\", reference_shift = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes on numbers of cases\n",
    "\n",
    "We are using there the same time shifts we used for the deaths.\n",
    "It's apparent that the different curves are slightly separated (but on a logarithmic scale it can means a bit more).\n",
    "My interpretation is that the testing policies of the different countries can be different, so you may have to compensate for this (South Korea and Italy did a lot of tests).\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}