notebooks/02_python_data_science_on_the_cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NEq8TAAeEYCx"
   },
   "source": [
    "## **Python for Data Analysis on the Cloud**\n",
    "\n",
    "# Get Started with Jupyter & Colab notebooks\n",
    "\n",
    "In this course, we will use the Jupyter notebook on the cloud.\n",
    "\n",
    "[The Jupyter notebook](https://jupyter.org) is an open-source web application that allows you to create and share documents that contain live code, equations, visualisations, and narrative text. \n",
    "\n",
    "There are many free services that allow you to run Jupyter notebooks on the cloud. We will use Colab and Binder.  \n",
    "\n",
    "[Colab](https://colab.research.google.com/notebooks/intro.ipynb#recent=true) is a free environment that runs Jupyter notebooks on the Google Cloud and requires no install or setup. You can view notebooks shared publicly without a Google sign-in. In order to execute and change code interactively, a Google account sign-in is required. You can find more information on how to create a Google account [here](https://support.google.com/accounts/answer/27441?hl=en). You can learn more about the Colab and how to open a new notebook [here](https://colab.research.google.com/notebooks/intro.ipynb#recent=true).\n",
    "\n",
    "[MyBinder](https://mybinder.org) is a free, community-led infrastructure that opens Jupyter notebooks in an interactive and reproducible environment and requires no install or setup. Binder requires no account registration or log-in to execute and change code interactively.\n",
    "\n",
    "<img src=\"https://jupyter.org/assets/homepage/jupyterpreview.webp\" align=\"left\" width=\"400\" height=\"200\"/>\n",
    "<img src=\"https://lh3.googleusercontent.com/edryGqlH18ng3ff18ffCbHuLEqfWpSJuO-1sejiV6t3ZYuUEpLYD4nqD1oha8yI4K4c9y1lqaseOckOrG7wZ-M4vaQ=w640-h400-e365-rj-sc0x00ffffff\" align=\"middle\" width=\"400\" height=\"200\"/>\n",
    "\n",
    "<br>\n",
    "\n",
    "<br>\n",
    "\n",
    "## Learning resources\n",
    "\n",
    "<i class=\"fas fa-code\"></i> [The Jupyter Notebook.](https://jupyter-notebook.readthedocs.io/en/stable/index.html) Jupyter Team.\n",
    "\n",
    "<i class=\"fas fa-code\"></i> [Jupyter Notebook Tutorial](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook). DataCamp. \n",
    "\n",
    "<i class=\"fas fa-play-circle\"></i> <i class=\"fas fa-code\"></i> [Get started with Google Colaboratory](https://www.youtube.com/watch?v=inN8seMm7UI). Jake VanderPlas, Coding TensorFlow.\n",
    "\n",
    "<i class=\"fas fa-play-circle\"></i> <i class=\"fas fa-code\"></i> [A hands-on introduction to Python for beginning programmers.](https://www.youtube.com/watch?v=MirG-vJOg04&t=1029s) Jessica McKellar, PyCon 2014.\n",
    "\n",
    "<i class=\"fas fa-book\"></i> <i class=\"fas fa-code\"></i> Charles Severance. [Python for Everybody: Exploring Data In Python 3.](https://www.py4e.com/book)\n",
    "\n",
    "<i class=\"fas fa-book\"></i> David Amos et al. [Python Basics: A Practical Introduction to Python.](https://realpython.com/products/python-basics-book/) The authors make available [sample chapters](https://static.realpython.com/python-basics-sample-chapters.pdf).  \n",
    "\n",
    "<i class=\"fas fa-play-circle\"></i> [Getting Started With the Open Science Framework (OSF)](https://youtu.be/2TV21gOzfhw). Center for Open Science.\n",
    "\n",
    "<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tN_tPZSA-28a"
   },
   "source": [
    "### Structure of Colab notebook\n",
    "The `Colab notebook` has a simple structure that consists of three parts:\n",
    "* Menubar\n",
    "* Toolbar\n",
    "* Cells\n",
    "\n",
    "For an example of those features, see an example Colab notebook [*Overview of Colaboratory Features*](https://colab.research.google.com/notebooks/basic_features_overview.ipynb) below.\n",
    "\n",
    "<img src=\"https://github.com/valdanchev/reproducible-data-science-python/blob/master/images/Colab_example.png?raw=true\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6PHCgWmVKmL_"
   },
   "source": [
    "### Manage Jupyter Notebook Files\n",
    "To open a new notebook:\n",
    "\n",
    "```\n",
    "File -> New notebook\n",
    "```\n",
    "To rename a notebook: \n",
    "```\n",
    "File -> Rename notebook\n",
    "```\n",
    "To access revision history:\n",
    "```\n",
    "File -> Revision history\n",
    "```\n",
    "To check whether the cells in your notebook are executable in linear order:\n",
    "```\n",
    "Runtime -> Restart and run all\n",
    "``` \n",
    "To share your Colab notebook and collaborate, you can create a shareable link by clicking the `Share` button at the top right of your Colab notebook and specify the type of access (e.g., Anyone with the link can view the notebook). You can then copy and share the link with your collaborators."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6VXxu00Lnca1"
   },
   "source": [
    "### Cells \n",
    "The notebook has two types of cells: code and text. You can add new cells by using the `+ Code` and `+ Text` buttons that are in the toolbar above the notebook and also appear when you hover between a pair of cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3k6AgkukRz1Y"
   },
   "source": [
    "# Python Coding for Data Analysis\n",
    "\n",
    "Below is a code cell, in which we type in the arithmetic expression `21 + 21`. \n",
    "\n",
    "The code is prefixed by a comment. Commenting your code is a good practice and part of your reproducible workflow. Comments in Python’s code cells start with a hashtag symbol # followed by a single space and some text. The text that follows the hashtag symbol on the same line is marked as a comment and is not therefore evaluated by the Python interpreter. Only the code (in this instance, “21 + 21”) is evaluated and the output (in this instance, “42”) will be displayed below the code cell.\n",
    "\n",
    "To execute the cell, press `Shift` + `Enter` or click the Play icon on the left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s_4RUyc6RcyZ",
    "outputId": "3eb126ce-9b73-48c5-fa78-a1366bbe5f2c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform a basic arithmetic operation of addition\n",
    "21 + 21"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aKpEc79sSRGc"
   },
   "source": [
    "Python _reads_ the code entered in the cell, _evaluates_ it, and _prints_ the result (42)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L2bHJ3659WhS"
   },
   "source": [
    "### Create a toy data set and perform basic data analysis\n",
    "Let's create a list of the whole numbers (or integers) `4, 2, 8, 6`. \n",
    "\n",
    "Lists are one of the [built-in data types](https://docs.python.org/3/tutorial/introduction.html#lists) in Python. Elements in a list are separated by comma `,` and are enclosed in square brackets `[]`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "w2nRUOALMbgx",
    "outputId": "38c67715-9061-4770-e56e-68fb8779abd2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 2, 8, 6]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 2, 8, 6]  # create a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The comment `# create a list` is an example of an inline comment. Inline comments refers to a code statement on the same line. Inline comments are separated by at least two spaces from the code statement. Similar to block comments, inline comments start with a hashtag symbol # followed by a single space and some text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YxRYhGclwR43"
   },
   "source": [
    "Let's assign the list of numbers to a variable called `even_numbers` using the =, which is called the assignment operator.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "V8U7mVCcab3t"
   },
   "outputs": [],
   "source": [
    "even_numbers = [4, 2, 8, 6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u99-72Djw2Vy"
   },
   "source": [
    "You can now apply built-in functions from the [Python Standard Library](https://docs.python.org/3/library/) to the variable `even_numbers`. A function is a block of code that:\n",
    "* takes input parameters  \n",
    "* performs a specific task\n",
    "* returns an output.\n",
    "\n",
    "Python has various built-in functions, including `min()`, `max()`, `sorted()`. Take the function `min()`. Using our example of even numbers above, the function `min()` will take as an input parameter the four numbers to compare, perform the comparison, and return the number with the lowest value.\n",
    "\n",
    "We call a function by writing the function name followed by parenthesis. The function `min()` takes only one input parameter, the input data in the form of a list or another Python data type we will discuss later in the course. When we call the function, we pass our list `even_numbers` as an argument inside the parentheses.\n",
    "\n",
    "An argument is different from a parameter, although the two terms are often used interchangeably. A parameter is the generic variable listed inside the parentheses of a function, whereas an argument is the actual value or data you pass to the function when you call it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ISD44V-ox3iu",
    "outputId": "bbb9f473-4421-4932-ecfe-b69210a14fd3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the number with the lowest value\n",
    "min(even_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pFNIOO1NSRx9"
   },
   "source": [
    "We can apply any other built-in Python function. For example, the `max()` function returns, intuitively, the number with the highest value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ourS3pE-yAeT",
    "outputId": "2c7288ed-4922-49cf-9e62-c73c20b47fff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the number with the highest value\n",
    "max(even_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yxV9tY0MzCjL"
   },
   "source": [
    "The function `sorted()` returns a sorted list of numbers in increasing order. We assign the resulting sorted list to a variable named `sorted_even_numbers`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QOnnJCOzyiPN",
    "outputId": "c9835a17-b734-483f-b880-29555b7ac7ab"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 4, 6, 8]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sort numbers in increasing order\n",
    "sorted_even_numbers = sorted(even_numbers)\n",
    "sorted_even_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kwQB_andIf5z"
   },
   "source": [
    "Functions often have multiple parameters. For example, in addition to your input data, the `sorted()` function takes two optional parameters. One of these parameters is `reverse`. If you type in `reverse = True`, the list will be sorted in descending order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8, 6, 4, 2]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(even_numbers, reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our list of even numbers is now sorted in descending order. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q1qlC2Ds0xQI"
   },
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/3/31/NumPy_logo_2020.svg\" width=\"150\" height=\"100\" align = \"right\">\n",
    "\n",
    "# Descriptive statistics with `NumPy` \n",
    "\n",
    "Functions for scientific computing, data analysis, machine learning, and statistical modeling are not in the Python Standard Library but are part of Python libraries or packages. For example, the Python library [`NumPy`](https://numpy.org) for scientific computing includes functions for computing the mean (average of the numbers) and standard deviation (variation or dispersion of the numbers from the mean).              \n",
    "\n",
    "To use `NumPy`, we first import the library and, by convention, give it the alias `np`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "r8vqdXJR0w4G"
   },
   "outputs": [],
   "source": [
    "# Import the NumPy library\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we imported the module NumPy as `np`, you can view help on the module by typing the module alias followed by a question mark ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mType:\u001b[0m        module\n",
       "\u001b[0;31mString form:\u001b[0m <module 'numpy' from '/Users/valentindanchev/opt/anaconda3/lib/python3.8/site-packages/numpy/__init__.py'>\n",
       "\u001b[0;31mFile:\u001b[0m        ~/opt/anaconda3/lib/python3.8/site-packages/numpy/__init__.py\n",
       "\u001b[0;31mDocstring:\u001b[0m  \n",
       "NumPy\n",
       "=====\n",
       "\n",
       "Provides\n",
       "  1. An array object of arbitrary homogeneous items\n",
       "  2. Fast mathematical operations over arrays\n",
       "  3. Linear Algebra, Fourier Transforms, Random Number Generation\n",
       "\n",
       "How to use the documentation\n",
       "----------------------------\n",
       "Documentation is available in two forms: docstrings provided\n",
       "with the code, and a loose standing reference guide, available from\n",
       "`the NumPy homepage <https://www.scipy.org>`_.\n",
       "\n",
       "We recommend exploring the docstrings using\n",
       "`IPython <https://ipython.org>`_, an advanced Python shell with\n",
       "TAB-completion and introspection capabilities.  See below for further\n",
       "instructions.\n",
       "\n",
       "The docstring examples assume that `numpy` has been imported as `np`::\n",
       "\n",
       "  >>> import numpy as np\n",
       "\n",
       "Code snippets are indicated by three greater-than signs::\n",
       "\n",
       "  >>> x = 42\n",
       "  >>> x = x + 1\n",
       "\n",
       "Use the built-in ``help`` function to view a function's docstring::\n",
       "\n",
       "  >>> help(np.sort)\n",
       "  ... # doctest: +SKIP\n",
       "\n",
       "For some objects, ``np.info(obj)`` may provide additional help.  This is\n",
       "particularly true if you see the line \"Help on ufunc object:\" at the top\n",
       "of the help() page.  Ufuncs are implemented in C, not Python, for speed.\n",
       "The native Python help() does not know how to view their help, but our\n",
       "np.info() function does.\n",
       "\n",
       "To search for documents containing a keyword, do::\n",
       "\n",
       "  >>> np.lookfor('keyword')\n",
       "  ... # doctest: +SKIP\n",
       "\n",
       "General-purpose documents like a glossary and help on the basic concepts\n",
       "of numpy are available under the ``doc`` sub-module::\n",
       "\n",
       "  >>> from numpy import doc\n",
       "  >>> help(doc)\n",
       "  ... # doctest: +SKIP\n",
       "\n",
       "Available subpackages\n",
       "---------------------\n",
       "doc\n",
       "    Topical documentation on broadcasting, indexing, etc.\n",
       "lib\n",
       "    Basic functions used by several sub-packages.\n",
       "random\n",
       "    Core Random Tools\n",
       "linalg\n",
       "    Core Linear Algebra Tools\n",
       "fft\n",
       "    Core FFT routines\n",
       "polynomial\n",
       "    Polynomial tools\n",
       "testing\n",
       "    NumPy testing tools\n",
       "f2py\n",
       "    Fortran to Python Interface Generator.\n",
       "distutils\n",
       "    Enhancements to distutils with support for\n",
       "    Fortran compilers support and more.\n",
       "\n",
       "Utilities\n",
       "---------\n",
       "test\n",
       "    Run numpy unittests\n",
       "show_config\n",
       "    Show numpy build configuration\n",
       "dual\n",
       "    Overwrite certain functions with high-performance Scipy tools\n",
       "matlib\n",
       "    Make everything matrices.\n",
       "__version__\n",
       "    NumPy version string\n",
       "\n",
       "Viewing documentation using IPython\n",
       "-----------------------------------\n",
       "Start IPython with the NumPy profile (``ipython -p numpy``), which will\n",
       "import `numpy` under the alias `np`.  Then, use the ``cpaste`` command to\n",
       "paste examples into the shell.  To see which functions are available in\n",
       "`numpy`, type ``np.<TAB>`` (where ``<TAB>`` refers to the TAB key), or use\n",
       "``np.*cos*?<ENTER>`` (where ``<ENTER>`` refers to the ENTER key) to narrow\n",
       "down the list.  To view the docstring for a function, use\n",
       "``np.cos?<ENTER>`` (to view the docstring) and ``np.cos??<ENTER>`` (to view\n",
       "the source code).\n",
       "\n",
       "Copies vs. in-place operation\n",
       "-----------------------------\n",
       "Most of the functions in `numpy` return a copy of the array argument\n",
       "(e.g., `np.sort`).  In-place versions of these functions are often\n",
       "available as array methods, i.e. ``x = np.array([1,2,3]); x.sort()``.\n",
       "Exceptions to this rule are documented.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dyFKPve2AKxD"
   },
   "source": [
    "We now use `np.` to append each function we will use from NumPy, for example the functions `mean()` and `std`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "v6Tm_QyH5X6e",
    "outputId": "1bd9e05c-ea17-48be-9dfc-9817ed421299"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the mean of our list of numbers\n",
    "np.mean(even_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VdHH0nye0W4a",
    "outputId": "69f6ae57-04f8-487e-c208-dacd06818e3b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.23606797749979"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the standard deviation of our list of numbers\n",
    "np.std(even_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting help\n",
    "\n",
    "In Jupyter and Colab, you can access help information by using the `help()` function or a question mark `?`. For example, to access help information about a function in the Python Standard Library, such as `min()`, you type in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on built-in function min in module builtins:\n",
      "\n",
      "min(...)\n",
      "    min(iterable, *[, default=obj, key=func]) -> value\n",
      "    min(arg1, arg2, *args, *[, key=func]) -> value\n",
      "    \n",
      "    With a single iterable argument, return its smallest item. The\n",
      "    default keyword-only argument specifies an object to return if\n",
      "    the provided iterable is empty.\n",
      "    With two or more arguments, return the smallest argument.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "min(iterable, *[, default=obj, key=func]) -> value\n",
       "min(arg1, arg2, *args, *[, key=func]) -> value\n",
       "\n",
       "With a single iterable argument, return its smallest item. The\n",
       "default keyword-only argument specifies an object to return if\n",
       "the provided iterable is empty.\n",
       "With two or more arguments, return the smallest argument.\n",
       "\u001b[0;31mType:\u001b[0m      builtin_function_or_method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "help(min)\n",
    "\n",
    "# Alternatively, you can symply type\n",
    "min?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ":::{note}\n",
    "In Jupyter, you access help by pressing `Shift` + `Tab` when you are typing in a cell in `edit` mode. See this [tutorial](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/) for the difference between `edit` mode and `command` mode.\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true,
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function mean in module numpy:\n",
      "\n",
      "mean(a, axis=None, dtype=None, out=None, keepdims=<no value>)\n",
      "    Compute the arithmetic mean along the specified axis.\n",
      "    \n",
      "    Returns the average of the array elements.  The average is taken over\n",
      "    the flattened array by default, otherwise over the specified axis.\n",
      "    `float64` intermediate and return values are used for integer inputs.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    a : array_like\n",
      "        Array containing numbers whose mean is desired. If `a` is not an\n",
      "        array, a conversion is attempted.\n",
      "    axis : None or int or tuple of ints, optional\n",
      "        Axis or axes along which the means are computed. The default is to\n",
      "        compute the mean of the flattened array.\n",
      "    \n",
      "        .. versionadded:: 1.7.0\n",
      "    \n",
      "        If this is a tuple of ints, a mean is performed over multiple axes,\n",
      "        instead of a single axis or all the axes as before.\n",
      "    dtype : data-type, optional\n",
      "        Type to use in computing the mean.  For integer inputs, the default\n",
      "        is `float64`; for floating point inputs, it is the same as the\n",
      "        input dtype.\n",
      "    out : ndarray, optional\n",
      "        Alternate output array in which to place the result.  The default\n",
      "        is ``None``; if provided, it must have the same shape as the\n",
      "        expected output, but the type will be cast if necessary.\n",
      "        See `ufuncs-output-type` for more details.\n",
      "    \n",
      "    keepdims : bool, optional\n",
      "        If this is set to True, the axes which are reduced are left\n",
      "        in the result as dimensions with size one. With this option,\n",
      "        the result will broadcast correctly against the input array.\n",
      "    \n",
      "        If the default value is passed, then `keepdims` will not be\n",
      "        passed through to the `mean` method of sub-classes of\n",
      "        `ndarray`, however any non-default value will be.  If the\n",
      "        sub-class' method does not implement `keepdims` any\n",
      "        exceptions will be raised.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    m : ndarray, see dtype parameter above\n",
      "        If `out=None`, returns a new array containing the mean values,\n",
      "        otherwise a reference to the output array is returned.\n",
      "    \n",
      "    See Also\n",
      "    --------\n",
      "    average : Weighted average\n",
      "    std, var, nanmean, nanstd, nanvar\n",
      "    \n",
      "    Notes\n",
      "    -----\n",
      "    The arithmetic mean is the sum of the elements along the axis divided\n",
      "    by the number of elements.\n",
      "    \n",
      "    Note that for floating-point input, the mean is computed using the\n",
      "    same precision the input has.  Depending on the input data, this can\n",
      "    cause the results to be inaccurate, especially for `float32` (see\n",
      "    example below).  Specifying a higher-precision accumulator using the\n",
      "    `dtype` keyword can alleviate this issue.\n",
      "    \n",
      "    By default, `float16` results are computed using `float32` intermediates\n",
      "    for extra precision.\n",
      "    \n",
      "    Examples\n",
      "    --------\n",
      "    >>> a = np.array([[1, 2], [3, 4]])\n",
      "    >>> np.mean(a)\n",
      "    2.5\n",
      "    >>> np.mean(a, axis=0)\n",
      "    array([2., 3.])\n",
      "    >>> np.mean(a, axis=1)\n",
      "    array([1.5, 3.5])\n",
      "    \n",
      "    In single precision, `mean` can be inaccurate:\n",
      "    \n",
      "    >>> a = np.zeros((2, 512*512), dtype=np.float32)\n",
      "    >>> a[0, :] = 1.0\n",
      "    >>> a[1, :] = 0.1\n",
      "    >>> np.mean(a)\n",
      "    0.54999924\n",
      "    \n",
      "    Computing the mean in float64 is more accurate:\n",
      "    \n",
      "    >>> np.mean(a, dtype=np.float64)\n",
      "    0.55000000074505806 # may vary\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get help about the NumPy function mean()\n",
    "help(np.mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Readability of your Python code\n",
    "\n",
    "To write readable, consistent, and clean code, it is important to consult the [PEP (Python Enhancement Proposal) 8 style guide for Python code](https://peps.python.org/pep-0008/). The guide describes the rules for writing a readable Python code. Below are outlined key rules to keep in mind when you write your Python code (for details and examples, check out the [PEP 8 style guide](https://peps.python.org/pep-0008/)).\n",
    "\n",
    "* **Naming conventions**\n",
    "    * Variable names (e.g., `even_numbers`) should be lowercase, with words separated by underscores as necessary to improve readability.\n",
    "    * Names to avoid (as, in some fonts, they are indistinguishable from the numerals one and zero): the characters ‘l’ (lowercase letter el), ‘O’ (uppercase letter oh), or ‘I’ (uppercase letter eye) as single character variable names.\n",
    "\n",
    "\n",
    "* **Comments**\n",
    "    * PEP 8 distinguishes block comments, inline comments, and documentation strings (or docstrings for short):\n",
    "        * Block comments in Python's code cells apply to the lines of code that follows them, and follow the same indentation as the code. Block comments are typically formed of complete sentences, each sentence starting with a capitalized word and ending in a period. Each line of a block comment starts with a hashtag symbol `#` followed by a single space and some text.\n",
    "        * Line comments appears on the same line as a statement and is separated by at least two spaces from the statement. Each line comment starts with a hashtag symbol `#` followed by a single space and some text.\n",
    "        * Dockstrings are used to document Python modules, functions, classes, or methods. Docstrings are surrounded by `\"\"\"triple double quotes\"\"\"`. You likely read docstrings more often (for example, when accessing help information about a function) than write them. See [PEP 8](https://peps.python.org/pep-0008/#documentation-strings) for details on docstrings.\n",
    "\n",
    "\n",
    "* **Maximum line length**\n",
    "    * Limit all lines to a maximum of 79 characters. To split a long command over multiple lines, one can break down code into readable statements using parenthesis. Another approach is to use the backslash symbol `\\`. \n",
    "    * Long blocks of text (comments) should be limited to 72 characters. \n",
    "    * When applicable and agreed upon, it would be okay to increase the line length limit up to 99 characters, provided that comments are still limited to 72 characters.\n",
    "    \n",
    "\n",
    "* **Indentation**\n",
    "    * Indentation in Python is the spaces at the beginning of a code line. Indentation is important not only because of code readability but particularly because in Python statements arranged at the same indentation level are considered to form part of a single code block. So, incorrect indentation in Python would likely produce an IndentationError or an errorless output.\n",
    "    * Use 4 spaces per indentation level.\n",
    "    \n",
    "\n",
    "* **Tabs or Spaces?**\n",
    "    * Spaces are the preferred indentation method.\n",
    "    * Python disallows mixing tabs and spaces for indentation.\n",
    "    \n",
    "    \n",
    "* **Whitespace in Expressions and Statements**\n",
    "    * Always surround these binary operators with a single space on either side: assignment (=), augmented assignment (+=, -= etc.), comparisons (==, <, >, !=, <>, <=, >=, in, not in, is, is not), Booleans (and, or, not). (e.g., `sorted_even_numbers = sorted(even_numbers)`)\n",
    "    * Don’t use spaces around the `=` sign when used to indicate a keyword argument (e.g., `reverse=True`)\n",
    "\n",
    "\n",
    "As mentioned in PEP 8, section *A Foolish Consistency is the Hobgoblin of Little Minds*, style guide recommendations may not be applicable in some circumstances. In such circumstances, you could look for examples from the Python community and use your best judgment.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running command-line commands in Jupyter/Colab notebook\n",
    "\n",
    "In addition to running Python code, we will also execute command-line commands in our Jupyter notebooks. Command-line commands are very useful for obtaining information (for example, you can check the version of your Jupyter notebook, Python, or packages), managing computer files, and installing Python packages. \n",
    "\n",
    "To execute command-line commands, you would typically need to use a command-line interface (CLI) such as the Terminal (macOS) or Command Prompt (Windows), which can be challenging. Fortunately,  [Jupyter notebook allows you to run command-line commands in the notebook code cells](https://anaconda.zendesk.com/hc/en-us/articles/360023858254-Executing-Terminal-Commands-in-Jupyter-Notebooks) by prepending an exclamation mark (`!`) to the beginning of the command. Any command appearing after the mark in the line will not be executed from the Python environment but from your operating system's command-line interface (CLI). You can think of the exclamation mark (`!`) as introducing command-line interface (You can learn more about the command-line interfaces from [this tutorial](http://swcarpentry.github.io/shell-novice/) by The Carpentries). \n",
    "\n",
    "As an example, you can determine the version of the Jupyter notebook you use by typing the line below, in which the question mark (`!`) is followed by the command `jupyter-notebook` and the flag `--version`. Command-line flags are used to specify options and modify command's execution. As an output, the command prints the version of your active Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.1.4\n"
     ]
    }
   ],
   "source": [
    "!jupyter-notebook --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing packages with `pip`\n",
    "\n",
    "Command-line tools are particularly useful for installing packages that are not part of the [Python standard library](https://docs.python.org/3/library/). To install packages, we will use the Python's packaging manager `pip`. A list of all Python packages available for instalation via `pip` can be found at https://pypi.org. \n",
    "\n",
    "Because `pip` is a command-line tool, we will prepend an exclamation mark (`!`) to the package installation command every time we use `pip`. The `pip install` command supports flags, for example we will specify the flag `-q` (short for `--quiet`) that hides output/warnings which may cause confusion initially. For example, to install the Python package for statistical modeling, `statsmodels`, we would type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q statsmodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command above installs the most recent version of a package, which you can determine (along other useful information about the package) by typing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: statsmodels\n",
      "Version: 0.12.2\n",
      "Summary: Statistical computations and models for Python\n",
      "Home-page: https://www.statsmodels.org/\n",
      "Author: \n",
      "Author-email: \n",
      "License: BSD License\n",
      "Location: /Users/valentindanchev/opt/anaconda3/lib/python3.8/site-packages\n",
      "Requires: numpy, pandas, patsy, scipy\n",
      "Required-by: dowhy\n"
     ]
    }
   ],
   "source": [
    "!pip show statsmodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are interested in installing a previous version of `statsmodels` (for example, for the purpose of reproducing a research project which uses a previous version), you can find the available previous versions at https://pypi.org and install a version of choice by typing the version number, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q statsmodels==0.12.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter magic commands\n",
    "\n",
    "Jupyter *magic commands* are special commands that add to the Python syntax and provide capabilities that help researchers solve various research problems related to data analysis and workflow. Magic commands are prefixed in the Jupyter notebook by a single `%` character when they operate on one line of code (known as line magics) or by double `%%` characters when they operate on multiple lines of code (known as cell magics).\n",
    "\n",
    "You can list all available magic commands by using the line magic `%lsmagic` as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "cell": {
        "!": "OSMagics",
        "HTML": "Other",
        "SVG": "Other",
        "bash": "Other",
        "capture": "ExecutionMagics",
        "debug": "ExecutionMagics",
        "file": "Other",
        "html": "DisplayMagics",
        "javascript": "DisplayMagics",
        "js": "DisplayMagics",
        "latex": "DisplayMagics",
        "markdown": "DisplayMagics",
        "perl": "Other",
        "prun": "ExecutionMagics",
        "pypy": "Other",
        "python": "Other",
        "python2": "Other",
        "python3": "Other",
        "ruby": "Other",
        "script": "ScriptMagics",
        "sh": "Other",
        "svg": "DisplayMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "writefile": "OSMagics"
       },
       "line": {
        "alias": "OSMagics",
        "alias_magic": "BasicMagics",
        "autoawait": "AsyncMagics",
        "autocall": "AutoMagics",
        "automagic": "AutoMagics",
        "autosave": "KernelMagics",
        "bookmark": "OSMagics",
        "cat": "Other",
        "cd": "OSMagics",
        "clear": "KernelMagics",
        "colors": "BasicMagics",
        "conda": "PackagingMagics",
        "config": "ConfigMagics",
        "connect_info": "KernelMagics",
        "cp": "Other",
        "debug": "ExecutionMagics",
        "dhist": "OSMagics",
        "dirs": "OSMagics",
        "doctest_mode": "BasicMagics",
        "ed": "Other",
        "edit": "KernelMagics",
        "env": "OSMagics",
        "gui": "BasicMagics",
        "hist": "Other",
        "history": "HistoryMagics",
        "killbgscripts": "ScriptMagics",
        "ldir": "Other",
        "less": "KernelMagics",
        "lf": "Other",
        "lk": "Other",
        "ll": "Other",
        "load": "CodeMagics",
        "load_ext": "ExtensionMagics",
        "loadpy": "CodeMagics",
        "logoff": "LoggingMagics",
        "logon": "LoggingMagics",
        "logstart": "LoggingMagics",
        "logstate": "LoggingMagics",
        "logstop": "LoggingMagics",
        "ls": "Other",
        "lsmagic": "BasicMagics",
        "lx": "Other",
        "macro": "ExecutionMagics",
        "magic": "BasicMagics",
        "man": "KernelMagics",
        "matplotlib": "PylabMagics",
        "mkdir": "Other",
        "more": "KernelMagics",
        "mv": "Other",
        "notebook": "BasicMagics",
        "page": "BasicMagics",
        "pastebin": "CodeMagics",
        "pdb": "ExecutionMagics",
        "pdef": "NamespaceMagics",
        "pdoc": "NamespaceMagics",
        "pfile": "NamespaceMagics",
        "pinfo": "NamespaceMagics",
        "pinfo2": "NamespaceMagics",
        "pip": "PackagingMagics",
        "popd": "OSMagics",
        "pprint": "BasicMagics",
        "precision": "BasicMagics",
        "prun": "ExecutionMagics",
        "psearch": "NamespaceMagics",
        "psource": "NamespaceMagics",
        "pushd": "OSMagics",
        "pwd": "OSMagics",
        "pycat": "OSMagics",
        "pylab": "PylabMagics",
        "qtconsole": "KernelMagics",
        "quickref": "BasicMagics",
        "recall": "HistoryMagics",
        "rehashx": "OSMagics",
        "reload_ext": "ExtensionMagics",
        "rep": "Other",
        "rerun": "HistoryMagics",
        "reset": "NamespaceMagics",
        "reset_selective": "NamespaceMagics",
        "rm": "Other",
        "rmdir": "Other",
        "run": "ExecutionMagics",
        "save": "CodeMagics",
        "sc": "OSMagics",
        "set_env": "OSMagics",
        "store": "StoreMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "tb": "ExecutionMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "unalias": "OSMagics",
        "unload_ext": "ExtensionMagics",
        "who": "NamespaceMagics",
        "who_ls": "NamespaceMagics",
        "whos": "NamespaceMagics",
        "xdel": "NamespaceMagics",
        "xmode": "BasicMagics"
       }
      },
      "text/plain": [
       "Available line magics:\n",
       "%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode\n",
       "\n",
       "Available cell magics:\n",
       "%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile\n",
       "\n",
       "Automagic is ON, % prefix IS NOT needed for line magics."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%lsmagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the popular magic commands we will use include:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Printing out the execution time of a Python command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 95 µs, sys: 53 µs, total: 148 µs\n",
      "Wall time: 153 µs\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time np.mean(even_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Listing all variables already defined in the current notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "even_numbers\t np\t sorted_even_numbers\t \n"
     ]
    }
   ],
   "source": [
    "%who"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Displaying matlpotlib/Seaborn graphs in the notebook (applies to older versions of Jupyter notebook, obsolete in newer versions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can read more about the Jupyter magic commands [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html#) and [here](https://jakevdp.github.io/PythonDataScienceHandbook/01.03-magic-commands.html).    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://pandas.pydata.org/docs/_static/pandas.svg\" width=\"150\" height=\"100\" align=\"right\">\n",
    "\n",
    "# Loading real-world data with `pandas`\n",
    "\n",
    "So far, we have used a toy data example. We will use the [`pandas`](https://pandas.pydata.org/docs/index.html) library to load a real-world data set. We will learn about pandas next session, here will just use key data loading functionality. Let's first import the `pandas` library and, by convention, give it the alias `pd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "dt7TlsRigs-L"
   },
   "outputs": [],
   "source": [
    "# Import the pandas library\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ourworldindata.org/uploads/2019/02/OurWorldinData-logo.png\" width=\"120\" height=\"80\" align=\"right\">\n",
    "\n",
    "## Data on Covid-19 by Our World in Data\n",
    "\n",
    "We will load and explore the Data on COVID-19 by [Our World in Data (OWID)](https://ourworldindata.org/coronavirus). Details about the data are available on this GitHub [repository](https://github.com/owid/covid-19-data/tree/master/public/data): \n",
    "> _It is updated daily and includes data on confirmed cases, deaths, hospitalizations, testing, and vaccinations as well as other variables of potential interest._\n",
    "\n",
    "The data on COVID-19 by Our World in Data is provided as a comma-separated values (CSV) file. We load the CSV file into Python using the `read_csv()` function from `pandas`. There is no need to download the file on your local computer or the cloud. We just specify the URL and use the code below to load the most recent online version of the data. We also assign the loaded data set to a variable called owid_covid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "owid_covid = pd.read_csv(\"https://covid.ourworldindata.org/data/owid-covid-data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now perform various operations on the data object by using the so called `methods`. Examples of `pandas` methods are `head()` and `tail()`; `head()` displays by default the top five rows of the data and `tail()` displays by default the last five rows. You can display a custom number of row by passing that number in in the brackets.\n",
    "\n",
    "Let's display the top five rows using the method `head()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iso_code</th>\n",
       "      <th>continent</th>\n",
       "      <th>location</th>\n",
       "      <th>date</th>\n",
       "      <th>total_cases</th>\n",
       "      <th>new_cases</th>\n",
       "      <th>new_cases_smoothed</th>\n",
       "      <th>total_deaths</th>\n",
       "      <th>new_deaths</th>\n",
       "      <th>new_deaths_smoothed</th>\n",
       "      <th>...</th>\n",
       "      <th>female_smokers</th>\n",
       "      <th>male_smokers</th>\n",
       "      <th>handwashing_facilities</th>\n",
       "      <th>hospital_beds_per_thousand</th>\n",
       "      <th>life_expectancy</th>\n",
       "      <th>human_development_index</th>\n",
       "      <th>excess_mortality_cumulative_absolute</th>\n",
       "      <th>excess_mortality_cumulative</th>\n",
       "      <th>excess_mortality</th>\n",
       "      <th>excess_mortality_cumulative_per_million</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AFG</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2020-02-24</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.746</td>\n",
       "      <td>0.5</td>\n",
       "      <td>64.83</td>\n",
       "      <td>0.511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AFG</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2020-02-25</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.746</td>\n",
       "      <td>0.5</td>\n",
       "      <td>64.83</td>\n",
       "      <td>0.511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AFG</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2020-02-26</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.746</td>\n",
       "      <td>0.5</td>\n",
       "      <td>64.83</td>\n",
       "      <td>0.511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AFG</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2020-02-27</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.746</td>\n",
       "      <td>0.5</td>\n",
       "      <td>64.83</td>\n",
       "      <td>0.511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AFG</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2020-02-28</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.746</td>\n",
       "      <td>0.5</td>\n",
       "      <td>64.83</td>\n",
       "      <td>0.511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  iso_code continent     location        date  total_cases  new_cases  \\\n",
       "0      AFG      Asia  Afghanistan  2020-02-24          5.0        5.0   \n",
       "1      AFG      Asia  Afghanistan  2020-02-25          5.0        0.0   \n",
       "2      AFG      Asia  Afghanistan  2020-02-26          5.0        0.0   \n",
       "3      AFG      Asia  Afghanistan  2020-02-27          5.0        0.0   \n",
       "4      AFG      Asia  Afghanistan  2020-02-28          5.0        0.0   \n",
       "\n",
       "   new_cases_smoothed  total_deaths  new_deaths  new_deaths_smoothed  ...  \\\n",
       "0                 NaN           NaN         NaN                  NaN  ...   \n",
       "1                 NaN           NaN         NaN                  NaN  ...   \n",
       "2                 NaN           NaN         NaN                  NaN  ...   \n",
       "3                 NaN           NaN         NaN                  NaN  ...   \n",
       "4                 NaN           NaN         NaN                  NaN  ...   \n",
       "\n",
       "   female_smokers  male_smokers  handwashing_facilities  \\\n",
       "0             NaN           NaN                  37.746   \n",
       "1             NaN           NaN                  37.746   \n",
       "2             NaN           NaN                  37.746   \n",
       "3             NaN           NaN                  37.746   \n",
       "4             NaN           NaN                  37.746   \n",
       "\n",
       "   hospital_beds_per_thousand  life_expectancy  human_development_index  \\\n",
       "0                         0.5            64.83                    0.511   \n",
       "1                         0.5            64.83                    0.511   \n",
       "2                         0.5            64.83                    0.511   \n",
       "3                         0.5            64.83                    0.511   \n",
       "4                         0.5            64.83                    0.511   \n",
       "\n",
       "   excess_mortality_cumulative_absolute  excess_mortality_cumulative  \\\n",
       "0                                   NaN                          NaN   \n",
       "1                                   NaN                          NaN   \n",
       "2                                   NaN                          NaN   \n",
       "3                                   NaN                          NaN   \n",
       "4                                   NaN                          NaN   \n",
       "\n",
       "   excess_mortality  excess_mortality_cumulative_per_million  \n",
       "0               NaN                                      NaN  \n",
       "1               NaN                                      NaN  \n",
       "2               NaN                                      NaN  \n",
       "3               NaN                                      NaN  \n",
       "4               NaN                                      NaN  \n",
       "\n",
       "[5 rows x 67 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the top five rows of the data set\n",
    "owid_covid.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iso_code</th>\n",
       "      <th>continent</th>\n",
       "      <th>location</th>\n",
       "      <th>date</th>\n",
       "      <th>total_cases</th>\n",
       "      <th>new_cases</th>\n",
       "      <th>new_cases_smoothed</th>\n",
       "      <th>total_deaths</th>\n",
       "      <th>new_deaths</th>\n",
       "      <th>new_deaths_smoothed</th>\n",
       "      <th>...</th>\n",
       "      <th>female_smokers</th>\n",
       "      <th>male_smokers</th>\n",
       "      <th>handwashing_facilities</th>\n",
       "      <th>hospital_beds_per_thousand</th>\n",
       "      <th>life_expectancy</th>\n",
       "      <th>human_development_index</th>\n",
       "      <th>excess_mortality_cumulative_absolute</th>\n",
       "      <th>excess_mortality_cumulative</th>\n",
       "      <th>excess_mortality</th>\n",
       "      <th>excess_mortality_cumulative_per_million</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>187564</th>\n",
       "      <td>ZWE</td>\n",
       "      <td>Africa</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>2022-05-14</td>\n",
       "      <td>249206.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>141.714</td>\n",
       "      <td>5482.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.286</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>30.7</td>\n",
       "      <td>36.791</td>\n",
       "      <td>1.7</td>\n",
       "      <td>61.49</td>\n",
       "      <td>0.571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187565</th>\n",
       "      <td>ZWE</td>\n",
       "      <td>Africa</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>2022-05-15</td>\n",
       "      <td>249206.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>122.000</td>\n",
       "      <td>5482.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.857</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>30.7</td>\n",
       "      <td>36.791</td>\n",
       "      <td>1.7</td>\n",
       "      <td>61.49</td>\n",
       "      <td>0.571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187566</th>\n",
       "      <td>ZWE</td>\n",
       "      <td>Africa</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>2022-05-16</td>\n",
       "      <td>249431.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>127.857</td>\n",
       "      <td>5484.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.714</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>30.7</td>\n",
       "      <td>36.791</td>\n",
       "      <td>1.7</td>\n",
       "      <td>61.49</td>\n",
       "      <td>0.571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187567</th>\n",
       "      <td>ZWE</td>\n",
       "      <td>Africa</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>2022-05-17</td>\n",
       "      <td>249748.0</td>\n",
       "      <td>317.0</td>\n",
       "      <td>158.000</td>\n",
       "      <td>5484.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.429</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>30.7</td>\n",
       "      <td>36.791</td>\n",
       "      <td>1.7</td>\n",
       "      <td>61.49</td>\n",
       "      <td>0.571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187568</th>\n",
       "      <td>ZWE</td>\n",
       "      <td>Africa</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>2022-05-18</td>\n",
       "      <td>250007.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>195.000</td>\n",
       "      <td>5486.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.714</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>30.7</td>\n",
       "      <td>36.791</td>\n",
       "      <td>1.7</td>\n",
       "      <td>61.49</td>\n",
       "      <td>0.571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       iso_code continent  location        date  total_cases  new_cases  \\\n",
       "187564      ZWE    Africa  Zimbabwe  2022-05-14     249206.0       75.0   \n",
       "187565      ZWE    Africa  Zimbabwe  2022-05-15     249206.0        0.0   \n",
       "187566      ZWE    Africa  Zimbabwe  2022-05-16     249431.0      225.0   \n",
       "187567      ZWE    Africa  Zimbabwe  2022-05-17     249748.0      317.0   \n",
       "187568      ZWE    Africa  Zimbabwe  2022-05-18     250007.0      259.0   \n",
       "\n",
       "        new_cases_smoothed  total_deaths  new_deaths  new_deaths_smoothed  \\\n",
       "187564             141.714        5482.0         0.0                1.286   \n",
       "187565             122.000        5482.0         0.0                0.857   \n",
       "187566             127.857        5484.0         2.0                0.714   \n",
       "187567             158.000        5484.0         0.0                0.429   \n",
       "187568             195.000        5486.0         2.0                0.714   \n",
       "\n",
       "        ...  female_smokers  male_smokers  handwashing_facilities  \\\n",
       "187564  ...             1.6          30.7                  36.791   \n",
       "187565  ...             1.6          30.7                  36.791   \n",
       "187566  ...             1.6          30.7                  36.791   \n",
       "187567  ...             1.6          30.7                  36.791   \n",
       "187568  ...             1.6          30.7                  36.791   \n",
       "\n",
       "        hospital_beds_per_thousand  life_expectancy  human_development_index  \\\n",
       "187564                         1.7            61.49                    0.571   \n",
       "187565                         1.7            61.49                    0.571   \n",
       "187566                         1.7            61.49                    0.571   \n",
       "187567                         1.7            61.49                    0.571   \n",
       "187568                         1.7            61.49                    0.571   \n",
       "\n",
       "        excess_mortality_cumulative_absolute  excess_mortality_cumulative  \\\n",
       "187564                                   NaN                          NaN   \n",
       "187565                                   NaN                          NaN   \n",
       "187566                                   NaN                          NaN   \n",
       "187567                                   NaN                          NaN   \n",
       "187568                                   NaN                          NaN   \n",
       "\n",
       "        excess_mortality  excess_mortality_cumulative_per_million  \n",
       "187564               NaN                                      NaN  \n",
       "187565               NaN                                      NaN  \n",
       "187566               NaN                                      NaN  \n",
       "187567               NaN                                      NaN  \n",
       "187568               NaN                                      NaN  \n",
       "\n",
       "[5 rows x 67 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the last five rows of the data set\n",
    "owid_covid.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Describe the Covid-19 data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xunUR0CxdM0Q"
   },
   "source": [
    "In addition to `pandas` methods, we can use `attributes` to access information about the metadata. For example, the attribute `shape` gives the dimensions of a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MYUJaeFLCeWw",
    "outputId": "d029ac25-328f-44e6-822a-e4f935e74e6a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(187569, 67)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of rows and columns in the data set\n",
    "owid_covid_shape = owid_covid.shape\n",
    "owid_covid_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UR1JhCuusdjD"
   },
   "source": [
    "The returned object is called tuples. Like lists, tuples contain a collection of data elements. But unlike lists, which are mutable, tuples are immutable, meaning that the element values cannot change. Also, compared to lists in which elements are inside square brackets `[]`, elements in tuples are inside parentheses `()`. \n",
    "\n",
    "To access a particular value in tuple, we use the square brackets. For example, to access the first element (i.e., number of rows) of the tuple `owid_covid_shape`, we type in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m71p9_JKwAna",
    "outputId": "c306b1b5-ec33-4c96-9553-352cbe26fe8f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "187569"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "owid_covid_shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7VgIAOWlwBTt"
   },
   "source": [
    "To access the second element (i.e., number of columns) of the tuple, we type in \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JzzSRF-BwIx2",
    "outputId": "03ef8160-f150-439d-8eec-dabfdc51d660"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "owid_covid.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5vPq9RUCwLZp"
   },
   "source": [
    "Note that indexing in Python starts from 0, so first element is index 0, second is index 1, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gqjA6sFbf15y"
   },
   "source": [
    "As of 11 May 2021, the data set contains 87,310 rows and 59 columns. But this is a live data set in which the number of rows are updated daily. To display these updates, we could use the `print()` function. The line of code below contains background text in quotes `''` and the up-to-date number of rows `owid_covid.shape[0]`  and number of columns `owid_covid.shape[1]` that will be inserted in the sentence each time the cell is executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CfMn4R9phEu7",
    "outputId": "a7312b52-98a6-4667-a45f-c65375da0913"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In the most current data on COVID-19 by Our World in Data, the number of rows is 187569 and the number of columns is 67\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"In the most current data on COVID-19 by Our World in Data, the number of rows is\",\n",
    "    owid_covid.shape[0],\n",
    "    \"and the number of columns is\",\n",
    "    owid_covid.shape[1],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0yDLuxDMeYVZ"
   },
   "source": [
    "In addition to the dimensions of the data set, we can access other metadata using attributes. For example, we can access the column labels of the data set using the attribute `columns`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3BgqbCKHek9z",
    "outputId": "8e845a7b-04d8-4645-cbf4-eff5d6bee7d5",
    "scrolled": true,
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['iso_code', 'continent', 'location', 'date', 'total_cases', 'new_cases',\n",
       "       'new_cases_smoothed', 'total_deaths', 'new_deaths',\n",
       "       'new_deaths_smoothed', 'total_cases_per_million',\n",
       "       'new_cases_per_million', 'new_cases_smoothed_per_million',\n",
       "       'total_deaths_per_million', 'new_deaths_per_million',\n",
       "       'new_deaths_smoothed_per_million', 'reproduction_rate', 'icu_patients',\n",
       "       'icu_patients_per_million', 'hosp_patients',\n",
       "       'hosp_patients_per_million', 'weekly_icu_admissions',\n",
       "       'weekly_icu_admissions_per_million', 'weekly_hosp_admissions',\n",
       "       'weekly_hosp_admissions_per_million', 'total_tests', 'new_tests',\n",
       "       'total_tests_per_thousand', 'new_tests_per_thousand',\n",
       "       'new_tests_smoothed', 'new_tests_smoothed_per_thousand',\n",
       "       'positive_rate', 'tests_per_case', 'tests_units', 'total_vaccinations',\n",
       "       'people_vaccinated', 'people_fully_vaccinated', 'total_boosters',\n",
       "       'new_vaccinations', 'new_vaccinations_smoothed',\n",
       "       'total_vaccinations_per_hundred', 'people_vaccinated_per_hundred',\n",
       "       'people_fully_vaccinated_per_hundred', 'total_boosters_per_hundred',\n",
       "       'new_vaccinations_smoothed_per_million',\n",
       "       'new_people_vaccinated_smoothed',\n",
       "       'new_people_vaccinated_smoothed_per_hundred', 'stringency_index',\n",
       "       'population', 'population_density', 'median_age', 'aged_65_older',\n",
       "       'aged_70_older', 'gdp_per_capita', 'extreme_poverty',\n",
       "       'cardiovasc_death_rate', 'diabetes_prevalence', 'female_smokers',\n",
       "       'male_smokers', 'handwashing_facilities', 'hospital_beds_per_thousand',\n",
       "       'life_expectancy', 'human_development_index',\n",
       "       'excess_mortality_cumulative_absolute', 'excess_mortality_cumulative',\n",
       "       'excess_mortality', 'excess_mortality_cumulative_per_million'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "owid_covid.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8OZ_HpRRe1I5"
   },
   "source": [
    "Display a concise summary of the DataFrame using the method `info()` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SBybgjsuCeRj",
    "outputId": "c0fd6788-7d0f-43cb-c83c-c6a8c086947c",
    "scrolled": true,
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 187569 entries, 0 to 187568\n",
      "Data columns (total 67 columns):\n",
      " #   Column                                      Non-Null Count   Dtype  \n",
      "---  ------                                      --------------   -----  \n",
      " 0   iso_code                                    187569 non-null  object \n",
      " 1   continent                                   176651 non-null  object \n",
      " 2   location                                    187569 non-null  object \n",
      " 3   date                                        187569 non-null  object \n",
      " 4   total_cases                                 180170 non-null  float64\n",
      " 5   new_cases                                   179954 non-null  float64\n",
      " 6   new_cases_smoothed                          178780 non-null  float64\n",
      " 7   total_deaths                                161800 non-null  float64\n",
      " 8   new_deaths                                  161792 non-null  float64\n",
      " 9   new_deaths_smoothed                         160633 non-null  float64\n",
      " 10  total_cases_per_million                     179338 non-null  float64\n",
      " 11  new_cases_per_million                       179122 non-null  float64\n",
      " 12  new_cases_smoothed_per_million              177953 non-null  float64\n",
      " 13  total_deaths_per_million                    160981 non-null  float64\n",
      " 14  new_deaths_per_million                      160973 non-null  float64\n",
      " 15  new_deaths_smoothed_per_million             159819 non-null  float64\n",
      " 16  reproduction_rate                           140137 non-null  float64\n",
      " 17  icu_patients                                24327 non-null   float64\n",
      " 18  icu_patients_per_million                    24327 non-null   float64\n",
      " 19  hosp_patients                               25587 non-null   float64\n",
      " 20  hosp_patients_per_million                   25587 non-null   float64\n",
      " 21  weekly_icu_admissions                       6067 non-null    float64\n",
      " 22  weekly_icu_admissions_per_million           6067 non-null    float64\n",
      " 23  weekly_hosp_admissions                      12103 non-null   float64\n",
      " 24  weekly_hosp_admissions_per_million          12103 non-null   float64\n",
      " 25  total_tests                                 76428 non-null   float64\n",
      " 26  new_tests                                   73456 non-null   float64\n",
      " 27  total_tests_per_thousand                    76428 non-null   float64\n",
      " 28  new_tests_per_thousand                      73456 non-null   float64\n",
      " 29  new_tests_smoothed                          98583 non-null   float64\n",
      " 30  new_tests_smoothed_per_thousand             98583 non-null   float64\n",
      " 31  positive_rate                               90878 non-null   float64\n",
      " 32  tests_per_case                              89458 non-null   float64\n",
      " 33  tests_units                                 101381 non-null  object \n",
      " 34  total_vaccinations                          50921 non-null   float64\n",
      " 35  people_vaccinated                           48496 non-null   float64\n",
      " 36  people_fully_vaccinated                     45947 non-null   float64\n",
      " 37  total_boosters                              23124 non-null   float64\n",
      " 38  new_vaccinations                            41640 non-null   float64\n",
      " 39  new_vaccinations_smoothed                   100308 non-null  float64\n",
      " 40  total_vaccinations_per_hundred              50921 non-null   float64\n",
      " 41  people_vaccinated_per_hundred               48496 non-null   float64\n",
      " 42  people_fully_vaccinated_per_hundred         45947 non-null   float64\n",
      " 43  total_boosters_per_hundred                  23124 non-null   float64\n",
      " 44  new_vaccinations_smoothed_per_million       100308 non-null  float64\n",
      " 45  new_people_vaccinated_smoothed              99293 non-null   float64\n",
      " 46  new_people_vaccinated_smoothed_per_hundred  99293 non-null   float64\n",
      " 47  stringency_index                            146120 non-null  float64\n",
      " 48  population                                  186420 non-null  float64\n",
      " 49  population_density                          167137 non-null  float64\n",
      " 50  median_age                                  154945 non-null  float64\n",
      " 51  aged_65_older                               153302 non-null  float64\n",
      " 52  aged_70_older                               154132 non-null  float64\n",
      " 53  gdp_per_capita                              154075 non-null  float64\n",
      " 54  extreme_poverty                             100625 non-null  float64\n",
      " 55  cardiovasc_death_rate                       154601 non-null  float64\n",
      " 56  diabetes_prevalence                         162150 non-null  float64\n",
      " 57  female_smokers                              116932 non-null  float64\n",
      " 58  male_smokers                                115329 non-null  float64\n",
      " 59  handwashing_facilities                      75957 non-null   float64\n",
      " 60  hospital_beds_per_thousand                  137178 non-null  float64\n",
      " 61  life_expectancy                             175401 non-null  float64\n",
      " 62  human_development_index                     150597 non-null  float64\n",
      " 63  excess_mortality_cumulative_absolute        6460 non-null    float64\n",
      " 64  excess_mortality_cumulative                 6460 non-null    float64\n",
      " 65  excess_mortality                            6460 non-null    float64\n",
      " 66  excess_mortality_cumulative_per_million     6460 non-null    float64\n",
      "dtypes: float64(62), object(5)\n",
      "memory usage: 95.9+ MB\n"
     ]
    }
   ],
   "source": [
    "owid_covid.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cnHa8HFVEbSb"
   },
   "source": [
    "Let's see what is the highest number of fully vaccinated adults in a country to date. We will use the variable `people_fully_vaccinated_per_hundred` and pandas' `max()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gfkiDHXcBOqa",
    "outputId": "04300058-f8c9-4184-dcbd-34dce71d90c7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "122.94"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "owid_covid[\"people_fully_vaccinated_per_hundred\"].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rQkzCTImFSio"
   },
   "source": [
    "Once we know the highest number of fully vaccinated adults per hundred, we determine the index of the observation (country) associated with that number using the pandas' method `idmax()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65772"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_highest_vaccination = owid_covid[\"people_fully_vaccinated_per_hundred\"].idxmax()\n",
    "index_highest_vaccination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gH6CRMimGVCt"
   },
   "source": [
    "Finally, we select the particular row (country in this case) by its location in the index using `iloc`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 115
    },
    "id": "sWzQKMoi_GGp",
    "outputId": "5b6daedc-99e8-4095-ac31-30c01a6304f1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iso_code</th>\n",
       "      <th>continent</th>\n",
       "      <th>location</th>\n",
       "      <th>date</th>\n",
       "      <th>total_cases</th>\n",
       "      <th>new_cases</th>\n",
       "      <th>new_cases_smoothed</th>\n",
       "      <th>total_deaths</th>\n",
       "      <th>new_deaths</th>\n",
       "      <th>new_deaths_smoothed</th>\n",
       "      <th>...</th>\n",
       "      <th>female_smokers</th>\n",
       "      <th>male_smokers</th>\n",
       "      <th>handwashing_facilities</th>\n",
       "      <th>hospital_beds_per_thousand</th>\n",
       "      <th>life_expectancy</th>\n",
       "      <th>human_development_index</th>\n",
       "      <th>excess_mortality_cumulative_absolute</th>\n",
       "      <th>excess_mortality_cumulative</th>\n",
       "      <th>excess_mortality</th>\n",
       "      <th>excess_mortality_cumulative_per_million</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65772</th>\n",
       "      <td>GIB</td>\n",
       "      <td>Europe</td>\n",
       "      <td>Gibraltar</td>\n",
       "      <td>2022-04-21</td>\n",
       "      <td>17706.0</td>\n",
       "      <td>212.0</td>\n",
       "      <td>30.286</td>\n",
       "      <td>101.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>79.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      iso_code continent   location        date  total_cases  new_cases  \\\n",
       "65772      GIB    Europe  Gibraltar  2022-04-21      17706.0      212.0   \n",
       "\n",
       "       new_cases_smoothed  total_deaths  new_deaths  new_deaths_smoothed  ...  \\\n",
       "65772              30.286         101.0         0.0                  0.0  ...   \n",
       "\n",
       "       female_smokers  male_smokers  handwashing_facilities  \\\n",
       "65772             NaN           NaN                     NaN   \n",
       "\n",
       "       hospital_beds_per_thousand  life_expectancy  human_development_index  \\\n",
       "65772                         NaN            79.93                      NaN   \n",
       "\n",
       "       excess_mortality_cumulative_absolute  excess_mortality_cumulative  \\\n",
       "65772                                   NaN                          NaN   \n",
       "\n",
       "       excess_mortality  excess_mortality_cumulative_per_million  \n",
       "65772               NaN                                      NaN  \n",
       "\n",
       "[1 rows x 67 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "owid_covid.iloc[[index_highest_vaccination]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vTUvMEQ1H0za"
   },
   "source": [
    "The above example is only for illustration of pandas functionality. We will unpack different approaches for data slicing and selecting with pandas in section _Data Design and Data Wrangling_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on exercise: Let's practice your Python skills\n",
    "\n",
    "1. Open a new `Colab` notebook.\n",
    "2. Add a new `Code` cell and include the following:\n",
    "    * Import the pandas library and add a comment describing your code\n",
    "    * Load the Data on COVID-19 by Our World in Data and assign the data to a variable name. Use an informative variable name that clearly describes what the data is about; you can use underscores to separate words in your variable name, e.g. COVID19_Data_Our_World_in_Data. Add a comment. \n",
    "\n",
    "3. For the loaded data set: \n",
    "    * Show the first 20 observations/rows\n",
    "    * Show the last 5 observations/rows\n",
    "\n",
    "4. When ready, share your Colab notebook with the class:\n",
    "    * Create a shareable link by clicking the Share button at the top right of your Colab notebook   \n",
    "    * Specify that Anyone with the link can view the notebook\n",
    "    * Share the link with the class.\n",
    "\n",
    "Before sharing your notebook, rerun your notebook from top to bottom using `Restart and run all` (under `Runtime` in the Colab menu bar) to ensure that your data analysis is computationally reproducible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G2nEJlaX-iAX"
   },
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/4/48/Markdown-mark.svg\" width=\"80\" height=\"40\" align=\"right\">\n",
    "\n",
    "# Markdown Text Cells\n",
    "\n",
    "Double click a text cell to see the Markdown syntax used to format the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6cdP1s2W_RRn"
   },
   "source": [
    "## **Headings**\n",
    "Headings begin with the hash symbol '#,' followed by the space. There are six Headings. To create the largest heading, add one hash symbol, and to create the smallest heading, add six hash symbols.\n",
    "\n",
    "```\n",
    "# Header 1, Title example (Not shown below)\n",
    "## Header 2, Subtitle example\n",
    "### Header 3,\n",
    "#### Header 4\n",
    "##### etc.\n",
    "```\n",
    "\n",
    "## Header 2, Subtitle example\n",
    "### Header 3,\n",
    "#### Header 4\n",
    "##### etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CE6YtY6oEKp1"
   },
   "source": [
    "## **Lists**\n",
    "\n",
    "```\n",
    "* Item one (Note the space after *)\n",
    "* Item two\n",
    "* Item three\n",
    "  * Sub-bullet\n",
    "    * Sub-sub-bullet\n",
    "\n",
    "```\n",
    "* Item one\n",
    "* Item two\n",
    "* Item three\n",
    "  * Sub-bullet\n",
    "    * Sub-sub-bullet\n",
    "  \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VLZzZSoyEJme"
   },
   "source": [
    "## **Bold and Italic Text**\n",
    "```\n",
    "**This text is bold**\n",
    "*This text is italic*\n",
    "~This was mistaken text~\n",
    "***This is both bold and italic text***\n",
    "```\n",
    "**This text is bold**\n",
    "\n",
    "*This text is italic*\n",
    "\n",
    "<del>A second line of stricken code</del>\n",
    "\n",
    "***This is both bold and italic text***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qsGBhKkrb3jZ"
   },
   "source": [
    "## **Hyperlinks**\n",
    "Markdown allows you to create links to content on the web using the following syntax: \n",
    "```\n",
    "[Link title](https://)\n",
    "```\n",
    "For example, a hyperlink to the [Project Jupyter](https://jupyter.org) would look like that: \n",
    "\n",
    "```\n",
    "[Project Jupyter](https://jupyter.org)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "opzpUQLcM-dY"
   },
   "source": [
    ":::{note}\n",
    "You can learn more about `Markdown` in `Jupyter` / `Colab` notebooks [here](https://www.datacamp.com/community/tutorials/markdown-in-jupyter-notebook), [here](https://www.earthdatascience.org/courses/intro-to-earth-data-science/file-formats/use-text-files/format-text-with-markdown-jupyter-notebook/) and [here](https://colab.research.google.com/notebooks/markdown_guide.ipynb#scrollTo=70pYkR9LiOV0). You can learn about the `Markdown` syntax in general [here](https://daringfireball.net/projects/markdown/syntax) and [here](https://www.markdownguide.org).\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PvvGkpxrWjEB"
   },
   "source": [
    "# Hands-on exercise: Let's practice your Markdown skills\n",
    "1. Create a new `Colab` notebook file.\n",
    "2. Add a new `Text` cell and include:\n",
    "* A notebook title (e.g. My Reproducible Research Workflow)\n",
    "* A bullet list with:\n",
    "  * A bold word for `Author`: and then add italised text for your name.\n",
    "  * A bold word for `Affiliation`: and then add italised text for your University and Department.\n",
    "  * A bold word for `Date`: and then add text for today’s date.\n",
    "* Add another Text cell and include:\n",
    "  * A list of at least two online datasets about Covid-19 you find interesting.\n",
    "  * Add a hyperlink to each database in your list and include the name of the database in the title of the hyperlink.\n",
    "* Add another Text cell and write a short research question you would be interesting in addressing using the Covid-19 datasets.\n",
    "\n",
    "When you complete your exercise, download your notebook and make it available on an Open Science Framework (OSF) repository. You can download your Colab notebook from `File` and then `Download .ipynb`. You can learn how to create an OSF repository and deposit your notebook from the tutorial [Getting Started With the Open Science Framework](https://www.youtube.com/watch?v=2TV21gOzfhw) by the Center for Open Science.   \n",
    "\n",
    "This exercise draws on Earth Lab's lesson [*Format Text In Jupyter Notebook With Markdown*](https://www.earthdatascience.org/courses/intro-to-earth-data-science/file-formats/use-text-files/format-text-with-markdown-jupyter-notebook/). "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "python_data_science_cloud.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}