pandas-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/CC_BY.png\"><br />\n",
    "\n",
    "Created by [Nathan Kelber](http://nkelber.com) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)<br />\n",
    "For questions/comments/improvements, email nathan.kelber@ithaka.org.<br />\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas 1\n",
    "\n",
    "**Description:** This notebook describes how to:\n",
    "* Create a Pandas Series or DataFrame\n",
    "* Accessing data rows using `.loc` and `.iloc`\n",
    "* Create filters using boolean operators\n",
    "* Changing data in the Series\n",
    "\n",
    "This is the first notebook in a series on learning to use Pandas. \n",
    "\n",
    "**Use Case:** For Learners (Detailed explanation, not ideal for researchers)\n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Knowledge Required:** \n",
    "* Python Basics ([Start Python Basics I](./python-basics-1.ipynb))\n",
    "\n",
    "**Knowledge Recommended:** \n",
    "* [Working with Dataset Files](./working-with-dataset-files.ipynb)\n",
    "\n",
    "**Completion Time:** 60 minutes\n",
    "\n",
    "**Data Format:** CSV (.csv)\n",
    "\n",
    "**Libraries Used:** Pandas\n",
    "\n",
    "**Research Pipeline:** None\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a Python data analysis and manipulation library. When it comes to viewing and manipulating data, most people are familiar with commercial spreadsheet software, such as Microsoft Excel or Google Sheets. While spreadsheet software and Pandas can accomplish similar tasks, each has significant advantages depending on the use-case.\n",
    "\n",
    "**Advantages of Spreadsheet Software**\n",
    "* Point and click\n",
    "* Easier to learn\n",
    "* Great for small datasets (<10,000 rows)\n",
    "* Better for browsing data\n",
    "\n",
    "**Advantages of Pandas**\n",
    "* More powerful data manipulation with Python\n",
    "* Can work with large datasets (millions of rows)\n",
    "* Faster for complicated manipulations\n",
    "* Better for cleaning and/or pre-processing data\n",
    "* Can automate workflows in a larger data pipeline\n",
    "\n",
    "In short, spreadsheet software is better for browsing small datasets and making moderate adjustments. Pandas is better for automating data cleaning processes that require large or complex data manipulation.\n",
    "\n",
    "Pandas can interpret a wide variety of data sources, including Excel files, CSV files, and Python objects like lists and dictionaries. Pandas converts these into two fundamental objects: \n",
    "\n",
    "* Data Series- a single column of data\n",
    "* DataFrame- a table of data containing multiple columns and rows\n",
    "\n",
    "This lesson introduces their basic affordances. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Series\n",
    "\n",
    "We can think of a Series as a single column of data. A DataFrame then is made by combining Series objects side-by-side into a table that has both height and width. Let's create a Series based on the world's ten most-populated countries [according to Wikipedia](https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population).\n",
    "\n",
    "|Population (in millions)|\n",
    "|---|\n",
    "|1,404|\n",
    "|1,366|\n",
    "|330|\n",
    "|269|\n",
    "|220|\n",
    "|211|\n",
    "|206|\n",
    "|169|\n",
    "|146|\n",
    "|127|\n",
    "\n",
    "We will put these population numbers into a Pandas Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pandas, `as pd` allows us to shorten typing `pandas` to `pd` when we call pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create our Series, we pass a list into the Series method:\n",
    "\n",
    "`variable_name = pd.Series([1, 2, 3])`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create a data series object in Pandas\n",
    "worldpop = pd.Series([1404, 1366, 330, 269, 220, 211, 206, 169, 146, 127])\n",
    "print(worldpop)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Underneath the Series is a `dtype` which describes the way the data is stored in the Series. Here we see `int64`, denoting the data are 64-bit integers. We can assign a name to our series using `.name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Give our series a name\n",
    "worldpop.name = 'World Population (In Millions)'\n",
    "print(worldpop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.iloc[]` Integer Location Selection\n",
    "\n",
    "To the left of each row in a Series are index numbers. The index numbers are similar to the index numbers for a Python list; they help us reference a particular row for data retrieval. Also, like a Python list, the index to a Series begins with 0. We can retrieve individual elements in a Series using the `.iloc` attribute, which stands for \"index location.\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the 4th element in our series\n",
    "worldpop.iloc[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like a python list, we can also slice a series into a smaller series. When slicing a Pandas series, the new series will not include the final index row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a slice of elements in our series\n",
    "# This slice will not include element 4\n",
    "worldpop.iloc[2:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, our Series has a numerical index like a Python list, but we can also give each row an identifier (like a key within a Python dictionary). We do this by using:\n",
    "\n",
    "`series_name.index = [name_1, name_2, name_3]`\n",
    "\n",
    "Since we are storing the populations of countries, it would also be helpful to include the name of each country within our index. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the index to use names instead of numerical indexes\n",
    "worldpop.index = [\n",
    "    'China',\n",
    "    'India',\n",
    "    'United States',\n",
    "    'Indonesia',\n",
    "    'Pakistan',\n",
    "    'Brazil',\n",
    "    'Nigeria',\n",
    "    'Bangladesh',\n",
    "    'Russia',\n",
    "    'Mexico'\n",
    "]\n",
    "\n",
    "worldpop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.loc[]` Location Selection\n",
    "Now we can also reference each element by its index name, similar to how we can supply a key to a dictionary to get a value. We use the `.loc[]` attribute to reference by name (as opposed to integer/index location using `.iloc[]`.\n",
    "\n",
    "Try finding the value for Nigeria using both `iloc[]` and `.loc[]` selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use `.iloc[]` to return the series value for Nigeria"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use `.loc[]` to return the series value for Nigeria"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of a value, we can return a new series by supplying a list. This will return the value *with the index names* as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a new series containing only Nigeria\n",
    "# Note that we use two sets of brackets\n",
    "\n",
    "worldpop.loc[['Nigeria']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a series value for Indonesia and Mexico\n",
    "worldpop.loc[['Indonesia', 'Mexico']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of supplying a list of every index name, we can use a slice notation using a `:`. There is, however, a significant difference in how this slice is created with *index names*: the final named index **is included**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a slice from Nigeria to Russia\n",
    "# This slice will include the final element!\n",
    "# This behavior is different than a list slice\n",
    "\n",
    "worldpop.loc['Nigeria':'Russia']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we created this Pandas series from a list, a series with index names is kind of like an ordered dictionary. Indeed, we could have created our Pandas series from a dictionary instead of a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series from a dictionary\n",
    "# Based on most populous cities in the world according to Wikipedia\n",
    "\n",
    "citiespop = pd.Series({\n",
    "    'Tokyo': 37,\n",
    "    'Delhi': 28,\n",
    "    'Shanghai': 25,\n",
    "    'São Paulo': 21,\n",
    "    'Mexico City': 21,\n",
    "    'Cairo': 20,\n",
    "    'Mumbai': 19,\n",
    "    'Beijing': 19,\n",
    "    'Dhaka': 19,\n",
    "    'Osaka': 19,\n",
    "}, name='World City Populations (In Millions)') # We can also specify the series name as an argument\n",
    "\n",
    "#Return the series\n",
    "citiespop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean Expressions\n",
    "\n",
    "We have seen already how we can select a particular value in a series by using an index name or number. We can also select particular values using Boolean expressions. An expression will evaluate to a Truth Table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Which countries have populations greater than 200 million?\n",
    "worldpop > 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing this expression into `.iloc[]`, we can retrieve just the rows that would evaluate to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate worldpop for `worldpop > 200`\n",
    "worldpop.loc[worldpop > 200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have not changed the values of `worldpop` but only evaluated an expression. `worldpop` remains the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "worldpop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to store the evaluation, we would need to use an assignment statement, either for `worldpop` or a new variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we wanted to save this to a new series variable\n",
    "new_series = worldpop.loc[worldpop > 200]\n",
    "\n",
    "new_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also evaluate multiple expressions, but there is a difference in syntax between Python generally and Pandas. Python Boolean expressions are written `and`, `or` and `not`. Pandas Boolean expressions are written `&`, `|`, and `~`.\n",
    "\n",
    "|Pandas Operator|Boolean|Requires|\n",
    "|---|---|---|\n",
    "|&|and|All must be `True`|\n",
    "|\\||or|If any are `True`|\n",
    "|~|not|The opposite|\n",
    "\n",
    "Try returning a series from `worldpop` using `.loc[]` for countries with populations either over 500 or under 250."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a series from `worldpop` with populations\n",
    "# over 500 or under 250"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying a Series\n",
    "\n",
    "So far, we have been returning expressions but not actually changing our original Pandas series object. We can use an initialization statement to make a change to the original series object The syntax is very similar to changing an item value in a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the population of China to 1500\n",
    "worldpop.loc['China'] = 1500\n",
    "worldpop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also change the value of multiple rows based on an expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the population of several countries based on an expression\n",
    "worldpop.loc[worldpop < 300] = 25\n",
    "worldpop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of Pandas Series\n",
    "\n",
    "* A series is a single column of data that may contain a Name\n",
    "* A particular row in a series can be referenced by index number or index name\n",
    "* Use `.iloc` to select a row by index number\n",
    "* Use `.loc` to select a row by index name\n",
    "* Use an initialization statement to change values\n",
    "* Boolean operators include & (and), | (or), ~ (negation)\n",
    "____\n",
    "Learn about Pandas DataFrames next in [Pandas 2 ->](./pandas-2.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "278px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}