lps-030/learn-python-series-030-data-science-pt1-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn Python Series (#30) - Data Science Part 1 - Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repository\n",
    "- https://github.com/pandas-dev/pandas\n",
    "- https://github.com/python/cpython\n",
    "\n",
    "### What will I learn?\n",
    "- You will learn what kind of toolset the `pandas` Python package is providing you with, how to install it (if you haven't installed it already in your current Python distribution), and import it into your projects;\n",
    "- how to convert data (either passed-in directly or read from another source) to a `pandas` DataFrame;\n",
    "- how to save data from a `pandas` DataFrame to an external file, such as CSV;\n",
    "- how to do some basic `pandas` data wrangling operations.\n",
    "\n",
    "### Requirements\n",
    "- A working modern computer running macOS, Windows or Ubuntu;\n",
    "- An installed Python 3(.7) distribution, such as (for example) the Anaconda Distribution;\n",
    "- The ambition to learn Python programming.\n",
    "\n",
    "### Difficulty\n",
    "- Beginner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Curriculum (of the `Learn Python Series`):\n",
    "\n",
    "- [Learn Python Series - Intro](https://steemit.com/utopian-io/@scipio/learn-python-series-intro)\n",
    "- [Learn Python Series (#2) - Handling Strings Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-2-handling-strings-part-1)\n",
    "- [Learn Python Series (#3) - Handling Strings Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-3-handling-strings-part-2)\n",
    "- [Learn Python Series (#4) - Round-Up #1](https://steemit.com/utopian-io/@scipio/learn-python-series-4-round-up-1)\n",
    "- [Learn Python Series (#5) - Handling Lists Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-5-handling-lists-part-1)\n",
    "- [Learn Python Series (#6) - Handling Lists Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-6-handling-lists-part-2)\n",
    "- [Learn Python Series (#7) - Handling Dictionaries](https://steemit.com/utopian-io/@scipio/learn-python-series-7-handling-dictionaries)\n",
    "- [Learn Python Series (#8) - Handling Tuples](https://steemit.com/utopian-io/@scipio/learn-python-series-8-handling-tuples)\n",
    "- [Learn Python Series (#9) - Using Import](https://steemit.com/utopian-io/@scipio/learn-python-series-9-using-import)\n",
    "- [Learn Python Series (#10) - Matplotlib Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-10-matplotlib-part-1)\n",
    "- [Learn Python Series (#11) - NumPy Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-11-numpy-part-1)\n",
    "- [Learn Python Series (#12) - Handling Files](https://steemit.com/utopian-io/@scipio/learn-python-series-12-handling-files)\n",
    "- [Learn Python Series (#13) - Mini Project - Developing a Web Crawler Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-13-mini-project-developing-a-web-crawler-part-1)\n",
    "- [Learn Python Series (#14) - Mini Project - Developing a Web Crawler Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-14-mini-project-developing-a-web-crawler-part-2)\n",
    "- [Learn Python Series (#15) - Handling JSON](https://steemit.com/utopian-io/@scipio/learn-python-series-15-handling-json)\n",
    "- [Learn Python Series (#16) - Mini Project - Developing a Web Crawler Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-16-mini-project-developing-a-web-crawler-part-3)\n",
    "- [Learn Python Series (#17) - Roundup #2 - Combining and analyzing any-to-any multi-currency historical data](https://steemit.com/utopian-io/@scipio/learn-python-series-17-roundup-2-combining-and-analyzing-any-to-any-multi-currency-historical-data)\n",
    "- [Learn Python Series (#18) - PyMongo Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-18-pymongo-part-1)\n",
    "- [Learn Python Series (#19) - PyMongo Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-19-pymongo-part-2)\n",
    "- [Learn Python Series (#20) - PyMongo Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-20-pymongo-part-3)\n",
    "- [Learn Python Series (#21) - Handling Dates and Time Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-21-handling-dates-and-time-part-1)\n",
    "- [Learn Python Series (#22) - Handling Dates and Time Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-22-handling-dates-and-time-part-2)\n",
    "- [Learn Python Series (#23) - Handling Regular Expressions Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-23-handling-regular-expressions-part-1)\n",
    "- [Learn Python Series (#24) - Handling Regular Expressions Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-24-handling-regular-expressions-part-2)\n",
    "- [Learn Python Series (#25) - Handling Regular Expressions Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-25-handling-regular-expressions-part-3)\n",
    "- [Learn Python Series (#26) - pipenv & Visual Studio Code](https://steemit.com/utopian-io/@scipio/learn-python-series-26-pipenv-and-visual-studio-code)\n",
    "- [Learn Python Series (#27) - Handling Strings Part 3 (F-Strings)](https://steemit.com/utopian-io/@scipio/learn-python-series-27-handling-strings-part-3-f-strings)\n",
    "- [Learn Python Series (#28) - Using Pickle and Shelve](https://steemit.com/utopian-io/@scipio/learn-python-series-28-using-pickle-and-shelve)\n",
    "- [Learn Python Series (#29) - Handling CSV](https://steemit.com/utopian-io/@scipio/learn-python-series-29-handling-csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional sample code files\n",
    "The full - and working! - iPython tutorial sample code file is included for you to download and run for yourself right here:\n",
    "\n",
    "### GitHub Account\n",
    "https://github.com/realScipio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn Python Series (#30) - Data Science Part 1 - Pandas\n",
    "Welcome to already episode #30 of the `Learn Python Series`! It's been a while since I've published my last (#29) tutorial episode on Python, after which I was busy with a number of projects including co-developing and running UA and @steem-ua together with @holger80.\n",
    "\n",
    "Not everybody realises that (although I can code) I'm not originally academically educated in Computer Sciences, ergo that I'm writing the `Learn Python Series` partially as a documentation project on my own Python research, study and development aspirations. By carefully writing these tutorials in a very structured format, almost (or even exactly) \"book-like\", I'm \"cementing\" my own Python knowledge and skills. The past months I've gained an interest in learning more about Data Science using Python, and I recently came to the conclusion my own \"research notes\" were beginning to pile and felt the need to better document my progress. How to do that better by resuming the `Learn Python Series`? So there you go...! ;-) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About Data Science\n",
    "Data Science is about gaining insights from (huge) amounts of (structured) data by analysing that data, and also to analytically and algorithmically solve complex problems, which insights and algorithmic solutions also have the potential to generate much value. When you dig into (large / big) data sets, you might be able to discover new insights that were previously hidden. The process of first exploring data, investigating that data to discover data characteristics and patterns, enriching that data with other data, often times requires a combination of both analytical skills and mathematical / business / tech creativity and skill. I suppose data science is positioned in the intersecting areas of those fields, which alligns with my own interests as well; which is why I find Data Science fascinating to learn more about, personally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About the Python package `pandas`\n",
    "`pandas` is a well-known and actively developed Python package which can be summarised as a \"data analysis, wrangling and management toolkit\"; I suppose you could call it \"Excel for Python\" in a way. `pandas` provides powerful and flexible methods and data formats to aid data science tasks, using Python and it's built on top of `numpy` (\"Numerical Python\", which we've already yet briefly talked about in episode #11 of the `Learn Python Series`).\n",
    "\n",
    "`pandas` is positioned (as opposed to NumPy itself) as a more \"high level\" data analysis / wrangling toolkit, and - like Excel or OpenOffice \"Calc\" - it works really well with \"tabular data\". Unlike Excel / Calc, `pandas` is able to handle really large data sets, with file sizes ranging from hundreds of MegaBytes to even Gigabytes (or more!); try working with (or even opening!) those on a regular Excel / Calc application running on a regular personal computer!\n",
    "\n",
    "`pandas` can therefore be used to -1- clean / munge / wrangle data sets, -2- analyse and (re-) model the data set, and -3- organise the data analysis (to plot, display in tabular form, and/or further process).\n",
    "\n",
    "In short `pandas` is really powerful and cool, so let's dive right in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing and importing `pandas`\n",
    "If you're working with the Anaconda Python distribution, the `pandas` package is already installed by default, so you only need to `import` it in your project. If you haven't already installed `pandas`, that's as simple as:\n",
    "\n",
    "`pip install pandas`\n",
    "\n",
    "Then, create a new Python file, give it a relevant name (for example `pandas_tut_1.py`) and then simply begin with:\n",
    "\n",
    "`import pandas as pd`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `pandas` Data Frame Basics\n",
    "A `DataFrame` is a `pandas` **data structure** to represent tabular data (like a CSV file or an Excel spreadsheet with named columns and rows). Shortly hereafter, we'll be covering how to read-in an existing CSV file and convert it to a DataFrame object, but let's begin with creating a simple example DataFrame from scratch.\n",
    "\n",
    "### the `.DataFrame()` constructor\n",
    "If we begin with a regular Python data object such as a dictionary, or a list of lists or tuples, `pandas` provides the `.DataFrame()` constructor to convert such data objects into a `pandas` DataFrame, for example like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "weather_dict = {\n",
    "    'day': ['1/1/2019', '1/2/2019', '1/3/2019', '1/4/2019', '1/5/2019'],\n",
    "    'temp_celsius': [3, 2, -1, 0, 4]\n",
    "}\n",
    "\n",
    "df1 = pd.DataFrame(data=weather_dict)\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Explanation:** after importing `pandas` as `pd`, and declaring a dictionary object with two keys (`day` and `temp_celsius`), each containing one list with 5 values, we then converted the `weather_dict` dictionary object into a DataFrame object (called `df1`).\n",
    "\n",
    "**Nota bene:** as always, I'm writing this tutorial itself using Jupyter Notebook, which contains both a Python interpreter, the markdown content, and a number of built-in Jupyter Notebook-specific methods and mechanisms. Running the above code inside a Jupyter Notebook prints/outputs the `df1` DataFrame simply by calling the variable `df1`. In case you want to print the `df1` DataFrame contents from the command line after having coded the above in an external code editor (e.g. Microsoft Visual Studio Code), then you need to do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        day  temp_celsius\n",
      "0  1/1/2019             3\n",
      "1  1/2/2019             2\n",
      "2  1/3/2019            -1\n",
      "3  1/4/2019             0\n",
      "4  1/5/2019             4\n"
     ]
    }
   ],
   "source": [
    "print(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(From here on I'm assuming you're following along on a Jupyter Notebook as well, hence I won't be explicitly printing the DataFrame objects every time in the remainder of this and following tutorial(s).)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Nota bene:** in this particular (dictionary) example, I've been using a \"top down\" approach, in which data is converted into a DataFrame object by **dictionary keys**. However, a more \"logical\" approach would be to insert that data \"row-by-row\", as the temperature value of \"3 degrees Celsius\" belongs to the associated data value \"1/1/2019\".\n",
    "\n",
    "Another way to construct the same DataFrame, is via a \"list of lists\", which are then given column names as an additional constructor argument, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "weather_list = [\n",
    "    ['1/1/2019', 3],\n",
    "    ['1/2/2019', 2],\n",
    "    ['1/3/2019', -1],\n",
    "    ['1/4/2019', 0],\n",
    "    ['1/5/2019', 4]\n",
    "]\n",
    "\n",
    "df2 = pd.DataFrame(data=weather_list, columns=['day', 'temp_celsius'])\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the `read_csv()` method\n",
    "As we've just learned, the `DataFrame()` constructor needs to be passed a `data=` argument, which is the Python object holding the (example) data. But of course when dealing with large data sets you're not going to declare all those values manually. Instead, you might have saved them already on disk and you like to read the data from disk to then convert to a DataFrame object.\n",
    "\n",
    "For exactly that purpose, `pandas` has the built-in method `read_csv()` (as well as a number of similar methods for other file types). Suppose in your current working directory exists the CSV file `weather.csv`, then you can construct the exact same DataFrame object like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df3 = pd.read_csv('weather.csv')\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the `to_csv()` method\n",
    "`pandas` also allows to go the opposite route: to export DataFrame objects and save them to disk as `.csv` files. The `to_csv()` is used for that.\n",
    "\n",
    "**Nota bene:** in order to save the `weather.csv` example file (that we just read via `read_csv()`) from the df2 DataFrame object we constructed before, it's convenient to not save the `0,1,2,3,4` **index values** to the CSV file (those index values are format-specific, and don't directly belong to the original data set). By default (at least in `pandas` version 0.24.0; the current version) those index values _would be_ exported to CSV and so are the column names / headers (the first row of the CSV file). While we do want those column name values included in the CSV file, but not the `pandas` default index values, we set the `index=` parameter to `None` (and leave the `header=` parameter as it is by default: `True`). As the first argument we pass the file name (and an optional filepath in case you want to save it in another directory as your current working directory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.to_csv('weather.csv', index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the above `to_csv()` code line, your file `'weather.csv'` should be saved as a valid CSV file, located in your current working directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the `.head()` and `.tail()` methods\n",
    "When working with large data sets, it's often times convenient to quickly inspect the data you're working with, without wanting to \"eye ball\" big amounts of data. To only display the top 5 lines of your DataFrame (including column names and index numbers) you can use the `.head()` method, and to only display the bottom 5 lines of your DataFrame you can use `.tail()`.\n",
    "\n",
    "**Nota bene:** Please note that our (very simple) example `weather.csv` data set only contains 5 rows in total for simplicity / explanatino matters, ergo, in thisspecific example case you wouldn't notice a difference when running either ... \n",
    "\n",
    "- `df3`, or\n",
    "- `df3.head()`, or\n",
    "- `df3.tail()`\n",
    "\n",
    "However, you can also pass an integer `N` to both `.head()` and `.tail()`, to only show N likes either at the top or bottom of your DataFrame, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.tail(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In these specific examples, by passing the integer value of `2` to both `.head(2)` and `.tail(2)` we only show the top and bottom 2 lines of the DataFrame, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index slicing\n",
    "If you're interested to only use a specific set of DataFrame rows, you can use index slices just like we've learned about already on regular Python lists.\n",
    "\n",
    "For example, if we only want to work with rows 1 and 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3[1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Nota bene:** the stop parameter is non-inclusive, ergo `df3[1:3]` means \"begin with row 1 and stop at row number 3\", hence, it shows rows 1 and 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you want to work with the entire DataFrame beginning with row number 2, then use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3[2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in case you want to work with the entire DataFrame until row number 3, then use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the `columns` attribute / property\n",
    "If you want to assign, return, or print all column names your DataFrame holds, call the `columns` attribute / property, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['day', 'temp_celsius'], dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the `shape` attribute / property\n",
    "Calling the `shape` property returns a tuple of the DataFrames \"size\" or \"shape\" in the form of `(num_rows, num_columns)`, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 3)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Nota bene:** it's more efficient / faster to call the `shape` property when determining the total amount of rows in a DataFrame than using `len(df3)` would be, although that works as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows via .shape: 5\n",
      "Number of rows via len(): 5\n"
     ]
    }
   ],
   "source": [
    "num_rows_shape, num_cols_shape = df3.shape\n",
    "print(f\"Number of rows via .shape: {num_rows_shape}\")\n",
    "\n",
    "num_rows_len = len(df3)\n",
    "print(f\"Number of rows via len(): {num_rows_len}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two syntaxes on selecting columns\n",
    "`pandas` allows for two syntaxes to selecting individual columns, being with the `.column_name` dot-property notation, and by using `['column_name']` squared brackets notation. E.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3\n",
       "1    2\n",
       "2   -1\n",
       "3    0\n",
       "4    4\n",
       "Name: temp_celsius, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.temp_celsius"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and also:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3\n",
       "1    2\n",
       "2   -1\n",
       "3    0\n",
       "4    4\n",
       "Name: temp_celsius, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['temp_celsius']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Nota bene:** I strongly recommend **to only use the squared bracket notation**, as your DataFrame column names could be identical to `pandas` built-in attribute / property names. Suppose for example you'd have a column named `columns` (for some reason or another), then calling `df3.columns` returns the `columns` property values, not the content of the `df3['columns']` DataFrame column!\n",
    "\n",
    "Also, in case your column names contain one or more spaces or other non-alphanumerical characters, the dot-property syntax doesn't work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting multiple DataFrame columns\n",
    "In case you want to select multiple DataFrame columns, but not all of them, then **pass a list** of column names instead. Again, on our very simple example file `weather.csv` we only have 2 columns containing data, so the example I'll give right now will return the same results (in this specific case) as calling the entire DataFrame object. In case there would be another column in the total DataFrame, then of course this technique only selects the column names passed as a list, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3[['day', 'temp_celsius']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting specific rows and specific columns\n",
    "In case you want to select both (one or more) columns and a slice of DataFrame rows, then combine the just explained techniques beginning with the columns, then the row slices, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    2\n",
       "2   -1\n",
       "Name: temp_celsius, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['temp_celsius'][1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Appending a new column using vectorised column operations\n",
    "As `pandas` is built on top of `numpy`, which very efficiently uses vectorised data array operations, so does `pandas` itself. Vectorisation means executing an operation on an entire column / array of data.\n",
    "\n",
    "Let's say, just to easily explain such a column-wise vectorisation process, we'd want to add another column to the DataFrame - called `\"temp_plus_one\"`, in which we want to store all the temperature values incremented by 1 celsius. To do that, first we simply name / assign that new / extra column, then reference (in this case) the `'temp_celsius'` column and add the value of `1` to it, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "      <th>temp_plus_one</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius  temp_plus_one\n",
       "0  1/1/2019             3              4\n",
       "1  1/2/2019             2              3\n",
       "2  1/3/2019            -1              0\n",
       "3  1/4/2019             0              1\n",
       "4  1/5/2019             4              5"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['temp_plus_one'] = df3['temp_celsius'] + 1\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DataFrame column \"temp_plus_one\" is now added to the original `df3` DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing an existing column from a DataFrame, using `.drop()`\n",
    "If we again like to remove the \"temp_plus_one\" column from the DataFrame, we can use the `.drop()` method. By default the `.drop()` method has an argument set as `axis=0` to imply removing one or more rows from the data set. If we want to drop a column, we either pass as argument `axis=1` or `axis='columns'`, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temp_celsius</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/1/2019</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/2/2019</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2019</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/4/2019</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/5/2019</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        day  temp_celsius\n",
       "0  1/1/2019             3\n",
       "1  1/2/2019             2\n",
       "2  1/3/2019            -1\n",
       "3  1/4/2019             0\n",
       "4  1/5/2019             4"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = df3.drop('temp_plus_one', axis='columns')\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What did we learn, hopefully?\n",
    "Data Science is (to me it is at least!) an extremely interesting topic, and the `pandas` Python package provides many powerful, relatively straightforward, and efficient tools for data analysis and manipulation. At `pandas`' core is the DataFrame object / data format, that you can create from and to regular Python data types (e.g. dictionaries, lists, tuples) and multiple file types (such as CSV, JSON and Excel).\n",
    "\n",
    "In this episode we covered some `pandas` basics (of course on every newly covered topics we need to start at the beginning!), and in the following episodes we'll gradually expand on the possibilities `pandas` has to offer and move on to intermediate and advanced skill techniques.\n",
    "\n",
    "### Thank you for your time!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}