JupyterNotebookForGreatGood/01 - Jupyter for python programming and data science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Programming and Data Science in Jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing and Viewing the Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a notebook, we can interactively build up an environment of functions, variables, objects, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we execute cells by pressing `Ctrl+Enter` in Command Mode. If you'd like to execute the currently selected cell and then highlight the next one, press `Shift+Enter`. Pressing this repeatedly is the fastest way to execute a chunk of code blocks in a line. And don't worry, executing a rendered Markdown cell doesn't hurt it, so you can just steamroll past them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lst = [\"one\", 2, \"III\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see what the current value of a variable is, we can either print it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or we make it the output of the last line of code in the cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def foo():\n",
    "    return lst\n",
    "\n",
    "foo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But then why didn't anything show up when we ran `print(lst)`? If the return value of the last line is `None`, nothing is displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And BTW, there's nothing to stop us from doing both!\n",
    "\n",
    "We get to see *two* outputs from our code cells: one is the `st`an`d`ard `out`, or `stdout`, which is where `print` puts things, and the other is the cell output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lst)\n",
    "lst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment is shared and interactive, so I can do something like the cell below, then run any of the cells above and get a new output, with `fore` at the end of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lst.append(\"fore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I could even put that cell all the way at the top, before the definition of `lst`, and it would still run.\n",
    "\n",
    "Beware though! When you shut down a Jupyter kernel, the environment disappears. If you aren't careful, getting the notebook back into the state it was in when you shut down can be very difficult. It's considered best practices to build the environment *linearly*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BTW, Python has lots of neat tools for inspecting objects in the environment, and Jupyter has special support for some of them.\n",
    "\n",
    "The built-in function `dir` returns a list of all of the attributes of an object (everything that can be accessed by `foo.something`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an attribute starts with a single `_`, that's a sign that you shouldn't care about it. If it starts with two `_`s, that means it's a special method that gets called by an operator or other special entity (i.e., not you), like `+` and `__add__`.\n",
    "\n",
    "So let's make a more succinct `dir`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shortdir(obj):\n",
    "    \"\"\"Same as built-in dir, but removes all items beginning with _\n",
    "    from the returned list.\n",
    "    \"\"\"\n",
    "    return [item for item in dir(obj) if not item.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shortdir(lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful for examining objects, but not functions or methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shortdir(shortdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something that's more useful for examining functions and methods is the `?` operator, which creates a tiny window at the bottom of your screen containing additional information.\n",
    "\n",
    "More useful is `??` which prints more verbosely. For functions, this includes the actual Python code that defined the function, which can be very useful when you forget the call signature or defaults of a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shortdir?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shortdir??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even faster then using any of these methods is the Jupyter tooltip. Pressing `Shift+Tab` while your text cursor is at the end of a word in Edit Mode will bring up the same information as `?`, but without having to execute the cell. Even more handy for quickly reminding yourself of how a function works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shortdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to emphasize: there's nothing magic here. Our homemade `shortdir` hooked into this system just fine. The one trick we needed to know was how to write a doc-string in Python, which is something we should be doing as often as possible!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting in Jupyter with Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's one tiny trick to get `matplotlib` working in Jupyter: you need the following line, called a `magic`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells Jupyter how to handle the display of `matplotlib` plots. We'll see an alternative to `inline` later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = np.arange(-3, 3, 0.01)\n",
    "plt.plot(xs, np.exp(-np.square(xs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we have this ugly `[<matplotlib.lines.Line2D at 0x ...]` above our plot, next to the `Out[n]`. Can you explain why that happens?\n",
    "\n",
    "We can fix it by adding a semi-colon to the end of the line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas in Jupyter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a dataset to get started. We'll use the [Iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) that inspired the GDSO logo. This dataset contains information about the petals and sepals (leaves underneath the petals) for three species of the *Iris* genus of flower.\n",
    "\n",
    "The dataset is available [at this link](https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv) if you'd like to download it by hand, otherwise, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if os.name is \"posix\":\n",
    "    !curl \"https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv\" -o res/data/iris.csv\n",
    "else:\n",
    "    !certutil.exe -urlcache -split -f \"https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv\" res/data/iris.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Side note on shell commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lines in the above cell that start with `!` execute in the command line. Executing command line utilities can be incredibly useful -- you can even perform your environment installations inside Jupyter!\n",
    "\n",
    "Even better, the execution of these tools can be controlled in Python. Above, I used an `if` statement to check whether the OS was of the `posix` family, which includes OS X and Linux, which determined the utility I used to download information from the web."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to our dataset: we can load the data into `pandas` and display the first five rows with `.head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"res/data/iris.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using a recent version of Jupyter, then this dataframe should be printed with some nice formatting: color behind alternate rows, and highlighting on hover. This is one of the benefits of Jupyter's all-HTML format -- outputs are (usually) in HTML, so all the power of CSS and even JavaScript can be used to prettify them and make them more useful.\n",
    "\n",
    "When presenting this material, this is where I stop and show folks how the dataframe head above and the matplotlib plot image are stored in the `.ipynb` file. The image is about 450 lines down, and the table 100 lines further. You should check out the files for yourself using a simple text editor (vim, emacs, nano, sublime, textedit, or notepad) to confirm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a powerful library for data science in Python, and you should learn it if you get the chance! I recommend [Brandon Rhodes' PyCon 2015 Tutorial](https://github.com/brandon-rhodes/pycon-pandas-tutorial), which is also an additional introduction to Jupyter notebooks.\n",
    "\n",
    "For now, we'll just use `.describe` to get some descriptive statistics on our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting in Jupyter with Seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a heads up, I strongly recommend that you check out the [seaborn data visualization library](https://seaborn.pydata.org/). Seaborn is a platform built on top of matplotlib in order to make data visualization a lot easier, as it is in `R` with `ggplot`. See below for examples of just how much easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df, hue=\"Name\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another tip: some folks find the matplotlib styling \"ugly\" (though it improved a lot in version 2.0), so try running the cell below, which changes the plotting aesthetics to a default designed by the seaborn folks, and then running those above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are lots of options for plot aesthetics available. One of my favorites, based on the style of plots in the XKCD webcomic, is set by the cell below. It doesn't work super well with seaborn, so try it on the pure matplotlib plot a few cells back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.xkcd();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}