python/1. Introduction to Python and IPython.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python and IPython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When people talk about Python, they can mean a couple of things. Python is \n",
    "1. A programming language\n",
    "2. The program, called an \"interpreter\" that runs code written in that language. Matlab and R are also interpreted languages. You can think of the interpreter as the \"engine\" that takes your code and uses it to make things happen.\n",
    "\n",
    "While there is only a single definition of the Python language [defined by the Python Software Foundation](http://www.python.org), there are many versions of the interpreter, written in different languages to run on different types of systems. (Curious fact: the \"standard\" Python interpreter, called CPython, is written in the C programming language, while program that compiles code written in the C programming language is itself written in C!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive vs script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python can be run in two basic modes. In the first, we would run a script written in python by calling the python interpreter from the command line:\n",
    "\n",
    "```\n",
    "$ python myprogram.py\n",
    "```\n",
    "\n",
    "For the second, we would type\n",
    "\n",
    "```\n",
    "$ python\n",
    "```\n",
    "\n",
    "at the command line to begin an *interactive* python session. In this session, we can execute commands in python interactively (just like the shell!), get the results, and use the output. You will know you are in the python shell when you see the prompt\n",
    "\n",
    "```\n",
    ">>>\n",
    "```\n",
    "instead of your normal shell prompt.\n",
    "\n",
    "Note that the ability to interactively run programs is one of the things that separates languages like python (and R and matlab) from compiled languages like C and Java. This is one of the major reasons we teach Python for data analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you read enough about Python, you will eventually see mention of differences with Python 2 and Python 3. Several years ago, the folks in charge of Python realized that they needed to make some serious changes in the language that would mean old software would no longer run, thus breaking what software engineers call \"backward compatibility.\" As a result, it's taken several years, but as of now, nearly all major Python packages have been ported to Python 3, and ongoing development of Python 2 packages has (or will soon be) discontinued.\n",
    "\n",
    "For this reason, you should begin by learning Python 3. For most Python you see in the wild, the differences are fairly minor, and code written for Python 2 can easily be made to run. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The ecosystem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python is a good language, and easier to learn than many. But learning to program isn't actually the hard part. The hard part is doing the actual analysis, which often means finding tools that make the analysis possible. Since most of us aren't professional-grade programmers, we use code written by others to do our science. And when that code isn't readily available, it makes our lives needlessly difficult.\n",
    "\n",
    "All of which is to say **we teach Python because Python has the tools for scientific computing**. There are lots and lots and lots of libraries for Python, and in the last several years, these have coalesced around a bunch of key technologies, including\n",
    "- the standard Python libraries, which define the language\n",
    "- `NumPy`, which defines a fast, efficient array that can be used for heavy number crunching\n",
    "- `SciPy`, which includes functions for signal processing, special functions, and statistics\n",
    "- `Pandas`, which defines a type of object called a data frame for organizing and manipulating data sets\n",
    "- `Matplotlib`, `bokeh`, `seaborn`, and a host more for plotting\n",
    "- `statsmodels` and `patsy` for defining and fitting statistical models like regressions\n",
    "- `Scikit-learn` for machine learning\n",
    "- `Scikit-image` and `PIL` for image processing\n",
    "- `Cython`, `PyPy`, `blaze`, and `Shed Skin` for making your code run *fast*\n",
    "- `SymPy` for computer algebra (think Mathematica)\n",
    "\n",
    "Even more important, if you work with neuroscience data, there are tremendously good libraries available:\n",
    "- for MRI: [NIPy](http://nipy.org) is a whole set of related tools for cleaning and analyzing MRI data\n",
    "- for EEG/MEG: there is less dominance here, but [MNE](http://martinos.org/mne/stable/mne-python.html) ([paper](http://journal.frontiersin.org/Journal/10.3389/fnins.2013.00267/full)) seems to be quite good\n",
    "- for spikes: Unfortunately, there isn't nearly as much here. This is because single unit electrophysiologists believe they are smarter than everyone else and generally refuse to run anything but their own code. In truth, it's because these data require a little less cleaning and processing. Once the spikes are sorted (lots of options there), some analysis is available [here](https://github.com/OpenElectrophy/OpenElectrophy) and [here](https://github.com/chrox/RealTimeElectrophy), while my code is [here](https://github.com/jmxpearson/physutils)\n",
    "- for coding behavioral experiments: [Psychophysics Toolbox](http://psychtoolbox.org) has been the go-to solution with Matlab. On the Python side, I like [PsychoPy](http://www.psychopy.org). (See also [my blog post](http://jmxpearson.com/2014/06/05/further-adventures-in-pythonism.html) on this.)\n",
    "- for simulations of neural networks, [Brian](http://briansimulator.org/) and [PyNN](http://neuralensemble.org/PyNN/) make what the formerly excruciating routine.\n",
    "- for Bayesian analysis, there are [PyStan](https://pystan.readthedocs.org/), [PyMC](http://pymc-devs.github.io/pymc/), and [emcee](http://dan.iel.fm/emcee/current/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sounds like a mess, I know, but you can visualize it a little like the figure below. There, arrows represent *dependencies*. For instance, arrows point from SciPy, Cython, and NumPy to Pandas because Pandas builds on all of these. Similarly, statsmodels builds on Pandas. As you can see NumPy is at the heart of many of these advanced tools.\n",
    "\n",
    "<img src=\"ecosystem.svg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### History"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost a decade ago, Fernando Perez began the [IPython project](http://www.ipython.org) to design a better version of the Python shell. As you'll recall, the shell is itself just another program. It gets input from the user, talks to the file system and operating system, and runs other commands/programs. You can view the python shell as a similar type of program, except, instead of talking to the operating system, the shell talks to the Python interpreter. IPython simply replaces the old Python shell with something much more flexible and powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More recently, IPython took a big step forward in releasing the IPython server and the IPython notebook, which is the system that generated this document. When you type\n",
    "\n",
    "```\n",
    "ipython notebook\n",
    "```\n",
    "\n",
    "at the command prompt, the IPython Notebook Server starts on your local machine, allowing you to interact with Python through your browser:\n",
    "<img src=\"ipython_local.svg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even more exciting, though, the IPython server *doesn't have to run on your machine*. You can interact with an IPython server running notebooks backed by a powerful computing cluster at a remote location!\n",
    "<img src=\"ipython_remote.svg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So why all the fuss?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IPython notebook has proven incredibly popular for a few reasons:\n",
    "* it allows us to mix figures, code, and text in the same document\n",
    "* it's easy to convert a notebook to pdf, html, and other document formats\n",
    "* with sites like [nbviewer](http://www.nbviewer.org), it's really easy to share with the world\n",
    "* because the notebook interface is language-agnostic, we can even use [Matlab](http://nbviewer.ipython.org/gist/anonymous/8940322) and [R](http://nbviewer.ipython.org/github/ipython/ipython/blob/3607712653c66d63e0d7f13f073bde8c0f209ba8/docs/examples/notebooks/rmagic_extension.ipynb) in the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll also note those spiffy menus and buttons at the top of the browser window. These allow you to do things like run the entire notebook, stop the notebook (if your code is taking too long for some reason), create or delete cells, and change what type of cell you're working on. There are also handy [keyboard shortcuts](http://nbviewer.ipython.org/github/adrn/ipython/blob/2.x/examples/Notebook/User%20Interface.ipynb) for many of these things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those who really want to dive in, I suggest the extended tutorial [here](http://ipython.org/videos.html), but let's go ahead and see what sorts of fun things are really very easy to do:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markdown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Markdown](http://daringfireball.net/projects/markdown/syntax) started out as a quick and dirty way to write HTML without all the clutter of tags. Write using something like normal text, and the program would convert it to a well-formatted web document. However, Markdown proved so successful that places like [GitHub](https://help.github.com/articles/github-flavored-markdown/) adopted and extended it, so that now, there are at least half a dozen markdown dialects out there, including code highlighting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Markdown is very easy to learn. For instance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# Pound signs indicate varying levels of header\n",
    "## this is a subheading\n",
    "### this is a sub-subheading\n",
    "\n",
    "lists work like this\n",
    "- one item\n",
    "- another item\n",
    "- etc.\n",
    "\n",
    "or this:\n",
    "1. first point\n",
    "1. second point\n",
    "1. notice these all start with 1? Markdown numbers automatically\n",
    "\n",
    "And [web links](http://www.duke.edu) are easy to do, too.\n",
    "\n",
    "Finally, you can get code highlighting like so:\n",
    "\n",
    "~~~python\n",
    "print \"Hello, world!\"\n",
    "~~~\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "becomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pound signs indicate varying levels of header\n",
    "## this is a subheading\n",
    "### this is a sub-subheading\n",
    "\n",
    "lists work like this\n",
    "- one item\n",
    "- another item\n",
    "- etc.\n",
    "\n",
    "or this:\n",
    "1. first point\n",
    "1. second point\n",
    "1. notice these all start with 1? Markdown numbers automatically\n",
    "\n",
    "And [web links](http://www.duke.edu) are easy to do, too.\n",
    "\n",
    "Finally, you can get code highlighting like so:\n",
    "\n",
    "~~~python\n",
    "print \"Hello, world!\"\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sweet! How does it work?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have your cell converted to Markdown in the notebook, you can either select \"Markdown\" from the dropdown box of cell types or hit Escape (so that your cursor disappears) and hit `m`. This will change the selector up top as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Another killer trick: Equations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those of you who know LaTeX, IPython uses MathJax to render equations in the browser:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "~~~\n",
    "Here is one of Maxwell's Equations:\n",
    "$$\n",
    "\\nabla \\times \\mathbf{E} +\\frac{\\partial \\mathbf{B}}{\\partial t} = \\mathbf{J}\n",
    "$$\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "becomes\n",
    "\n",
    "Here is one of Maxwell's Equations:\n",
    "$$\n",
    "\\nabla \\times \\mathbf{E} +\\frac{\\partial \\mathbf{B}}{\\partial t} = \\mathbf{J}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But wait, there's more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember all that `bash` we learned? You can execute shell commands through the notebook by starting your code cell with a `!` (pronounced \"bang\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basic Data Analysis.ipynb\r\n",
      "Basic Programming in Python.ipynb\r\n",
      "Introduction to Python and IPython.ipynb\r\n",
      "Working with Array Data.ipynb\r\n",
      "\u001b[34mdata\u001b[m\u001b[m\r\n",
      "ecosystem.svg\r\n",
      "ipython_communication.svg\r\n",
      "ipython_local.svg\r\n",
      "ipython_remote.svg\r\n",
      "lemurfig.pdf\r\n",
      "lemurs.py\r\n",
      "lemurs.pyc\r\n",
      "marmoset.jpg\r\n",
      "summary_data.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/jmxp/code/DIBS_materials/python\r\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython is magic!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally (or sort of finally; there's much, much more), IPython has a set of extensions to the normal Python shell called [magics](http://ipython.org/ipython-doc/dev/interactive/tutorial.html#magic-functions). These come in especially handy for running scripts saved in `.py` files, debugging, and interfacing with R or Matlab."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}