notebooks/Storing Git commit information into Pandas' DataFrame.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Software version control systems contain a huge amount of evolutionary data. It's very common to mine these repositories to gain some insight about how the development of a software product works. But there is the need for some preprocessing of that data to avoid false analysis.\n",
    "\n",
    "That’s why I show you how to read the commit information of a Git repository into Pandas’ DataFrame!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idea\n",
    "The main idea is to use an existing Git library for Python that provides the necessary (and hopefully efficient) access to a Git repository. In this notebook, we'll use [GitPython](http://gitpython.readthedocs.io/en/stable/intro.html), because at first glance it seems easy to use and to do the things we need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our implementation strategy is straightforward: We try to avoid any functions as much as possible but try to use all the processing power Pandas delivers. The So let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create an initial DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we import our two main libraries for analysis: Pandas and GitPython."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With GitPython, you can access a Git repository via a <tt>Repo</tt> object. That's your entry point to the world of Git.\n",
    "\n",
    "For this notebook, we analyze the Sprint PetClinic repository that can be easily cloned to your local computer with a\n",
    "<pre>git clone https://github.com/spring-projects/spring-petclinic.git</pre>\n",
    "\n",
    "<tt>Repo</tt> needs at least the directory to your Git repository. I've added an additional argument <tt>odbt</tt> with the <tt>git.GitCmdObjectDB</tt>. With this, GitPython will be using a more performant approach for retrieving all the data (see [doc](http://gitpython.readthedocs.io/en/stable/tutorial.html#object-databases) for more details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<git.Repo \"C:\\dev\\repos\\spring-petclinic\\.git\">"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repo = git.Repo(r'C:\\dev\\repos\\spring-petclinic', odbt=git.GitCmdObjectDB)\n",
    "repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To transform the complete repository into Pandas' <tt>DataFrame</tt>, we simply iterate over all commits of the <tt>master</tt> branch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ffa967c94b65a70ea6d3b44275632821838d9fd3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fd1c742d4f8d193eb935519909c15302b783cd52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f792522b3dffca918f52010c8593999088034e19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>75912a06c5613a2ea1305ad4d8ad6bc4be7765ce</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>443d35eae23c874ed38305fbe75216339c41beaf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        raw\n",
       "0  ffa967c94b65a70ea6d3b44275632821838d9fd3\n",
       "1  fd1c742d4f8d193eb935519909c15302b783cd52\n",
       "2  f792522b3dffca918f52010c8593999088034e19\n",
       "3  75912a06c5613a2ea1305ad4d8ad6bc4be7765ce\n",
       "4  443d35eae23c874ed38305fbe75216339c41beaf"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "commits = pd.DataFrame(repo.iter_commits('master'), columns=['raw'])\n",
    "commits.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our <tt>raw</tt> column now contains all the commits as PythonGit's <tt>Commit</tt> Objects (to be more accurate: references to these objects). The string representation is coincidental the SHA key of the commit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Investigate commit data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the last commit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<git.Commit \"ffa967c94b65a70ea6d3b44275632821838d9fd3\">"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit = commits.ix[0, 'raw']\n",
    "last_commit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such a <tt>Commit</tt> object is our entry point for retrieving further data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wraps a git Commit object.\n",
      "\n",
      "    This class will act lazily on some of its attributes and will query the\n",
      "    value on demand only if it involves calling the git binary.\n"
     ]
    }
   ],
   "source": [
    "print(last_commit.__doc__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It provides all data we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('tree',\n",
       " 'author',\n",
       " 'authored_date',\n",
       " 'author_tz_offset',\n",
       " 'committer',\n",
       " 'committed_date',\n",
       " 'committer_tz_offset',\n",
       " 'message',\n",
       " 'parents',\n",
       " 'encoding',\n",
       " 'gpgsig')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.__slots__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "E. g. basic data like the commit message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'spring-petclinic-angular1 repo renamed to spring-petclinic-angularjs'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the date of the commit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2017, 4, 12, 21, 41, tzinfo=<git.objects.util.tzoffset object at 0x0000025943CEC198>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.committed_datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some information about the author."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Antoine Rey'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.author.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'antoine.rey@gmail.com'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.author.email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or file statistics about the commit,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'readme.md': {'deletions': 1, 'insertions': 1, 'lines': 2}}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_commit.stats.files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fill the DataFrame with data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check how fast we can retrieve all the authors from the commit's data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 62.5 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "commits['author'] = commits['raw'].apply(lambda x: x.author.name)\n",
    "commits.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go further and retrieve some more data (<tt>DataFrame</tt> is transposed / rotated via a <tt>T</tt> for displaying reasons)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 78.1 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "commits['email'] = commits['raw'].apply(lambda x: x.author.email)\n",
    "commits['committed_date'] = commits['raw'].apply(lambda x: pd.to_datetime(x.committed_datetime))\n",
    "commits['message'] = commits['raw'].apply(lambda x: x.message)\n",
    "commits['sha'] = commits['raw'].apply(lambda x: str(x))\n",
    "commits.head(2).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dead easy and reasonable fast, but what about the modified files? Let's challenge our computer a little bit more by extracting the statistics data about every commit. The <tt>Stats</tt> object contains all the touched files per commit including the information about the number of lines that were either inserted or deleted.\n",
    "\n",
    "Additionally, we need some tricks to get the data we need. For this, I guide you step by step through this approach. The main idea is to retrieve the real statistics data (not only the object's references) and temporarily store these statistics information as Pandas' <tt>Series</tt>. Then we take another round to transform this data to use it in <tt>DataFrame</tt>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cracking the <tt>stats</tt> files statistic object\n",
    "This step is a little bit tricky and was found only by a good amount of trial and error. But it works in the end as we will see. The goal is to unpack the information in the <tt>stats</tt> object into nice columns of out <tt>DataFrame</tt> via the <tt>Series#apply</tt> method. I'll show you step by step how this works in principle (albeit it will work a little bit different when using the <tt>apply</tt> approach).\n",
    "\n",
    "As seen above, we have access to every file modification of each commit. In the end, it's a dictionary with the filename as the key and a dictionary of the change attributes as values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'src/main/webapp/WEB-INF/tags/menu.tag': {'deletions': 2,\n",
       "  'insertions': 2,\n",
       "  'lines': 4}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_commit = commits.ix[56, 'raw']\n",
    "some_commit.stats.files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the dictionary of dictionaries in two steps. We have to keep in mind that all tricky data transformation is highly dependent on the right <tt>index</tt>. But first things first.\n",
    "\n",
    "First, to the outer dictionary: We create a <tt>Series</tt> of the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "src/main/webapp/WEB-INF/tags/menu.tag    {'insertions': 2, 'deletions': 2, 'lines': 4}\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_as_series = pd.Series(some_commit.stats.files)\n",
    "dict_as_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, we wrap that series into a <tt>DataFrame</tt> (for <tt>index</tt> reasons):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>src/main/webapp/WEB-INF/tags/menu.tag</th>\n",
       "      <td>{'insertions': 2, 'deletions': 2, 'lines': 4}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                   0\n",
       "src/main/webapp/WEB-INF/tags/menu.tag  {'insertions': 2, 'deletions': 2, 'lines': 4}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_as_series_wrapped_in_dataframe = pd.DataFrame(dict_as_series)\n",
    "dict_as_series_wrapped_in_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, some magic occurs. We stack the <tt>DataFrame</tt>, meaning that we put our columns into our <tt>index</tt> which becomes a <tt>MultiIndex</tt>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "src/main/webapp/WEB-INF/tags/menu.tag  0    {'insertions': 2, 'deletions': 2, 'lines': 4}\n",
       "dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_dataframe = dict_as_series_wrapped_in_dataframe.stack()\n",
    "stacked_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex(levels=[['src/main/webapp/WEB-INF/tags/menu.tag'], [0]],\n",
       "           labels=[[0], [0]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_dataframe.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With some manipulation of the <tt>index</tt>, we achive what we need: an expansion of the rows for each file in a commit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>level_1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>src/main/webapp/WEB-INF/tags/menu.tag</td>\n",
       "      <td>{'insertions': 2, 'deletions': 2, 'lines': 4}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       level_0  \\\n",
       "level_1                                          \n",
       "0        src/main/webapp/WEB-INF/tags/menu.tag   \n",
       "\n",
       "                                                     0  \n",
       "level_1                                                 \n",
       "0        {'insertions': 2, 'deletions': 2, 'lines': 4}  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_dataframe.reset_index().set_index('level_1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this (dirty?) trick, we achieved that all files from the <tt>stats</tt> object can be assigned to the original <tt>index</tt> of our <tt>DataFrame</tt>.\n",
    "\n",
    "In the context of a call with the <tt>apply</tt> method, the command looks a little bit different, but in the end, the result is the same (I took a commit with multiple modified files from the <tt>DataFrame</tt> just to show the tranformation a little bit better):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_1</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>readme.md</td>\n",
       "      <td>{'insertions': 2, 'deletions': 2, 'lines': 4}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>src/main/java/org/springframework/samples/petc...</td>\n",
       "      <td>{'insertions': 111, 'deletions': 0, 'lines': 111}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>src/main/webapp/WEB-INF/web.xml</td>\n",
       "      <td>{'insertions': 0, 'deletions': 118, 'lines': 118}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              level_1  \\\n",
       "64                                          readme.md   \n",
       "64  src/main/java/org/springframework/samples/petc...   \n",
       "64                    src/main/webapp/WEB-INF/web.xml   \n",
       "\n",
       "                                                    0  \n",
       "64      {'insertions': 2, 'deletions': 2, 'lines': 4}  \n",
       "64  {'insertions': 111, 'deletions': 0, 'lines': 111}  \n",
       "64  {'insertions': 0, 'deletions': 118, 'lines': 118}  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(commits[64:65]['raw'].apply(\n",
    "    lambda x: pd.Series(x.stats.files)).stack()).reset_index(level=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 23.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "stats = pd.DataFrame(commits['raw'].apply(\n",
    "    lambda x: pd.Series(x.stats.files)).stack()).reset_index(level=1)\n",
    "stats = stats.rename(columns={ 'level_1' : 'filename', 0 : 'stats_modifications'})\n",
    "stats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, this takes almost 30 seconds on my machine :-( (Help needed! Maybe there is a better way for doing this)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we extract the data from the <tt>stats_modification</tt> column. We do this by simply wrapping the dictionary in a <tt>Series</tt>, that will return the data needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "deletions     1\n",
       "insertions    1\n",
       "lines         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(stats.ix[0, 'stats_modifications'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With an <tt>apply</tt>, it looks a little bit different because we are applying the <tt>lambda</tt> function along the <tt>DataFrame</tt>'s <tt>index</tt>.\n",
    "\n",
    "We get a warning because there seems to be a problem with the ordering of the index. But I haven't found any errors so far with this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>deletions</th>\n",
       "      <th>insertions</th>\n",
       "      <th>lines</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>21</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   deletions  insertions  lines\n",
       "0          1           1      2\n",
       "1          0           1      1\n",
       "2          0          10     10\n",
       "2          3          21     24\n",
       "2          3           0      3\n",
       "3          1           1      2\n",
       "3          9          11     20"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_modifications = stats['stats_modifications'].apply(lambda x: pd.Series(x))\n",
    "stats_modifications.head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We join the newly created data with the existing one with a <tt>join</tt> method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>stats_modifications</th>\n",
       "      <th>deletions</th>\n",
       "      <th>insertions</th>\n",
       "      <th>lines</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>readme.md</td>\n",
       "      <td>{'insertions': 1, 'deletions': 1, 'lines': 2}</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>{'insertions': 1, 'deletions': 0, 'lines': 1}</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>{'insertions': 10, 'deletions': 0, 'lines': 10}</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>{'insertions': 10, 'deletions': 0, 'lines': 10}</td>\n",
       "      <td>3</td>\n",
       "      <td>21</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>{'insertions': 10, 'deletions': 0, 'lines': 10}</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename                              stats_modifications  deletions  \\\n",
       "0  readme.md    {'insertions': 1, 'deletions': 1, 'lines': 2}          1   \n",
       "1    pom.xml    {'insertions': 1, 'deletions': 0, 'lines': 1}          0   \n",
       "2    pom.xml  {'insertions': 10, 'deletions': 0, 'lines': 10}          0   \n",
       "2    pom.xml  {'insertions': 10, 'deletions': 0, 'lines': 10}          3   \n",
       "2    pom.xml  {'insertions': 10, 'deletions': 0, 'lines': 10}          3   \n",
       "\n",
       "   insertions  lines  \n",
       "0           1      2  \n",
       "1           1      1  \n",
       "2          10     10  \n",
       "2          21     24  \n",
       "2           0      3  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = stats.join(stats_modifications)\n",
    "stats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we get rid of the now obsolete <tt>stats_modifications</tt> columns..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>deletions</th>\n",
       "      <th>insertions</th>\n",
       "      <th>lines</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>readme.md</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>3</td>\n",
       "      <td>21</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pom.xml</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  deletions  insertions  lines\n",
       "0  readme.md          1           1      2\n",
       "1    pom.xml          0           1      1\n",
       "2    pom.xml          0          10     10\n",
       "2    pom.xml          3          21     24\n",
       "2    pom.xml          3           0      3"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del(stats['stats_modifications'])\n",
    "stats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...we join the existing <tt>DataFrame</tt> with the <tt>stats</tt> information (transposed for displaying reasons)..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>raw</th>\n",
       "      <td>ffa967c94b65a70ea6d3b44275632821838d9fd3</td>\n",
       "      <td>fd1c742d4f8d193eb935519909c15302b783cd52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <td>Antoine Rey</td>\n",
       "      <td>Antoine Rey</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>email</th>\n",
       "      <td>antoine.rey@gmail.com</td>\n",
       "      <td>antoine.rey@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>committed_date</th>\n",
       "      <td>2017-04-12 21:41:00+02:00</td>\n",
       "      <td>2017-03-06 08:12:14+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>message</th>\n",
       "      <td>spring-petclinic-angular1 repo renamed to spri...</td>\n",
       "      <td>Do not fail maven build when git directing is ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sha</th>\n",
       "      <td>ffa967c94b65a70ea6d3b44275632821838d9fd3</td>\n",
       "      <td>fd1c742d4f8d193eb935519909c15302b783cd52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>filename</th>\n",
       "      <td>readme.md</td>\n",
       "      <td>pom.xml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>deletions</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>insertions</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lines</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                0  \\\n",
       "raw                      ffa967c94b65a70ea6d3b44275632821838d9fd3   \n",
       "author                                                Antoine Rey   \n",
       "email                                       antoine.rey@gmail.com   \n",
       "committed_date                          2017-04-12 21:41:00+02:00   \n",
       "message         spring-petclinic-angular1 repo renamed to spri...   \n",
       "sha                      ffa967c94b65a70ea6d3b44275632821838d9fd3   \n",
       "filename                                                readme.md   \n",
       "deletions                                                       1   \n",
       "insertions                                                      1   \n",
       "lines                                                           2   \n",
       "\n",
       "                                                                1  \n",
       "raw                      fd1c742d4f8d193eb935519909c15302b783cd52  \n",
       "author                                                Antoine Rey  \n",
       "email                                       antoine.rey@gmail.com  \n",
       "committed_date                          2017-03-06 08:12:14+00:00  \n",
       "message         Do not fail maven build when git directing is ...  \n",
       "sha                      fd1c742d4f8d193eb935519909c15302b783cd52  \n",
       "filename                                                  pom.xml  \n",
       "deletions                                                       0  \n",
       "insertions                                                      1  \n",
       "lines                                                           1  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "commits = commits.join(stats)\n",
    "commits.head(2).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and come to an end by deleting the <tt>raw</tt> data column, too (and also transposed for displaying reasons)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <td>Antoine Rey</td>\n",
       "      <td>Antoine Rey</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>email</th>\n",
       "      <td>antoine.rey@gmail.com</td>\n",
       "      <td>antoine.rey@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>committed_date</th>\n",
       "      <td>2017-04-12 21:41:00+02:00</td>\n",
       "      <td>2017-03-06 08:12:14+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>message</th>\n",
       "      <td>spring-petclinic-angular1 repo renamed to spri...</td>\n",
       "      <td>Do not fail maven build when git directing is ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sha</th>\n",
       "      <td>ffa967c94b65a70ea6d3b44275632821838d9fd3</td>\n",
       "      <td>fd1c742d4f8d193eb935519909c15302b783cd52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>filename</th>\n",
       "      <td>readme.md</td>\n",
       "      <td>pom.xml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>deletions</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>insertions</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lines</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                0  \\\n",
       "author                                                Antoine Rey   \n",
       "email                                       antoine.rey@gmail.com   \n",
       "committed_date                          2017-04-12 21:41:00+02:00   \n",
       "message         spring-petclinic-angular1 repo renamed to spri...   \n",
       "sha                      ffa967c94b65a70ea6d3b44275632821838d9fd3   \n",
       "filename                                                readme.md   \n",
       "deletions                                                       1   \n",
       "insertions                                                      1   \n",
       "lines                                                           2   \n",
       "\n",
       "                                                                1  \n",
       "author                                                Antoine Rey  \n",
       "email                                       antoine.rey@gmail.com  \n",
       "committed_date                          2017-03-06 08:12:14+00:00  \n",
       "message         Do not fail maven build when git directing is ...  \n",
       "sha                      fd1c742d4f8d193eb935519909c15302b783cd52  \n",
       "filename                                                  pom.xml  \n",
       "deletions                                                       0  \n",
       "insertions                                                      1  \n",
       "lines                                                           1  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del(commits['raw'])\n",
    "commits.head(2).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we're finished! A <tt>DataFrame</tt> that contains all the repository information needed for further analysis!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2228366 entries, 0 to 557\n",
      "Data columns (total 9 columns):\n",
      "author            object\n",
      "email             object\n",
      "committed_date    object\n",
      "message           object\n",
      "sha               object\n",
      "filename          object\n",
      "deletions         float64\n",
      "insertions        float64\n",
      "lines             float64\n",
      "dtypes: float64(3), object(6)\n",
      "memory usage: 170.0+ MB\n"
     ]
    }
   ],
   "source": [
    "commits.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end, we still have our commits from the beginning, but with all information that we can work on in another notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "558"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(commits.index.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Store for later usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, we just store the <tt>DataFrame</tt> into a h5 format with compression for later usage (we get a warning because of the string objects we're using, but that's no problem AFAIK)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\dev\\Anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:1101: PerformanceWarning: \n",
      "your performance may suffer as PyTables will pickle object types that it cannot\n",
      "map directly to c-types [inferred_type->mixed,key->block1_values] [items->['author', 'email', 'committed_date', 'message', 'sha', 'filename']]\n",
      "\n",
      "  return pytables.to_hdf(path_or_buf, key, self, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "commits.to_hdf(\"data/commits.h5\", 'commits', mode='w', complevel=9, complib='zlib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# All in one code block\n",
    "This notebook is really long because it includes a lot of explanations. But if you just need the code to extract a Git repository, here it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\dev\\Anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:1101: PerformanceWarning: \n",
      "your performance may suffer as PyTables will pickle object types that it cannot\n",
      "map directly to c-types [inferred_type->mixed,key->block1_values] [items->['author', 'email', 'committed_date', 'message', 'sha', 'filename']]\n",
      "\n",
      "  return pytables.to_hdf(path_or_buf, key, self, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import git\n",
    "\n",
    "repo = git.Repo(r'C:\\dev\\repos\\spring-petclinic', odbt=git.GitCmdObjectDB)\n",
    "\n",
    "commits = pd.DataFrame(repo.iter_commits('master'), columns=['raw'])\n",
    "commits['author'] = commits['raw'].apply(lambda x: x.author.name)\n",
    "commits['email'] = commits['raw'].apply(lambda x: x.author.email)\n",
    "commits['committed_date'] = commits['raw'].apply(lambda x: pd.to_datetime(x.committed_datetime))\n",
    "commits['message'] = commits['raw'].apply(lambda x: x.message)\n",
    "commits['sha'] = commits['raw'].apply(lambda x: str(x))\n",
    "\n",
    "stats = pd.DataFrame(commits['raw'].apply(lambda x: pd.Series(x.stats.files)).stack()).reset_index(level=1)\n",
    "stats = stats.rename(columns={ 'level_1' : 'filename', 0 : 'stats_modifications'})\n",
    "stats_modifications = stats['stats_modifications'].apply(lambda x: pd.Series(x))\n",
    "stats = stats.join(stats_modifications)\n",
    "del(stats['stats_modifications'])\n",
    "\n",
    "commits = commits.join(stats)\n",
    "del(commits['raw'])\n",
    "\n",
    "commits.to_hdf(\"data/commits.h5\", 'commits', mode='w', complevel=9, complib='zlib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "I hope you aren't demotivated now by my Pandas' approach for extracting data from Git repositories. Agreed, the <tt>stats</tt> object is little unconventional to work with (and there may be better ways for doing it), but I think in the end, the result is pretty useful."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}