Lecture notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Welcome to Advanced Software Engineering\n",
    "## Analyzing Version Control System Data\n",
    "\n",
    "![](https://comidoc.net/static/assets/thumbs/480/webp/2267906_505c.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning Objectives\n",
    "\n",
    "After this session (+ exercises) the student will be able to:\n",
    "\n",
    "  * Understand how research analyzing VCS histories collects the underlying data.\n",
    "  * Export the history from Git repositories.\n",
    "  * Apply scripts and programs in various languages to clean and pre-process the exported data.\n",
    "  * Apply scripts and programs in various languages to analyze Git VCS data with respect to certain hypotheses/research questions.\n",
    "  * Interpret the analysis results to either better understand current practices or to suggest actionable changes of current practices in software engineering teams.\n",
    "  \n",
    "-------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What are we doing today?\n",
    "\n",
    "## Motivation\n",
    "\n",
    "You all read the paper C. Bird et al. [_\"Don't Touch My Code! Examining the Effects of Ownership on Software Quality\"_](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bird2011dtm.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "\n",
    "IFrame('./bird2011dtm.pdf', width='100%', height=850)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What did you take away from the paper?\n",
    "\n",
    "<!--\n",
    "  * Relationship between single/shared person size of commits vs. failure rates\n",
    "    - The more minor contirbutors the higher the failure rate\n",
    "  * Interesting: how ownership is defined\n",
    "    - Correlation of low-knowledge work and failure rates\n",
    "  * Main cause of failures how people commit but not who owns the most\n",
    "  * Ownership by commits in single files?\n",
    "    - No mapping between authors and specific files?\n",
    "    - ???\n",
    "-->\n",
    "\n",
    "  * \n",
    "  * \n",
    "  * "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### Some paragraphs that I think are important...\n",
    "\n",
    "  > \"How much does ownership affect quality?\"\n",
    "    \n",
    "  > Ownership is a general term used to describe whether one person has responsibility for a software component, or if there is no one clearly responsible developer. Within Microsoft, we have found that when more people work on a binary, it has more failures.\n",
    "  \n",
    "  > Interestingly, unlike some aspects of software which are known to be related to defects such as dependency complexity, or size, ownership is something that can be deliberately changed by modifying processes and policies. Thus, the answer to the question: “How much does ownership affect quality?” is important as it is actionable. Managers and team leads can make better decisions about how to govern a project by knowing the answer. \n",
    "  \n",
    "   \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "  > We require several types of data. The most important data are the commit histories and software failures. Software repositories record the contents of every change made to a piece of software, along with the change author, the time of change, and an associated log message that may be indicative of the type of change (e.g. introducing a feature, or fixing a bug). We collected the number of changes made by each developer to each source file and used a mapping of source files to binaries in order to determine the number of changes made by each developer to each binary. Although the source code management system uses branches heavily, we only recorded changes from developers that were edits to the source code. Branching operations (e.g. branching and merging) were not counted as changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### After reading the paper, do you know how the researchers collected the data on which they base their analysis?\n",
    "\n",
    "The big issue is now:\n",
    "<p style=\"padding:6px; color: grey; background-color: white; border: orange 2px solid\">\"Alright, I can read the article and understand the experiment setup and results, but how would I run a similar experiment in my organization?\"</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# What are we doing today?\n",
    "\n",
    "## What are we really doing now?\n",
    "\n",
    "We will -on a running example- perform a set of analyses of data from Git, a particular version control system (VCS). Everything in this lecture should be -with minor adaptations- applicable to other VCS, such as Mercurial, Microsoft TFS, etc.\n",
    "\n",
    "You will learn how to:\n",
    "\n",
    "  * Export the history from Git repositories.\n",
    "  * Apply scripts and programs in various languages to clean and pre-process the exported data.\n",
    "  * Apply scripts and programs in various languages to analyze Git VCS data with respect to certain hypotheses/research questions.\n",
    "  * Interpret the analysis results to either better understand current practices or to suggest actionable changes of current practices in software engineering teams.\n",
    "  \n",
    "## Working together now\n",
    "\n",
    "As you have various operating systems installed on your computers and since I cannot support all of these in such a short time, we will work together on a remotely hosted virtual machine Linux environment.\n",
    "\n",
    "Navigate to: https://github.com/HelgeCPH/2019_ase_behavioural_analysis and press the button that says `launch binder`. In case you have a Linux or Unix installed on your computer you can run everything on your local machine right away.\n",
    "\n",
    "After waiting a short while click on the file called `Lecture notes.ipynb` now you should see a similar screen to mine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "--------------------------------------------------------\n",
    "\n",
    "\n",
    "# Getting the VCS Data\n",
    "\n",
    "In the lecture we use [Apache Airflow](https://airflow.apache.org) as an example software system.\n",
    "\n",
    "  > Airflow is a platform to programmatically author, schedule and monitor workflows.\n",
    "\n",
    "The project is accessible on Github: https://github.com/apache/airflow. That is, we make use of the Git VCS during this lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/apache/airflow.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -ltrha airflow/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Moving back in time \n",
    "\n",
    "We want to all work on the same \"view\" of the software so let's go back to the state of the repository on Friday Sept. 30th."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ checkout $(git -C airflow/ rev-list -n 1 --before=\"2023-10-20\" main)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "If you want to perform an analyzes for a certain period only, you can add the `--after` switch too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## The revisions of a software system, organizational history\n",
    "\n",
    "```bash\n",
    "git log\n",
    "```\n",
    "\n",
    "Note, you have to switch into the directory of the repository that you want to study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ log | head -30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What do we see and how to read it?\n",
    "\n",
    "```\n",
    "commit 9a5ba2c5375fe2d8fdde294f474c3977e9512747\n",
    "Author: anukrati1507 <47574465+anukrati1507@users.noreply.github.com>\n",
    "Date:   Fri Oct 20 15:32:43 2023 +0530\n",
    "\n",
    "    Adding name to company list (#35078)\n",
    "    \n",
    "    Co-authored-by: anukrati15 <anukrati.saxena@cloudera.com>\n",
    "\n",
    "commit c0a3930f064a94e56b58075d579d38de0d2a444c\n",
    "Author: Shubham Raj <48172486+shubhamraj-git@users.noreply.github.com>\n",
    "Date:   Fri Oct 20 15:21:58 2023 +0530\n",
    "\n",
    "    Added shubhamraj-git to the company list (#35077)\n",
    "\n",
    "commit 9a65fea6b01de51bcbefa3c104920544ccf3b39e\n",
    "Author: Samit Maharjan <mahjsamit_2002@outlook.com>\n",
    "Date:   Fri Oct 20 15:09:31 2023 +0530\n",
    "\n",
    "    Added new user under Cloudera to INTHEWILD.md (#35076)\n",
    "    \n",
    "    Co-authored-by: samit-maharjan <samit.maharjan@cloudera.com>\n",
    "\n",
    "commit a66437222b5f8250fd9dfd26925c8ba9017eebb2\n",
    "Author: Andrey Anshin <Andrey.Anshin@taragol.is>\n",
    "Date:   Fri Oct 20 13:33:22 2023 +0400\n",
    "\n",
    "    Use base classes for AWS Lambda Operators/Sensors (#34890)\n",
    "    \n",
    "    * Use base classes for AWS Lambda Operators/Sensors\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note, we are looking on the master/main branch only at the moment. That is, you see only commit messages on that single branch (we will talk more about these in three weeks).\n",
    "\n",
    "```bash\n",
    "git log --all\n",
    "```\n",
    "\n",
    "```bash\n",
    "git log --branches --remotes --tags --graph --oneline --decorate\n",
    "```\n",
    "\n",
    "\n",
    "### What are all these switches?\n",
    "\n",
    "You can read the help of `git log` via:\n",
    "\n",
    "```bash\n",
    "git help log\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "All the `git log`s we looked at so far are meant for humans to read and interpret.\n",
    "\n",
    "However, we want to automatically analyze the logs to infer information out of it that is hidden in the logs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "---------------------------------------------\n",
    "\n",
    "# Collecting, Cleansing, and Pre-processing the Data\n",
    "\n",
    "## Exporting the `git log` into a machine readable format\n",
    "\n",
    "Read the `PRETTY FORMATS` section of the `git log` help for possible placeholders in the format string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ log --all --pretty=format:'%s' > data/all_commit_msgs.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head data/all_commit_msgs.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ log --all --pretty=format:'\"%h\",\"%s\"' > data/all_commit_msgs.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head data/all_commit_msgs.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ log --pretty=format:'\"%h\",\"%an\",\"%ad\"' \\\n",
    "    --date=short \\\n",
    "    --numstat > \\\n",
    "    data/airflow_evo.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -lh data/airflow_evo.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head -30 data/airflow_evo.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not completely suited to our needs yet. Let's convert it into a CSV file with one file per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python evo_log_to_csv.py data/airflow_evo.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -ltrh data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head data/airflow_evo.log.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not completely nice either, what are the following \"weird\" files?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"=>\" data/airflow_evo.log.csv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to \"clean up\" commits in which files have been moved so that we have a view of the current repository contents and do not lose old commit information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python repair_git_move.py data/airflow_evo.log.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -ltrh data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head data/airflow_evo.log_repaired.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"=>\" data/airflow_evo.log_repaired.csv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------------\n",
    "\n",
    "\n",
    "\n",
    "# Shell-based Analysis\n",
    "\n",
    "## How many revisions do we have in the repository?\n",
    "\n",
    "That is: How much does the software change?\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "  > In general, process measures based on the change history are more useful in predicting fault rates than product metrics of the code: For instance, the number of times code has been changed is a better indication of how many faults it will contain than is its length\n",
    "  >\n",
    "  > [Graves et al. _Predicting fault incidence using software change history_](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.116.9414&rep=rep1&type=pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cut -d, -f 1 data/airflow_evo.log_repaired.csv | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 1 | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 1 | sort | uniq | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How many revisions of each file do we have?\n",
    "\n",
    "For the twenty most often changed files, we could formulate a query as in the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 7 | sort | uniq -c | sort -nr | tail -n +2 | head -30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shell-based Analysis\n",
    "\n",
    "## Who are the ten persons that contribute the most?\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "  > Our conclusion: adding more programmers improves the chances that a FOSS project will be successful.\n",
    "  >\n",
    "  > Schweik et al. _Brooks' Versus Linus' Law: An Empirical Test of Open Source Projects_\n",
    "\n",
    "\n",
    "  > ...files with changes from nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer than nine developers, indicating that many developers changing code may have a detrimental effect on the system's security\n",
    "  >\n",
    "  > [_Secure Open Source Collaboration: An Empirical Study of Linus’ Law_](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.965.7992&rep=rep1&type=pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq -c | sort -n | tail -n10 | tac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, on MacOS and other Unixes you can let tail reverse directly and you do not have the `tac` command by default.\n",
    "\n",
    "```bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq -c | sort -n | tail -r -n10\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq -c | sort -n | tail -n10 | tac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn!\n",
    "\n",
    "  * How many contributors are there in total?\n",
    " \n",
    "<!--tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq | wc -l -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  * Who is contributing the least?\n",
    "\n",
    "<!-- tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq -c | sort -n -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  * Google-fu question: What is the median contribution value?\n",
    "  \n",
    "<!-- tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | sort | uniq -c | cut -d '\"' -f 1 | python -c\"import sys,statistics;print(statistics.median([int(i) for i in sys.stdin.readlines()]))\" -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tail -n +2 data/airflow_evo.log_repaired.csv | cut -d, -f 2 | ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scripted Analysis 1: \n",
    "\n",
    "## What are people doing?\n",
    "\n",
    "We can analyze the commit messages to get a feeling for what people are doing.\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "  > If we look at the commit cloud, we see that certain terms dominate. What you'll learn right now is by no means scientific, but it's a useful heuristic: the words that stand out tell you where you spend your time.\n",
    "  >\n",
    "  > A. Tornhill _Your Code as a Crime Scene_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cat data/all_commit_msgs.txt | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"[Ff]ix\" data/all_commit_msgs.txt | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"[Ff]ix\" data/all_commit_msgs.txt | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"[Ee]rror\" data/all_commit_msgs.txt | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "grep \"[Bb]ug\" data/all_commit_msgs.txt | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "With a bit enhanced word natural-language processing (NLP) based word extraction, we can condense the following word frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python word_freq.py data/all_commit_msgs.txt nlp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's create a wordcloud for the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python wordcloud_gen.py data/all_commit_msgs.txt nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -ltrh out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](out/all_commit_msgs.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scripted Analysis 2 \n",
    "\n",
    "## Sentiment Analysis of Commit Messages\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "Commit messages can give insight into the state of your team and likely also into the state of your software.\n",
    "\n",
    "For example they can shed light on when your team members are \"working best\" and in which setup:\n",
    "\n",
    "  > Our results show that projects developed in Java tend to have more negative commit comments, and that projects that have more distributed teams tend to have a higher positive polarity in their emotional content. Additionally, we found that commit comments written on Mondays tend to a more negative emotion.\n",
    "  >\n",
    "  > [E. Guzman et al. _\"Sentiment Analysis of Commit Comments in GitHub: An Empirical Study\"_](https://www.researchgate.net/profile/Emitza_Guzman/publication/266657943_Sentiment_analysis_of_commit_comments_in_GitHub_An_empirical_study/links/5b8305ba4585151fd134f10c/Sentiment-analysis-of-commit-comments-in-GitHub-An-empirical-study.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python commit_sentiments.py data/all_commit_msgs.csv > data/all_commit_sentiment_msgs.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -ltrh data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head data/all_commit_sentiment_msgs.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "df = pd.read_csv('data/all_commit_sentiment_msgs.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which are the commits containing negative commit messages?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.polarity < -0.5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which are the commits containing positive commit messages?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.polarity > 0.5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Possible Project:**\n",
    "\n",
    "Train an NLP sentiment analysis model on a properly annotated commit history to get better results.\n",
    "\n",
    "#### Which is the commit with the most negative commit message?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_negative = df[df.polarity == df.polarity.min()]\n",
    "print(most_negative.msg.values)\n",
    "most_negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the corresponding commit?\n",
    "\n",
    "Now we are crossing information from two datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git -C airflow/ log d4dfe2654"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_df = pd.read_csv('data/airflow_evo.log_repaired.csv')\n",
    "complete_df[complete_df.hash == most_negative.index[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scripted Analysis 2 \n",
    "\n",
    "## A Project's Bus-factor\n",
    "\n",
    "\n",
    "One of the early mentions of the term in a real project [_\"If Guido was hit by a bus?\"_](https://legacy.python.org/search/hypermail/python-1994q2/1040.html)\n",
    "\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "  > ...the number of people on your team that have to be hit by a truck (or quit) before the project is in serious trouble...\n",
    "  >\n",
    "  > L. Williams and R. Kessler, Pair Programming Illuminated. Wesley, 2003.\n",
    "Addison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with computing code ownership. There are different possibilities to define and measure code ownership:\n",
    "\n",
    "a)\n",
    "  > One measure of ownership is  how  much  of  the  development  activity  for  a  component comes from one developer.  If one developer makes 80% of the changes to a component, then we say that the component has high ownership.  The other way that we measure ownership  is  by  determining  how  many  low-expertise  developers are working on a component.  If many developers are all making few changes to a component, then there are many non-experts working on the component and we label the component as having low ownership\n",
    "  >\n",
    "  > C. Bird et al. [_\"Don't Touch My Code! Examining the Effects of Ownership on Software Quality\"_]( https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bird2011dtm.pdf)\n",
    "\n",
    "\n",
    "  > Proportion of Ownership – The proportion of ownership  (or  simply  ownership)  of  a  contributor  for a particular component is the ratio of number of commits that the contributor has made relative to the total number of commits for that component.\n",
    "  >\n",
    "  > C. Bird et al. [_\"Don't Touch My Code! Examining the Effects of Ownership on Software Quality\"_]( https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bird2011dtm.pdf)\n",
    "\n",
    "b)\n",
    "\n",
    "There are more elaborate measures for code ownership in the literature, for example:\n",
    "\n",
    "  > $DOA = 3.293 + 1.098 ∗ FA + 0.164 ∗ DL − 0.321 ∗ ln(1 + AC )$\n",
    "  >\n",
    "  > The degree of authorship of a developer d in a file f depends on three factors: first authorship (FA), number of deliveries (DL), and number of acceptances (AC). If d is the author of f, FA is 1; otherwise it is 0; DL is the number of changes in f made by D; and AC is the number of changes in f made by other developers.\n",
    "  > \n",
    "  > [G. Avelino et al. _What is the Truck Factor of Popular GitHub Applications? A First Assessment_](https://peerj.com/preprints/1233.pdf)\n",
    "  \n",
    "c) \n",
    "\n",
    "In *Your Code as a Crime Scene*, A. Tornhill suggests to compute a simple measure -which he calls it *knowledge ownership*- that counts the total amount of lines added through the history of a file. And additionally, per author the amounts of lines added by that author. Out of the two values he computes a the _knowledge ownership_ rate.\n",
    "\n",
    "\n",
    "\n",
    "For the following example, we go for the latter definition and in our computation we just say that the author with the highest _knowledge ownership_ rate on a file \"owns\" that file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = 'INTHEWILD.md'\n",
    "complete_df[complete_df.current == fname]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_rows = []\n",
    "\n",
    "for fname in set(complete_df.current.values):\n",
    "    view = complete_df[complete_df.current == fname]\n",
    "    sum_series = view.groupby(['author']).added.sum()\n",
    "    view_df = sum_series.reset_index(name='sum_added')\n",
    "    total_added = view_df.sum_added.sum()\n",
    "    \n",
    "    if total_added > 0:  # For binaries there are no lines counted\n",
    "        view_df['owning_percent'] = view_df.sum_added / total_added\n",
    "        owning_author = view_df.loc[view_df.owning_percent.idxmax()]\n",
    "        new_rows.append((fname, owning_author.author, owning_author.sum_added,\n",
    "                         total_added, owning_author.owning_percent))\n",
    "    # All binaries are silently skipped in this report..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "owner_df = pd.DataFrame(new_rows, columns=['artifact', 'main_dev', 'added', 'total_added', 'owner_rate'])\n",
    "owner_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "owner_freq_series = owner_df.groupby(['main_dev']).artifact.count()\n",
    "owner_freq_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "owner_freq_df = owner_freq_series.reset_index(name='owns_no_artifacts')\n",
    "owner_freq_df.sort_values(by='owns_no_artifacts', inplace=True)\n",
    "owner_freq_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "fig = plt.figure(figsize=(12, 3))\n",
    "plt.xticks(rotation=45)\n",
    "plt.xticks(size=3)\n",
    "plt.plot(owner_freq_df.main_dev, owner_freq_df.owns_no_artifacts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can -similar to \n",
    "<a href=\"https://ieeexplore.ieee.org/stamp)/stamp.jsp?arnumber=7503718\">G. Avelino et al. *A novel approach for estimating Truck Factors*</a>- start \"removing\" low-contributing authors from the dataset as long as still more than half of the files have an owner. The amount of remaining owners is the bus-factor of that project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "no_artifacts = len(owner_df.artifact)\n",
    "half_no_artifacts = no_artifacts // 2\n",
    "count = 0\n",
    "\n",
    "for owner, freq in owner_freq_df.values:\n",
    "    no_artifacts -= freq\n",
    "    if no_artifacts < half_no_artifacts:\n",
    "        break\n",
    "    else:\n",
    "        count += 1\n",
    "\n",
    "busfactor = len(owner_freq_df.main_dev) - count\n",
    "print(f\"The bus factor of Apache Airflow is: {busfactor}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git Analysis Beyound Shell-Scripts\n",
    "\n",
    "## Co-changeing files Analysis\n",
    "\n",
    "### Why does it matter?\n",
    "\n",
    "\n",
    "  > Our software evolution analysis approach enabled us to detect shortcomings of [software] such as architectural weaknesses, poorly designed inheritance hierarchies, or blurred interfaces of modules.\n",
    "  >\n",
    "  > H. Gall et al. _CVS release history data for detecting logical couplings_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic usage example of `PyDriller`:\n",
    "\n",
    "  > PyDriller is a Python framework that helps developers on mining software repositories. With PyDriller you can easily extract information from any Git repository, such as commits, developers, modifications, diffs, and source codes, and quickly export CSV files.\n",
    "  >\n",
    "  > https://pydriller.readthedocs.io/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydriller import Repository\n",
    "\n",
    "\n",
    "path_to_repo = \"./airflow\"\n",
    "for commit in Repository(path_to_repo).traverse_commits():\n",
    "    if \"[bot]\" in commit.author.name:\n",
    "        print(commit.hash, commit.author_date, commit.author.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from datetime import datetime\n",
    "\n",
    "\n",
    "\n",
    "def commits_to_df(path_to_repo, from_dt=None, to_dt=None):\n",
    "    commits = []\n",
    "    if from_dt and to_dt:\n",
    "        repo = Repository(path_to_repo, since=from_dt, to=to_dt)\n",
    "    else:\n",
    "        repo = Repository(path_to_repo)\n",
    "    for commit in tqdm(repo.traverse_commits()):\n",
    "        for f in commit.modified_files:\n",
    "            commits.append((commit.hash, commit.author_date, commit.author.name, f.old_path, f.new_path))\n",
    "    df = pd.DataFrame(commits, columns=[\"sha\", \"date\", \"author\", \"old_path\", \"new_path\"])\n",
    "    return df\n",
    "\n",
    "\n",
    "df = commits_to_df(path_to_repo, from_dt=datetime(2021, 1, 1, 0, 0, 0), to_dt=datetime(2021, 2, 1, 0, 0, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[(df.old_path == df.new_path) & (df.new_path.str.endswith(\".py\"))]\n",
    "df.drop(\"old_path\", axis=1, inplace=True)\n",
    "df.rename(columns = {\"new_path\":\"path\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code is adapted from https://www.feststelltaste.de/spotting-co-changing-files/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commits_df = pd.merge(df[[\"sha\", \"path\"]], df[[\"sha\", \"path\"]], left_on='sha', right_on='sha', suffixes=['','_other'], how='outer')\n",
    "commits_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commits_df = commits_df[commits_df[\"path\"] != commits_df[\"path_other\"]]\n",
    "commits_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commit_coupling = commits_df.groupby([\"path\", \"path_other\"]).count()\n",
    "commit_coupling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commit_coupling['all_changes'] = commit_coupling.groupby([\"path\"]).sha.transform('sum')\n",
    "commit_coupling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commit_coupling[\"ratio\"] = commit_coupling[\"sha\"] / commit_coupling[\"all_changes\"]\n",
    "commit_coupling.reset_index().sort_values(by=[\"ratio\", \"path\"], ascending=False, inplace=True)\n",
    "commit_coupling.rename(columns={\"sha\" : \"cochanging\"}, inplace=True)\n",
    "commit_coupling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commit_coupling.cochanging.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------------------------------------------\n",
    "\n",
    "\n",
    "# Your turn!\n",
    "\n",
    "![](http://giphygifs.s3.amazonaws.com/media/11M1k4fIwVqPF6/giphy.gif)\n",
    "\n",
    "Chose one or more software systems that are under version control with Git.\n",
    "\n",
    "Use the version history as done during class (if you want to in combination with the provided Python scripts) to analyze VCS history according to one of the small projects below.\n",
    "\n",
    "Base your analysis on the file `<your_system>_evo.log.csv` that you can create similar to the examples above like:\n",
    "\n",
    "```bash\n",
    "git clone <url to>/<your_system>.git\n",
    "\n",
    "git -C <your_system>/ log --pretty=format:'\"%h\",\"%an\",\"%ad\"' \\\n",
    "    --date=short \\\n",
    "    --numstat > \\\n",
    "    data/<your_system>_evo.log\n",
    "\n",
    "python evo_log_to_csv.py data/<your_system>_evo.log\n",
    "python repair_git_move.py data/<your_system>_evo.log.csv\n",
    "```\n",
    "\n",
    "You can implement your analysis in the languages and with technologies of your choice. (Likely your favorite scripting language comes in handy here.)\n",
    "\n",
    "If you do not like the shell-based analysis that relies on Git logs, you may want to consider tools, such as, [PyDriller](https://github.com/ishepard/pydriller), which provide a more high-level API for many of the actions presented in this class.\n",
    "\n",
    "\n",
    "\n",
    "I suggest that you could work on one of the following problems:\n",
    "\n",
    "\n",
    "### A) Can you suggest a team structure for a project?\n",
    "\n",
    "Persons who work often on the same artifacts should perhaps work physically together and not be distributed around the globe, see e.g., [M. Penta et al. _The Effect of Communication Overhead on Software Maintenance Project Staffing: a Search-Based Approach_](http://crest.cs.ucl.ac.uk/fileadmin/crest/sebasepaper/DiPentaHAQ07.pdf) and N. Nagappan et al. [_The Influence of Organizational Structure on Software Quality_](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2008-11.pdf).\n",
    "\n",
    "Write a script/program that suggests teams of size five to ten persons.\n",
    "\n",
    "Find a suitable categorization for all persons that rarely contribute to a project.\n",
    "\n",
    "Chapters 11-13 in A. Tornhill _Your Code as a Crime Scene_ will likely form a good basis for the task.\n",
    "\n",
    "\n",
    "### B) Hotspot Analysis\n",
    "\n",
    "Based on A. Tornhill _Your Code as a Crime Scene_ chapters 3-5, write a script/program that identifies and visualizes hotspots (often changing and complex artifacts)\n",
    "\n",
    "You can make use of [`cloc`](https://github.com/AlDanial/cloc) (installed into this VM) to compute the size (proxy for complexity) of a file:\n",
    "\n",
    "```bash\n",
    "./cloc --csv --by-file airflow\n",
    "```\n",
    "\n",
    "Alternatively, you can write another small script that computes the whitespace complexity of a file, as in chapter 6\n",
    "\n",
    "### C) Change TF Code Ownership Metric\n",
    "\n",
    "Change the above implementation of the Truck Factor metric so that it uses an ownership metric as suggested by Bird et al. That is, do not rely on the number of added lines but instead the number of changes (commits) of files.\n",
    "\n",
    "How do the resulting Truck Factor values change? Shall we use one ownership metric over the other?\n",
    "\n",
    "\n",
    "\n",
    "<!--\n",
    "### C) Architectural Decay\n",
    "\n",
    "Investigate potential architectural decay based on measures for temporal coupling chapters 7-8 (and 9-10) in A. Tornhill _Your Code as a Crime Scene_.\n",
    "\n",
    "Write a script/program that suggests artifacts that seem to be prone to architectural decay when their temporal couples increase drastically over an analysis period.\n",
    "-->\n",
    "\n",
    "\n",
    "\n",
    "### D) Reproduce Paper Results\n",
    "\n",
    "Take the initial paper from https://peerj.com/preprints/1233.pdf and the final version <a href=\"https://ieeexplore.ieee.org/stamp)/stamp.jsp?arnumber=7503718\">G. Avelino et al. _A novel approach for estimating Truck Factors_</a>. Download the given repositories, implement a computation of ownership as described in the papers, and finally compare how the bus-factors developed since the papers were published to now.\n",
    "\n",
    "You will find their tool implementation on Github: [https://github.com/aserg-ufmg/Truck-Factor](https://github.com/aserg-ufmg/Truck-Factor)\n",
    "\n",
    "### E) Relationships of Contributors and Components\n",
    "\n",
    "Bird et al. have the hypothesis that:\n",
    "\n",
    "  > Minor contributors to components will be Major contributors to other components that are related through dependency relationships\n",
    "  \n",
    "Does that hold for open-source projects too? Think about a simple definition of \"component\" that allows you to generate results quickly.\n",
    "\n",
    "### E) Freestyle\n",
    "\n",
    "Formulate a problem that you want to investigate based on the history of a VCS. An example for such a problem formulation could be:\n",
    "  \n",
    "  * Can we find a relation between sentiments in commit messages and complexity of the files in the commit?\n",
    "\n",
    "Write a script/program that implements the analysis for the given problem and generates suitable result data or plots.\n",
    "\n",
    "\n",
    "\n",
    "<!--\n",
    "## Formalities\n",
    "\n",
    "  * Chose a project now and work on it for the rest of today's session.\n",
    "-->\n",
    "<!--  * I will be there for the session on Thursday to help with practicalities. -->\n",
    "<!--\n",
    "* Prepare a small presentation of your results and findings for next Monday. We will start the session with a small presentation per group.\n",
    "  -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I do not have Linux on my computer...\n",
    "\n",
    "All of the scripts and advise in this material assumes that you are working on a Linux/Unix environment. In case you are on Windows, there are many different ways of setting up such an environment. You can find a guide on setting up virtual machines with -amongst others- Ubuntu Linux here: http://itu.dk/people/ropf/blog/vagrant_install.html. I recommend that you setup such an environment as you as software engineers should be comfortable and able to work in various environments depending on the needs of your future companies/clients.\n",
    "\n",
    "For small analysis that do not require a lot of resources, you might want to fork and adapt this repository, so that you can run your code on mybinder."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}