microtask-2/microtask-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## microtask-2\n",
    "\n",
    "Create a Python script to execute Perceval via its Python interface using the GitLab and GitHub backends."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Perceval is a Python module for retrieving data from various data source like Git repositories and GitHub, GitLab projects, Discourse, StackOverflow. You can get to know more about the project from [chaoss/grimoirelab-perceval](https://github.com/chaoss/grimoirelab-perceval).\n",
    "\n",
    "In this notebook, we will use the github and gitlab backend modules to extract the information from a selected repository. The documentation to the perceval package can be found at https://perceval.readthedocs.io/en/latest/perceval.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import json\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GitHub Backend\n",
    "\n",
    "The GitHub repository we are gonna target is [amfoss/gitlit](https://github.com/amfoss/gitlit).\n",
    "\n",
    "> The documentation for the GitHub Backend can be found [here](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#module-perceval.backends.core.github)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need the personal access token to use the GitHub Backend of Perceval without having any problem with the GitHub API. You can generate a new token [here](https://github.com/settings/tokens/new). Please follow these steps if you need any help, [GitHub: PAT Help](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line).\n",
    "\n",
    "One of the latest features added to the GitHub backend is the possiblity to pass a list of tokens.\n",
    "[chaoss/grimoirelab-perceval#546](https://github.com/chaoss/grimoirelab-perceval/issues/546)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_TOKENS = [\"12345678abcdefgh\", \"12345678abcdefgh\"]\n",
    "# your github api tokens goes here\n",
    "\n",
    "OWNER = \"amfoss\"\n",
    "REPOSITORY = \"gitlit\"\n",
    "\n",
    "from_date = datetime(2018,1,1)\n",
    "to_date = datetime(2020,1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from perceval.backends.core.github import GitHub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "github_backend = GitHub(owner = OWNER,\n",
    "                        repository = REPOSITORY,\n",
    "                        api_token = API_TOKENS,\n",
    "                        sleep_for_rate = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's printout some basic information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "amfoss\n",
      "gitlit\n",
      "https://github.com/amfoss/gitlit\n"
     ]
    }
   ],
   "source": [
    "print(github_backend.owner)\n",
    "print(github_backend.repository)\n",
    "print(github_backend.origin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['issue', 'pull_request', 'repository']\n"
     ]
    }
   ],
   "source": [
    "print(github_backend.categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetching Issues from GitHub Backend\n",
    "\n",
    "The Issues of a project can be fetched using GitHub Backend. [Reference](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#perceval.backends.core.github.GitHub.CATEGORIES).\n",
    "\n",
    "\n",
    "We can use the fetch function with the argument `category = 'issue'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ISSUES COUNT: 34\n"
     ]
    }
   ],
   "source": [
    "issues = github_backend.fetch(\n",
    "    category = 'issue', \n",
    "    from_date = from_date,\n",
    "    to_date = to_date\n",
    "    )\n",
    "\n",
    "issues_list = list(issues)\n",
    "print(\"ISSUES COUNT:\", len(issues_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In GitHub, every pull request is an issue, but not every issue is a pull request. For this reason, \"shared\" actions for both features, like manipulating assignees, labels and milestones, are provided within the Issues API.\n",
    "\n",
    "https://developer.github.com/v3/pulls/#labels-assignees-and-milestones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the information which can be drawn from the data that was fetched by the Perceval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['backend_name', 'backend_version', 'perceval_version', 'timestamp', 'origin', 'uuid', 'updated_on', 'classified_fields_filtered', 'category', 'search_fields', 'tag', 'data'])\n",
      "\n",
      "dict_keys(['url', 'repository_url', 'labels_url', 'comments_url', 'events_url', 'html_url', 'id', 'node_id', 'number', 'title', 'user', 'labels', 'state', 'locked', 'assignee', 'assignees', 'milestone', 'comments', 'created_at', 'updated_at', 'closed_at', 'author_association', 'body', 'reactions', 'user_data', 'assignee_data', 'assignees_data', 'comments_data', 'reactions_data'])\n",
      "\n",
      "dict_keys(['login', 'id', 'node_id', 'avatar_url', 'gravatar_id', 'url', 'html_url', 'followers_url', 'following_url', 'gists_url', 'starred_url', 'subscriptions_url', 'organizations_url', 'repos_url', 'events_url', 'received_events_url', 'type', 'site_admin'])\n"
     ]
    }
   ],
   "source": [
    "issue = issues_list[-2]\n",
    "\n",
    "# pprint(issue)\n",
    "\n",
    "print(issue.keys())\n",
    "print()\n",
    "print(issue['data'].keys())\n",
    "print()\n",
    "print(issue['data']['user'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The title of the issue: Add Logo to NavBar\n",
      "The issue is opened by: harshithpabbati\n",
      "The issue is created at: 2019-10-09T08:31:06Z\n",
      "The status of the issue is: open\n",
      "The URL of the issue is: https://github.com/amfoss/GitLit/issues/34\n"
     ]
    }
   ],
   "source": [
    "print(\"The title of the issue:\", issue['data']['title'])\n",
    "print(\"The issue is opened by:\", issue['data']['user']['login'])\n",
    "print(\"The issue is created at:\", issue['data']['created_at'])\n",
    "print(\"The status of the issue is:\", issue['data']['state'])\n",
    "print(\"The URL of the issue is:\", issue['data']['html_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetching Pull Requests from GitHub Backend\n",
    "\n",
    "The PRs of a project can be fetched using GitHub Backend. [Reference](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#perceval.backends.core.github.GitHub.CATEGORIES).\n",
    "\n",
    "We can use the fetch function with the argument `category = 'pull_request'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PRS COUNT: 33\n"
     ]
    }
   ],
   "source": [
    "prs = github_backend.fetch(\n",
    "    category = 'pull_request', \n",
    "    from_date = from_date,\n",
    "    to_date = to_date\n",
    "    )\n",
    "\n",
    "prs_list = list(prs)\n",
    "print(\"PRS COUNT:\", len(prs_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the information which can be drawn from the data that was fetched by the Perceval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['backend_name', 'backend_version', 'perceval_version', 'timestamp', 'origin', 'uuid', 'updated_on', 'classified_fields_filtered', 'category', 'search_fields', 'tag', 'data'])\n",
      "\n",
      "dict_keys(['url', 'id', 'node_id', 'html_url', 'diff_url', 'patch_url', 'issue_url', 'number', 'state', 'locked', 'title', 'user', 'body', 'created_at', 'updated_at', 'closed_at', 'merged_at', 'merge_commit_sha', 'assignee', 'assignees', 'requested_reviewers', 'requested_teams', 'labels', 'milestone', 'commits_url', 'review_comments_url', 'review_comment_url', 'comments_url', 'statuses_url', 'head', 'base', '_links', 'author_association', 'merged', 'mergeable', 'rebaseable', 'mergeable_state', 'merged_by', 'comments', 'review_comments', 'maintainer_can_modify', 'commits', 'additions', 'deletions', 'changed_files', 'user_data', 'review_comments_data', 'reviews_data', 'requested_reviewers_data', 'merged_by_data', 'commits_data'])\n",
      "\n",
      "dict_keys(['login', 'id', 'node_id', 'avatar_url', 'gravatar_id', 'url', 'html_url', 'followers_url', 'following_url', 'gists_url', 'starred_url', 'subscriptions_url', 'organizations_url', 'repos_url', 'events_url', 'received_events_url', 'type', 'site_admin'])\n"
     ]
    }
   ],
   "source": [
    "pr = prs_list[1]\n",
    "\n",
    "# pprint(pr)\n",
    "\n",
    "print(pr.keys())\n",
    "print()\n",
    "print(pr['data'].keys())\n",
    "print()\n",
    "print(pr['data']['user'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The title of pull request: Update README.md\n",
      "The pull request is opened by: akhilam512\n",
      "The pull request is created at: 2018-12-15T14:00:01Z\n",
      "The status of the pull request is: closed\n",
      "Additions: 4 \n",
      "Deletions: 1\n",
      "The URL of the pull request is: https://github.com/amfoss/GitLit/pull/2\n"
     ]
    }
   ],
   "source": [
    "print(\"The title of pull request:\", pr['data']['title'])\n",
    "print(\"The pull request is opened by:\", pr['data']['user']['login'])\n",
    "print(\"The pull request is created at:\", pr['data']['created_at'])\n",
    "print(\"The status of the pull request is:\", pr['data']['state'])\n",
    "print(\"Additions:\", pr['data']['additions'] , \"\\nDeletions:\", pr['data']['deletions'])\n",
    "print(\"The URL of the pull request is:\", pr['data']['html_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GitLab Backend\n",
    "\n",
    "The GitLab project we are gonna target is [amfoss/gitlit](https://gitlab.com/amfoss/gitlit).\n",
    "\n",
    "> The documentation for the GitLab Backend can be found [here](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#module-perceval.backends.core.gitlab)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need the personal access token to use the GitHub Backend of Perceval without having any problem with the GitHub API. You can generate a new token [here](https://gitlab.com/profile/personal_access_tokens). Please follow these steps if you need any help, [GitLab: PAT Docs](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_TOKEN = \"12345678abcdefgh\"\n",
    "# your gitlab api token goes here\n",
    "\n",
    "OWNER = \"amfoss\"\n",
    "REPOSITORY = \"gitlit\"\n",
    "\n",
    "from_date = datetime(2018,1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from perceval.backends.core.gitlab import GitLab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "gitlab_backend = GitLab(owner = OWNER,\n",
    "                        repository = REPOSITORY,\n",
    "                        api_token = API_TOKEN,\n",
    "                        sleep_for_rate = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's printout some basic information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "amfoss\n",
      "gitlit\n",
      "https://gitlab.com/amfoss/gitlit\n"
     ]
    }
   ],
   "source": [
    "print(gitlab_backend.owner)\n",
    "print(gitlab_backend.repository)\n",
    "print(gitlab_backend.origin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['issue', 'merge_request']\n"
     ]
    }
   ],
   "source": [
    "print(gitlab_backend.categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetching Issues from GitLab Backend\n",
    "\n",
    "The Issues of a project can be fetched using GitLab Backend. [Reference](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#perceval.backends.core.gitlab.GitLab.CATEGORIES).\n",
    "\n",
    "\n",
    "We can use the fetch function with the argument `category = 'issue'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ISSUES COUNT: 1\n"
     ]
    }
   ],
   "source": [
    "issues = gitlab_backend.fetch(\n",
    "    category = 'issue', \n",
    "    from_date = from_date\n",
    "    )\n",
    "\n",
    "issues_list = list(issues)\n",
    "print(\"ISSUES COUNT:\", len(issues_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the information which can be drawn from the data that was fetched by the Perceval.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['backend_name', 'backend_version', 'perceval_version', 'timestamp', 'origin', 'uuid', 'updated_on', 'classified_fields_filtered', 'category', 'search_fields', 'tag', 'data'])\n",
      "\n",
      "dict_keys(['id', 'iid', 'project_id', 'title', 'description', 'state', 'created_at', 'updated_at', 'closed_at', 'closed_by', 'labels', 'milestone', 'assignees', 'author', 'assignee', 'user_notes_count', 'merge_requests_count', 'upvotes', 'downvotes', 'due_date', 'confidential', 'discussion_locked', 'web_url', 'time_stats', 'task_completion_status', 'has_tasks', '_links', 'references', 'moved_to_id', 'weight', 'epic_iid', 'epic', 'notes_data', 'award_emoji_data'])\n",
      "\n",
      "dict_keys(['id', 'name', 'username', 'state', 'avatar_url', 'web_url'])\n"
     ]
    }
   ],
   "source": [
    "issue = issues_list[0]\n",
    "\n",
    "# pprint(issue)\n",
    "\n",
    "print(issue.keys())\n",
    "print()\n",
    "print(issue['data'].keys())\n",
    "print()\n",
    "print(issue['data']['author'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The title of the issue: Add Logo to NavBar\n",
      "The issue is opened by: amfoss_in\n",
      "The issue is created at: 2019-10-09T08:31:06.000Z\n",
      "The status of the issue is: opened\n",
      "The URL of the issue is: https://gitlab.com/amfoss/GitLit/issues/34\n"
     ]
    }
   ],
   "source": [
    "print(\"The title of the issue:\", issue['data']['title'])\n",
    "print(\"The issue is opened by:\", issue['data']['author']['username'])\n",
    "print(\"The issue is created at:\", issue['data']['created_at'])\n",
    "print(\"The status of the issue is:\", issue['data']['state'])\n",
    "print(\"The URL of the issue is:\", issue['data']['web_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetching Merge Requests from GitLab Backend\n",
    "\n",
    "The MRs of a project can be fetched using GitLab Backend. [Reference](https://perceval.readthedocs.io/en/latest/perceval.backends.core.html#perceval.backends.core.gitlab.GitLab.CATEGORIES).\n",
    "\n",
    "\n",
    "We can use the fetch function with the argument `category = 'merge_request'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MRS COUNT: 33\n"
     ]
    }
   ],
   "source": [
    "mrs = gitlab_backend.fetch(\n",
    "    category = 'merge_request', \n",
    "    from_date = from_date\n",
    "    )\n",
    "\n",
    "mrs_list = list(mrs)\n",
    "print(\"MRS COUNT:\", len(mrs_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the information which can be drawn from the data that was fetched by the Perceval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['backend_name', 'backend_version', 'perceval_version', 'timestamp', 'origin', 'uuid', 'updated_on', 'classified_fields_filtered', 'category', 'search_fields', 'tag', 'data'])\n",
      "\n",
      "dict_keys(['id', 'iid', 'project_id', 'title', 'description', 'state', 'created_at', 'updated_at', 'merged_by', 'merged_at', 'closed_by', 'closed_at', 'target_branch', 'source_branch', 'user_notes_count', 'upvotes', 'downvotes', 'assignee', 'author', 'assignees', 'source_project_id', 'target_project_id', 'labels', 'work_in_progress', 'milestone', 'merge_when_pipeline_succeeds', 'merge_status', 'sha', 'merge_commit_sha', 'squash_commit_sha', 'discussion_locked', 'should_remove_source_branch', 'force_remove_source_branch', 'reference', 'references', 'web_url', 'time_stats', 'squash', 'task_completion_status', 'has_conflicts', 'blocking_discussions_resolved', 'approvals_before_merge', 'subscribed', 'changes_count', 'latest_build_started_at', 'latest_build_finished_at', 'first_deployed_to_production_at', 'pipeline', 'head_pipeline', 'diff_refs', 'merge_error', 'user', 'notes_data', 'award_emoji_data', 'versions_data'])\n",
      "\n",
      "dict_keys(['id', 'name', 'username', 'state', 'avatar_url', 'web_url'])\n"
     ]
    }
   ],
   "source": [
    "mr = mrs_list[1]\n",
    "\n",
    "# pprint(mr)\n",
    "\n",
    "print(mr.keys())\n",
    "print()\n",
    "print(mr['data'].keys())\n",
    "print()\n",
    "print(mr['data']['author'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The title of merge request: Update README.md\n",
      "The merge request is opened by: akhilkg\n",
      "The merge request is created at: 2018-12-15T14:00:01.000Z\n",
      "The status of the merge request is: merged\n",
      "The URL of the merge request is: https://gitlab.com/amfoss/GitLit/-/merge_requests/2\n"
     ]
    }
   ],
   "source": [
    "print(\"The title of merge request:\", mr['data']['title'])\n",
    "print(\"The merge request is opened by:\", mr['data']['author']['username'])\n",
    "print(\"The merge request is created at:\", mr['data']['created_at'])\n",
    "print(\"The status of the merge request is:\", mr['data']['state'])\n",
    "print(\"The URL of the merge request is:\", mr['data']['web_url'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}