GitHub/GitHub_Create_leaderboard_of_contributors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f334b21-2372-4531-926a-5eef1f8e3392",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-26T08:07:21.267353Z",
     "iopub.status.busy": "2023-01-26T08:07:21.267092Z",
     "iopub.status.idle": "2023-01-26T08:07:21.281180Z",
     "shell.execute_reply": "2023-01-26T08:07:21.280278Z",
     "shell.execute_reply.started": "2023-01-26T08:07:21.267289Z"
    },
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"GitHub.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/GitHub.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3994648-c20b-4e20-88a0-d3488f232043",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# GitHub - Create leaderboard of contributors\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=GitHub+-+Create+leaderboard+of+contributors:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe58f42b-a421-4a61-8227-2fdf6410d15d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #github #repos #commits #stats #naas_drivers #leaderboard #commitsPoints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d4b05e-f205-46a0-90aa-2bd75bcfa402",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Suhas B](https://www.linkedin.com/in/suhasbrao/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53ded090-2e67-419c-9f85-e6a186d7fa95",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-04-12 (Created: 2023-02-01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6720b93e-aa24-4b59-9e96-f15d64ac3436",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b3c5395-9e7a-4b62-9d9d-fb7cd224a0ec",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2c71cac5-4b22-4c35-891f-54868e92d186",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import plotly.express as px\n",
    "from naas_drivers import github\n",
    "import naas\n",
    "\n",
    "import requests\n",
    "from urllib.parse import urlencode"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23739f25-7db1-4411-b612-bd49afda2fb8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Setup Github\n",
    "**How to find your personal access token on Github?**\n",
    "\n",
    "- First we need to create a personal access token to get the details of our organization from here: https://github.com/settings/tokens\n",
    "- You will be asked to select scopes for the token. Which scopes you choose will determine what information and actions you will be able to perform against the API.\n",
    "- You should be careful with the ones prefixed with write:, delete: and admin: as these might be quite destructive.\n",
    "- You can find description of each scope in docs here (https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2c10d40-5bd3-44ee-a8cd-093b96d9c238",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Github repository url\n",
    "REPO_URL = \"https://github.com/jupyter-naas/awesome-notebooks\"\n",
    "\n",
    "# Github token\n",
    "GITHUB_TOKEN = naas.secret.get(\"Git_Token\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b341b929-2a16-461f-aac9-91aef332aa9b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b9f2577-9040-4ceb-a65f-003a9d3710e9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get actor/user who closed the issue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9e5899fa-84b6-4e59-a2e0-f0acc36c3517",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to get the actor for closing an issue\n",
    "def get_actor_for_closed_issue(events_url, git_obj):\n",
    "    '''\n",
    "    This function is used to get actor for a closed issues\n",
    "    It gives the name of a person who closed the issue.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    events_url: str:\n",
    "        events_url from Github.\n",
    "        Example : \"https://api.github.com/repos/jupyter-naas/awesome-notebooks/issues/1401/events\"\n",
    "    \n",
    "    git_obj: object of naas_driver.github\n",
    "    '''\n",
    "    \n",
    "    url = events_url\n",
    "    res = requests.get(url, headers=git_obj.headers)\n",
    "    try:\n",
    "        res.raise_for_status()\n",
    "    except requests.HTTPError as e:\n",
    "        raise(e)\n",
    "    \n",
    "    res_json = res.json()\n",
    "    if len(res_json) == 0:\n",
    "        return\n",
    "    \n",
    "    \n",
    "    actor = None\n",
    "    for events in res_json:\n",
    "        if events[\"event\"] == \"closed\":\n",
    "            actor= events[\"actor\"][\"login\"]\n",
    "    \n",
    "    return actor\n",
    "# get_actor_for_closed_issue(\"https://api.github.com/repos/jupyter-naas/awesome-notebooks/issues/1392/events\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd182a5f-4d2b-432d-819c-28c1188bf834",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get All Issues (both open and closed issues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "12992028-00ae-4a81-b30c-5358db5be496",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_all_issues(repo_url):\n",
    "    '''\n",
    "    This function retrives all the issues of a repository and returns a \n",
    "    dataframe with the following columns:\n",
    "    \n",
    "    link_to_the_issue\n",
    "    issue_title\n",
    "    issue_number\n",
    "    issue_state\n",
    "    issue_creator\n",
    "    issue_closed_by\n",
    "    last_created_date\n",
    "    last_created_time\n",
    "    last_updated_date\n",
    "    last_updated_time\n",
    "    \n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    repo_url: str:\n",
    "        Repository url from Github.\n",
    "        Example : \"https://github.com/jupyter-naas/awesome-notebooks\"\n",
    "    '''\n",
    "    \n",
    "    git_obj = github.connect(GITHUB_TOKEN)\n",
    "    repository = git_obj.get_repository_url(repo_url)\n",
    "    df = pd.DataFrame()\n",
    "    page, idx = 1, 0\n",
    "    while True:\n",
    "        params = {\n",
    "                \"per_page\": \"100\",\n",
    "                \"page\": page,\n",
    "            }\n",
    "        \n",
    "        # Api to get open issues from github\n",
    "        url = f\"https://api.github.com/repos/{repository}/issues?state=all&{urlencode(params, safe='(),')}\"\n",
    "        res = requests.get(url, headers=git_obj.headers)\n",
    "        try:\n",
    "            res.raise_for_status()\n",
    "        except requests.HTTPError as e:\n",
    "            raise(e)\n",
    "        \n",
    "        res_json = res.json()\n",
    "        if len(res_json) == 0:\n",
    "            break\n",
    "        \n",
    "        for issue in res_json:\n",
    "            # Fetch all the issues, check node_id to see if it is an issue or PR\n",
    "            \n",
    "            if(issue[\"node_id\"].startswith(\"I_\")):\n",
    "                \n",
    "                df.loc[idx,'link_to_the_issue'], df.loc[idx, 'issue_number'] = issue['html_url'], issue['number']\n",
    "                df.loc[idx, 'issue_title'], df.loc[idx, 'issue_state'] = issue['title'], issue['state']\n",
    "   \n",
    "                df.loc[idx, \"issue_creator\"] = issue[\"user\"].get(\"login\")\n",
    "                \n",
    "                # Create a cloumn that stores the name of the person who closed the issue\n",
    "                # else if the issue is Open the column value will be None\n",
    "                df.loc[idx, \"issue_closed_by\"] = get_actor_for_closed_issue( issue[\"events_url\"], git_obj )\n",
    "                \n",
    "                df.loc[idx, 'last_created_date'] = issue.get('created_at').strip('Z').split('T')[0]\n",
    "                df.loc[idx, 'last_created_time'] = issue.get('created_at').strip('Z').split('T')[-1]\n",
    "                df.loc[idx, 'last_updated_date'] = issue.get('updated_at').strip('Z').split('T')[0]\n",
    "                df.loc[idx, 'last_updated_time'] = issue.get('updated_at').strip('Z').split('T')[-1]\n",
    "                \n",
    "                \n",
    "                idx +=1\n",
    "        page+=1\n",
    "    return df\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "882b90f1-d443-41b8-a57a-1b2de73d238a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get Issue/PR comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "04751a7f-57b6-4f4f-b915-58e35e6cb166",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_issue_pr_comments(repo_url):\n",
    "    '''\n",
    "    The function retrieves all the issue comments and pr comments for a given r\n",
    "    repository url.\n",
    "    The function returns a dataframe with following columns for an issue comment/pr comment\n",
    "    \n",
    "    comment_id\n",
    "    issue_url\n",
    "    comment_by\n",
    "    comment_body\n",
    "    '''\n",
    "    git_obj = github.connect(GITHUB_TOKEN)\n",
    "    repository = git_obj.get_repository_url(repo_url)\n",
    "    df = pd.DataFrame()\n",
    "    page, idx = 1, 0\n",
    "    while True:\n",
    "        params = {\n",
    "                \"per_page\": \"100\",\n",
    "                \"page\": page,\n",
    "            }\n",
    "        \n",
    "        # Api to get comments from github\n",
    "       \n",
    "        url = f\"https://api.github.com/repos/{repository}/issues/comments?{urlencode(params, safe='(),')}\"\n",
    "        res = requests.get(url, headers=git_obj.headers)\n",
    "        try:\n",
    "            res.raise_for_status()\n",
    "        except requests.HTTPError as e:\n",
    "            raise(e)\n",
    "        \n",
    "        res_json = res.json()\n",
    "        if len(res_json) == 0:\n",
    "            break\n",
    "        \n",
    "        for issue_comment in res_json:\n",
    "            df.loc[idx, \"comment_id\"] = issue_comment[\"id\"]\n",
    "            df.loc[idx, \"issue_url\"] = issue_comment[\"issue_url\"]\n",
    "            df.loc[idx, \"comment_by\"] = issue_comment[\"user\"][\"login\"]\n",
    "            df.loc[idx, \"comment_body\"] = issue_comment[\"body\"]\n",
    "            \n",
    "            idx +=1\n",
    "        page += 1\n",
    "        \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd1dc0b-d631-459d-a5a8-bb1f5d276ffb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get Contributor Details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "15f047b6-136d-47b2-8af8-322a28c847ca",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_contributors_details(repo_url):\n",
    "    '''\n",
    "    The function retrieves contributors for a given repository url\n",
    "    This function returns a dataframe of contributors with following columns\n",
    "    \n",
    "    contributor_name\n",
    "    issues_created\n",
    "    issues_closed\n",
    "    issue_pr_comments\n",
    "    commits\n",
    "    \n",
    "    '''\n",
    "    git_obj = github.connect(GITHUB_TOKEN)\n",
    "    repository = git_obj.get_repository_url(repo_url)\n",
    "    df = pd.DataFrame()\n",
    "    page, idx = 1, 0\n",
    "    while True:\n",
    "        params = {\n",
    "                \"per_page\": \"100\",\n",
    "                \"page\": page,\n",
    "            }\n",
    "        url = f\"https://api.github.com/repos/{repository}/contributors?{urlencode(params, safe='(),')}\"\n",
    "        res = requests.get(url, headers=git_obj.headers)\n",
    "        try:\n",
    "            res.raise_for_status()\n",
    "        except requests.HTTPError as e:\n",
    "            raise(e)\n",
    "        \n",
    "        res_json = res.json()\n",
    "        if len(res_json) == 0:\n",
    "            break\n",
    "        \n",
    "        for contributor in res_json:\n",
    "            df.loc[idx, \"contributor_name\"] = contributor[\"login\"]\n",
    "            # create a column to store number of issues created by a contributor\n",
    "            df.loc[idx, \"issues_created\"] = len(df_all_issues.loc[df_all_issues[\"issue_creator\"]==contributor[\"login\"]])\n",
    "            # create a column to store number of issues closed by a contributor\n",
    "            \n",
    "            df.loc[idx, \"issues_closed\"] = len(df_all_issues.loc[df_all_issues[\"issue_closed_by\"]==contributor[\"login\"]])\n",
    "            \n",
    "            # Need to get df for issue comments and PR comments\n",
    "            df.loc[idx, \"issue_pr_comments\"] = len(df_all_comments.loc[df_all_comments[\"comment_by\"] == contributor[\"login\"]])\n",
    "            \n",
    "            # add a column to store number of commits made by a contributor\n",
    "            df.loc[idx, \"commits\"] = len(git_obj.repos.get_commits(REPO_URL, contributor[\"login\"]))\n",
    "            idx +=1\n",
    "        \n",
    "        page += 1\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ab6634-931c-445b-8b28-6a2c7cb11a30",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get leaderboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4a803d9b-261b-49de-b4bb-1bc80df121ec",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_leaderboard(df):\n",
    "    '''\n",
    "    The function orders the given dataframe based on pts column in descending order.\n",
    "    '''\n",
    "    return df.sort_values(by=\"pts\", ascending=False).reset_index(drop=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0473d647-98bd-467f-b96e-a67b75389a96",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Set Key Variables from function call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ddf7e829-0c0f-4b58-9fa4-92c485f20258",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_all_issues = get_all_issues(REPO_URL)\n",
    "print(\"Total open Issues fetched:\", len(df_all_issues))\n",
    "# df_all_issues.head()\n",
    "\n",
    "df_all_comments = get_issue_pr_comments(REPO_URL)\n",
    "# df_all_comments.head()\n",
    "\n",
    "df_contributors = get_contributors_details(REPO_URL)\n",
    "# df_contributors.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3ca51f-382c-40fc-bc85-bf51699b3ade",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Add pts column to contributor dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66198144-fe23-491c-a93f-ada487f34ecf",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Add a column of points for contributors df\n",
    "# pts is calculated by multiplying a pre-defined wieght for each of the columns\n",
    "# Feel free to modify the weight \n",
    "df_contributors[\"pts\"] = 1*df_contributors[\"issues_created\"] + 1*df_contributors[\"issues_closed\"] + \\\n",
    "                        0.5*df_contributors[\"issue_pr_comments\"] + 1*df_contributors[\"commits\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19744942-2de2-4211-ac6a-2fc1e119bc2c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d2fb673a-494b-4362-b7e5-d4547a8b6b9a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_leaderboard = get_leaderboard(df_contributors)\n",
    "print(\"Here are the top 10 contributors \\n\")\n",
    "df_leaderboard.head(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "58d4c2b5a700d444c161aa052156ca47ffb7642a2e44c771125cac9a6fe04edc",
   "notebook_path": "GitHub/GitHub_Create_leaderboard_of_contributors.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}