GitHub/GitHub_Track_notebooks_created_over_time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3908d09-b571-4bf1-90d4-4144b4217018",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"GitHub.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/GitHub.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da3d8a29-e10c-4c8d-a012-5f1114d86fad",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# GitHub - Track notebooks created over time\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=GitHub+-+Track+notebooks+created+over+time:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd9e8fd-6ceb-4c0e-b059-5635d04b5878",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #github #repos #commits #notebooks #operations #analytics #html #plotly #csv #image #png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa199566-beb5-4560-8b15-6084447cc0c2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Florent Ravenel](https://www.linkedin.com/in/florent-ravenel/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a44799-2590-4d4c-8d76-f291390971e8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-04-12 (Created: 2022-03-18)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "naas-description",
   "metadata": {
    "papermill": {},
    "tags": [
     "description"
    ]
   },
   "source": [
    "**Description:** This notebook allows users to track changes to their notebooks over time using GitHub."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef49c8ec-8f57-4ec1-b681-c8f763b4d946",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5a034dd-2a9d-4fdd-b1f6-7a2b58fd3cdd",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "import os\n",
    "from naas_drivers import github\n",
    "import plotly.graph_objects as go\n",
    "import pydash as _pd\n",
    "from urllib.parse import urlencode\n",
    "from datetime import datetime, timedelta\n",
    "import naas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ad0485-44eb-428d-bfe3-45391fe1d371",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Setup Github\n",
    "**How to find your personal access token on Github?**\n",
    "\n",
    "- First we need to create a personal access token to get the details of our organization from here: https://github.com/settings/tokens\n",
    "- You will be asked to select scopes for the token. Which scopes you choose will determine what information and actions you will be able to perform against the API.\n",
    "- You should be careful with the ones prefixed with write:, delete: and admin: as these might be quite destructive.\n",
    "- You can find description of each scope in docs here (https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4bd695e-f7bd-4cde-85a2-050e98bae3b6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Inputs\n",
    "REPO_URL = \"https://github.com/jupyter-naas/awesome-notebooks\"\n",
    "GITHUB_TOKEN = \"ghp_COJiJEU4cQR4rjsxxxxx\"\n",
    "\n",
    "# Outputs\n",
    "chart_title = \"Notebooks created since naas launch\"\n",
    "name_output = f\"awesome-notebooks\"\n",
    "csv_output = f\"{name_output}.csv\"\n",
    "html_output = f\"{name_output}.html\"\n",
    "image_output = f\"{name_output}.png\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d64d497-7281-407a-b216-c58137dfc469",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7abf49a0-a5f8-43d1-af1f-be26f39fa952",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-22T13:48:53.656270Z",
     "iopub.status.busy": "2022-01-22T13:48:53.656043Z",
     "iopub.status.idle": "2022-01-22T13:48:53.658818Z",
     "shell.execute_reply": "2022-01-22T13:48:53.658211Z",
     "shell.execute_reply.started": "2022-01-22T13:48:53.656247Z"
    },
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get commits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "796a40ff-75a4-4f09-aa7a-f1b7dda682d9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_commits = github.connect(GITHUB_TOKEN).repos.get_commits(REPO_URL)\n",
    "print(\"Total commits:\", len(df_commits))\n",
    "df_commits.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ea2f696-6e79-4a7d-953a-4b0bebd224ab",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Filter on merge PR and date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c06cda12-e0f3-41da-9143-906514dd69e2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_commits_merge(df):\n",
    "    df_pr = df[\n",
    "        (df.MESSAGE.str[:5] == \"Merge\") & (df.VERIFICATION_STATUS == True)\n",
    "    ].reset_index(drop=True)\n",
    "    print(\"Total Merged PR:\", len(df_pr))\n",
    "\n",
    "    df_pr[\"DATE\"] = df_pr[\"AUTHOR_DATE\"].dt.strftime(\"%Y-%m-%d\")\n",
    "    df_pr = (\n",
    "        df_pr[[\"DATE\", \"COMMITTER_NAME\", \"ID\"]]\n",
    "        .drop_duplicates(\"DATE\")\n",
    "        .reset_index(drop=True)\n",
    "    )\n",
    "    print(\"Total Merged PR (unique date):\", len(df_pr))\n",
    "    return df_pr\n",
    "\n",
    "\n",
    "df_pr = get_commits_merge(df_commits)\n",
    "df_pr.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f39e6f-2a9e-4a2b-b1b7-48e892b2aed1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get notebooks for each commits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42489e05-c892-4c87-a210-4efc199cbbeb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_notebooks(commit_id):\n",
    "    notebooks = []\n",
    "    headers = {\"Authorization\": f\"token {GITHUB_TOKEN}\"}\n",
    "    url = f\"https://api.github.com/repos/jupyter-naas/awesome-notebooks/git/trees/{commit_id}?recursive=1\"\n",
    "    res = requests.get(url, headers=headers)\n",
    "    try:\n",
    "        res.raise_for_status()\n",
    "    except requests.HTTPError as e:\n",
    "        raise (e)\n",
    "    res_json = res.json()\n",
    "\n",
    "    trees = res_json.get(\"tree\")\n",
    "    for file in trees:\n",
    "        if (\n",
    "            \".github\" not in file.get(\"path\")\n",
    "            and \".gitignore\" not in file.get(\"path\")\n",
    "            and \"/\" in file.get(\"path\")\n",
    "        ):\n",
    "            if file.get(\"path\").endswith(\".ipynb\"):\n",
    "                temp = file.get(\"path\").split(\"/\")\n",
    "                if temp == -1:\n",
    "                    data = {\"root\": None, \"subdir\": file.get(\"path\")}\n",
    "                    notebooks.append(data)\n",
    "                else:\n",
    "                    last_folder = \"\"\n",
    "                    file_name = temp[-1]\n",
    "                    temp.pop()\n",
    "                    for folder in temp:\n",
    "                        last_folder += \"/\" + folder\n",
    "                    root = last_folder[1:]\n",
    "                    data = {\"root\": root, \"subdir\": file_name}\n",
    "                    notebooks.append(data)\n",
    "    return pd.DataFrame(notebooks)\n",
    "\n",
    "\n",
    "def tracks_notebooks(df):\n",
    "    df_tracks = pd.DataFrame()\n",
    "\n",
    "    for _, row in df.iterrows():\n",
    "        date = row.DATE\n",
    "        commit_id = row.ID\n",
    "        try:\n",
    "            df_track = get_notebooks(commit_id)\n",
    "            df_track[\"DATE\"] = date\n",
    "            df_track[\"ID\"] = commit_id\n",
    "            # Concat\n",
    "            df_tracks = pd.concat([df_tracks, df_track], axis=0)\n",
    "        except Exception as e:\n",
    "            print(f\"Error on {date} - {commit_id}\", e)\n",
    "    print(\"Total notebooks tracked:\", len(df_tracks))\n",
    "    return df_tracks\n",
    "\n",
    "\n",
    "df_tracks = tracks_notebooks(df_pr)\n",
    "df_tracks.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e889ef-c50d-4e59-a523-e8ce955c576d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Create trend dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9deb6e43-abba-49ec-8603-c02bf14fe90e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_trend(df, date_col_name=\"DATE\", date_val_name=\"ID\", date_order=\"asc\"):\n",
    "    # Cleaning\n",
    "    df = df.rename(columns={date_val_name: \"VALUE\"})\n",
    "\n",
    "    df = df.groupby(date_col_name, as_index=False).agg({\"VALUE\": \"count\"})\n",
    "    d = datetime.now().date()\n",
    "    d2 = df.loc[df.index[0], date_col_name]\n",
    "    idx = pd.date_range(d2, d, freq=\"D\")\n",
    "\n",
    "    df.set_index(date_col_name, drop=True, inplace=True)\n",
    "    df.index = pd.DatetimeIndex(df.index)\n",
    "    df = df.reindex(idx, fill_value=0)\n",
    "    df[date_col_name] = pd.DatetimeIndex(df.index)\n",
    "    df = df.reset_index(drop=True)\n",
    "    for _, row in df.iterrows():\n",
    "        if _ > 0:\n",
    "            n_1 = df.loc[df.index[_ - 1], \"VALUE\"]\n",
    "            n = df.loc[df.index[_], \"VALUE\"]\n",
    "            if n == 0:\n",
    "                df.loc[_, \"VALUE\"] = n_1\n",
    "    df[\"GROUP\"] = \"Actual\"\n",
    "\n",
    "    # Create line to target\n",
    "    value_target = 1000\n",
    "    date_target = \"2022-12-31\"\n",
    "    df_target = df[-1:].reset_index(drop=True)\n",
    "    d = datetime.strptime(date_target, \"%Y-%m-%d\")\n",
    "    d2 = df_target.loc[df_target.index[0], date_col_name]\n",
    "    idx = pd.date_range(d2, d, freq=\"D\")\n",
    "    df_target.set_index(date_col_name, drop=True, inplace=True)\n",
    "    df_target.index = pd.DatetimeIndex(df_target.index)\n",
    "    df_target = df_target.reindex(idx, fill_value=1000)\n",
    "    df_target[date_col_name] = pd.DatetimeIndex(df_target.index)\n",
    "    df_target = df_target.reset_index(drop=True)\n",
    "\n",
    "    # Get target value\n",
    "    to_go = value_target - df_target.loc[df_target.index[0], \"VALUE\"]\n",
    "    var = to_go / (len(idx) - 1)\n",
    "    for _, row in df_target.iterrows():\n",
    "        if _ > 0:\n",
    "            n_1 = df_target.loc[df_target.index[_ - 1], \"VALUE\"]\n",
    "            df_target.loc[_, \"VALUE\"] = n_1 + var\n",
    "    df_target[\"GROUP\"] = \"Forecast\"\n",
    "\n",
    "    # Concat data\n",
    "    df = pd.concat([df, df_target]).reset_index(drop=True)\n",
    "    df = df[[date_col_name, \"GROUP\", \"VALUE\"]]\n",
    "\n",
    "    # Calc variation\n",
    "    for idx, row in df.iterrows():\n",
    "        if idx == 0:\n",
    "            value_n1 = 0\n",
    "        else:\n",
    "            value_n1 = df.loc[df.index[idx - 1], \"VALUE\"]\n",
    "        df.loc[df.index[idx], \"VALUE_COMP\"] = value_n1\n",
    "    df[\"VARV\"] = df[\"VALUE\"] - df[\"VALUE_COMP\"]\n",
    "    df[\"VARP\"] = df[\"VARV\"] / abs(df[\"VALUE_COMP\"])\n",
    "\n",
    "    # Prep data\n",
    "    df[\"VALUE_D\"] = df[\"VALUE\"].map(\"{:,.0f}\".format).str.replace(\",\", \" \")\n",
    "    df[\"VARV_D\"] = df[\"VARV\"].map(\"{:,.0f}\".format).str.replace(\",\", \" \")\n",
    "    df.loc[df[\"VARV\"] >= 0, \"VARV_D\"] = \"+\" + df[\"VARV_D\"]\n",
    "    df[\"VARP_D\"] = df[\"VARP\"].map(\"{:,.2%}\".format).str.replace(\",\", \" \")\n",
    "    df.loc[df[\"VARP\"] >= 0, \"VARP_D\"] = \"+\" + df[\"VARP_D\"]\n",
    "\n",
    "    df.loc[df[\"GROUP\"] == \"Forecast\", \"TEXT\"] = \"\"\n",
    "    df.loc[df[\"GROUP\"] == \"Actual\", \"TEXT\"] = (\n",
    "        df[\"VALUE_D\"]\n",
    "        + \" notebooks as of \"\n",
    "        + df[\"DATE\"].dt.strftime(\"%Y-%m-%d\")\n",
    "        + \" (\"\n",
    "        + df[\"VARV_D\"]\n",
    "        + \" vs yesterday)\"\n",
    "    )\n",
    "    return df\n",
    "\n",
    "\n",
    "df_trend = get_trend(df_tracks)\n",
    "df_trend"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dc1e74c-90af-4e87-b131-06415220d4d6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3d2e573-0d96-4ec6-9361-376d28438604",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Plotting a line chart for notebook commits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a34f5de4-bec4-4735-bc75-614a6ddfe2e9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_linechart(df, label, group, value, text, title):\n",
    "    # Init\n",
    "    fig = go.Figure()\n",
    "\n",
    "    # Create fig\n",
    "    df1 = df[df[group] == \"Actual\"]\n",
    "    last_text = df1.loc[df1.index[-1], text]\n",
    "    fig.add_trace(\n",
    "        go.Scatter(\n",
    "            name=\"Actual\",\n",
    "            x=df1[label],\n",
    "            y=df1[value],\n",
    "            mode=\"lines\",\n",
    "            hovertext=df1[text],\n",
    "            hoverinfo=\"text\",\n",
    "            line=dict(color=\"royalblue\"),\n",
    "        )\n",
    "    )\n",
    "    df2 = df[df[group] == \"Forecast\"]\n",
    "    fig.add_trace(\n",
    "        go.Scatter(\n",
    "            name=\"Target\",\n",
    "            x=df2[label],\n",
    "            y=df2[value],\n",
    "            mode=\"lines\",\n",
    "            hovertext=df2[text],\n",
    "            hoverinfo=\"text\",\n",
    "            line=dict(color=\"red\", dash=\"dot\"),\n",
    "        )\n",
    "    )\n",
    "    fig.update_layout(\n",
    "        #         title=f\"<b>Tracks of notebooks created since naas launch </b><br><span style='font-size: 13px;'>Total notebooks as of today: {notebooks} (+{var} vs yesterday)</span>\",\n",
    "        title=f\"<b>{title}</b><br><span style='font-size: 13px;'>{last_text}</span>\",\n",
    "        title_font=dict(family=\"Verdana\", size=18, color=\"black\"),\n",
    "        plot_bgcolor=\"#ffffff\",\n",
    "        width=1200,\n",
    "        height=800,\n",
    "        paper_bgcolor=\"white\",\n",
    "        xaxis_title=\"Date\",\n",
    "        xaxis_title_font=dict(family=\"Verdana\", size=11, color=\"black\"),\n",
    "        yaxis_title=\"No. of notebooks\",\n",
    "        yaxis_title_font=dict(family=\"Verdana\", size=11, color=\"black\"),\n",
    "        margin_pad=10,\n",
    "    )\n",
    "    fig.show()\n",
    "    return fig\n",
    "\n",
    "\n",
    "fig = create_linechart(df_trend, \"DATE\", \"GROUP\", \"VALUE\", \"TEXT\", chart_title)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5aec6d1-8946-4b17-8835-1384826d4230",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Save and share your csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d93671b1-e7fe-4f40-a234-4256accdd911",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save your dataframe in CSV\n",
    "df_trend.to_csv(csv_output, index=False)\n",
    "\n",
    "# Share output with naas\n",
    "naas.asset.add(csv_output)\n",
    "\n",
    "# -> Uncomment the line below to remove your asset\n",
    "# naas.asset.delete(csv_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "febc0cd5-92da-42a2-98f1-ef058d7c1ff7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Save and share your graph in HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "123d965c-3842-46fc-9457-a5bda30d71ba",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save your graph in HTML\n",
    "fig.write_html(html_output)\n",
    "\n",
    "# Share output with naas\n",
    "naas.asset.add(html_output, params={\"inline\": True})\n",
    "\n",
    "# -> Uncomment the line below to remove your asset\n",
    "# naas.asset.delete(html_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08dcaf24-7c07-4c13-8c72-088d91a7b77c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Save and share your graph in image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3152e531-fd70-4da6-9292-82517470f1e4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save your graph in PNG\n",
    "fig.write_image(image_output)\n",
    "\n",
    "# Share output with naas\n",
    "naas.asset.add(image_output, params={\"inline\": True})\n",
    "\n",
    "# -> Uncomment the line below to remove your asset\n",
    "# naas.asset.delete(image_output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "e6462e6eaca51db7dbf96bbd6be22be706fa160145a35d73671984ccb506f093",
   "notebook_path": "GitHub/GitHub_Track_notebooks_created_over_time.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}