collect_users/analyze_users.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6617d56b",
   "metadata": {},
   "source": [
    "# Analysis of user collection\n",
    "\n",
    "In this notebook, the retrieved results of the user collection is analyzed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "635b6d3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import time\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d921730",
   "metadata": {},
   "source": [
    "Configure modules like pandas and plotting libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e12160d",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = [12, 8]\n",
    "pd.set_option('display.min_rows', 24)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0efae96b",
   "metadata": {},
   "source": [
    "Folders and constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74fd74bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "fp_figs = Path(\"figs\")\n",
    "\n",
    "annotated_users = Path(\"results\", \"users_enriched_summer2021.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49d80e8d",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "In this section, output data of the various methods is collected and combined. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a189654b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_files = glob.glob(\"methods/*/results/*.csv\")\n",
    "\n",
    "for i, d in enumerate(data_files):\n",
    "    print(i+1, d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206bba3c",
   "metadata": {},
   "source": [
    "All files is loaded and combined in a single dataset. The dataset contains the variables:\n",
    "- `source` The source is the method used to find the user, e.g. paperswithcode, github_search_users, github_search_topics. The name is derived from the file name of the results file.\n",
    "- `service` The service the user profile is found, e.g. github.com, github.warwick.ac.uk, gitlab.\n",
    "- `date` The date of collection. This can be useful when updating results.\n",
    "- `user_id` The user handle of the found user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f16f7232",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_user_names = pd.concat(\n",
    "    [pd.read_csv(fp) for fp in data_files], \n",
    "    axis=0, \n",
    "    keys=data_files, \n",
    "    names=[\"source\", \"row\"]\n",
    ") \\\n",
    "    .reset_index(\"source\") \\\n",
    "    .reset_index(drop=True)\n",
    "\n",
    "df_user_names[\"source\"] = df_user_names[\"source\"].apply(lambda x: Path(x).stem)\n",
    "df_user_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e86f42",
   "metadata": {},
   "source": [
    "## Findability\n",
    "\n",
    "The following results shows how many times user handles are found with each retrieval method. If a user is found with multiple methods, the findability of the user can be consider to be better. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7bb1a4",
   "metadata": {},
   "source": [
    "### Cross table user and method\n",
    "\n",
    "Create a cross table with service and user on one axis and the extraction method on the other axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1613a425",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df_name_method_crosstab = df_user_names \\\n",
    "    .drop(\"date\", axis=1) \\\n",
    "    .groupby([\"service\", \"user_id\", \"source\"]) \\\n",
    "    .size() \\\n",
    "    .unstack(\"source\") \\\n",
    "    .fillna(0) \\\n",
    "    .applymap(int)\n",
    "\n",
    "df_name_method_crosstab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a3ad6a",
   "metadata": {},
   "source": [
    "The following example can be uncommented to find the findabilty of a specific project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "627cd832",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_name_method_crosstab.loc[('github.com', 'asreview')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dce9541",
   "metadata": {},
   "source": [
    "### Findability per user\n",
    "\n",
    "The following table show which users are found with most collection strategies. The count indicates the number of methods, and the relative score the number relative to the total number of collection strategies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "804c4aef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute the count\n",
    "df_name_findability = (df_name_method_crosstab > 0) \\\n",
    "    .astype(int) \\\n",
    "    .sum(axis=1) \\\n",
    "    .sort_values(ascending=False) \\\n",
    "    .to_frame(name=\"count\")\n",
    "\n",
    "# compute relative score\n",
    "df_name_findability[\"relative\"] = df_name_findability[\"count\"] / len(data_files)\n",
    "\n",
    "df_name_findability\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fdc9cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieval_count = df_name_findability[\"count\"].value_counts().to_frame(name=\"Retrieval count\")\n",
    "\n",
    "sns.barplot(\n",
    "    x=retrieval_count.index.astype(str), \n",
    "    y=retrieval_count[\"Retrieval count\"]\n",
    ")\n",
    "plt.title(\"Retrieval count for each user in dataset\")\n",
    "plt.savefig(Path(fp_figs, 'user_collection_user_findability.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0934aae",
   "metadata": {},
   "source": [
    "### Findability per method\n",
    "\n",
    "This table indicates how succeful a method is collecting users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23136aa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_method_findability = (df_name_method_crosstab > 0) \\\n",
    "    .astype(int) \\\n",
    "    .sum(axis=0) \\\n",
    "    .sort_values(ascending=False) \\\n",
    "    .to_frame(name=\"count\")\n",
    "\n",
    "df_method_findability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21be8f84",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_method_findability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68bddfe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.barplot(\n",
    "    x=df_method_findability.index, \n",
    "    y=df_method_findability[\"count\"]\n",
    ")\n",
    "plt.title(\"Retrieval count for each method in dataset\")\n",
    "plt.savefig(Path(fp_figs, 'user_collection_method_findability.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4311df7",
   "metadata": {},
   "source": [
    "## Filtering of users\n",
    "\n",
    "Not all users collected in the previous steps are relevant to the analysis. Some are not (or no longer) part of the organisation, or excluded for other reasons (for example students)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2e9bb52",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_annotated_users = pd.read_excel(annotated_users)\n",
    "\n",
    "df_annotated_users[\n",
    "    [\"user_id\", \n",
    "     \"is_student\", \n",
    "     \"is_employee\", \n",
    "     \"is_currently_employed\", \n",
    "     \"is_research_group\", \n",
    "     \"final_decision\"\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad800df2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_annotated_users_included = df_annotated_users[df_annotated_users[\"final_decision\"] == 1]\n",
    "\n",
    "print(\"The number of included users is\", len(df_annotated_users_included))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc0ecfe",
   "metadata": {},
   "source": [
    "## Properties of users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8854f817",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_annotated_users_included \\\n",
    "    .loc[df_annotated_users_included[\"final_decision\"] == 1, \"is_research_group\"] \\\n",
    "    .fillna(0) \\\n",
    "    .value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "837dc7d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a boxplot with swarm\n",
    "ax = sns.boxplot(x='public_repos', data=df_annotated_users_included)\n",
    "ax = sns.swarmplot(x='public_repos', data=df_annotated_users_included, color=\".25\")\n",
    "\n",
    "# output users with most public repos\n",
    "df_annotated_users_included[['user_id', 'public_repos']].sort_values('public_repos', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "237f17b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x='public_gists', data=df_annotated_users_included)\n",
    "df_annotated_users_included[['user_id', 'public_gists']].sort_values('public_gists', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ac0bf4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = sns.boxplot(x='followers', data=df_annotated_users_included, showfliers=False)\n",
    "\n",
    "df_annotated_users_included[['user_id', 'followers']].sort_values('followers', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e85a1a02",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x=df_annotated_users_included['following'], showfliers = False)\n",
    "df_annotated_users_included[['user_id', 'following']].sort_values('following', ascending=False).head(10)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}