openalex_hkust_research_2014-2023.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb29e2a9-4b5f-4ed9-ac21-076addfa8915",
   "metadata": {},
   "source": [
    "## Visualizing Institutional Research Using Open Data\n",
    "\n",
    "**Code for data collection and cleaning**\n",
    "\n",
    "(Note: I am a research support librarian who's still learning Python for bibliometric analysis. Feel free to [email me](mailto:lbaster@ust.hk) if you spot any errors. Much appreciated :))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f528f10-e636-42ce-926d-1ecfbc681bf1",
   "metadata": {},
   "source": [
    "### Table of Content\n",
    "\n",
    "**[S1: How are we progressing toward open access?](#S1)**<br>\n",
    "**[S2: Which journals have we published in the most over the last 10 years?](#S2)**<br>\n",
    "**[S3: How has our contribution to the Sustainable Development Goals (SDG) evolved over the years?](#S3)**<br>\n",
    "**[S4: Which of our publications have had the most impact over the years?](#S4)**<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5daf03d8-27b2-492c-8c5e-59c34e6089c1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9718e35-b2ea-41ca-8ceb-187af383b178",
   "metadata": {},
   "source": [
    "### <a id=\"S1\">S1: How are we progressing toward open access?</a>\n",
    "\n",
    "#### HKUST journal & conference articles, 2014-2023, by OA status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c86462c8-a558-479e-becc-2e02c7643a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "# Create a function to build the API URL\n",
    "def build_url(base_url, endpoint, filters, group_by=None):\n",
    "    filters_str = \",\".join(f\"{key}:{value}\" for key, value in filters.items())\n",
    "    url = f\"{base_url}/{endpoint}?filter={filters_str}\"\n",
    "    if group_by:\n",
    "        url += f\"&group_by={group_by}\"\n",
    "    return url\n",
    "\n",
    "# Create a function to get data from the API\n",
    "def get_data_from_api(url):\n",
    "    response = requests.get(url)\n",
    "    if response.ok:\n",
    "        data = response.json()\n",
    "        return data.get('group_by') or data\n",
    "    return []\n",
    "\n",
    "# Create a function to create and merge data frames\n",
    "def create_and_merge_dfs(data_info):\n",
    "    dfs = [\n",
    "        pd.DataFrame(data)[['key', 'count']]\n",
    "            .rename(columns={'key': 'publication_year', 'count': col_name})\n",
    "            .fillna(0)\n",
    "        for data, col_name in data_info\n",
    "    ]\n",
    "\n",
    "    merged_df = dfs[0]\n",
    "    for df in dfs[1:]:\n",
    "        merged_df = pd.merge(merged_df, df, on='publication_year', how='outer').fillna(0)\n",
    "    return merged_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feeab27c-f59b-4ca9-a31b-6c524668b5e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters\n",
    "base_url = 'https://api.openalex.org'\n",
    "endpoint = 'works'\n",
    "ror_id = '00q4vv597'  # HKUST ROR\n",
    "pub_year_range = '2014-2023'\n",
    "pub_type = 'article'\n",
    "source_type = 'journal|conference'\n",
    "group_by = 'publication_year'\n",
    "\n",
    "# Common filters\n",
    "common_filters = {\n",
    "    'institutions.ror': ror_id,\n",
    "    'publication_year': pub_year_range,\n",
    "    'type': pub_type,\n",
    "    'primary_location.source.type': source_type,\n",
    "    'is_paratext': 'false'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db0876cd-282e-4b91-9eee-0bc9a9a25d10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OA Statuses\n",
    "oa_statuses = ['gold', 'hybrid', 'bronze', 'green', '!gold|!hybrid|!bronze']\n",
    "\n",
    "# Column name mapping\n",
    "column_names = {\n",
    "    'gold': 'gold_oa',\n",
    "    'hybrid': 'hybrid_oa',\n",
    "    'bronze': 'bronze_oa',\n",
    "    'green': 'green_oa',\n",
    "    '!gold|!hybrid|!bronze': 'green_only_oa'\n",
    "}\n",
    "\n",
    "# Collect data for each OA status including non-OA\n",
    "data_info = []\n",
    "for oa_status in oa_statuses + [None]:\n",
    "    filters = common_filters.copy()\n",
    "    filters['is_oa'] = 'true' if oa_status is not None else 'false'\n",
    "    if oa_status:\n",
    "        filters['open_access.oa_status'] = oa_status\n",
    "    \n",
    "    url = build_url(base_url, endpoint, filters, group_by)\n",
    "    print(f'{oa_status} url: {url}')\n",
    "    \n",
    "    data = get_data_from_api(url)\n",
    "    col_name = column_names.get(oa_status, 'non_oa')\n",
    "    data_info.append((data, col_name))\n",
    "\n",
    "# Create and merge dataframes\n",
    "df = create_and_merge_dfs(data_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd3c2a4d-0221-4468-909a-05a2a4c88443",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sorted = df.sort_values('publication_year', ascending=False)\n",
    "df_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4748260f-3cc2-4552-ae3a-2e345eeb7607",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sorted.to_csv('works_by_oa_status.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e18d80f-8715-46e4-8fb3-609f8b71e77b",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50a42856-73f1-4dd3-ac55-f86468fe8772",
   "metadata": {},
   "source": [
    "### <a id=\"S2\">S2: Which journals have we published in the most over the last 10 years?\n",
    "\n",
    "#### HKUST journal articles, 2014-2023, by source (journal)\n",
    "Reference: [openalex-api-tutorials](https://github.com/ourresearch/openalex-api-tutorials/blob/main/notebooks/institutions/uw-collaborators.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0db0dfd2-da33-419b-9ade-3779c215a445",
   "metadata": {},
   "outputs": [],
   "source": [
    "ust_id = \"https://ror.org/00q4vv597\"\n",
    "email = \"#your email\"\n",
    "\n",
    "# specify endpoint\n",
    "endpoint = 'works'\n",
    "\n",
    "# build the 'filter' parameter\n",
    "filters = \",\".join((\n",
    "    f'institutions.ror:{ust_id}',\n",
    "    'is_paratext:false',\n",
    "    'type_crossref:journal-article', \n",
    "    'publication_year:2014-2023'\n",
    "))\n",
    "\n",
    "# put the URL together\n",
    "filtered_works_url = f'https://api.openalex.org/{endpoint}?filter={filters}'\n",
    "\n",
    "if email:\n",
    "    filtered_works_url += f\"&mailto={email}\"\n",
    "print(f'complete URL with filters:\\n{filtered_works_url}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bfcd4e9-5911-4e5c-ad9f-f49dfb678903",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "r = requests.get(filtered_works_url)\n",
    "results_page = r.json()\n",
    "print(f\"works retrieved: \\n{results_page['meta']['count']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71cef73f-6dac-49af-a188-4ca3d85b8490",
   "metadata": {},
   "outputs": [],
   "source": [
    "# there are 25 results per page\n",
    "results_page['meta']['count'] / 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4699285-db14-408d-af1c-bf3b69bcf615",
   "metadata": {},
   "outputs": [],
   "source": [
    "cursor = '*'\n",
    "\n",
    "select = \",\".join((\n",
    "    'id',\n",
    "    'ids',\n",
    "    'title',\n",
    "    'display_name',\n",
    "    'publication_year',\n",
    "    'publication_date',\n",
    "    'primary_location',\n",
    "    'open_access',\n",
    "    'authorships',\n",
    "    'cited_by_count',\n",
    "    'is_retracted',\n",
    "    'is_paratext',\n",
    "))\n",
    "\n",
    "# loop through pages\n",
    "works = []\n",
    "loop_index = 0\n",
    "while cursor:\n",
    "    \n",
    "    # set cursor value and request page from OpenAlex\n",
    "    url = f'{filtered_works_url}&select={select}&cursor={cursor}'\n",
    "    page_with_results = requests.get(url).json()\n",
    "    \n",
    "    results = page_with_results['results']\n",
    "    works.extend(results)\n",
    "\n",
    "    # update cursor to meta.next_cursor\n",
    "    cursor = page_with_results['meta']['next_cursor']\n",
    "    loop_index += 1\n",
    "    if loop_index in [5, 10, 20, 50, 100] or loop_index % 500 == 0:\n",
    "        print(f'{loop_index} api requests made so far')\n",
    "print(f'done. made {loop_index} api requests. collected {len(works)} works')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46ee59db-b472-4b61-8524-8706307cf29a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save raw data to pickle\n",
    "import pickle\n",
    "\n",
    "# uncomment these lines and run to save the results so we won't have to fetch them\n",
    "# again next time we run the notebook\n",
    "\n",
    "# pickle file in the current folder\n",
    "import os\n",
    "with open('hkust_j_2014-2023.pickle', 'wb') as outf:\n",
    "    pickle.dump(works, outf, protocol=pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb653744-3988-4eb3-b1d5-ee08d4fcdd57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OR uncomment these lines and run to load the saved results\n",
    "with open('hkust_j_2014-2023.pickle', 'rb') as f:\n",
    "    works = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb68931c-093a-4eb3-8214-41a24d8d0283",
   "metadata": {},
   "outputs": [],
   "source": [
    "# works[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69dee5ac-412e-4c92-ac98-00c293c7b201",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = []\n",
    "\n",
    "for work in works:\n",
    "    work_id = work.get('id')\n",
    "    work_ids = work.get('ids', {})\n",
    "    work_doi = work_ids.get('doi')\n",
    "    work_publication_year = work.get('publication_year')\n",
    "    work_title = work.get('title')\n",
    "\n",
    "    # Initialize other variables that might be populated from different fields\n",
    "    source_display_name = None\n",
    "    host_organization_name = None\n",
    "    host_organization_lineage_names = None\n",
    "\n",
    "    # Check if primary_location and source exist and are not None\n",
    "    primary_location = work.get('primary_location')\n",
    "    if primary_location:\n",
    "        source = primary_location.get('source')\n",
    "        if source:\n",
    "            source_display_name = source.get('display_name')\n",
    "            issn_l = source.get('issn_l')\n",
    "            issn_value = source.get('issn', [])\n",
    "            issn = \"; \".join(issn_value) if isinstance(issn_value, (list, tuple)) else str(issn_value)\n",
    "            host_organization_name = source.get('host_organization_name')\n",
    "            host_organization_lineage_names = \", \".join(source.get('host_organization_lineage_names', []))\n",
    "\n",
    "    # Append a dictionary with the required information to data list\n",
    "    data.append({\n",
    "        'openalex_id': work_id,\n",
    "        # 'title': work_title,\n",
    "        'pub_year': work_publication_year,\n",
    "        'journal': source_display_name,\n",
    "        'issn_l': issn_l,\n",
    "        'publisher': host_organization_name,\n",
    "        'publisher_full': host_organization_lineage_names,\n",
    "        'doi': work_doi\n",
    "    })\n",
    "\n",
    "df = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "293b5d04-327b-4f89-9433-15631d340297",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f7faa87-f83a-48a0-b03e-5d7aea8649c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the pivot table for counts of openalex_id\n",
    "pivot_table = df.pivot_table(index='journal', columns='pub_year', \n",
    "                             values='openalex_id', aggfunc='count', fill_value=0)\n",
    "\n",
    "# Assuming each journal has a single publisher, we take the first publisher for each journal\n",
    "publishers = df.drop_duplicates(subset='journal').set_index('journal')['publisher']\n",
    "\n",
    "# Join the pivot table with the publisher series\n",
    "final_df = pivot_table.join(publishers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de59ada6-98c9-4654-b54e-7437924e91a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_publisher_name(publisher):\n",
    "    if publisher is None:\n",
    "        return publisher\n",
    "\n",
    "    mapping = {\n",
    "        \"SpringerNature\": [\"springer\", \"nature\", \"biomed central\", \"bmc\"],\n",
    "        \"Elsevier\": [\"elsevier\", \"RELX\"],\n",
    "        \"Wiley\": [\"wiley\", \"hindawi\"],\n",
    "        \"IEEE\": [\"IEEE\", \"Institute of Electrical and Electronics Engineers\"],\n",
    "        \"MDPI\": [\"Multidisciplinary Digital Publishing Institute\", \"mdpi\"], \n",
    "        \"Taylor & Francis\": [\"Routledge\", \"Francis\"],\n",
    "        \"SAGE\": [\"SAGE\"],\n",
    "        \"De Gruyter\": [\"De Gruyter\"],\n",
    "        \"Emerald\": [\"Emerald\"],\n",
    "        \"CUP\": [\"cambridge university press\", \"cup\"],\n",
    "        \"OUP\": [\"oxford university press\", \"oup\"],\n",
    "        \"ACS\": [\"American Chemical Society\"],\n",
    "        \"RSC\": [\"Royal Society of Chemistry\"],\n",
    "        \"ACM\": [\"Association for Computing Machinery\"],\n",
    "        \"APS\": [\"American Physical Society\"],\n",
    "        \"IOP\": [\"IOP\", \"Institute of Physics\"],\n",
    "        \"ASCE\": [\"American Society of Civil Engineers\"],\n",
    "        \"IET\": [\"Institution of Engineering and Technology\"],\n",
    "        \"AOM\": [\"Academy of Management\"],\n",
    "        \"AAAS\": [\"American Association for the Advancement of Science\"],\n",
    "        \"INFORMS\": [\"Institute for Operations Research and the Management Sciences\"],\n",
    "        \"Frontiers\": [\"Frontiers\"],\n",
    "        \"eLife\": [\"eLife\"],        \n",
    "        \"PLoS\": [\"Public Library of Science\"],  \n",
    "    }\n",
    "    \n",
    "    publisher_lower = publisher.lower()\n",
    "    for normalized_name, variants in mapping.items():\n",
    "        if any(variant.lower() in publisher_lower for variant in variants):\n",
    "            return normalized_name\n",
    "    return publisher  # Return the original name if no match is found\n",
    "\n",
    "# Now apply the map function with the updated normalize_publisher_name function\n",
    "publishers = publishers.map(normalize_publisher_name)\n",
    "\n",
    "# Join the pivot table with the normalized publishers Series\n",
    "final_df = pivot_table.join(publishers)\n",
    "\n",
    "# Sort the DataFrame by the year 2023 in descending order\n",
    "sorted_final_df = final_df.sort_values(by=2023, ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e42475-e37b-43a5-af86-a2c470b144f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_final_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6d6f179-4c5c-4630-9f8b-ca200e93788c",
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_final_df.to_csv('j_by_source_publisher.csv', index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b597440-e7af-4a8e-ba36-61501c9b2d45",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55e2a884-c34a-48b7-b1cc-4abcb572c018",
   "metadata": {},
   "source": [
    "### <a id=\"S3\">S3: How has our contribution to the Sustainable Development Goals (SDG) evolved over the years?\n",
    "\n",
    "#### HKUST journal + conference articles, 2014-2023, by SDG areas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d2e16ed-42bc-403c-84d8-3a9a3f4b961b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "# Create a function to build the API URL\n",
    "def build_url(ror_id, pub_year, publication_type, group_by=None):\n",
    "    # specify endpoint\n",
    "    endpoint = 'works'\n",
    "\n",
    "    # build the base 'filter' parameter list with required filters\n",
    "    filters = [\n",
    "        f'institutions.ror:{ror_id}',\n",
    "        f'publication_year:{pub_year}',\n",
    "        f'type:{publication_type}',\n",
    "        'is_paratext:false'\n",
    "    ]\n",
    "    \n",
    "    # join the filter elements into a single string\n",
    "    filters_str = \",\".join(filters)\n",
    "\n",
    "    # put the URL together\n",
    "    filtered_works_url = f'https://api.openalex.org/{endpoint}?filter={filters_str}'\n",
    "\n",
    "    # add 'group_by' to URL if provided\n",
    "    if group_by:\n",
    "        filtered_works_url += f'&group_by={group_by}'\n",
    "\n",
    "    return filtered_works_url\n",
    "\n",
    "# Create a function to get data from the API\n",
    "def get_data_from_api(url):\n",
    "    response = requests.get(url)\n",
    "    data = response.json()\n",
    "    if 'group_by' in data:\n",
    "        return data['group_by']\n",
    "    return data or []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a67a240b-3553-4090-aa49-4fe269f2f4ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set parameters\n",
    "ror_id = '00q4vv597'\n",
    "publication_type = 'article'\n",
    "group_by = 'sustainable_development_goals.id'\n",
    "years = range(2014, 2024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f83a230c-4b61-45f1-9688-b1fcfa6091ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import reduce\n",
    "\n",
    "# Create a function to create a dataframe\n",
    "def create_df(data, col_name):\n",
    "    df = pd.DataFrame(data)[['key', 'key_display_name', 'count']]\n",
    "    df = df.rename(columns={'key_display_name': 'SDG', 'count': col_name})\n",
    "    df = df.set_index('key')\n",
    "    return df.fillna(0)\n",
    "\n",
    "# Loop through each year, get data and merge into a single dataframe\n",
    "dfs = []\n",
    "for year in years:\n",
    "    url = build_url(ror_id, year, publication_type, group_by)\n",
    "    data = get_data_from_api(url)\n",
    "    df = create_df(data, str(year))\n",
    "    dfs.append(df)\n",
    "    \n",
    "df = reduce(lambda left, right: pd.merge(left, right, on='SDG', how='outer'), dfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f78743b-1733-4160-90f6-c8976480056d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sorted_2023 = df.sort_values('2023', ascending=False)\n",
    "df_sorted_2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d8cdda2-e863-4bec-b7b9-370b36ac1a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the first 7 rows\n",
    "top_rows = df_sorted_2023.iloc[:7]\n",
    "\n",
    "# Sum the rest of the rows\n",
    "other_sdg_data = df_sorted_2023.iloc[7:].sum().to_frame().T\n",
    "\n",
    "# Concatenate the top rows with the summed 'Other SDGs' row\n",
    "result_df = pd.concat([top_rows, other_sdg_data], ignore_index=True)\n",
    "result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1791fb11-9549-4044-9c71-7d65aa1f4938",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trans = result_df.transpose()\n",
    "\n",
    "new_header = df_trans.iloc[0]\n",
    "df_trans = df_trans[1:]\n",
    "df_trans.columns = new_header\n",
    "df_new = df_trans.reset_index()\n",
    "df_new.rename(columns={'index': 'publication_year', df_new.columns[-1]: 'Other SDGs'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99ff1e50-7cee-4984-a6e1-c7758faeef41",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all HKUST articles by year\n",
    "ror_id = '00q4vv597'\n",
    "publication_type = 'article'\n",
    "pub_year = '2014-2023'\n",
    "group_by = 'publication_year'\n",
    "\n",
    "url = build_url(ror_id, pub_year, publication_type, group_by)\n",
    "data = get_data_from_api(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c41fe4bf-3bc9-407f-bcb6-e9d363a0d0f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all = pd.DataFrame(data)[['key', 'count']].rename(columns={'key': 'publication_year', 'count': 'all articles'})\n",
    "df_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f05c2ad6-7978-4d53-a28e-29544a1e43a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count Non SDGs articles by year\n",
    "merged_df = pd.merge(df_new, df_all, on='publication_year', how='outer') \n",
    "\n",
    "merged_df['Non SDGs'] = merged_df['all articles'] - merged_df.iloc[:, 1:9].sum(axis=1)\n",
    "merged_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a398523-b6fc-4b2c-825e-dee35e02ee6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df.to_csv('works_by_sdg.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5986b6ff-7f3b-484b-a2b8-34b1e1e33cc1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065da128-9a43-424d-8e19-b406e94c7948",
   "metadata": {},
   "source": [
    "### <a id=\"S4\">S4: Which of our publications have had the most impact over the years?\n",
    "\n",
    "#### HKUST journal  & conference articles, 2014-2023, by citations\n",
    "Reference: [openalex-api-tutorials](https://github.com/ourresearch/openalex-api-tutorials/blob/main/notebooks/institutions/uw-collaborators.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2c1deb1-0e74-4717-bd5f-1d626fb77fdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "ust_id = \"https://ror.org/00q4vv597\"\n",
    "email = \"#your email\"\n",
    "\n",
    "# specify endpoint\n",
    "endpoint = 'works'\n",
    "\n",
    "# build the 'filter' parameter\n",
    "filters = \",\".join((\n",
    "    f'institutions.ror:{ust_id}',\n",
    "    'is_paratext:false',\n",
    "    'type:article', \n",
    "    'publication_year:2014-2023'\n",
    "))\n",
    "\n",
    "# put the URL together\n",
    "filtered_works_url = f'https://api.openalex.org/{endpoint}?filter={filters}'\n",
    "\n",
    "if email:\n",
    "    filtered_works_url += f\"&mailto={email}\"\n",
    "print(f'complete URL with filters:\\n{filtered_works_url}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "028ae4e9-13eb-45d1-86f2-3bba9b12ef57",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "r = requests.get(filtered_works_url)\n",
    "results_page = r.json()\n",
    "print(f\"works retrieved: \\n{results_page['meta']['count']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b48bb4d0-d4ca-43a2-ac13-cec9d9d25340",
   "metadata": {},
   "outputs": [],
   "source": [
    "# there are 25 results per page\n",
    "results_page['meta']['count'] / 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22638492-9d52-449c-a584-b3a4dc94e4ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "cursor = '*'\n",
    "\n",
    "select = \",\".join((\n",
    "    'id',\n",
    "    'ids',\n",
    "    'title',\n",
    "    'display_name',\n",
    "    'publication_year',\n",
    "    'publication_date',\n",
    "    'primary_location',\n",
    "    'open_access',\n",
    "    'authorships',\n",
    "    'cited_by_count',\n",
    "    'counts_by_year',\n",
    "    'is_retracted',\n",
    "    'is_paratext',\n",
    "    'concepts',\n",
    "))\n",
    "\n",
    "# loop through pages\n",
    "works = []\n",
    "loop_index = 0\n",
    "while cursor:\n",
    "    \n",
    "    # set cursor value and request page from OpenAlex\n",
    "    url = f'{filtered_works_url}&select={select}&cursor={cursor}'\n",
    "    page_with_results = requests.get(url).json()\n",
    "    \n",
    "    results = page_with_results['results']\n",
    "    works.extend(results)\n",
    "\n",
    "    # update cursor to meta.next_cursor\n",
    "    cursor = page_with_results['meta']['next_cursor']\n",
    "    loop_index += 1\n",
    "    if loop_index in [5, 10, 20, 50, 100] or loop_index % 250 == 0:\n",
    "        print(f'{loop_index} api requests made so far')\n",
    "print(f'done. made {loop_index} api requests. collected {len(works)} works')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c904156-97c6-4ec5-89f2-1b288306f5cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save raw data to pickle\n",
    "import pickle\n",
    "\n",
    "# uncomment these lines and run to save the results so we won't have to fetch them\n",
    "# again next time we run the notebook\n",
    "\n",
    "# # pickle file in the current folder\n",
    "# import os\n",
    "# with open('hkust_2014-2023_cites_concepts.pickle', 'wb') as outf:\n",
    "#     pickle.dump(works, outf, protocol=pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16e5f6b4-6e56-4112-8ddd-7ae5e1b2fb8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OR uncomment these lines and run to load the saved results\n",
    "with open('hkust_2014-2023_cites_concepts.pickle', 'rb') as f:\n",
    "    works = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e7772ef-a455-4add-8b14-247de3a669f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# works[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d796933-51d6-43c7-a2fa-605f162afdfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Define your concept of interest\n",
    "# concept_of_interest = 'artificial intelligence'  # Set to None to include all works\n",
    "concept_of_interest =  None\n",
    "\n",
    "data = []\n",
    "\n",
    "for work in works:\n",
    "    # If a concept of interest is specified, check for it; otherwise, include all works.\n",
    "    if concept_of_interest is None or any(concept.get('display_name') == concept_of_interest for concept in work.get('concepts', [])):\n",
    "        work_id = work.get('ids', {}).get('openalex')\n",
    "        work_doi = work.get('ids', {}).get('doi')\n",
    "        work_publication_year = work.get('publication_year')\n",
    "        work_title = work.get('title')\n",
    "        cited_by_count = work.get('cited_by_count')\n",
    "        counts_by_year_dict = {count['year']: count['cited_by_count'] for count in work.get('counts_by_year', [])}\n",
    "\n",
    "        # Initialize columns for each year\n",
    "        year_columns = {str(year): counts_by_year_dict.get(year, 0) for year in range(2014, 2024)}\n",
    "        \n",
    "        # Get the first author's name and surname, if available\n",
    "        authorships = work.get('authorships', [])\n",
    "        first_author_name = authorships[0]['author']['display_name'] if authorships else None\n",
    "        \n",
    "        # Extract the surname by splitting the name and taking the last part\n",
    "        first_author_surname = first_author_name.split()[-1] if first_author_name else None\n",
    "\n",
    "        # Concatenate surname with publication year to form author_year\n",
    "        author_year = f\"{first_author_surname}_{work_publication_year}\" if first_author_surname else None\n",
    "        # Initialize other variables that might be populated from different fields\n",
    "        source_display_name = None\n",
    "\n",
    "        # Check if primary_location and source exist and are not None\n",
    "        primary_location = work.get('primary_location')\n",
    "        if primary_location:\n",
    "            source = primary_location.get('source')\n",
    "            if source:\n",
    "                source_display_name = source.get('display_name')\n",
    "\n",
    "        # Append a dictionary with the required information to data list\n",
    "        data.append({\n",
    "            'openalex_id': work_id,\n",
    "            'first_author': first_author_name,\n",
    "            'author_year': author_year,\n",
    "            'title': work_title,\n",
    "            'pub_year': work_publication_year,\n",
    "            'journal': source_display_name,\n",
    "            'doi': work_doi,\n",
    "            'cited_by_count': cited_by_count,\n",
    "            **year_columns  # Unpack the year_columns dictionary into the data dictionary\n",
    "        })\n",
    "\n",
    "# Create a DataFrame from the data list\n",
    "df = pd.DataFrame(data)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95f1c091-087c-4a99-96b3-59fbb072b69d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count no. of works with total citations over 10\n",
    "cites_over_10 = len(df[df['cited_by_count'] >= 10])\n",
    "cites_over_10 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dec0f52e-dffc-4b41-b658-0994a39e25c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cites_over_10 = df[df['cited_by_count'] >= 10]\n",
    "df_cites_over_10.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b7afaac-a248-4a34-924b-44c9ed830a1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cites_over_10.to_csv('works_by_citations_over_10.csv', index=False, encoding='utf-8-sig')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}