harvest-pandora-subject-collections.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a86913b-9a41-4013-b700-1696982517d7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Harvest Pandora subjects and collections\n",
    "\n",
    "This notebook harvests Pandora's navigation hierarchy, saving the connections between subjects, collections, and titles.\n",
    "\n",
    "The [Pandora](http://pandora.nla.gov.au/) selective web archive assigns archived titles to subject and collection groupings. These curated collections help researchers find archived websites relating to specific topics or events, such as [election campaigns](http://pandora.nla.gov.au/subject/6). This notebook creates two datasets containing details of all Pandora's subjects and collections. The datasets can be used to [assemble subject-based collections of archived websites for research](https://glam-workbench.net/trove-web-archives/create-datasets/).\n",
    "\n",
    "## Pandora vs Trove\n",
    "\n",
    "The relationship between Pandora and Trove is a bit confusing. While the websites archived in Pandora are now part of the Australian Web Archive, and are searchable through Trove, not all of Pandora's metadata can be accessed through the Trove web interface.\n",
    "\n",
    "Trove's [Categories](https://trove.nla.gov.au/landing/categories) tab includes a link to [Archived Webpage Collections](https://webarchive.nla.gov.au/collection). This collection hierarchy is basically the same as Pandora's – combining Pandora's subjects, subcategories, and collections into a single structure. However, it only includes links to titles that are part of collections. This is important, as less than half of Pandora's selected titles seem to be assigned to collections.\n",
    "\n",
    "I originally started harvesting the collections from Trove, but eventually realised that I was missing out on titles that had been grouped by subject, but were not part of collections. As a result, I shifted approaches to scrape the data from Pandora directly.\n",
    "\n",
    "## Subjects, Collections, and Titles\n",
    "\n",
    "There are two levels of subject headings in Pandora. The top-level headings are displayed on the Pandora home page, for example, [Arts](http://pandora.nla.gov.au/subject/2) and [Politics](http://pandora.nla.gov.au/subject/21). The top-level headings can include sub-categories. For example, 'Arts' includes sub-categories for 'Architecture' and 'Dance'. Both the top-level subjects and sub-categories can include collections and titles.\n",
    "\n",
    "Collections are more fine-grained groupings of titles, often related to specific events or activities. Collections can include sub-collections. In Pandora's web interface, the sub-collections are displayed as sub-headings on the collection page, but in the backend each sub-collection has its own identifier. For example, the 'Galleries' collection, includes a list of gallery websites divided into sub-collections by the state in which they're located. Both collections and sub-collections can contain titles.\n",
    "\n",
    "Collections can appear in multiple subjects and sub-categories. This means that the harvesting process saves duplicate copies of collections that need to be removed.\n",
    "\n",
    "Titles are also a type of group, bringing together webpage snapshots over time. They can also link urls where the addresses or domains of resources have changed. As a result, each title can be associated with multiple urls. This notebook doesn't harvest the full title details, it simply links title identifiers with subjects and collections. See [Harvest the full collection of Pandora titles](harvest-pandora-titles.ipynb) for more.\n",
    "\n",
    "Titles can be linked to any level in this hierarchy. So to assemble a complete list of titles under a subject such as 'Arts', you need to get all the titles from 'Arts', all of the titles from all of the sub-categories under 'Arts', and all of the titles from all of the collections and sub-collections under both 'Arts' and its subcategories. See [Create archived url datasets from Pandora's collections and subjects](create-datasets.ipynb) for an example of this.\n",
    "\n",
    "For more on Pandora's approach to describing collections see [Creating Structure in Web Archives With Collections: Different Concepts From Web Archivists](https://doi.org/10.48550/arXiv.2209.08649).\n",
    "\n",
    "## Datasets\n",
    "\n",
    "This notebook creates two datasets in `ndjson` format (one JSON object per line):\n",
    "\n",
    "- `pandora-subjects.ndjson`\n",
    "- `pandora-collections.ndjson`\n",
    "\n",
    "The `pandora-subjects.ndjson` file includes the following fields:\n",
    "\n",
    "- `name` – subject heading\n",
    "- `id` – subject identifier in the form `/subject/[number]`\n",
    "- `subcategories` – list of subcategory identifiers\n",
    "- `collections` – list of collection identifiers\n",
    "- `titles` – list of title identifiers\n",
    "\n",
    "The `pandora-collections.ndjson` file includes the following fields:\n",
    "\n",
    "- `name` – collection/subcollection name\n",
    "- `id` – collection identifier in the form `/col/[number]`\n",
    "- `subcollections` – list of subcollection identifiers\n",
    "- `titles` – list of title identifiers\n",
    "\n",
    "Pre-harvested versions of these datasets are available from the [Pandora collections data](https://glam-workbench.net/trove-web-archives/pandora-collections-data/) section of the GLAM Workbench.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dabdf361-6f0a-4c9c-91bd-14a92b45e6a9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "from dotenv import load_dotenv\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d6f6094-ce03-48d7-9ef2-7cdac14dba11",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class SubjectHarvester:\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        subject_output=\"pandora-subjects.ndjson\",\n",
    "        collection_output=\"pandora-collections.ndjson\",\n",
    "        sample=None,\n",
    "    ):\n",
    "        self.subject_output = subject_output\n",
    "        self.collection_output = collection_output\n",
    "        self.sample = sample\n",
    "\n",
    "    def get_title_ids(self, page_id):\n",
    "        \"\"\"\n",
    "        Get the TEP identifiers for all the titles on the specified page.\n",
    "        Excludes titles in subcollections as they will can be harvested separately.\n",
    "        \"\"\"\n",
    "        title_ids = []\n",
    "        page = 1\n",
    "        # Subjects can have multiple pages of titles, so we'll go through page by page\n",
    "        # until there's no more titles\n",
    "        while page:\n",
    "            response = requests.get(f\"http://pandora.nla.gov.au{page_id}/{page}\")\n",
    "            soup = BeautifulSoup(response.text, \"lxml\")\n",
    "            # we only want the first itemlist containing titles\n",
    "            # subsequent titles will be part of subcollections\n",
    "            title_links = []\n",
    "            for item_list in soup.find_all(\"div\", class_=\"itemlist\"):\n",
    "                # This checks if the title list has an h1 tag before it\n",
    "                # which indicates its actually a subcollection\n",
    "                if not (\n",
    "                    item_list.find_previous_sibling(\"h1\")\n",
    "                    and item_list.find_previous_sibling(\"h1\").name == \"h1\"\n",
    "                ):\n",
    "                    # Extract the TEP ids from the links\n",
    "                    title_links = item_list.find_all(\"a\", href=re.compile(r\"/tep/\\d+\"))\n",
    "                    for title_link in title_links:\n",
    "                        title_ids.append(title_link[\"href\"])\n",
    "            # Continue if it's a subject page and there were title links on this page\n",
    "            if title_links and \"/col/\" not in page_id:\n",
    "                page += 1\n",
    "            else:\n",
    "                page = None\n",
    "            time.sleep(0.5)\n",
    "        return title_ids\n",
    "\n",
    "    def harvest_subcategories(self, subject_id):\n",
    "        \"\"\"\n",
    "        Harvest details of sub-categories from a subject page.\n",
    "        \"\"\"\n",
    "        subject_ids = []\n",
    "        # Get the subject page\n",
    "        response = requests.get(f\"http://pandora.nla.gov.au{subject_id}\")\n",
    "        soup = BeautifulSoup(response.text, \"lxml\")\n",
    "        # Get all the links to subcategories\n",
    "        subject_links = soup.find_all(\"a\", href=re.compile(r\"/subject/\\d+$\"))\n",
    "        # Process all the sub-categories\n",
    "        for subject_link in subject_links:\n",
    "            subject_name = \" \".join(subject_link.stripped_strings)\n",
    "            subject_id = subject_link[\"href\"]\n",
    "            # Get collections\n",
    "            collection_ids = self.harvest_collections(subject_id)\n",
    "            # Get titles\n",
    "            title_ids = self.get_title_ids(subject_id)\n",
    "            with Path(self.subject_output).open(\"a\") as subjects_file:\n",
    "                subjects_file.write(\n",
    "                    json.dumps(\n",
    "                        {\n",
    "                            \"name\": subject_name,\n",
    "                            \"id\": subject_id,\n",
    "                            \"collections\": collection_ids,\n",
    "                            \"titles\": title_ids,\n",
    "                            \"subcategories\": []\n",
    "                        }\n",
    "                    )\n",
    "                    + \"\\n\"\n",
    "                )\n",
    "            subject_ids.append(subject_id)\n",
    "        return subject_ids\n",
    "\n",
    "    def harvest_subcollections(self, coll_id, coll_name):\n",
    "        \"\"\"\n",
    "        Harvest sub-collections from a collection page.\n",
    "        \"\"\"\n",
    "        collection_ids = []\n",
    "        # Get the collection page\n",
    "        response = requests.get(f\"http://pandora.nla.gov.au{coll_id}\")\n",
    "        soup = BeautifulSoup(response.text, \"lxml\")\n",
    "        # Sub-collections are included in the collection pages and identified with h1 headings.\n",
    "        # The h1 headings include a name attribute that is set to the sub-collection id.\n",
    "        # You can use the id to request a page that just has the subcollection.\n",
    "        # First get all the h1 tags\n",
    "        for subc in soup.find_all(\"h1\"):\n",
    "            # Get the id value from the name attribute\n",
    "            sub_link = subc.find(\"a\", {\"name\": re.compile(r\"\\d+\")})\n",
    "            if sub_link:\n",
    "                sub_name = sub_link.string\n",
    "                # Add the collection name to the sub collection name (if it's not already there)\n",
    "                if coll_name not in sub_name:\n",
    "                    sub_name = f\"{coll_name} - {sub_name}\"\n",
    "                # Use the sub-collection id to get a list of titles in the sub-collection\n",
    "                sub_id = f\"/col/{sub_link['name']}\"\n",
    "                title_ids = self.get_title_ids(sub_id)\n",
    "                with Path(self.collection_output).open(\"a\") as collections_file:\n",
    "                    collections_file.write(\n",
    "                        json.dumps(\n",
    "                            {\n",
    "                                \"name\": sub_name,\n",
    "                                \"id\": sub_id,\n",
    "                                \"titles\": title_ids,\n",
    "                                \"subcollections\": [],\n",
    "                            }\n",
    "                        )\n",
    "                        + \"\\n\"\n",
    "                    )\n",
    "                collection_ids.append(sub_id)\n",
    "        return collection_ids\n",
    "\n",
    "    def harvest_collections(self, subject_id):\n",
    "        \"\"\"\n",
    "        Harvest details of collections from a subject, or sub-category page.\n",
    "        \"\"\"\n",
    "        collection_ids = []\n",
    "        # Get the subject page\n",
    "        response = requests.get(f\"http://pandora.nla.gov.au{subject_id}\")\n",
    "        soup = BeautifulSoup(response.text, \"lxml\")\n",
    "        # Get all of the links to collection pages\n",
    "        collection_links = soup.find_all(\"a\", href=re.compile(r\"/col/\\d+$\"))\n",
    "        # Process each collection page\n",
    "        for coll_link in collection_links:\n",
    "            coll_name = \" \".join(coll_link.stripped_strings)\n",
    "            coll_id = coll_link[\"href\"]\n",
    "            # Get any sub-collections\n",
    "            subcollection_ids = self.harvest_subcollections(coll_id, coll_name)\n",
    "            # Get titles\n",
    "            title_ids = self.get_title_ids(coll_id)\n",
    "            with Path(self.collection_output).open(\"a\") as collections_file:\n",
    "                collections_file.write(\n",
    "                    json.dumps(\n",
    "                        {\n",
    "                            \"name\": coll_name,\n",
    "                            \"id\": coll_id,\n",
    "                            \"subcollections\": subcollection_ids,\n",
    "                            \"titles\": title_ids,\n",
    "                        }\n",
    "                    )\n",
    "                    + \"\\n\"\n",
    "                )\n",
    "            collection_ids.append(coll_id)\n",
    "        return collection_ids\n",
    "\n",
    "    def harvest(self):\n",
    "        \"\"\"\n",
    "        Start the harvest by getting the top-level subjects on the Pandora home page\n",
    "        and work down the hierarchy from there.\n",
    "        \"\"\"\n",
    "        # Remove old data files\n",
    "        Path(self.subject_output).unlink(missing_ok=True)\n",
    "        Path(self.collection_output).unlink(missing_ok=True)\n",
    "        # Get the Pandora home page\n",
    "        response = requests.get(\"http://pandora.nla.gov.au/\")\n",
    "        soup = BeautifulSoup(response.text, \"lxml\")\n",
    "        # Find the list of subjects\n",
    "        subject_list = soup.find(\"div\", class_=\"browseSubjects\").find_all(\"li\")\n",
    "        # Process each top-level subject\n",
    "        for subject in tqdm(subject_list[: self.sample]):\n",
    "            subject_link = subject.find(\"a\")\n",
    "            subject_name = \" \".join(subject_link.stripped_strings)\n",
    "            subject_id = subject_link[\"href\"]\n",
    "            # Get subcategories\n",
    "            subcategory_ids = self.harvest_subcategories(subject_id)\n",
    "            # Get collections\n",
    "            subcollection_ids = self.harvest_collections(subject_id)\n",
    "            # Get titles\n",
    "            title_ids = self.get_title_ids(subject_id)\n",
    "            with Path(self.subject_output).open(\"a\") as subjects_file:\n",
    "                subjects_file.write(\n",
    "                    json.dumps(\n",
    "                        {\n",
    "                            \"name\": subject_name,\n",
    "                            \"id\": subject_id,\n",
    "                            \"subcategories\": subcategory_ids,\n",
    "                            \"collections\": subcollection_ids,\n",
    "                            \"titles\": title_ids,\n",
    "                        }\n",
    "                    )\n",
    "                    + \"\\n\"\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07258429-8f1e-4acd-a778-ce2beaa884d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "harvester = SubjectHarvester()\n",
    "harvester.harvest()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd2f2fe-08a1-41c7-827a-968620b1635f",
   "metadata": {},
   "source": [
    "## Remove duplicate collections\n",
    "\n",
    "Collections can appear under multiple subjects, so there will be duplicates in the collections dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "983b6be0-3e05-49e6-8793-e824284a1344",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "dfc = pd.read_json(\"pandora-collections.ndjson\", lines=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57cbbb9f-1feb-4026-97e3-76fd92eb246f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "dfc.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31e50849-089a-4788-b047-546cef627e0c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "dfc.drop_duplicates(subset=[\"id\"], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4692b6c-7be9-442c-a143-d9f807ccde20",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "dfc.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37499022-7212-48e5-9c40-e4d8ca4ab21e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "dfc.to_json(\"pandora-collections.ndjson\", orient=\"records\", lines=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "150d9e88-3e3d-43ed-82a9-75cb778f677d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# IGNORE CELL --TESTING ONLY\n",
    "if os.getenv(\"GW_STATUS\") == \"dev\":\n",
    "\n",
    "    harvester = SubjectHarvester(\n",
    "        subject_output=\"pandora-subjects-test.ndjson\",\n",
    "        collection_output=\"pandora-collections-test.ndjson\",\n",
    "        sample=1,\n",
    "    )\n",
    "    harvester.harvest()\n",
    "\n",
    "    Path(\"pandora-subjects-test.ndjson\").unlink(missing_ok=True)\n",
    "    Path(\"pandora-collections-test.ndjson\").unlink(missing_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3e531ff-31cb-49ef-b9a7-06bad671cede",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherratt](https://timsherratt.au/) for the [GLAM Workbench](https://glam-workbench.net/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "position": 1,
  "rocrate": {
   "action": [
    {
     "description": "This dataset contains details of the subject and collection groupings used by Pandora to organise archived web resource titles.",
     "isPartOf": "https://github.com/GLAM-Workbench/trove-web-archives-collections",
     "mainEntityOfPage": "https://glam-workbench.net/trove-web-archives/pandora-collections-data",
     "result": [
      {
       "license": "https://creativecommons.org/publicdomain/zero/1.0/",
       "url": "https://github.com/GLAM-Workbench/trove-web-archives-collections/raw/main/pandora-subjects.ndjson"
      },
      {
       "license": "https://creativecommons.org/publicdomain/zero/1.0/",
       "url": "https://github.com/GLAM-Workbench/trove-web-archives-collections/raw/main/pandora-collections.ndjson"
      }
     ]
    }
   ],
   "author": [
    {
     "mainEntityOfPage": "https://timsherratt.au",
     "name": "Sherratt, Tim",
     "orcid": "https://orcid.org/0000-0001-7956-4498"
    }
   ],
   "description": "This notebook harvests Pandora's navigation hierarchy, saving the connections between subjects, collections, and titles.\n\nThe [Pandora](http://pandora.nla.gov.au/) selective web archive assigns archived titles to subject and collection groupings. These curated collections help researchers find archived websites relating to specific topics or events, such as [election campaigns](http://pandora.nla.gov.au/subject/6). This notebook creates two datasets containing details of all Pandora's subjects and collections. The datasets can be used to [assemble subject-based collections of archived websites for research](https://glam-workbench.net/trove-web-archives/create-datasets/).",
   "mainEntityOfPage": "https://glam-workbench.net/trove-web-archives/harvest-pandora-subject-collections/",
   "name": "Harvest Pandora subjects and collections"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}