notebooks/community/wayback-machine-downloader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wayback Machine Downloader\n",
    "\n",
    "A command line tool to download an entire website from the Internet Archive Wayback Machine.\n",
    "\n",
    "In this notebook, we will install the tool, download an archived website and save it to your computer.\n",
    "\n",
    "Read [more on the tool's repository](https://github.com/hartator/wayback-machine-downloader)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important note on the tool status\n",
    "The wayback machine now enforces stricter rate limits to the requested content. At the time of writing, the original tool has not been updated to accommodate this, but the open source code community has produced an alternative version, we have created a copy of this fixed version and that's the code we will download and install here, the installation process is still a single click but looks much more complex because it does not come directly from the official code distribution platform, which is outdated.\n",
    "\n",
    "To learn more about this read [this discussion on GitHub](https://github.com/hartator/wayback-machine-downloader/pull/268)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1 - install the fixed version of the code\n",
    "We will download the fixed code, and install the needed things for [Ruby](https://www.ruby-lang.org/en/) tools like this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# download the code\n",
    "git init .\n",
    "git remote add -t \\* -f origin https://github.com/msramalho/wayback-machine-downloader\n",
    "git checkout configurable_delay\n",
    "\n",
    "# install necessary dependencies\n",
    "apt install rubygems && gem install bundler && bundle install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# call the help command to validate the installation\n",
    "! ./bin/wayback_machine_downloader --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2 - download a simple website\n",
    "We will download the archived contents of [example.com](https://example.com/) with the default options.\n",
    "\n",
    "You can easily replace it with another website on the cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "WEBSITE_TO_DOWNLOAD = \"example.com\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "! ./bin/wayback_machine_downloader $WEBSITE_TO_DOWNLOAD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The downloaded website's contents are inside a new folder `websites`. \n",
    "\n",
    "Now that the download is completed, let's compress the pages into a .zip file so you can download it to your computer and take it from there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3 - download faster\n",
    "One useful configuration option is `--concurrency` which is 1 by default but can be increased so that websites can be fetched faster, particularly if these have hundreds or thousands of pages. We will set it to `16` but note that you should test what works for your notebook environment, also pay attention to potential rate limits from Wayback. \n",
    "\n",
    "We noted that changing concurrency from `1` (default) to `16` improved the tool's performance from `139s` to `94s`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ./bin/wayback_machine_downloader --concurrency 16 $WEBSITE_TO_DOWNLOAD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the `--help` command for other useful configurations like restricting the timestamps between which to fetch content, or exclusion/inclusion criteria for pages.\n",
    "\n",
    "If you are only interested in knowing which pages exist, perhaps you're looking for specifically named pages only, you can pass the `--list` option and then inspect the results. You can then go directly to a specific page on your browser or, if you detect a useful pattern, use the `--only` flag to filter those only."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "open-source-research-notebooks-4sg58OrJ",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}