examples/ID_to_Rsync_Link.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding a volume's Extracted Features data from a HathiTrust volume ID\n",
    "\n",
    "The Extracted Features Dataset is organized using a pairtree structure, which allows us to find an exact volume from its volume id.\n",
    "\n",
    "## Command Line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you install the HTRC Feature Reader, a command line utility is installed, `htid2rsync`, which converts one or more volume ids to paths in the rsync pairtree. For help, run ```bash htid2rsync --help```.\n",
    "\n",
    "Here is a basic example of using `htid2rsync` with two volume ids:\n",
    "\n",
    "```bash\n",
    "$ htid2rsync miun.adx6300.0001.001 hvd.32044010273894\n",
    "miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2\n",
    "hvd/pairtree_root/32/04/40/10/27/38/94/32044010273894/hvd.32044010273894.json.bz2\n",
    "```\n",
    "\n",
    "This should work on all operating systems. If not, leave a bug report at https://github.com/htrc/htrc-feature-reader.\n",
    "\n",
    "On Unix or Linux command lines, you can send these files directly into RSync by specifying `--files-from=-`, which tells Rsync to listen for a file list sent in from the previous command with a pipe (`|`):\n",
    "\n",
    "```bash\n",
    "$ htid2rsync miun.adx6300.0001.001 hvd.32044010273894 | rsync -azv --files-from=- data.sharc.hathitrust.org::features/ my-local-folder/\n",
    "```\n",
    "\n",
    "### Loading volume ids from a file\n",
    "\n",
    "If you have a file of volume ids, one per line, use ```--from-file filename```, or just `-f filename`.\n",
    "\n",
    "```bash\n",
    "$ htid2rsync --from-file volumeids.txt\n",
    "\n",
    "$ htid2rsync --f volumeids.txt\n",
    "```\n",
    "\n",
    "\n",
    "### Piping in volume ids from STDIN\n",
    "\n",
    "If you are supplying volume ids from STDIN rather than a file, use ```bash --from-file -```. For example:\n",
    "\n",
    "```bash\n",
    "$ some_cmd_that_outputs_volume_ids | htid2rsync --from-file -\n",
    "```\n",
    "\n",
    "### Saving to an output file\n",
    "\n",
    "Supply `--outfile` or `-o` to save the output to a text file. e.g.\n",
    "\n",
    "```bash\n",
    "$ htid2rsync --outfile paths.txt miun.adx6300.0001.001\n",
    "\n",
    "$ htid2rsync -o paths.txt miun.adx6300.0001.001\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from htrc_features import utils\n",
    "utils.id_to_rsync('miun.adx6300.0001.001')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Without the HTRC Feature Reader Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filepath to sync Extracted Features files through RSync follows a [pairtree format](https://wiki.ucop.edu/display/Curation/PairTree?preview=/14254128/16973838/PairtreeSpec.pdf).\n",
    "\n",
    "Because the official pairtree library is only compatible with Python 2.X, we recommend just using the following functions for encoding a volume id to a filename-friendly format and converting the safe id to a file path: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def id_encode(id):\n",
    "    return id.replace(\":\", \"+\").replace(\"/\", \"=\").replace(\".\", \",\")\n",
    "\n",
    "def id2path(id):\n",
    "    clean_id = id_encode(id)\n",
    "    path = []\n",
    "    while len(clean_id) > 0:\n",
    "        val, clean_id = clean_id[:2], clean_id[2:]\n",
    "        path.append(val)\n",
    "    return '/'.join(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method brings it all together, generating the pairtree path which can then be downloaded through RSync:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def id_to_rsync(htid):\n",
    "    '''\n",
    "    Take an HTRC id and convert it to an Rsync location for syncing Extracted\n",
    "    Features\n",
    "    '''\n",
    "    libid, volid = htid.split('.', 1)\n",
    "    volid_clean = id_encode(volid)\n",
    "    filename = '.'.join([libid, volid_clean, 'json.bz2'])\n",
    "    path = '/'.join([libid, 'pairtree_root', id2path(volid).replace('\\\\', '/'),\n",
    "                     volid_clean, filename])\n",
    "    return path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_to_rsync('miun.adx6300.0001.001')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Extracted Features for this volume can be downloaded using RSync:\n",
    "\n",
    "```\n",
    "rsync -azv data.sharc.hathitrust.org::pd-features/{{URL}} .\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compiling and downloading a list of volumes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`select.txt` contains a set of ids for 10k HathiTrust Digital Library volumes in the PZ class (_Fiction and juvenile belles lettres_), that were collected from the HTRC from though its Solr Proxy:\n",
    "\n",
    "http://chinkapin.pti.indiana.edu:9994/solr/meta/select/?q=callnumber:PZ*&wt=csv&fl=id&rows=10000\n",
    "\n",
    "Here is what the id's look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mdp.39015030727963', 'uc2.ark:/13960/t5cc0v81s', 'miun.adx6300.0001.001']\n"
     ]
    }
   ],
   "source": [
    "idlist = open(\"select.txt\", \"r+\").readlines()\n",
    "idlist = [id.strip() for id in idlist[1:]]\n",
    "print(idlist[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mdp/pairtree_root/39/01/50/30/72/79/63/39015030727963/mdp.39015030727963.json.bz2',\n",
       " 'uc2/pairtree_root/ar/k+/=1/39/60/=t/5c/c0/v8/1s/ark+=13960=t5cc0v81s/uc2.ark+=13960=t5cc0v81s.json.bz2']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rsynclist = [id_to_rsync(v) for v in idlist]\n",
    "rsynclist[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also write the full list of our desired volume urls to a file and tell rsync to download from that list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write to file\n",
    "rsyncf = open('rsync-urls.txt', 'w+')\n",
    "rsyncf.write(\"\\n\".join(rsynclist))\n",
    "rsyncf.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Syncing from a file of URLs can be done as follows:\n",
    "\n",
    "```\n",
    "rsync -azv --files-from=rsync-urls.txt data.sharc.hathitrust.org::pd-features/ files/\n",
    "```\n",
    "\n",
    "If you don't need the full pairtree directory structure, it can be flattened to a single folder. This example uses [GNU Parallel](http://www.gnu.org/software/parallel/), available for Linux or Mac OS, or installed on Cygwin in Windows.\n",
    "\n",
    "```\n",
    "find sample-files/ -type -f | parallel --eta mv {} just-one-folder/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explanation of ID-to-URL encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Institution:\tmiun\n",
      "Id:\t\tadx6300.0001.001\n"
     ]
    }
   ],
   "source": [
    "htid = 'miun.adx6300.0001.001'\n",
    "libid, volid = htid.split('.', 1)\n",
    "print(\"Institution:\\t%s\\nId:\\t\\t%s\" % (libid, volid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HathiTrust ID has a library id and an identifier within that library. In this case, _miun_ notes the origin of the file as Michigan University."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filename:\tmiun.adx6300,0001,001.json.bz2\n"
     ]
    }
   ],
   "source": [
    "volid_clean = id_encode(volid)\n",
    "filename = \".\".join([libid, volid_clean, 'json.bz2'])\n",
    "print(\"Filename:\\t%s\" % filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some IDs don't play nice with filesystems and need to be encoded in a cleaner format: \":\" becomes \"+\", \"/\" becomes \"=\", and \".\" becomes \",\" (as above). The institution identifier and the encoded version of the institution's local id become the filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pairtree Root:\t['miun', 'pairtree_root']\n",
      "Full Path:\tmiun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2\n"
     ]
    }
   ],
   "source": [
    "pairtree_root = [libid, 'pairtree_root']\n",
    "path = '/'.join(pairtree_root + [id2path(volid).replace('\\\\', '/'),\n",
    "                                 volid_clean, filename])\n",
    "print(\"Pairtree Root:\\t%s\" % pairtree_root)\n",
    "print(\"Full Path:\\t%s\" % path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The institutional id is split up and encoded, then recombined into a path."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}