notebooks/Demo-reading-pagexml-files-from-archive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d6715cf8",
   "metadata": {},
   "source": [
    "# Reading and analysing archives with many PageXML files\n",
    "\n",
    "This tutorial demonstrates how to read larger sets of [PageXML](https://www.primaresearch.org/tools/PAGELibraries) files that are combined in zip or tar archives. \n",
    "\n",
    "As an example of a zipped archive, this tutorial uses two datasets provided by the [National Archives of the Netherlands](https://www.nationaalarchief.nl/en) (NA) via their HTR repository on [Zenodo](https://zenodo.org/): https://zenodo.org/record/6414086#.Y8Elk-zMIUo. The repository contains many other HTR PageXML datasets that NA made available.\n",
    "\n",
    "The datasets contains HTR output in [PageXML](https://www.primaresearch.org/tools/PAGELibraries) format of scans from the following archive: \n",
    "- (small) _Notaris mr. D.A.M.de Fremery te Assen, 1899-1915, 114.11, 1_ ([EAD](https://www.drentsarchief.nl/onderzoeken/archiefstukken?mivast=34&mizig=210&miadt=34&micode=0114.11&miview=inv2)). This is an archive maintained by the [Drents Archief](https://www.drentsarchief.nl).\n",
    "- (medium) _Verspreide West-Indische stukken, 1614-1875, 1.05.06, 1-1413_ ([EAD](https://www.nationaalarchief.nl/onderzoeken/archief/1.05.06/invnr/%401?query=1.05.06&search-type=inventory)). This is an archive maintained by the [Nationaal Archief](https://www.nationaalarchief.nl/en). \n",
    "\n",
    "You can download the datasets via the following URLs:\n",
    "- https://zenodo.org/record/6414086/files/HTR%20results%20DA%200114.11%20PAGE.zip?download=1\n",
    "- https://zenodo.org/record/6414086/files/HTR%20results%201.05.06%20PAGE.zip?download=1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28955c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84d5e74d",
   "metadata": {},
   "source": [
    "## Extracting PageXML files from Zip/Tar files\n",
    "\n",
    "The example zip files contains of smaller zip files, that each have a number of PageXML files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ba14c6ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object read_page_archive_files at 0x123bed460>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pagexml.helper.file_helper import read_page_archive_files\n",
    "\n",
    "da_archive_file = '../data/HTR results DA 0114.11 PAGE.zip'\n",
    "na_archive_file = '../data/HTR results 1.05.06 PAGE.zip'\n",
    "\n",
    "read_page_archive_files(da_archive_file)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f3e2597b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source_file': ['../data/HTR results DA 0114.11 PAGE.zip', '1.zip'],\n",
       " 'archived_filename': 'NL-AsnDA_0114.11_1_0001.xml',\n",
       " 'archived_filepath': 'NL-AsnDA_0114.11_1_0001.xml'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_fileinfo, page_data = next(read_page_archive_files(da_archive_file))\n",
    "\n",
    "# file information of the first PageXML file in the zip file\n",
    "page_fileinfo\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "825fe0d4",
   "metadata": {},
   "source": [
    "The fileinfo contains three fields:\n",
    "\n",
    "1. `source_file`: The name of the zip/tar archive file that the PageXML file was extracted from. If the archive file consists of zip/tar archvie files, this property will be a list of the hierarchy of zip/tar files that the PageXML file was extracted from.\n",
    "2. `archived_filename`: the name of the PageXML file\n",
    "3. `archived_filepath`: the full path of the PageXML in the zip/tar archive.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048646cd",
   "metadata": {},
   "source": [
    "The file data contains the PageXML content in byte string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "434993db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<PcGts xmlns=\"http://schema.primaresearch.org/PAGE/gts/pagecontent/2013-07-15\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://schema.primaresearch.org/PAGE/gts/pagecontent/2013-07-15 http://schema.primaresearch.org/PAGE/gts/pagecontent/2013-07-15/pagecontent.xsd\">\\n  <Metadata>\\n    <Creator>prov=READ-COOP:name=PyLaia@Transkribus:model_id=21087:version=0.0.1\\nprov=University of Rostock/Institute of Mathematics/CITlab|PLANET AI GmbH/Tobias Gruening/tobias.gruening@planet-ai.de:name=de.uros.citlab.module.baseline2polygon.B2PSeamMultiOriented:v=2.6.6\\nprov=University of Rostock/Institute of Mathematics/CITlab|PLANET AI GmbH/Tobias Gruening/tobias.gruening@planet-ai.de:name=/net_tf/LA73_249_0mod360.pb:de.uros.citlab.segmentation.CITlab_LA_ML:v=2.6.6\\nTranskribus\\n</Creator>\\n    <Created>2020-09-16T15:29:40.548+02:00</Created>\\n    <LastChange>2020-09-16T15:29:51.557+02:00</LastChange>\\n  </Metadata>\\n  <Page imageFilename=\"NL-AsnDA_0114.11_1_0001.jpg\" imageWidth=\"2'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the content of the first PageXML file in the zip file\n",
    "page_data[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "48510b8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PageXMLScan(\n",
       "\tid=NL-AsnDA_0114.11_1_0001.jpg, \n",
       "\ttype=['pagexml_doc', 'text_region', 'scan'], \n",
       "\tstats={\"lines\": 49, \"words\": 306, \"text_regions\": 3, \"columns\": 0, \"extra\": 0, \"pages\": 0}\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pagexml.parser import parse_pagexml_file\n",
    "\n",
    "scan = parse_pagexml_file(pagexml_file=page_fileinfo['archived_filename'], pagexml_data=page_data)\n",
    "scan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4318eb9",
   "metadata": {},
   "source": [
    "The West-Indische Stukken archive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "53168e39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source_file': ['../data/HTR results 1.05.06 PAGE.zip', '1.zip'],\n",
       " 'archived_filename': 'NL-HaNA_1.05.06_1_0001.xml',\n",
       " 'archived_filepath': 'NL-HaNA_1.05.06_1_0001.xml'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_fileinfo, page_data = next(read_page_archive_files(na_archive_file))\n",
    "page_fileinfo\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fa2a3f06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PageXMLScan(\n",
       "\tid=NL-HaNA_1.05.06_1_0001.jpg, \n",
       "\ttype=['pagexml_doc', 'text_region', 'scan'], \n",
       "\tstats={\"lines\": 33, \"words\": 165, \"text_regions\": 1, \"columns\": 0, \"extra\": 0, \"pages\": 0}\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan = parse_pagexml_file(pagexml_file=page_fileinfo['archived_filename'], pagexml_data=page_data)\n",
    "\n",
    "scan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1a25b5c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'lines': 33,\n",
       " 'words': 165,\n",
       " 'text_regions': 1,\n",
       " 'columns': 0,\n",
       " 'extra': 0,\n",
       " 'pages': 0}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan.stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd788fb9",
   "metadata": {},
   "source": [
    "The static `.stats` function returns a dictionary with a count of the number of text regions, lines and words (elements that are part of the PageXML specification), as well as the three elements `pages`, `columns` and `extra`. The latter three are sub-classes of PageXMLTextRegion that are not part of the PageXML spec but are part of the physical structure domain. \n",
    "\n",
    "Often scan images represent an opening of a book, consisting of two physical pages (odd and even numbered, left and right, ...). If you know where the page split is, you can split the scan and create two PageXMLPage documents. Pages can have `PageXMLColumn`s as well as extra `PageXMLTextRegion`s such as headers, footers and marginalia. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "aa6c9e08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         Jieem — Nederland\n",
      " kopij om te leggen nevens de minuut.\n",
      "\n",
      "11 Octob. 1614.\n",
      "\n",
      "               Die Staten Generaal der Vereenichden\n",
      "\n",
      "           Nederlanden, Allen den ghenen die dese\n",
      "           jegenwoordige sullen werden gethoont,\n",
      "          doen te weten: Alsoo Gerrit Jacobs\n",
      "           Witssen, Oud Borgermeester der stadt\n",
      "           Amstelredam, Jonas Witssen, Simon\n",
      "           Monisen, reeders van den schepe ge„\n",
      "          naempt het Vosken, daer schipper op is\n",
      "          geweest Jan de With, Hans Hongers, Pau„\n",
      "          lus Pelgrom, Lambrecht van Tweenhuijsen\n",
      "          reeders van de twee seepen genaempt\n",
      "  ƒ88488   den Tijger ende de Fortuijn, daer schip\n",
      " Raem.\n",
      "           pers op zijn geweest Adriaen Bloek\n",
      "          ende Henrick Corstiaenssen, Arnoult\n",
      "          van Lijbergen, Wessel Schenck, Hans\n",
      "          Claessen ende Berent Sweertssen, ree„\n",
      "          ders van het schip genaempt den\n",
      "          Nachtegael, daer schipper op is geweest\n",
      "          Thijs Volkertssen, coopluijden der voors:\n",
      "          stadt Amstelredam, -en de Pieter Clements\n",
      "          Brouwer, Jan Clementssen Kies en de\n",
      "          Cornelis Volckertsen, coopluijden der\n",
      "          stede Hoorn, reeders van den schepe\n",
      "          genaempt de Fortuijn, daer, schipper\n",
      "         op is geweest Cornelis Jacobssen Maij:\n",
      "          Alle als nu te samen vereenight en\n",
      "          eene compargnie, ons reverentelijck\n",
      "                                           vert.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pagexml.helper.pagexml_helper import pretty_print_textregion\n",
    "\n",
    "pretty_print_textregion(scan)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3d9bc0a",
   "metadata": {},
   "source": [
    "## Examing the structure of each archive file\n",
    "\n",
    "In these example datasets, the zip file contains one or more sub-archives of HTR data, each sub-archive contained in a smaller zip file. From the hierarchy of `source_file`s, it is possible to group the PageXML files per sub-archive.\n",
    "\n",
    "Because the PageXML objects can be big and archives can contain tens or hundreds of thousands of PageXML files, extracting them all can therefore take a significant amount of time, it can be handy to first get an overview of the structure of these zip files and see how many scans there are. \n",
    "\n",
    "You can pass `filenames_only=True` to the `read_page_archive_files` function to extract only the filenames. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3f36ef8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "scans = defaultdict(lambda: defaultdict(set))\n",
    "\n",
    "archives = [\n",
    "    {'id': 'NL-AsnDA_0114.11', 'file': '../data/HTR results DA 0114.11 PAGE.zip'},\n",
    "    {'id': 'NL-HaNA_1.05.06', 'file': '../data/HTR results 1.05.06 PAGE.zip'}    \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "25bfe3ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "extracting PageXML files from ../data/HTR results DA 0114.11 PAGE.zip: 630it [00:00, 5559.87it/s]\n",
      "extracting PageXML files from ../data/HTR results 1.05.06 PAGE.zip: 15857it [00:02, 6463.84it/s]\n"
     ]
    }
   ],
   "source": [
    "for archive in archives:\n",
    "    for page_fileinfo, _ in read_page_archive_files(archive['file'], \n",
    "                                                            filenames_only=True,\n",
    "                                                            show_progress=True):\n",
    "        volume = f\"{archive['id']}_{page_fileinfo['source_file'][-1].replace('.zip', '')}\"\n",
    "        scans[archive['id']][volume].add(page_fileinfo['archived_filename'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3c1a0d3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "archive NL-AsnDA_0114.11\n",
      "\tvolume: NL-AsnDA_0114.11_1\tnumber of scans: 630\n",
      "\n",
      "archive NL-HaNA_1.05.06\n",
      "\tvolume: NL-HaNA_1.05.06_1\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_10\tnumber of scans: 53\n",
      "\tvolume: NL-HaNA_1.05.06_100\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1005\tnumber of scans: 18\n",
      "\tvolume: NL-HaNA_1.05.06_1007\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_101\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_102\tnumber of scans: 39\n",
      "\tvolume: NL-HaNA_1.05.06_103\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_1038\tnumber of scans: 45\n",
      "\tvolume: NL-HaNA_1.05.06_1039\tnumber of scans: 55\n",
      "\tvolume: NL-HaNA_1.05.06_104\tnumber of scans: 158\n",
      "\tvolume: NL-HaNA_1.05.06_105\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_1050\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_1051\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1053\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1055\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_106\tnumber of scans: 24\n",
      "\tvolume: NL-HaNA_1.05.06_107\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1076\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_1077\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_1078\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1079\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_108\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1080\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1081\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_1082\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1083\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_1084\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1086\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1087\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1088\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1089\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_108A\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_109\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1092\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1093\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1096\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_11\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_110\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1103\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1104\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_111\tnumber of scans: 23\n",
      "\tvolume: NL-HaNA_1.05.06_1111\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1113\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1118\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_112\tnumber of scans: 46\n",
      "\tvolume: NL-HaNA_1.05.06_113\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_114\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1145\tnumber of scans: 28\n",
      "\tvolume: NL-HaNA_1.05.06_1146\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_1148\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_115\tnumber of scans: 42\n",
      "\tvolume: NL-HaNA_1.05.06_1151\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1153\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_1154\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1155\tnumber of scans: 120\n",
      "\tvolume: NL-HaNA_1.05.06_1157\tnumber of scans: 29\n",
      "\tvolume: NL-HaNA_1.05.06_1159\tnumber of scans: 24\n",
      "\tvolume: NL-HaNA_1.05.06_116\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_1163\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1165A\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_1166\tnumber of scans: 407\n",
      "\tvolume: NL-HaNA_1.05.06_1167\tnumber of scans: 583\n",
      "\tvolume: NL-HaNA_1.05.06_1168\tnumber of scans: 33\n",
      "\tvolume: NL-HaNA_1.05.06_1168A\tnumber of scans: 39\n",
      "\tvolume: NL-HaNA_1.05.06_1169\tnumber of scans: 47\n",
      "\tvolume: NL-HaNA_1.05.06_117\tnumber of scans: 49\n",
      "\tvolume: NL-HaNA_1.05.06_1170\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1171\tnumber of scans: 338\n",
      "\tvolume: NL-HaNA_1.05.06_1172\tnumber of scans: 225\n",
      "\tvolume: NL-HaNA_1.05.06_1173\tnumber of scans: 243\n",
      "\tvolume: NL-HaNA_1.05.06_1174\tnumber of scans: 222\n",
      "\tvolume: NL-HaNA_1.05.06_1177\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1178\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_118\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_1182\tnumber of scans: 448\n",
      "\tvolume: NL-HaNA_1.05.06_1183\tnumber of scans: 266\n",
      "\tvolume: NL-HaNA_1.05.06_1184\tnumber of scans: 201\n",
      "\tvolume: NL-HaNA_1.05.06_1189\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_119\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1190\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1191\tnumber of scans: 246\n",
      "\tvolume: NL-HaNA_1.05.06_1192\tnumber of scans: 33\n",
      "\tvolume: NL-HaNA_1.05.06_1193\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_1195\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1196\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1197\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1197A\tnumber of scans: 35\n",
      "\tvolume: NL-HaNA_1.05.06_1198\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1199\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_12\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_120\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1200\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_1201\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1202\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1203\tnumber of scans: 67\n",
      "\tvolume: NL-HaNA_1.05.06_1205\tnumber of scans: 248\n",
      "\tvolume: NL-HaNA_1.05.06_1206\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1207\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1208\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1209\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_121\tnumber of scans: 25\n",
      "\tvolume: NL-HaNA_1.05.06_1210\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1211\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1212\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1213\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1214\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1215\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1216\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_1217\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1218\tnumber of scans: 148\n",
      "\tvolume: NL-HaNA_1.05.06_1219\tnumber of scans: 25\n",
      "\tvolume: NL-HaNA_1.05.06_122\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_1220\tnumber of scans: 109\n",
      "\tvolume: NL-HaNA_1.05.06_1221\tnumber of scans: 99\n",
      "\tvolume: NL-HaNA_1.05.06_1222\tnumber of scans: 71\n",
      "\tvolume: NL-HaNA_1.05.06_1223\tnumber of scans: 49\n",
      "\tvolume: NL-HaNA_1.05.06_1224\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1225\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1226\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1227\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1228\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1229\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_123\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1230\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1231\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1232\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1233\tnumber of scans: 27\n",
      "\tvolume: NL-HaNA_1.05.06_1234\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_1235\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1236\tnumber of scans: 31\n",
      "\tvolume: NL-HaNA_1.05.06_1237\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_1238\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1239\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_124\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1240\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_1241\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1242\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1243\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_1244\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_1245\tnumber of scans: 18\n",
      "\tvolume: NL-HaNA_1.05.06_1246\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1247\tnumber of scans: 20\n",
      "\tvolume: NL-HaNA_1.05.06_1248\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_125\tnumber of scans: 37\n",
      "\tvolume: NL-HaNA_1.05.06_1254\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_126\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_127\tnumber of scans: 121\n",
      "\tvolume: NL-HaNA_1.05.06_128\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_129\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_13\tnumber of scans: 80\n",
      "\tvolume: NL-HaNA_1.05.06_130\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_132\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_1328\tnumber of scans: 57\n",
      "\tvolume: NL-HaNA_1.05.06_1329\tnumber of scans: 56\n",
      "\tvolume: NL-HaNA_1.05.06_133\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1333\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1335\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_134\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1342\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1343\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1344\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1345\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1346\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_1347\tnumber of scans: 22\n",
      "\tvolume: NL-HaNA_1.05.06_135\tnumber of scans: 45\n",
      "\tvolume: NL-HaNA_1.05.06_1354\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1359\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_136\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1360\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1360A\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1361\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_1364\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_1365\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1366\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1367\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_137\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_1370\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1371\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1377\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1378\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_138\tnumber of scans: 14\n",
      "\tvolume: NL-HaNA_1.05.06_1382\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_1383\tnumber of scans: 29\n",
      "\tvolume: NL-HaNA_1.05.06_1386\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1388\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_139\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1391\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_1393\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1394\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1395\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1396\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1397\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_140\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_1401\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_1402\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1403\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1406\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_1406A\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_1408\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_1409\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_141\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_1410\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_1411\tnumber of scans: 85\n",
      "\tvolume: NL-HaNA_1.05.06_1412\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_1413\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_142\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_143\tnumber of scans: 22\n",
      "\tvolume: NL-HaNA_1.05.06_144\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_146\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_147\tnumber of scans: 75\n",
      "\tvolume: NL-HaNA_1.05.06_148\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_149\tnumber of scans: 81\n",
      "\tvolume: NL-HaNA_1.05.06_150\tnumber of scans: 28\n",
      "\tvolume: NL-HaNA_1.05.06_151\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_152\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_153\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_154\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_155\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_156\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_157\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_158\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_159\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_160\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_161\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_164\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_169\tnumber of scans: 12\n",
      "\tvolume: NL-HaNA_1.05.06_171\tnumber of scans: 12\n",
      "\tvolume: NL-HaNA_1.05.06_172\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_173\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_173A\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_174\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_175\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_176\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_177\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_177A\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_184\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_185\tnumber of scans: 57\n",
      "\tvolume: NL-HaNA_1.05.06_186\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_187\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_2\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_21\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_213\tnumber of scans: 54\n",
      "\tvolume: NL-HaNA_1.05.06_214\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_215\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_216\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_217\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_218\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_219\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_22\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_220\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_226A\tnumber of scans: 191\n",
      "\tvolume: NL-HaNA_1.05.06_23\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_231\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_233\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_236\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_24\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_246\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_246A\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_247\tnumber of scans: 26\n",
      "\tvolume: NL-HaNA_1.05.06_248\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_249\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_25\tnumber of scans: 24\n",
      "\tvolume: NL-HaNA_1.05.06_259\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_26\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_260\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_261\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_262\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_263\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_264\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_265\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_266\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_268\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_269\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_27\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_270\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_271\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_272\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_273\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_275\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_278\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_279\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_287\tnumber of scans: 67\n",
      "\tvolume: NL-HaNA_1.05.06_292\tnumber of scans: 57\n",
      "\tvolume: NL-HaNA_1.05.06_3\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_304\tnumber of scans: 35\n",
      "\tvolume: NL-HaNA_1.05.06_305\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_306\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_307\tnumber of scans: 210\n",
      "\tvolume: NL-HaNA_1.05.06_314\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_315\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_318\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_325\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_326\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_327\tnumber of scans: 34\n",
      "\tvolume: NL-HaNA_1.05.06_328\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_329\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_330\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_333\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_334\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_339\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_343\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_344\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_353\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_35A\tnumber of scans: 45\n",
      "\tvolume: NL-HaNA_1.05.06_36\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_37\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_373\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_376\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_377\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_378\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_380\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_381\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_382\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_384\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_385\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_386\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_387A\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_388\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_389\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_392\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_394\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_395\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_399\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_40\tnumber of scans: 146\n",
      "\tvolume: NL-HaNA_1.05.06_405A\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_409\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_40A\tnumber of scans: 126\n",
      "\tvolume: NL-HaNA_1.05.06_41\tnumber of scans: 164\n",
      "\tvolume: NL-HaNA_1.05.06_414\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_415\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_416\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_42\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_426\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_43\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_44\tnumber of scans: 672\n",
      "\tvolume: NL-HaNA_1.05.06_45\tnumber of scans: 488\n",
      "\tvolume: NL-HaNA_1.05.06_452\tnumber of scans: 87\n",
      "\tvolume: NL-HaNA_1.05.06_453\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_454\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_456\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_457\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_46\tnumber of scans: 18\n",
      "\tvolume: NL-HaNA_1.05.06_462\tnumber of scans: 97\n",
      "\tvolume: NL-HaNA_1.05.06_464\tnumber of scans: 154\n",
      "\tvolume: NL-HaNA_1.05.06_467\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_469\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_471\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_472\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_476\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_478\tnumber of scans: 59\n",
      "\tvolume: NL-HaNA_1.05.06_479\tnumber of scans: 142\n",
      "\tvolume: NL-HaNA_1.05.06_479A\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_47B\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_48\tnumber of scans: 48\n",
      "\tvolume: NL-HaNA_1.05.06_481\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_482\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_483\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_485\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_487\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_488\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_489\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_49\tnumber of scans: 126\n",
      "\tvolume: NL-HaNA_1.05.06_496\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_50\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_501\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_502\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_503\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_51\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_515A\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_52\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_524\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_525\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_53\tnumber of scans: 340\n",
      "\tvolume: NL-HaNA_1.05.06_531\tnumber of scans: 86\n",
      "\tvolume: NL-HaNA_1.05.06_534\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_535\tnumber of scans: 18\n",
      "\tvolume: NL-HaNA_1.05.06_536\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_54\tnumber of scans: 58\n",
      "\tvolume: NL-HaNA_1.05.06_545A\tnumber of scans: 32\n",
      "\tvolume: NL-HaNA_1.05.06_549\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_555\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_557\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_56\tnumber of scans: 88\n",
      "\tvolume: NL-HaNA_1.05.06_562\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_564A\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_567\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_567A\tnumber of scans: 74\n",
      "\tvolume: NL-HaNA_1.05.06_57\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_572\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_573\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_574\tnumber of scans: 33\n",
      "\tvolume: NL-HaNA_1.05.06_577\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_578\tnumber of scans: 26\n",
      "\tvolume: NL-HaNA_1.05.06_579\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_583\tnumber of scans: 36\n",
      "\tvolume: NL-HaNA_1.05.06_585\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_59\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_6\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_60\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_606\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_607\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_608\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_61\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_613\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_617\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_62\tnumber of scans: 89\n",
      "\tvolume: NL-HaNA_1.05.06_63\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_64\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_65\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_66\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_67\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_674\tnumber of scans: 214\n",
      "\tvolume: NL-HaNA_1.05.06_676\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_677\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_679\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_683\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_689\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_69\tnumber of scans: 22\n",
      "\tvolume: NL-HaNA_1.05.06_690\tnumber of scans: 23\n",
      "\tvolume: NL-HaNA_1.05.06_691\tnumber of scans: 22\n",
      "\tvolume: NL-HaNA_1.05.06_7\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_70\tnumber of scans: 14\n",
      "\tvolume: NL-HaNA_1.05.06_703\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_708\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_709\tnumber of scans: 14\n",
      "\tvolume: NL-HaNA_1.05.06_71\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_710\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_711\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_712\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_713\tnumber of scans: 246\n",
      "\tvolume: NL-HaNA_1.05.06_714\tnumber of scans: 27\n",
      "\tvolume: NL-HaNA_1.05.06_72\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_720\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_721\tnumber of scans: 20\n",
      "\tvolume: NL-HaNA_1.05.06_722\tnumber of scans: 53\n",
      "\tvolume: NL-HaNA_1.05.06_723\tnumber of scans: 75\n",
      "\tvolume: NL-HaNA_1.05.06_724\tnumber of scans: 59\n",
      "\tvolume: NL-HaNA_1.05.06_727\tnumber of scans: 26\n",
      "\tvolume: NL-HaNA_1.05.06_728\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_729\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_73\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_730\tnumber of scans: 23\n",
      "\tvolume: NL-HaNA_1.05.06_731\tnumber of scans: 115\n",
      "\tvolume: NL-HaNA_1.05.06_740\tnumber of scans: 22\n",
      "\tvolume: NL-HaNA_1.05.06_748\tnumber of scans: 159\n",
      "\tvolume: NL-HaNA_1.05.06_749A\tnumber of scans: 55\n",
      "\tvolume: NL-HaNA_1.05.06_75\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_750A\tnumber of scans: 43\n",
      "\tvolume: NL-HaNA_1.05.06_757\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_76\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_771\tnumber of scans: 10\n",
      "\tvolume: NL-HaNA_1.05.06_781\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_784\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_8\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_801\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_808\tnumber of scans: 308\n",
      "\tvolume: NL-HaNA_1.05.06_809\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_812\tnumber of scans: 416\n",
      "\tvolume: NL-HaNA_1.05.06_813\tnumber of scans: 83\n",
      "\tvolume: NL-HaNA_1.05.06_814\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_876\tnumber of scans: 79\n",
      "\tvolume: NL-HaNA_1.05.06_878\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_879\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_880\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_881\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_882\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_883\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_884\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_885\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_886\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_887\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_888\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_889\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_890\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_891\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_892\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_893\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_894\tnumber of scans: 71\n",
      "\tvolume: NL-HaNA_1.05.06_9\tnumber of scans: 21\n",
      "\tvolume: NL-HaNA_1.05.06_901\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_902\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_903\tnumber of scans: 15\n",
      "\tvolume: NL-HaNA_1.05.06_904\tnumber of scans: 236\n",
      "\tvolume: NL-HaNA_1.05.06_905\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_907\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_918\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_921\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_922\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_923\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_924\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_925\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_926\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_928\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_929\tnumber of scans: 16\n",
      "\tvolume: NL-HaNA_1.05.06_929A\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_929B\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_930\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_931\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_932\tnumber of scans: 39\n",
      "\tvolume: NL-HaNA_1.05.06_933\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_934\tnumber of scans: 44\n",
      "\tvolume: NL-HaNA_1.05.06_936\tnumber of scans: 18\n",
      "\tvolume: NL-HaNA_1.05.06_937\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_938\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_939\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_940\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_941\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_942\tnumber of scans: 11\n",
      "\tvolume: NL-HaNA_1.05.06_946\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_95\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_951\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_954\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_955\tnumber of scans: 6\n",
      "\tvolume: NL-HaNA_1.05.06_96\tnumber of scans: 2\n",
      "\tvolume: NL-HaNA_1.05.06_963\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_964\tnumber of scans: 4\n",
      "\tvolume: NL-HaNA_1.05.06_965\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_97\tnumber of scans: 9\n",
      "\tvolume: NL-HaNA_1.05.06_970\tnumber of scans: 29\n",
      "\tvolume: NL-HaNA_1.05.06_971\tnumber of scans: 23\n",
      "\tvolume: NL-HaNA_1.05.06_972\tnumber of scans: 41\n",
      "\tvolume: NL-HaNA_1.05.06_973\tnumber of scans: 8\n",
      "\tvolume: NL-HaNA_1.05.06_975\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_978\tnumber of scans: 3\n",
      "\tvolume: NL-HaNA_1.05.06_979\tnumber of scans: 13\n",
      "\tvolume: NL-HaNA_1.05.06_98\tnumber of scans: 17\n",
      "\tvolume: NL-HaNA_1.05.06_980\tnumber of scans: 19\n",
      "\tvolume: NL-HaNA_1.05.06_981\tnumber of scans: 5\n",
      "\tvolume: NL-HaNA_1.05.06_983\tnumber of scans: 312\n",
      "\tvolume: NL-HaNA_1.05.06_983A\tnumber of scans: 102\n",
      "\tvolume: NL-HaNA_1.05.06_984\tnumber of scans: 7\n",
      "\tvolume: NL-HaNA_1.05.06_99\tnumber of scans: 40\n"
     ]
    }
   ],
   "source": [
    "for archive_id in scans:\n",
    "    print(f'\\narchive {archive_id}')\n",
    "    for volume in scans[archive_id]:\n",
    "        print(f'\\tvolume: {volume}\\tnumber of scans: {len(scans[archive_id][volume])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166b11c7",
   "metadata": {},
   "source": [
    "It turns out that the Notarial archive zip file only contains a single zip file with a volume of 630 PageXML files, while the West Indische Stukken has many smaller volumes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "683ba2cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "extracting PageXML files from ../data/HTR results DA 0114.11 PAGE.zip: 630it [00:09, 67.00it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "defaultdict(int,\n",
       "            {'lines': 56700,\n",
       "             'words': 238926,\n",
       "             'text_regions': 2484,\n",
       "             'columns': 0,\n",
       "             'extra': 0,\n",
       "             'pages': 0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "archive_id = 'NL-AsnDA_0114.11'\n",
    "archive_file = '../data/HTR results DA 0114.11 PAGE.zip'\n",
    "archive_stats = defaultdict(int)\n",
    "\n",
    "for page_fileinfo, page_data in read_page_archive_files(archive_file, \n",
    "                                                        show_progress=True):\n",
    "    scan = parse_pagexml_file(pagexml_file=page_fileinfo['archived_filename'], pagexml_data=page_data)\n",
    "    scan_stats = scan.stats\n",
    "    for field in scan_stats:\n",
    "        archive_stats[field] += scan_stats[field]\n",
    "\n",
    "archive_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3085840c",
   "metadata": {},
   "source": [
    "The 630 PageXML files in the dataset of the notarial archive of D.A.M. de Fremery contains close to 240,000 words. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e58572a",
   "metadata": {},
   "source": [
    "## Transforming XML into plain text line format files\n",
    "\n",
    "If you want to work on the text of the PageXML files, it can be useful to make a plain text format that is faster to process and uses less memory. The PageXML tools package contains functionality to extract the text lines from PageXML files into a simple tab-separated value format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c17ebe04",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pagexml.helper.text_helper import make_line_format_file\n",
    "from pagexml.parser import parse_pagexml_files_from_archive\n",
    "\n",
    "\n",
    "for archive in archives:\n",
    "    line_tsv_file = f\"../data/line_format-{archive['id']}.tsv.gz\"\n",
    "    page_docs = parse_pagexml_files_from_archive(archive['file'])\n",
    "    #make_line_format_file(page_extractor, line_tsv_file, add_bounding_box=True)\n",
    "    make_line_format_file(page_docs, line_tsv_file, add_bounding_box=True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e10570e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'doc_id': 'NL-HaNA_1.05.06_1_0001.jpg', 'textregion_id': 'r1', 'line_id': 'r1l1', 'text': 'Jieem — Nederland', 'doc_box': '0,0,2522,3957', 'textregion_box': '120,152,2333,3493', 'line_box': '586,136,1108,113'}\n"
     ]
    }
   ],
   "source": [
    "from pagexml.helper.text_helper import LineReader\n",
    "\n",
    "line_tsv_file = '../data/line_format-NL-HaNA_1.05.06.tsv.gz'\n",
    "\n",
    "reader = LineReader(pagexml_line_files=line_tsv_file)\n",
    "for line in reader:\n",
    "    print(line)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "453ee171",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e8c23fe",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e66cff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c9f7963",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}