examples/basic_read_write.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `seqp`: how to save text as token IDs and load them back"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook illustrates the use of [`seqp`](https://github.com/noe/seqp) to write text to binary form as token IDs, and to later read the data back."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's download some text data to encode. Universal Declaration of Human Rights (UDHR) will be our choice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "!wget -q http://research.ics.aalto.fi/cog/data/udhr/txt/eng.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the first lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Universal Declaration of Human Rights\n",
      "Preamble Whereas recognition of the inherent dignity and of the equal and inalienable rights of all members of the human family is the foundation of freedom, justice and peace in the world, Whereas disregard and contempt for human rights have resulted in barbarous acts which have outraged the conscience of mankind, and the advent of a world in which human beings shall enjoy freedom of speech and belief and freedom from fear and want has been proclaimed as the highest aspiration of the common people, Whereas it is essential, if man is not to be compelled to have recourse, as a last resort, to rebellion against tyranny and oppression, that human rights should be protected by the rule of law, Whereas it is essential to promote the development of friendly relations between nations, Whereas the peoples of the United Nations have in the Charter reaffirmed their faith in fundamental human rights, in the dignity and worth of the human person and in the equal rights of men and women and have determined to promote social progress and better standards of life in larger freedom, Whereas Member States have pledged themselves to achieve, in cooperation with the United Nations, the promotion of universal respect for and observance of human rights and fundamental freedoms, Whereas a common understanding of these rights and freedoms is of the greatest importance for the full realization of this pledge, Now, therefore, The General Assembly, Proclaims this Universal Declaration of Human Rights as a common standard of achievement for all peoples and all nations, to the end that every individual and every organ of society, keeping this Declaration constantly in mind, shall strive by\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "file_name = 'eng.txt'\n",
    "max_lines_to_print = 2\n",
    "\n",
    "with open(file_name) as f:\n",
    "    for line, idx in zip(f, range(max_lines_to_print)):\n",
    "        print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary extraction\n",
    "\n",
    "Now, we will first extract a vocabulary that allows us to have a mapping from word to integer number.\n",
    "\n",
    "Normally, this would involve a proper tokenization step, but since that is not the focus of this notebook, we will simply split on white space and punctuation by means of a regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from seqp.vocab import Vocabulary, VocabularyCollector\n",
    "\n",
    "collector = VocabularyCollector()\n",
    "\n",
    "with open(file_name) as f:\n",
    "    for line in f:\n",
    "        line = line.strip().lower()\n",
    "        # tokenize words (taken from https://stackoverflow.com/a/8930959/674487)\n",
    "        tokens = re.findall(r\"\\w+|[^\\w\\s]\", line, re.UNICODE)\n",
    "        for token in tokens:\n",
    "            collector.add_symbol(token)\n",
    "\n",
    "vocab = collector.consolidate(max_num_symbols=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write text as HDF5 records\n",
    "\n",
    "We will iterate again the lines in the file and, with the vocabulary extracted above, we will turn each line into a list of integer numbers and store them in an HDF5 file.\n",
    "\n",
    "Each record stored in the file has an associated unique key (an integer number). Although you can use whatever number as key, a sensible choice for line-oriented storage is to use the line number, like we do here.\n",
    "\n",
    "Note that we save the vocabulary along with the records, in the **metadata** part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from seqp.hdf5 import Hdf5RecordWriter\n",
    "\n",
    "output_file = 'udhr_eng.hdf5'\n",
    "\n",
    "with Hdf5RecordWriter(output_file) as writer, open(file_name) as f:\n",
    "\n",
    "    # save vocabulary along with the records\n",
    "    writer.add_metadata({'vocab': vocab.to_json()})\n",
    "\n",
    "    for line in f:\n",
    "        line = line.strip().lower()\n",
    "        tokens = re.findall(r\"\\w+|[^\\w\\s]\", line, re.UNICODE)\n",
    "        token_ids = vocab.encode(tokens, add_eos=False, use_unk=True)\n",
    "        writer.write(np.array(token_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read back HDF5 records\n",
    "\n",
    "First, we extract the vocabulary back from the file metadata, and then we iterate through the few first records and print them.\n",
    "\n",
    "`RecordReader` offers an iterator to tuples of sequence index and length. With the sequence index, you can `retrieve` the sequence itself.\n",
    "\n",
    "With the vocabulary, we convert back the token IDs to tokens and print them to show that the text is properly preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "universal declaration of human rights\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "from seqp.hdf5 import Hdf5RecordReader\n",
    "\n",
    "with Hdf5RecordReader(output_file) as reader:\n",
    "\n",
    "    vocab = Vocabulary.from_json(reader.metadata('vocab'))\n",
    "\n",
    "    for idx, length in reader.indexes_and_lengths():\n",
    "        if idx >= max_lines_to_print:\n",
    "            break\n",
    "        token_ids = reader.retrieve(idx).tolist()\n",
    "        tokens = vocab.decode(token_ids)\n",
    "        print(\" \".join(tokens))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}