polish_roberta_vocab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PoLitBert - Polish RoBERT'a model \n",
    "\n",
    "## Preparation of vocabulary and encoding the data\n",
    "\n",
    "Used corpuses:\n",
    "* Wikipedia, Link: \n",
    "* Oscar\n",
    "* Polish Books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usefull resources\n",
    "* https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md\n",
    "* https://github.com/musixmatchresearch/umberto/issues/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import csv\n",
    "import sys\n",
    "import datetime as dt\n",
    "import os\n",
    "from pathlib import Path\n",
    "import re\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "import mmap"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create vocabulary\n",
    "\n",
    "### Prepare data for vocab\n",
    "\n",
    "Separate text file for training vocabulary has been created with one sentence per line.\n",
    "We used polish sentence tokenizer with [additional abbreviations](https://gist.github.com/ksopyla/f05fe2f48bbc9de895368b8a7863b5c3)\n",
    "typical for the Polish language.\n",
    "Sentencepiece model is capable of handling around 12.000.000 sentences, so larger files are not necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the BPE vocabulary model\n",
    "\n",
    "We used the [SentencePiece](https://github.com/google/sentencepiece) segmentation model trained from raw\n",
    "sentences with fixed final vocabulary size - 32K and 50K unique tokens.\n",
    "\n",
    "Training and segmentation can be done in two ways:\n",
    "- as a python module,\n",
    "- as a command-line tool.\n",
    "\n",
    "To use it as a command-line it should be installed from source, which is described in the\n",
    "[build the C++ version from source](https://github.com/google/sentencepiece#c-from-source) section of the documentation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training SentencePiece vocab using command line\n",
    "\n",
    "* 32k vocab:\n",
    "```\n",
    "spm_train \\\n",
    "    --input=./data/corpus_raw/corpus_books_wiki_12M_lines.txt \\\n",
    "    --max_sentence_length=4192\\\n",
    "    --model_prefix=./data/vocab/books_wikipedia_v32k_sen10M.spm.bpe \\\n",
    "    --vocab_size=32000 \\\n",
    "    --model_type=bpe \\\n",
    "    --shuffle_input_sentence=true \\\n",
    "    --input_sentence_size=10000000 \\\n",
    "    --bos_id=0 --eos_id=1 --pad_id=2 --unk_id=3\n",
    "```\n",
    "\n",
    "* 50k vocab:\n",
    "```\n",
    "spm_train \\\n",
    "    --input=./data/corpus_raw/corpus_books_wiki_12M_lines.txt \\\n",
    "    --max_sentence_length=4192\\\n",
    "    --model_prefix=./data/vocab/books_wikipedia_v50k_sen10M.spm.bpe \\\n",
    "    --vocab_size=50000 \\\n",
    "    --model_type=bpe \\\n",
    "    --shuffle_input_sentence=true \\\n",
    "    --input_sentence_size=10000000 \\\n",
    "    --bos_id=0 --eos_id=1 --pad_id=2 --unk_id=3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Training SentencePiece vocab with Python module\n",
    "\n",
    "Below, for reference, an example of how to prepare a SP model if Python script is preferred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import sentencepiece as spm\n",
    "\n",
    "vocab_size = 32000\n",
    "model_type = \"bpe\"  \n",
    "iss = 10_000_000\n",
    "\n",
    "data_file = './data/corpus_raw/corpus_books_wiki_12M_lines.txt'\n",
    "\n",
    "tok_model = f\"books_wikipedia_v32k_sen10M\"\n",
    "tok_model = os.path.abspath(f\"./data/vocab/{tok_model}\")\n",
    "\n",
    "piece_options = ' --bos_id=0 --eos_id=1 --pad_id=2 --unk_id=3 --shuffle_input_sentence=true'\n",
    "\n",
    "cmd = f\"--input={data_file} --model_prefix={tok_model} --num_threads=4 --vocab_size={vocab_size}  --input_sentence_size={iss}\" + piece_options\n",
    "print(cmd)\n",
    "\n",
    "start = dt.datetime.now()\n",
    "print(start)\n",
    "spm.SentencePieceTrainer.train(cmd)\n",
    "end = dt.datetime.now()\n",
    "\n",
    "print(f\"Created vocab of {vocab_size} tokens from {data_file}, took {end-start}.\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "32000\n['▁Będąc', '▁młodym', '▁programi', 'stą', '▁(', 'ho', 'ho', '),', '▁czy', 't', 'ałem', '▁\"', 'D', 'zia', 'dy', '\"', '▁w', '▁1983', 'r', '.']\n"
    }
   ],
   "source": [
    "# Example segmentation usage:\n",
    "\n",
    "# make segmenter instance and load the model file (m.model)\n",
    "sp = spm.SentencePieceProcessor()\n",
    "sp.load(f\"{tok_model}.model\")\n",
    "\n",
    "# verify vocab size\n",
    "print(sp.get_piece_size())\n",
    "\n",
    "# encode: text => id\n",
    "text = \"\"\"Będąc młodym programistą (hoho), czytałem \"Dziady\" w 1983r.\"\"\"\n",
    "print(sp.encode_as_pieces(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairseq vocab\n",
    "\n",
    "Usage of sentencepiece the model's with fairseq requires changing the separator used in the dictionary.\n",
    "All _\\t_ characters should be replaced with _whitespace_ in the vocab file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "for vocab_size in (\"32k\", \"50k\"):\n",
    "    vocab_file = f\"./data/vocab/books_wikipedia_v{vocab_size}_sen10M.spm.bpe.vocab\"\n",
    "\n",
    "    p = Path(vocab_file)\n",
    "\n",
    "    output_path = f\"{p.with_suffix('')}_fair.vocab\"\n",
    "    with open(output_path, 'w+') as output_file:\n",
    "        with open(vocab_file) as f:\n",
    "\n",
    "            text = f.read().replace('\\t', ' ')\n",
    "            output_file.write(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encode data with sentence piece model\n",
    "\n",
    "Encoding prepared training and test datasets with SentencePiece tokenizer. Both, for 32k and 50k vocabularies.\n",
    "\n",
    "* 32k vocab:\n",
    "\n",
    "```\n",
    "DATA_PATH=./data/wiki_books_oscar/\n",
    "VOCAB_SIZE=32k\n",
    "\n",
    "for SPLIT in test train ; do \\\n",
    "    spm_encode \\\n",
    "    --model=./data/vocab/books_wikipedia_v${VOCAB_SIZE}_sen10M.spm.bpe.model \\\n",
    "    --extra_options=bos:eos \\\n",
    "    --output_format=piece \\\n",
    "    < ${DATA_PATH}corpus_wiki_books_oscar_${SPLIT}.txt \\\n",
    "    > ${DATA_PATH}corpus_wiki_books_oscar_${SPLIT}_${VOCAB_SIZE}.txt.bpe\n",
    "done\n",
    "```\n",
    "\n",
    "* 50k vocab:\n",
    "\n",
    "```\n",
    "DATA_PATH=./data/wiki_books_oscar/\n",
    "VOCAB_SIZE=50k\n",
    "\n",
    "for SPLIT in test train ; do \\\n",
    "    spm_encode \\\n",
    "    --model=./data/vocab/books_wikipedia_v${VOCAB_SIZE}_sen10M.spm.bpe.model \\\n",
    "    --extra_options=bos:eos \\\n",
    "    --output_format=piece \\\n",
    "    < ${DATA_PATH}corpus_wiki_books_oscar_${SPLIT}.txt \\\n",
    "    > ${DATA_PATH}corpus_wiki_books_oscar_${SPLIT}_${VOCAB_SIZE}.txt.bpe\n",
    "done\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Data binarization with Fairseq\n",
    "\n",
    "### Fairseq-preprocessing bpe encoded and splited data\n",
    "\n",
    "* Data processed with 32k vocab:\n",
    "\n",
    "```\n",
    "DATA_PATH=./data/wiki_books_oscar/\n",
    "VOCAB_SIZE=32k\n",
    "\n",
    "fairseq-preprocess \\\n",
    "    --only-source \\\n",
    "    --srcdict ./vocab/books_wikipedia_v${VOCAB_SIZE}_sen10M.spm.bpe_fair.vocab \\\n",
    "    --trainpref ${DATA_PATH}corpus_wiki_books_oscar_train_vocab${VOCAB_SIZE}.txt.bpe \\\n",
    "    --validpref ${DATA_PATH}corpus_wiki_books_oscar_test_vocab${VOCAB_SIZE}.txt.bpe \\\n",
    "    --destdir ${DATA_PATH}vocab${VOCAB_SIZE} \\\n",
    "    --workers 8\n",
    "```\n",
    "\n",
    "* Data processed with 50k vocab:\n",
    "\n",
    "```\n",
    "DATA_PATH=./data/wiki_books_oscar/\n",
    "VOCAB_SIZE=50k\n",
    "\n",
    "fairseq-preprocess \\\n",
    "    --only-source \\\n",
    "    --srcdict ./vocab/books_wikipedia_v${VOCAB_SIZE}_sen10M.spm.bpe_fair.vocab \\\n",
    "    --trainpref ${DATA_PATH}corpus_wiki_books_oscar_train_vocab${VOCAB_SIZE}.txt.bpe \\\n",
    "    --validpref ${DATA_PATH}corpus_wiki_books_oscar_test_vocab${VOCAB_SIZE}.txt.bpe \\\n",
    "    --destdir ${DATA_PATH}vocab${VOCAB_SIZE} \\\n",
    "    --workers 8\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "version": "3.7.3-final"
  },
  "orig_nbformat": 2,
  "file_extension": ".py",
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3,
  "kernelspec": {
   "name": "python37364bitherbertpipenvf409fddaf3f446fd8dcf7490c441f6bd",
   "display_name": "Python 3.7.3 64-bit ('herbert': pipenv)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}