labs/lab2/lab2_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "clsA66QtojHf"
   },
   "source": [
    "# Practice session 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8MgmSagHojHk"
   },
   "source": [
    "## Preprocessing text data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EEUYFKS6ojHl"
   },
   "source": [
    "**This notebook is recommended to run in [Google Colab](https://colab.research.google.com). Towards the end of the class we will use the GPUs provided there to make our model train faster. Check that your runtime type is GPU before you start training.**\n",
    "\n",
    "**If you have a local GPU and a working Sockeye installation, feel free run it locally. (If you have a working Sockeye installation but no GPU, you can also run everything locally, but the last part - training the model - will be very slow.)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jhdcgs7TojHm"
   },
   "source": [
    "We have learned how to setup our environment and use Sockeye to train a sequence-to-sequence model. However, it is not all there is to creating an actual machine translation system. Unlike sequences of integers, natural language texts are complicated and messy. We need to preprocess them before we can use them for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yIGnh5iqojHn"
   },
   "source": [
    "### 1. Get the data\n",
    "\n",
    "Download the Estonian-English parallel corpus and move it into `/data`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 286
    },
    "colab_type": "code",
    "id": "tU_Mgjt3ojHt",
    "outputId": "326c5b28-7322-4b3d-8622-c49d256c9c80"
   },
   "outputs": [],
   "source": [
    "!wget http://opus.nlpl.eu/download.php?f=Europarl/v8/moses/en-et.txt.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ex5mGrWlojHx"
   },
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "!mv download.php\\?f\\=Europarl%2Fv8%2Fmoses%2Fen-et.txt.zip data/en-et.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "giLd-OmyojH0"
   },
   "source": [
    "Uncompress the data, delete unnecessary files, move the files we need into `data/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 118
    },
    "colab_type": "code",
    "id": "t5ayoFMCojH3",
    "outputId": "c09b219b-5d3e-4466-8252-5ba5323984cf"
   },
   "outputs": [],
   "source": [
    "!unzip data/en-et.zip\n",
    "!rm LICENSE\n",
    "!rm README\n",
    "!rm Europarl.en-et.xml\n",
    "!mv Europarl* data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QOwB-xMVojH7"
   },
   "source": [
    "Give our files shorter names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CSD2BS9nojH8"
   },
   "outputs": [],
   "source": [
    "!mv data/Europarl.en-et.en data/europarl.en\n",
    "!mv data/Europarl.en-et.et data/europarl.et"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U3CUkbJdojH_"
   },
   "source": [
    "### 2. Look at the data\n",
    "\n",
    "You can see that the parallel corpus is named Europarl. It is one of several corpora that are commonly used.\n",
    "\n",
    "Let's check how many lines our files contain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "n8Yx2tO3ojIB",
    "outputId": "2ab4a143-93c1-4ee9-dce3-701340e80f30"
   },
   "outputs": [],
   "source": [
    "!wc -l data/europarl*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b_BHyJVyojIE"
   },
   "source": [
    "As you may have guessed, \"Europarl\" comes from \"European Parliament\". This corpus contains translations of parliament proceedings. It is a convenient resource, as all parliament proceedings have to be traslated into all official languages of the European Union. Which is great for us!\n",
    "\n",
    "Let's see what some random sentence pairs from this corpus look like. First, let's shuffle and merge the source and target files horizontally (each line of the resulting file will contain a source line and a target line, separated by a tab):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tUs7R44AojIF"
   },
   "outputs": [],
   "source": [
    "!paste data/europarl.et data/europarl.en | shuf > data/shuf-europarl.both"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "94vxDhKIojII"
   },
   "source": [
    "Now let's print out several sentence pairs in a readable format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "id": "5dAhkPleojIJ",
    "outputId": "54ba73b0-0206-437f-85b4-73a121ec3481"
   },
   "outputs": [],
   "source": [
    "with open('data/shuf-europarl.both', 'r', encoding='utf8') as fh:\n",
    "    for i in range(5):\n",
    "        et_sentence, en_sentence = fh.readline().strip().split('\\t')\n",
    "        print('ET: {}\\nEN: {}\\n'.format(et_sentence, en_sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jKHOHx7EojIN"
   },
   "source": [
    "**Question.** What kind of language do these sentences exhibit? Do the translations look good to you? If you do not speak Estonian, how did you judge that?\n",
    "\n",
    "In this practice session, we will only use a small subset of the Europarl corpus for training. However, note that for training a real MT system, even the whole Europarl corpus would not be considered a lot of data.\n",
    "\n",
    "Let's separate the data again, and keep 20000 lines for the training set, 1000 lines for the development set, and 500 lines for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3KyUwMw1ojIO"
   },
   "outputs": [],
   "source": [
    "!sed -n 1,20000p data/shuf-europarl.both | cut -f 1 > data/train.et\n",
    "!sed -n 1,20000p data/shuf-europarl.both | cut -f 2 > data/train.en\n",
    "!sed -n 20001,21000p data/shuf-europarl.both | cut -f 1 > data/dev.et\n",
    "!sed -n 20001,21000p data/shuf-europarl.both | cut -f 2 > data/dev.en\n",
    "!sed -n 21001,21500p data/shuf-europarl.both | cut -f 1 > data/test.et\n",
    "!sed -n 21001,21500p data/shuf-europarl.both | cut -f 2 > data/test.en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sdIOzwpiojIR"
   },
   "source": [
    "### 3. Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SBCfmmlLojIS"
   },
   "source": [
    "Now that we have looked at the data, we can start preprocessing it.\n",
    "\n",
    "Real data is always messy. Some sentence pairs may be misaligned, some tranlations may be missing. It is always a good idea to take a look at your data and make sure that it is mostly OK.\n",
    "\n",
    "We can get rid of sentence pairs that are almost defnitely bad. These pairs are those where at least one side (source or target) is empty, at least one side is longer than 100 words, or one side contains at least 9 times more words that the other. You are provided with a Python script which does that, let's apply it to our files.\n",
    "\n",
    "If you are using Colab, don't forget to upload the script `cleaning.py` first. The script is included in the practice session materials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "77gT9YC-ojIU",
    "outputId": "551469bc-911a-454c-9e83-cbf885e036bd"
   },
   "outputs": [],
   "source": [
    "!python cleaning.py --corpora data/train --output data/cleaned-train --src_lang et --tgt_lang en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ereXmmrNojIX"
   },
   "source": [
    "Let's check how many sentence pairs are left:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "PPaA29lbojIZ",
    "outputId": "96c62873-7985-4168-bcb4-68e523aaf88a"
   },
   "outputs": [],
   "source": [
    "!wc -l data/cleaned-train*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rDGuEyiMojId"
   },
   "source": [
    "### 4. Truecasing\n",
    "\n",
    "Now we need to truecase the text.\n",
    "\n",
    "**Question.** Truecasing has been mentioned in the lectures. What is it? Why do we need it?\n",
    "\n",
    "We will use our own [TartuNLP truecaser](https://github.com/TartuNLP/truecaser). (If you happen to find any bugs, feel free to open an issue or a pull request.)\n",
    "\n",
    "First, get the truecaser code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 118
    },
    "colab_type": "code",
    "id": "Iv3lB_tEojIe",
    "outputId": "f98365f2-99f8-4ba9-93f5-66aba7e47b36"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/TartuNLP/truecaser.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P86PsCyaojIj"
   },
   "source": [
    "Let's create a directory where we'll keep all the preprocessing models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mpJElw0jojIk"
   },
   "outputs": [],
   "source": [
    "!mkdir preproc-models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pilL2LucojIm"
   },
   "source": [
    "Now let's train two truecasing models, one for Estonian and one for English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 151
    },
    "colab_type": "code",
    "id": "yYLRbiRAojIn",
    "outputId": "09faa02c-11c6-4f8c-c269-6b66fc883486"
   },
   "outputs": [],
   "source": [
    "!python truecaser/learntc.py data/cleaned-train.et preproc-models/tc-et\n",
    "!python truecaser/learntc.py data/cleaned-train.en preproc-models/tc-en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eS6dR5UxojIt"
   },
   "source": [
    "Now we can apply the models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "zATC8NDOojIu",
    "outputId": "8ee2374f-fc7c-4cbe-ce02-2bc59ff9daf4"
   },
   "outputs": [],
   "source": [
    "!python truecaser/applytc.py preproc-models/tc-et data/cleaned-train.et > data/tc-cleaned-train.et\n",
    "!python truecaser/applytc.py preproc-models/tc-en data/cleaned-train.en > data/tc-cleaned-train.en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eNMOyDwvojIx"
   },
   "source": [
    "Compare the first sentences of the English training file before and after truecasing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 101
    },
    "colab_type": "code",
    "id": "Vz7Ps3mLojIy",
    "outputId": "e419bbb7-602c-42eb-f528-0c4b57c9a30e"
   },
   "outputs": [],
   "source": [
    "!head -5 data/cleaned-train.en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 101
    },
    "colab_type": "code",
    "id": "ik0QTosdojI2",
    "outputId": "b3bdd21c-ffd4-4966-ba02-b11b423dbc48"
   },
   "outputs": [],
   "source": [
    "!head -5 data/tc-cleaned-train.en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DoonxUBJojI4"
   },
   "source": [
    "**Question.** Did our model make any unexpected changes in the first examples? If it has, can you guess why this happened?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xEZsJQvPojI5"
   },
   "source": [
    "### 5. Subword segmentation\n",
    "\n",
    "The last preprocessing step is subword segmentation. Words will be split into smaller parts based on character co-occurrence frequency. The most common words will remain in one piece, and rare words will be broken into several units.\n",
    "\n",
    "**Question.** Why do we need subword segmentation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PYKCx5kFojI6"
   },
   "source": [
    "**Note.** In a typical natural language processing pipeline, the first pre-processing step would be **tokenization**. Its main task is to turn a string into a list of tokens, in other words, to separate words from punctuation marks (e.g. `Hi, Mary!` $\\rightarrow$ `[\"Hi\", \",\", \"Mary\", \"!\"]`). We are not doing tokenization in this tutorial because SentencePiece can handle untokenized text. One benefit of using SentencePiece for both tokenization and subword segmentation is that it is fully reversible. Our current pipeline may not be typical for other NLP tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YBqmgFjPojI7"
   },
   "source": [
    "First, install SentencePiece, one of the popular options for subword segmentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "colab_type": "code",
    "id": "WRAgVFUiojI7",
    "outputId": "590dd941-1ee9-4f42-f048-ffeae4f91715"
   },
   "outputs": [],
   "source": [
    "!pip install sentencepiece"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "06y-bEOlojJA"
   },
   "source": [
    "Now we can learn a model for splitting our text into subwords. Note that it is common to have a joint vocabulary for source and target languages.\n",
    "\n",
    "SentencePiece is not very straightforward to use, so we have provided you with a script that does everything you need.\n",
    "\n",
    "If you are using Colab, don't forget to upload the script `word-pieces.py` first. The script is included in the practice session materials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "id": "PorJ_PrXojJA",
    "outputId": "7794b888-ee1d-4a6d-ee73-2ec31aac1f4f"
   },
   "outputs": [],
   "source": [
    "!python word-pieces.py -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B2evHXbPojJD"
   },
   "source": [
    "Let's train a model with 4000 subwords. This means that SentencePiece will split the words until the size of vocabulary reduces to 4000. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "WHUFuktBojJE",
    "outputId": "6d86c0e5-d605-4364-919b-2db7d95a713a"
   },
   "outputs": [],
   "source": [
    "!python word-pieces.py --action train --size 4000 --corpora data/tc-cleaned-train.* --model preproc-models/sp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "osfnmb86ojJH"
   },
   "source": [
    "Now we can apply our model to the training files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "ANEGQir1ojJI",
    "outputId": "ab62df2f-0492-4ecc-9a9c-a41badf74031"
   },
   "outputs": [],
   "source": [
    "!python word-pieces.py --action split --corpora data/tc-cleaned-train.* --model preproc-models/sp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MO-dtHQeojJL"
   },
   "source": [
    "### 5. Repeat for dev sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nOe-DYo7ojJL"
   },
   "source": [
    "Now, repeat the preprocessing steps for the development sets. Note that you do not need to learn new models, but only to apply the ones that were learned using the training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "Y-gEjzpCojJM",
    "outputId": "5cc9fc15-6889-41f1-9450-99f04024913f"
   },
   "outputs": [],
   "source": [
    "### YOUR CODE ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HZpFW07CojJX"
   },
   "source": [
    "### 6. Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZDqVEEke6CpO"
   },
   "source": [
    "If you are working in Colab, install Sockeye. Check that your runtime type is GPU before you install the GPU version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 692
    },
    "colab_type": "code",
    "id": "PX2A8DqBqODX",
    "outputId": "126bf310-144d-456e-afc1-6841e8434d3d"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/awslabs/sockeye/master/requirements/requirements.gpu-cu100.txt\n",
    "!pip install sockeye --no-deps -r requirements.gpu-cu100.txt\n",
    "!rm requirements.gpu-cu100.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Djl0sGAojJZ"
   },
   "source": [
    "There is one final preprocessing step left. It is not necessary, but with large datasets it makes life easier. If you do data preparation beforehand, you will not have to spend time preparing data for Sockeye when you start training. For big models this means that if training fails, you will know about it right away and not in a few hours. The following command will serialize data in matrix format and split it into shards, if necessary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 474
    },
    "colab_type": "code",
    "id": "FrL7GmGIojJa",
    "outputId": "07317990-561f-4eac-9656-8562591f5a32"
   },
   "outputs": [],
   "source": [
    "!python -m sockeye.prepare_data --source data/sp-tc-cleaned-train.et \\\n",
    "                                --target data/sp-tc-cleaned-train.en \\\n",
    "                                --shared-vocab \\\n",
    "                                --output prepared_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MLcEFbDQojJd"
   },
   "source": [
    "### 7. Train a translation model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Vukv9BgkojJe"
   },
   "source": [
    "Now that we have preprocessed some texts, we are finally ready to train a translation model. It will not be very good, because we are only using 20,000 sentence pairs for training and we do not have a lot of time, but nevertheless it should learn something useful.\n",
    "\n",
    "The following command will train a 1-layer bi-LSTM encoder, 1-layer LSTM decoder with dot attention for 30 epochs.\n",
    "\n",
    "If you are using Colab, before you start training, check that you have selected runtime type 'GPU'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "dFazEdTtojJf",
    "outputId": "30edbd55-3cfd-4c08-8802-7535cbf20321"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "start = time.time()\n",
    "!python -m sockeye.train --prepared-data prepared_data \\\n",
    "                         --validation-source data/sp-tc-cleaned-dev.et \\\n",
    "                         --validation-target data/sp-tc-cleaned-dev.en \\\n",
    "                         --shared-vocab \\\n",
    "                         --output et2en_model \\\n",
    "                         --encoder rnn \\\n",
    "                         --decoder rnn \\\n",
    "                         --num-embed 256 \\\n",
    "                         --rnn-num-hidden 512 \\\n",
    "                         --num-layers 1:1 \\\n",
    "                         --rnn-attention-type dot \\\n",
    "                         --max-seq-len 60 \\\n",
    "                         --decode-and-evaluate 500 \\\n",
    "                         --batch-type word \\\n",
    "                         --batch-size 8000 \\\n",
    "                         --max-num-epochs 30 \\\n",
    "                         --checkpoint-interval 200 \\\n",
    "                         --initial-learning-rate 0.002\n",
    "\n",
    "print(time.time() - start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ozt_aoprHaTX"
   },
   "source": [
    "You probably got best validation BLEU below 0.05, which indicates that our model did not learn to translate well yet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wcdiy8Ok9pU1"
   },
   "source": [
    "### 8. Translate something"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gcPWKth09v0A"
   },
   "source": [
    "Before we can translate a sentence, we need to preprocess it in the same way as we did the training and development sets.\n",
    "\n",
    "First, create a file containing the lines \"Tere!\" and \"See on lause.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O6M2h9rd-W8r"
   },
   "outputs": [],
   "source": [
    "with open('data/input.et', 'w', encoding='utf8') as fh:\n",
    "    fh.write('\\n'.join([\"Tere!\", \"See on lause.\"]) + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2MIQmJ8p_Oz5"
   },
   "source": [
    "Then preprocess it: truecase and split into subwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "pTQ7x6L-_UQe",
    "outputId": "94f1d418-2de2-42b6-b7ce-bc5417fe7bc4"
   },
   "outputs": [],
   "source": [
    "!python truecaser/applytc.py preproc-models/tc-et data/input.et > data/tc-input.et\n",
    "!python word-pieces.py --action split --corpora data/tc-input.et --model preproc-models/sp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-nZ2xezPIaRR"
   },
   "source": [
    "Now we can translate it. To get a readable sentence, we also need to reverse SentencePiece splitting afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 591
    },
    "colab_type": "code",
    "id": "y_YXsGZkojJi",
    "outputId": "6bcaa406-f20c-403e-ecdc-3f5966d72626"
   },
   "outputs": [],
   "source": [
    "!python -m sockeye.translate --models et2en_model --input data/sp-tc-input.et --output data/hypothesis.en\n",
    "!python word-pieces.py --action restore --corpora data/hypothesis.en --model preproc-models/sp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "96v78hQ7Ikhe"
   },
   "source": [
    "Let's see what we got:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "u5eXrHIP_tqU",
    "outputId": "1faf8910-dda1-4299-a6bb-85f0139e2208"
   },
   "outputs": [],
   "source": [
    "with open('data/de-sp-hypothesis.en', 'r', encoding='utf8') as fh:\n",
    "    print(fh.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ICVisgP7In_9"
   },
   "source": [
    "You can probably see that our model generates readable English text, but it is not necessarily a translation of the input. The language model component is already OK, but the conditioning part is not working yet. You will fix it when you train a bigger baseline with more data."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zFR5AcWkGsTo"
   },
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "lab2_part1_data_ta.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}