Multilingual Embeddings - 3. Transfer Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilingual Embeddings - 3. Transfer Learning\n",
    "\n",
    "One of the most promising applications of multilingual word embeddings is cross-lingual transfer learning. Suppose you would like to train an NLP model in several languages, but you only have training data in one language. Collecting new training data for each of your target languages can be expensive, and translating every text you'd like to process is rather cumbersome. With cross-lingual word embeddings, however, you can transfer your model from one language to the other very efficiently. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "We'll start by downloading two sets of embeddings from the [MUSE library](https://github.com/facebookresearch/MUSE) by Facebook Research. These are fastText Wikipedia supervised word embeddings for 30 languages, aligned in a single vector space. In this notebook, we'll work with the English and French embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-09-18 11:23:44--  https://s3.amazonaws.com/arrival/embeddings/wiki.multi.en.vec\n",
      "Resolving s3.amazonaws.com... 52.216.22.53\n",
      "Connecting to s3.amazonaws.com|52.216.22.53|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 628614720 (599M) [text/plain]\n",
      "Saving to: '/tmp/wiki.multi.en.vec'\n",
      "\n",
      "/tmp/wiki.multi.en. 100%[=====================>] 599.49M  9.08MB/s   in 71s    \n",
      "\n",
      "2018-09-18 11:24:57 (8.48 MB/s) - '/tmp/wiki.multi.en.vec' saved [628614720/628614720]\n",
      "\n",
      "--2018-09-18 11:24:57--  https://s3.amazonaws.com/arrival/embeddings/wiki.multi.fr.vec\n",
      "Resolving s3.amazonaws.com... 54.231.80.243\n",
      "Connecting to s3.amazonaws.com|54.231.80.243|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 628737710 (600M) [text/plain]\n",
      "Saving to: '/tmp/wiki.multi.fr.vec'\n",
      "\n",
      "/tmp/wiki.multi.fr. 100%[=====================>] 599.61M  9.66MB/s   in 77s    \n",
      "\n",
      "2018-09-18 11:26:16 (7.74 MB/s) - '/tmp/wiki.multi.fr.vec' saved [628737710/628737710]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/arrival/embeddings/wiki.multi.en.vec -O /tmp/wiki.multi.en.vec\n",
    "!wget https://s3.amazonaws.com/arrival/embeddings/wiki.multi.fr.vec -O /tmp/wiki.multi.fr.vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading vectors from /tmp/wiki.multi.en.vec\n",
      "Loading vectors from /tmp/wiki.multi.fr.vec\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_vectors(embedding_file_path):\n",
    "    print(\"Loading vectors from\", embedding_file_path)\n",
    "    embeddings = []\n",
    "    word2id = {}\n",
    "    with open(embedding_file_path, 'r', encoding='utf-8') as f:\n",
    "        next(f)\n",
    "        for i, line in enumerate(f):\n",
    "            word, emb = line.rstrip().split(' ', 1)\n",
    "            emb = np.fromstring(emb, sep=' ')\n",
    "            assert word not in word2id, 'word found twice'\n",
    "            embeddings.append(emb)\n",
    "            word2id[word] = len(word2id)\n",
    "\n",
    "    embeddings = np.vstack(embeddings)\n",
    "    return embeddings, word2id\n",
    "\n",
    "embeddings_en, embedding_word2id_en = load_vectors(\"/tmp/wiki.multi.en.vec\")\n",
    "embeddings_fr, embedding_word2id_fr = load_vectors(\"/tmp/wiki.multi.fr.vec\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "We'll use [Keras](https://keras.io/) to train a simple model for sentiment analysis. This model will tell us whether a text contains a positive or a negative opinion. As our training and test data, we'll take the popular IMDB dataset of movie reviews. \n",
    "\n",
    "After downloading the data, we apply the traditional preprocessing steps. We'll work with a vocabulary of 10,000 words, cut off long texts after 500 words and pad all shorter texts to that same number of tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "VOCABULARY_SIZE = 10000\n",
    "INDEX_FROM = 3\n",
    "START_INDEX = 1\n",
    "OOV_INDEX = 2\n",
    "EMBEDDING_DIM = 300\n",
    "SEQ_LENGTH = 500\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(path=\"imdb.npz\",\n",
    "                                                      num_words=VOCABULARY_SIZE,\n",
    "                                                      skip_top=0,\n",
    "                                                      maxlen=None,\n",
    "                                                      seed=113,\n",
    "                                                      start_char=START_INDEX,\n",
    "                                                      oov_char=OOV_INDEX,\n",
    "                                                      index_from=INDEX_FROM)\n",
    "\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=SEQ_LENGTH)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained word embeddings\n",
    "\n",
    "To be able to transfer our model between languages, its embedding layer needs to project the words into a multilingual vector space. We therefore initialize it with (a subset of) the multilingual word embeddings we downloaded earlier. \n",
    "\n",
    "First, we need to find out what words the indices in the preprocessed IMDB data stand for. Then we create an embedding matrix where each row contains the embedding of the word indexed by its row number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_embedding_matrix(target_word2id, embedding_word2id, embeddings, num_rows, num_columns):\n",
    "    embedding_matrix = np.zeros((num_rows, num_columns))\n",
    "    for word, i in target_word2id.items():\n",
    "        if i >= num_rows:\n",
    "            continue\n",
    "        if word in embedding_word2id: \n",
    "            embedding_matrix[i] = embeddings[embedding_word2id[word]]\n",
    "    return embedding_matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id_en = imdb.get_word_index()\n",
    "word2id_en = {k:(v+INDEX_FROM) for k,v in word2id_en.items()}\n",
    "word2id_en[\"<PAD>\"] = 0\n",
    "word2id_en[\"<START>\"] = START_INDEX\n",
    "word2id_en[\"<UNK>\"] = OOV_INDEX\n",
    "\n",
    "embedding_matrix_en = create_embedding_matrix(word2id_en, embedding_word2id_en, \n",
    "                                              embeddings_en, VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training our model\n",
    "\n",
    "Now it's time to train our model. We'll work with the basic convolutional network that is described [in this blogpost](https://machinelearningmastery.com/predict-sentiment-movie-reviews-using-deep-learning/). Its single-value output ranges from 0 to 1 and indicates whether a text is more positive (1) or negative (0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 500, 300)          3000600   \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 500, 32)           28832     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 250, 32)           0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 8000)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 250)               2000250   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 251       \n",
      "=================================================================\n",
      "Total params: 5,029,933\n",
      "Trainable params: 2,029,333\n",
      "Non-trainable params: 3,000,600\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Flatten\n",
    "from keras.layers.convolutional import Conv1D\n",
    "from keras.layers.convolutional import MaxPooling1D\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM, input_length=SEQ_LENGTH, weights=[embedding_matrix_en], trainable=False))\n",
    "model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))\n",
    "model.add(MaxPooling1D(pool_size=2))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(250, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll train this model with some standard settings. We'll use a batch size of 64, and stop iterating over the training data when the validation loss does not improve anymore. We'll end up with an accuracy of around 86%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 120s 5ms/step - loss: 0.4991 - acc: 0.7361 - val_loss: 0.3561 - val_acc: 0.8426\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 115s 5ms/step - loss: 0.3144 - acc: 0.8680 - val_loss: 0.3134 - val_acc: 0.8657\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 109s 4ms/step - loss: 0.2632 - acc: 0.8916 - val_loss: 0.3208 - val_acc: 0.8624\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 113s 5ms/step - loss: 0.2005 - acc: 0.9206 - val_loss: 0.3358 - val_acc: 0.8608\n",
      "25000/25000 [==============================] - 39s 2ms/step\n",
      "Accuracy: 86.08%\n"
     ]
    }
   ],
   "source": [
    "NUM_EPOCHS=20\n",
    "BATCH_SIZE=64\n",
    "\n",
    "earlystop = EarlyStopping(monitor='val_loss', patience=2)\n",
    "\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), \n",
    "          epochs=NUM_EPOCHS, batch_size=BATCH_SIZE, callbacks=[earlystop])\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few sanity checks will show the model does what it's supposed to do: a positive English sentence such as \"it was a great movie\" gets a score close to 1, while a negative English sentence such as \"it was a horrible movie\" gets a score closer to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.88016844],\n",
       "       [0.15926579]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_texts = [\"it was a great movie\".split(), \"it was a horrible movie\".split()]\n",
    "test_vectors = [[word2id_en.get(w, OOV_INDEX) for w in text] for text in test_texts]\n",
    "\n",
    "padded_test_vectors = sequence.pad_sequences(test_vectors, maxlen=SEQ_LENGTH)\n",
    "\n",
    "model.predict(padded_test_vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transferring our model\n",
    "\n",
    "Obviously the above model cannot be used to classify French texts. Most French words are not present in our English vocabulary, and even if they were, it's questionable whether their English embedding would contain the correct information. To transfer our model from English to French, we need to replace its English embedding layer by a French embedding layer.\n",
    "\n",
    "We're again going to work with a vocabulary of 10,000 words. Unfortunately, we don't have a relevant set of French texts to fit this vocabulary on. Therefore we'll install Luminoso Insight's useful [wordfreq](https://github.com/LuminosoInsight/wordfreq) library. This library contains word frequency information for many Western languages. In particular, `top_n_list` will give us the most frequent `n` words for a language, as measured in a set of large corpora. For example, it tells us the most frequent three English words are *the*, *of* and *to*, while the most frequent French words are *de*, *la* and *le*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: wordfreq in /Users/yvespeirsman/anaconda3/lib/python3.6/site-packages (2.1.0)\n",
      "Requirement already satisfied: langcodes>=1.4.1 in /Users/yvespeirsman/anaconda3/lib/python3.6/site-packages (from wordfreq) (1.4.1)\n",
      "Requirement already satisfied: msgpack in /Users/yvespeirsman/anaconda3/lib/python3.6/site-packages (from wordfreq) (0.5.6)\n",
      "Requirement already satisfied: regex==2018.02.21 in /Users/yvespeirsman/anaconda3/lib/python3.6/site-packages (from wordfreq) (2018.2.21)\n",
      "Requirement already satisfied: marisa-trie in /Users/yvespeirsman/anaconda3/lib/python3.6/site-packages (from langcodes>=1.4.1->wordfreq) (0.7.5)\n"
     ]
    }
   ],
   "source": [
    "!pip3 install wordfreq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'of', 'to', 'and', 'a', 'in', 'i', 'is', 'that', 'for']\n",
      "['de', 'la', 'le', 'et', 'l', 'à', 'les', 'est', 'des', 'en']\n"
     ]
    }
   ],
   "source": [
    "from wordfreq import top_n_list\n",
    "\n",
    "print(top_n_list('en', 10))\n",
    "print(top_n_list('fr', 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll construct our vocabulary from the most frequent 10,000 French words, and use it to create a French embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id_fr = {word: idx+INDEX_FROM for idx, word in enumerate(top_n_list('fr', VOCABULARY_SIZE))}\n",
    "word2id_fr[\"<PAD>\"] = 0\n",
    "word2id_fr[\"<START>\"] = START_INDEX\n",
    "word2id_fr[\"<UNK>\"] = OOV_INDEX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix_fr = create_embedding_matrix(word2id_fr, embedding_word2id_fr, \n",
    "                                              embeddings_fr, VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to replace the weights of the original embedding layer in our model by this new French embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix_fr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And lo and behold, the model we just trained on English texts can now suddenly take French texts as input and classify them correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.724368  ],\n",
       "       [0.28478226]], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_texts_fr = [\"c était un film merveilleux\".split(), \"c était un film horrible\".split()]\n",
    "test_vectors_fr = [[word2id_fr.get(w, OOV_INDEX) for w in text] for text in test_texts_fr]\n",
    "\n",
    "padded_test_vectors_fr = sequence.pad_sequences(test_vectors_fr, maxlen=SEQ_LENGTH)\n",
    "\n",
    "model.predict(padded_test_vectors_fr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Multilingual word embeddings offer us a simple and efficient way of transferring a model from one language to the other. This comes in handy when you need to apply your model to several languages, but you only have training data in one of them. \n",
    "\n",
    "Needless to say, transferring a model like this is not without its problems. After all, two different languages don't just have different vocabularies. They can also display variation in word order, etc. This can be problematic for models such as recurrent neural networks, but also for convolutional neural networks, which take the n-grams in a text as their input. \n",
    "\n",
    "When the two languages are reasonably close in linguistic terms, you should be able to get decent performance from transferred model. However, it's very likely a model that has been trained on data in the same language, will likely do better. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}