1_Bidirectional_LSTM_Eng_to_French.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translating French to English and back with Pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import re, pickle, collections, bcolz, numpy as np, keras, sklearn, math, operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Gavrilov\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:860: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import word2vec\n",
    "import gensim.models.keyedvectors\n",
    "import torch, torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path='C:/Users/Gavrilov/My Projects/Translation_system_Neural_Net/fr-en-109-corpus/'\n",
    "dpath = 'C:/Users/Gavrilov/My Projects/Translation_system_Neural_Net/translate/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Prepare corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The French-English parallel corpus can be downloaded from http://www.statmt.org/wmt10/training-giga-fren.tar. It was created by Chris Callison-Burch, who crawled millions of web pages and then used 'a set of simple heuristics to transform French URLs onto English URLs (i.e. replacing \"fr\" with \"en\" and about 40 other hand-written rules), and assume that these documents are translations of each other'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fname=path+'giga-fren.release2.fixed'\n",
    "en_fname = fname+'.en'\n",
    "fr_fname = fname+'.fr'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To make this problem a little simpler so we can train our model more quickly, we'll just learn to translate questions that begin with 'Wh' (e.g. what, why, where which). Here are our regexps that filter the sentences we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "re_eq = re.compile('^(Wh[^?.!]+\\?)') #read chek: in english look things start with Wh and end with question mark\n",
    "re_fq = re.compile('^([^?.!]+\\?)') #read chek: in french it can be anything at all but it should end with question mark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lines = ((re_eq.search(eq), re_fq.search(fq)) \n",
    "         for eq, fq in zip(open(en_fname, encoding='utf8'), open(fr_fname, encoding='utf8'))) \n",
    "#once you go open in Python, that return iterator that we can loop through\n",
    "#when we zip two together we got english questions and french questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52331"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qs = [(e.group(), f.group()) for e,f in lines if e and f]; len(qs) \n",
    "#then we just go through and run the regex, and then return the ones that both of the regex's match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('What is light ?', 'Qu’est-ce que la lumière?'),\n",
       " ('Who are we?', 'Où sommes-nous?'),\n",
       " ('Where did we come from?', \"D'où venons-nous?\"),\n",
       " ('What would we do without it?', 'Que ferions-nous sans elle ?'),\n",
       " ('What is the absolute location (latitude and longitude) of Badger, Newfoundland and Labrador?',\n",
       "  'Quelle sont les coordonnées (latitude et longitude) de Badger, à Terre-Neuve-etLabrador?'),\n",
       " ('What is the major aboriginal group on Vancouver Island?',\n",
       "  'Quel est le groupe autochtone principal sur l’île de Vancouver?')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qs[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Because it takes a while to load the data, we save the results to make it easier to load in later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pickle.dump(qs, open(dpath+'fr-en-qs.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "qs = pickle.load(open(dpath+'fr-en-qs.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "en_qs, fr_qs = zip(*qs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Because we are translating at word level, we need to tokenize the text first. (Note that it is also possible to translate at character level, which doesn't require tokenizing.) There are many tokenizers available, but we found we got best results using these simple heuristics.\n",
    "P.S. We also could use NLTK that has a lot of tokenizers in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "re_apos = re.compile(r\"(\\w)'s\\b\")         # make 's a separate word\n",
    "re_mw_punc = re.compile(r\"(\\w[’'])(\\w)\")  # other ' in a word creates 2 words\n",
    "re_punc = re.compile(\"([\\\"().,;:/_?!—])\") # add spaces around punctuation\n",
    "re_mult_space = re.compile(r\"  *\")        # replace multiple spaces with just one\n",
    "def simple_toks(sent):\n",
    "    sent = re_apos.sub(r\"\\1 's\", sent)\n",
    "    sent = re_mw_punc.sub(r\"\\1 \\2\", sent)\n",
    "    sent = re_punc.sub(r\" \\1 \", sent).replace('-', ' ')\n",
    "    sent = re_mult_space.sub(' ', sent)\n",
    "    return sent.lower().split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['qu’', 'est', 'ce', 'que', 'la', 'lumière', '?'],\n",
       " ['où', 'sommes', 'nous', '?'],\n",
       " [\"d'\", 'où', 'venons', 'nous', '?'],\n",
       " ['que', 'ferions', 'nous', 'sans', 'elle', '?']]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fr_qtoks = list(map(simple_toks, fr_qs)); fr_qtoks[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['what', 'is', 'light', '?'],\n",
       " ['who', 'are', 'we', '?'],\n",
       " ['where', 'did', 'we', 'come', 'from', '?'],\n",
       " ['what', 'would', 'we', 'do', 'without', 'it', '?']]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_qtoks = list(map(simple_toks, en_qs)); en_qtoks[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['rachel', \"'s\", 'baby', 'is', 'cuter', 'than', 'other', \"'s\", '.']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_toks(\"Rachel's baby is cuter than other's.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Special tokens used to pad the end of sentences, and to mark the start of a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "PAD = 0; SOS = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Enumerate the unique words (*vocab*) in the corpus, and also create the reverse map (word->index). Then use this mapping to encode every sentence as a list of int indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def toks2ids(sents):\n",
    "    voc_cnt = collections.Counter(t for sent in sents for t in sent) #create our vocabulary\n",
    "    vocab = sorted(voc_cnt, key=voc_cnt.get, reverse=True)\n",
    "    vocab.insert(PAD, \"<PAD>\") #insert padding character\n",
    "    vocab.insert(SOS, \"<SOS>\") #insert start of stream character\n",
    "    w2id = {w:i for i,w in enumerate(vocab)} #create reverse mappig from words to id\n",
    "    ids = [[w2id[t] for t in sent] for sent in sents] #return list of tokens\n",
    "    return ids, vocab, w2id, voc_cnt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fr_ids, fr_vocab, fr_w2id, fr_counts = toks2ids(fr_qtoks) \n",
    "#returning list of ID for each sentence, vocabulary, reverse vocabulary, and frequency count for vocabulary\n",
    "en_ids, en_vocab, en_w2id, en_counts = toks2ids(en_qtoks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Stanford's GloVe word vectors can be downloaded from https://nlp.stanford.edu/projects/glove/ (in the code below we have preprocessed them into a bcolz array). We use these because each individual word has a single word vector, which is what we need for translation. Word2vec, on the other hand, often uses multi-word phrases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('C:/Users/Gavrilov/.keras/datasets/glove_6B/glove.6B.100d.txt', 'r', encoding=\"utf8\") as f: \n",
    "    lines = [line.split() for line in f] #lines.split() -without value '\\n' because each iteration gives line and that is fine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "glove_words = [elem[0] for elem in lines]\n",
    "glove_words_idx = {elem:idx for idx,elem in enumerate(glove_words)} #is elem:idx equal to glove_words_idx[elem]=idx?\n",
    "glove_vecs = np.stack(np.array(elem[1:], dtype=np.float32) for elem in lines) #np.float32 -standard double-precision floating point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pickle.dump(glove_words, open('C:/Users/Gavrilov/.keras/datasets/glove_6B/glove.6B.100d.txt'+'_glove_words.pkl', 'wb'))\n",
    "pickle.dump(glove_words_idx, open('C:/Users/Gavrilov/.keras/datasets/glove_6B/glove.6B.100d.txt'+'_glove_words_idx.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving array using specific function\n",
    "def save_array(fname, arr):\n",
    "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
    "    c.flush()\n",
    "save_array('C:/Users/Gavrilov/.keras/datasets/glove_6B/glove.6B.100d.txt'+'_glove_vecs'+'.dat', glove_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def load_glove(loc):\n",
    "    return (bcolz.open(loc+'_glove_vecs.dat')[:],\n",
    "        pickle.load(open(loc+'_glove_words.pkl','rb'), encoding='utf8'),\n",
    "        pickle.load(open(loc+'_glove_words_idx.pkl','rb'), encoding='utf8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "en_vecs, en_wv_word, en_wv_idx = load_glove('C:/Users/Gavrilov/.keras/datasets/glove_6B/glove.6B.100d.txt')\n",
    "en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}\n",
    "n_en_vec, dim_en_vec = en_vecs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.32306999, -0.87616003,  0.21977   ,  0.25268   ,  0.22976001,\n",
       "        0.73879999, -0.37954   , -0.35306999, -0.84368998, -1.11129999,\n",
       "       -0.30265999,  0.33177999, -0.25113001,  0.30447999, -0.077491  ,\n",
       "       -0.89815003,  0.092496  , -1.14069998, -0.58323997,  0.66869003,\n",
       "       -0.23122001, -0.95854998,  0.28262001, -0.078848  ,  0.75314999,\n",
       "        0.26583999,  0.34220001, -0.33949   ,  0.95608002,  0.065641  ,\n",
       "        0.45747   ,  0.39835   ,  0.57964998,  0.39267001, -0.21851   ,\n",
       "        0.58794999, -0.55998999,  0.63367999, -0.043983  , -0.68730998,\n",
       "       -0.37841001,  0.38025999,  0.61641002, -0.88269001, -0.12346   ,\n",
       "       -0.37928   , -0.38317999,  0.23868001,  0.66850001, -0.43320999,\n",
       "       -0.11065   ,  0.081723  ,  1.15690005,  0.78957999, -0.21223   ,\n",
       "       -2.3211    , -0.67806   ,  0.44560999,  0.65706998,  0.1045    ,\n",
       "        0.46217   ,  0.19912   ,  0.25802001,  0.057194  ,  0.53443003,\n",
       "       -0.43133   , -0.34311   ,  0.59789002, -0.58416998,  0.068995  ,\n",
       "        0.23943999, -0.85180998,  0.30379   , -0.34176999, -0.25746   ,\n",
       "       -0.031101  , -0.16285001,  0.45168999, -0.91627002,  0.64521003,\n",
       "        0.73281002, -0.22752   ,  0.30226001,  0.044801  , -0.83740997,\n",
       "        0.55005997, -0.52506   , -1.73570001,  0.47510001, -0.70486999,\n",
       "        0.056939  , -0.71319997,  0.089623  ,  0.41394001, -1.33630002,\n",
       "       -0.61914998, -0.33089   , -0.52881002,  0.16483   , -0.98878002], dtype=float32)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_w2v['king']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Also we need to grab French word vectors which we using from http://fauconnier.github.io/index.html \n",
    "direct link http://embeddings.org/frWac_non_lem_no_postag_no_phrase_200_skip_cut100.bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "w2v_path='C:/Users/Gavrilov/.keras/datasets/frWac_non_lem_no_postag_no_phrase_200_skip_cut100.bin'\n",
    "#fr_model = word2vec.Word2Vec.load_word2vec_format(w2v_path, binary=True)\n",
    "fr_model = gensim.models.KeyedVectors.load_word2vec_format(w2v_path, binary=True)\n",
    "fr_voc = fr_model.vocab\n",
    "dim_fr_vec = 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We need to map each word index in our vocabs to their word vector. Not every word in our vocabs will be in our word vectors, since our tokenization approach won't be identical to the word vector creators - in these cases we simply create a random vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def create_emb(w2v, targ_vocab, dim_vec):\n",
    "    vocab_size = len(targ_vocab)\n",
    "    emb = np.zeros((vocab_size, dim_vec)) #creating big array of zeros\n",
    "    found=0 #keeping track of founfed words in the loop\n",
    "\n",
    "    for i, word in enumerate(targ_vocab):\n",
    "        try: emb[i] = w2v[word]; found+=1 #filling array of zeros with word vectors if we can\n",
    "        except KeyError: emb[i] = np.random.normal(scale=0.6, size=(dim_vec,)) #if we can't we just stick a random vector there instead\n",
    "\n",
    "    return emb, found"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((19549, 100), 17251)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_embs, found = create_emb(en_w2v, en_vocab, dim_en_vec); en_embs.shape, found"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((26709, 200), 21878)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fr_embs, found = create_emb(fr_model, fr_vocab, dim_fr_vec); fr_embs.shape, found"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Prep data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Each sentence has to be of equal length. Keras has a convenient function `pad_sequences` to truncate and/or pad each sentence as required - even although we're not using keras for the neural net, we can still use any functions from it we need!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((52331, 30), (52331, 30), (19549, 100))"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "maxlen = 30\n",
    "en_padded = pad_sequences(en_ids, maxlen, 'int64', \"post\", \"post\")\n",
    "fr_padded = pad_sequences(fr_ids, maxlen, 'int64', \"post\", \"post\")\n",
    "en_padded.shape, fr_padded.shape, en_embs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n = int(len(en_ids)*0.9)\n",
    "idxs = np.random.permutation(len(en_ids))\n",
    "fr_train, fr_test = fr_padded[idxs][:n], fr_padded[idxs][n:]\n",
    "en_train, en_test = en_padded[idxs][:n], en_padded[idxs][n:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(47097, 30)"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras_tqdm import TQDMNotebookCallback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "parms = {'verbose': 0, 'callbacks': [TQDMNotebookCallback()]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fr_wgts = [fr_embs.T, np.zeros((len(fr_vocab,)))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding, Bidirectional, LSTM, TimeDistributed, Dense, Activation, Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp = Input((maxlen,))\n",
    "x = Embedding(len(en_vocab), dim_en_vec, input_length=maxlen,\n",
    "              weights=[en_embs], trainable=False)(inp)\n",
    "x = Bidirectional(LSTM(128, return_sequences=True))(x)\n",
    "x = Bidirectional(LSTM(128, return_sequences=True))(x)\n",
    "x = LSTM(128, return_sequences=True)(x)\n",
    "x = TimeDistributed(Dense(dim_fr_vec))(x)\n",
    "x = TimeDistributed(Dense(len(fr_vocab), weights=fr_wgts))(x)\n",
    "x = Activation('softmax')(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(inp, x)\n",
    "model.compile('adam', 'sparse_categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import keras.backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "K.set_value(model.optimizer.lr, 1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47018a4b93aa452f903493646b701723",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a11a04365474a4b877e6d5c76114456",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-178-5c491b4561f4>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m hist=model.fit(en_train, np.expand_dims(fr_train,-1), batch_size=64, epochs=20, **parms, \n\u001b[1;32m----> 2\u001b[1;33m                validation_data=[en_test, np.expand_dims(fr_test,-1)])\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\training.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[0;32m   1655\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1656\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1657\u001b[1;33m                               validation_steps=validation_steps)\n\u001b[0m\u001b[0;32m   1658\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1659\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[1;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[0;32m   1225\u001b[0m                             val_outs = self._test_loop(val_f, val_ins,\n\u001b[0;32m   1226\u001b[0m                                                        \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1227\u001b[1;33m                                                        verbose=0)\n\u001b[0m\u001b[0;32m   1228\u001b[0m                             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mval_outs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1229\u001b[0m                                 \u001b[0mval_outs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mval_outs\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\training.py\u001b[0m in \u001b[0;36m_test_loop\u001b[1;34m(self, f, ins, batch_size, verbose, steps)\u001b[0m\n\u001b[0;32m   1368\u001b[0m                     \u001b[0mins_batch\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_slice_arrays\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_ids\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1369\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1370\u001b[1;33m                 \u001b[0mbatch_outs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1371\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch_outs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1372\u001b[0m                     \u001b[1;32mif\u001b[0m \u001b[0mbatch_index\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m   2355\u001b[0m         \u001b[0msession\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2356\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[1;32m-> 2357\u001b[1;33m                               **self.session_kwargs)\n\u001b[0m\u001b[0;32m   2358\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2359\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36mrun\u001b[1;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m    887\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    888\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[1;32m--> 889\u001b[1;33m                          run_metadata_ptr)\n\u001b[0m\u001b[0;32m    890\u001b[0m       \u001b[1;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    891\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_run\u001b[1;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m   1118\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1119\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[1;32m-> 1120\u001b[1;33m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[0;32m   1121\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1122\u001b[0m       \u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_do_run\u001b[1;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m   1315\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1316\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[1;32m-> 1317\u001b[1;33m                            options, run_metadata)\n\u001b[0m\u001b[0;32m   1318\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1319\u001b[0m       \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_do_call\u001b[1;34m(self, fn, *args)\u001b[0m\n\u001b[0;32m   1321\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1322\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1323\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1324\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1325\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[1;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[0;32m   1300\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[0;32m   1301\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1302\u001b[1;33m                                    status, run_metadata)\n\u001b[0m\u001b[0;32m   1303\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1304\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msession\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "hist=model.fit(en_train, np.expand_dims(fr_train,-1), batch_size=64, epochs=20, **parms, \n",
    "               validation_data=[en_test, np.expand_dims(fr_test,-1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights(dpath+'trans.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sent2ids(sent):\n",
    "    sent = simple_toks(sent)\n",
    "    ids = [en_w2id[t] for t in sent]\n",
    "    return pad_sequences([ids], maxlen, padding=\"post\", truncating=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def en2fr(sent): \n",
    "    ids = sent2ids(sent)\n",
    "    tr_ids = np.argmax(model.predict(ids), axis=-1)\n",
    "    return ' '.join(fr_vocab[i] for i in tr_ids[0] if i>0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'quelles est les les de ? ?'"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en2fr(\"what is the size of canada?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the model (just one epoch this time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0db3748393f4f1386805b4190502b7f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4ff886dada542a7baff9bb969a75edf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread Thread-6:\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\Gavrilov\\Anaconda3\\lib\\threading.py\", line 916, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"C:\\Users\\Gavrilov\\Anaconda3\\lib\\site-packages\\tqdm\\_tqdm.py\", line 144, in run\n",
      "    for instance in self.tqdm_cls._instances:\n",
      "  File \"C:\\Users\\Gavrilov\\Anaconda3\\lib\\_weakrefset.py\", line 60, in __iter__\n",
      "    for itemref in self.data:\n",
      "RuntimeError: Set changed size during iteration\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "hist=model.fit(en_train, np.expand_dims(fr_train,-1), batch_size=64, epochs=1, **parms, \n",
    "               validation_data=[en_test, np.expand_dims(fr_test,-1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights(dpath+'trans.h')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'quel est ce que ? ? ?'"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en2fr('what is the size of canada?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}