predict.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN = 'predict'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if your GPU is busy you can use CPU for predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['THEANO_FLAGS'] = 'device=cpu,floatX=float32'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1.0.4'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate headlines using the \"simple\" model from http://arxiv.org/pdf/1512.01712v1.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use indexing of tokens from [vocabulary-embedding](./vocabulary-embedding.ipynb) this does not clip the indexes of the words to `vocab_size`.\n",
    "\n",
    "Use the index of outside words to replace them with several `oov` words (`oov` , `oov0`, `oov`...) that appear in the same description and headline. This will allow headline generator to replace the oov with the same word in the description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "FN0 = 'vocabulary-embedding'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we will generate predictions using the model generated in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN1 = 'train'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "input data (`X`) is made from `maxlend` description words followed by `eos`\n",
    "followed by headline words followed by `eos`\n",
    "if description is shorter than `maxlend` it will be left padded with `empty`\n",
    "if entire data is longer than `maxlen` it will be clipped and if it is shorter it will be padded.\n",
    "\n",
    "labels (`Y`) are the headline words followed by `eos` and clipped or padded to `maxlenh`\n",
    "\n",
    "In other words the input is made from a `maxlend` half in which the description is padded from the left\n",
    "and a `maxlenh` half in which `eos` is followed by a headline followed by another `eos` if there is enough space.\n",
    "\n",
    "The labels match only the second half and \n",
    "the first label matches the `eos` at the start of the second half (following the description in the first half)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the model parameters should be identical with what used in training but notice that `maxlend` is flexible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "maxlend=50 # 0 - if we dont want to use description at all\n",
    "maxlenh=25\n",
    "maxlen = maxlend + maxlenh\n",
    "rnn_size = 512\n",
    "rnn_layers = 3  # match FN1\n",
    "batch_norm=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the out of the first `activation_rnn_size` nodes from the top layer will be used for activation and the rest will be used to select predicted word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "activation_rnn_size = 40 if maxlend else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# training parameters\n",
    "seed=42\n",
    "p_W, p_U, p_dense, p_emb, weight_decay = 0, 0, 0, 0, 0\n",
    "optimizer = 'adam'\n",
    "batch_size=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nb_train_samples = 30000\n",
    "nb_val_samples = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# read word embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cPickle as pickle\n",
    "\n",
    "with open('data/%s.pkl'%FN0, 'rb') as fp:\n",
    "    embedding, idx2word, word2idx, glove_idx2idx = pickle.load(fp)\n",
    "vocab_size, embedding_size = embedding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_unknown_words = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dimension of embedding space for words 100\n",
      "vocabulary size 40000 the last 10 words can be used as place holders for unknown/oov words\n",
      "total number of different words 523734 523734\n",
      "number of words outside vocabulary which we can substitue using glove similarity 122377\n",
      "number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov) 361357\n"
     ]
    }
   ],
   "source": [
    "print 'dimension of embedding space for words',embedding_size\n",
    "print 'vocabulary size', vocab_size, 'the last %d words can be used as place holders for unknown/oov words'%nb_unknown_words\n",
    "print 'total number of different words',len(idx2word), len(word2idx)\n",
    "print 'number of words outside vocabulary which we can substitue using glove similarity', len(glove_idx2idx)\n",
    "print 'number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov)',len(idx2word)-vocab_size-len(glove_idx2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(nb_unknown_words):\n",
    "    idx2word[vocab_size-1-i] = '<%d>'%i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(vocab_size-nb_unknown_words, len(idx2word)):\n",
    "    idx2word[i] = idx2word[i]+'^'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "empty = 0\n",
    "eos = 1\n",
    "idx2word[empty] = '_'\n",
    "idx2word[eos] = '~'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing import sequence\n",
    "from keras.utils import np_utils\n",
    "import random, sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prt(label, x):\n",
    "    print label+':',\n",
    "    for w in x:\n",
    "        print idx2word[w],\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation, Dropout, RepeatVector, Merge, TimeDistributedDense\n",
    "from keras.layers.recurrent import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.regularizers import l2\n",
    "from keras.layers.core import Lambda\n",
    "import keras.backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# seed weight initialization\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "regularizer = l2(weight_decay) if weight_decay else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## rnn model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start with a stacked LSTM, which is identical to the bottom of the model used in training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rnn_model = Sequential()\n",
    "rnn_model.add(Embedding(vocab_size, embedding_size,\n",
    "                        input_length=maxlen,\n",
    "                        W_regularizer=regularizer, dropout=p_emb, weights=[embedding], mask_zero=True,\n",
    "                        name='embedding_1'))\n",
    "for i in range(rnn_layers):\n",
    "    lstm = LSTM(rnn_size, return_sequences=True, # batch_norm=batch_norm,\n",
    "                W_regularizer=regularizer, U_regularizer=regularizer,\n",
    "                b_regularizer=regularizer, dropout_W=p_W, dropout_U=p_U,\n",
    "                name='lstm_%d'%(i+1)\n",
    "                  )\n",
    "    rnn_model.add(lstm)\n",
    "    rnn_model.add(Dropout(p_dense, name='dropout_%d'%(i+1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use the bottom weights from the trained model, and save the top weights for later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "def str_shape(x):\n",
    "    return 'x'.join(map(str,x.shape))\n",
    "\n",
    "def inspect_model(model):\n",
    "    print model.name\n",
    "    for i,l in enumerate(model.layers):\n",
    "        print i, 'cls=%s name=%s'%(type(l).__name__, l.name)\n",
    "        weights = l.get_weights()\n",
    "        for weight in weights:\n",
    "            print str_shape(weight),\n",
    "        print\n",
    "\n",
    "def load_weights(model, filepath):\n",
    "    \"\"\"Modified version of keras load_weights that loads as much as it can\n",
    "    if there is a mismatch between file and model. It returns the weights\n",
    "    of the first layer in which the mismatch has happened\n",
    "    \"\"\"\n",
    "    print 'Loading', filepath, 'to', model.name\n",
    "    flattened_layers = model.layers\n",
    "    with h5py.File(filepath, mode='r') as f:\n",
    "        # new file format\n",
    "        layer_names = [n.decode('utf8') for n in f.attrs['layer_names']]\n",
    "\n",
    "        # we batch weight value assignments in a single backend call\n",
    "        # which provides a speedup in TensorFlow.\n",
    "        weight_value_tuples = []\n",
    "        for name in layer_names:\n",
    "            print name\n",
    "            g = f[name]\n",
    "            weight_names = [n.decode('utf8') for n in g.attrs['weight_names']]\n",
    "            if len(weight_names):\n",
    "                weight_values = [g[weight_name] for weight_name in weight_names]\n",
    "                try:\n",
    "                    layer = model.get_layer(name=name)\n",
    "                except:\n",
    "                    layer = None\n",
    "                if not layer:\n",
    "                    print 'failed to find layer', name, 'in model'\n",
    "                    print 'weights', ' '.join(str_shape(w) for w in weight_values)\n",
    "                    print 'stopping to load all other layers'\n",
    "                    weight_values = [np.array(w) for w in weight_values]\n",
    "                    break\n",
    "                symbolic_weights = layer.trainable_weights + layer.non_trainable_weights\n",
    "                weight_value_tuples += zip(symbolic_weights, weight_values)\n",
    "                weight_values = None\n",
    "        K.batch_set_value(weight_value_tuples)\n",
    "    return weight_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data/train.hdf5 to sequential_1\n",
      "embedding_1\n",
      "lstm_1\n",
      "dropout_1\n",
      "lstm_2\n",
      "dropout_2\n",
      "lstm_3\n",
      "dropout_3\n",
      "simplecontext_1\n",
      "timedistributed_1\n",
      "failed to find layer timedistributed_1 in model\n",
      "weights 944x40000 40000\n",
      "stopping to load all other layers\n"
     ]
    }
   ],
   "source": [
    "weights = load_weights(rnn_model, 'data/%s.hdf5'%FN1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(944, 40000), (40000,)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[w.shape for w in weights]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## headline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A special layer that reduces the input just to its headline part (second half).\n",
    "For each word in this part it concatenate the output of the previous layer (RNN)\n",
    "with a weighted average of the outputs of the description part.\n",
    "In this only the last `rnn_size - activation_rnn_size` are used from each output.\n",
    "The first `activation_rnn_size` output is used to computer the weights for the averaging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "context_weight = K.variable(1.)\n",
    "head_weight = K.variable(1.)\n",
    "cross_weight = K.variable(0.)\n",
    "\n",
    "def simple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):\n",
    "    desc, head = X[:,:maxlend], X[:,maxlend:]\n",
    "    head_activations, head_words = head[:,:,:n], head[:,:,n:]\n",
    "    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]\n",
    "    \n",
    "    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot\n",
    "    # activation for every head word and every desc word\n",
    "    activation_energies = K.batch_dot(head_activations, desc_activations, axes=([2],[2]))\n",
    "    # make sure we dont use description words that are masked out\n",
    "    assert mask.ndim == 2\n",
    "    activation_energies = K.switch(mask[:, None, :maxlend], activation_energies, -1e20)\n",
    "    \n",
    "    # for every head word compute weights for every desc word\n",
    "    activation_energies = K.reshape(activation_energies,(-1,maxlend))\n",
    "    activation_weights = K.softmax(activation_energies)\n",
    "    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))\n",
    "\n",
    "    # for every head word compute weighted average of desc words\n",
    "    desc_avg_word = K.batch_dot(activation_weights, desc_words, axes=([2],[1]))\n",
    "    return K.concatenate((context_weight*desc_avg_word, head_weight*head_words))\n",
    "\n",
    "\n",
    "class SimpleContext(Lambda):\n",
    "    def __init__(self,**kwargs):\n",
    "        super(SimpleContext, self).__init__(simple_context,**kwargs)\n",
    "        self.supports_masking = True\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        return input_mask[:, maxlend:]\n",
    "    \n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        nb_samples = input_shape[0]\n",
    "        n = 2*(rnn_size - activation_rnn_size)\n",
    "        return (nb_samples, maxlenh, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(rnn_model)\n",
    "\n",
    "if activation_rnn_size:\n",
    "    model.add(SimpleContext(name='simplecontext_1'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we are not going to fit so we dont care about loss and optimizer\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "944"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 2*(rnn_size - activation_rnn_size)\n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "perform the top dense of the trained model in numpy so we can play around with exactly how it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# out very own softmax\n",
    "def output2probs(output):\n",
    "    output = np.dot(output, weights[0]) + weights[1]\n",
    "    output -= output.max()\n",
    "    output = np.exp(output)\n",
    "    output /= output.sum()\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def output2probs1(output):\n",
    "    output0 = np.dot(output[:n//2], weights[0][:n//2,:])\n",
    "    output1 = np.dot(output[n//2:], weights[0][n//2:,:])\n",
    "    output = output0 + output1 # + output0 * output1\n",
    "    output += weights[1]\n",
    "    output -= output.max()\n",
    "    output = np.exp(output)\n",
    "    output /= output.sum()\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lpadd(x, maxlend=maxlend, eos=eos):\n",
    "    \"\"\"left (pre) pad a description to maxlend and then add eos.\n",
    "    The eos is the input to predicting the first word in the headline\n",
    "    \"\"\"\n",
    "    assert maxlend >= 0\n",
    "    if maxlend == 0:\n",
    "        return [eos]\n",
    "    n = len(x)\n",
    "    if n > maxlend:\n",
    "        x = x[-maxlend:]\n",
    "        n = maxlend\n",
    "    return [empty]*(maxlend-n) + x + [eos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samples = [lpadd([3]*26)]\n",
    "# pad from right (post) so the first maxlend will be description followed by headline\n",
    "data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(data[:,maxlend] == eos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 75), [51])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape,map(len, samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 25, 944)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = model.predict(data, verbose=0, batch_size=1)\n",
    "probs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# variation to https://github.com/ryankiros/skip-thoughts/blob/master/decoding/search.py\n",
    "def beamsearch(predict, start=[empty]*maxlend + [eos], avoid=None, avoid_score=1,\n",
    "               k=1, maxsample=maxlen, use_unk=True, oov=vocab_size-1, empty=empty, eos=eos, temperature=1.0):\n",
    "    \"\"\"return k samples (beams) and their NLL scores, each sample is a sequence of labels,\n",
    "    all samples starts with an `empty` label and end with `eos` or truncated to length of `maxsample`.\n",
    "    You need to supply `predict` which returns the label probability of each sample.\n",
    "    `use_unk` allow usage of `oov` (out-of-vocabulary) label in samples\n",
    "    \"\"\"\n",
    "    def sample(energy, n, temperature=temperature):\n",
    "        \"\"\"sample at most n different elements according to their energy\"\"\"\n",
    "        n = min(n,len(energy))\n",
    "        prb = np.exp(-np.array(energy) / temperature )\n",
    "        res = []\n",
    "        for i in xrange(n):\n",
    "            z = np.sum(prb)\n",
    "            r = np.argmax(np.random.multinomial(1, prb/z, 1))\n",
    "            res.append(r)\n",
    "            prb[r] = 0. # make sure we select each element only once\n",
    "        return res\n",
    "\n",
    "    dead_samples = []\n",
    "    dead_scores = []\n",
    "    live_samples = [list(start)]\n",
    "    live_scores = [0]\n",
    "\n",
    "    while live_samples:\n",
    "        # for every possible live sample calc prob for every possible label \n",
    "        probs = predict(live_samples, empty=empty)\n",
    "        assert vocab_size == probs.shape[1]\n",
    "\n",
    "        # total score for every sample is sum of -log of word prb\n",
    "        cand_scores = np.array(live_scores)[:,None] - np.log(probs)\n",
    "        cand_scores[:,empty] = 1e20\n",
    "        if not use_unk and oov is not None:\n",
    "            cand_scores[:,oov] = 1e20\n",
    "        if avoid:\n",
    "            for a in avoid:\n",
    "                for i, s in enumerate(live_samples):\n",
    "                    n = len(s) - len(start)\n",
    "                    if n < len(a):\n",
    "                        # at this point live_sample is before the new word,\n",
    "                        # which should be avoided, is added\n",
    "                        cand_scores[i,a[n]] += avoid_score\n",
    "        live_scores = list(cand_scores.flatten())\n",
    "        \n",
    "\n",
    "        # find the best (lowest) scores we have from all possible dead samples and\n",
    "        # all live samples and all possible new words added\n",
    "        scores = dead_scores + live_scores\n",
    "        ranks = sample(scores, k)\n",
    "        n = len(dead_scores)\n",
    "        dead_scores = [dead_scores[r] for r in ranks if r < n]\n",
    "        dead_samples = [dead_samples[r] for r in ranks if r < n]\n",
    "        \n",
    "        live_scores = [live_scores[r-n] for r in ranks if r >= n]\n",
    "        live_samples = [live_samples[(r-n)//vocab_size]+[(r-n)%vocab_size] for r in ranks if r >= n]\n",
    "\n",
    "        # live samples that should be dead are...\n",
    "        # even if len(live_samples) == maxsample we dont want it dead because we want one\n",
    "        # last prediction out of it to reach a headline of maxlenh\n",
    "        def is_zombie(s):\n",
    "            return s[-1] == eos or len(s) > maxsample\n",
    "        \n",
    "        # add zombies to the dead\n",
    "        dead_scores += [c for s, c in zip(live_samples, live_scores) if is_zombie(s)]\n",
    "        dead_samples += [s for s in live_samples if is_zombie(s)]\n",
    "        \n",
    "        # remove zombies from the living \n",
    "        live_scores = [c for s, c in zip(live_samples, live_scores) if not is_zombie(s)]\n",
    "        live_samples = [s for s in live_samples if not is_zombie(s)]\n",
    "\n",
    "    return dead_samples, dead_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# !pip install python-Levenshtein"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def keras_rnn_predict(samples, empty=empty, model=model, maxlen=maxlen):\n",
    "    \"\"\"for every sample, calculate probability for every possible label\n",
    "    you need to supply your RNN model and maxlen - the length of sequences it can handle\n",
    "    \"\"\"\n",
    "    sample_lengths = map(len, samples)\n",
    "    assert all(l > maxlend for l in sample_lengths)\n",
    "    assert all(l[maxlend] == eos for l in samples)\n",
    "    # pad from right (post) so the first maxlend will be description followed by headline\n",
    "    data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "    probs = model.predict(data, verbose=0, batch_size=batch_size)\n",
    "    return np.array([output2probs(prob[sample_length-maxlend-1]) for prob, sample_length in zip(probs, sample_lengths)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_fold(xs):\n",
    "    \"\"\"convert list of word indexes that may contain words outside vocab_size to words inside.\n",
    "    If a word is outside, try first to use glove_idx2idx to find a similar word inside.\n",
    "    If none exist then replace all accurancies of the same unknown word with <0>, <1>, ...\n",
    "    \"\"\"\n",
    "    xs = [x if x < vocab_size-nb_unknown_words else glove_idx2idx.get(x,x) for x in xs]\n",
    "    # the more popular word is <0> and so on\n",
    "    outside = sorted([x for x in xs if x >= vocab_size-nb_unknown_words])\n",
    "    # if there are more than nb_unknown_words oov words then put them all in nb_unknown_words-1\n",
    "    outside = dict((x,vocab_size-1-min(i, nb_unknown_words-1)) for i, x in enumerate(outside))\n",
    "    xs = [outside.get(x,x) for x in xs]\n",
    "    return xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_unfold(desc,xs):\n",
    "    # assume desc is the unfolded version of the start of xs\n",
    "    unfold = {}\n",
    "    for i, unfold_idx in enumerate(desc):\n",
    "        fold_idx = xs[i]\n",
    "        if fold_idx >= vocab_size-nb_unknown_words:\n",
    "            unfold[fold_idx] = unfold_idx\n",
    "    return [unfold.get(x,x) for x in xs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import Levenshtein\n",
    "\n",
    "def gensamples(X=None, X_test=None, Y_test=None, avoid=None, avoid_score=1, skips=2, k=10, batch_size=batch_size, short=True, temperature=1., use_unk=True):\n",
    "    if X is None or isinstance(X,int):\n",
    "        if X is None:\n",
    "            i = random.randint(0,len(X_test)-1)\n",
    "        else:\n",
    "            i = X\n",
    "        print 'HEAD %d:'%i,' '.join(idx2word[w] for w in Y_test[i])\n",
    "        print 'DESC:',' '.join(idx2word[w] for w in X_test[i])\n",
    "        sys.stdout.flush()\n",
    "        x = X_test[i]\n",
    "    else:\n",
    "        x = [word2idx[w.rstrip('^')] for w in X.split()]\n",
    "        \n",
    "    if avoid:\n",
    "        # avoid is a list of avoids. Each avoid is a string or list of word indeicies\n",
    "        if isinstance(avoid,str) or isinstance(avoid[0], int):\n",
    "            avoid = [avoid]\n",
    "        avoid = [a.split() if isinstance(a,str) else a for a in avoid]\n",
    "        avoid = [vocab_fold([w if isinstance(w,int) else word2idx[w] for w in a])\n",
    "                 for a in avoid]\n",
    "\n",
    "    print 'HEADS:'\n",
    "    samples = []\n",
    "    if maxlend == 0:\n",
    "        skips = [0]\n",
    "    else:\n",
    "        skips = range(min(maxlend,len(x)), max(maxlend,len(x)), abs(maxlend - len(x)) // skips + 1)\n",
    "    for s in skips:\n",
    "        start = lpadd(x[:s])\n",
    "        fold_start = vocab_fold(start)\n",
    "        sample, score = beamsearch(predict=keras_rnn_predict, start=fold_start, avoid=avoid, avoid_score=avoid_score,\n",
    "                                   k=k, temperature=temperature, use_unk=use_unk)\n",
    "        assert all(s[maxlend] == eos for s in sample)\n",
    "        samples += [(s,start,scr) for s,scr in zip(sample,score)]\n",
    "\n",
    "    samples.sort(key=lambda x: x[-1])\n",
    "    codes = []\n",
    "    for sample, start, score in samples:\n",
    "        code = ''\n",
    "        words = []\n",
    "        sample = vocab_unfold(start, sample)[len(start):]\n",
    "        for w in sample:\n",
    "            if w == eos:\n",
    "                break\n",
    "            words.append(idx2word[w])\n",
    "            code += chr(w//(256*256)) + chr((w//256)%256) + chr(w%256)\n",
    "        if short:\n",
    "            distance = min([100] + [-Levenshtein.jaro(code,c) for c in codes])\n",
    "            if distance > -0.6:\n",
    "                print score, ' '.join(words)\n",
    "        #         print '%s (%.2f) %f'%(' '.join(words), score, distance)\n",
    "        else:\n",
    "                print score, ' '.join(words)\n",
    "        codes.append(code)\n",
    "    return samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seed = 8\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = \"* Billy Joel is looking for a buyer in Sagaponack^ . Now that he and wife Katie Lee Joel are splitting up , the singer is planning to sell the two oceanfront^ properties he bought for her in 2007 . The four-bedroom mansion ( No . 1 ) and smaller beach bungalow^ ( No . 2 ) will be listed with Corcoran 's Biana^ Stepanian^ for a combined $ 35 million . * Richard Bressler^ , the former CFO of Viacom and now a managing\"\n",
    "Y = \"Billy Joel Lists in Sagaponack^\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "18.3972350549 Gossip Roundup : It 's Not a Enough\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X=X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = \"18 Cake GIFs That 'll Make You Moist\"\n",
    "Y = \"Is it 350degF^ in here or is it just me ?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "8.39775133658 The Cake\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"President Barack Obama 's re-election campaign is fundraising off of comments on Obama 's birth certificate by Mitt Romney 's son Matt .\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "6.44595342034 <0>^\n",
      "7.14836262441 Bacon Cake\n",
      "7.15618682446 Cake Cake\n",
      "10.4518038648 Bacon Cake !\n",
      "12.3920855886 <0>^ , Bacon\n",
      "15.6066176586 Cake Cake , <0>^\n",
      "16.6810159629 Cake Cake , Cake !\n",
      "20.0855381511 <0>^ , Bacon and Cake\n",
      "21.7307348662 Cake Cake , Cake , and <0>^\n",
      "24.5682745072 <0>^ , Bacon and Cake .\n",
      "27.2642401814 Cake Cake , Cake , and <0>^ Puppies\n",
      "28.0683858812 <0>^ , Bacon Is Not , Too !\n",
      "28.1353368097 <0>^ , Bacon Is Not Super Bowl\n",
      "30.0097000925 <0>^ , Bacon Is Not , And !\n",
      "33.728532485 <0>^ , Bacon Is Not , Here 's What Cake\n",
      "36.8273877234 Cake Cake , Cake , and <0>^ Best Wedding\n",
      "37.9201195189 Cake Cake , Cake , and <0>^ Best Wedding Moments\n",
      "38.8532513543 <0>^ , Bacon Is Not , Here 's What Cake Is Like\n",
      "44.855803035 Cake Cake , Cake , and <0>^ Best Wedding Moments EVER .\n",
      "58.2777635869 <0>^ , Bacon Is Not , Here 's What Cake Cake Can Internet ( NSFW )\n"
     ]
    }
   ],
   "source": [
    "gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1, use_unk=True, short=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"What have you been listening to this year ? If you want to find out using cold , hard evidence , then Spotify 's new Year in Music tool will tell you .\"\n",
    "Y = \"Spotify Will Make You Smarter for Your App\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "12.9753409925 How To Make A <0>^\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "headline = samples[0][0][len(samples[0][1]):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'How To Make A <0>^ ~'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(idx2word[w] for w in headline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "avoid = headline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "15.1214181555 Spotify Is the <0>^ Deal of the Day\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, avoid=avoid, avoid_score=.1, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "avoid = samples[0][0][len(samples[0][1]):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "15.8060034323 The 10 Most Popular Songs Of All Time\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, avoid=avoid, avoid_score=.1, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def wsimple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):\n",
    "    desc, head = X[:,:maxlend], X[:,maxlend:]\n",
    "    head_activations, head_words = head[:,:,:n], head[:,:,n:]\n",
    "    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]\n",
    "    \n",
    "    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot\n",
    "    # activation for every head word and every desc word\n",
    "    activation_energies = K.batch_dot(head_activations, desc_activations, axes=([2],[2]))\n",
    "    # make sure we dont use description words that are masked out\n",
    "    assert mask.ndim == 2\n",
    "    activation_energies = K.switch(mask[:, None, :maxlend], activation_energies, -1e20)\n",
    "    \n",
    "    # for every head word compute weights for every desc word\n",
    "    activation_energies = K.reshape(activation_energies,(-1,maxlend))\n",
    "    activation_weights = K.softmax(activation_energies)\n",
    "    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))\n",
    "\n",
    "    return activation_weights\n",
    "\n",
    "\n",
    "class WSimpleContext(Lambda):\n",
    "    def __init__(self):\n",
    "        super(WSimpleContext, self).__init__(wsimple_context)\n",
    "        self.supports_masking = True\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        return input_mask[:, maxlend:]\n",
    "    \n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        nb_samples = input_shape[0]\n",
    "        n = 2*(rnn_size - activation_rnn_size)\n",
    "        return (nb_samples, maxlenh, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wmodel = Sequential()\n",
    "wmodel.add(rnn_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wmodel.add(WSimpleContext())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wmodel.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seed = 8\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "context_weight.set_value(np.float32(1.))\n",
    "head_weight.set_value(np.float32(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"Representatives of the groups depicted in The Revenant^ spoke with BuzzFeed News about the actor 's Golden Globes speech calling on listeners to `` protect ... indigenous lands . ''\"\n",
    "Y = \"Native American Groups Officially Respond To Leonardo DiCaprio 's Call To Action\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "20.5102220631 How The Golden Globes Will Make You Cry\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sample = samples[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Representatives of the groups depicted in The trainwreck spoke with BuzzFeed News about the actor 's Golden Globes speech calling on listeners to `` protect ... indigenous lands . '' ~ How The Golden Globes Will Make You Cry ~\""
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([idx2word[w] for w in sample])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 75)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = sequence.pad_sequences([sample], maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 25, 50)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = wmodel.predict(data, verbose=0, batch_size=1)\n",
    "weights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 8)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "startd = np.where(data[0,:] != empty)[0][0]\n",
    "lenh = np.where(data[0,maxlend+1:] == eos)[0][0]\n",
    "startd, lenh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/udi/anaconda2/lib/python2.7/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.\n",
      "  warnings.warn(self.msg_depr % (key, alt_key))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEACAYAAABWLgY0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGI9JREFUeJzt3X+M3PV95/HnyzhYtQnONmS9cXKbklhpaeRiSJu0yZ7E\n3eKWpCq22qjXn0BOnKr0ekGphPBVquKqp8qmf7Q5nQ4pKs05EnupSptiWve8lKCVti35ZezMcY4T\n4yQQ7F03mPgHxNiY9/2xs2ZYZnc+s/OdmfdOXw8JMd/vfsfz5LPD11+/d2asiMDMzAbTqn4HmJlZ\n9/gkb2Y2wHySNzMbYD7Jm5kNMJ/kzcwGmE/yZmYDrOVJXtK7JT0h6UD936clfVzSkKRJSUck7Ze0\nvhfBZmZWTu28Tl7SKuC7wPuB3wGei4h7Jd0DDEXEju5kmpnZcrQ7rrkZeCoingG2AXvq+/cA26sM\nMzOzzrV7kv8PwET99oaImAWIiBlguMowMzPrXPFJXtIbgFuBv6zvWjjn8ecjmJkls7qNYz8EfDUi\nvlffnpW0ISJmJY0AJ5vd6dZbb43z588zMjICwLp169i0aRNbtmwB4ODBgwA93T569Cgf+chH+vb4\nzbbn92Xpmd9+8MEH+/79Wrjt71/Z9sK2fvdAzudT4xr1+/u1f/9+AEZGRli3bh333Xef6FDxD14l\n/W/g/0TEnvr2buBUROxe6gevt912W3zqU5/qtLNSu3btYseOXD8jztgEObvcVMZN5TJ23XXXXXz2\ns5/t+CRfNK6RtJa5H7r+dcPu3cBWSUeAcWBXs/vOzMx02li5p59+ut8Jr5OxCXJ2uamMm8pl7apC\n0bgmIl4E3rJg3ynmTvxmZpbUFTt37uzqA5w8eXLnDTfc0NXHaNf69esZHR3td8ZrZGyCnF1uKuOm\nchm7Tpw4wQc+8IE/6PTXaevNUMvx6KOPxo033tjVxzAzGzQHDhxgfHy8NzP5TjT+9DqL6enpfie8\nTsYmyNnlpjJuKpe1qwr+gDIzswHmcY2ZWUIrZlxjZmb945l8EhmbIGeXm8q4qVzWrir4St7MbIB5\nJm9mlpBn8mZm1pJn8klkbIKcXW4q46ZyWbuq4Ct5M7MB5pm8mVlCnsmbmVlLnsknkbEJcna5qYyb\nymXtqoKv5M3MBphn8mZmCVU1k2/nL/JethcuXALgylXiDav9hwczs17pyUz+Ew9/g088/A2OPf+D\nbj9ckYzzt4xNkLPLTWXcVC5rVxV6ciX/7efPA3DplV48mpmZzevJTH7Hgbmx0p/+wrv58Q3ruvp4\nZmaDwK+TNzOzlvw6+SQyNkHOLjeVcVO5rF1VKDrJS1ov6S8lHZb0pKT3SxqSNCnpiKT9ktZ3O9bM\nzNpTNJOX9L+AqYj4jKTVwDrg94DnIuJeSfcAQxGxY+F9PZM3M2tfz2bykq4G/m1EfAYgIl6OiNPA\nNmBP/bA9wPZOY8zMrFol45prge9J+oykA5I+LWktsCEiZgEiYgYYbnZnz+TLZGyCnF1uKuOmclm7\nqlDyOvnVwI3Af46Ir0j6E2AHsHDO03TuMzU1xbHjk6wZGuH+77yZTRuvYfPmzYyNjQGvLm4vt2u1\nWl8fv9n2vCw989u1Wi1Vj79/K3s74/OpUT97pqenmZiYAGB0dJTh4WHGx8fpVMuZvKQNwD9HxDvr\n22PMneTfBdwUEbOSRoDHIuK6hff3TN7MrH09m8nXRzLPSHp3fdc48CSwF7ijvu924KFOY8zMrFql\nr5P/OPCApIPA9cAfAbuBrZKOMHfi39Xsjp7Jl8nYBDm73FTGTeWydlWh6LNrIuIQ8FNNvnRztTlm\nZlYlf3aNmVlC/uwaMzNryZ9dk0TGJsjZ5aYybiqXtasKvpI3MxtgnsmbmSXkmbyZmbXkmXwSGZsg\nZ5ebyripXNauKvhK3sxsgHkmb2aWkGfyZmbWkmfySWRsgpxdbirjpnJZu6rgK3kzswHmmbyZWUKe\nyZuZWUueySeRsQlydrmpjJvKZe2qgq/kzcwGmGfyZmYJeSZvZmYteSafRMYmyNnlpjJuKpe1qwq+\nkjczG2CeyZuZJeSZvJmZteSZfBIZmyBnl5vKuKlc1q4qrC45SNK3gdPAK8DFiHifpCHgL4B3AN8G\nfjkiTnep08zMlqFoJi/pGPDeiHi+Yd9u4LmIuFfSPcBQROxYeF/P5M3M2tfrmbyaHLsN2FO/vQfY\n3mmMmZlVq/QkH8Ajkr4s6c76vg0RMQsQETPAcLM7eiZfJmMT5OxyUxk3lcvaVYWimTzwwYg4Iekt\nwKSkI8yd+Bs1nftMTU1x7Pgka4ZGuP87b2bTxmvYvHkzY2NjwKuL28vtWq3W18dvtj0vS8/8dq1W\nS9Xj79/K3s74fGrUz57p6WkmJiYAGB0dZXh4mPHxcTrV9uvkJX0SOAfcCdwUEbOSRoDHIuK6hcd7\nJm9m1r6ezeQlrZV0Vf32OuBngRqwF7ijftjtwEOdxpiZWbVKZvIbgGlJTwCPAw9HxCSwG9haH92M\nA7ua3dkz+TIZmyBnl5vKuKlc1q4qtJzJR8S3gC1N9p8Cbu5GlJmZVcOfXWNmlpA/u8bMzFryZ9ck\nkbEJcna5qYybymXtqoKv5M3MBphn8mZmCXkmb2ZmLXkmn0TGJsjZ5aYybiqXtasKvpI3Mxtgnsmb\nmSXkmbyZmbXkmXwSGZsgZ5ebyripXNauKvhK3sxsgHkmb2aWkGfyZmbWkmfySWRsgpxdbirjpnJZ\nu6rgK3kzswHmmbyZWUKeyZuZWUueySeRsQlydrmpjJvKZe2qgq/kzcwGmGfyZmYJeSZvZmYteSaf\nRMYmyNnlpjJuKpe1qwrFJ3lJqyQdkLS3vj0kaVLSEUn7Ja3vXqaZmS1H8Uxe0ieA9wJXR8StknYD\nz0XEvZLuAYYiYsfC+3kmb2bWvp7O5CW9Hfgw8GcNu7cBe+q39wDbO40xM7NqlY5r/gS4G2i87N8Q\nEbMAETEDDDe7o2fyZTI2Qc4uN5VxU7msXVVY3eoAST8PzEbEQUk3LXFo07nP1NQUx45PsmZohPu/\n82Y2bbyGzZs3MzY2Bry6uL3crtVqfX38ZtvzsvTMb9dqtVQ9/v6t7O2Mz6dG/eyZnp5mYmICgNHR\nUYaHhxkfH6dTLWfykv4I+A3gZeCHgDcCnwd+ErgpImYljQCPRcR1C+/vmbyZWft6NpOPiN+LiNGI\neCfwK8AXIuI3gYeBO+qH3Q481GmMmZlVq5PXye8Ctko6AozXt1/HM/kyGZsgZ5ebyripXNauKrSc\nyTeKiClgqn77FHBzN6LMzKwa/uwaM7OE/Nk1ZmbWkj+7JomMTZCzy01l3FQua1cVfCVvZjbAPJM3\nM0vIM3kzM2vJM/kkMjZBzi43lXFTuaxdVfCVvJnZAPNM3swsIc/kzcysJc/kk8jYBDm73FTGTeWy\ndlXBV/JmZgPMM3kzs4Q8kzczs5Z6PJMPDh0/y6HjZzlx5qVuP/SiMs7fMjZBzi43lXFTuaxdVejp\nlfz3z7/M3fuOcve+o5w8d6GXD21m9q9ST2fyO7dey85HvgXAH394E9dvfGNXH9vMbKXyTN7MzFry\n6+STyNgEObvcVMZN5bJ2VcFX8mZmA8wzeTOzhDyTNzOzljyTTyJjE+TsclMZN5XL2lWFlid5SWsk\nfVHSE5Jqkj5Z3z8kaVLSEUn7Ja3vfq6ZmbWjaCYvaW1EvCjpCuAfgY8DvwQ8FxH3SroHGIqIHQvv\n65m8mVn7ejqTj4gX6zfXAKuBALYBe+r79wDbO40xM7NqFZ3kJa2S9AQwAzwSEV8GNkTELEBEzADD\nze7rmXyZjE2Qs8tNZdxULmtXFVaXHBQRrwA3SLoa+Lyk9zB3Nf+aw5rdd2pqimPHJ1kzNMIDT72J\nmeMXWbtxE7AJeHVxx8bGerZdq9V6+ngl2/Oy9Mxv12q1VD3+/q3s7YzPp0b97JmenmZiYgKA0dFR\nhoeHGR8fp1Ntv05e0u8DLwJ3AjdFxKykEeCxiLhu4fGeyZuZta9nM3lJ18y/ckbSDwFbgcPAXuCO\n+mG3Aw91GmNmZtUqmcm/FXhM0kHgi8D+iNgH7Aa2SjoCjAO7mt3ZM/kyGZsgZ5ebyripXNauKrSc\nyUdEDbixyf5TwM3diDIzs2r4s2vMzBLyZ9eYmVlL/uyaJDI2Qc4uN5VxU7msXVXwlbyZ2QDzTN7M\nLCHP5M3MrCXP5JPI2AQ5u9xUxk3lsnZVwVfyZmYDzDN5M7OEPJM3M7OWPJNPImMT5OxyUxk3lcva\nVQVfyZuZDTDP5M3MEvJM3szMWvJMPomMTZCzy01l3FQua1cVfCVvZjbAPJM3M0vIM3kzM2vJM/kk\nMjZBzi43lXFTuaxdVfCVvJnZAPNM3swsIc/kzcysJc/kk8jYBDm73FTGTeWydlWh5Ule0tslfUHS\nk5Jqkj5e3z8kaVLSEUn7Ja3vfq6ZmbWj5Uxe0ggwEhEHJV0FfBXYBnwUeC4i7pV0DzAUETsW3t8z\neTOz9vVsJh8RMxFxsH77HHAYeDtzJ/o99cP2ANs7jTEzs2q1NZOX9CPAFuBxYENEzMLcbwTAcLP7\neCZfJmMT5OxyUxk3lcvaVYXVpQfWRzUPAndFxDlJC+c8Tec+U1NTHDs+yZqhER546k3MHL/I2o2b\ngE3Aq4s7NjbWs+1ardbTxyvZnpelZ367Vqul6vH3b2VvZ3w+Nepnz/T0NBMTEwCMjo4yPDzM+Pg4\nnSp6nbyk1cDfAn8fEZ+q7zsM3BQRs/W5/WMRcd3C+3omb2bWvl6/Tv7Pgf83f4Kv2wvcUb99O/BQ\npzFmZlatkpdQfhD4deDfS3pC0gFJtwC7ga2SjgDjwK5m9+90Jn/izEscOn6WQ8fPcuLMSx39WvMy\nzt8yNkHOLjeVcVO5rF1VaDmTj4h/BK5Y5Ms3V5vzeifPXeDufUeBuRHPW69e0+2HNDMbGF1/x+uW\nLVu6/RBtm/+hRyYZmyBnl5vKuKlc1q4qFL+6pttOnHmJk+cuADB81ZW+Yjczq0Caz66ZH8vcve/o\n5ZN9t2Scv2VsgpxdbirjpnJZu6rgT6E0MxtgnsknkbEJcna5qYybymXtqoKv5M3MBliamXwvZZy/\nZWyCnF1uKuOmclm7quAreTOzAeaZfBIZmyBnl5vKuKlc1q4q+ErezGyAeSafRMYmyNnlpjJuKpe1\nqwq+kjczG2CeySeRsQlydrmpjJvKZe2qgq/kzcwGWN9m8les4vLnxB86fpYLl15p+rXG/VXJOH/L\n2AQ5u9xUxk3lsnZVoW+fQnn6/CX+4B++dXn7kzdf2/RrjfvNzKw9nsknkbEJcna5qYybymXtqoJn\n8mZmA8yvk08iYxPk7HJTGTeVy9pVBV/Jm5kNMM/kk8jYBDm73FTGTeWydlXBV/JmZgPMM/kkMjZB\nzi43lXFTuaxdVWh5kpd0v6RZSV9r2DckaVLSEUn7Ja3vbqaZmS1HyZX8Z4CfW7BvB/APEfGjwBeA\n/7rYnT2TL5OxCXJ2uamMm8pl7apCy5N8REwDzy/YvQ3YU7+9B9hecZeZmVVguTP54YiYBYiIGWB4\nsQM9ky+TsQlydrmpjJvKZe2qQlWfXROLfWFqaopjxydZMzTCA0+9iZnjF1m7cRPUP5PmzFP13wQa\ntmtvmgU2AFD7yuOceeoEV79rbuwz/82Y/+PVcrZrtVpH9+/G9rwsPfPbtVotVY+/fyt7O+PzqVE/\ne6anp5mYmABgdHSU4eFhxsfH6ZQiFj0/v3qQ9A7g4Yj4ifr2YeCmiJiVNAI8FhHXNbvvo48+GjsO\nCICdW69l5yOvfvDYwg8oa/xQsma3//jDm7h+4xuX919qZraCHDhwgPHxcXX665SOa1T/Z95e4I76\n7duBhzoNMTOz6pW8hHIC+Cfg3ZKelvRRYBewVdIRYLy+3ZRn8mUyNkHOLjeVcVO5rF1VaDmTj4hf\nW+RLN1fcYmZmFfNn1ySRsQlydrmpjJvKZe2qQt/+ZqhOnTjzEifPXQBg+KoreevVa/pcZGaWz4r9\n7JqT5y5w976j3L3v6OWTfamM87eMTZCzy01l3FQua1cV/CmUZmYDrOvjmi1btvC5A9X8WlesgkPH\nzwJw4dIrTfeXjG4a529Zxj5ZZ4IZu9xUxk3lsnZVYUVdyZ8+f+nyiObCpWi6v93RTSdjHzOz7Fbs\nTL4TGedvGZsgZ5ebyripXNauKqyoK3kzM2uPXyefRMYmyNnlpjJuKpe1qwq+kjczG2CeySeRsQly\ndrmpjJvKZe2qwop9x2sGWV5+aWa2GM/kO1Dlyy+zzgQzdrmpjJvKZe2qgmfyZmYDbOBm8vPvfj10\n/CxPfe/Fy7dPnHnp8jEP7X/s8v7Gd842OnHmpab37ZasM8GMXW4q46ZyWbuqMHAz+dPnLy36VwfO\nz8y//4OL3L3v6OVjmpkfxSy8r5nZSvKvcia/5X0/0++E18k6E8zY5aYybiqXtasKA3cl34nFPgCt\nl/yKHTOr0sDN5Esc/NI/N92/2Aeg9cL8TDDbB6ZlnFW6qYybymXtqoJfXWNmNsA8k29T46tuqhzp\nLHcm2O1XAWWcVbqpTMbnVMZ1grxdVfCVfJsaxym9Hum06skw3rGVz8+pwdLRSV7SLZK+Lukbku5p\ndsxKmsn3U9aZYMYuN5VxU7msXVVY9kle0irgfwA/B7wH+FVJP7bwuKNHjy6/rkuOfv3Jfie8Tq1W\n63dCUxm73FTGTeUydlV1gdzJSyjfB3wzIr4DIOlzwDbg640HvfDCCx08RHecO3sWrik/vt2XVi72\nMsil9n/ju//S8h24rV5a2di51HHt/LqnT59+zTFXXXkF5y5cKv5va/dxS/Y/e/JUW+vSC6dPn+7L\n4y5lJTQ1fv+gf9/Dxq4sz6lDhw5V8ut0cpJ/G/BMw/Z3mTvxD5yF76JtZbF3yy61f/Kbp3hy39GO\n3oHb2LnUce3+uo3HLPYu4nbfIbycNZrf/576bzLLeVzLpfH7Bzm+h4P2nOr6m6FmZmb4rTvfNvdg\nUrcfrsjMs89A63N1T730/Ey/E5p6+umn+53wOjPPPtP6oB7LuE5uKpe1qwqKWN4rRCT9NLAzIm6p\nb+8AIiJ2Nx73sY99LBpHNtdff33fX1Z58ODBvjcslLEJcna5qYybymXoOnjw4GtGNOvWreO+++7r\n+Mq4k5P8FcARYBw4AXwJ+NWIONxplJmZVWPZ45qIuCTpd4BJ5l6lc79P8GZmuSz7St7MzPLr5HXy\n90ualfS1JY7575K+KemgpC0N+1u+iapHTTc07P+2pEOSnpD0pV41SfpRSf8k6byk313wtb6sU4um\nrqxTYdev1R/7kKRpST/R8LV+rdVSTf16Tt3a+LiSPtjwtX6t01JNfVmnhuN+StJFSb/YsK8r61RB\nV/trFRHL+gcYA7YAX1vk6x8C/q5++/3A4/Xbq4CjwDuANwAHgR9bbkcVTfXtY8BQFR1tNl0DvBf4\nQ+B3G/b3c52aNnVznQq7fhpYX799S5LnVNOmPj+n1jbc3gwcTrBOTZv6uU4Na/Io8LfAL3Z7nTrp\nWu5aLftKPiKmgeeXOGQb8Nn6sV8E1kvaQMObqCLiIjD/JqqOddAEILrwWT6tmiLiexHxVeDlBV/q\n2zot0QRdWqfCrscjYv5dK48z914N6O9aLdYE/XtOvdiweRUw/w67fq7TYk3Qp3Wq+y/Ag8DJhn1d\nW6cOu2AZa9XNDyhr9mapty2xvxcWPvazDY8dwCOSvizpP/WoZyn9XKelZFmnO4G/r9/OslaNTdDH\ntZK0XdJh4GHgP9Z393WdFmmCPq2TpI3A9oi4j7mT57x+r9NiXbCMterl3wyV451Qi/tgRJyQ9Bbm\nFvFw/Xdce62+r5Okfwd8lLk/9qawSFPf1ioi/gb4G0ljwH8DtvbicZeyRFO/1ulPgUrn7RVZ2NV4\n7mx7rbp5kn8W+DcN22+v77sSGG2yvxcWayIiTtT//S+SPs/cH9n6eZJ/lv6t06L6vU71H2x+Grgl\nIub/yNvXtVqkqe9rVX/saUnvlPTDJHlONTZFxKk+rtNPAp+TJOZ+DvUhSS/T/3Vq1nUxIvYuZ606\nHdeIxa/Q9wK3weV3x34/ImaBLwObJL1D0pXAr9SPrUrbTZLWSrqqvn8d8LPA/+1R08Lj5vVznZo2\n9WCdluySNAr8FfCbEfFUw5f6tlaLNfXzOSXpXQ23bwSujIhT9Hedmjb1c50i4p31f65lbv792xGx\nl+6v07K6lrtWy76SlzQB3AS8WdLTwCeZu0qPiPh0ROyT9GFJR4EXmPujLNHFN1EttwnYAHxeUjC3\nJg9ExGQvmuo/+P0K8EbgFUl3AT8eEef6tU6LNQFvoUvrVNIF/D7ww8D/rF/lXIyI9/XzObVYE318\nTgG/JOk24ALwA+CXob//7y3WRH/XqdHlNwx1c5066WKZa+U3Q5mZDTD/9X9mZgPMJ3kzswHmk7yZ\n2QDzSd7MbID5JG9mNsB8kjczG2A+yZuZDTCf5M3MBtj/B79t6qFCm6dWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12a3d3b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.hist(np.array(weights[0,:lenh,startd:].flatten()+1), bins=100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "def heat(sample,weights,dark=0.3):\n",
    "    weights = (weights - weights.min())/(weights.max() - weights.min() + 1e-4)\n",
    "    html = ''\n",
    "    fmt = ' <span style=\"background-color: #{0:x}{0:x}ff\">{1}</span>'\n",
    "    for t,w in zip(sample,weights):\n",
    "        c = int(256*((1.-dark)*(1.-w)+dark))\n",
    "        html += fmt.format(c,idx2word[t])\n",
    "    display(HTML(html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       " <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #fafaff\">Representatives</span> <span style=\"background-color: #f4f4ff\">of</span> <span style=\"background-color: #ffffff\">the</span> <span style=\"background-color: #f0f0ff\">groups</span> <span style=\"background-color: #f7f7ff\">depicted</span> <span style=\"background-color: #ffffff\">in</span> <span style=\"background-color: #fdfdff\">The</span> <span style=\"background-color: #f0f0ff\">trainwreck</span> <span style=\"background-color: #f7f7ff\">spoke</span> <span style=\"background-color: #ffffff\">with</span> <span style=\"background-color: #ececff\">BuzzFeed</span> <span style=\"background-color: #c7c7ff\">News</span> <span style=\"background-color: #fefeff\">about</span> <span style=\"background-color: #ffffff\">the</span> <span style=\"background-color: #b9b9ff\">actor</span> <span style=\"background-color: #ffffff\">'s</span> <span style=\"background-color: #5555ff\">Golden</span> <span style=\"background-color: #4c4cff\">Globes</span> <span style=\"background-color: #d6d6ff\">speech</span> <span style=\"background-color: #f9f9ff\">calling</span> <span style=\"background-color: #fefeff\">on</span> <span style=\"background-color: #f8f8ff\">listeners</span> <span style=\"background-color: #fdfdff\">to</span> <span style=\"background-color: #e8e8ff\">``</span> <span style=\"background-color: #f0f0ff\">protect</span> <span style=\"background-color: #f2f2ff\">...</span> <span style=\"background-color: #f1f1ff\">indigenous</span> <span style=\"background-color: #ffffff\">lands</span> <span style=\"background-color: #fcfcff\">.</span> <span style=\"background-color: #f5f5ff\">''</span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heat(sample, weights[0,-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "columns = [idx2word[data[0,i]] for i in range(startd,maxlend)]\n",
    "rows = [idx2word[data[0,i]] for i in range(maxlend+1,maxlend+lenh+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(weights[0,:lenh,startd:],columns=columns,index=rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAccAAAGZCAYAAAD8ePo1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XlcE9f6P/DPBFCRRXAB/eJye6lLK+KCWm0tWpdq3UFE\n1NpWvG4Vra0rKIpVoWrV2qrXevW6oC217oA7ttpaNxRrtXWrtqgoLqissiW/P/gxl0ASZmISkvB5\n+8rrRcLJ5AlO8sw585wzgkqlUoGIiIhEiooOgIiIyNwwORIREZXC5EhERFQKkyMREVEpTI5ERESl\nMDkSERGVYmvsF8i686es9rmPHstq7/jSPyW3zXv2VNa2BRt5xw52TjVktYfMWTQFWRmy2gu28v57\nbapVl9W+IDNdVvuMm8my2rt6e8lqL8eva2JltW85rp+s9veOnpTVfsnyI7LaR/53rKz2VWvWlNVe\nsJG376iUhfK2r7CR3LYw97m8bQuCrPYQ5H3Olbk5strbOjrLai/3eyE/45ms9g71PWW1l8O7UWe9\nn3vx72MGjOTFGT05EhFR5SD7wMSMcViViIgswvHjx9GrVy/07NkTa9eu1dru4sWLaN68OQ4dOiT7\nucWYHImIyCAEQaH3rTxKpRLz58/H+vXrERcXh/j4ePz5Z9nTdkqlEkuXLkWnTp1kP7ckJkciIjJ7\nFy9eRKNGjeDh4QE7Ozv06dMHCQkJZdpFR0ejZ8+eqFniPLvU55bE5EhERAahgKD3rTypqamoV6+e\neN/d3R0PHjwo0+bIkSMYNmyY7OeWxoIcIiIyiIouyImMjMS0adMMsi0mRyIiMgiFzGkxcri7uyMl\nJUW8n5qaCjc3N7U2ly5dwscffwyVSoUnT57g+PHjsLGxkfTc0pgciYjI7LVo0QLJycm4e/cu6tSp\ng/j4eCxbtkytTcnziKGhoXjrrbfQrVs3FBYWlvvc0pgciYjIIIw5rGpjY4Pw8HAEBwdDpVIhICAA\nnp6eiImJgSAIGDJkiOzn6iIpOSYnJ6Nhw4by3gkREZEB+fr6wtfXV+2xoKAgjW2joqLKfa4ukpJj\nWFgY7t+/jxYtWqBt27Zo27YtmjZtKvlFiIjI+gkSqk4thaTkuGXLFuTl5eG3337DmTNnMHbsWGRn\nZ+PMmTPGjo+IiCyEMQtyTE1SckxMTMS5c+eQmJiIjIwMdOnSBW3btjV2bEREZEEqeiqHIUlKju+9\n9x6aN2+OsWPHwtfXF1WqVDF2XERERBVGUnI8deoUzp8/j7Nnz2Lz5s1QKBRo1aoVJk+ebOz4iIjI\nQigqW8/R2dkZDRo0wL1793D//n0kJSWhoKDA2LERERFVCEnJsVu3bvjnP/8JHx8fDB06FFFRURxa\nJSIiNYIVLdctKTkePnwYCoX1vGkiIjI8ayrIkZTxHjx4gAkTJqBjx47o2LEjJk6ciPv37xs7NiIi\nsiAKQdD7Zm4kJcfQ0FB07doVP/30E3766Se89dZbCA0NNXZsREREFUJSckxLS8OgQYNga2sLW1tb\n+Pv7Iy0tzdixERGRBRFe4J+5kZQcXVxcsGfPHhQWFqKwsBB79uyBi4uLsWMjIiKqEJKSY2RkJPbv\n34833ngDnTp1wsGDB8ss6kpERJWbQlDofTM3kqpVPTw8sGbNGmPHQkREFsyaqlV1Jsf58+frfLOz\nZ882eEBERGSZzLHqVF86k6OXl5f481dffYWJEycaPSAiIqKKpjM5+vn5iT9v2rRJ7T4REVFJ5lh1\nqi/JZ0GtaSyZiIhIF0kFOUREROUxx6pTfelMjq1btxZ7jM+fP0ebNm0AACqVCoIg4Pz588aPkIiI\nLII1jTDqTI5JSUmmioOIiCycNVWrWk8fmIiIyEB4zpGIiAyiUlarEhERVRbsORIRkUFUmoIcIiIi\nqaypIIfJkYiIDILnHImIiKyY8XuOMrvZyvwCIwUCqJSFstoLCus5CjIHCjvzGaiwd65q1O0X5OQb\ndfsWv2+qVNKbyvzcQmEjq7kApbztG5lKZV7xyGFNK+RYzzshIiIyEPM5lCciIovGalUiIqJSWK1K\nRERUCqtViYiIrBh7jkREZBDWNKzKniMREVEp7DkSEZFBsFqViIioFGsaVmVyJCIig6hU1aq3bt3C\n+++/j759+wIArly5gtWrVxs9MCIisiwKQdD7Zm7KTY7h4eGYMmUKbG2LOpnNmjXDvn37jB4YERFR\nRSk3Oebk5MDb21vtMRsbeQv7EhERWZJyzzm6uroiOTlZrEI6cOAA6tSpY/TAiIjIslSqatW5c+ci\nPDwcN2/exJtvvon69etjyZIlpoiNiIgsiDmeO9RXucmxQYMG2LhxI7Kzs6FUKuHo6GiKuIiIyMJY\nU7VquckxLy8PBw8exN27d1FQ8L8LEYeEhBg1MCIisiyVquc4fvx4ODk5oXnz5qhSpYopYiIiIqpQ\n5SbH1NRUrF+/3hSxEBERmYVyp3K0bt0aV69eNUUsRERkwQRB0PtmbrT2HPv27QtBEFBYWIidO3ei\nfv36asOqsbGxJgmQiIgsQ6U455iamordu3ebMhYiIrJg5tgD1JfW5Fi/fn14eHiYMhYiIrJgxp7K\ncfz4cURGRkKlUmHQoEEYM2aM2u8TEhKwYsUKKBQK2NraIjQ0FD4+Prh//z6mT5+Ox48fQ6FQYPDg\nwXjvvfd0vpbW5Pj48WNs2LBB6xNHjhwp820RERHpR6lUYv78+di4cSPc3NwQEBCAbt26wdPTU2zz\n+uuvo1u3bgCAq1evYvLkydi/fz9sbGwQGhqKV155BVlZWfD398cbb7yh9tzStCZHpVKJrKwsA741\nIiIi/Vy8eBGNGjUSRzT79OmDhIQEtQRnb28v/pydnQ2FoqjmtE6dOuKypw4ODvD09MSDBw/0S451\n6tThRH8iIpJMYcRR1dTUVNSrV0+87+7ujt9++61MuyNHjmDp0qVIS0vD2rVry/z+zp07uHLlSpkL\napSmdSqHSqWSEzcREVVy5jCVo3v37ti/fz9WrVqFL774Qu13WVlZmDRpEsLCwuDg4KBzO1p7jhs3\nbjRIoEREVDkYcyqHu7s7UlJSxPupqalwc3PT2r5t27a4ffs2nj59ChcXFxQUFGDSpEkYMGAAunfv\nXu7rae05uri4yAydiIgqM2P2HFu0aIHk5GTcvXsXeXl5iI+PF4tviiUnJ4s/X758Gfn5+WIuCwsL\nw8svv4z3339f0nspd/k4IiKiimZjY4Pw8HAEBwdDpVIhICAAnp6eiImJgSAIGDJkCA4ePIg9e/bA\nzs4OVatWFYdVz507h9jYWDRp0gQDBw6EIAj4+OOP4evrq/X1mByJiMgi+Pr6lkloQUFB4s+jR4/G\n6NGjyzzPx8cHf/zxh6zXYnIkIiKDUFSm6zkSERFJUSmWjyMiIpKjUiw8TkREJIcV5cbyr+dIRERU\n2TA5EhERlWL0YVVFlaqy2ldz177igUZy+vEyl8RTVK0mLxYjK8zNldXe1ta4/70qpVJW++cPn8lq\nX+MVWc1lsa9h3P/bZ/czZLW/l/FUVntlfr6s9sYe7xIEmcfZMuIRFDbyNi27vby/jaKqffmNXoDc\n+G2qGTceOXjOkYiIqBRjX8/RlJgciYjIIDiVg4iIqJRKN6yalpaGbdu24e7duygoKBAfj4qKMlpg\nREREFUVScvzwww/h4+ODjh07wsZG3sliIiKqHKyo4ygtOebk5GDatGnGjoWIiMgsSKq/7tKlC44d\nO2bsWIiIyIIpBEHvm7mR1HPcvHkzvv76a9jZ2cHOzg4qlQqCIOD8+fPGjo+IiCxEpZvKkZSUZOw4\niIjIwpljD1BfkoZVVSoV9uzZg1WrVgEA7t27h4sXLxo1MCIioooiKTlGRETgwoULiIuLAwBUr14d\n8+bNM2pgRERkWQRB/5u5kZQcL168iLlz56Jq1aJ1UmvUqIF8uWs7EhERWQhJ5xxtbW1RWFgoLg2U\nlpYGhYIX9CAiov+pdMvHjRgxAhMmTMDjx4+xfPlyHDhwAJMnTzZ2bEREZEGsqSBHUnLs378/mjdv\njlOnTkGlUmH16tXw9PQ0dmxERGRBrCg36k6OT5/+7xpztWrVQp8+fdR+5+LiYrzIiIiIKojO5Ojv\n7w9BEKBSqXDv3j04OzsDANLT01GvXj0cPXrUJEESEZH5qzTDqsXJb/bs2ejRowc6d+4MADh27BgS\nEhKMHx0REVEFkFRy+uuvv4qJEQA6d+7MVXOIiEiN8AL/zI2kghw3NzesXr0a/fv3BwDExsbCzc3N\nqIEREZFlsaapHJJ6jkuXLkVaWhpCQkIQEhKCx48fY+nSpcaOjYiILIhC0P9mbiT1HF1cXDB79mxj\nx0JERGQWdCbHcePG6XzymjVrDBoMERFZLmsaVtWZHIODg00VBxERkdnQmRzbt28v/pyXl4e//voL\nAPDSSy/Bzs7OqIEREZFlqTQ9x2KnT5/GzJkz4eHhIS4IsGjRIrRr187Y8RERkYUwx8IafUlKjosW\nLcL69evxz3/+EwBw69YtTJkyBTt37jRqcEREZDmsqecoaSpHfn6+mBiBomFVXs+RiIislaSeo5eX\nF2bNmiUuArB37154eXkZNTAiIrIsVtRxlJYc582bh61bt2LLli0AAB8fHwwbNsyogREREVUUncOq\nR44cwdatW1GlShWMHDkS9+7dw+XLlxEdHc0rchARkRqFIOh9Mzc6k+O6devQtWtX8X5+fj527tyJ\n6OhofPvtt0YPjoiILEelWXg8Pz8f9erVE+/7+PjAxcUFLi4uyMnJMXpwRERkOcywA6g3nT3H9PR0\ntftz5swRf05LSzNORERERBVMZ3L09vbGtm3byjweExMDb29vowVFRESWx5rOOeocVg0LC8OECRMQ\nGxuL5s2bAwAuX76MvLw8rFq1yiQBEhERmZrO5FirVi3ExMTg5MmTuHHjBgCgc+fO6Nixo0mCIyIi\ny2FNK+RImufYsWNHvROiwlbSS4hyZJ7LrOLiKj0WC18sXZmbJ+8Jjk7GCeT/U1SpIqt9FRcHI0Ui\nX42X3Y26fffmdWW1v/74lqz2T3//W1Z7NxcXWe1tqlWX1V6lUspqL0hbnAuA/O+QwrxcWe1tqlSV\n1V6lLJTVHrDs7x05rCg3SkuORERE5bGmnqP0wzciIqJKgj1HIiIyCGu6ZBV7jkRERKVISo779+9H\nZmYmAGD16tUICQnB5cuXjRoYERFZFkEQ9L6ZG0nJcfXq1XB0dERiYiJOnjyJgIAAREREGDk0IiKy\nJIKg/83cSEqONjY2AIBjx44hMDAQXbp04cWOiYhIjbFXyDl+/Dh69eqFnj17Yu3atWV+Hxsbi/79\n+6N///4YOnQorl69qvZ7pVIJPz8/jBs3rvz3IiUgd3d3zJkzB/v27UPnzp2Rl5cHpVLevCYiIiJ9\nKZVKzJ8/H+vXr0dcXBzi4+Px559/qrVp0KABtm7dir1792L8+PEIDw9X+/3mzZvh6ekp6fUkJccv\nvvgCnTp1wvr16+Hs7IynT59i+vTpEt8SERFVBsY853jx4kU0atQIHh4esLOzQ58+fZCQkKDWplWr\nVnBychJ/Tk1NFX93//59HDt2DIMHD5b0XiQlR3t7e9SsWRPnzp0DANja2qJRo0aSXoCIiOhFpaam\nql1C0d3dHQ8ePNDa/vvvv4evr694PzIyEtOnT5dc/CMpOa5cuRLr1q0Tx3jz8/Mxbdo0SS9ARESV\ng7kU5Jw6dQo7d+7E1KlTAQA//vgjateujVdeeQUqlUrSNiQtAnD48GHs3r0bfn5+AIoydlZWlp5h\nExGRNTLmlAx3d3ekpKSI91NTU+Hm5lam3ZUrVzBnzhysW7cONWrUAACcP38eR48exbFjx5Cbm4us\nrCxMnz4dixcv1vp6knqOdnZ2auPC2dnZst4UERFZP2P2HFu0aIHk5GTcvXsXeXl5iI+PR7du3dTa\npKSkYNKkSVi8eDEaNmwoPv7JJ5/gxx9/REJCApYtW4bXXntNZ2IEJPYc33nnHcyZMwfp6enYtm0b\nduzYgcDAQClPJSIiemE2NjYIDw9HcHAwVCoVAgIC4OnpiZiYGAiCgCFDhmD16tV49uwZ5s2bB5VK\nBVtbW2zfvl2v15OUHEeNGoUTJ07AwcEBt27dwqRJk/DGG2/o9YJERGSdpM5X1Jevr69akQ0ABAUF\niT8vWLAACxYs0LmN9u3bo3379uW+luSFx5s0aYLnz59DEAQ0adJE6tOIiIgsjqRzjt9//z0GDx6M\nw4cP4+DBgxgyZIjeXVUiIrJO5lKtagiSeo7r1q3Drl274OrqCgB48uQJgoKCEBAQYNTgiIjIcpjj\nAuL6kpQcXV1d4eDgIN53cHAQEyURERFgnj1AfelMjhs2bAAANGzYEIGBgejWrRsEQUBCQgKaNm1q\nkgCJiIhMTWdyLJ7o37BhQ7U5I6XnlhAREVWaYdWQkBC1+8XJsuQQKxERkbWRdM7x2rVrmD59Op49\newag6BzkokWL0LhxY6MGR0RElsOKOo7SkuOcOXMwc+ZMdOjQAQBw+vRphIeHIyYmxqjBERGR5TD2\nIgCmJCk5Zmdni4kRAF577TWur0pERGqsKDdKS44NGjTAqlWrMGDAAADA3r170aBBA6MGRkREVFEk\nrZATGRmJJ0+eYOLEiZg4cSLS0tIQGRlp7NiIiMiCFF+9SZ+buZHUc6xRowZmz55t7FiIiIjMgs7k\nOG7cOJ1PXrNmjUGDISIiy2WGHUC96UyOwcHBZR4r7v6qVCrjRERERBbJHIdH9aUzOaanpyM1NRXD\nhw8HAAQEBCAtLQ2CIGDq1KkmCZCIiCyDFeVG3QU569atQ9euXcX7+fn52LFjB6KjoznHkYiIrJbO\nnmN+fj7q1asn3vfx8YGrqytcXV2Rk5Nj9OCIiMhyWNOwqs6eY3p6utr9OXPmiD+npaUZJyIiIqIK\npjM5ent7Y9u2bWUej4mJgbe3t9GCIiIiyyMI+t/Mjc5h1bCwMEyYMAGxsbFo3rw5AODy5cvIy8vD\nqlWrTBIgERFZBmsaVtWZHGvVqoWYmBicPHkSN27cAAB07twZHTt2NElwRERkOawoN0pbIadjx45M\niEREVGlISo6mlJ8h72ofyvx8yW0Lc/Nkbbvw+XNZ7RW168hqD5kLKdi5ushq//z+fVntq9evL6u9\nqqBAVnu571fO/61gI2mZYJHCTt6ur1IWympv51BNVvvYFdNktf/rfIqs9s6e9cpvVILcfaEwR97n\nVlGlqvRtPzduZby8/1n5VIXyPieqQnkRFebK+54ypkp3ySoiIqLyWFFulHZVDiIiosqEPUciIjKI\nSlOtSkREJJUV5UYOqxIREZXGniMRERmEoLCeriOTIxERGQSHVYmIiKwYe45ERGQQrFYlIiIqxYpy\nI4dViYiIStPZc5w/f77ObvLs2bMNHhAREVmmSjOs6uXlZao4iIjIwllRbtSdHP38/EwVBxERkdnQ\nmRzHjRun88lr1qwxaDBERGTBrKjrqDM5BgcHmyoOIiKycJXmnGP79u1NFQcREZHZ0JkcP/roI6xY\nsQL9+vXT+PvY2FijBEVERJbHijqOupPjq6++iosXL2LlypWwteV6AUREpF2lWXj86dOniIyMxM2b\nN9GkSRO0adMGrVu3RuvWreHi4mKqGImIiExKZ3KcMWMGACAvLw+XLl1CUlISdu7cifDwcDg7O2Pf\nvn0mCZKIiMxfpRlWLZabm4vMzExkZGQgIyMDbm5uaNq0qbFjIyIiC1JpqlXDw8Nx/fp1ODg4oGXL\nlmjdujVGjhyJGjVqmCo+IiIik9O58HhKSgry8vJQp04duLu7o27dunB2djZVbEREZEEEQf+budHZ\nc1y/fj1UKhWuX7+OpKQkbNiwAdeuXYOLiwtatWqFSZMmmSpOIiIyc5VmWBUoerNNmjSBs7MznJyc\n4OjoiB9//BEXL15kciQiIqukMzlu3rwZSUlJSEpKgq2trTiNIyAgAE2aNDFVjEREZAGsqOOoOzne\nvXsXvXr1QmhoKNzc3EwVExERWSBjD6seP34ckZGRUKlUGDRoEMaMGaP2+5s3byIsLAyXL1/GJ598\ngpEjR4q/y8jIwKxZs3D9+nUoFApERkaiZcuWWl9LZ3IMDQ19wbdCRET04pRKJebPn4+NGzfCzc0N\nAQEB6NatGzw9PcU2Li4umD17No4cOVLm+QsXLkTnzp3x5ZdfoqCgAM+fP9f5ejqrVYmIiCRTvMCt\nHBcvXkSjRo3g4eEBOzs79OnTBwkJCWptatasCS8vrzLLnWZmZiIxMRGDBg0CANja2sLR0bHct0JE\nRPTCBEHQ+1ae1NRU1KtXT7zv7u6OBw8eSIrrzp07cHV1RWhoKPz8/BAeHs6eIxERVW4FBQX4/fff\nMWzYMOzatQvVqlXD2rVrdT6HyZGIiAzCmIsAuLu7IyUlRbyfmpoquVC0bt26qFu3Llq0aAEA6Nmz\nJ37//Xedz2FyJCIigzDmsGqLFi2QnJyMu3fvIi8vD/Hx8ejWrZvW9iqVSvy5du3aqFevHm7dugUA\nOHXqlFohjya8SCMREZk9GxsbhIeHIzg4GCqVCgEBAfD09ERMTAwEQcCQIUPw6NEjDBo0CFlZWVAo\nFNi8eTPi4+Ph4OCA2bNnY+rUqSgoKECDBg0QFRWl8/WYHImIyCCMvQiAr68vfH191R4LCgoSf65d\nuzaOHTum8bnNmjXDjh07JL8WkyMRERmGFS2RY3bJ0c6putG2rSoskNVeUaWKzBdQld+mZHOlvPaQ\n2d7O2Une9mVS5ufLam/M/1u5f3tjy0x5Iqt94XN5+2aTLrrPl5Rm6+ggq73cfVPuviDYSv/qyX30\nWNa2barby2ovl8LOTlZ7mypVZbVXyvyekvu3J2nMLjkSEZFlEhTsORIREamxolFVTuUgIiIqjT1H\nIiIyiEp1sWMiIiIprCg3cliViIioNPYciYjIMKyo6ygrOebk5MDe3rhziIiIyDJZ01QOScOq58+f\nR+/evfHOO+8AAK5cuYKIiAhjxkVERFRhJCXHqKgorF+/Hi4uLgCK1qhLTEw0amBERGRZjHnJKlOT\nPKxa8grMAKBQsJaHiIhKMMcspydJybFevXo4f/48BEFAfn4+Nm/eXO61sIiIiCyVpO5fREQEtm7d\nitTUVPj6+uKPP/7AnDlzjB0bERFZkEo3rKpQKLB06VK1x27fvg1XV1ejBEVERJan0lWrjhs3DpmZ\nmeL9P//8E+PHjzdaUERERBVJcnIcN24csrKycOnSJUyaNAlLliwxdmxERGRBBEHQ+2ZuJA2rdunS\nBQUFBRg1ahSysrKwcuVKvPTSS8aOjYiILIn55Ti96UyO8+fPV8voGRkZaNCgAbZu3QoAmD17tnGj\nIyIiqgA6k6OXl5fa/ebNmxs1GCIislzmODyqL53J0c/Pz1RxEBGRhas0ybHYX3/9hWXLluHGjRvI\nzc0VH09ISDBaYERERBVFUrVqaGgohg4dChsbG2zevBkDBw5E//79jR0bERFZEsUL3MyMpJByc3PR\nsWNHAICHhwcmTpyIY8eOGTUwIiKyLJVuKkeVKlWgVCrRqFEjbNmyBe7u7sjKyjJ2bERERBVCUs8x\nLCwMOTk5mD17Ni5fvow9e/Zg0aJFxo6NiIgsSKXrOXp7ewMAHBwcEBUVZdSAiIjIQplfjtObzuQ4\nbtw4nU9es2aNQYMhIiIyBzqT44ULF1CvXj306dMHLVu2hEqlMlVcRERkYazpqhw6k+OJEydw4sQJ\nxMfHIy4uDp07d0bfvn3RuHFjU8VHRESWwgzPHepLZ0GOjY0NfH19sWjRImzbtg2NGjXCiBEjsGXL\nFlPFR0REZHLlFuTk5eXhxx9/RFxcHO7evYsRI0agR48epoiNiIgsiBV1HHUnx+nTp+P69evw9fVF\nSEgImjRpYqq4iIjIwpjjlAx96UyOe/fuhb29Pf766y9ER0eLj6tUKgiCgPPnzxs9QCIiIlPTmRyv\nXLliqjiIiMjSVZZqVSIiIqmsaVjVDNdCJyIiqljsORIRkWFYT8fR+MlRpZS3qk41tzqy2itspb8F\n2+rVZW1bsLOT115hI6s9UCivuczxfEW1arLay49fHmWBvPcr5/9WrvzMHFnt5f5trp6+K6u9k3NV\nWe1reTeU1d7WwUFWe7l/e0Fuexvpny1lfoGsbdvKHNqTu1+qbOR9p8nddxQ28v6WKiN+TuTisCoR\nEZEVM59DDiIismiVZm1VIiIiyaxoWJXJkYiIDILnHImIiKwYe45ERGQY1tNxZM+RiIioNPYciYjI\nIFitSkREVJoVFeQwORIRkUGwWpWIiMiKldtzXLNmjcbHx40bZ/BgiIjIghn5nOPx48cRGRkJlUqF\nQYMGYcyYMWXaLFiwAMePH4e9vT0+++wzvPLKKwCAjRs3Yvv27RAEAU2aNEFUVBSqVKmi9bXK7Tkq\nFArxVlBQgCNHjuDvv/9+gbdHREQkj1KpxPz587F+/XrExcUhPj4ef/75p1qbY8eOITk5GYcOHcKn\nn36KuXPnAgBSU1MRHR2NnTt3IjY2FoWFhdi3b5/O1yu351g6M48ePRr/+te/5L4vIiKycsY853jx\n4kU0atQIHh4eAIA+ffogISEBnp6eYpuEhAQMHDgQANCyZUtkZGTg0aNHAIqSa05ODhQKBZ4/fw43\nNzedryf7nGNeXh7u378v92lERGTthBe4lSM1NRX16tUT77u7u+PBgwdqbR48eIC6deuqtUlNTYW7\nuztGjhyJLl26wNfXF05OTnj99dd1vl65PceBAweKRwOFhYV4+PAhxo4dW/47ISKiSsVcq1XT09OR\nkJCAH374AU5OTpg0aRJiY2PRr18/rc8pNzl++eWX4s82NjaoU6eOzpOYREREhubu7o6UlBTxfmpq\napmhUTc3N7WRzfv378Pd3R2//PILGjRoABcXFwBAjx49kJSUpDM5ljus2rBhQ+Tm5uLEiRM4fvw4\nkpOTZb8pIiKqBBSC/rdytGjRAsnJybh79y7y8vIQHx+Pbt26qbXp1q0bdu/eDQC4cOECnJ2dUbt2\nbfzf//35DEQhAAAgAElEQVQffv31V+Tm5kKlUuHUqVNq5yo1KbfnuGXLFnzzzTfo3r07AOCjjz7C\nsGHDMHz48HLfDBERkSHY2NggPDwcwcHBUKlUCAgIgKenJ2JiYiAIAoYMGYLOnTvj2LFj6NGjB+zt\n7REVFQUA8Pb2Rs+ePTFw4EDY2tri1VdfRWBgoM7XE1QqlUpXg379+iEmJgYODg4AgKysLAQFBSE2\nNlbSG3r+WF7xjjI3R1Z7WwcnyW3zM57J2rZgZycvlmrVZbVXKQtltS/My5XVXi658ec9TZPV/vnD\nx7LaOzduLKu9HE9+uyyrvau3l6z2CZ9+J6u9k3NVWe1fHeAtq719vbrlNyrBpqq9rPb5menyti9j\nX8u8eVPWtqu4OMtqryyQ9zlUyDytVK12HVntlfl5stoXPpf3nelQX3eP6UWk/vSj3s91f7OLweIw\nBEnLx9mVSBJ2MhMGERFVEmZakKOPcpNj//79ERgYiLfffhsAcOTIEXEeCRERUTFzrVbVh9aCnNGj\nR+POnTsYPXo05s2bB3t7e9jb2yMiIgKjRo0yZYxEREQmpbXn6O/vj1GjRmHgwIH417/+hZYtW5oy\nLiIisjSV4XqO77zzDnx9fbF69WoMGjQIAwYMgELxv47myJEjTRIgERFZBmsaVtV5ztHOzg729vbI\ny8tDVlaWWnIkIiKyVlqT4/Hjx/HZZ5+ha9eu2LVrF+zt5ZV2ExFRJVMZeo5r1qzBihUr0NiIc82I\niMh6CJXhnOM333xjyjiIiIjMhqRFAIiIiMpVGYZViYiI5LCmalWWnxIREZXCniMRERmGFfUcmRyJ\niMggrKlalcOqREREpbDnSEREhsFhVSIiolKsKDlyWJWIiKgUs+s55j5Ok9Xe1sFJcltlQYGsbSuM\nfRQkd/uFhTK3b9xjH8HGRlZ7ha289sZkW72aUbdfr6GzrPaPUjJltc/PyJbVvpqbUlZ72VQqo21a\n7n6mLJD3OVHYyfsaNPp+LPdza0a9NWua52h2yZGIiCwUq1WJiIisF3uORERkEIKRT+WYEpMjEREZ\nhhWdc7SeNE9ERGQg7DkSEZFBsFqViIioNFarEhERWS/2HImIyCA4rEpERFSaFSVHDqsSERGVIik5\nRkdH49mzZ8aOhYiILJmg0P9mZiQNqz569AgBAQF49dVXMWjQILz55ptWNbZMREQvTqhs1aoff/wx\nDh06hICAAOzatQtvv/02li1bhuTkZGPHR0REZHKS+7KCIKBOnTqoXbs2bGxs8OzZM0yaNAmLFy82\nZnxERGQpBEH/m5mRNKy6adMm7NmzB66urggICMD06dNhZ2cHpVKJt99+G9OnTzd2nEREZOas6XSb\npOSYnp6Or776Ch4eHmqPKxQKfP3110YJjIiIqKKUO6xaWFiI+Pj4MomxmKenp8GDIiIiC2RF1arl\nRmRjY4OXXnoJKSkppoiHiIgslKAQ9L6ZG8nDqn369IG3tzfs7e3Fx9esWWO0wIiIiCqKzuT4999/\n49GjR/joo4/UHk9MTESdOnWMGhgREVmYylKQExkZiU8++QRNmzZVe7xGjRpYvnw5Bg8ebNTgiIjI\nclhTtarOc46PHj0qkxgBoGnTprh7967RgiIiIqpIOnuOGRkZWn/3/PlzgwdDREQWzAyrTvWl8514\neXlh27ZtZR7//vvv0bx5c6MFRUREFkgh6H8zMzp7jmFhYQgJCUFsbKyYDC9duoT8/HysXLnSJAES\nERGZms7kWLt2bcTExODUqVO4fv06AKBz587o2LGjSYIjIiLLYU0FOZLmOXbo0AEdOnQwdixERGTJ\nKss5RyIiospIUs+RiIioPJVuWJWIiKhcHFYlIiKyXuw5EhGRQZjj1TX0xeRIRESGYUXnHDmsSkRE\nVAp7jkREZBCCFRXkMDkSEZFhWNGwqqBSqVQVHQQREZE5sZ4+MBERkYEwORIREZXC5EhERFQKkyMR\nEVEpTI5ERESlMDkSERGVwuRIRERUCpMjERFRKSZNjvv370dmZiYAYPXq1QgJCcHly5fLtJs2bRoA\nYNOmTZK3/ejRI4SFheFf//oXAODGjRv4/vvvtbbftGkTMjMzoVKpEBYWBj8/P/z8889y3o4k58+f\nR2xsLHbv3i3eSrt16xbef/999O3bFwBw5coVrF69Wud2Nb23zz//XGt7uX8fqbHrKzk5GXl5eQCA\n06dPY/PmzUhPT9fafv/+/ZIeA+S918LCQixatEhu+Dh37hyys7MBAHv27EFUVBTu3r0rezuaSP2c\n6Ovp06dlbvn5+Vrb5+XlITY2FmvWrMHKlSvFmzbZ2dlQKpUAivbthIQEndt/9OgRfvjhB/zwww94\n/Pix/m9MC03fI3K+W8ojd1+W216Thw8f6h8wSWLS5Lh69Wo4OjoiMTERJ0+eREBAACIiIsq0u3z5\nMlJTU7Fjxw48e/aszAdZk5kzZ6JTp0548OABAOAf//gHNm/erDWWHTt2wNHRET///DPS09OxePFi\nLF26tEy7+fPnY8GCBVpvukybNg2LFy/GuXPn8Ntvv+G3337DpUuXyrQLDw/HlClTYGtbtJpfs2bN\nsG/fPp3bPnToEPbu3SvenzdvHtLS0rS2l/v3kRp769at0aZNG603bSZOnAiFQoG///4bc+bMwb17\n9zBlyhSt7deuXSvpMbnv1cbGBufOndP6utpERETA3t4eV65cwYYNG9CwYUPMmDFDY9u0tDSsWbMG\n4eHhCA0NFW/aSP2cFLt16xZmz56N4OBgvPfee+JNG39/f3Ts2BE9e/ZEz5490bFjR3Tt2hV+fn4a\n/4/Hjx+PhIQE2NjYoHr16uJNm3fffRe5ublITU3FqFGjsGfPHsycOVNj23379mHw4ME4cOAA9u/f\nL/6sSWFhodbX1EXTQd2uXbv02pYmcvdlue01mTVrVpnHRowYgffeew+TJk2S/R6oLJOurWpjYwMA\nOHbsGAIDA9GlSxd88cUXZdoFBQXhgw8+wO3bt+Hv7y8+rlKpIAgCEhISyjznyZMn6N27t/iFaWtr\nC4VCe+4vXjXv2LFjGDBgABo3bgxNK+l5eXkBKOpF3bhxA7179wYAHDhwAJ6enjrf76VLl7Bv3z4I\n5aw3mJOTA29vb7XHiv9W2nz11VcYP348FAoFfvrpJzg5OSEyMlJre7l/H6mxJyUlAQC++OIL1KlT\nBwMGDAAA7N27V+fRrUKhgK2tLQ4fPox3330XI0aMwMCBA8u0O3bsGI4fP47U1FS1g5HMzEytfyO5\n7/WVV17BuHHj0KtXL7Uv/bffflvrc2xtbSEIAo4cOYLhw4dj8ODB2L59u8a2H374IXx8fNCxY8dy\n/18B6Z+TYh999BGCgoIQGBio830We/3119GzZ0+8+eabAICff/4Zhw4dgr+/P+bNm1eml52amor1\n69eXu91iKpUK9vb22L59O4YOHYrRo0eL+0Vpa9aswfbt21GrVi0ARQcSH3zwAXr16qXW7saNG1i2\nbFm5IyolxcXFIS4uDnfu3MG4cePEx7OyslCjRg3J2wGKPm8TJ07U+Dup+7K+7TXRdGD42WefASj/\nu4OkMWlydHd3x5w5c3DixAmMHj0aeXl54vBLScVHvnPnzsXQoUNx9uxZAEC7du3QrFkzjduuXr06\nnjx5In6ZX7hwAU5OTlpj8fLyQnBwMO7cuYMpU6YgMzNT4xeLn58fAODbb7/FN998I/bugoKCMHz4\ncJ3vt3Hjxnj48CHc3Nx0tnN1dUVycrIY+4EDB1CnTh2NbUv2nBcsWIAJEyagTZs2CAkJwdOnT+Hi\n4qLxeXL/PlJjL3b06FG1nuywYcPQv39/fPTRRxrb29raIi4uDrt378a///1vAEBBQUGZdu7u7vDy\n8sLRo0fRvHlz8XEHBwetvS+57zUvLw+urq44ffq02uO6kqODgwO+/vprxMbGYsuWLVAqlRrjB4oO\nfopPFUgh9XNSzNbWFsOGDZO8/V9//VXtQKNTp05YtGgRPv30U3G4r6TWrVvj6tWraNq0qaTtq1Qq\nJCUlITY2FgsXLgQArfGrVCoxMQKAi4uLxoPUqKgoLFmyRNLrl4y7Tp06ePLkCYKDg8XHHRwcJL+X\nYiX3vdKk7sv6tpfKw8PjhbdB/2PShcdzcnLw008/oUmTJvjHP/6BBw8e4Nq1a+jUqZPG9ps2bcL2\n7dvRo0cPqFQqHDlyBIGBgRgxYkSZtpcvX8b8+fNx/fp1NG7cGE+ePMGKFSu0JlOlUok//vgDDRo0\ngLOzM548eYLU1FSt7Xv27InvvvtOTD7Pnj1DYGAgDh48qPX9jhgxAleuXIG3tzfs7OzEx9esWaPW\n7vbt2wgPD0dSUhKcnZ1Rv359LFmyBPXr1y+zza5du0IQBLEXXfK/T1uvWs7fp/gIOysrS1LsxYKC\ngjBs2DD06dMHgiAgLi4O33zzDWJiYjS2v3HjBmJiYtCqVSv07dsXt2/fxv79+zFmzBiN7QsKCsQD\nk/LI3Rf08fDhQ8TFxaFFixZo27YtUlJScObMGY09gOXLl6NNmzbo3LmzpG1L/ZwUHyhFR0ejZs2a\n6NGjB6pUqSL+XtuBUnBwMDp06IA+ffoAKBra/OWXX7Bu3ToEBASUGXLs3bs3/v77b9SvX19t+7Gx\nsRq3f/bsWaxfvx5t2rTBmDFjcPv2bWzatAmzZ88u03bx4sW4evWqWixNmzYtczBRWFiod4/o9u3b\ncHNzQ9WqVQEAz58/x6NHjzR+vvQhd1+W216q4u+GmjVrlltPQOUz+VU5EhMT8ffff2PQoEFIS0tD\nVlYWGjRooLFtv3798N1334lDXdnZ2RgyZIjWD2VBQQFu3boFlUqFl156Se1LXZNDhw7h3LlzEAQB\nPj4+6NGjh9a2O3bswMqVK/Haa69BpVLh7NmzmDhxotiz1OTMmTMaH2/fvr3Gx4sLGRwdHXXGrQ+V\nSoXCwkK1v49KpVL7stMVczFtsd+5cwcLFy7E+fPnIQgC2rRpg7CwMK1fQCdPnkTr1q1RrVo1SfEX\nf/BL03YwIGdfuH//PubPn4/z588DANq2bYtZs2ahbt26kmIrT+vWrZGTkwM7OzvY2dmJBzbFr6eJ\nlM9JyQOl0nQdKKWlpWHVqlXiudbikQdHR0fcu3cPjRo1Umt/9+5dpKenIzExEUDRCI6Tk5PGnkph\nYSE+//xzredfS1uyZAlatmwpxtK2bVtcuHBBY09b3wTp7++PmJgYcV/Py8vD0KFDsWPHDsnb+OGH\nH/DWW2/Jfm2yXCZNjitXrsSlS5dw69YtHDx4EKmpqfjoo4+09i769euH7du3i0d8ubm5CAgI0Joc\nz58/j7t376qduNc2lh8REYHk5GS1I9aGDRti7ty5WuN/+PAhfv31VwBAy5YttQ59ypWXl4eDBw/i\n7t27asMrISEhWp+zdetW9OvXD87OzgCKerJxcXFah3pDQ0MRFRUl3s/Ozsb48eO1Vu0tWbKkzBeU\npsf0NWPGDFy4cAE1atSAj48P2rVrBx8fH63ngp48eSL+nJeXh/379+PZs2dah23l7AsjR45E3759\n1c6XxsbGYsOGDWXamuLoXO7nxNjkjOAAQGBgILZt2yZp235+fmV6qv369SvzGdfnnGOxAQMGYM+e\nPWqP9e/fX+00QHm+/PJLrYUuUg/c+vXrp/M1tH2vUcUw6TnHw4cPY/fu3WJvy93dHVlZWVrb+/v7\nY/DgwWKP7siRIxg0aJDGttOmTcPt27fRrFkz8ehSEAStX4inTp3C/v37xZ3az89PLLbRRKVS4Zdf\nfsHt27cREhKClJQUXLx4sUwhDQAMHToU3377LVq3bq32odHWYxg/fjycnJzQvHnzMj05bbZt26aW\nCGvUqIHvv/9ea3J0d3dHREQEIiIi8OzZM4wdOxaDBw/Wuv1ffvmlzGPHjx/Xmhxv3bqFiIgIPH78\nGHFxcbhy5QqOHj2KDz/8UGP74ukTqampOHjwID799FM8ePAAv//+u8b2rq6uavc/+OAD+Pv7a0yO\ncveFtLQ0tf3K399f60HD0aNHNT6ui0qlwt69e3Hnzh1MmDAB9+7dw8OHDzXuO4D8z4ncA6Vbt27h\nv//9b5mDMW0Vvdu3b1cbwRk9ejSGDBmiNTlKKXD65ptv8O233+L27dtqSSMrK0tjlbM+5xyL1axZ\nEwkJCejWrRuAou+R0vtTeYYOHar1dyV7oCUP3EorPiWxdetWAFA7GCuv8I1Mz6TJ0c7ODoIgiDtC\n8TwxbUaOHIn27duLQy5RUVF49dVXNbaVWl1ZrFGjRkhJSRGHhu7du4d//OMfWttHRERAoVDg1KlT\nCAkJgYODAyZOnKhxaObbb78F8L9KzvLIrQYEis6ZFidboGjISddcssmTJ2Px4sWYM2cOLl++jDFj\nxqBnz55l2sn90ioWHh6O6dOnY86cOQCKpqNMnTpVa3Lcs2cPzp07h2vXrsHFxQXDhw9H27ZttW6/\n5Dw/pVKJS5cuaS1ikLsvuLi4YM+ePeI807i4OK3n6/RRct+ZMGECqlevjnnz5mkd1pP7OZF7oFRc\n3Tp48GBJ1a2AegVkeUObUgqc+vXrB19fXyxbtkxtGoODg4PGv/3atWv1Puc4b948TJ06FZ9++ikE\nQUDdunVlz22dPXs2vv76a42/k3rgVvxd88svv6hNL5k2bRr8/PwwdepUWTGRcZk0Ob7zzjuYM2cO\n0tPTsW3bNuzYsQOBgYE6n9O8eXOdlWLF5FZXZmVloXfv3uLR+2+//QYvLy+xIKV04cnFixexa9cu\nsfdRo0YNnclIDrnVgEBRheHkyZMRFBQEAIiJiRFL80s6dOiQ+HPLli2xevVqeHt7QxAEHDp0qExF\nptwvrWJyp6NERkaiYcOGCAoKwmuvvVZucURxmTpQVO3n4eGhdXqD3H0hMjIS8+fPR1RUFARBQOvW\nrdVe70XJ3Xfkfk7kHijJrW6VM4IDQG34XhsnJyc4OTlh2bJlkmJ4kekJDRs2xLZt28Tet4ODg+xt\naEuMgLwDN6BoJOHcuXPw8fEBUHQKQFc1MlUMkybHUaNG4cSJE3BwcMCtW7cwadIkvPHGGy+0zZLV\nlX369JFcXSl3oqytrS0KCwvFL6C0tDTJR93a9O3bF4IgoLCwEDt37pRcDQgUHW3GxMSIvdTXX39d\n4zDpDz/8oHb/1VdfRUFBgfh46eQoCALq168v9gBL0jVVRM50FKBoZZDr16/j7NmzWL58Of7++2+8\n9NJLWofOoqOjtW6rmL77wv3798v87ty5c6hXr165rymF3H1H7udE6oFSsbfeegtbt26VXN0qZwQH\nkD/EbmyPHj3CsmXL8ODBA6xbtw43btxAUlKSztMKcsg5cAOAhQsXIiwsTFyhy9nZWeccZaoYJi3I\n2bBhA3r37g13d3eDbfPMmTNQqVT4/PPP1c6HFT9mqKKJvXv3Yt++ffj999/h5+eHAwcO4KOPPtJ5\nnrI87dq107kkW3nzlp4/f46UlBT885//1DuG0saOHYuvv/5aYyWkrgpIOdNRgKJJ/OfOncPZs2eR\nmJiIJ0+eoFWrVlqHuzIyMrBy5Upxzmv79u0xYcIEtfmL+u4LmopCND2mL037zuTJk/HOO+9ofc7D\nhw9x8eJFCIKAFi1a6DzQUCqViImJwalTpwD870BJW2+ra9euZR7T9X8r17vvvisOsRfv33379kVc\nXJxBti/Xv/71L/j7+2PNmjXYu3cvCgoK4OfnV+EFMBkZGQCgcw4uVRyT9hyzsrIQHByMGjVqoHfv\n3ujVqxdq1679QtssnlpQUFBQZprB8+fPtT6vZLFMfn4+CgoKYG9vr7W8vn///mjevDlOnToFlUqF\n1atXl7tCTnnq16+v98TdhIQELF68GPn5+Th69Cj++OMPrFixQmfvSMp0heLhozZt2qB9+/bw8fGR\n9D4bNGiAjRs3Sp6OMnToUPj4+MDHxwfvvvtuudMmwsLC0LhxY6xYsQJA0TnL0NBQtTU+5e4LSUlJ\nSEpKQlpamlplamZmpt5LlWkid9/5/vvvsWrVKnTo0AEqlQoLFizAhx9+iICAAI3tFQoF/P390aFD\nB0kHSvoUFcmhz4pPxiR3xSS5pBy4laRPdTqZnkmTY0hICEJCQnDlyhXs379f/FLcuHGj3tvUt4Ck\nZLGMSqVCQkICLly4oLX9tGnTsGTJErUvteLH9PX48WON0wWKjRw5UuvvVq1ahe3bt4sVg6+88orO\nha9DQ0PRt29fMbns3bsXoaGhWl8/ICAAiYmJWLBgAZKTk9G8eXP4+Pjg/fff19he7tBV8VF7Tk4O\n7O3ttcZdLDk5GV999ZV4PyQkpMySZHL3hfz8fGRnZ6OwsFCtGtTR0RFffvlluTGVp+RqRrVq1RKn\nDRX/Ttsw5rp167Br1y6x0OPJkycICgrSmhylHiidPHkSHTt2VDsPXZKuFYHkkDvEbmxyV0ySS8qB\nW0n6VKeT6Zk0ORarVasWateuDRcXlxdehV/fApKSBEFA9+7dsXLlSq0VYzdu3FC7X1hY+MJXSlAq\nlTpL9HWxtbWV9QGXM10BADp06IB27drht99+w+nTpxETE4Nr165pTY4zZ84Uh66AosW+P/74Y63J\nMSkpCbNmzUJ2djZ+/PFHXLlyBTExMVoX2K5WrRoSExPFitZz586VWUBA7r7Qvn17tG/fHn5+fkZZ\nesvf318cmr5375441SI9PR316tXT2oNzdXVVKxpxcHDQOfVA6oHS2bNn0bFjxzLnoYsZKjnOnTsX\n4eHhuHnzJt58801xiL2izJw5E+PHj0dycjKCgoLEFZMMRcqBW0n6VKeT6Zk0OW7duhUHDhxAWloa\nevXqhQULFuDll19+oW3KrXorVvLoubjCrHixgZK+/vprrFmzBrm5uWLvo3hlmfIqbctTp04dvYdS\nXn75ZcTGxqKwsBB//fUXoqOj0bp1a63t5U5XeP/995GTk4NWrVqhbdu2aotDayJ36CoyMhLr16/H\n+PHjARRN/ShegUWTiIgIzJgxQyxiqFGjRpmKUn33hdmzZ2PFihVq8wQ/+eSTF/4CK05+s2fPRo8e\nPcTl444dO6bx/F5xL75hw4YIDAxEt27dxHOBuiqZpR4oFRehSakmfRFyh9iNrXHjxtiyZUuZ1aEM\nRcqBW0n6VKeT6Zk0Od6/fx9hYWF45ZVXTPmyGpU8eraxsYGHh4fG1TfGjh2LsWPHYunSpbIvK1Oe\nF/mAhoeHY82aNahSpQqmTJmCTp066awGLDldASg6p6jrS7Jp06a4fPkyrl+/LiYdXcu96TN0Vboa\ntLwrZ+zdu1e8zqEhv3CfPHkiJkagaKqFIa8rWHqh786dO2u89mbxKELDhg3RsGFD8fHiyevaSD1Q\n0jWED+gexpfD3M6pDRkyBLt27ULjxo3FxwxZcDVv3jxMnz5d54FbSefOncOuXbvg4eEhuTqdTM8k\nyTEzMxOOjo4YNWoUAJS5JqMhJ1xLJffo2dvbGxkZGeIXfnp6Os6cOYPu3bvrHYO+51oLCwvx5Zdf\nYsaMGfj4448lPcfDw0NrsY4mYWFhAIr+73bt2oWwsDA8fPhQ4/X+APlDV/Xq1RPXYc3Pz8fmzZt1\nFqk8efJEXA+0eO3WCRMmyF7pRBOFQoGUlBT83//9H4CidWINuWKJm5sbVq9ejf79+wMo+hLUNAez\ndPKQOi9P6oGSvkP4cpnLObWHDx8iNTUVz58/x++//y4ejGZmZiInJ8dgr9OsWTNZB27/+c9/DPba\nZDwmmcqh7/QAY5K72LSm9RkHDhyocyqGMclZvxIommqxcOFCXLhwAYIgoFWrVggLC9O66PuWLVuQ\nmJiIy5cvw8PDA23bthWvSaiNnMW+09LSsHDhQpw8eRIqlQpvvPEGZs2apTXZjRw5Em3btlVLMGfO\nnHmhYq5ix48fx5w5c9CuXTtxgvann36qc66gHE+fPsXKlSvFYeO2bdsiJCRE60HhtWvXMH36dHEJ\nMldXVyxatEit56OJ1OImY6vIaRsl7dq1Czt37sSlS5fE67ICRQcb/v7+BjvHqm9P+fHjx8jNzRXv\nFx+ckXkw+VU5zIWcxaYBzYsha3rMVObOnYvU1FTJF+gNDAzEsGHDxHOO8fHx2LJli9a5f+vXr0fb\ntm3RvHlznZeKWrZsGT755BMAwIkTJ154UQdtNH3hGvLv//jxY3z33Xd49dVX8fz5c9SqVQvt2rUz\nyLblCgoKwuTJk9GhQwcARQsmLF++XOvC4+fPn8fs2bPLLW4qObSriaZLSukjPDwc7777rtmcUzt4\n8KDGpRINZdSoUWJPueSUlZLXkCwpISEBixYtwoMHD1CzZk2kpKTA09MT8fHxRouR5DPpOcf333+/\nTIWkpsdMQW71ppeXF6KiosT1Krdu3SppWTtjkXuB3pycHLWFtwcMGKCz4KR4CLw8P/30k5gcP//8\n83KTo7bydqBoFGHChAkaf/fGG28gPj5enDh/4MABrdcBlev777/H5s2bcf/+fTRr1gy//vorWrVq\npXUhbqlKXn1eE23D3NnZ2WJiBIDXXntN5/qqUVFRkoqbTLW/mts5tY4dOyIqKkryPES55Fafrlix\nAt999x1GjhyJ3bt349SpU7KuEEKmYZLkmJubi5ycHDx58gTPnj1TG/tPTU01RQhlyK3eDA8Px+rV\nqzF58mQARV/WmpZYM5WIiAiN1bXa+Pr6Yu3atejduzcEQcC+ffvQuXNn8fyvqc77luzlFsvOzsaO\nHTvw9OnTMsmxeLEGlUqFTZs2Yfr06QCKzrtWr15d8nUDddm8eTO2b9+OwMBAREdH488//8Ty5ctf\neLvaeg7ladCgAVatWqU2qqFt+LuYlOImXdceNSRzO6c2a9YsWfMQ5ZJbfWprawtXV1colUoolUp0\n6NCBy8eZIZMkx5iYGGzatAkPHjyAv7+/mBwdHR3x7rvvmiKEMjQtNq2rSKd69epmtWp+3759UatW\nLcNWZfIAABMYSURBVLRt21Y8H6jrSHj//v0Aiv4viotNVCoV4uPjX+i8b/FCBiqVSuOiBqUrIEsm\njMzMTGzevBk7d+5E7969NSaTkos1PH36FH///bfaeRpDqFKlinigkZeXB09PT9y6deuFt1tylZ68\nvDz89ddfAFDu+djIyEh89dVXmDhxIgDAx8dH55en3OKmtLQ0/Oc//8GNGzfU/pYv2lMu5uHhofFi\nzRVF7jxEueT2lJ2dnZGVlYV27dph6tSpqFmzpsaDRqpYJj3nGB0drfUacKZUWFiI6OhofPDBB+W2\nXbhwIWbNmqV1iExOBaihpaSkIDExEefPn8fx48fh5ORUpmio2L59++Dr6wtHR0esWrUKv//+Oz78\n8MMXHmor7+hbU1HC06dPsWHDBsTGxsLPzw/vvfee1oscF9M09Nm6dWuDDMlPmDABUVFR2LRpE06d\nOgVnZ2cUFBQYrAd0+vRpzJw5Ex4eHuKCAIsWLTLYOc2SxU1KpRKdOnXSWdwUHByMd955B//9738x\nb9487Nq1CzVr1jTYhazN7WLNQ4YMwbRp09TmIS5evBjfffedQbavbWUqbQtLZGdno2rVqlCpVIiN\njUVGRgb69etnkMprMhyTF+Rcu3YNN27cQF5enviYtovQGlNAQAC2b99ebrviSrczZ85o/H3pNTxN\n5f79+0hMTMSZM2dw9epV1KhRAz4+Phg7dqzG9sXFK4mJiVixYgVGjRqFVatWGWxh9nv37pUZ2nv4\n8GGZZcMWLVqEw4cPIzAwEMOHD5d8+aB+/fqJQ5979uwRhz4NNTRW7MyZM8jIyMCbb75psGkI/v7+\n+Pzzz8V1T2/duoUpU6Zg586dau30PUepTzw7d+5UK2gaNGiQ1utLyjVgwADxYs3F1dwVWbx25coV\ncR4iUNRz++yzz9CsWTODvYamnnJ5Q+Fk3kxakLNy5UqcPn0af/75Jzp37ozjx4/Dx8enQpJjmzZt\n8Omnn6J3795q5e+le1LFJeDt27dHXl4ebt68CUEQ8NJLL1XoHK4uXbqgRYsWGDt2LD799NNy2xdX\n0R07dgyBgYHo0qWLzsvqyNW9e3f07NkTCxcuFP+eY8aMKTPResOGDahSpQr+/e9/q33ZF1+PUNvC\n78Ya+izNGAc7+fn5aguCv/TSSxqvt6hpWLnkELg2cqfpFFcfu7m54ccff4Sbm5vGK9frS+7Fmo1J\nqVTi5s2bRltAAlDvKQ8aNAj5+fniJeVKKnmxg5LK2/epYpg0OR48eBB79uzBwIEDERUVhUePHhls\nKEeuP/74AwDEBaaLd1Bt511+/PFHzJ07Fw0bNoRKpcKdO3cwb948cUkwU9u9ezfOnTuHuLg4/Oc/\n/0GjRo3Qrl07rWuZuru7Y86cOThx4gRGjx6NvLw8g15gtXHjxvDx8cGwYcOwYsUK8e9U2pUrV/Ta\nft26dZGeno7u3btj5MiRcHZ2tph5YV5eXpg1a5Y4R3Pv3r1q8+6KpaenIzU1VayIDggIQFpaGgRB\n0Hm+e8qUKRg2bJjYi46Pj8cnn3yidVRg/PjxyMjIwIwZMzB//nxkZWWJiz4YgqaLNRvq2olyKRQK\nrFu3Dr179zbaMnaHDx8We8pA0WdN0znWkufPyfyZNDlWrVoVCoUCtra2yMzMRK1atXDv3j1ThiB6\n66231BYkEAQBjo6O+OOPPzQub/fZZ59h8+bNaNSoEYCik/xjxoypsOTYrFkzNGjQAA0aNMC5c+ew\nd+9enD17VuuX0BdffIGffvoJwcHBcHZ2xoMHD8TKT0MQBAHDhw9Hs2bNMG7cOEydOtWgq8ysWrUK\nADBx4kS89tpr4tCnJZg3bx62bt2KLVu2AIB4EFHaunXr1Kpk8/PzsWPHDuTk5CA0NFTr9R/lTtM5\ncOAAfHx80KRJE0RHR+Pp06dYtGiRxus86sMYFzV/Ea+//jrWr19fZpTIUBXa5tRTJsMxaXL08vJC\neno6Bg8eDH9/f1SvXl3nYtnGdPnyZVy6dAldu3aFSqXCDz/8gKZNmyImJga9evXC6NGj1do7ODiI\niREoKreXer7MGPz9/ZGfn4/WrVvDx8cHW7Zs0XllCXt7e7U5kG5ubhqXMNNX8UGGj48PNm7ciMmT\nJ+PmzZsG235JFXWeV64jR46IPcGRI0ciPj4eaWlp+P3331G3bl306tVLrX1+fr7aeVsfHx+4urrC\n1dVV53JncqfpXL16VW0tWRcXF3EkxRCWLFmCadOmqSXE4scqwr59+yAIAr755hu1xw21MpemnvKL\nXpSAKl6FrZBz584dZGZmGvSkuBzDhw/H2rVrxQSXlZWFsWPHYt26dfD398e+ffvU2s+dOxcpKSl4\n5513IAgCDhw4gHr16uH1118HYLjL/UiVlpaGmjVrmvQ1dSldfFNQUICkpKQKW2XGHAQFBWH58uVi\nwhswYAA2bdqE7OxshIaGlqm07dGjBw4fPqxxW927d8eRI0c0/q5kj6/0OUpN03T69++P6OhosUL4\n6dOnGDFihMEKZjQt6l2RBTnPnz/HN998I67L27ZtWwQFBem8coZcJ06cwM8//wwA6NSpU4X2lMkw\nKmyFnPr165d5zJQeP36sVlBjZ2eHR48eoVq1ahoLbfLy8lC7dm1xlY2aNWsiNzdXvLqHqZOjo6Mj\nYmNjzebKB76+vhg1ahSmTJkCQRBga2uLyMhIg135wBJp6gm6uLjAxcVFY0/Q29sb27ZtK9PriImJ\ngbe3t9bXmTp1qqxpOsHBwRgyZIjYcz1w4EC5lbJS6HvhcWObMWMGHB0dxWlkcXFxmDFjhkGv6fjG\nG28wIVqZSrtCTr9+/cRr5gFF197r27cvsrOzNU6gNvY18OQylysfFHv55ZehVCoRHByM5cuXw8XF\nxaDXzLNE6enpavdLrqiUlpZWpn1YWBgmTJiA2NhYMbFdvnwZeXl54jlXTf7973+jd+/eSExMxKlT\npzBq1ChERERoLcgZOHAgvLy8cOrUKQBF1ZYvel1VwDAXHjeG69evq40EdejQAb179zbY9jVVoTo5\nOcHLywszZ87klA4LVWlXyJkwYQJ8fX3F8ul58+ahRYsWAIClS5eWaX/r1i1ERETg8ePHiIuLw5Ur\nV3D06FGd11A0JnO7mritrS2mT5+Offv2Yfjw4Vi0aJFBC3IskdyeYK1atRATE4OTJ0/ixo0bAIqu\n/ajrSiiAftN0Xn75ZYMkxJKKr/s5efJk1KlTB1WqVMHp06dx9epVDBw4UO08pym9+uqruHDhAlq1\nagWg6PqamqqF9fX++++jbt26aov6Jycno3nz5ggLC0N0dLTBXotMp1KukKOPd999F9OnT8ecOXPE\nic0VeWkec7vyQcnLd127dg1TpkzBvXv3NC6AXVk8fvwYEyZMgN3/a+9uQqL6wjCAP6NlToohVkoa\naJD4EWJGUSYGVovJMhFbSEmLSNGFVmZQOZVKVGRBEphpEJWa+reoTMUoWii2SQ2LggzJSQUzP8bP\nqWbufxFe1LTJ0nNnmucHgo7SfQnlnXPvOc+7cOG0K8GlS5fOyXUSExPh7u6O+vp63L9/H46OjoiN\njVUszHr37t2oqKhAR0cHEhISEBERgdbWVsUyVzUaDdra2uSjP52dnfDx8ZHPe/7ts9CoqKif/q/H\nR9xN9z2yDkKfOcbHx6OxsREdHR0wGo3y60qEAMzW6OjoT+/2J46nEc3SJh9MHIfk6+uL4uJiReZ0\nWpI/XQnO1nwf05mt8eNatbW12LdvH+Lj4xX9Gy8sLJzXf1+tVqOqqmrSM9zxwApbv3tizYQ2x/T0\ndOh0Ovj5+cmNRaVSWUVzdHV1RXt7u/zLXlNT81M0mkiWNvmgtbVVbgA02aZNm+a8IU4038d0ZmvB\nggWorKzEgwcPkJeXBwCTNo2J9qsjTnMhJycHZ8+eRWZmppxQdPHiRYyNjUGr1c7rtWn+CL2tqtFo\n5DNH1kan00Gr1aKpqQkuLi7w8vJCTk7OvP/hTTU0NARnZ2f5DNtUSm18yM7Olj83GAxoaGhAYGCg\nnEBEtqO1tRV3795FcHAwdu7cCZ1Oh+rqaiQkJChdGtFvE9ocU1JSkJGRoei72tmaOoJpbGwMJpNJ\nHjEzdSTTfEtMTER+fj4iIiImJfwA059pU4per8fhw4ctatMQ0VwqKCjAwYMHkZ2dPe0b/oyMDAWq\norki9LZqX18fIiMjERQUNGmenZJjn8wZz0hsa2tDS0sLtm7dCkmS8PDhQ3l3q0j5+fkAfhw9sWRq\ntRqfPn1SugwSKDU1FVeuXJl0xnEipZ6Hz5fxI19zufOVLIfQlaOljX2ajb179yI/P18OLx4aGkJi\nYiKKiooUq2lgYOCn4b9KJdJMPEQuSRJaW1uh0WgsakA0za/u7m4sX7581vMNiSyR0JXjhg0b0NHR\ngY8fPyI0NBSjo6OTdq1asp6enkm7Qh0cHNDT06NYPdMN/w0ODp6zae6zNXHckr29PTw9PeHh4aFI\nLaSM8cclttIERc3fJGUIbY5lZWUoLS3FwMCAHMp8+vRpReLjZis6OhqxsbHYvn07gB+h0jExMYrV\nc+vWLXn47+3bt+Xhv0qZuPrv7e3lVHMbZGvzCsffENbW1qKnp0ceSfb48WO4ubkpWRrNAaHNsaio\nCOXl5XJiiLe397QxWpYoKSkJ4eHh8qH2c+fOISAgQLF6RA3/Nae5uRmXLl3CkiVLkJycjGPHjqGv\nrw8mkwkXLlxAeHi48JpIGbY2r3D8DeH58+dx7949+fWIiAhF3zjT3BDaHB0cHCbdmlTy7NOfCAwM\nnDHMWTRLGf6blZWFI0eOYHBwEPv370dBQQGCg4Px4cMHpKWlsTnSP290dBQ6nU7OUNXpdL8cMUbW\nQWhzXL9+Pa5du4axsTHU19ejuLh4zgas2hpLGf5rNBoRFhYGAMjNzZXzK6cLbyf6Fx0/fhzx8fFY\nuXIlJElCZ2cnMjMzlS6L/pLQ5nj06FH8999/8PX1RWlpKbZs2TLj5HqamdFoRGRkJGpqagAou9vX\nzs5O/nzqfDxrDHsgmq3w8HDU1tbKw71XrVplEZNy6O8oNuy4v79f3mlJs5eUlAStVqvIrdSJ/P39\noVarIUkSDAaD3CAlScLXr1/x5s0bResjEsFaM6NpZsKDx/Py8vD9+3fExMTAzc0Na9euxYkTJ0SW\n8U/Q6/VyoIJarZZfF719/O3bt0KvR2RprDkzmmYmtDkODg7C2dkZ5eXliI6ORkpKyoxpGvRrBoNB\nTssBfqzUcnJyFKyIyDa9fv3aajOjaWZCm6PRaER3dzeqq6tx6NAhkZf+5xiNxp+eNY6NjSlUDZHt\nWr16NT5//mxVmdFkntDmmJycjAMHDiAkJARBQUHQ6XTw9vYWWYLVKy4uRklJCXQ63aRV9/DwMEJC\nQhSsjMg2WWNmNJmn2IYc+jODg4MYGBjA5cuXkZaWJr/u5OSk2LgqIltmzZnRNDOhzbGtrQ1nzpzB\nly9fUFlZiXfv3uHZs2dITk4WVQIREZFZduZ/ZO5otVqkpaVhwYIfd3P9/PxQVVUlsgQiojkRFxcH\n4EembEhIiPwx/jVZN6HPHEdHRxEUFDTptfGtz0RE1qSkpASA7WXK2gqhK0dXV1e0t7fLW55ramqw\nbNkykSUQERGZJfSZo06ng1arRVNTE1xcXODl5YWcnBybmf9GRETWQdhtVZPJhJaWFty8eRMjIyMw\nmUxwdnYWdXkiIqLfJuy2qp2dHQoLCwEAixcvZmMkIiKLJfSZY2hoKG7cuIGuri709/fLH0RERJZE\n6DPH6WY3qlQqPH36VFQJREREZjEhh4iIaAqh5xwNBgOKi4vx8uVLqFQqrFu3DnFxcVi0aJHIMoiI\niH5J6MoxNTUVTk5OiIqKAgBUVlZCr9cjNzdXVAlERERmCV05vn//flJc3MaNG7Fjxw6RJRAREZkl\ndLdqQEAAmpub5a9fvXqFNWvWiCyBiIjILKG3VTUaDdra2rBixQoAQGdnJ3x8fOQg8kePHokqhYiI\naEZCm2NHR8cvv88YOSIisgRCb6t6enqiq6sLL168gKenJ9RqNUwmEzw9PdkYiYjIYghtjlevXkVh\nYSGuX78OAPj27RvS09NFlkBERGSW0Ob45MkT5OXlQa1WAwDc3d0xPDwssgQiIiKzhDbHhQsXQqVS\nyfMcR0ZGRF6eiIjotwg956jRaHDq1Cno9XqUlZWhoqICe/bsEVkCERGRWcKzVevr61FXVwcACAsL\nw+bNm0VenoiIyCxFg8dNJhMqKyvlODkiIiJLIOSZ49DQEPLz85GVlYW6ujpIkoQ7d+5g27ZtqK6u\nFlECERHRbxOyckxKSsKSJUsQHByMhoYG9Pb2QpIknDx5Ev7+/vN9eSIiolkR0hx37dolR8MZjUaE\nhYXh+fPnHFVFREQWScht1fHsVACwt7eHh4cHGyMREVksIStHf39/+eC/JEkwGAxwdHSEJElQqVRo\nbGyc7xKIiIh+m6K7VYmIiCyR0IQcIiIia8DmSERENAWbIxER0RRsjkRERFOwORIREU3xPxL/B6Mc\ndTjeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x133beb950>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(df);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}