attention-text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('./data/labeledTrainData.tsv.zip', 'r') as z:\n",
    "    z.extractall('./data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment                                             review\n",
       "0  5814_8          1  With all this stuff going down at the moment w...\n",
       "1  2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "2  7759_3          0  The film starts with a manager (Nicholas Bell)...\n",
       "3  3630_4          0  It must be assumed that those who praised this...\n",
       "4  9495_8          1  Superbly trashy and wondrously unpretentious 8..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(data_train.shape)\n",
    "data_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['review'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.review.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "\n",
    "def clean_str(string):\n",
    "    \"\"\"\n",
    "    Tokenization/string cleaning for all datasets \n",
    "    Every dataset is lower cased\n",
    "    \"\"\"\n",
    "    string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", string)     \n",
    "    string = re.sub(r\"\\'s\", \" \\'s\", string) \n",
    "    string = re.sub(r\"\\'ve\", \" \\'ve\", string) \n",
    "    string = re.sub(r\"n\\'t\", \" n\\'t\", string) \n",
    "    string = re.sub(r\"\\'re\", \" \\'re\", string) \n",
    "    string = re.sub(r\"\\'d\", \" \\'d\", string) \n",
    "    string = re.sub(r\"\\'ll\", \" \\'ll\", string) \n",
    "    string = re.sub(r\",\", \" , \", string) \n",
    "    string = re.sub(r\"!\", \" ! \", string) \n",
    "    string = re.sub(r\"\\(\", \" \\( \", string) \n",
    "    string = re.sub(r\"\\)\", \" \\) \", string) \n",
    "    string = re.sub(r\"\\?\", \" \\? \", string) \n",
    "    string = re.sub(r\"\\s{2,}\", \" \", string)    \n",
    "    return string.strip().lower()\n",
    "\n",
    "texts = []\n",
    "labels = []\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    text = BeautifulSoup(data_train.review[idx], \"lxml\")\n",
    "    texts.append(clean_str(text.get_text()))\n",
    "    labels.append(data_train.sentiment[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"with all this stuff going down at the moment with mj i 've started listening to his music , watching the odd documentary here and there , watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj 's feeling towards the press and also the obvious message of drugs are bad m'kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for 20 minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans \\\\? nah , joe pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates mj 's music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also , the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line , this movie is for people who like mj on one level or another \\\\( which i think is most people \\\\) if not , then stay away it does try and give off a wholesome message and ironically mj 's bestest buddy in this movie is a girl ! michael jackson is truly one of the most talented people ever to grace this planet but is he guilty \\\\? well , with all the attention i 've gave this subject hmmm well i do n't know because people can be different behind closed doors , i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "from keras.layers import Embedding\n",
    "from keras.layers import Dense, Input, Flatten, concatenate, TimeDistributed\n",
    "from keras.layers import Conv1D, MaxPooling1D, Dropout, Concatenate, SpatialDropout1D, GlobalAveragePooling1D, GlobalMaxPooling1D\n",
    "from keras.layers import Bidirectional, LSTM, GRU, CuDNNLSTM, CuDNNGRU\n",
    "from keras.models import Model\n",
    "\n",
    "from keras import backend as K\n",
    "from keras.engine.topology import Layer, InputSpec\n",
    "from keras import initializers\n",
    "\n",
    "MAX_SEQUENCE_LENGTH = 1000\n",
    "MAX_NB_WORDS = 20000\n",
    "EMBEDDING_DIM = 100\n",
    "VALIDATION_SPLIT = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\keras_preprocessing\\text.py:175: UserWarning: The `nb_words` argument in `Tokenizer` has been renamed `num_words`.\n",
      "  warnings.warn('The `nb_words` argument in `Tokenizer` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 81988 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(nb_words=MAX_NB_WORDS)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (25000, 1000)\n",
      "Shape of label tensor: (25000, 2)\n"
     ]
    }
   ],
   "source": [
    "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive and negative reviews in traing and validation set \n",
      "[10073.  9927.]\n",
      "[2427. 2573.]\n"
     ]
    }
   ],
   "source": [
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
    "\n",
    "x_train = data[:-nb_validation_samples]\n",
    "y_train = labels[:-nb_validation_samples]\n",
    "x_val = data[-nb_validation_samples:]\n",
    "y_val = labels[-nb_validation_samples:]\n",
    "\n",
    "print('Number of positive and negative reviews in traing and validation set ')\n",
    "print(y_train.sum(axis=0))\n",
    "print(y_val.sum(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 400000 word vectors in Glove 6B 100d.\n"
     ]
    }
   ],
   "source": [
    "GLOVE_DIR = \"../pre_trained word embeddings/glove.6B\"\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt'), encoding='utf-8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "print('Total %s word vectors in Glove 6B 100d.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = Embedding(len(word_index) + 1,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Theano as backend\n",
    "class AttLayer(Layer):\n",
    "    def __init__(self, **kwargs):\n",
    "        self.init = initializers.get('normal')\n",
    "        #self.input_spec = [InputSpec(ndim=3)]\n",
    "        super(AttLayer, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape)==3\n",
    "        #self.W = self.init((input_shape[-1],1))\n",
    "        self.W = self.init((input_shape[-1],))\n",
    "        #self.input_spec = [InputSpec(shape=input_shape)]\n",
    "        self.trainable_weights = [self.W]\n",
    "        super(AttLayer, self).build(input_shape)  # be sure you call this somewhere!\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        eij = K.tanh(K.dot(x, self.W))\n",
    "        \n",
    "        ai = K.exp(eij)\n",
    "        weights = ai/K.sum(ai, axis=1).dimshuffle(0,'x')\n",
    "        \n",
    "        weighted_input = x*weights.dimshuffle(0,1,'x')\n",
    "        return weighted_input.sum(axis=1)\n",
    "\n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        return (input_shape[0], input_shape[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Tensorflow as backend\n",
    "# https://github.com/richliao/textClassifier/issues/13#issuecomment-377323318\n",
    "class AttLayer(Layer):\n",
    "    def __init__(self, **kwargs):\n",
    "        self.init = initializers.get('normal')\n",
    "        #self.input_spec = [InputSpec(ndim=3)]\n",
    "        super(AttLayer, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape)==3\n",
    "        #self.W = self.init((input_shape[-1],1))\n",
    "        self.W = self.init((input_shape[-1],))\n",
    "        #self.input_spec = [InputSpec(shape=input_shape)]\n",
    "        self.trainable_weights = [self.W]\n",
    "        super(AttLayer, self).build(input_shape)  # be sure you call this somewhere!\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        eij = K.tanh(K.squeeze(K.dot(x, K.expand_dims(self.W)), axis=-1))\n",
    "        \n",
    "        ai = K.exp(eij)\n",
    "        weights = ai/K.expand_dims(K.sum(ai, axis=1),1)\n",
    "        \n",
    "        weighted_input = x*K.expand_dims(weights,2)\n",
    "        return K.sum(weighted_input, axis=1)\n",
    "\n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        return (input_shape[0], input_shape[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttLayer(Layer):\n",
    "    def __init__(self, attention_dim):\n",
    "        self.init = initializers.get('normal')\n",
    "        self.supports_masking = True\n",
    "        self.attention_dim = attention_dim\n",
    "        super(AttLayer, self).__init__()\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape) == 3\n",
    "        self.W = K.variable(self.init((input_shape[-1], self.attention_dim)))\n",
    "        self.b = K.variable(self.init((self.attention_dim, )))\n",
    "        self.u = K.variable(self.init((self.attention_dim, 1)))\n",
    "        self.trainable_weights = [self.W, self.b, self.u]\n",
    "        super(AttLayer, self).build(input_shape)\n",
    "\n",
    "    def compute_mask(self, inputs, mask=None):\n",
    "        return mask\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        # size of x :[batch_size, sel_len, attention_dim]\n",
    "        # size of u :[batch_size, attention_dim]\n",
    "        # uit = tanh(xW+b)\n",
    "        uit = K.tanh(K.bias_add(K.dot(x, self.W), self.b))\n",
    "        ait = K.dot(uit, self.u)\n",
    "        ait = K.squeeze(ait, -1)\n",
    "\n",
    "        ait = K.exp(ait)\n",
    "\n",
    "        if mask is not None:\n",
    "            # Cast the mask to floatX to avoid float64 upcasting in theano\n",
    "            ait *= K.cast(mask, K.floatx())\n",
    "        ait /= K.cast(K.sum(ait, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n",
    "        ait = K.expand_dims(ait)\n",
    "        weighted_input = x * ait\n",
    "        output = K.sum(weighted_input, axis=1)\n",
    "\n",
    "        return output\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return (input_shape[0], input_shape[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - attention GRU network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 1000, 200)         121200    \n",
      "_________________________________________________________________\n",
      "att_layer_1 (AttLayer)       (None, 200)               20200     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,340,702\n",
      "Trainable params: 8,340,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:14: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.4143 - acc: 0.7980 - val_loss: 0.3316 - val_acc: 0.8580\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.2492 - acc: 0.8982 - val_loss: 0.2708 - val_acc: 0.8908\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.1908 - acc: 0.9245 - val_loss: 0.2374 - val_acc: 0.9082\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.1473 - acc: 0.9433 - val_loss: 0.2282 - val_acc: 0.9110\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.1056 - acc: 0.9618 - val_loss: 0.2360 - val_acc: 0.9172\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0717 - acc: 0.9756 - val_loss: 0.3170 - val_acc: 0.9088\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0429 - acc: 0.9867 - val_loss: 0.3978 - val_acc: 0.9078\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0245 - acc: 0.9926 - val_loss: 0.4569 - val_acc: 0.8994\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0128 - acc: 0.9962 - val_loss: 0.6374 - val_acc: 0.8984\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0063 - acc: 0.9986 - val_loss: 0.6661 - val_acc: 0.9056\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x29661f31630>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_gru)\n",
    "preds = Dense(2, activation='softmax')(l_att)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - attention GRU network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - attention GRU network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 1000, 200)         121200    \n",
      "_________________________________________________________________\n",
      "att_layer_1 (AttLayer)       (None, 200)               20200     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,340,702\n",
      "Trainable params: 8,340,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:15: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.4028 - acc: 0.8090 - val_loss: 0.3140 - val_acc: 0.8760\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.2531 - acc: 0.8980 - val_loss: 0.2896 - val_acc: 0.8766\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.1931 - acc: 0.9250 - val_loss: 0.2176 - val_acc: 0.9130\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.1511 - acc: 0.9445 - val_loss: 0.2365 - val_acc: 0.9096\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.1135 - acc: 0.9600 - val_loss: 0.2374 - val_acc: 0.9120\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0811 - acc: 0.9723 - val_loss: 0.2763 - val_acc: 0.9056\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0525 - acc: 0.9831 - val_loss: 0.3241 - val_acc: 0.9076\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0313 - acc: 0.9905 - val_loss: 0.4193 - val_acc: 0.9024\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0176 - acc: 0.9951 - val_loss: 0.5928 - val_acc: 0.8946\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0106 - acc: 0.9967 - val_loss: 0.5550 - val_acc: 0.8978\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1b9e3e10438>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_gru)\n",
    "l_dropout = Dropout(0.2)(l_att)\n",
    "preds = Dense(2, activation='softmax')(l_dropout)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - attention GRU network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - attention GRU network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 1000, 200)         121200    \n",
      "_________________________________________________________________\n",
      "att_layer_2 (AttLayer)       (None, 200)               20200     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,340,702\n",
      "Trainable params: 8,340,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:15: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.2513 - acc: 0.8897 - val_loss: 0.2434 - val_acc: 0.9054\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0943 - acc: 0.9660 - val_loss: 0.2964 - val_acc: 0.9054\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0518 - acc: 0.9816 - val_loss: 0.3474 - val_acc: 0.9014\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0294 - acc: 0.9901 - val_loss: 0.4211 - val_acc: 0.8990\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0165 - acc: 0.9954 - val_loss: 0.5425 - val_acc: 0.8940\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0096 - acc: 0.9976 - val_loss: 0.6227 - val_acc: 0.9008\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0061 - acc: 0.9979 - val_loss: 0.7327 - val_acc: 0.8912\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0052 - acc: 0.9987 - val_loss: 0.8789 - val_acc: 0.8876\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0034 - acc: 0.9992 - val_loss: 0.7707 - val_acc: 0.8980\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.0024 - acc: 0.9993 - val_loss: 0.9038 - val_acc: 0.9028\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2966d0a29e8>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_gru)\n",
    "l_dropout = Dropout(0.5)(l_att)\n",
    "preds = Dense(2, activation='softmax')(l_dropout)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - attention GRU network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - attention GRU network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 1000, 200)         121200    \n",
      "_________________________________________________________________\n",
      "att_layer_2 (AttLayer)       (None, 200)               20200     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,340,702\n",
      "Trainable params: 8,340,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:15: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.2345 - acc: 0.8959 - val_loss: 0.2277 - val_acc: 0.9122\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0684 - acc: 0.9763 - val_loss: 0.3465 - val_acc: 0.9014\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0218 - acc: 0.9934 - val_loss: 0.3927 - val_acc: 0.9014\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0095 - acc: 0.9972 - val_loss: 0.4575 - val_acc: 0.8980\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0050 - acc: 0.9985 - val_loss: 0.5368 - val_acc: 0.9026\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0032 - acc: 0.9990 - val_loss: 0.6356 - val_acc: 0.9026\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0033 - acc: 0.9988 - val_loss: 0.6061 - val_acc: 0.8958\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 4.2733e-04 - acc: 0.9999 - val_loss: 0.6952 - val_acc: 0.9062\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 5.1345e-05 - acc: 1.0000 - val_loss: 0.7572 - val_acc: 0.9032\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 2.1196e-05 - acc: 1.0000 - val_loss: 0.7759 - val_acc: 0.9040\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1b9e4dfc588>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_gru)\n",
    "l_dropout = Dropout(0.2)(l_att)\n",
    "preds = Dense(2, activation='softmax')(l_dropout)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - attention GRU network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - attention GRU network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_3 (Bidirection (None, 1000, 200)         121200    \n",
      "_________________________________________________________________\n",
      "att_layer_3 (AttLayer)       (None, 200)               20200     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,340,702\n",
      "Trainable params: 8,340,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:15: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.1802 - acc: 0.9134 - val_loss: 0.3161 - val_acc: 0.9062\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0184 - acc: 0.9939 - val_loss: 0.3991 - val_acc: 0.9048\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0058 - acc: 0.9983 - val_loss: 0.5415 - val_acc: 0.8988\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 9.3853e-04 - acc: 0.9998 - val_loss: 0.6161 - val_acc: 0.9032\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 1.4262e-04 - acc: 1.0000 - val_loss: 0.6886 - val_acc: 0.9034\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1ba0ce455c0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_gru)\n",
    "l_dropout = Dropout(0.5)(l_att)\n",
    "preds = Dense(2, activation='softmax')(l_dropout)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - attention GRU network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=5, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Self Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "from keras.engine.topology import Layer\n",
    "\n",
    "class Position_Embedding(Layer):\n",
    "    \n",
    "    def __init__(self, size=None, mode='sum', **kwargs):\n",
    "        self.size = size #必须为偶数\n",
    "        self.mode = mode\n",
    "        super(Position_Embedding, self).__init__(**kwargs)\n",
    "        \n",
    "    def call(self, x):\n",
    "        if (self.size == None) or (self.mode == 'sum'):\n",
    "            self.size = int(x.shape[-1])\n",
    "        batch_size,seq_len = K.shape(x)[0],K.shape(x)[1]\n",
    "        position_j = 1. / K.pow(10000., \\\n",
    "                                 2 * K.arange(self.size / 2, dtype='float32' \\\n",
    "                               ) / self.size)\n",
    "        position_j = K.expand_dims(position_j, 0)\n",
    "        position_i = K.cumsum(K.ones_like(x[:,:,0]), 1)-1 #K.arange不支持变长，只好用这种方法生成\n",
    "        position_i = K.expand_dims(position_i, 2)\n",
    "        position_ij = K.dot(position_i, position_j)\n",
    "        position_ij = K.concatenate([K.cos(position_ij), K.sin(position_ij)], 2)\n",
    "        if self.mode == 'sum':\n",
    "            return position_ij + x\n",
    "        elif self.mode == 'concat':\n",
    "            return K.concatenate([position_ij, x], 2)\n",
    "        \n",
    "    def compute_output_shape(self, input_shape):\n",
    "        if self.mode == 'sum':\n",
    "            return input_shape\n",
    "        elif self.mode == 'concat':\n",
    "            return (input_shape[0], input_shape[1], input_shape[2]+self.size)\n",
    "\n",
    "\n",
    "class Attention(Layer):\n",
    "\n",
    "    def __init__(self, nb_head, size_per_head, **kwargs):\n",
    "        self.nb_head = nb_head\n",
    "        self.size_per_head = size_per_head\n",
    "        self.output_dim = nb_head*size_per_head\n",
    "        super(Attention, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.WQ = self.add_weight(name='WQ', \n",
    "                                  shape=(input_shape[0][-1], self.output_dim),\n",
    "                                  initializer='glorot_uniform',\n",
    "                                  trainable=True)\n",
    "        self.WK = self.add_weight(name='WK', \n",
    "                                  shape=(input_shape[1][-1], self.output_dim),\n",
    "                                  initializer='glorot_uniform',\n",
    "                                  trainable=True)\n",
    "        self.WV = self.add_weight(name='WV', \n",
    "                                  shape=(input_shape[2][-1], self.output_dim),\n",
    "                                  initializer='glorot_uniform',\n",
    "                                  trainable=True)\n",
    "        super(Attention, self).build(input_shape)\n",
    "        \n",
    "    def Mask(self, inputs, seq_len, mode='mul'):\n",
    "        if seq_len == None:\n",
    "            return inputs\n",
    "        else:\n",
    "            mask = K.one_hot(seq_len[:,0], K.shape(inputs)[1])\n",
    "            mask = 1 - K.cumsum(mask, 1)\n",
    "            for _ in range(len(inputs.shape)-2):\n",
    "                mask = K.expand_dims(mask, 2)\n",
    "            if mode == 'mul':\n",
    "                return inputs * mask\n",
    "            if mode == 'add':\n",
    "                return inputs - (1 - mask) * 1e12\n",
    "                \n",
    "    def call(self, x):\n",
    "        #如果只传入Q_seq,K_seq,V_seq，那么就不做Mask\n",
    "        #如果同时传入Q_seq,K_seq,V_seq,Q_len,V_len，那么对多余部分做Mask\n",
    "        if len(x) == 3:\n",
    "            Q_seq,K_seq,V_seq = x\n",
    "            Q_len,V_len = None,None\n",
    "        elif len(x) == 5:\n",
    "            Q_seq,K_seq,V_seq,Q_len,V_len = x\n",
    "        #对Q、K、V做线性变换\n",
    "        Q_seq = K.dot(Q_seq, self.WQ)\n",
    "        Q_seq = K.reshape(Q_seq, (-1, K.shape(Q_seq)[1], self.nb_head, self.size_per_head))\n",
    "        Q_seq = K.permute_dimensions(Q_seq, (0,2,1,3))\n",
    "        K_seq = K.dot(K_seq, self.WK)\n",
    "        K_seq = K.reshape(K_seq, (-1, K.shape(K_seq)[1], self.nb_head, self.size_per_head))\n",
    "        K_seq = K.permute_dimensions(K_seq, (0,2,1,3))\n",
    "        V_seq = K.dot(V_seq, self.WV)\n",
    "        V_seq = K.reshape(V_seq, (-1, K.shape(V_seq)[1], self.nb_head, self.size_per_head))\n",
    "        V_seq = K.permute_dimensions(V_seq, (0,2,1,3))\n",
    "        #计算内积，然后mask，然后softmax\n",
    "        A = K.batch_dot(Q_seq, K_seq, axes=[3,3]) / self.size_per_head**0.5\n",
    "        A = K.permute_dimensions(A, (0,3,2,1))\n",
    "        A = self.Mask(A, V_len, 'add')\n",
    "        A = K.permute_dimensions(A, (0,3,2,1))    \n",
    "        A = K.softmax(A)\n",
    "        #输出并mask\n",
    "        O_seq = K.batch_dot(A, V_seq, axes=[3,2])\n",
    "        O_seq = K.permute_dimensions(O_seq, (0,2,1,3))\n",
    "        O_seq = K.reshape(O_seq, (-1, K.shape(O_seq)[1], self.output_dim))\n",
    "        O_seq = self.Mask(O_seq, Q_len, 'mul')\n",
    "        return O_seq\n",
    "        \n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return (input_shape[0][0], input_shape[0][1], self.output_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            (None, 1000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1000, 100)    8198900     input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "attention_3 (Attention)         (None, 1000, 128)    38400       embedding_1[2][0]                \n",
      "                                                                 embedding_1[2][0]                \n",
      "                                                                 embedding_1[2][0]                \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_3 (Glo (None, 128)          0           attention_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 128)          0           global_average_pooling1d_3[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 2)            258         dropout_3[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 8,237,558\n",
      "Trainable params: 8,237,558\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 886s 44ms/step - loss: 0.3783 - acc: 0.8212 - val_loss: 0.2826 - val_acc: 0.8830\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 882s 44ms/step - loss: 0.2131 - acc: 0.9169 - val_loss: 0.2584 - val_acc: 0.8940\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 883s 44ms/step - loss: 0.1257 - acc: 0.9531 - val_loss: 0.3041 - val_acc: 0.8830\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 898s 45ms/step - loss: 0.0657 - acc: 0.9777 - val_loss: 0.4544 - val_acc: 0.8784\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 893s 45ms/step - loss: 0.0302 - acc: 0.9904 - val_loss: 0.6209 - val_acc: 0.8658\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1d1745cbc50>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "\n",
    "#embedded_sequences = Position_Embedding()(embedded_sequences) #增加Position_Embedding能轻微提高准确率\n",
    "O_seq = Attention(8,16)([embedded_sequences, embedded_sequences, embedded_sequences])\n",
    "O_seq = GlobalAveragePooling1D()(O_seq)\n",
    "O_seq = Dropout(0.2)(O_seq)\n",
    "preds = Dense(2, activation='softmax')(O_seq)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "             optimizer='adam',\n",
    "             metrics=['accuracy'])\n",
    "\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=5, batch_size=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_4 (InputLayer)            (None, 1000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1000, 100)    8198900     input_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "position__embedding_1 (Position (None, 1000, 100)    0           embedding_1[3][0]                \n",
      "__________________________________________________________________________________________________\n",
      "attention_4 (Attention)         (None, 1000, 128)    38400       position__embedding_1[0][0]      \n",
      "                                                                 position__embedding_1[0][0]      \n",
      "                                                                 position__embedding_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_4 (Glo (None, 128)          0           attention_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 128)          0           global_average_pooling1d_4[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 2)            258         dropout_4[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 8,237,558\n",
      "Trainable params: 8,237,558\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "20000/20000 [==============================] - 898s 45ms/step - loss: 0.2377 - acc: 0.8946 - val_loss: 0.3168 - val_acc: 0.8806\n",
      "Epoch 2/3\n",
      "20000/20000 [==============================] - 896s 45ms/step - loss: 0.0800 - acc: 0.9705 - val_loss: 0.5898 - val_acc: 0.8410\n",
      "Epoch 3/3\n",
      "20000/20000 [==============================] - 888s 44ms/step - loss: 0.0334 - acc: 0.9878 - val_loss: 0.6799 - val_acc: 0.8574\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1d1734daf28>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "\n",
    "embedded_sequences = Position_Embedding()(embedded_sequences) #增加Position_Embedding能轻微提高准确率\n",
    "O_seq = Attention(8,16)([embedded_sequences, embedded_sequences, embedded_sequences])\n",
    "O_seq = GlobalAveragePooling1D()(O_seq)\n",
    "O_seq = Dropout(0.2)(O_seq)\n",
    "preds = Dense(2, activation='softmax')(O_seq)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "             optimizer='adam',\n",
    "             metrics=['accuracy'])\n",
    "\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=3, batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HRNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 81988 unique tokens.\n",
      "Shape of data tensor: (25000, 15, 100)\n",
      "Shape of label tensor: (25000, 2)\n",
      "Number of positive and negative reviews in traing and validation set\n",
      "[ 9998. 10002.]\n",
      "[2502. 2498.]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer, text_to_word_sequence\n",
    "from nltk import tokenize\n",
    "\n",
    "MAX_SENT_LENGTH = 100\n",
    "MAX_SENTS = 15\n",
    "\n",
    "reviews = []\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    text = BeautifulSoup(data_train.review[idx], \"lxml\")\n",
    "    text = clean_str(text.get_text())\n",
    "    texts.append(text)\n",
    "    sentences = tokenize.sent_tokenize(text)\n",
    "    reviews.append(sentences)\n",
    "    labels.append(data_train.sentiment[idx])\n",
    "\n",
    "data = np.zeros((len(texts), MAX_SENTS, MAX_SENT_LENGTH), dtype='int32')\n",
    "\n",
    "for i, sentences in enumerate(reviews):\n",
    "    for j, sent in enumerate(sentences):\n",
    "        if j < MAX_SENTS:\n",
    "            wordTokens = text_to_word_sequence(sent)\n",
    "            k = 0\n",
    "            for _, word in enumerate(wordTokens):\n",
    "                if k < MAX_SENT_LENGTH and tokenizer.word_index[word] < MAX_NB_WORDS:\n",
    "                    data[i, j, k] = tokenizer.word_index[word]\n",
    "                    k = k + 1\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Total %s unique tokens.' % len(word_index))\n",
    "\n",
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
    "\n",
    "x_train = data[:-nb_validation_samples]\n",
    "y_train = labels[:-nb_validation_samples]\n",
    "x_val = data[-nb_validation_samples:]\n",
    "y_val = labels[-nb_validation_samples:]\n",
    "\n",
    "print('Number of positive and negative reviews in traing and validation set')\n",
    "print(y_train.sum(axis=0))\n",
    "print(y_val.sum(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = Embedding(len(word_index) + 1,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SENT_LENGTH,\n",
    "                            trainable=True,\n",
    "                            mask_zero=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, 15, 100)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_1 (TimeDist (None, 15, 200)           8319500   \n",
      "_________________________________________________________________\n",
      "bidirectional_4 (Bidirection (None, 200)               180600    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,500,502\n",
      "Trainable params: 8,500,502\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "model fitting - Hierachical attention network\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 125s 6ms/step - loss: 0.5248 - acc: 0.7323 - val_loss: 0.4053 - val_acc: 0.8270\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 121s 6ms/step - loss: 0.3296 - acc: 0.8594 - val_loss: 0.4344 - val_acc: 0.8120\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.2492 - acc: 0.8996 - val_loss: 0.3192 - val_acc: 0.8744\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.1835 - acc: 0.9298 - val_loss: 0.3468 - val_acc: 0.8696\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.1294 - acc: 0.9527 - val_loss: 0.3730 - val_acc: 0.8686\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 124s 6ms/step - loss: 0.0801 - acc: 0.9729 - val_loss: 0.4922 - val_acc: 0.8670\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 124s 6ms/step - loss: 0.0423 - acc: 0.9864 - val_loss: 0.5912 - val_acc: 0.8586\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 123s 6ms/step - loss: 0.0197 - acc: 0.9942 - val_loss: 0.7258 - val_acc: 0.8602\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 125s 6ms/step - loss: 0.0096 - acc: 0.9971 - val_loss: 1.2480 - val_acc: 0.8450\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 123s 6ms/step - loss: 0.0065 - acc: 0.9985 - val_loss: 1.0831 - val_acc: 0.8530\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x16e6d6dc518>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_input = Input(shape=(MAX_SENT_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sentence_input)\n",
    "l_lstm = Bidirectional(GRU(100))(embedded_sequences)\n",
    "sentEncoder = Model(sentence_input, l_lstm)\n",
    "\n",
    "review_input = Input(shape=(MAX_SENTS,MAX_SENT_LENGTH), dtype='int32')\n",
    "review_encoder = TimeDistributed(sentEncoder)(review_input)\n",
    "l_lstm_sent = Bidirectional(GRU(100))(review_encoder)\n",
    "preds = Dense(2, activation='softmax')(l_lstm_sent)\n",
    "model = Model(review_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "model.summary()\n",
    "print(\"model fitting - Hierachical attention network\")\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HATT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttLayer(Layer):\n",
    "    def __init__(self, attention_dim):\n",
    "        self.init = initializers.get('normal')\n",
    "        self.supports_masking = True\n",
    "        self.attention_dim = attention_dim\n",
    "        super(AttLayer, self).__init__()\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape) == 3\n",
    "        self.W = K.variable(self.init((input_shape[-1], self.attention_dim)))\n",
    "        self.b = K.variable(self.init((self.attention_dim, )))\n",
    "        self.u = K.variable(self.init((self.attention_dim, 1)))\n",
    "        self.trainable_weights = [self.W, self.b, self.u]\n",
    "        super(AttLayer, self).build(input_shape)\n",
    "\n",
    "    def compute_mask(self, inputs, mask=None):\n",
    "        return mask\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        # size of x :[batch_size, sel_len, attention_dim]\n",
    "        # size of u :[batch_size, attention_dim]\n",
    "        # uit = tanh(xW+b)\n",
    "        uit = K.tanh(K.bias_add(K.dot(x, self.W), self.b))\n",
    "        ait = K.dot(uit, self.u)\n",
    "        ait = K.squeeze(ait, -1)\n",
    "\n",
    "        ait = K.exp(ait)\n",
    "\n",
    "        if mask is not None:\n",
    "            # Cast the mask to floatX to avoid float64 upcasting in theano\n",
    "            ait *= K.cast(mask, K.floatx())\n",
    "        ait /= K.cast(K.sum(ait, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n",
    "        ait = K.expand_dims(ait)\n",
    "        weighted_input = x * ait\n",
    "        output = K.sum(weighted_input, axis=1)\n",
    "\n",
    "        return output\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return (input_shape[0], input_shape[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Tensorflow as backend\n",
    "# https://github.com/richliao/textClassifier/issues/13#issuecomment-377323318\n",
    "class AttLayer2(Layer):\n",
    "    def __init__(self, **kwargs):\n",
    "        self.init = initializers.get('normal')\n",
    "        self.supports_masking = True\n",
    "        #self.input_spec = [InputSpec(ndim=3)]\n",
    "        super(AttLayer2, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape)==3\n",
    "        #self.W = self.init((input_shape[-1],1))\n",
    "        self.W = self.init((input_shape[-1],))\n",
    "        #self.input_spec = [InputSpec(shape=input_shape)]\n",
    "        self.trainable_weights = [self.W]\n",
    "        super(AttLayer2, self).build(input_shape)  # be sure you call this somewhere!\n",
    "        \n",
    "    def compute_mask(self, inputs, mask=None):\n",
    "        return mask\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        eij = K.tanh(K.squeeze(K.dot(x, K.expand_dims(self.W)), axis=-1))\n",
    "        \n",
    "        ai = K.exp(eij)\n",
    "        \n",
    "        if mask is not None:\n",
    "            # Cast the mask to floatX to avoid float64 upcasting in theano\n",
    "            ai *= K.cast(mask, K.floatx())\n",
    "        \n",
    "        weights = ai/K.expand_dims(K.sum(ai, axis=1),1)\n",
    "        \n",
    "        weighted_input = x*K.expand_dims(weights,2)\n",
    "        return K.sum(weighted_input, axis=1)\n",
    "\n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        return (input_shape[0], input_shape[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras.backend as K\n",
    "from keras.layers import Layer\n",
    "from keras import initializers, regularizers, constraints\n",
    "\n",
    "def dot_product(x, kernel):\n",
    "    \"\"\"\n",
    "    https://github.com/richliao/textClassifier/issues/13#issuecomment-377323318\n",
    "    Wrapper for dot product operation, in order to be compatible with both\n",
    "    Theano and Tensorflow\n",
    "    Args:\n",
    "        x (): input\n",
    "        kernel (): weights\n",
    "    Returns:\n",
    "    \"\"\"\n",
    "    if K.backend() == 'tensorflow':\n",
    "        return K.squeeze(K.dot(x, K.expand_dims(kernel)), axis=-1)\n",
    "    else:\n",
    "        return K.dot(x, kernel)\n",
    "    \n",
    "class AttentionWithContext(Layer):\n",
    "    \"\"\"\n",
    "    Attention operation, with a context/query vector, for temporal data.\n",
    "    Supports Masking.\n",
    "    Follows the work of Yang et al. [https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf]\n",
    "    \"Hierarchical Attention Networks for Document Classification\"\n",
    "    by using a context vector to assist the attention\n",
    "    # Input shape\n",
    "        3D tensor with shape: `(samples, steps, features)`.\n",
    "    # Output shape\n",
    "        2D tensor with shape: `(samples, features)`.\n",
    "    \n",
    "    How to use:\n",
    "    Just put it on top of an RNN Layer (GRU/LSTM/SimpleRNN) with return_sequences=True.\n",
    "    The dimensions are inferred based on the output shape of the RNN.\n",
    "    \n",
    "    Note: The layer has been tested with Keras 2.0.6\n",
    "    \n",
    "    Example:\n",
    "        model.add(LSTM(64, return_sequences=True))\n",
    "        model.add(AttentionWithContext())\n",
    "        # next add a Dense layer (for classification/regression) or whatever...\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, return_coefficients=False,\n",
    "                 W_regularizer=None, u_regularizer=None, b_regularizer=None,\n",
    "                 W_constraint=None, u_constraint=None, b_constraint=None,\n",
    "                 bias=True, **kwargs):\n",
    "        self.supports_masking = True\n",
    "        self.return_coefficients = return_coefficients\n",
    "        self.init = initializers.get('glorot_uniform')\n",
    "        \n",
    "        self.W_regularizer = regularizers.get(W_regularizer)\n",
    "        self.u_regularizer = regularizers.get(u_regularizer)\n",
    "        self.b_regularizer = regularizers.get(b_regularizer)\n",
    "        \n",
    "        self.W_constraint = constraints.get(W_constraint)\n",
    "        self.u_constraint = constraints.get(u_constraint)\n",
    "        self.b_constraint = constraints.get(b_constraint)\n",
    "        \n",
    "        self.bias = bias\n",
    "        super(AttentionWithContext, self).__init__(**kwargs)\n",
    "    \n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape) == 3\n",
    "        \n",
    "        self.W = self.add_weight((input_shape[-1], input_shape[-1],),\n",
    "                                 initializer=self.init,\n",
    "                                 name='{}_W'.format(self.name),\n",
    "                                 regularizer=self.W_regularizer,\n",
    "                                 constraint=self.W_constraint)\n",
    "        if self.bias:\n",
    "            self.b = self.add_weight((input_shape[-1],),\n",
    "                                     initializer='zero',\n",
    "                                     name='{}_b'.format(self.name),\n",
    "                                     regularizer=self.b_regularizer,\n",
    "                                     constraint=self.b_constraint)\n",
    "        \n",
    "        self.u = self.add_weight((input_shape[-1],),\n",
    "                                 initializer=self.init,\n",
    "                                 name='{}_u'.format(self.name),\n",
    "                                 regularizer=self.u_regularizer,\n",
    "                                 constraint=self.u_constraint)\n",
    "        \n",
    "        super(AttentionWithContext, self).build(input_shape)\n",
    "    \n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        # do not pass the mask to the next layers\n",
    "        return None\n",
    "    \n",
    "    def call(self, x, mask=None):\n",
    "        uit = dot_product(x, self.W)\n",
    "        \n",
    "        if self.bias:\n",
    "            uit += self.b\n",
    "        \n",
    "        uit = K.tanh(uit)\n",
    "        ait = dot_product(uit, self.u)\n",
    "        \n",
    "        a = K.exp(ait)\n",
    "        \n",
    "        # apply mask after the exp. will be re-normalized next\n",
    "        if mask is not None:\n",
    "            # Cast the mask to floatX to avoid float64 upcasting in theano\n",
    "            a *= K.cast(mask, K.floatx())\n",
    "        \n",
    "        # in some cases especially in the early stages of training the sum may be almost zero\n",
    "        # and this results in NaN's. A workaround is to add a very small positive number ε to the sum.\n",
    "        # a /= K.cast(K.sum(a, axis=1, keepdims=True), K.floatx())\n",
    "        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n",
    "        \n",
    "        a = K.expand_dims(a)\n",
    "        weighted_input = x * a\n",
    "        \n",
    "        if self.return_coefficients:\n",
    "            return [K.sum(weighted_input, axis=1), a]\n",
    "        else:\n",
    "            return K.sum(weighted_input, axis=1)\n",
    "    \n",
    "    \n",
    "    \n",
    "    def compute_output_shape(self, input_shape):\n",
    "        if self.return_coefficients:\n",
    "            return [(input_shape[0], input_shape[-1]), (input_shape[0], input_shape[-1], 1)]\n",
    "        else:\n",
    "            return input_shape[0], input_shape[-1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Dimensions must be equal, but are 15 and 100 for 'attention_with_context_3/mul' (op: 'Mul') with input shapes: [?,15], [?,15,100].",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36m_create_c_op\u001b[1;34m(graph, node_def, inputs, control_inputs)\u001b[0m\n\u001b[0;32m   1575\u001b[0m   \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1576\u001b[1;33m     \u001b[0mc_op\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mc_api\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_FinishOperation\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mop_desc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1577\u001b[0m   \u001b[1;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mInvalidArgumentError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m: Dimensions must be equal, but are 15 and 100 for 'attention_with_context_3/mul' (op: 'Mul') with input shapes: [?,15], [?,15,100].",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-58-5455dd8d33f8>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[0mreview_encoder\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mTimeDistributed\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msentEncoder\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreview_input\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[0ml_lstm_sent\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mBidirectional\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mGRU\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreturn_sequences\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreview_encoder\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0ml_att_sent\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mAttentionWithContext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ml_lstm_sent\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     14\u001b[0m \u001b[0mpreds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mDense\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mactivation\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'softmax'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ml_att_sent\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[0mmodel\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mModel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreview_input\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpreds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\base_layer.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs, **kwargs)\u001b[0m\n\u001b[0;32m    455\u001b[0m             \u001b[1;31m# Actually call the layer,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    456\u001b[0m             \u001b[1;31m# collecting output(s), mask(s), and shape(s).\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 457\u001b[1;33m             \u001b[0moutput\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    458\u001b[0m             \u001b[0moutput_mask\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompute_mask\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mprevious_mask\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    459\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-54-1547fbe96d8f>\u001b[0m in \u001b[0;36mcall\u001b[1;34m(self, x, mask)\u001b[0m\n\u001b[0;32m    102\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mmask\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    103\u001b[0m             \u001b[1;31m# Cast the mask to floatX to avoid float64 upcasting in theano\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 104\u001b[1;33m             \u001b[0ma\u001b[0m \u001b[1;33m*=\u001b[0m \u001b[0mK\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmask\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mK\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfloatx\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    105\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    106\u001b[0m         \u001b[1;31m# in some cases especially in the early stages of training the sum may be almost zero\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\u001b[0m in \u001b[0;36mbinary_op_wrapper\u001b[1;34m(x, y)\u001b[0m\n\u001b[0;32m    848\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mname_scope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mop_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    849\u001b[0m       \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 850\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    851\u001b[0m       \u001b[1;32melif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msparse_tensor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSparseTensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    852\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\u001b[0m in \u001b[0;36m_mul_dispatch\u001b[1;34m(x, y, name)\u001b[0m\n\u001b[0;32m   1092\u001b[0m   \u001b[0mis_tensor_y\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1093\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0mis_tensor_y\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1094\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mgen_math_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmul\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1095\u001b[0m   \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1096\u001b[0m     \u001b[1;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msparse_tensor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSparseTensor\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Case: Dense * Sparse.\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py\u001b[0m in \u001b[0;36mmul\u001b[1;34m(x, y, name)\u001b[0m\n\u001b[0;32m   5247\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0m_ctx\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0m_ctx\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_eager_context\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_eager\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   5248\u001b[0m     _, _, _op = _op_def_lib._apply_op_helper(\n\u001b[1;32m-> 5249\u001b[1;33m         \"Mul\", x=x, y=y, name=name)\n\u001b[0m\u001b[0;32m   5250\u001b[0m     \u001b[0m_result\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_op\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   5251\u001b[0m     \u001b[0m_inputs_flat\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_op\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\u001b[0m in \u001b[0;36m_apply_op_helper\u001b[1;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[0;32m    785\u001b[0m         op = g.create_op(op_type_name, inputs, output_types, name=scope,\n\u001b[0;32m    786\u001b[0m                          \u001b[0minput_types\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minput_types\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattrs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mattr_protos\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 787\u001b[1;33m                          op_def=op_def)\n\u001b[0m\u001b[0;32m    788\u001b[0m       \u001b[1;32mreturn\u001b[0m \u001b[0moutput_structure\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mop_def\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_stateful\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mop\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    789\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\u001b[0m in \u001b[0;36mnew_func\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    452\u001b[0m                 \u001b[1;34m'in a future version'\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mdate\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m'after %s'\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mdate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    453\u001b[0m                 instructions)\n\u001b[1;32m--> 454\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    455\u001b[0m     return tf_decorator.make_decorator(func, new_func, 'deprecated',\n\u001b[0;32m    456\u001b[0m                                        _add_deprecated_arg_notice_to_docstring(\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36mcreate_op\u001b[1;34m(***failed resolving arguments***)\u001b[0m\n\u001b[0;32m   3153\u001b[0m           \u001b[0minput_types\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minput_types\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3154\u001b[0m           \u001b[0moriginal_op\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_default_original_op\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3155\u001b[1;33m           op_def=op_def)\n\u001b[0m\u001b[0;32m   3156\u001b[0m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_create_op_helper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mret\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcompute_device\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcompute_device\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3157\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mret\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def)\u001b[0m\n\u001b[0;32m   1729\u001b[0m           op_def, inputs, node_def.attr)\n\u001b[0;32m   1730\u001b[0m       self._c_op = _create_c_op(self._graph, node_def, grouped_inputs,\n\u001b[1;32m-> 1731\u001b[1;33m                                 control_input_ops)\n\u001b[0m\u001b[0;32m   1732\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1733\u001b[0m     \u001b[1;31m# Initialize self._outputs.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36m_create_c_op\u001b[1;34m(graph, node_def, inputs, control_inputs)\u001b[0m\n\u001b[0;32m   1577\u001b[0m   \u001b[1;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mInvalidArgumentError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1578\u001b[0m     \u001b[1;31m# Convert to ValueError for backwards compatibility.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1579\u001b[1;33m     \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1580\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1581\u001b[0m   \u001b[1;32mreturn\u001b[0m \u001b[0mc_op\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Dimensions must be equal, but are 15 and 100 for 'attention_with_context_3/mul' (op: 'Mul') with input shapes: [?,15], [?,15,100]."
     ]
    }
   ],
   "source": [
    "MAX_SENTS = 15\n",
    "MAX_SENT_LENGTH =100\n",
    "\n",
    "sentence_input = Input(shape=(MAX_SENT_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sentence_input)\n",
    "l_lstm = Bidirectional(GRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_att = AttLayer(100)(l_lstm)\n",
    "sentEncoder = Model(sentence_input, l_att)\n",
    "\n",
    "review_input = Input(shape=(MAX_SENTS,MAX_SENT_LENGTH), dtype='int32')\n",
    "review_encoder = TimeDistributed(sentEncoder)(review_input)\n",
    "l_lstm_sent = Bidirectional(GRU(100, return_sequences=True))(review_encoder)\n",
    "l_att_sent = AttentionWithContext(100)(l_lstm_sent)\n",
    "preds = Dense(2, activation='softmax')(l_att_sent)\n",
    "model = Model(review_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "model.summary()\n",
    "print(\"model fitting - Hierachical attention network\")\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=1, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_19 (InputLayer)        (None, 15, 100)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_15 (TimeDis (None, 15, 200)           8313020   \n",
      "_________________________________________________________________\n",
      "bidirectional_19 (Bidirectio (None, 15, 200)           180600    \n",
      "_________________________________________________________________\n",
      "time_distributed_16 (TimeDis (None, 15, 200)           40200     \n",
      "_________________________________________________________________\n",
      "att_layer_16 (AttLayer)      (None, 200)               2020      \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,536,242\n",
      "Trainable params: 8,536,242\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "model fitting - Hierachical attention network\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:26: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n"
     ]
    },
    {
     "ename": "InvalidArgumentError",
     "evalue": "Inputs to operation bidirectional_19/while/Select_1 of type Select must have the same size and shape.  Input 0: [50,10000] != input 1: [50,100]\n\t [[Node: bidirectional_19/while/Select_1 = Select[T=DT_FLOAT, _class=[\"loc:@training_4/RMSprop/gradients/AddN_7\"], _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](bidirectional_19/while/Tile, bidirectional_19/while/add_6, bidirectional_19/while/Switch_3:1)]]\n\t [[Node: loss_4/mul/_1385 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_6143_loss_4/mul\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-35-237dab69ce1d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     24\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"model fitting - Hierachical attention network\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     25\u001b[0m model.fit(x_train, y_train, validation_data=(x_val, y_val),\n\u001b[1;32m---> 26\u001b[1;33m           nb_epoch=10, batch_size=50)\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\training.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[0;32m   1035\u001b[0m                                         \u001b[0minitial_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1036\u001b[0m                                         \u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1037\u001b[1;33m                                         validation_steps=validation_steps)\n\u001b[0m\u001b[0;32m   1038\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1039\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\engine\\training_arrays.py\u001b[0m in \u001b[0;36mfit_loop\u001b[1;34m(model, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[0;32m    197\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    198\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 199\u001b[1;33m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    200\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    201\u001b[0m                 \u001b[1;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mo\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mout_labels\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m   2664\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_legacy_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2665\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2666\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2667\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2668\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mpy_any\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mis_tensor\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py\u001b[0m in \u001b[0;36m_call\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m   2634\u001b[0m                                 \u001b[0msymbol_vals\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2635\u001b[0m                                 session)\n\u001b[1;32m-> 2636\u001b[1;33m         \u001b[0mfetched\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2637\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mfetched\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2638\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1380\u001b[0m           ret = tf_session.TF_SessionRunCallable(\n\u001b[0;32m   1381\u001b[0m               \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1382\u001b[1;33m               run_metadata_ptr)\n\u001b[0m\u001b[0;32m   1383\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1384\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py\u001b[0m in \u001b[0;36m__exit__\u001b[1;34m(self, type_arg, value_arg, traceback_arg)\u001b[0m\n\u001b[0;32m    517\u001b[0m             \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    518\u001b[0m             \u001b[0mcompat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc_api\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 519\u001b[1;33m             c_api.TF_GetCode(self.status.status))\n\u001b[0m\u001b[0;32m    520\u001b[0m     \u001b[1;31m# Delete the underlying status object from memory otherwise it stays alive\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    521\u001b[0m     \u001b[1;31m# as there is a reference to status from this from the traceback due to\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m: Inputs to operation bidirectional_19/while/Select_1 of type Select must have the same size and shape.  Input 0: [50,10000] != input 1: [50,100]\n\t [[Node: bidirectional_19/while/Select_1 = Select[T=DT_FLOAT, _class=[\"loc:@training_4/RMSprop/gradients/AddN_7\"], _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](bidirectional_19/while/Tile, bidirectional_19/while/add_6, bidirectional_19/while/Switch_3:1)]]\n\t [[Node: loss_4/mul/_1385 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_6143_loss_4/mul\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]"
     ]
    }
   ],
   "source": [
    "from keras.layers import TimeDistributed\n",
    "\n",
    "sentence_input = Input(shape=(MAX_SENT_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sentence_input)\n",
    "l_lstm = Bidirectional(GRU(100, return_sequences=True))(embedded_sequences)\n",
    "l_dense = TimeDistributed(Dense(200))(l_lstm)\n",
    "l_att = AttLayer(10)(l_dense)\n",
    "sentEncoder = Model(sentence_input, l_att)\n",
    "\n",
    "review_input = Input(shape=(MAX_SENTS,MAX_SENT_LENGTH), dtype='int32')\n",
    "review_encoder = TimeDistributed(sentEncoder)(review_input)\n",
    "l_lstm_sent = Bidirectional(GRU(100, return_sequences=True))(review_encoder)\n",
    "l_dense_sent = TimeDistributed(Dense(200))(l_lstm_sent)\n",
    "l_att_sent = AttLayer(10)(l_dense_sent)\n",
    "preds = Dense(2, activation='softmax')(l_att_sent)\n",
    "model = Model(review_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "print(\"model fitting - Hierachical attention network\")\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}