Build LSTM logic for local chatbot model

[jeff1evesque]

Currently the Vagrantfile deploys a prebuild chatbot, executed by run.py. However, we need to adjust our run.py to optionally build our own model. This can be implemented by invoking an optional flag, when set to True will instead create our own local chatbot model. Though many different solutions exists, we will initially try to implement keras to reduce syntax requirements.

[jeff1evesque]

ef6f289: both tensorflow and keras need to be compatible versions.

[jeff1evesque]

Running the --train locally, on the entire --insert case, results in MemoryError:

root@development:/vagrant# python3 run.py --train
Using TensorFlow backend.
vocabulary size: 21458
Traceback (most recent call last):
  File "run.py", line 120, in <module>
    main(op='train')
  File "run.py", line 70, in main
    model = train(posts, comments, cwd=cwd)
  File "/vagrant/chatbot/app/train.py", line 56, in train
    posts_train = create_posts(posts, vocab_size, post_maxlen, word2idx)
  File "/vagrant/chatbot/app/train.py", line 138, in create_posts
    post_idx = np.zeros(shape=(len(posts), post_maxlen, vocab_size))
MemoryError

If we decrease our vocab size:

$ git diff chatbot/
diff --git a/chatbot/app/train.py b/chatbot/app/train.py
index eb707da..f512eaa 100644
--- a/chatbot/app/train.py
+++ b/chatbot/app/train.py
@@ -50,7 +50,7 @@ def train(
     word2idx = {w:(i+1) for i,(w,_) in enumerate(counter.most_common())}
     idx2word = {v:k for k,v in word2idx.items()}
     idx2word[0] = 'PAD'
-    vocab_size = len(word2idx) + 1
+    vocab_size = int((len(word2idx) + 1)/1000)
     print('vocabulary size: {vocab}'.format(vocab=vocab_size))

     posts_train = create_posts(posts, vocab_size, post_maxlen, word2idx)

We result with an out of bounds error:

root@development:/vagrant# python3 run.py --train
Using TensorFlow backend.
vocabulary size: 21
i: 0, w: Your
Traceback (most recent call last):
  File "run.py", line 120, in <module>
    main(op='train')
  File "run.py", line 70, in main
    model = train(posts, comments, cwd=cwd)
  File "/vagrant/chatbot/app/train.py", line 56, in train
    posts_train = create_posts(posts, vocab_size, post_maxlen, word2idx)
  File "/vagrant/chatbot/app/train.py", line 140, in create_posts
    post = encode(posts[p], post_maxlen,vocab_size, word2idx)
  File "/vagrant/chatbot/app/train.py", line 128, in encode
    indices[i, word2idx[w]] = 1
IndexError: index 486 is out of bounds for axis 1 with size 21

[jeff1evesque]

We temporarily moved all our sample dataset except the first month into chatbot/data2:

root@development:/vagrant# date
Thu Dec 27 04:10:13 UTC 2018
$ git status
On branch feature-74
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   chatbot/data/reddit-2005-12
        deleted:    chatbot/data/reddit-2006-01
        deleted:    chatbot/data/reddit-2006-02
        deleted:    chatbot/data/reddit-2006-03
        deleted:    chatbot/data/reddit-2006-04
        deleted:    chatbot/data/reddit-2006-05

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        chatbot/data2/

no changes added to commit (use "git add" and/or "git commit -a")

Then, we removed approximately half the existing content from chatbot/data/reddit-2006-05, then executed our python run.py --insert followed by python run.py --train:

root@development:/vagrant# python3 run.py --train
Using TensorFlow backend.
vocabulary size: 1865
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 10, 1865)          0
_________________________________________________________________
lstm_1 (LSTM)                (None, 128)               1020928
_________________________________________________________________
repeat_vector_1 (RepeatVecto (None, 20, 128)           0
_________________________________________________________________
time_distributed_1 (TimeDist (None, 20, 1865)          240585
_________________________________________________________________
activity_regularization_1 (A (None, 20, 1865)          0
_________________________________________________________________
activation_1 (Activation)    (None, 20, 1865)          0
=================================================================
Total params: 1,261,513
Trainable params: 1,261,513
Non-trainable params: 0
_________________________________________________________________
None
Train on 54 samples, validate on 14 samples
Epoch 1/1
2018-12-27 04:12:37.075971: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2
32/54 [================>.............] - ETA: 1s - loss: 26.9587 - acc: 0.0000e+00Epoch 00001: saving model to /vagrant/model/checkpoint.ckpt
54/54 [==============================] - 4s 66ms/step - loss: 23.9524 - acc: 0.0000e+00 - val_loss: 14.3032 - val_acc: 0.0000e+00
root@development:/vagrant# ls -l model/
total 29644
-rwxrwxrwx 1 vagrant vagrant 15165772 Dec 27 04:12 chatbot.h5
-rwxrwxrwx 1 vagrant vagrant 15165772 Dec 27 04:12 checkpoint.ckpt
-rwxrwxrwx 1 vagrant vagrant    18185 Dec 27 04:12 idx2word.pkl