Feature Extraction and Summarization with Sequence to Sequence Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Feature Extraction and Summarization with Sequence to Sequence Learning.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "czSR_WOeHqea",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# KDD 2018 Hands-On Tutorial  https://kddseq2seq.com/\n",
        "\n",
        "Feature Extraction and Summarization With Sequence-to-Sequence Learning\n",
        "\n",
        "\n",
        "### Pre-requisites\n",
        "\n",
        "The target audience of this tutorial are moderately skilled users who have some familiarity with neural networks and are comfortable writing code.  These blog posts are good background for this tutorial:\n",
        "\n",
        "- [How To Create Data Products That Are Magical Using Sequence-to-Sequence Models](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8)\n",
        "\n",
        "- [How To Create Natural Language Semantic Search For Arbitrary Objects With Deep Learning](https://towardsdatascience.com/semantic-code-search-3cd6d244a39c)\n",
        "\n",
        "### Google Colab Notebooks\n",
        "\n",
        "This tutorial can be run in Google Colab notebooks, which provides a free gpu-enabled Jupyter Notebook on the cloud.  **You can open this notebook in Colab  by following [this link](https://colab.research.google.com/github/hohsiangwu/kdd-2018-hands-on-tutorials/blob/master/Feature%20Extraction%20and%20Summarization%20with%20Sequence%20to%20Sequence%20Learning.ipynb).**"
      ]
    },
    {
      "metadata": {
        "id": "3ZqOslw71bkc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# What we will go through today\n",
        "\n",
        "1. Language Model\n",
        "  * Self-supervised learning\n",
        "  * Sequence generation\n",
        "  * Pooling to get representations\n",
        "2. Sequence to Sequence Model\n",
        "  * Machine translation\n",
        "  * Encoder to get representations\n",
        "3. Joint Vector Space"
      ]
    },
    {
      "metadata": {
        "id": "-Tv_cXn6Gt0P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Motivating Example: Semantic Code Search\n",
        "\n",
        "Yes, this is a gif of a notebook inside another notebook.\n",
        "\n",
        "Motivation:  What if you could search code semantically instead of keyword search?  "
      ]
    },
    {
      "metadata": {
        "id": "KH8jBx7tGEQN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://github.com/hamelsmu/code_search/raw/master/gifs/live_search.gif?sanitize=true)\n",
        "\n",
        "A detailed, open source end to end tutorial on how to create semantic code search yourself is [here](https://towardsdatascience.com/semantic-code-search-3cd6d244a39c)."
      ]
    },
    {
      "metadata": {
        "id": "Q-gKT37Didb7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Setup Notebook\n",
        "\n",
        "Install [ktext](https://github.com/hamelsmu/ktext) and [annoy](https://github.com/spotify/annoy)."
      ]
    },
    {
      "metadata": {
        "id": "E7l80u-0fyHK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -q ktext\n",
        "!pip install -q annoy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "W-KjInFk0v8l",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "448a2cdc-62ad-4898-ec91-e38ce12fb25e"
      },
      "cell_type": "code",
      "source": [
        "import json\n",
        "from urllib.request import urlopen\n",
        "\n",
        "from annoy import AnnoyIndex\n",
        "from keras import optimizers\n",
        "from keras.layers import Input, Dense, LSTM, GRU, Embedding, Lambda, BatchNormalization\n",
        "from keras.models import load_model, Model\n",
        "from keras import optimizers\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.utils import get_file, to_categorical\n",
        "from ktext.preprocess import processor\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import random\n",
        "from tqdm import tqdm"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "iZzcs-PDPZqn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data sets"
      ]
    },
    {
      "metadata": {
        "id": "McfeOLlh0v8x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## [CoNaLa](https://conala-corpus.github.io/)\n",
        "\n",
        "Challenge designed to test systems for generating program snippets from natural language.\n",
        "\n",
        "\n",
        "### Preview of the CoNaLa Dataset\n",
        "\n",
        "```\n",
        "{\n",
        "  \"question_id\": 36875258,\n",
        "  \"intent\": \"copying one file's contents to another in python\", \n",
        "  \"rewritten_intent\": \"copy the content of file 'file.txt' to file 'file2.txt'\", \n",
        "  \"snippet\": \"shutil.copy('file.txt', 'file2.txt')\", \n",
        "}\n",
        "\n",
        "{\n",
        "  \"intent\": \"How do I check if all elements in a list are the same?\", \n",
        "  \"rewritten_intent\": \"check if all elements in list `mylist` are the same\", \n",
        "  \"snippet\": \"len(set(mylist)) == 1\", \n",
        "  \"question_id\": 22240602\n",
        "}\n",
        "\n",
        "{\n",
        "  \"intent\": \"Iterate through words of a file in Python\", \n",
        "  \"rewritten_intent\": \"get a list of words `words` of a file 'myfile'\", \n",
        "  \"snippet\": \"words = open('myfile').read().split()\", \n",
        "  \"question_id\": 7745260\n",
        "}\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "JAAOUd-k0v8x",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "70aa77ed-af9a-4f05-96f7-94f2582f0a8e"
      },
      "cell_type": "code",
      "source": [
        "!wget http://www.phontron.com/download/conala-corpus-v1.1.zip\n",
        "!unzip -o conala-corpus-v1.1.zip"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2018-08-22 09:02:34--  http://www.phontron.com/download/conala-corpus-v1.1.zip\r\n",
            "Resolving www.phontron.com (www.phontron.com)... 208.113.196.149\n",
            "Connecting to www.phontron.com (www.phontron.com)|208.113.196.149|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 52105440 (50M) [application/zip]\n",
            "Saving to: ‘conala-corpus-v1.1.zip.3’\n",
            "\n",
            "conala-corpus-v1.1. 100%[===================>]  49.69M  31.3MB/s    in 1.6s    \n",
            "\n",
            "2018-08-22 09:02:36 (31.3 MB/s) - ‘conala-corpus-v1.1.zip.3’ saved [52105440/52105440]\n",
            "\n",
            "Archive:  conala-corpus-v1.1.zip\n",
            "  inflating: conala-corpus/conala-mined.jsonl  \n",
            "  inflating: conala-corpus/conala-train.json  \n",
            "  inflating: conala-corpus/conala-test.json  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hFzeLKxg0v81",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with open('conala-corpus/conala-mined.jsonl', 'r') as f:\n",
        "    lines = [json.loads(line) for line in f.readlines()]\n",
        "source_docs = [line['snippet'] for line in lines]\n",
        "target_docs = [line['intent'] for line in lines]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QsXzuRH50v83",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with open('conala-corpus/conala-train.json', 'r') as f:\n",
        "    lines = json.load(f)\n",
        "train_source_docs = [line['snippet'] for line in lines]\n",
        "train_target_docs = [line['intent'] for line in lines]\n",
        "test_docs = [line['rewritten_intent'] for line in lines if line['rewritten_intent']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-t0x8A5w0v87",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with open('conala-corpus/conala-test.json', 'r') as f:\n",
        "    lines = json.load(f)\n",
        "test_source_docs = [line['snippet'] for line in lines]\n",
        "test_target_docs = [line['intent'] for line in lines]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EhmB0qpr3RD5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Other Data Sources (For Later Use)\n",
        "\n",
        "The below datasets are alternate sources of data for this same exercise.  We will not be reviewing these data as part of this tutorial.  However, we encourage you to inspect these data for additional practice and to get more intuition regarding these techniques.  Practicing with these other datasets will  give you confidence regarding the general application of the techniques we are teaching in this tutorial."
      ]
    },
    {
      "metadata": {
        "id": "oQLV9P8h0v8q",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### [English to French](http://www.manythings.org/anki/)"
      ]
    },
    {
      "metadata": {
        "id": "1uK2D4090v8r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# !wget http://www.manythings.org/anki/fra-eng.zip\n",
        "# !unzip -o fra-eng.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8UMp9B4M0v8u",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# with open('fra.txt', 'r') as f:\n",
        "#     lines = f.readlines()\n",
        "# target_docs, source_docs = zip(*[line.strip().split('\\t') for line in lines])\n",
        "# target_docs = list(set(target_docs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bw33N_AcPZqp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### GitHub issues data"
      ]
    },
    {
      "metadata": {
        "id": "GSR2uek3PZqq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# issues = pd.read_csv('https://storage.googleapis.com/kubeflow-examples/github-issue-summarization-data/github-issues.zip')\n",
        "# source_docs = list(issues.body)\n",
        "# target_docs = list(issues.issue_title)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AP58__OgPZqt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Python (function, docstring) pairs\n",
        "\n",
        "Purpose of this dataset is to see if you can generate the docstring of a python function or method by looking at the code."
      ]
    },
    {
      "metadata": {
        "id": "izpU8EXGPZqu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# f = urlopen('https://storage.googleapis.com/kubeflow-examples/code_search/data/train.function')\n",
        "# source_docs = [line.decode('utf-8') for line in f.readlines()]\n",
        "# f = urlopen('https://storage.googleapis.com/kubeflow-examples/code_search/data/train.docstring')\n",
        "# target_docs = [line.decode('utf-8') for line in f.readlines()]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8JykNMln0v9I",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Use subset of the data\n",
        "\n",
        "We will use only of the training set in the interest of brevity.  However, we can use the full dataset in a subsequent pass if desired."
      ]
    },
    {
      "metadata": {
        "id": "Hj1tDqDFPZqx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "source_docs = source_docs[:200000]\n",
        "target_docs = target_docs[:200000]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lxYjchOjPZqz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Language Model\n",
        "\n",
        "What is a language model?\n",
        "\n",
        "![alt text](https://cdn-images-1.medium.com/max/1440/1*XGfyUGtWq0yZ4RfufYfbRw.jpeg)\n",
        "\n",
        "Source: https://medium.com/paper-club/language-modeling-survey-333077e43dd9"
      ]
    },
    {
      "metadata": {
        "id": "COkelzEXPZq0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Preprocessing\n",
        "Tokenize, generate vocabulary, apply padding and vectorize."
      ]
    },
    {
      "metadata": {
        "id": "PYXCJuN69MOo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Keras Text Pre-Processing Primer\n",
        "\n",
        "Now that we have gathered the data, we need to prepare the data for the modeling. Before jumping into the code, let’s warm up with a toy example of two documents:\n",
        "\n",
        "```\n",
        "[“The quick brown fox jumped over the lazy dog 42 times.”, “The dog is lazy”]\n",
        "```\n",
        "\n",
        "Below is a rough outline of the steps I will take in order to pre-processes this raw text:\n",
        "\n",
        "**1. Clean text:** in this step, we want to remove or replace specific characters and lower case all the text. This step is discretionary and depends on the size of the data and the specifics of your domain. In this toy example, I lower-case all characters and replace numbers with *number* in the text. In the real data, I handle more scenarios.\n",
        "\n",
        "[“the quick brown fox jumped over the lazy dog *number* times”, “the dog is lazy”]\n",
        "\n",
        "\n",
        "**3. Tokenize:** split each document into a list of words\n",
        "\n",
        "```\n",
        "[[‘the’, ‘quick’, ‘brown’, ‘fox’, ‘jumped’, ‘over’, ‘the’, ‘lazy’, ‘dog’, ‘*number*’, ‘times’], [‘the’, ‘dog’, ‘is’, ‘lazy’]]\n",
        "```\n",
        "\n",
        "**4. Build vocabulary:** You will need to represent each distinct word in your corpus as an integer, which means you will need to build a map of token -> integers. Furthermore, I find it useful to reserve an integer for rare words that occur below a certain threshold as well as 0 for padding (see next step). After you apply a token -> integer mapping, your data might look like this:\n",
        "\n",
        "```\n",
        "[[2, 3, 4, 5, 6, 7, 2, 8, 9, 10, 11], [2, 9, 12, 8]]\n",
        "```\n",
        "\n",
        "**5. Padding:** 5. Padding: You will have documents that have different lengths. There are many strategies on how to deal with this for deep learning, however for this tutorial I will pad and truncate documents such that they are all transformed to the same length for simplicity. You can decide to pad (with zeros) and truncate your document at the beginning or end, which I will refer to as “pre” and “post” respectively. After pre-padding our toy example, the data might look like this:\n",
        "\n",
        "```\n",
        "[[2, 3, 4, 5, 6, 7, 2, 8, 9, 10, 11], [0, 0, 0, 0, 0, 0, 0, 2, 9, 12, 8]]\n",
        "```\n",
        "\n",
        "A reasonable way to decide your target document length is to build a histogram of document lengths and choose a sensible number. (Note that the above example has padded the data in front but we could also pad at the end. We will discuss this more in the next section)."
      ]
    },
    {
      "metadata": {
        "id": "6b7dZOxkDutv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Inspect the raw text of source and target documents:\n",
        "\n",
        "Source docs:\n"
      ]
    },
    {
      "metadata": {
        "id": "AdyYWgT_Eu13",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "a00a10a8-bcdc-4d57-b092-75d03b482c8d"
      },
      "cell_type": "code",
      "source": [
        "for x in source_docs[:10]:\n",
        "  print(x)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "sorted(l, key=lambda x: (-int(x[1]), x[0]))\n",
            "[int(x) for x in str(num)]\n",
            "c.decode('unicode_escape')\n",
            "parser.add_argument('-t', dest='table', help='', nargs='+')\n",
            "datetime.datetime.strptime(s, '%Y-%m-%dT%H:%M:%SZ')\n",
            "np.array(x._data).reshape(x.size[::-1]).T\n",
            "soup.get_text().replace('\\n', '\\n\\n')\n",
            "re.sub('(?<!\\\\S)((\\\\S+)(?:\\\\s+\\\\2))(?:\\\\s+\\\\2)+(?!\\\\S)', '\\\\1', s)\n",
            "mylist.sort(key=lambda d: (d['weight'], d['factor']))\n",
            "itertools.combinations\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hT8NcmEU4C4A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Target docs:"
      ]
    },
    {
      "metadata": {
        "id": "Ww4AHHUX4CPL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "6a2db83f-676a-40a2-dc9f-ee3e8e7feea1"
      },
      "cell_type": "code",
      "source": [
        "target_docs[:10]"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Sort a nested list by two elements',\n",
              " 'converting integer to list in python',\n",
              " 'Converting byte string in unicode string',\n",
              " 'List of arguments with argparse',\n",
              " 'How to convert a Date string to a DateTime object?',\n",
              " 'How to efficiently convert Matlab engine arrays to numpy ndarray?',\n",
              " 'Converting html to text with Python',\n",
              " 'regex for repeating words in a string in Python',\n",
              " 'Ordering a list of dictionaries in python',\n",
              " 'Two Combination Lists from One List']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "LTxQMjaa4WsL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In order to pre-process this data, we will use the [`ktext` package](https://github.com/hamelsmu/ktext).   `ktext` helps accomplish the pre-processing steps outlined in the previous section. This library is a thin wrapper around keras and spacy text processing utilities, and leverages python process-based-threading to speed things up. It also chains all of the pre-processing steps together and provides a bunch of convenience functions. Warning: this package is under development so use with caution outside this tutorial (pull requests are welcome!). To learn more about how this library works, look at this [tutorial](https://github.com/hamelsmu/ktext/blob/master/notebooks/Tutorial.ipynb) (but for now I suggest reading ahead)."
      ]
    },
    {
      "metadata": {
        "id": "uHOMxrONPZq3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "bdf737c1-73fd-4d1e-fa40-0b9f57965a21"
      },
      "cell_type": "code",
      "source": [
        "proc = processor(hueristic_pct_padding=.7, keep_n=5000)\n",
        "vecs = proc.fit_transform(target_docs)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:....tokenizing data\n",
            "WARNING:root:Setting maximum document length to 10 based upon hueristic of 0.7 percentile.\n",
            " See full histogram by insepecting the `document_length_stats` attribute.\n",
            "WARNING:root:(1/2) done. 45 sec\n",
            "WARNING:root:....building corpus\n",
            "WARNING:root:(2/2) done. 1 sec\n",
            "WARNING:root:Finished parsing 200,000 documents.\n",
            "WARNING:root:...fit is finished, beginning transform\n",
            "WARNING:root:...padding data\n",
            "WARNING:root:done. 2 sec\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "5htxJQw81RLS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "assert vecs.shape[0] == len(target_docs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0jld4HT0Dp_r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The above code cleans, tokenizes, and applies pre-padding and post-truncating such that each document length is equal to the 70th percentile of document lengths, which is an arbitrary choice. I made decisions about padding length by studying histograms of document length provided by ktext. Furthermore, only the top 5,000 words in the vocabulary are retained and remaining words are set to the index 1 which correspond to rare words (this was another arbitrary choice). \n",
        "\n",
        "Below is an example where tokens are mapped to integers."
      ]
    },
    {
      "metadata": {
        "id": "-ogmGKqUC-Jd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "9d180555-9b39-475f-e4fe-3ad0ebece752"
      },
      "cell_type": "code",
      "source": [
        "print('original list: ', target_docs[0])\n",
        "print('tokenized list: ', vecs[0])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "original list:  Sort a nested list by two elements\n",
            "tokenized list:  [  0   0   0 135   2 145  11  45  37  62]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ldna19QcEXYp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can see the most common words here, by calling the `token_count_pandas()` method."
      ]
    },
    {
      "metadata": {
        "id": "dNOluhQpD8Fq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "outputId": "5e1eaf26-a7cb-4430-e071-569f1c0c62db"
      },
      "cell_type": "code",
      "source": [
        "proc.token_count_pandas().head(20)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>a</th>\n",
              "      <td>103489</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>python</th>\n",
              "      <td>96545</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>in</th>\n",
              "      <td>94008</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>to</th>\n",
              "      <td>92498</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>how</th>\n",
              "      <td>70208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>of</th>\n",
              "      <td>44738</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>with</th>\n",
              "      <td>31986</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>the</th>\n",
              "      <td>26095</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>from</th>\n",
              "      <td>23301</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>list</th>\n",
              "      <td>22951</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>i</th>\n",
              "      <td>19227</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>using</th>\n",
              "      <td>17161</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>and</th>\n",
              "      <td>15000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>string</th>\n",
              "      <td>14001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>file</th>\n",
              "      <td>13059</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>number</th>\n",
              "      <td>12683</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>for</th>\n",
              "      <td>12596</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>do</th>\n",
              "      <td>11755</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>can</th>\n",
              "      <td>11042</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>an</th>\n",
              "      <td>10842</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         count\n",
              "a       103489\n",
              "python   96545\n",
              "in       94008\n",
              "to       92498\n",
              "how      70208\n",
              "of       44738\n",
              "with     31986\n",
              "the      26095\n",
              "from     23301\n",
              "list     22951\n",
              "i        19227\n",
              "using    17161\n",
              "and      15000\n",
              "string   14001\n",
              "file     13059\n",
              "number   12683\n",
              "for      12596\n",
              "do       11755\n",
              "can      11042\n",
              "an       10842"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "metadata": {
        "id": "Ab3JU2L5Ey9_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Furthermore, the documents in our corpus have different lengths. By setting `hueristic_pct_padding=.7`, `ktext` will truncate and pad all sequences to the 70th percentile length. However, it can be useful to sanity check a histogram of lengths. We inspect the `document_length_stats` property below which displays a histogram of document lengths."
      ]
    },
    {
      "metadata": {
        "id": "ZIH4R1HREiTO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "outputId": "f6d6eb5a-fa8c-45ee-eb88-af5d1f2deed6"
      },
      "cell_type": "code",
      "source": [
        "proc.document_length_stats"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bin</th>\n",
              "      <th>doc_count</th>\n",
              "      <th>cumsum_pct</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0</td>\n",
              "      <td>60</td>\n",
              "      <td>0.000300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5</td>\n",
              "      <td>70368</td>\n",
              "      <td>0.352140</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>10</td>\n",
              "      <td>100814</td>\n",
              "      <td>0.856210</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>15</td>\n",
              "      <td>25273</td>\n",
              "      <td>0.982575</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>20</td>\n",
              "      <td>3166</td>\n",
              "      <td>0.998405</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>25</td>\n",
              "      <td>275</td>\n",
              "      <td>0.999780</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>30</td>\n",
              "      <td>44</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   bin  doc_count  cumsum_pct\n",
              "6    0         60    0.000300\n",
              "0    5      70368    0.352140\n",
              "1   10     100814    0.856210\n",
              "2   15      25273    0.982575\n",
              "3   20       3166    0.998405\n",
              "5   25        275    0.999780\n",
              "4   30         44    1.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "SCNn24hRDxeF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "It is useful to keep track of the maximum length and the unique number of tokens in the corpus for later purposes."
      ]
    },
    {
      "metadata": {
        "id": "wnjhEa-NPZq6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "8fa05687-e8fe-4f3c-87ea-9fc5a9ba742f"
      },
      "cell_type": "code",
      "source": [
        "vocab_size = max(proc.id2token.keys()) + 1\n",
        "max_length = proc.padding_maxlen\n",
        "\n",
        "print('vocab size: ', vocab_size)\n",
        "print('max length allowed for documents: ', max_length)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "vocab size:  5002\n",
            "max length allowed for documents:  10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "srQ2EApW0v9X",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Language model architecture"
      ]
    },
    {
      "metadata": {
        "id": "V31lnrq70v9T",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Prepare training data for language model."
      ]
    },
    {
      "metadata": {
        "id": "Ioq7Sd9EPZq8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "ea91f806-7123-42ae-8260-4b6f49b6165c"
      },
      "cell_type": "code",
      "source": [
        "sequences = []\n",
        "for arr in tqdm(vecs):\n",
        "    non_zero = (arr != 0).argmax()\n",
        "    for i in range(non_zero, len(arr)):\n",
        "        sequences.append(arr[:i+1])\n",
        "sequences = pad_sequences(sequences, maxlen=max_length, padding='pre')\n",
        "sequences = np.array(sequences)\n",
        "X, y = sequences[:,:-1], sequences[:,-1]\n",
        "# y = to_categorical(y, num_classes=vocab_size)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 200000/200000 [00:01<00:00, 129203.69it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "WKW7TCl6PZq_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        },
        "outputId": "8ad8f59f-e862-41dc-b710-84d279ed0969"
      },
      "cell_type": "code",
      "source": [
        "i = Input(shape=(max_length-1,))\n",
        "x = Embedding(vocab_size, 256, input_length=max_length-1)(i)\n",
        "x = LSTM(256, return_sequences=True)(x)\n",
        "last_timestep = Lambda(lambda x: x[:, -1, :])(x)\n",
        "last_timestep = Dense(vocab_size, activation='softmax')(last_timestep)\n",
        "model = Model(i, last_timestep)\n",
        "model.summary()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         (None, 9)                 0         \n",
            "_________________________________________________________________\n",
            "embedding_1 (Embedding)      (None, 9, 256)            1280512   \n",
            "_________________________________________________________________\n",
            "lstm_1 (LSTM)                (None, 9, 256)            525312    \n",
            "_________________________________________________________________\n",
            "lambda_1 (Lambda)            (None, 256)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 5002)              1285514   \n",
            "=================================================================\n",
            "Total params: 3,091,338\n",
            "Trainable params: 3,091,338\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jnVeYIZV0v9a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training\n",
        "\n",
        "Now that we have created our architecture, we can train our model.  \n",
        "\n",
        "**We are going to train this model for only 1 epoch for illustration purposes.  We will load a pre-trained model that was run for 15 epochs.**"
      ]
    },
    {
      "metadata": {
        "id": "Psve_dWVPZrB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        },
        "outputId": "2bd77410-1a1a-43b5-b88c-68867d4b555e"
      },
      "cell_type": "code",
      "source": [
        "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "history = model.fit(X, y, epochs=1, batch_size=500, validation_split=0.1)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 1466839 samples, validate on 162983 samples\n",
            "Epoch 1/1\n",
            "1466839/1466839 [==============================] - 134s 91us/step - loss: 4.9483 - acc: 0.2260 - val_loss: 4.4829 - val_acc: 0.2671\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "gV_UzGzxGMot",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Load the cached model.  **Warning:** we can only use the cached model if the original `target_docs` variable was subset to the first 200k rows.  I will explain this more in the tutorial."
      ]
    },
    {
      "metadata": {
        "id": "wVv_1rA6GAn3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# This will load the cached model\n",
        "\n",
        "assert len(target_docs) == 200000, 'target_docs should be truncated to the first 200k rows to use the cached model.'\n",
        "\n",
        "fname = get_file(fname='kdd_lm_v2.h5', origin='https://storage.googleapis.com/kdd-seq2seq-2018/kdd_lm_v2.h5', )\n",
        "model = load_model(fname)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cFH1DmGHPZrE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Generate sequence\n",
        "\n",
        "The goal of a language model is to predict the next word in a sequence. To sanity check the language model, we will see what kind of sentence is generated when we start with a a seed word of 'is'. We are looking to see if the sentence generated appears to be sampled from the distribution of the data.\n",
        "\n",
        "In other words does the sentence generated look like it was written by the same author(s) pertaining to the same domain as the training corpus?"
      ]
    },
    {
      "metadata": {
        "id": "2R-2I_HRPZrE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def generate_seq(model, proc, n_words, seed_text):\n",
        "    in_text = seed_text\n",
        "    for _ in range(n_words):\n",
        "        vec = proc.transform([in_text])[:,1:]\n",
        "        index = np.argmax(model.predict(vec, verbose=0), axis=1)[0]\n",
        "        out_word = ''\n",
        "        if index == 1:\n",
        "            out_word = '_unk_'\n",
        "        else:\n",
        "            out_word = proc.id2token[index]\n",
        "        in_text += ' ' + out_word\n",
        "    return in_text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MCDhy30KqF2j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "See what sentence is generated from language model, seeded witht he word `is`."
      ]
    },
    {
      "metadata": {
        "id": "8wpHSxIOPZrH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1ff04a71-4929-49df-bcca-531ef9d4bb92"
      },
      "cell_type": "code",
      "source": [
        "generate_seq(model, proc, max_length, 'is')"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'is there a way to use ribbon toolbars in tkinter python'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "zzNZqTqVPZrJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Generate sentence embeddings\n",
        "\n",
        "One of the goals of training the language model is learning reprsentations of sentences in our corpus. "
      ]
    },
    {
      "metadata": {
        "id": "aF90GHpSraaK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "There are a plethora of general purpose pre-trained models that will generate high-quality embeddings of phrases (also called sentence embeddings). [This article](https://medium.com/huggingface/universal-word-sentence-embeddings-ce48ddc8fc3a) provides a great overview of the landscape. For example, Google’s universal sentence encoder works very well for many use cases and is available on [Tensorflow Hub](https://www.tensorflow.org/hub/modules/google/universal-sentence-encoder/1).\n",
        "\n",
        "Despite the convenience of these pre-trained models, it can be advantageous to train a model that captures the domain-specific vocabulary and semantics of docstrings. There are many techniques one can use to create sentence embeddings. These range from simple approaches, like averaging word vectors to more sophisticated techniques like those used in the construction of the universal sentence encoder.\n",
        "\n",
        "For this tutorial, we will leverage a the language model we trained earlier to generate embeddings for sentences.  It is important to carefully consider the corpus you use for training when building a language model. Ideally, you want to use a corpus that is of a similar domain to your downstream problem so you can adequately capture the relevant semantics and vocabulary. For example, a great corpus for this problem would be stack overflow data, since that is a forum that contains an extremely rich discussion of code. However, in order to keep this tutorial simple, we re-use the set of docstrings as our corpus. This is sub-optimal as discussions on stack overflow often contain richer semantic information than what is in a one-line docstring. We leave it as an exercise for the reader to examine the impact on the final outcome by using an alternate corpus.\n",
        "\n",
        "After we train the language model, our next task is to use this model to generate an embedding for each sentence. A common way of doing this is to summarize the hidden states of the language model.   A simple approach is to use aggregate stastics like the mean, max, or the sum of all the hidden states. There are other approaches that are outside the scope of this tutorial, and will discuss if time permits."
      ]
    },
    {
      "metadata": {
        "id": "fW1UeLKWqkMp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The below code extracts the hidden states from the encoder when given an input. There is one hidden state for each word in the sentence."
      ]
    },
    {
      "metadata": {
        "id": "Y-Rig9cePZrM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "embedding_model = Model(inputs=model.inputs, outputs=model.layers[-3].output)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "i6aZsEL24Vdt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can extract values from intermediate layers of this language model, and use those as sentence embeddings.  Here is how you can do that concretely with the language model we trained:"
      ]
    },
    {
      "metadata": {
        "id": "X-FFznLQPZrO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "outputId": "944430e0-ecdf-4f64-ee73-3f36eb417e48"
      },
      "cell_type": "code",
      "source": [
        "input_sequence = test_docs[random.randint(0, len(test_docs))]\n",
        "print('input sequence: ', input_sequence, '\\n\\nhidden states:\\n')\n",
        "vec = proc.transform([input_sequence])[:,1:]\n",
        "embedding_model.predict(vec)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "input sequence:  check whether file \"/path/to/file\" exists \n",
            "\n",
            "hidden states:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[[-4.06504214e-01,  0.00000000e+00, -0.00000000e+00, ...,\n",
              "          0.00000000e+00, -0.00000000e+00,  0.00000000e+00],\n",
              "        [-7.11692631e-01,  1.15522444e-01, -0.00000000e+00, ...,\n",
              "          0.00000000e+00, -0.00000000e+00,  0.00000000e+00],\n",
              "        [-0.00000000e+00,  6.63353682e-01, -8.49576890e-01, ...,\n",
              "          9.85891759e-01, -9.28904057e-01, -0.00000000e+00],\n",
              "        ...,\n",
              "        [-9.94829893e-01,  9.99988079e-01, -9.77638876e-04, ...,\n",
              "          5.43718189e-02, -0.00000000e+00,  0.00000000e+00],\n",
              "        [-9.94820595e-01,  0.00000000e+00,  7.61590958e-01, ...,\n",
              "          0.00000000e+00, -7.75839329e-01,  9.99471366e-01],\n",
              "        [-0.00000000e+00,  9.99999762e-01,  4.55019414e-01, ...,\n",
              "          0.00000000e+00, -0.00000000e+00,  8.32423046e-02]]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "metadata": {
        "id": "PBksOOsJrTkk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's extract the hidden states for all the sentences in our training data."
      ]
    },
    {
      "metadata": {
        "id": "haQjC19y0v9l",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "test_vecs = proc.transform(test_docs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rLeLELte0v9n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "hidden_states = embedding_model.predict(test_vecs[:, 1:])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9GjRzXnG4fT9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As mentioned earlier, we can compute aggregate statistics over the hidden states.  This is how you can do that:"
      ]
    },
    {
      "metadata": {
        "id": "SCf0kjpr0v9p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mean_vecs = np.mean(hidden_states, axis=1)\n",
        "max_vecs = np.max(hidden_states, axis=1)\n",
        "sum_vecs = np.sum(hidden_states, axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "giC48Kt00v9s",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Application - Nearest Neighbor Search\n",
        "\n",
        "Now that we have a way to represent each sentence as a vector, we can use this representation on many kinds of downstream tasks. One such task is finding a similar sentence to any given sentence.  \n",
        "\n",
        "\n",
        "### Build vector indices\n",
        "\n",
        "We will first place all the vectorized sentences in a special data structure that allows for fast nearest neighbor lookups. We will use [annoy](https://github.com/spotify/annoy) for this purpose."
      ]
    },
    {
      "metadata": {
        "id": "afBInH4C0v9s",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "2c4f24e9-bafc-4303-e8c3-19e680cfe76d"
      },
      "cell_type": "code",
      "source": [
        "dimension = hidden_states.shape[-1]\n",
        "index = AnnoyIndex(dimension)\n",
        "for i, v in enumerate(sum_vecs):\n",
        "    index.add_item(i, v)\n",
        "index.build(10)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "metadata": {
        "id": "pp2KAWNj0v9u",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Search nearest neighbors"
      ]
    },
    {
      "metadata": {
        "id": "tr_tKEDx0v9w",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "27be9780-eb8f-4202-8005-2d593d757a76"
      },
      "cell_type": "code",
      "source": [
        "input_sequence = test_docs[random.randint(0, len(test_docs))]\n",
        "print('Query: ', input_sequence)\n",
        "\n",
        "vec = proc.transform([input_sequence])[:,1:]\n",
        "vec = np.sum(embedding_model.predict(vec), axis=1)\n",
        "ids, _ = index.get_nns_by_vector(vec.T, 10, include_distances=True)\n",
        "\n",
        "print('\\nSearch Results:')\n",
        "[test_docs[i] for i in ids][1:]"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Query:  remove uppercased characters in string `s`\n",
            "\n",
            "Search Results:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"remove duplicate characters from string 'ffffffbbbbbbbqqq'\",\n",
              " 'remove first word in string `s`',\n",
              " 'remove all whitespaces in string `sentence`',\n",
              " 'remove newlines and whitespace from string `yourstring`',\n",
              " 'get digits in string `my_string`',\n",
              " 'Remove string \"1\" from string `string`',\n",
              " 'remove backslashes from string `result`',\n",
              " 'remove None value from list `L`',\n",
              " \"remove newline in string 'Mac EOL\\\\r'\"]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "metadata": {
        "id": "pnPW20o9PZrR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Sequence to Sequence Model"
      ]
    },
    {
      "metadata": {
        "id": "VpxbvU6OCDje",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "A [sequence to sequence model](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8) allows you to take an input sequence (source), and predict an output sequence (target).  These sequences can be anything, however we will focus on natural language for this tutorial. Sequence-to-sequence models have been used with great success in summarizing texts as well as generating translations from one language to another. For this tutorial, we will demonstrate a very creative task: given a snippet of code, we will train a model that generates a description of that code!\n",
        "\n",
        "### Sequence to Sequence Primer\n",
        "\n",
        "There are many variants of seq2seq models, however we will walk through one of the most simplest forms: an encoder-decoder network using RNNs.\n",
        "\n",
        "#### Training\n",
        "\n",
        "The decoder receives the ground truth, shifted by one time-step (is allowed to see the ground-truth of the previous time step).  This is called teacher forcing.   \n",
        "\n",
        "![alt text](https://blog.keras.io/img/seq2seq/seq2seq-teacher-forcing.png)\n",
        "\n",
        "#### Inference\n",
        "\n",
        "At inference time, we will not be able to see the ground truth from the last time step.   Therefore we can use the last predicted output in place of the previous time step's ground truth.  We will generate our sequence this way using a greedy approach, stopping only when we either reach a maximum length or predict a special <stop> token.  There are more sophisticated ways of generating sequences such as using [beam search](https://en.wikipedia.org/wiki/Beam_search) that we will not cover in this tutorial.\n",
        "\n",
        "![alt text](https://blog.keras.io/img/seq2seq/seq2seq-inference.png)\n",
        "\n",
        "Credit: https://blog.keras.io/category/tutorials.html\n",
        "\n",
        "\n",
        "**Building a neural network architecture is like stacking lego bricks.** For beginners, it can be useful to think of each layer as an API: you send the API some data and then the API returns some data. Thinking of things this way frees you from becoming overwhelmed, and you can build your understanding of things slowly. It is important to understand two concepts:\n",
        "\n",
        "the shape of data that each layer expects, and the shape of data the layer will return. (When you stack many layers on top of each other, the input and output shapes must be compatible, like legos).\n",
        "conceptually, what will the output(s) of a layer represent? What does the output of a subset of stacked layers represent?\n",
        "\n",
        "Let's take a look at the data we want to use. The `source` is the snippet of code and the `target` is the description of that code."
      ]
    },
    {
      "metadata": {
        "id": "ESQyR0l2CH01",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "19095f1e-2f0d-43cb-91e2-93feebddf3ba"
      },
      "cell_type": "code",
      "source": [
        "print('source (code input): ', source_docs[2])\n",
        "print('target (description output): ', target_docs[2])"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "source (code input):  c.decode('unicode_escape')\n",
            "target (description output):  Converting byte string in unicode string\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BQsnI6720v9z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Preprocessing\n",
        "\n",
        "Similar to previous excercises, we must pre-process the raw strings into a format that can be utilized by our model. One such format is to map each word in our corpus to a unique integer value, which we will refer to as a vocabulary. If the source and target are from the same distribution, (which they are not in this example) the vocabulary can be shared.\n",
        "\n",
        "\n",
        "Concretely, we will tokenize, generate vocabulary, apply padding and vectorize. These steps are as follows:\n",
        "\n",
        "**1. Tokenize:** Process of parsing strings into discrete words or tokens.\n",
        "\n",
        "**2. Generate Vocabulary:** Assign each token to a unique integer, rare-occuring tokens may be assigned to the same integer.\n",
        "\n",
        "**3. Padding:** We standardize the sequence length of each example to be the same by truncating and padding each example to the same lentgh.\n",
        "\n",
        "The `ktext` package helps us accomplish these steps."
      ]
    },
    {
      "metadata": {
        "id": "s_nE-SeKPZrS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "def9f314-8c1a-41ef-d241-9e14e03e4ab1"
      },
      "cell_type": "code",
      "source": [
        "source_proc = processor(hueristic_pct_padding=.7, keep_n=15000)\n",
        "source_vecs = source_proc.fit_transform(source_docs)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:....tokenizing data\n",
            "WARNING:root:Setting maximum document length to 10 based upon hueristic of 0.7 percentile.\n",
            " See full histogram by insepecting the `document_length_stats` attribute.\n",
            "WARNING:root:(1/2) done. 41 sec\n",
            "WARNING:root:....building corpus\n",
            "WARNING:root:(2/2) done. 1 sec\n",
            "WARNING:root:Finished parsing 200,000 documents.\n",
            "WARNING:root:...fit is finished, beginning transform\n",
            "WARNING:root:...padding data\n",
            "WARNING:root:done. 1 sec\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "rdSNtX3MOjIX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that we will pre-process the source documents in the same way as the language model.  The target documents, however will be processed in the same way with some subtle differences. "
      ]
    },
    {
      "metadata": {
        "id": "ePq40vC1Oew0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "4ea70752-16b8-4b14-c94a-21fb108bfd0c"
      },
      "cell_type": "code",
      "source": [
        "target_proc = processor(append_indicators=True, hueristic_pct_padding=.7, keep_n=12000, padding ='post')\n",
        "target_vecs = target_proc.fit_transform(target_docs)"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:....tokenizing data\n",
            "WARNING:root:Setting maximum document length to 15 based upon hueristic of 0.7 percentile.\n",
            " See full histogram by insepecting the `document_length_stats` attribute.\n",
            "WARNING:root:(1/2) done. 45 sec\n",
            "WARNING:root:....building corpus\n",
            "WARNING:root:(2/2) done. 1 sec\n",
            "WARNING:root:Finished parsing 200,000 documents.\n",
            "WARNING:root:...fit is finished, beginning transform\n",
            "WARNING:root:...padding data\n",
            "WARNING:root:done. 2 sec\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "qkfDO83HE8Xc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Above, we passed some additional parameters:\n",
        "\n",
        " - **append_indicators=True** will append the tokens ‘_start_’ and ‘_end_’ to the start and end of each document, respectively.\n",
        " \n",
        " - **padding=’post’** means that zero padding will be added to the end of the document instead of default of ‘pre’.\n",
        " \n",
        " \n",
        " The reason for processing the target documents in this way is that we want our model to know when the first letter of the docstring is supposed to occur, and also learn to predict when the end of a phrase should be. This will make more sense in the next section where model architecture is discussed.\n",
        "\n",
        "Additionally, we will use teacher forcing for the decoder of the sequence to sequence model, so we will offset the target sequence by one."
      ]
    },
    {
      "metadata": {
        "id": "3b0wFryNPZrU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "encoder_input_data = source_vecs\n",
        "encoder_seq_len = encoder_input_data.shape[1]\n",
        "\n",
        "decoder_input_data = target_vecs[:, :-1]\n",
        "decoder_target_data = target_vecs[:, 1:]\n",
        "\n",
        "num_encoder_tokens = max(source_proc.id2token.keys()) + 1\n",
        "num_decoder_tokens = max(target_proc.id2token.keys()) + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FCkBg1_1PZrZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Encoder model"
      ]
    },
    {
      "metadata": {
        "id": "wvDkJOqoFHhf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The role of the encoder is to extract features and generate a representation of the input sequence, which in this case is a snippet of code. "
      ]
    },
    {
      "metadata": {
        "id": "nThkgdknPZra",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_emb_dim=512\n",
        "hidden_state_dim=1024\n",
        "encoder_seq_len=encoder_seq_len\n",
        "num_encoder_tokens=num_encoder_tokens\n",
        "num_decoder_tokens=num_decoder_tokens\n",
        "\n",
        "encoder_inputs = Input(shape=(encoder_seq_len,), name='Encoder-Input')\n",
        "x = Embedding(num_encoder_tokens, word_emb_dim, name='Body-Word-Embedding', mask_zero=False)(encoder_inputs)\n",
        "x = BatchNormalization(name='Encoder-Batchnorm-1')(x)\n",
        "_, state_h = GRU(hidden_state_dim, return_state=True, name='Encoder-Last-GRU', dropout=.5)(x)\n",
        "encoder_model = Model(inputs=encoder_inputs, outputs=state_h, name='Encoder-Model')\n",
        "seq2seq_encoder_out = encoder_model(encoder_inputs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LrcGequcQFCT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292
        },
        "outputId": "60ce65bc-f9c8-49e3-a3ad-926634b472e9"
      },
      "cell_type": "code",
      "source": [
        "encoder_model.summary()"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "Encoder-Input (InputLayer)   (None, 10)                0         \n",
            "_________________________________________________________________\n",
            "Body-Word-Embedding (Embeddi (None, 10, 512)           7681024   \n",
            "_________________________________________________________________\n",
            "Encoder-Batchnorm-1 (BatchNo (None, 10, 512)           2048      \n",
            "_________________________________________________________________\n",
            "Encoder-Last-GRU (GRU)       [(None, 1024), (None, 102 4721664   \n",
            "=================================================================\n",
            "Total params: 12,404,736\n",
            "Trainable params: 12,403,712\n",
            "Non-trainable params: 1,024\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8PnsgReVPZre",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Decoder model"
      ]
    },
    {
      "metadata": {
        "id": "t9IbLqTCFQr3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The role of the decoder is to generate a description of the code conditioned on the features extracted by the encoder."
      ]
    },
    {
      "metadata": {
        "id": "bu-KgmfjPZre",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "decoder_inputs = Input(shape=(None,), name='Decoder-Input')\n",
        "dec_emb = Embedding(num_decoder_tokens, word_emb_dim, name='Decoder-Word-Embedding', mask_zero=False)(decoder_inputs)\n",
        "dec_bn = BatchNormalization(name='Decoder-Batchnorm-1')(dec_emb)\n",
        "decoder_gru = GRU(hidden_state_dim, return_state=True, return_sequences=True, name='Decoder-GRU', dropout=.5)\n",
        "decoder_gru_output, _ = decoder_gru(dec_bn, initial_state=seq2seq_encoder_out)\n",
        "x = BatchNormalization(name='Decoder-Batchnorm-2')(decoder_gru_output)\n",
        "decoder_dense = Dense(num_decoder_tokens, activation='softmax', name='Final-Output-Dense')\n",
        "decoder_outputs = decoder_dense(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WfH-WvGwPZrg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Sequence to sequence model"
      ]
    },
    {
      "metadata": {
        "id": "kW3bg3XnFYqx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can connect the encoder and decoder together to create the sequence to sequence model."
      ]
    },
    {
      "metadata": {
        "id": "xsNgKYlKPZri",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "seq2seq_model = Model([encoder_inputs, decoder_inputs], decoder_outputs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "26w4XvF7S337",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Summary of model architecture:"
      ]
    },
    {
      "metadata": {
        "id": "WZ1W2IrGSqZO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "outputId": "497f5d48-969d-42a1-af65-8f87ab2700fd"
      },
      "cell_type": "code",
      "source": [
        "seq2seq_model.summary()"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Decoder-Input (InputLayer)      (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Word-Embedding (Embeddi (None, None, 512)    5846528     Decoder-Input[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-Input (InputLayer)      (None, 10)           0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Batchnorm-1 (BatchNorma (None, None, 512)    2048        Decoder-Word-Embedding[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-Model (Model)           (None, 1024)         12404736    Encoder-Input[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-GRU (GRU)               [(None, None, 1024), 4721664     Decoder-Batchnorm-1[0][0]        \n",
            "                                                                 Encoder-Model[1][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Batchnorm-2 (BatchNorma (None, None, 1024)   4096        Decoder-GRU[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Final-Output-Dense (Dense)      (None, None, 11419)  11704475    Decoder-Batchnorm-2[0][0]        \n",
            "==================================================================================================\n",
            "Total params: 34,683,547\n",
            "Trainable params: 34,679,451\n",
            "Non-trainable params: 4,096\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "nvT-vl0TRMHI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://raw.githubusercontent.com/hohsiangwu/kdd-2018-hands-on-tutorials/master/images/seq2seq_model_architecture.svg?sanitize=true)"
      ]
    },
    {
      "metadata": {
        "id": "WT8maZlD0v-A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training"
      ]
    },
    {
      "metadata": {
        "id": "JxHP00h6FgL0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The below hyperparameters were found through some trial and error.\n",
        "\n",
        "**We will only train for one epoch in this tutorial for illustration purposes and load relevant model from cache.**"
      ]
    },
    {
      "metadata": {
        "id": "HgUemXCbPZrk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        },
        "outputId": "dabc98d6-f04b-4112-c565-541b3f2d2d8a"
      },
      "cell_type": "code",
      "source": [
        "batch_size = 1024\n",
        "epochs = 1\n",
        "\n",
        "seq2seq_model.compile(optimizer=optimizers.Nadam(lr=0.00005), loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
        "history = seq2seq_model.fit([encoder_input_data, decoder_input_data],\n",
        "                            np.expand_dims(decoder_target_data, -1),\n",
        "                            batch_size=batch_size,\n",
        "                            epochs=epochs,\n",
        "                            validation_split=0.1)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 180000 samples, validate on 20000 samples\n",
            "Epoch 1/1\n",
            "180000/180000 [==============================] - 266s 1ms/step - loss: 4.9942 - acc: 0.4057 - val_loss: 3.8051 - val_acc: 0.4637\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1ktyoRomRbOV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Load cached model. \n",
        "\n",
        "Due to anticipated compute constraints, we will allow you to load a pre-trained model, that was trained for 15 epochs."
      ]
    },
    {
      "metadata": {
        "id": "VmZ4a7tcRWbR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "outputId": "90dd6945-7a36-414a-e4bc-a9c73bc8a9d1"
      },
      "cell_type": "code",
      "source": [
        "!wget https://storage.googleapis.com/kdd-seq2seq-2018/kdd_seq2seq_weights.h5"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2018-08-22 09:12:36--  https://storage.googleapis.com/kdd-seq2seq-2018/kdd_seq2seq_weights.h5\r\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.20.128, 2607:f8b0:400e:c09::80\r\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.20.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 138765796 (132M) [application/octet-stream]\n",
            "Saving to: ‘kdd_seq2seq_weights.h5’\n",
            "\n",
            "kdd_seq2seq_weights 100%[===================>] 132.34M  42.2MB/s    in 3.1s    \n",
            "\n",
            "2018-08-22 09:12:39 (42.2 MB/s) - ‘kdd_seq2seq_weights.h5’ saved [138765796/138765796]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "q1mH7noRRXa-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "seq2seq_model.load_weights('kdd_seq2seq_weights.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Pau6RSHNTxIX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Recommendation of keeping track of different experiments:\n",
        "\n",
        "http://wandb.com\n",
        "\n",
        "(We will not be covering this in the tutorial)"
      ]
    },
    {
      "metadata": {
        "id": "1lbdGPHE0v-B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Extract encoder and decoder models"
      ]
    },
    {
      "metadata": {
        "id": "xcl66yGRF11e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To prepare the model for inference (to make predictions), we have to re-assemble it (with its trained weights intact) such that the decoder uses the last prediction as input rather than being fed the right answer for the previous time step, as illustrated below:\n",
        "\n",
        "![alt text](https://blog.keras.io/img/seq2seq/seq2seq-inference.png)"
      ]
    },
    {
      "metadata": {
        "id": "UAqk3fIwPZrn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def extract_decoder_model(model):\n",
        "    latent_dim = model.get_layer('Encoder-Model').output_shape[-1]\n",
        "    decoder_inputs = model.get_layer('Decoder-Input').input\n",
        "    dec_emb = model.get_layer('Decoder-Word-Embedding')(decoder_inputs)\n",
        "    dec_bn = model.get_layer('Decoder-Batchnorm-1')(dec_emb)\n",
        "    gru_inference_state_input = Input(shape=(latent_dim,), name='hidden_state_input')\n",
        "    gru_out, gru_state_out = model.get_layer('Decoder-GRU')([dec_bn, gru_inference_state_input])\n",
        "    dec_bn2 = model.get_layer('Decoder-Batchnorm-2')(gru_out)\n",
        "    dense_out = model.get_layer('Final-Output-Dense')(dec_bn2)\n",
        "    decoder_model = Model([decoder_inputs, gru_inference_state_input], [dense_out, gru_state_out])\n",
        "    return decoder_model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PU8JuOvfGKta",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "One side effect of training a sequence-to-sequence model in this way is that the encoder can be re-used as a general purpose feature extractor. We extract the encoder below for this purpose in a later exercise."
      ]
    },
    {
      "metadata": {
        "id": "Xe0_KakGPZrp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "7d7ceacc-de7b-4403-c7a7-ff326c3cece9"
      },
      "cell_type": "code",
      "source": [
        "encoder_model = seq2seq_model.get_layer('Encoder-Model')\n",
        "for layer in encoder_model.layers:\n",
        "    layer.trainable = False\n",
        "\n",
        "decoder_model = extract_decoder_model(seq2seq_model)\n",
        "decoder_model.summary()"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Decoder-Input (InputLayer)      (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Word-Embedding (Embeddi (None, None, 512)    5846528     Decoder-Input[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Batchnorm-1 (BatchNorma (None, None, 512)    2048        Decoder-Word-Embedding[1][0]     \n",
            "__________________________________________________________________________________________________\n",
            "hidden_state_input (InputLayer) (None, 1024)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-GRU (GRU)               [(None, None, 1024), 4721664     Decoder-Batchnorm-1[1][0]        \n",
            "                                                                 hidden_state_input[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Decoder-Batchnorm-2 (BatchNorma (None, None, 1024)   4096        Decoder-GRU[1][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Final-Output-Dense (Dense)      (None, None, 11419)  11704475    Decoder-Batchnorm-2[1][0]        \n",
            "==================================================================================================\n",
            "Total params: 22,278,811\n",
            "Trainable params: 22,275,739\n",
            "Non-trainable params: 3,072\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZFdwR6UR0v-L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Predict code descriptions using the trained sequence-to-sequence model\n",
        "\n",
        "You will see that the predicted descriptions are not perfect, but seem to be picking up on correlations between common code token sequences and natural language descriptions of that code.\n",
        "\n",
        "Feel free to run the below block of code as many times as you want. A new random sample from the test set will be drawn each time."
      ]
    },
    {
      "metadata": {
        "id": "JvK1eN9fPZrr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "outputId": "f170ee29-8c70-4b02-b1ac-82f860d8c61b"
      },
      "cell_type": "code",
      "source": [
        "i = random.randint(0, len(test_source_docs))\n",
        "\n",
        "max_len = target_proc.padding_maxlen\n",
        "raw_input_text = test_source_docs[i]\n",
        "\n",
        "raw_tokenized = source_proc.transform([raw_input_text])\n",
        "encoding = encoder_model.predict(raw_tokenized)\n",
        "original_encoding = encoding\n",
        "state_value = np.array(target_proc.token2id['_start_']).reshape(1, 1)\n",
        "\n",
        "decoded_sentence = []\n",
        "stop_condition = False\n",
        "while not stop_condition:\n",
        "    preds, st = decoder_model.predict([state_value, encoding])\n",
        "    pred_idx = np.argmax(preds[:, :, 2:]) + 2\n",
        "    pred_word_str = target_proc.id2token[pred_idx]\n",
        "\n",
        "    if pred_word_str == '_end_' or len(decoded_sentence) >= max_len:\n",
        "        stop_condition = True\n",
        "        break\n",
        "    decoded_sentence.append(pred_word_str)\n",
        "\n",
        "    # update the decoder for the next word\n",
        "    encoding = st\n",
        "    state_value = np.array(pred_idx).reshape(1, 1)\n",
        "\n",
        "print('sample code from test set:\\n------------------------\\n', raw_input_text)\n",
        "print('\\nground truth:\\n------------------------\\n', test_target_docs[i])\n",
        "print('\\npredicted description:\\n------------------------')\n",
        "print(' '.join(decoded_sentence))"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "sample code from test set:\n",
            "------------------------\n",
            " len(my_tuple)\n",
            "\n",
            "ground truth:\n",
            "------------------------\n",
            " Getting the length of an array\n",
            "\n",
            "predicted description:\n",
            "------------------------\n",
            "how to convert a list of tuples to a list of tuples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tf2bITb3Ny7H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Generate Embeddings\n",
        "\n",
        "We need two embeddings\n",
        "\n",
        "1.  Embeddings for the code snippets, from the seq2seq encoder.\n",
        "\n",
        "2. Embeddings for the docstrings, from the language model.\n",
        "\n",
        "\n",
        "\n",
        "Embeddings for the code snippets:"
      ]
    },
    {
      "metadata": {
        "id": "qYPM3Tqf0v-P",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_source_emb = encoder_model.predict(source_proc.transform(source_docs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "U9M3Uvslgnp9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Embeddings for the target documents, which are natural language summaries (like docstrings):"
      ]
    },
    {
      "metadata": {
        "id": "geb3SOyk0v-Q",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_target_vecs = proc.transform(target_docs)\n",
        "hidden_states = embedding_model.predict(train_target_vecs[:, 1:])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0cFA5iSciFOt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Summarize the hidden states from the language model."
      ]
    },
    {
      "metadata": {
        "id": "sFlbJj02iKTv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mean_vecs = np.mean(hidden_states, axis=1)\n",
        "max_vecs = np.max(hidden_states, axis=1)\n",
        "sum_vecs = np.sum(hidden_states, axis=1)\n",
        "train_target_emb = mean_vecs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zO9Cyj3qiP7F",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Check the shapes of each embedding."
      ]
    },
    {
      "metadata": {
        "id": "e_f5ptIL0v-R",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "990e80f4-d667-4ed1-bf1b-fea1351087c8"
      },
      "cell_type": "code",
      "source": [
        "print('source embedding shape on training set: ', train_source_emb.shape)\n",
        "print('target embedding shape on training set: ', train_target_emb.shape)"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "source embedding shape on training set:  (200000, 1024)\n",
            "target embedding shape on training set:  (200000, 200)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8xOKEWFj0v-S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Construct a Joint Vector Space (Semantic Code Search)\n",
        "\n",
        "Right now we have a way of representing:\n",
        "- a blob of code as a vector using the encoder of the sequence-to-sequence model, and \n",
        "- the code descriptions as a vector using the language model.\n",
        "\n",
        "However, these two vector spaces are not related to eachother. It can be useful to project the vectors for code and descriptions into the same space so that we can search code with natural language. There are many ways of accomplishing this task, however we will demonstrate a technique inspired from [this paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41473.pdf), where we use regression to \"pull\" these vectors into the same space.  This idea is further illustrated below:\n",
        "\n",
        "![alt text](https://cdn-images-1.medium.com/max/1280/1*zhLXNHK8ILaYV8tT-jDlOQ.png)"
      ]
    },
    {
      "metadata": {
        "id": "taGJtaGIZWCg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Review of the high-level process:  How do we build a joint vector space?  \n",
        "\n",
        "Surprise! You are almost there!  We have already completed steps 1 - 3 as illustrated below.   \n",
        "\n",
        "![alt text](https://raw.githubusercontent.com/hohsiangwu/kdd-2018-hands-on-tutorials/master/images/joint_space_diagram.svg?sanitize=true)"
      ]
    },
    {
      "metadata": {
        "id": "kEuoPiJnebrm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Most of the pieces for this step come from prior steps in this tutorial. In this step, we will fine-tune the seq2seq model  to predict docstring embeddings instead of docstrings. "
      ]
    },
    {
      "metadata": {
        "id": "-noqpPJY0v-T",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "outputId": "56b6d571-203f-443e-bdc8-5b4b06b67754"
      },
      "cell_type": "code",
      "source": [
        "inp = Input(shape=(train_source_emb.shape[1],))\n",
        "x = Dense(train_target_emb.shape[1], use_bias=False)(inp)\n",
        "# x = BatchNormalization()(x)\n",
        "# x = Dense(train_target_emb.shape[1])(x)\n",
        "modal_model = Model([inp], x)\n",
        "modal_model.summary()"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         (None, 1024)              0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 200)               204800    \n",
            "=================================================================\n",
            "Total params: 204,800\n",
            "Trainable params: 204,800\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7zALzqWO0v-U",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1020
        },
        "outputId": "a0b0bb83-3c1b-4dbf-e78a-b354c53ea52a"
      },
      "cell_type": "code",
      "source": [
        "modal_model.compile(optimizer=optimizers.Nadam(lr=0.0001), loss='cosine_proximity', metrics=['accuracy'])\n",
        "\n",
        "batch_size = 1200\n",
        "epochs = 20\n",
        "history = modal_model.fit([train_source_emb], train_target_emb,\n",
        "                          batch_size=batch_size, epochs=epochs, validation_split=0.1)"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 180000 samples, validate on 20000 samples\n",
            "Epoch 1/20\n",
            "180000/180000 [==============================] - 2s 11us/step - loss: -0.6479 - acc: 0.0472 - val_loss: -0.7270 - val_acc: 0.0734\n",
            "Epoch 2/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7389 - acc: 0.0820 - val_loss: -0.7373 - val_acc: 0.0864\n",
            "Epoch 3/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7461 - acc: 0.0900 - val_loss: -0.7410 - val_acc: 0.0909\n",
            "Epoch 4/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7492 - acc: 0.0956 - val_loss: -0.7430 - val_acc: 0.0965\n",
            "Epoch 5/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7511 - acc: 0.0993 - val_loss: -0.7442 - val_acc: 0.1002\n",
            "Epoch 6/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7523 - acc: 0.1018 - val_loss: -0.7451 - val_acc: 0.1022\n",
            "Epoch 7/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7533 - acc: 0.1031 - val_loss: -0.7457 - val_acc: 0.1032\n",
            "Epoch 8/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7540 - acc: 0.1042 - val_loss: -0.7462 - val_acc: 0.1044\n",
            "Epoch 9/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7546 - acc: 0.1049 - val_loss: -0.7466 - val_acc: 0.1048\n",
            "Epoch 10/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7550 - acc: 0.1066 - val_loss: -0.7470 - val_acc: 0.1059\n",
            "Epoch 11/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7555 - acc: 0.1071 - val_loss: -0.7472 - val_acc: 0.1067\n",
            "Epoch 12/20\n",
            "180000/180000 [==============================] - 2s 10us/step - loss: -0.7558 - acc: 0.1078 - val_loss: -0.7475 - val_acc: 0.1086\n",
            "Epoch 13/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7562 - acc: 0.1084 - val_loss: -0.7477 - val_acc: 0.1089\n",
            "Epoch 14/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7565 - acc: 0.1088 - val_loss: -0.7479 - val_acc: 0.1078\n",
            "Epoch 15/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7567 - acc: 0.1091 - val_loss: -0.7481 - val_acc: 0.1104\n",
            "Epoch 16/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7570 - acc: 0.1098 - val_loss: -0.7483 - val_acc: 0.1103\n",
            "Epoch 17/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7572 - acc: 0.1099 - val_loss: -0.7484 - val_acc: 0.1103\n",
            "Epoch 18/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7574 - acc: 0.1106 - val_loss: -0.7485 - val_acc: 0.1103\n",
            "Epoch 19/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7576 - acc: 0.1109 - val_loss: -0.7487 - val_acc: 0.1099\n",
            "Epoch 20/20\n",
            "180000/180000 [==============================] - 1s 8us/step - loss: -0.7578 - acc: 0.1112 - val_loss: -0.7488 - val_acc: 0.1102\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "YBUfnKEIaMrw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Application - Semantic Search"
      ]
    },
    {
      "metadata": {
        "id": "bPaOVL3sw30g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Warning: the results of this portion are minimal and not very good.  We leave it as an exercise to the reader to improve the results.  A more thorough example of this application is [here](https://towardsdatascience.com/semantic-code-search-3cd6d244a39c)."
      ]
    },
    {
      "metadata": {
        "id": "8EBmYWFstUQk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**1. Vectorize all of the code.**  We are going to push all the code through the encoder -> then modal_model to project the code into the same vector-space as our phrase embeddings. "
      ]
    },
    {
      "metadata": {
        "id": "U-u1WX4EQa83",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3675f131-dfe1-420e-a6d6-97d49415a0d8"
      },
      "cell_type": "code",
      "source": [
        "vec_code = modal_model.predict(encoder_model.predict(source_proc.transform(source_docs)))\n",
        "vec_code.shape"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(200000, 200)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "metadata": {
        "id": "JCfFh08PtlPp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**2. Place all vectorized code into a search index.** We are going to use Spotify's [annoy](https://github.com/spotify/annoy).  There are other libraries that are good like [nmslib](https://github.com/nmslib/nmslib)."
      ]
    },
    {
      "metadata": {
        "id": "P3fJvwROaZNo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a01bf3c0-4075-4575-ddb2-7c180125ed07"
      },
      "cell_type": "code",
      "source": [
        "dimension = vec_code.shape[-1]\n",
        "index = AnnoyIndex(dimension)\n",
        "for i, v in enumerate(vec_code):\n",
        "    index.add_item(i, v)\n",
        "index.build(10)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "metadata": {
        "id": "VU02muFRuuoI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**3. Create Helper Function To Vectorize Queries.** we are going to use the language model to create \"sentence embeddings\" for our queries.  "
      ]
    },
    {
      "metadata": {
        "id": "YPas7DOZb-KV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def vectorize_string(txt):\n",
        "  vec = proc.transform([txt])[:,1:]\n",
        "  emb = np.mean(embedding_model.predict(vec), axis=1)\n",
        "  return emb"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3S7D_Stgu6oP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**4.  Given a query, lookup the closest code in vector space.** . We can wrap everything in one function that:\n",
        "\n",
        "- vectorizes a string query\n",
        "- lookup the nearest vectorized code (which is in the same vector-space as the string query)\n",
        "- retrieve the original code and display it for the user."
      ]
    },
    {
      "metadata": {
        "id": "m4GJg9GZcOgE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def search(inp):\n",
        "  inp = vectorize_string(inp)\n",
        "  ids, dist = index.get_nns_by_vector(inp.squeeze(), 10, include_distances=True)\n",
        "\n",
        "  for i, dist in zip(ids, dist):\n",
        "    print(f'dist: {dist:.2f}\\n{source_docs[i]}\\n')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WjJclTwivRdF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### The Demo"
      ]
    },
    {
      "metadata": {
        "id": "L2fOY4Dvhwmd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 547
        },
        "outputId": "f9dc1b70-a24d-4bd5-ac1e-38059b6dea6c"
      },
      "cell_type": "code",
      "source": [
        "search('read csv file into pandas dataframe')"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dist: 0.45\n",
            "pandas.read_json('input.json').to_excel('output.xlsx')\n",
            "\n",
            "dist: 0.45\n",
            "return pd.read_json(json.dumps(r)).unstack()\n",
            "\n",
            "dist: 0.46\n",
            "data_df = pd.read_json(data_json_str)\n",
            "\n",
            "dist: 0.46\n",
            "workbook = xlrd.open_workbook('test.xlsx')\n",
            "\n",
            "dist: 0.46\n",
            "wb = xlrd.open_workbook('cell_formula_test.xlsx')\n",
            "\n",
            "dist: 0.46\n",
            "workbook = xlrd.open_workbook('test.xlsm')\n",
            "\n",
            "dist: 0.46\n",
            "data = pd.read_json('/path/to/file.json', lines=True)\n",
            "\n",
            "dist: 0.46\n",
            "worksheet.write(row, col + 1, item)\n",
            "\n",
            "dist: 0.46\n",
            "worksheet.write(row, col + 1, item)\n",
            "\n",
            "dist: 0.46\n",
            "df2 = pd.read_json('test_data_01.json')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ib2Pzzl5tE3H",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 547
        },
        "outputId": "aed9305b-4a13-4adf-cae1-505675ecc4ee"
      },
      "cell_type": "code",
      "source": [
        "search('sort list in ascending order')"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dist: 0.50\n",
            "bids.append(bid)\n",
            "\n",
            "dist: 0.52\n",
            "bids.append(int(bid))\n",
            "\n",
            "dist: 0.53\n",
            "brr.sort()\n",
            "\n",
            "dist: 0.53\n",
            "brr[:] = brr[::-1]\n",
            "\n",
            "dist: 0.53\n",
            "brr = brr[::-1]\n",
            "\n",
            "dist: 0.54\n",
            "intbids.append(int(bid))\n",
            "\n",
            "dist: 0.54\n",
            "res1[:, (k)] = U[:, :, (k)].dot(V[:, (k)])\n",
            "\n",
            "dist: 0.54\n",
            "bid = int(bid)\n",
            "\n",
            "dist: 0.54\n",
            "bid = int(bid)\n",
            "\n",
            "dist: 0.55\n",
            "yield t[:j] + (t[i],) + t[j:]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "p2USqf4029Nq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# What's next"
      ]
    },
    {
      "metadata": {
        "id": "aZMUgbDM3CaX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "1. Optimization\n",
        "  - Hyperparameter tuning\n",
        "    - Grid search\n",
        "    - Bayesian optimization\n",
        "  - [Fairseq](https://github.com/pytorch/fairseq): Dynamic padding, multi-GPU, multiple instances\n",
        "2. Try different model architecture\n",
        "  - Transformer: [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf)\n",
        "3. Try different preprocessing steps\n",
        "  - Introduce more priors based on domain specific knowledge\n",
        "    - [Graph neural network](https://github.com/Microsoft/gated-graph-neural-network-samples)"
      ]
    },
    {
      "metadata": {
        "id": "M41W_C4M3D7C",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}