Ngram_Language_Model_with_NLTK.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4rc1"
    },
    "colab": {
      "name": "Ngram Language Model with NLTK.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mohameddhameem/LearnPyTorch/blob/master/Ngram_Language_Model_with_NLTK.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "65559326e76d2b7841244e1dd6ea448671411b95",
        "id": "aO7EQ7JVcJOb",
        "colab_type": "text"
      },
      "source": [
        "# N-grams Language Models (N-grams LM)\n",
        "\n",
        "Nowadays, everything seems to be going neural... \n",
        "\n",
        "Traditionally, we can use n-grams to generate language models to predict which word comes next given a history of words. \n",
        "\n",
        "We'll use the `lm` module in `nltk` to get a sense of how non-neural language modelling is done.\n",
        "\n",
        "(**Source:** The content in this notebook is largely based on [language model tutorial in NLTK documentation by Ilia Kurenkov](https://github.com/nltk/nltk/blob/develop/nltk/lm/__init__.py))"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "10bbe6360f9659217f9f40455301aa72d6b614a6",
        "id": "FzgZn2DQcJOO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "outputId": "f64e05be-d667-4ae3-a801-e556f29a5eba"
      },
      "source": [
        "!pip install -U pip\n",
        "!pip install -U dill\n",
        "!pip install -U nltk"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting pip\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)\n",
            "\r\u001b[K     |▎                               | 10kB 17.1MB/s eta 0:00:01\r\u001b[K     |▌                               | 20kB 3.1MB/s eta 0:00:01\r\u001b[K     |▊                               | 30kB 4.2MB/s eta 0:00:01\r\u001b[K     |█                               | 40kB 3.0MB/s eta 0:00:01\r\u001b[K     |█▏                              | 51kB 2.9MB/s eta 0:00:01\r\u001b[K     |█▍                              | 61kB 3.5MB/s eta 0:00:01\r\u001b[K     |█▋                              | 71kB 3.7MB/s eta 0:00:01\r\u001b[K     |█▉                              | 81kB 4.1MB/s eta 0:00:01\r\u001b[K     |██                              | 92kB 4.5MB/s eta 0:00:01\r\u001b[K     |██▎                             | 102kB 4.7MB/s eta 0:00:01\r\u001b[K     |██▌                             | 112kB 4.7MB/s eta 0:00:01\r\u001b[K     |██▊                             | 122kB 4.7MB/s eta 0:00:01\r\u001b[K     |███                             | 133kB 4.7MB/s eta 0:00:01\r\u001b[K     |███▏                            | 143kB 4.7MB/s eta 0:00:01\r\u001b[K     |███▍                            | 153kB 4.7MB/s eta 0:00:01\r\u001b[K     |███▋                            | 163kB 4.7MB/s eta 0:00:01\r\u001b[K     |███▉                            | 174kB 4.7MB/s eta 0:00:01\r\u001b[K     |████                            | 184kB 4.7MB/s eta 0:00:01\r\u001b[K     |████▎                           | 194kB 4.7MB/s eta 0:00:01\r\u001b[K     |████▌                           | 204kB 4.7MB/s eta 0:00:01\r\u001b[K     |████▊                           | 215kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████                           | 225kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████▎                          | 235kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████▌                          | 245kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████▊                          | 256kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████                          | 266kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████▏                         | 276kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████▍                         | 286kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████▋                         | 296kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████▉                         | 307kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████                         | 317kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████▎                        | 327kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████▌                        | 337kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████▊                        | 348kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████                        | 358kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████▏                       | 368kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████▍                       | 378kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████▋                       | 389kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████▉                       | 399kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████                       | 409kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████▎                      | 419kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 430kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 440kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████                      | 450kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████▎                     | 460kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████▌                     | 471kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 481kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████                     | 491kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 501kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████▍                    | 512kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████▋                    | 522kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 532kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████                    | 542kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████▎                   | 552kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████▌                   | 563kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 573kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 583kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████▏                  | 593kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████▍                  | 604kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████▋                  | 614kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████▉                  | 624kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 634kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 645kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████▌                 | 655kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 665kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████                 | 675kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████▎                | 686kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████▌                | 696kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████▊                | 706kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████                | 716kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 727kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████▍               | 737kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████▋               | 747kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████▉               | 757kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 768kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████▎              | 778kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████▌              | 788kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████▊              | 798kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 808kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████▏             | 819kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████▍             | 829kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████▋             | 839kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████▉             | 849kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 860kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████▎            | 870kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 880kB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████▉            | 890kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████            | 901kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 911kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▌           | 921kB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▊           | 931kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████           | 942kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████▏          | 952kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████▍          | 962kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████▋          | 972kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████▉          | 983kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 993kB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████▎         | 1.0MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████▌         | 1.0MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████▊         | 1.0MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 1.0MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████▏        | 1.0MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████▍        | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████▋        | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████▉        | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████▋       | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████▉       | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▎      | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▌      | 1.1MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▊      | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▏     | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▍     | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▋     | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▉     | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▎    | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▌    | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▊    | 1.2MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▏   | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▍   | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▋   | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▉   | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▋  | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▉  | 1.3MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▎ | 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▊ | 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▏| 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▍| 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▋| 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▉| 1.4MB 4.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 1.4MB 4.7MB/s \n",
            "\u001b[?25hInstalling collected packages: pip\n",
            "  Found existing installation: pip 19.3.1\n",
            "    Uninstalling pip-19.3.1:\n",
            "      Successfully uninstalled pip-19.3.1\n",
            "Successfully installed pip-20.0.2\n",
            "Requirement already up-to-date: dill in /usr/local/lib/python3.6/dist-packages (0.3.1.1)\n",
            "Collecting nltk\n",
            "  Downloading nltk-3.4.5.zip (1.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.5 MB 4.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.12.0)\n",
            "Building wheels for collected packages: nltk\n",
            "  Building wheel for nltk (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for nltk: filename=nltk-3.4.5-py3-none-any.whl size=1449904 sha256=3f29d4644825b2ded67675ec9ddc0bd37468791bd65198ae180392fdd4381b48\n",
            "  Stored in directory: /root/.cache/pip/wheels/e3/c9/b0/ed26a73ef75a53145820825afa8e2d2c9b30fe9f6c10cd3202\n",
            "Successfully built nltk\n",
            "Installing collected packages: nltk\n",
            "  Attempting uninstall: nltk\n",
            "    Found existing installation: nltk 3.2.5\n",
            "    Uninstalling nltk-3.2.5:\n",
            "      Successfully uninstalled nltk-3.2.5\n",
            "Successfully installed nltk-3.4.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "1b213fb0dd5534ce82d6f1c716e9695ba5cf9758",
        "id": "v5R89ym9cJOd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.util import pad_sequence\n",
        "from nltk.util import bigrams\n",
        "from nltk.util import ngrams\n",
        "from nltk.util import everygrams\n",
        "from nltk.lm.preprocessing import pad_both_ends\n",
        "from nltk.lm.preprocessing import flatten"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "800f7632b6a834d36b95211a912b390fcb9dc11b",
        "id": "XRJFlK25cJOo",
        "colab_type": "text"
      },
      "source": [
        "If we want to train a bigram model, we need to turn this text into bigrams. Here's what the first sentence of our text would look like if we use the `ngrams` function from NLTK for this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "1c1716261478af5e75bba7799bfefc10bd1ea4ea",
        "id": "hcNftyI0cJOp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = [['a', 'b', 'c'], ['a', 'c', 'd', 'c', 'e', 'f']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "a5a1c775bab7cf9c67656d4349d8bfca02a80738",
        "id": "mBB_v_lXcJOt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ee0bc1b6-1e7c-4d70-fe48-934d57a723be"
      },
      "source": [
        "list(bigrams(text[0]))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('a', 'b'), ('b', 'c')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "0219988318e39dd0576913b8087af8b35cbdab2f",
        "id": "fdtYmxPQcJOy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "00148a44-f601-48e2-ffa8-a48005b76d71"
      },
      "source": [
        "list(ngrams(text[1], n=3))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('a', 'c', 'd'), ('c', 'd', 'c'), ('d', 'c', 'e'), ('c', 'e', 'f')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "e3a92b25dc25f3ae86fd265880b7f410e981c670",
        "id": "6z_pXLuycJO0",
        "colab_type": "text"
      },
      "source": [
        "Notice how \"b\" occurs both as the first and second member of different bigrams but \"a\" and \"c\" don't? \n",
        "\n",
        "Wouldn't it be nice to somehow indicate how often sentences start with \"a\" and end with \"c\"?\n",
        "\n",
        "\n",
        "A standard way to deal with this is to add special \"padding\" symbols to the sentence before splitting it into ngrams. Fortunately, NLTK also has a function for that, let's see what it does to the first sentence.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "1ec58c94f58429c9160b4496c939f211f88ade54",
        "id": "rt-d1IOxcJO1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1349c92c-82c6-4c0c-de3c-9f8f5a13a8cf"
      },
      "source": [
        "from nltk.util import pad_sequence\n",
        "list(pad_sequence(text[0],\n",
        "                  pad_left=True, left_pad_symbol=\"<s>\",\n",
        "                  pad_right=True, right_pad_symbol=\"</s>\",\n",
        "                  n=2)) # The n order of n-grams, if it's 2-grams, you pad once, 3-grams pad twice, etc. "
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', 'a', 'b', 'c', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "ae38644238aa46ac100381c0de046972090b093b",
        "id": "14K44sTxcJO3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "df7f3d21-0c6f-4f82-ee93-a22cea7c32c2"
      },
      "source": [
        "padded_sent = list(pad_sequence(text[0], pad_left=True, left_pad_symbol=\"<s>\", \n",
        "                                pad_right=True, right_pad_symbol=\"</s>\", n=2))\n",
        "list(ngrams(padded_sent, n=2))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "4cda3fcbd1b8ab48431240e04168011953ebc913",
        "id": "fvioLWyicJO6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8aaa67c0-f8a3-4a27-fca0-328d3d30314b"
      },
      "source": [
        "list(pad_sequence(text[0],\n",
        "                  pad_left=True, left_pad_symbol=\"<s>\",\n",
        "                  pad_right=True, right_pad_symbol=\"</s>\",\n",
        "                  n=3)) # The n order of n-grams, if it's 2-grams, you pad once, 3-grams pad twice, etc. "
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', '<s>', 'a', 'b', 'c', '</s>', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "bd572a57371a716471d156144e7ebc136072de01",
        "id": "N2zVIdRbcJO8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "1faaf9d1-343c-4994-d622-fcb6fa90b6e3"
      },
      "source": [
        "padded_sent = list(pad_sequence(text[0], pad_left=True, left_pad_symbol=\"<s>\", \n",
        "                                pad_right=True, right_pad_symbol=\"</s>\", n=3))\n",
        "list(ngrams(padded_sent, n=3))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('<s>', '<s>', 'a'),\n",
              " ('<s>', 'a', 'b'),\n",
              " ('a', 'b', 'c'),\n",
              " ('b', 'c', '</s>'),\n",
              " ('c', '</s>', '</s>')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "e4d80e5707a2efa7ea5cee601d07bca23bc0ae87",
        "id": "lxTz_6BicJO-",
        "colab_type": "text"
      },
      "source": [
        "Note the `n` argument, that tells the function we need padding for bigrams.\n",
        "\n",
        "Now, passing all these parameters every time is tedious and in most cases they can be safely assumed as defaults anyway.\n",
        "\n",
        "Thus the `nltk.lm` module provides a convenience function that has all these arguments already set while the other arguments remain the same as for `pad_sequence`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "eb64c3c2a205c5fc5e1fce6f63c3e9038f0d8c4a",
        "id": "OfhjdnLIcJO_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8a8a70b0-559f-42f3-a5b4-8e343537b78a"
      },
      "source": [
        "from nltk.lm.preprocessing import pad_both_ends\n",
        "list(pad_both_ends(text[0], n=2))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', 'a', 'b', 'c', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2d2581fae73d38d623a77b5d523c957df2dc8478",
        "id": "2zAhbvoXcJPC",
        "colab_type": "text"
      },
      "source": [
        "Combining the two parts discussed so far we get the following preparation steps for one sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "1168917cc340d400f6dba9b2703561785481d8e4",
        "id": "_AjXsHWLcJPD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6c54ce21-41ee-4c37-bb12-e17af05db972"
      },
      "source": [
        "list(bigrams(pad_both_ends(text[0], n=2)))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2778a5e5b8d95bc395ba41af491818140c7f680e",
        "id": "f9Jtwy-mcJPF",
        "colab_type": "text"
      },
      "source": [
        "To make our model more robust we could also train it on unigrams (single words) as well as bigrams, its main source of information.\n",
        "NLTK once again helpfully provides a function called `everygrams`.\n",
        "\n",
        "While not the most efficient, it is conceptually simple."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "dfdc9effca718a7881f1fdc4562cdfb29164b96c",
        "id": "s0t6WKTGcJPG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "33b3dd67-c917-4c19-afb5-b64f83ed11e6"
      },
      "source": [
        "from nltk.util import everygrams\n",
        "padded_bigrams = list(pad_both_ends(text[0], n=2))\n",
        "list(everygrams(padded_bigrams, max_len=2))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('<s>',),\n",
              " ('a',),\n",
              " ('b',),\n",
              " ('c',),\n",
              " ('</s>',),\n",
              " ('<s>', 'a'),\n",
              " ('a', 'b'),\n",
              " ('b', 'c'),\n",
              " ('c', '</s>')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "d74b0ba683b733d5a83756361a905625a841e68d",
        "id": "7UX4ViUacJPI",
        "colab_type": "text"
      },
      "source": [
        "We are almost ready to start counting ngrams, just one more step left.\n",
        "\n",
        "During training and evaluation our model will rely on a vocabulary that defines which words are \"known\" to the model.\n",
        "\n",
        "To create this vocabulary we need to pad our sentences (just like for counting ngrams) and then combine the sentences into one flat stream of words.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "310c937ea7cc847a26f6c3d08c7169e8cd6b354a",
        "id": "eOTYB9AscJPI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "596043f2-63d6-48a3-c913-f146affe7a94"
      },
      "source": [
        "from nltk.lm.preprocessing import flatten\n",
        "list(flatten(pad_both_ends(sent, n=2) for sent in text))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', 'a', 'b', 'c', '</s>', '<s>', 'a', 'c', 'd', 'c', 'e', 'f', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "5bab7d656c474d945669e6a86bf4beec3a44189e",
        "id": "dFYdsvQgcJPK",
        "colab_type": "text"
      },
      "source": [
        "In most cases we want to use the same text as the source for both vocabulary and ngram counts.\n",
        "\n",
        "Now that we understand what this means for our preprocessing, we can simply import a function that does everything for us."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "147ccdd961ac97492ee9f0eadf3dfce7325c6790",
        "id": "CV0_obsUcJPK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.lm.preprocessing import padded_everygram_pipeline\n",
        "train, vocab = padded_everygram_pipeline(2, text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "b1b15504f2fb4487deec19de1c5ca786e49a42d6",
        "id": "aG4PbcYocJPM",
        "colab_type": "text"
      },
      "source": [
        "So as to avoid re-creating the text in memory, both `train` and `vocab` are lazy iterators. They are evaluated on demand at training time.\n",
        "\n",
        "For the sake of understanding the output of `padded_everygram_pipeline`, we'll \"materialize\" the lazy iterators by casting them into a list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "aa18a8a4a700926c30488c94cf519d8900de8ddf",
        "id": "DsNrKXsGcJPN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        },
        "outputId": "e8b46b6a-c9f7-486b-926f-5dbd4fa5c21e"
      },
      "source": [
        "training_ngrams, padded_sentences = padded_everygram_pipeline(2, text)\n",
        "for ngramlize_sent in training_ngrams:\n",
        "    print(list(ngramlize_sent))\n",
        "    print()\n",
        "print('#############')\n",
        "list(padded_sentences)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('<s>',), ('a',), ('b',), ('c',), ('</s>',), ('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]\n",
            "\n",
            "[('<s>',), ('a',), ('c',), ('d',), ('c',), ('e',), ('f',), ('</s>',), ('<s>', 'a'), ('a', 'c'), ('c', 'd'), ('d', 'c'), ('c', 'e'), ('e', 'f'), ('f', '</s>')]\n",
            "\n",
            "#############\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', 'a', 'b', 'c', '</s>', '<s>', 'a', 'c', 'd', 'c', 'e', 'f', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "4638a871a58694a77473a590728ebf25047cd6eb",
        "id": "Lt35fRLmcJPP",
        "colab_type": "text"
      },
      "source": [
        "## Lets get some real data and tokenize it"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "c679084566e711286d861a622e21ca5c04db7bee",
        "id": "HAa_rDV_cJPP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try: # Use the default NLTK tokenizer.\n",
        "    from nltk import word_tokenize, sent_tokenize \n",
        "    # Testing whether it works. \n",
        "    # Sometimes it doesn't work on some machines because of setup issues.\n",
        "    word_tokenize(sent_tokenize(\"This is a foobar sentence. Yes it is.\")[0])\n",
        "except: # Use a naive sentence tokenizer and toktok.\n",
        "    import re\n",
        "    from nltk.tokenize import ToktokTokenizer\n",
        "    # See https://stackoverflow.com/a/25736515/610569\n",
        "    sent_tokenize = lambda x: re.split(r'(?<=[^A-Z].[.?]) +(?=[A-Z])', x)\n",
        "    # Use the toktok tokenizer that requires no dependencies.\n",
        "    toktok = ToktokTokenizer()\n",
        "    word_tokenize = word_tokenize = toktok.tokenize"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "9bb181f26fb777464b7d4e6d08beac724864cefe",
        "id": "NYsO3aErcJPR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import requests\n",
        "import io #codecs\n",
        "\n",
        "\n",
        "# Text version of https://kilgarriff.co.uk/Publications/2005-K-lineer.pdf\n",
        "if os.path.isfile('language-never-random.txt'):\n",
        "    with io.open('language-never-random.txt', encoding='utf8') as fin:\n",
        "        text = fin.read()\n",
        "else:\n",
        "    url = \"https://gist.githubusercontent.com/alvations/53b01e4076573fea47c6057120bb017a/raw/b01ff96a5f76848450e648f35da6497ca9454e4a/language-never-random.txt\"\n",
        "    text = requests.get(url).content.decode('utf8')\n",
        "    with io.open('language-never-random.txt', 'w', encoding='utf8') as fout:\n",
        "        fout.write(text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "af4102fcbf643e09319ca6394c175ca64082fd19",
        "id": "hY65XoyVcJPT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Tokenize the text.\n",
        "tokenized_text = [list(map(str.lower, word_tokenize(sent))) \n",
        "                  for sent in sent_tokenize(text)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "9daa634f65b63882f7192a9489fba198970f3771",
        "id": "BLFT7iqJcJPU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        },
        "outputId": "fb6c64fe-1bb1-4bf0-f35e-d592a147d888"
      },
      "source": [
        "tokenized_text[0]"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['language',\n",
              " 'is',\n",
              " 'never',\n",
              " ',',\n",
              " 'ever',\n",
              " ',',\n",
              " 'ever',\n",
              " ',',\n",
              " 'random',\n",
              " 'adam',\n",
              " 'kilgarriff',\n",
              " 'abstract',\n",
              " 'language',\n",
              " 'users',\n",
              " 'never',\n",
              " 'choose',\n",
              " 'words',\n",
              " 'randomly',\n",
              " ',',\n",
              " 'and',\n",
              " 'language',\n",
              " 'is',\n",
              " 'essentially',\n",
              " 'non-random',\n",
              " '.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "91c93337cfea53b0c15a6e8617395dab872fb140",
        "id": "WPJOO3bzcJPW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "714080e9-ba26-43e4-fcd2-63bc02a97a97"
      },
      "source": [
        "print(text[:500])"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                       Language is never, ever, ever, random\n",
            "\n",
            "                                                               ADAM KILGARRIFF\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Abstract\n",
            "Language users never choose words randomly, and language is essentially\n",
            "non-random. Statistical hypothesis testing uses a null hypothesis, which\n",
            "posits randomness. Hence, when we look at linguistic phenomena in cor-\n",
            "pora, the null hypothesis will never be true. Moreover, where there is enough\n",
            "data, we shall (almost) always be able to establish \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "3eef00b8b1a95043e1c8bdd6912ae22060198e5c",
        "id": "z06qzUnXcJPY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Preprocess the tokenized text for 3-grams language modelling\n",
        "n = 3\n",
        "train_data, padded_sents = padded_everygram_pipeline(n, tokenized_text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "32c5ad89e3e61b143e940d50a0d0ee602dadfe3f",
        "id": "mnvRBG1DcJPZ",
        "colab_type": "text"
      },
      "source": [
        "# Training an N-gram Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "ff1ff506e3a14df8283cb93b0a45f86862d3e3c7",
        "id": "4WFrg0oNcJPa",
        "colab_type": "text"
      },
      "source": [
        "Having prepared our data we are ready to start training a model. As a simple example, let us train a Maximum Likelihood Estimator (MLE).\n",
        "\n",
        "We only need to specify the highest ngram order to instantiate it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "2635458b0f3fed618bfae6a0705ccb11555bfb0b",
        "id": "kmbVyTd2cJPb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.lm import MLE\n",
        "model = MLE(n) # Lets train a 3-grams model, previously we set n=3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "69cdd0e5babaf059a7b9bc87dc0a6e261cef6cb2",
        "id": "GVy5jJjKcJPh",
        "colab_type": "text"
      },
      "source": [
        "Initializing the MLE model, creates an empty vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "93b95f89f020585dabfbcd6c41273c823a91e882",
        "id": "aF8G9jYocJPi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8527abbc-4795-4a42-d557-ffde46bdd923"
      },
      "source": [
        "len(model.vocab)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "fb567e332a0089c86e388a8c93f32d1a31737a29",
        "id": "x861E4DccJPl",
        "colab_type": "text"
      },
      "source": [
        "... which gets filled as we fit the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "a31ecc7f6df30a42df9ed6a79039b50f734299cf",
        "id": "N0z81-MAcJPl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e05bb456-d785-4fe1-949d-7dc29acedf77"
      },
      "source": [
        "model.fit(train_data, padded_sents)\n",
        "print(model.vocab)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Vocabulary with cutoff=1 unk_label='<UNK>' and 1429 items>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "6fd7e68c2cdacbe1901fd3b8c05c440706d19100",
        "id": "Xo1jP-uTcJPn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "eb2aad12-38d6-46c9-a8c3-11d21835ac11"
      },
      "source": [
        "len(model.vocab)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1429"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "74622c58a2df2ac08c8b0c79644d1f87f4014f44",
        "id": "GuCdUJaAcJPp",
        "colab_type": "text"
      },
      "source": [
        "The vocabulary helps us handle words that have not occurred during training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "e5670e34f9d1950f6a1515d57db18092d39484f2",
        "id": "VN8BFNgScJPq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "aea12312-f0e7-4348-ed75-bda1024d788a"
      },
      "source": [
        "print(model.vocab.lookup(tokenized_text[0]))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('language', 'is', 'never', ',', 'ever', ',', 'ever', ',', 'random', 'adam', 'kilgarriff', 'abstract', 'language', 'users', 'never', 'choose', 'words', 'randomly', ',', 'and', 'language', 'is', 'essentially', 'non-random', '.')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "cfec70768e8976073dda05f702f3f210a33dc1c3",
        "id": "LqrzKlkDcJPr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6cd203d2-283c-4b77-a2a2-b380e465f4e3"
      },
      "source": [
        "# If we lookup the vocab on unseen sentences not from the training data, \n",
        "# it automatically replace words not in the vocabulary with `<UNK>`.\n",
        "print(model.vocab.lookup('language is never random lah .'.split()))"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('language', 'is', 'never', 'random', '<UNK>', '.')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "83a71e4b25b53795dedce91c1a5686b7e22279e0",
        "id": "nsOFC251cJPt",
        "colab_type": "text"
      },
      "source": [
        "Moreover, in some cases we want to ignore words that we did see during training but that didn't occur frequently enough, to provide us useful information. \n",
        "\n",
        "You can tell the vocabulary to ignore such words using the `unk_cutoff` argument for the vocabulary lookup, To find out how that works, check out the docs for the [`nltk.lm.vocabulary.Vocabulary` class](https://github.com/nltk/nltk/blob/develop/nltk/lm/vocabulary.py)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YVH8h00Yd5Im",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a8ac8dca-dbf6-403a-d2e2-c31de242a9e7"
      },
      "source": [
        "print(model.vocab.lookup('language is never random lah .'.split()))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('language', 'is', 'never', 'random', '<UNK>', '.')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "b0310a2f3c7d9e90f4574b44838336ca1e4ef2ef",
        "id": "EXCdtChhcJPu",
        "colab_type": "text"
      },
      "source": [
        "**Note:** For more sophisticated ngram models, take a look at [these objects from `nltk.lm.models`](https://github.com/nltk/nltk/blob/develop/nltk/lm/models.py):\n",
        "\n",
        " - `Lidstone`: Provides Lidstone-smoothed scores.\n",
        " - `Laplace`: Implements Laplace (add one) smoothing.\n",
        " - `InterpolatedLanguageModel`: Logic common to all interpolated language models (Chen & Goodman 1995).\n",
        " - `WittenBellInterpolated`: Interpolated version of Witten-Bell smoothing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "af5a0851a2f8707ea2e172681342ed3ecd872328",
        "id": "ku88y_NpcJPu",
        "colab_type": "text"
      },
      "source": [
        "# Using the N-gram Language Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "d098fd4686bcdc0ab9aa72dbbd4b62a17ee2c332",
        "id": "o59mM_A-cJPv",
        "colab_type": "text"
      },
      "source": [
        "When it comes to ngram models the training boils down to counting up the ngrams from the training corpus."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "bfc60d61539269298390044c0d3415bfd28e2b1e",
        "id": "DVByw6MucJPv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "23afdfbe-69e3-40af-c9f1-a9c46dde9177"
      },
      "source": [
        "print(model.counts)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<NgramCounter with 3 ngram orders and 18687 ngrams>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "e9b8cac68de80aee9c7cf7b7f2faf2b199ad27cc",
        "id": "wBEdcoG-cJPw",
        "colab_type": "text"
      },
      "source": [
        "This provides a convenient interface to access counts for unigrams..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "90f4611580c41118747a42cfac4c9d729f02523f",
        "id": "vdpfyo3acJPx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "921d990f-afbe-48d4-aabc-5e156e253248"
      },
      "source": [
        "model.counts['language'] # i.e. Count('language')"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "25"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "ccdf1bf83ba16f0a43f04eb96ca224a968cc81d2",
        "id": "hWAnkon5cJPz",
        "colab_type": "text"
      },
      "source": [
        "...and bigrams for the phrase \"language is\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "3ba55a83d7c052d1f14923c4046a5f6c86d72c8c",
        "id": "Xm-ixD57cJPz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "70af01b6-2898-4022-b48f-20cbca915707"
      },
      "source": [
        "model.counts[['language']]['is'] # i.e. Count('is'|'language')"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "11"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "88bceda0a01e9f2642dd30a6b341c24a606720b9",
        "id": "nauix6pjcJP1",
        "colab_type": "text"
      },
      "source": [
        "... and trigrams for the phrase \"language is never\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "813e30e1fb153002cfb020d430def2510bf43bbc",
        "id": "WBUwylo8cJP1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cc772821-fa37-45b7-d8ce-3ed34ec45659"
      },
      "source": [
        "model.counts[['language', 'is']]['never'] # i.e. Count('never'|'language is')"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "7"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2c4338d351cfc3cd785673ad9581ecab161b24c6",
        "id": "SEnuvLitcJP3",
        "colab_type": "text"
      },
      "source": [
        "And so on. However, the real purpose of training a language model is to have it score how probable words are in certain contexts.\n",
        "\n",
        "This being MLE, the model returns the item's relative frequency as its score."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "051ef5a06c004a8e8ddb6168ca318bc8c0c9abf4",
        "id": "QRP_QNs0cJP4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "32d323e8-ece7-4aab-e57e-2934cc324318"
      },
      "source": [
        "model.score('language') # P('language')"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.003916040100250626"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "ab3d48919b8f074342e624a1da441621829a4dbd",
        "id": "MNDOLZkecJP5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "dc362bac-2de1-4df4-9046-eed5f4b46343"
      },
      "source": [
        "model.score('is', 'language'.split())  # P('is'|'language')"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.44"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "73955908d499f459e19c2cbfaec7d94c513acb91",
        "id": "Lt3F74v4cJP6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "31fd4078-6671-4cb7-fbb1-8c2674c1d64f"
      },
      "source": [
        "model.score('never', 'language is'.split())  # P('never'|'language is')"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.6363636363636364"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "946dde75e8d5b8d8878271359b0d91926abeb199",
        "id": "IYOU5VPzcJP8",
        "colab_type": "text"
      },
      "source": [
        "Items that are not seen during training are mapped to the vocabulary's \"unknown label\" token.  This is \"<UNK>\" by default.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "44038e8cd8d078b734a6b1a803d0795517e2fa82",
        "id": "g-bPFAAGcJP8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c55669c4-1c04-4120-8ccf-9697af068438"
      },
      "source": [
        "model.score(\"<UNK>\") == model.score(\"lah\")"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "f2f170a0bbca4b33b5eb1a55a55f0e1b1b18a634",
        "id": "WC24kB01cJP_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ced9fa7d-dcb3-49e4-f40a-44e408b767a5"
      },
      "source": [
        "model.score(\"<UNK>\") == model.score(\"leh\")"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "9107d1a7c27f30a449e9f370002a6406d5a1396d",
        "id": "5YBBdTqqcJQB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bcb379f0-f472-4dc4-b7c5-0b994bc113eb"
      },
      "source": [
        "model.score(\"<UNK>\") == model.score(\"lor\")"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "1a9ad0be90c72ece6c2a1c00314ea60f41a00f81",
        "id": "nxq_XgyLcJQC",
        "colab_type": "text"
      },
      "source": [
        "To avoid underflow when working with many small score values it makes sense to take their logarithm. \n",
        "\n",
        "For convenience this can be done with the `logscore` method.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "20ef81b7e161df3b9ab27236c33559713b3077ce",
        "id": "__twANUccJQD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6018824d-d17b-444a-b604-d34ee6eefacb"
      },
      "source": [
        "model.logscore(\"never\", \"language is\".split())"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "-0.6520766965796932"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "18f9e0a8d0aba302532b8a84f342d1bf4d3e202a",
        "id": "f96MsatYcJQE",
        "colab_type": "text"
      },
      "source": [
        "# Generation using N-gram Language Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "f893159fe093aef484c07b37d922de4ac5727834",
        "id": "9vf2wwIqcJQE",
        "colab_type": "text"
      },
      "source": [
        "One cool feature of ngram models is that they can be used to generate text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "98989ec4bae592fc98332e759daf9e42bac4213e",
        "id": "XCfQN-c3cJQF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "7e4bb959-dc6f-4822-b6d1-b094f48c3012"
      },
      "source": [
        "print(model.generate(20, random_seed=7))"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['ate', 'inferences', 'are', 'drawn.', '2', '.', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "3b8d07eaf3afae978131573ae127fa0ec3f2e50d",
        "id": "vmrNr_EIcJQH",
        "colab_type": "text"
      },
      "source": [
        "We can do some cleaning to the generated tokens to make it human-like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "73d0d5e0029e64876100e0f2e368b4835a99efcd",
        "id": "WqEZj3O1cJQH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
        "\n",
        "detokenize = TreebankWordDetokenizer().detokenize\n",
        "\n",
        "def generate_sent(model, num_words, random_seed=42):\n",
        "    \"\"\"\n",
        "    :param model: An ngram language model from `nltk.lm.model`.\n",
        "    :param num_words: Max no. of words to generate.\n",
        "    :param random_seed: Seed value for random.\n",
        "    \"\"\"\n",
        "    content = []\n",
        "    for token in model.generate(num_words, random_seed=random_seed):\n",
        "        if token == '<s>':\n",
        "            continue\n",
        "        if token == '</s>':\n",
        "            break\n",
        "        content.append(token)\n",
        "    return detokenize(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "949378240cbc8247579b79946b113a3afd039b39",
        "id": "bPkDIZw6cJQJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1eff5f64-dbfd-4e04-aca1-e7ceeb603256"
      },
      "source": [
        "generate_sent(model, 20, random_seed=7)"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'ate inferences are drawn. 2.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "9041c514e0458236132fe9b3c42ac2ce651beb92",
        "id": "lU28eZw8cJQK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "36cbf9f8-b7bf-4ed2-f8c1-aa9fb86600c2"
      },
      "source": [
        "print(model.generate(28, random_seed=0))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['the', 'scf-verb', 'link', 'is', 'motivated', '.', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "523e7c2373f6a4c4b6a542f1e82369fedb6cbd21",
        "id": "zcIPP4ZKcJQM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a8f126fd-7a99-414d-ce81-f3db006f0562"
      },
      "source": [
        "generate_sent(model, 28, random_seed=0)"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'the scf-verb link is motivated.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "bce2885bd22d0bb1f79c3861f10fde7df6714e40",
        "id": "6NDW8RnGcJQO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7fd9cd04-44be-440c-fbd6-2b9b35a82ac4"
      },
      "source": [
        "generate_sent(model, 20, random_seed=1)"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'29⫺50. manning, christopher and hinrich schütze 1999 foundations of statistical independence.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "90ddb0e7d0fb52bc77d08425331944c55ee0885e",
        "id": "_cUAXxTkcJQQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bc624a8c-1569-4a55-fc7d-9d220d2a6718"
      },
      "source": [
        "generate_sent(model, 20, random_seed=30)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'information between two rare events, the null hypthesis regarding the two objections, while both valid, are not'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "ba9546c6800d9e0b0f0af4f0fd31a141af93b12f",
        "id": "hdKeQjgtcJQS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2d5a0b36-b6bc-4916-8b67-d7a1c395ad6d"
      },
      "source": [
        "generate_sent(model, 20, random_seed=42)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'not.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WCqNhUPccJQT",
        "colab_type": "text"
      },
      "source": [
        "# Saving the model \n",
        "\n",
        "The native Python's pickle may not save the lambda functions in the  model, so we can use the `dill` library in place of pickle to save and load the language model.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OeLh59SycJQT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import dill as pickle \n",
        "\n",
        "with open('kilgariff_ngram_model.pkl', 'wb') as fout:\n",
        "    pickle.dump(model, fout)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5440PRszcJQV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with open('kilgariff_ngram_model.pkl', 'rb') as fin:\n",
        "    model_loaded = pickle.load(fin)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nqhbFlVucJQX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "31f042d8-5971-4bba-c1cb-30e316c729d6"
      },
      "source": [
        "generate_sent(model_loaded, 20, random_seed=42)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'not.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "e4f13ed7359c87e397229104c1bcf18eb20603ad",
        "id": "XHxYICcQcJQY",
        "colab_type": "text"
      },
      "source": [
        "# Lets try some generating with Donald Trump data!!!\n",
        "\n",
        "\n",
        "**Dataset:** https://www.kaggle.com/kingburrito666/better-donald-trump-tweets#Donald-Tweets!.csv\n",
        "\n",
        "\n",
        "In this part, I'll be munging that data as how I would be doing it at work. \n",
        "I've really no seen the data before but I hope this session would be helpful for you to see how to approach new datasets with the skills you have."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "caf5fea33f84e06c3cf613136ec33f80e774fc98",
        "id": "XwjJH9YTcJQY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "outputId": "8e22af58-1b5d-4461-8ac2-c25f43c21e2c"
      },
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('Donald-Tweets!.csv')\n",
        "df.head()"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Time</th>\n",
              "      <th>Tweet_Text</th>\n",
              "      <th>Type</th>\n",
              "      <th>Media_Type</th>\n",
              "      <th>Hashtags</th>\n",
              "      <th>Tweet_Id</th>\n",
              "      <th>Tweet_Url</th>\n",
              "      <th>twt_favourites_IS_THIS_LIKE_QUESTION_MARK</th>\n",
              "      <th>Retweets</th>\n",
              "      <th>Unnamed: 10</th>\n",
              "      <th>Unnamed: 11</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>16-11-11</td>\n",
              "      <td>15:26:37</td>\n",
              "      <td>Today we express our deepest gratitude to all ...</td>\n",
              "      <td>text</td>\n",
              "      <td>photo</td>\n",
              "      <td>ThankAVet</td>\n",
              "      <td>7.970000e+17</td>\n",
              "      <td>https://twitter.com/realDonaldTrump/status/797...</td>\n",
              "      <td>127213</td>\n",
              "      <td>41112</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>16-11-11</td>\n",
              "      <td>13:33:35</td>\n",
              "      <td>Busy day planned in New York. Will soon be mak...</td>\n",
              "      <td>text</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.970000e+17</td>\n",
              "      <td>https://twitter.com/realDonaldTrump/status/797...</td>\n",
              "      <td>141527</td>\n",
              "      <td>28654</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>16-11-11</td>\n",
              "      <td>11:14:20</td>\n",
              "      <td>Love the fact that the small groups of protest...</td>\n",
              "      <td>text</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.970000e+17</td>\n",
              "      <td>https://twitter.com/realDonaldTrump/status/797...</td>\n",
              "      <td>183729</td>\n",
              "      <td>50039</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>16-11-11</td>\n",
              "      <td>2:19:44</td>\n",
              "      <td>Just had a very open and successful presidenti...</td>\n",
              "      <td>text</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.970000e+17</td>\n",
              "      <td>https://twitter.com/realDonaldTrump/status/796...</td>\n",
              "      <td>214001</td>\n",
              "      <td>67010</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>16-11-11</td>\n",
              "      <td>2:10:46</td>\n",
              "      <td>A fantastic day in D.C. Met with President Oba...</td>\n",
              "      <td>text</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.970000e+17</td>\n",
              "      <td>https://twitter.com/realDonaldTrump/status/796...</td>\n",
              "      <td>178499</td>\n",
              "      <td>36688</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       Date      Time  ... Unnamed: 10 Unnamed: 11\n",
              "0  16-11-11  15:26:37  ...         NaN         NaN\n",
              "1  16-11-11  13:33:35  ...         NaN         NaN\n",
              "2  16-11-11  11:14:20  ...         NaN         NaN\n",
              "3  16-11-11   2:19:44  ...         NaN         NaN\n",
              "4  16-11-11   2:10:46  ...         NaN         NaN\n",
              "\n",
              "[5 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "a15d9d1a754ca357ae79e8390069b08b05320458",
        "id": "NHKeaSLVcJQa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "trump_corpus = list(df['Tweet_Text'].apply(word_tokenize))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "8724b8724cc52b685205047bd1fda649545a6a24",
        "id": "FJXmk5z-cJQb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Preprocess the tokenized text for 3-grams language modelling\n",
        "n = 3\n",
        "train_data, padded_sents = padded_everygram_pipeline(n, trump_corpus)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "c5010b607062f6f2dd0ccf011bb9004e1e3dcee7",
        "id": "yCQW-Ib7cJQb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.lm import MLE\n",
        "trump_model = MLE(n) # Lets train a 3-grams model, previously we set n=3\n",
        "trump_model.fit(train_data, padded_sents)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "8036f1ae5e0e1438d1d2981026766cdacd4479aa",
        "id": "qa8wJ0rMcJQc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "186c1568-c821-4482-8efd-78d792a13e80"
      },
      "source": [
        "generate_sent(trump_model, num_words=40, random_seed=42)"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'do, and almost everyone of my beautiful mother, amazing father, @realDonaldTrump leading GOP in #Wisconsin.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "defe4e4899cb14300710eb0c7786a6f2e894455d",
        "id": "hXbSyYRacJQe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bc57881c-1d57-4f63-8f73-46b968ac584b"
      },
      "source": [
        "generate_sent(trump_model, num_words=10, random_seed=0)"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'pretty sad situation. Go Jeb! He is heads above'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "eba7934efa5a5e6b818e951c5daec9e9f621ca26",
        "id": "9SZ-wjAGcJQf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "21fb68ed-c608-40f3-ce7d-53917f80b44f"
      },
      "source": [
        "generate_sent(trump_model, num_words=50, random_seed=10)"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'and hard for Hillary vs Bernie but do not like Ted Cruz would speak behind my back, just like our government! Lower taxes! https://t.co/ZwIkqNH2FX'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "3ec48d8d44b2f57b16249e12b845f43f52faadca",
        "id": "L8Q1VxmScJQg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5de415aa-85ca-41a7-ee10-a4c8d79f921e"
      },
      "source": [
        "print(generate_sent(trump_model, num_words=100, random_seed=52))"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "with @DonaldJTrumpJr &amp; videos on the BORDER tomorrow. Will be in South Bend, Indiana! #Trump2016 #MakeAmericaGreatAgain\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "2567d003d69c26233b915df03509b3fd014beed4",
        "id": "XZtUi_mNcJQh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}