11 - Lesson - Natural Language Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "colab": {
      "name": "11 - Lesson - Natural Language Processing.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/adel-nouar/ML_with_Rune/blob/main/11%20-%20Lesson%20-%20Natural%20Language%20Processing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ba959ea8"
      },
      "source": [
        "# Natural Language Processing\n",
        "### Goal of lesson\n",
        "- How the simple syntax of language can be parsed\n",
        "- What Context-Free Grammar (CFG) is\n",
        "- Use it to parse text\n",
        "- Understand text in trigrams\n",
        "- See how it can be used to generate predictions\n",
        "\n",
        "### What is Natural Language Processing\n",
        "- Automatic computational processing of human languages\n",
        "- Includes \n",
        "    - Algorithms that take human written language as input\n",
        "    - Algorithms that produce natural text\n",
        "\n",
        "- Examples include\n",
        "    - Automatic summarization\n",
        "    - Language identification\n",
        "    - Translation"
      ],
      "id": "ba959ea8"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "88cafa61"
      },
      "source": [
        "### Syntax\n",
        "- One basic description of a language's syntax is the sequence in which the subject, verb, and object usually appear in sentences.\n",
        "\n",
        "### Formal Grammar\n",
        "- A system of rules for genrating sentences in a language\n",
        "- A grammar is usually thought of as a language generator ([wiki](https://en.wikipedia.org/wiki/Formal_grammar))"
      ],
      "id": "88cafa61"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "338af3bc"
      },
      "source": [
        "### Context-Free Grammar (CFG)\n",
        "- A formal grammar is \"context free\" if its production rules can be applied regardless of the context of a nonterminal ([wiki](https://en.wikipedia.org/wiki/Context-free_grammar))."
      ],
      "id": "338af3bc"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eec8e4f6"
      },
      "source": [
        "> #### Programming Notes:\n",
        "> - Libraries used\n",
        ">     - [**nltk**](https://www.nltk.org) - Natural Language Toolkit\n",
        ">     - [**os**](https://docs.python.org/3/library/os.html) Miscellaneous operating system interfaces\n",
        ">     - [**collections**](https://docs.python.org/3/library/collections.html) Container datatypes\n",
        ">     - [**markovify**](https://pypi.org/project/markovify/) A simple, extensible Markov chain generato\n",
        "> - Functionality and concepts used\n",
        ">     - [**ChartParser**](https://tedboy.github.io/nlps/generated/generated/nltk.ChartParser.html) generic chart parser\n",
        ">     - **List Comprehension** to convert data ([Lecture on **List Comprehension**](https://youtu.be/vCYEvtfXdig))\n",
        ">     - [**Counter**](https://docs.python.org/3/library/collections.html#collections.Counter) a dict subclass for counting hashable objects\n",
        ">     - [**markovify.Text**](https://pypi.org/project/markovify/) to create your Markov Model"
      ],
      "id": "eec8e4f6"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9ea2f28f",
        "outputId": "d0d0e18b-ad1d-422e-e5e9-3950f794deb2"
      },
      "source": [
        "#!pip install nltk"
      ],
      "id": "9ea2f28f",
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (3.2.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nltk) (1.15.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7d8eea04"
      },
      "source": [
        "import nltk"
      ],
      "id": "7d8eea04",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "06e74e9f",
        "outputId": "dce603be-39f7-4bc4-a724-7cec15df70f1"
      },
      "source": [
        "grammar = nltk.CFG.fromstring(\"\"\"\n",
        "    S -> NP VP\n",
        "\n",
        "    NP -> D N | N\n",
        "    VP -> V | V NP\n",
        "\n",
        "    D -> \"the\" | \"a\" | \"an\"\n",
        "    N -> \"she\" | \"city\" | \"car\" | \"town\"\n",
        "    V -> \"saw\" | \"walked\" | \"liked\"   \n",
        "\"\"\")\n",
        "\n",
        "parser = nltk.ChartParser(grammar)\n",
        "\n",
        "sentence = input().split()\n",
        "\n",
        "for tree in parser.parse(sentence):\n",
        "    tree.pretty_print()"
      ],
      "id": "06e74e9f",
      "execution_count": null,
      "outputs": [
        {
          "name": "stdin",
          "output_type": "stream",
          "text": [
            " she liked town\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "      S           \n",
            "  ____|____        \n",
            " |         VP     \n",
            " |     ____|___    \n",
            " NP   |        NP \n",
            " |    |        |   \n",
            " N    V        N  \n",
            " |    |        |   \n",
            "she liked     town\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6fa8235"
      },
      "source": [
        "### Challenge with CFG\n",
        "- You need to encode all possibilities\n",
        "\n",
        "### Idea\n",
        "- Understand text in small subsets\n",
        "- **$n$-gram**\n",
        "    - a contiguous sequence of $n$ items from a sample text\n",
        "- **Word $n$-gram**\n",
        "    - a contiguous sequence of $n$ words from a sample text\n",
        "- **unigram**\n",
        "    - 1 items in sequence\n",
        "- **bigram**\n",
        "    - 2 items in sequence\n",
        "- **trigram**\n",
        "    - 3 items in sequence"
      ],
      "id": "b6fa8235"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "30889d25"
      },
      "source": [
        "### Word Tokenization\n",
        "- the task of splitting a sequence of words into tokens\n",
        "\n",
        "- Considerations: comma, punktuation, etc."
      ],
      "id": "30889d25"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d4ea65e3"
      },
      "source": [
        "import os\n",
        "from collections import Counter"
      ],
      "id": "d4ea65e3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d3d5f1b8",
        "outputId": "89e0f9f8-29f0-497e-e658-dcb7e0b54764"
      },
      "source": [
        "nltk.download('punkt')"
      ],
      "id": "d3d5f1b8",
      "execution_count": null,
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /Users/adel/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "be1f0f82"
      },
      "source": [
        "content = []\n",
        "for filename in os.listdir('files/holmes/'):\n",
        "    with open(f'files/holmes/{filename}') as f:\n",
        "        content.append(f.read())"
      ],
      "id": "be1f0f82",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fcdbe57",
        "outputId": "f1320056-29cb-4bdb-e806-2b9df6345241"
      },
      "source": [
        "len(content)"
      ],
      "id": "7fcdbe57",
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "21"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a58c7ddd"
      },
      "source": [
        "corpus = []\n",
        "for item in content:\n",
        "    corpus.extend([word.lower() for word in nltk.word_tokenize(item) if any(c.isalpha() for c in word)])"
      ],
      "id": "a58c7ddd",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "79dc5211",
        "outputId": "797567d9-43fe-49ce-9fa3-e60c214a8407"
      },
      "source": [
        "len(corpus)"
      ],
      "id": "79dc5211",
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "178205"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "99ed6d7f-d2cf-4972-9fa2-b6f9217db404"
      },
      "source": [
        "ngrams = Counter(nltk.ngrams(corpus, 3))"
      ],
      "id": "99ed6d7f-d2cf-4972-9fa2-b6f9217db404",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d9b296ab",
        "outputId": "012efff2-f997-4b5a-d6c8-fe706cc98e71"
      },
      "source": [
        "for ngram, freq in ngrams.most_common(10):\n",
        "    print(f'{freq}: {ngram}')"
      ],
      "id": "d9b296ab",
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "80: ('it', 'was', 'a')\n",
            "71: ('one', 'of', 'the')\n",
            "65: ('i', 'think', 'that')\n",
            "59: ('out', 'of', 'the')\n",
            "55: ('that', 'it', 'was')\n",
            "55: ('that', 'he', 'had')\n",
            "55: ('there', 'was', 'a')\n",
            "55: ('that', 'he', 'was')\n",
            "52: ('it', 'is', 'a')\n",
            "49: ('i', 'can', 'not')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1373e8d6"
      },
      "source": [
        "### Markov Model\n",
        "- A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous even ([wiki](https://en.wikipedia.org/wiki/Markov_chain))\n",
        "- Or as the example above:\n",
        "    - Given any two words -> you have probabilities for the next word"
      ],
      "id": "1373e8d6"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2866684b-342c-4008-addd-a674f1ca58c0",
        "outputId": "5a206a23-e7a1-4e29-ecc1-fe1be2942f87",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install markovify"
      ],
      "id": "2866684b-342c-4008-addd-a674f1ca58c0",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting markovify\n",
            "  Downloading markovify-0.9.3.tar.gz (28 kB)\n",
            "Collecting unidecode\n",
            "  Downloading Unidecode-1.3.2-py3-none-any.whl (235 kB)\n",
            "\u001b[K     |████████████████████████████████| 235 kB 7.8 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: markovify\n",
            "  Building wheel for markovify (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for markovify: filename=markovify-0.9.3-py3-none-any.whl size=18622 sha256=2102137628f3f373007089936354b022f89979a047b489f2565bb3ec2af0f169\n",
            "  Stored in directory: /root/.cache/pip/wheels/d9/f0/5b/748a27bdf2496bd4df51acb9442dae516efce507ff4849813e\n",
            "Successfully built markovify\n",
            "Installing collected packages: unidecode, markovify\n",
            "Successfully installed markovify-0.9.3 unidecode-1.3.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c260c02b"
      },
      "source": [
        "import markovify"
      ],
      "id": "c260c02b",
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3e9fc8c"
      },
      "source": [
        "with open('files/shakespeare.txt') as f:\n",
        "    text = f.read()"
      ],
      "id": "c3e9fc8c",
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "354f2d51"
      },
      "source": [
        "model = markovify.Text(text)"
      ],
      "id": "354f2d51",
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4f76620f",
        "outputId": "35adb1df-cf94-4f13-9d14-b51df05a9281",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "model.make_sentence()"
      ],
      "id": "4f76620f",
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'When I will bethink me.'"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    }
  ]
}