how_spacy_works.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "!pip install spacy"
      ],
      "metadata": {
        "id": "thfJmLCzjUwR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download en_core_web_lg"
      ],
      "metadata": {
        "id": "jhERXFghjq8j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BvmHjwyji_Km"
      },
      "source": [
        "### Predicting Part-of-speech Tags"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YPTDAidDi_Km",
        "outputId": "daf09e56-7991-4295-aa7c-17ac81afa3a4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The DET\n",
            "doctor NOUN\n",
            "finished VERB\n",
            "her PRON\n",
            "work NOUN\n"
          ]
        }
      ],
      "source": [
        "import spacy\n",
        "\n",
        "nlp = spacy.load(\"en_core_web_lg\")\n",
        "\n",
        "doc = nlp(\"The doctor finished her work\")\n",
        "\n",
        "for token in doc:\n",
        "    print(token.text, token.pos_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xq9N6f0ui_Km"
      },
      "source": [
        "### Predicting Syntatic Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LIy-k9-oi_Kn",
        "outputId": "914f211e-736f-404a-9e51-f0c7a13e3fd1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The DET det doctor\n",
            "doctor NOUN nsubj finished\n",
            "finished VERB ROOT finished\n",
            "her PRON poss work\n",
            "work NOUN dobj finished\n"
          ]
        }
      ],
      "source": [
        "for token in doc:\n",
        "    print(token.text, token.pos_, token.dep_, token.head.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u1ZIACNri_Kn",
        "outputId": "25e4087e-34e8-46f0-a381-d1781a7f6b58"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The DET det doctor\n",
            "doctor NOUN nsubj finished\n",
            "finished VERB ROOT finished\n",
            "her PRON poss work\n",
            "work NOUN dobj finished\n",
            ", PUNCT punct finished\n",
            "but CCONJ cc finished\n",
            "the DET det designer\n",
            "designer NOUN nsubj was\n",
            "was AUX conj finished\n",
            "not PART neg was\n",
            "happy ADJ acomp was\n",
            "about ADP prep happy\n",
            "it PRON pobj about\n"
          ]
        }
      ],
      "source": [
        "doc_longer = nlp(\"The doctor finished her work, but the designer was not happy about it\")\n",
        "\n",
        "for token in doc_longer:\n",
        "    print(token.text, token.pos_, token.dep_, token.head.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WaIdbPDHi_Kn"
      },
      "source": [
        "### The explain method"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "j-4Z6oFvi_Kn",
        "outputId": "fde521ab-4ea0-41c0-98c8-9fd299eef96b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'coordinating conjunction'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "spacy.explain('CCONJ')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0A1rWQAEi_Kn"
      },
      "source": [
        "### Predicting named entities in context"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hkbzy_sai_Kn",
        "outputId": "b064b560-3322-4beb-c4f5-b7c84fca6de6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Apple ORG\n"
          ]
        }
      ],
      "source": [
        "text = \"New iPhone X release date leaked as Apple reveals pre-orders by mistake\"\n",
        "\n",
        "# Process the text\n",
        "doc = nlp(text)\n",
        "\n",
        "# Iterate over the entities\n",
        "for ent in doc.ents:\n",
        "    # print the entity text and label\n",
        "    print(ent.text, ent.label_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fRHo_NIi_Ko"
      },
      "source": [
        "### Match patterns\n",
        "\n",
        "Match exact token texts:\n",
        "[{'ORTH':  'iPhone'}, {'ORTH': 'X'}]\n",
        "\n",
        "Match lexical attributes:\n",
        "[{'LOWER':  'iphone'}, {'LOWER': 'x'}]\n",
        "\n",
        "Match any token attributes:\n",
        "[{'LEMMA':  'buy'}, {'POS': 'NOUN'}]\n",
        "This pattern would match phrases like \"buying milk\" or \"bought flowers\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YXHfj1DOi_Ko",
        "outputId": "a4574e3b-9a1f-4d87-973d-03a5be7ccf43"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "iPhone X\n"
          ]
        }
      ],
      "source": [
        "from spacy.matcher import Matcher\n",
        "\n",
        "matcher = Matcher(nlp.vocab)\n",
        "\n",
        "pattern = [{'ORTH':  'iPhone'}, {'ORTH': 'X'}]\n",
        "matcher.add('IPHONE_PATTERN', [pattern]) # first arg: unique id\n",
        "\n",
        "doc = nlp('New iPhone X release date leaked')\n",
        "\n",
        "matches = matcher(doc)\n",
        "\n",
        "for match_id, start, end in matches:\n",
        "    matched_span = doc[start:end]\n",
        "    print(matched_span.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "UYzuHUuvi_Ko"
      },
      "outputs": [],
      "source": [
        "### Matching lexical attributes\n",
        "\n",
        "pattern = [ # matches the tokens '2018 FIFA World Cup:'\n",
        "    {'IS_DIGIT': True},\n",
        "    {'LOWER': 'fifa'},\n",
        "    {'LOWER': 'world'},\n",
        "    {'LOWER': 'cup'},\n",
        "    {'IS_PUNCT': True},\n",
        "] \n",
        "\n",
        "doc = nlp('2018 FIFA World Cup: France won!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ts899CcNi_Ko"
      },
      "source": [
        "Using operators and quantifiers: \n",
        "\n",
        "- `{'OP': '!'}` = Negation: match 0 times\n",
        "- `{'OP': '?'}` = Optional: match 0 or 1 times\n",
        "- `{'OP': '+'}` = Match 1 or more times\n",
        "- `{'OP': '*'}` = Match 0 or more times"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "MJpDtb6Gi_Ko"
      },
      "outputs": [],
      "source": [
        "# Two tokens whose lowercase forms match 'iphone' and 'x'\n",
        "pattern1 = [{'LOWER': 'iphone'}, {'LOWER': 'x'}]\n",
        "\n",
        "# Token whose lowercase form matches 'iphone' and an optional digit\n",
        "pattern2 = [{'LOWER': 'iphone'}, {'IS_DIGIT': True, 'OP': '?'}]\n",
        "\n",
        "# Add patterns to the matcher\n",
        "matcher.add('GADGET', [pattern1, pattern2])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_b1CKpkTi_Ko"
      },
      "source": [
        "### Doc and Span"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hn90cxRki_Kp",
        "outputId": "87f17cfc-c9ff-4a9f-ce2b-39ae960af3a7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('David Bowie', 'PERSON')]\n"
          ]
        }
      ],
      "source": [
        "# Import the Doc and Span classes\n",
        "from spacy.tokens import Doc, Span\n",
        "\n",
        "# Create a doc from the words and spaces\n",
        "doc = Doc(nlp.vocab, words=['I', 'like', 'David', 'Bowie'], spaces=[True, True, True, False])\n",
        "\n",
        "# Create a span for \"David Bowie\" from the doc and assign it the label \"PERSON\"\n",
        "span = Span(doc, 2, 4, label='PERSON')\n",
        "\n",
        "# Add the span to the doc's entities\n",
        "doc.ents = [span]\n",
        "\n",
        "# Print entities' text and labels\n",
        "print([(ent.text, ent.label_) for ent in doc.ents])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OUztJowai_Kp"
      },
      "source": [
        "### Check structure"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "DSxziZjLi_Kp"
      },
      "outputs": [],
      "source": [
        "doc = nlp(\"Berlin is a nice city\")\n",
        "\n",
        "# Iterate over the tokens\n",
        "for token in doc:\n",
        "    # Check if the current token is a proper noun\n",
        "    if token.pos_ == 'PROPN':\n",
        "        # Check if the next token is a verb\n",
        "        if doc[token.i + 1].pos_ == 'VERB':\n",
        "            print('Found a verb after a proper noun!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmEQfkDDi_Kp"
      },
      "source": [
        "### Check similarity\n",
        "By default SpaCy uses cosine similarity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3rJXrQddi_Kp",
        "outputId": "a52fde32-f3ed-4899-d2f7-cb0cbd9a56e3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8698332283318978\n"
          ]
        }
      ],
      "source": [
        "doc1 = nlp(\"I like pizza\")\n",
        "doc2 = nlp(\"I like fast food\")\n",
        "\n",
        "print(doc1.similarity(doc2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WVKcM-dHi_Kp"
      },
      "source": [
        "### Adding statistical predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QgukPSz-i_Kp",
        "outputId": "ecd5bf77-ef0e-4331-f1ee-027053e5317f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Matched span:  Golden Retriever\n",
            "Root token:  Retriever\n",
            "Root head token:  have\n",
            "Previous token:  a DET\n"
          ]
        }
      ],
      "source": [
        "matcher = Matcher(nlp.vocab)\n",
        "\n",
        "matcher.add('DOG', [[{'LOWER': 'golden'}, {'LOWER': 'retriever'}]])\n",
        "doc = nlp('I have a Golden Retriever')\n",
        "\n",
        "for match_id, start, end in matcher(doc):\n",
        "    span = doc[start:end]\n",
        "    print('Matched span: ', span.text)\n",
        "\n",
        "    #Get the span's root token and root head token\n",
        "    print('Root token: ', span.root.text)\n",
        "    print('Root head token: ', span.root.head.text)\n",
        "\n",
        "    #Get previous token and its POS tag\n",
        "    print('Previous token: ', doc[start - 1].text, doc[start - 1].pos_)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4q2wCKBdi_Kq"
      },
      "source": [
        "##### Using Phrase Matcher"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H0JqFL-pi_Kq",
        "outputId": "e39c17a0-4e75-49a9-d812-9be1511f41f8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Matched span:  Golden Retriever\n",
            "Root token:  Retriever\n",
            "Root head token:  have\n",
            "Previous token:  a DET\n"
          ]
        }
      ],
      "source": [
        "from spacy.matcher import PhraseMatcher\n",
        "\n",
        "matcher = PhraseMatcher(nlp.vocab)\n",
        "pattern = nlp('Golden Retriever')\n",
        "matcher.add('DOG', [pattern])\n",
        "\n",
        "doc = nlp('I have a Golden Retriever')\n",
        "\n",
        "for match_id, start, end in matcher(doc):\n",
        "    span = doc[start:end]\n",
        "    print('Matched span: ', span.text)\n",
        "\n",
        "    #Get the span's root token and root head token\n",
        "    print('Root token: ', span.root.text)\n",
        "    print('Root head token: ', span.root.head.text)\n",
        "\n",
        "    #Get previous token and its POS tag\n",
        "    print('Previous token: ', doc[start - 1].text, doc[start - 1].pos_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KEOlXQ-2i_Kq"
      },
      "source": [
        "### Built-in pipeline components\n",
        "\n",
        "The part-of-speech tagger sets the token dot tag attribute. \n",
        "\n",
        "`tagger -> part-of-speech tagger -> Token.tag`\n",
        "\n",
        "\n",
        "The depdendency parser adds the token dot dep and token dot head attributes and is also responsible for detecting sentences and base noun phrases, also known as noun chunks.\n",
        "\n",
        "`parser -> dependency parser -> Token.dep | Token.head | Doc.sents | Doc.noun_chunks`\n",
        "\n",
        "\n",
        "The named entity recognizer adds the detected entities to the doc dot ents property. It also sets entity type attributes on the tokens that indicate if a token is part of an entity or not.\n",
        "\n",
        "`ner -> named entity recognizer -> Doc.ents | Token.ent_iob | Token.ent_type`\n",
        "\n",
        "\n",
        "Finally, the text classifier sets category labels that apply to the whole text, and adds them to the doc dot cats property. Because text categories are always very specific, the text classifier is not included in any of the pre-trained models by default. But you can use it to train your own system.\n",
        "\n",
        "`textcat -> text classifier -> Doc.cats`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tvgXWgDxi_Kq",
        "outputId": "17fa3faa-edf6-4002-dcb8-f7772152a35b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']\n",
            "[('tok2vec', <spacy.pipeline.tok2vec.Tok2Vec object at 0x7f4338e5da60>), ('tagger', <spacy.pipeline.tagger.Tagger object at 0x7f4338e5dad0>), ('parser', <spacy.pipeline.dep_parser.DependencyParser object at 0x7f4338b00550>), ('attribute_ruler', <spacy.pipeline.attributeruler.AttributeRuler object at 0x7f4338a9ef00>), ('lemmatizer', <spacy.lang.en.lemmatizer.EnglishLemmatizer object at 0x7f4338aa4f00>), ('ner', <spacy.pipeline.ner.EntityRecognizer object at 0x7f4338b00850>)]\n"
          ]
        }
      ],
      "source": [
        "# Print the names of the pipeline components\n",
        "print(nlp.pipe_names)\n",
        "\n",
        "# Print the full pipeline of (name, component) tuples\n",
        "print(nlp.pipeline)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EZ9kgxjWi_Kq"
      },
      "source": [
        "### Custom components\n",
        "\n",
        "Why? \n",
        "\n",
        "- Make a function execute automatically when you call nlp\n",
        "\n",
        "- Add your own metadata to documents and tokens\n",
        "\n",
        "- Updating built-in attributes like doc.ents\n",
        "\n",
        "\n",
        "```\n",
        "def custom_component(doc):\n",
        "    #do something\n",
        "    return doc\n",
        "\n",
        "nlp.add_pipeline(custom_component)\n",
        "```\n",
        "\n",
        "Setting last to True will add the component last in the pipeline. This is the default behavior.\n",
        "\n",
        "`last -> if true, add last -> ex: nlp.add_pipe(component, last=True)`\n",
        "\n",
        "\n",
        "Setting first to True will add the component first in the pipeline, right after the tokenizer. \n",
        "\n",
        "`first -> if true, add first -> ex: nlp.add_pipe(component, first=True)`\n",
        "\n",
        "\n",
        "The \"before\" and \"after\" arguments let you define the name of an existing component to add the new component before or after. For example, before equals \"ner\" will add it before the named entity recognizer. The other component to add the new component before or after needs to exist, though – otherwise, spaCy will raise an error.\n",
        "`before -> add before component -> ex: nlp.add_pipe(component, before='ner')`\n",
        "\n",
        "`after -> add after component -> ex: nlp.add_pipe(component, after='tagger')`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BdKMHOgti_Kr",
        "outputId": "8e3e665d-b2bf-4823-a159-656820aa7adc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pipeline:  ['custom_component', 'tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']\n"
          ]
        }
      ],
      "source": [
        "from spacy.language import Language\n",
        "\n",
        "@Language.component('custom_component')\n",
        "def custom_component(doc):\n",
        "    #Print the doc's length\n",
        "    print('Doc length: ', len(doc))\n",
        "\n",
        "    #important: have to return the modified doc\n",
        "    return doc\n",
        "\n",
        "#Add the component first in the pipeline\n",
        "nlp.add_pipe('custom_component', first=True)\n",
        "\n",
        "#Print the pipeline component names\n",
        "print('Pipeline: ', nlp.pipe_names)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KhNirbgRi_Kr",
        "outputId": "d6c2cc61-e9fd-481a-a8dd-cf5fe9db1651"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  2\n"
          ]
        }
      ],
      "source": [
        "doc = nlp('Hello world')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ckvmRdjni_Kr",
        "outputId": "89e3870f-530d-417c-a9ff-3c1d7bf07c12"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  8\n",
            "[('Golden Retriever', 'ANIMAL')]\n"
          ]
        }
      ],
      "source": [
        "# Define the custom component\n",
        "@Language.component('animal_component')\n",
        "def animal_component(doc):\n",
        "    # Create a Span for each match and assign the label 'ANIMAL'\n",
        "    # and overwrite the doc.ents with the matched spans\n",
        "    doc.ents = [Span(doc, start, end, label='ANIMAL')\n",
        "                for match_id, start, end in matcher(doc)]\n",
        "    return doc\n",
        "    \n",
        "# Add the component to the pipeline after the 'ner' component \n",
        "nlp.add_pipe('animal_component', after='ner')\n",
        "\n",
        "# Process the text and print the text and label for the doc.ents\n",
        "doc = nlp(\"I have a cat and a Golden Retriever\")\n",
        "print([(ent.text, ent.label_) for ent in doc.ents])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-bScY9Wi_Kr"
      },
      "source": [
        "### Setting custom attributes\n",
        "\n",
        "- Add custom metadata to documents, tokens and spans \n",
        "\n",
        "- Accessible via the ._ property"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xs53erW2i_Kr",
        "outputId": "acaf1a76-224e-4960-c89c-38e6598cda3d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  5\n"
          ]
        }
      ],
      "source": [
        "from spacy.tokens import Doc, Token, Span\n",
        "\n",
        "\n",
        "Doc.set_extension('title', default=None)\n",
        "\n",
        "# set extension on the Token with default value\n",
        "Token.set_extension('is_color', default=False)\n",
        "doc = nlp(\"The sky is blue.\")\n",
        "\n",
        "#overwrite extension attribute value\n",
        "doc[3]._.is_color = True\n",
        "\n",
        "Span.set_extension('has_color', default=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWdE_awTi_Kr"
      },
      "source": [
        "- Define a getter and an optional setter function \n",
        "\n",
        "- Getter only is called when you retrieve the attribute value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jLCHJYs4i_Ks",
        "outputId": "f998b938-4981-4e6f-a457-36ce525e0006"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  5\n",
            "True - blue\n"
          ]
        }
      ],
      "source": [
        "#define getter function\n",
        "def get_is_color(token):\n",
        "    colors = ['red', 'yellow', 'blue']\n",
        "    return token.text in colors\n",
        "\n",
        "#set extension on the Token with getter\n",
        "Token.set_extension('is_color', getter=get_is_color, force=True)\n",
        "\n",
        "doc = nlp(\"The sky is blue.\")\n",
        "\n",
        "print(doc[3]._.is_color, '-', doc[3].text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "utYmu4cfi_Ks"
      },
      "source": [
        "If you want to set extension attributes on a Span, you almost always want to use a property extension with a getter. Otherwise, you'd have to update *every possible span ever* by hand to set all the values. In this example, the \"get has color\" function takes the span and returns whether the text of any of the tokens is in the list of colors. After we've processed the doc, we can check different slices of the doc and the custom \"has color\" property returns whether the span contains a color token or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "naAioynPi_Ks",
        "outputId": "36cf6a73-6489-4346-ed07-19dff7a05f20"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  5\n",
            "True - sky is blue\n",
            "False - The sky\n"
          ]
        }
      ],
      "source": [
        "from spacy.tokens import Span\n",
        "\n",
        "def get_has_color(span):\n",
        "    colors = ['red', 'yellow', 'blue']\n",
        "    return any(token.text in colors for token in span)\n",
        "\n",
        "#set extension on the Span with Getter\n",
        "Span.set_extension('has_color', getter=get_has_color, force=True)\n",
        "\n",
        "doc = nlp(\"The sky is blue.\")\n",
        "\n",
        "print(doc[1:4]._.has_color, '-', doc[1:4].text)\n",
        "print(doc[0:2]._.has_color, '-', doc[0:2].text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_2vhB3Gi_Ks"
      },
      "source": [
        "Method extensions make the extension attribute a callable method. You can then pass one or more arguments to it, and compute attribute values dynamically – for example, based on a certain argument or setting. In this example, the method function checks whether the doc contains a token with a given text. The first argument of the method is always the object itself – in this case, the Doc. It's passed in automatically when the method is called. All other function arguments will be arguments on the method extension. In this case, \"token text\". Here, the custom \"has token\" method returns True for the word \"blue\" and False for the word \"cloud\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lZcJp5ETi_Ks",
        "outputId": "03c8fba6-9468-4442-d30f-986afa9e41ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  5\n",
            "True - blue\n",
            "False - cloud\n"
          ]
        }
      ],
      "source": [
        "from spacy.tokens import Doc\n",
        "\n",
        "#define methods with arguments\n",
        "def has_token(doc, token_text):\n",
        "    in_doc = token_text in [token.text for token in doc]\n",
        "    return in_doc\n",
        "\n",
        "#set extension on the Doc with method\n",
        "Doc.set_extension('has_token', method=has_token, force=True)\n",
        "doc = nlp('The sky is blue.')\n",
        "\n",
        "print(doc._.has_token('blue'), '- blue')\n",
        "print(doc._.has_token('cloud'), '- cloud')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q-17QKkqi_Ks",
        "outputId": "fccd391d-beb1-4452-dbdc-910f65623fe1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  8\n",
            "<strong>Hello world</strong>\n"
          ]
        }
      ],
      "source": [
        "# Define the method\n",
        "def to_html(span, tag):\n",
        "    # Wrap the span text in a HTML tag and return it\n",
        "    return '<{tag}>{text}</{tag}>'.format(tag=tag, text=span.text)\n",
        "\n",
        "# Register the Span property extension 'to_html' with the method to_html\n",
        "Span.set_extension('to_html', method=to_html, force=True)\n",
        "\n",
        "# Process the text and call the to_html method on the span with the tag name 'strong'\n",
        "doc = nlp(\"Hello world, this is a sentence.\")\n",
        "span = doc[0:2]\n",
        "print(span._.to_html('strong'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0uMoiYJi_Ks"
      },
      "source": [
        "### Processing large volumes of text\n",
        "\n",
        "- Use `nlp.pipe` method \n",
        "\n",
        "- Processes texts as a stream, yields Doc objects\n",
        "\n",
        "- Much faster than calling `nlp` on each text\n",
        "\n",
        "\n",
        "BAD \n",
        "```\n",
        "docs = [nlp(text) for text in LOTS_OF_TEXTS]\n",
        "```\n",
        "\n",
        "GOOD\n",
        "```\n",
        "docs = list(nlp.pipe(LOTS_OF_TEXTS))\n",
        "```\n",
        "\n",
        "\n",
        "-   Setting `as_tuples=True` on `nlp.pipe` lets you pass in (text, context) tuples\n",
        "\n",
        "- Yields (doc, context) tuples\n",
        "\n",
        "- Useful for associating metadata with the doc "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vREzTkePi_Kt",
        "outputId": "f4ab567c-d2fc-45a4-aa49-1737503c50ff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  4\n",
            "Doc length:  3\n",
            "This is a text 15\n",
            "And another text 16\n"
          ]
        }
      ],
      "source": [
        "data = [\n",
        "    ('This is a text', {'id': 1, 'page_number': 15}),\n",
        "    ('And another text', {'id': 2, 'page_number': 16}),\n",
        "]\n",
        "\n",
        "for doc, context in nlp.pipe(data, as_tuples=True):\n",
        "    print(doc.text, context['page_number'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCFMPY6ui_Kt"
      },
      "source": [
        "### Using only the tokenizer\n",
        "\n",
        "- User `nlp.make_doc` to turn a text in to a Doc object\n",
        "\n",
        "Sometimes you already have a model loaded to do other processing, but you only need the tokenizer for one particular text. Running the whole pipeline is unnecessarily slow, because you'll be getting a bunch of predictions from the model that you don't need.\n",
        "\n",
        "If you only need a tokenized Doc object, you can use the nlp dot make doc method instead, which takes a text and returns a Doc. This is also how spaCy does it behind the scenes: nlp dot make doc turns the text into a Doc before the pipeline components are called.\n",
        "\n",
        "BAD\n",
        "`doc = nlp(\"Hello world\")`\n",
        "\n",
        "GOOD\n",
        "`doc = nlp.make_doc(\"Hello world\")`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GsyTpiGQi_Kt"
      },
      "source": [
        "### Disable pipeline components\n",
        "\n",
        "- Use `nlp.disable_pipes` to temporarily disable one or more pipes\n",
        "\n",
        "```\n",
        "with nlp.disable_pipes('tagger', 'parser'):\n",
        "    #Process the text and print the entities\n",
        "    doc = nlp(text)\n",
        "    print(doc.ents)\n",
        "```\n",
        "\n",
        "- restores them after the `with` block\n",
        "\n",
        "- only run the remaining components"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-pzw70zi_Kt"
      },
      "source": [
        "### Why updating the model? \n",
        "\n",
        "- Better results on your specific domain\n",
        "\n",
        "- Learn classification schemes specifically for your problem\n",
        "\n",
        "- Essential for text classification \n",
        "\n",
        "- Very useful for named entity recognition\n",
        "\n",
        "- Less critical for part-of-speech tagging and dependency parsing\n",
        "\n",
        "How? \n",
        "\n",
        "1. Initialize the model weights randomly with `nlp.begin_training`\n",
        "\n",
        "2. Predict a few examples with the current weights by calling `nlp.update`\n",
        "\n",
        "3. Compare prediction with true labels \n",
        "\n",
        "4. Calculate how to change weights to improve predictions\n",
        "\n",
        "5. Update weights slightly\n",
        "\n",
        "6. Go back to 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5dDZKmfi_Kt"
      },
      "source": [
        "#### Example: Training the entity recognizer \n",
        "\n",
        "- The entity recognizer tags words and phrases in context\n",
        "\n",
        "- Each token can only be part of one entity\n",
        "\n",
        "- Examples need to come with context: \n",
        "```\n",
        "(\"iPhone X is coming\", {'entities': [(0, 8, 'GADGET')]})\n",
        "```\n",
        "\n",
        "- Texts with no entities are also important: \n",
        "```\n",
        "(\"I need a new phone! Any tips?\", {'entities': []})\n",
        "```\n",
        "\n",
        "- The goal is to teach the model to generalize "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lwVuFzx4i_Kt",
        "outputId": "8b714ebc-25f4-4c3d-a071-61fd9abb64ce"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  6\n",
            "Doc length:  4\n",
            "Doc length:  10\n",
            "Doc length:  6\n",
            "Doc length:  7\n",
            "Doc length:  9\n",
            "How to preorder the iPhone X [(4, 6, 'GADGET'), (4, 5, 'GADGET')]\n",
            "iPhone X is coming [(0, 2, 'GADGET'), (0, 1, 'GADGET')]\n",
            "Should I pay $1,000 for the iPhone X? [(7, 9, 'GADGET'), (7, 8, 'GADGET')]\n",
            "The iPhone 8 reviews are here [(1, 2, 'GADGET'), (1, 3, 'GADGET')]\n",
            "Your iPhone goes up to 11 today [(1, 2, 'GADGET')]\n",
            "I need a new phone! Any tips? []\n"
          ]
        }
      ],
      "source": [
        "matcher_c = Matcher(nlp.vocab)\n",
        "\n",
        "#Creating training data\n",
        "TEXTS = ['How to preorder the iPhone X', 'iPhone X is coming', 'Should I pay $1,000 for the iPhone X?', 'The iPhone 8 reviews are here', 'Your iPhone goes up to 11 today', 'I need a new phone! Any tips?']\n",
        "\n",
        "# Two tokens whose lowercase forms match 'iphone' and 'x'\n",
        "pattern1 = [{'LOWER': 'iphone'}, {'LOWER': 'x'}]\n",
        "\n",
        "# Token whose lowercase form matches 'iphone' and an optional digit\n",
        "pattern2 = [{'LOWER': 'iphone'}, {'IS_DIGIT': True, 'OP': '?'}]\n",
        "\n",
        "# Add patterns to the matcher\n",
        "matcher_c.add('GADGET', [pattern1, pattern2])\n",
        "\n",
        "for doc in nlp.pipe(TEXTS):\n",
        "    # Find the matches in the doc\n",
        "    matches = matcher_c(doc)\n",
        "    \n",
        "    # Get a list of (start, end, label) tuples of matches in the text\n",
        "    entities = [(start, end, 'GADGET') for match_id, start, end in matches]\n",
        "    print(doc.text, entities)  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1s3jNpm6i_Kt",
        "outputId": "1e00d15d-7095-4f4f-cbe4-b327b0669c82"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Doc length:  6\n",
            "Doc length:  4\n",
            "Doc length:  10\n",
            "Doc length:  6\n",
            "Doc length:  7\n",
            "Doc length:  9\n",
            "('How to preorder the iPhone X', {'entities': []})\n",
            "('iPhone X is coming', {'entities': []})\n",
            "('Should I pay $1,000 for the iPhone X?', {'entities': []})\n",
            "('The iPhone 8 reviews are here', {'entities': []})\n",
            "('Your iPhone goes up to 11 today', {'entities': []})\n",
            "('I need a new phone! Any tips?', {'entities': []})\n"
          ]
        }
      ],
      "source": [
        "TRAINING_DATA = []\n",
        "\n",
        "# Create a Doc object for each text in TEXTS\n",
        "for doc in nlp.pipe(TEXTS):\n",
        "    # Match on the doc and create a list of matched spans\n",
        "    spans = [doc[start:end] for match_id, start, end in matcher(doc)]\n",
        "    # Get (start character, end character, label) tuples of matches\n",
        "    entities = [(span.start_char, span.end_char, 'GADGET') for span in spans]\n",
        "    \n",
        "    # Format the matches as a (doc.text, entities) tuple\n",
        "    training_example = (doc.text, {'entities': entities})\n",
        "    # Append the example to the training data\n",
        "    TRAINING_DATA.append(training_example)\n",
        "    \n",
        "print(*TRAINING_DATA, sep='\\n')    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7egJO8Ni_Ku"
      },
      "source": [
        "The steps of a training loop: \n",
        "\n",
        "1. Loop for a number of times.\n",
        "The training loop is a series of steps that's performed to train or update a model. We usually need to perform it several times, for multiple iterations, so that the model can learn from it effectively. If we want to train for 10 iterations, we need to loop 10 times. \n",
        "\n",
        "\n",
        "2. Shuffle the training data. \n",
        "To prevent the model from getting stuck in a suboptimal solution, we randomly shuffle the data for each iteration. This is a very common strategy when doing stochastic gradient descent. \n",
        "\n",
        "\n",
        "3. Divide the data into batches. \n",
        "Next, we divide the training data into batches of several examples, also known as minibatching. This makes it easier to make a more accurate estimate of the gradient.\n",
        "\n",
        "\n",
        "4. Update the model for each batch. \n",
        "Finally, we update the model for each batch, and start the loop again until we've reached the last iteration.\n",
        "\n",
        "\n",
        "4. Save the updated model. \n",
        "We can then save the model to a directory and use it in spaCy."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "# Example loop\n",
        "from __future__ import annotations\n",
        "import random\n",
        "\n",
        "\n",
        "TRAINING_DATA = [\n",
        "    (\"How to preorder the iPhone X\", {'entities': [(20, 28, 'GADGET')]})\n",
        "    # and many more examples...\n",
        "]\n",
        "\n",
        "# loop for 10 iterations\n",
        "for i in range(10):\n",
        "    #shuffle the training data\n",
        "    random.shuffle(TRAINING_DATA)\n",
        "\n",
        "    #create batches and iterate over them\n",
        "    for batch in spacy.util.minibatch(TRAINING_DATA, size=len(TRAINING_DATA)):\n",
        "        #split the batch in texts in annotations\n",
        "        text = [text for text, annotation in batch]\n",
        "        annotations = [annotation for text, annotation in batch]\n",
        "\n",
        "        #update the model\n",
        "        nlp.update(texts, annotations)\n",
        "\n",
        "# save the model\n",
        "nlp.to_disk(path_to_model)        \n",
        "```"
      ],
      "metadata": {
        "id": "zYfpzgVWm5ln"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "# Setting up a new pipeline from scratch\n",
        "\n",
        "# start with blank english model\n",
        "nlp = spacy.blank('en')\n",
        "\n",
        "#create blank entity recognizer and add it to the pipeline\n",
        "ner = nlp.create_pipe('ner')\n",
        "nlp.add_pipe(ner)\n",
        "\n",
        "#add new label\n",
        "ner.add_label('GADGET')\n",
        "\n",
        "#start the training\n",
        "nlp.begin_training()\n",
        "\n",
        "#train for 10 iterations\n",
        "for itn in range(10):\n",
        "    random.shuffle(examples)\n",
        "\n",
        "    #divide examples into batches\n",
        "    for batch in spacy.util.minibatch(examples, size=2):\n",
        "        texts = [text for text, annotation in batch]\n",
        "        annotations = [annotation for text, annotation in batch]\n",
        "\n",
        "        #update the model\n",
        "        nlp.update(texts, annotations)\n",
        "```"
      ],
      "metadata": {
        "id": "f95cy939nGOJ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yhWM8TKi_Ku"
      },
      "source": [
        "### Training best practices\n",
        "\n",
        "- Problem 1: Models can \"forget\" things\n",
        "    - Statistical models can learn lots of things – but it doesn't mean that they won't unlearn them. If you're updating an existing model with new data, especially new labels, it can overfit and adjust *too much* to the new examples. For instance, if you're only updating it with examples of \"website\", it may \"forget\" other labels it previously predicted correctly – like \"person\". This is also known as the catastrophic forgetting problem.\n",
        "    - Existing model can overfit on new data\n",
        "        - Ex: if you only update it with WEBSITE, it can \"unlearn\" what a PERSON is\n",
        "    - Also known as \"catastrophic forgetting\" problem\n",
        "    \n",
        "- Solution 1: Mix in previously correct predictions\n",
        "    - To prevent this, make sure to always mix in examples of what the model previously got correct. If you're training a new category \"website\", also include examples of \"person\". spaCy can help you with this. You can create those additional examples by running the existing model over data and extracting the entity spans you care about. You can then mix those examples in with your existing data and update the model with annotations of all labels.\n",
        "    - For example, if you are training WEBSITE, also include examples of PERSON\n",
        "    - Run existing spaCy model over data and extract all other relevant entities\n",
        "```\n",
        "    BAD \n",
        "    TRAINING_DATA = [\n",
        "        ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]})\n",
        "    ]\n",
        "\n",
        "    GOOD \n",
        "    TRAINING_DATA = [\n",
        "        ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]}),\n",
        "        ('Obama is a person', {'entities': [(0, 5, 'PERSON')]}),\n",
        "    ]\n",
        "```    \n",
        "\n",
        "\n",
        "- Problem 2: Models can't learn everything\n",
        "    - Another common problem is that your model just won't learn what you want it to. spaCy's models make predictions based on the local context – for example, for named entities, the surrounding words are most important. If the decision is difficult to make based on the context, the model can struggle to learn it. The label scheme also needs to be consistent and not too specific.\n",
        "    - spaCy's models make predictions based on local context\n",
        "    - Model can struggle to learn if decision is difficult to make based on context\n",
        "    - Label scheme need to be consistent and not too specific: \n",
        "        - Ex: CLOTHING is better than ADULT_CLOTHING and CHILDRENS_CLOTHING\n",
        "\n",
        "- Solution 2: Plan your label scheme carefully\n",
        "    - Before you start training and updating models, it's worth taking a step back and planning your label scheme. Try to pick categories that are reflected in the local context and make them more generic if possible. You can always add a rule-based system later to go from generic to specific.\n",
        "    - Pick categories that are reflected in local context\n",
        "    - More generic is better than too specific\n",
        "    - Use rules to go from generic labels to specific categories         \n",
        "\n",
        "```\n",
        "    BAD \n",
        "    LABELS = ['ADULT_SHOES', 'CHILDREN_SHOES', 'BANDS_I_LIKE']\n",
        "\n",
        "    GOOD \n",
        "    LABELS = ['CLOTHING', 'BANDS']\n",
        "```      "
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3.8.13 ('mypython3')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "4bdafb2bf0c7bd936307a677d16fd9d95eb603ddfd0469c9830015bd83c0c8fa"
      }
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}