Chapter_04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a neural network model\n",
    "\n",
    "https://course.spacy.io/chapter4\n",
    "\n",
    "In this chapter, you'll learn how to update spaCy's statistical models to customize them for your use case – for example, to predict a new entity type in online comments. You'll write your own training loop from scratch, and understand the basics of how training works, along with tips and tricks that can make your custom NLP projects more successful.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and updating models\n",
    "\n",
    "Welcome to the final chapter, which is about one of the most exciting aspects of modern NLP: training your own models!\n",
    "\n",
    "In this lesson, you'll learn about training and updating spaCy's neural network models and the data you need for it – focusing specifically on the named entity recognizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why updating the model?\n",
    "\n",
    "* Better results on your specific domain\n",
    "* Learn classification schemes specifically for your problem\n",
    "* Essential for text classification\n",
    "* Very useful for named entity recognition\n",
    "* Less critical for part-of-speech tagging and dependency parsing\n",
    "\n",
    "Before we get starting with explaining how, it's worth taking a second to ask ourselves: Why would we want to update the model with our own examples? Why can't we just rely on pre-trained models?\n",
    "\n",
    "Statistical models make predictions based on the examples they were trained on.\n",
    "\n",
    "You can usually make the model more accurate by showing it examples from your domain.\n",
    "\n",
    "You often also want to predict categories specific to your problem, so the model needs to learn about them.\n",
    "\n",
    "This is essential for text classification, very useful for entity recognition and a little less critical for tagging and parsing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How training works\n",
    "\n",
    "1. Initialize the model weights randomly with nlp.begin_training\n",
    "\n",
    "2. Predict a few examples with the current weights by calling nlp.update\n",
    "\n",
    "3. Compare prediction with true labels\n",
    "\n",
    "4. Calculate how to change weights to improve predictions\n",
    "\n",
    "5. Update weights slightly\n",
    "\n",
    "6. Go back to 2.\n",
    "\n",
    "spaCy supports updating existing models with more examples, and training new models.\n",
    "\n",
    "If we're not starting with a pre-trained model, we first initialize the weights randomly.\n",
    "\n",
    "Next, we call nlp dot update, which predicts a batch of examples with the current weights.\n",
    "\n",
    "The model then checks the predictions against the correct answers, and decides how to change the weights to achieve better predictions next time.\n",
    "\n",
    "Finally, we make a small correction to the current weights and move on to the next batch of examples.\n",
    "\n",
    "We continue calling nlp dot update for each batch of examples in the data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an illustration showing the process.\n",
    "\n",
    "<img src=\"img/trainning1.png\" />\n",
    "\n",
    "The training data are the examples we want to update the model with.\n",
    "\n",
    "The text should be a sentence, paragraph or longer document. For the best results, it should be similar to what the model will see at runtime.\n",
    "\n",
    "The label is what we want the model to predict. This can be a text category, or an entity span and its type.\n",
    "\n",
    "The gradient is how we should change the model to reduce the current error. It's computed when we compare the predicted label to the true label.\n",
    "\n",
    "After training, we can then save out an updated model and use it in our application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Training data: Examples and their annotations.\n",
    "\n",
    "* Text: The input text the model should predict a label for.\n",
    "\n",
    "* Label: The label the model should predict.\n",
    "\n",
    "* Gradient: How to change the weights.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the entity recognizer\n",
    "\n",
    "Let's look at an example for a specific component: the entity recognizer.\n",
    "\n",
    "The entity recognizer takes a document and predicts phrases and their labels. This means that the training data needs to include texts, the entities they contain, and the entity labels.\n",
    "\n",
    "Entities can't overlap, so each token can only be part of one entity.\n",
    "\n",
    "Because the entity recognizer predicts entities in context, it also needs to be trained on entities and their surrounding context.\n",
    "\n",
    "The easiest way to do this is to show the model a text and a list of character offsets. For example, \"iPhone X\" is a gadget, starts at character 0 and ends at character 8.\n",
    "\n",
    "It's also very important for the model to learn words that aren't entities.\n",
    "\n",
    "In this case, the list of span annotations will be empty.\n",
    "\n",
    "Our goal is to teach the model to recognize new entities in similar contexts, even if they weren't in the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The entity recognizer tags words and phrases in context\n",
    "\n",
    "* Each token can only be part of one entity\n",
    "\n",
    "* Examples need to come with context\n",
    "\n",
    "(\"iPhone X is coming\", {'entities': [(0, 8, 'GADGET')]})\n",
    "\n",
    "* Texts with no entities are also important\n",
    "\n",
    "(\"I need a new phone! Any tips?\", {'entities': []})\n",
    "\n",
    "* Goal: teach the model to generalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The training data\n",
    "\n",
    "The training data tells the model what we want it to predict. This could be texts and named entities we want to recognize, or tokens and their correct part-of-speech tags.\n",
    "\n",
    "To update an existing model, we can start with a few hundred to a few thousand examples.\n",
    "\n",
    "To train a new category we may need up to a million.\n",
    "\n",
    "spaCy's pre-trained English models for instance were trained on 2 million words labelled with part-of-speech tags, dependencies and named entities.\n",
    "\n",
    "Training data is usually created by humans who assign labels to texts.\n",
    "\n",
    "This is a lot of work, but can be semi-automated – for example, using spaCy's Matcher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Examples of what we want the model to predict in context\n",
    "\n",
    "* Update an existing model: a few hundred to a few thousand examples\n",
    "    \n",
    "* Train a new category: a few thousand to a million examples\n",
    "    \n",
    "* spaCy's English models: 2 million words\n",
    "\n",
    "* Usually created manually by human annotators\n",
    "\n",
    "* Can be semi-automated – for example, using spaCy's Matcher!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to get started and prepare the training data. Let's look at some examples and create a small dataset for a new entity type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Purpose of training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While spaCy comes with a range of pre-trained models to predict linguistic annotations, you almost always want to fine-tune them with more examples. You can do this by training them with more labelled data.\n",
    "\n",
    "What does training not help with?\n",
    "\n",
    "( ) Improve model accuracy on your data.\n",
    "\n",
    "( ) Learn new classification schemes.\n",
    "\n",
    "(X) Discover patterns in unlabelled data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating training data\n",
    "\n",
    "spaCy’s rule-based *Matcher*  is a great way to quickly create training data for named entity models. A list of sentences is available as the variable *TEXTS*. You can print it the IPython shell to inspect it. We want to find all mentions of different iPhone models, so we can create training data to teach a model to recognize them as 'GADGET'.\n",
    "\n",
    "Write a pattern for two tokens whose lowercase forms match 'iphone' and 'x'.\n",
    "Write a pattern for two tokens: one token whose lowercase form matches 'iphone' and an optional digit using the '?' operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from spacy.matcher import Matcher\n",
    "from spacy.lang.en import English\n",
    "\n",
    "with open(\"exercises/iphone.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "nlp = English()\n",
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "# Two tokens whose lowercase forms match 'iphone' and 'x'\n",
    "pattern1 = [{\"LOWER\": \"iphone\"}, {\"LOWER\": \"x\"}]\n",
    "\n",
    "# Token whose lowercase form matches 'iphone' and an optional digit\n",
    "pattern2 = [{\"LOWER\": \"iphone\"}, {\"IS_DIGIT\": True, \"OP\": \"?\"}]\n",
    "\n",
    "# Add patterns to the matcher\n",
    "matcher.add(\"GADGET\", None, pattern1, pattern2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s use the match patterns we’ve created in the previous exercise to bootstrap a set of training examples. A list of sentences is available as the variable TEXTS.\n",
    "\n",
    "* Create a doc object for each text using nlp.pipe.\n",
    "* Match on the doc and create a list of matched spans.\n",
    "* Get (start character, end character, label) tuples of matched spans.\n",
    "* Format each example as a tuple of the text and a dict, mapping 'entities' to the entity tuples.\n",
    "* Append the example to TRAINING_DATA and inspect the printed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "document: How to preorder the iPhone X\n",
      "[iPhone X, iPhone]\n",
      "[(20, 28, 'GADGET'), (20, 26, 'GADGET')]\n",
      "('How to preorder the iPhone X', {'entities': [(20, 28, 'GADGET'), (20, 26, 'GADGET')]})\n",
      "\n",
      "document: iPhone X is coming\n",
      "[iPhone X, iPhone]\n",
      "[(0, 8, 'GADGET'), (0, 6, 'GADGET')]\n",
      "('iPhone X is coming', {'entities': [(0, 8, 'GADGET'), (0, 6, 'GADGET')]})\n",
      "\n",
      "document: Should I pay $1,000 for the iPhone X?\n",
      "[iPhone X, iPhone]\n",
      "[(28, 36, 'GADGET'), (28, 34, 'GADGET')]\n",
      "('Should I pay $1,000 for the iPhone X?', {'entities': [(28, 36, 'GADGET'), (28, 34, 'GADGET')]})\n",
      "\n",
      "document: The iPhone 8 reviews are here\n",
      "[iPhone 8]\n",
      "[(4, 12, 'GADGET')]\n",
      "('The iPhone 8 reviews are here', {'entities': [(4, 12, 'GADGET')]})\n",
      "\n",
      "document: Your iPhone goes up to 11 today\n",
      "[iPhone]\n",
      "[(5, 11, 'GADGET')]\n",
      "('Your iPhone goes up to 11 today', {'entities': [(5, 11, 'GADGET')]})\n",
      "\n",
      "document: I need a new phone! Any tips?\n",
      "[]\n",
      "[]\n",
      "('I need a new phone! Any tips?', {'entities': []})\n",
      "\n",
      "('How to preorder the iPhone X', {'entities': [(20, 28, 'GADGET'), (20, 26, 'GADGET')]})\n",
      "('iPhone X is coming', {'entities': [(0, 8, 'GADGET'), (0, 6, 'GADGET')]})\n",
      "('Should I pay $1,000 for the iPhone X?', {'entities': [(28, 36, 'GADGET'), (28, 34, 'GADGET')]})\n",
      "('The iPhone 8 reviews are here', {'entities': [(4, 12, 'GADGET')]})\n",
      "('Your iPhone goes up to 11 today', {'entities': [(5, 11, 'GADGET')]})\n",
      "('I need a new phone! Any tips?', {'entities': []})\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from spacy.matcher import Matcher\n",
    "from spacy.lang.en import English\n",
    "\n",
    "with open(\"exercises/iphone.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "nlp = English()\n",
    "matcher = Matcher(nlp.vocab)\n",
    "pattern1 = [{\"LOWER\": \"iphone\"}, {\"LOWER\": \"x\"}]\n",
    "pattern2 = [{\"LOWER\": \"iphone\"}, {\"IS_DIGIT\": True, \"OP\": \"?\"}]\n",
    "matcher.add(\"GADGET\", None, pattern1, pattern2)\n",
    "\n",
    "TRAINING_DATA = []\n",
    "\n",
    "# Create a Doc object for each text in TEXTS\n",
    "for doc in nlp.pipe(TEXTS):\n",
    "    print(\"document: \"+doc.text)\n",
    "    # Match on the doc and create a list of matched spans\n",
    "    spans = [doc[start:end] for match_id, start, end in matcher(doc)]\n",
    "    print(spans)\n",
    "    # Get (start character, end character, label) tuples of matches\n",
    "    entities = [(span.start_char, span.end_char, \"GADGET\") for span in spans]\n",
    "    print(entities)\n",
    "    # Format the matches as a (doc.text, entities) tuple\n",
    "    training_example = (doc.text, {\"entities\": entities})\n",
    "    print(training_example)\n",
    "    # Append the example to the training data\n",
    "    TRAINING_DATA.append(training_example)\n",
    "    print()\n",
    "print(*TRAINING_DATA, sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The training loop\n",
    "\n",
    "While some other libraries give you one method that takes care of training a model, spaCy gives you full control over the training loop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The steps of a training loop:\n",
    "    \n",
    "1 Loop for a number of times.\n",
    "\n",
    "2 Shuffle the training data.\n",
    "\n",
    "3 Divide the data into batches.\n",
    "\n",
    "4 Update the model for each batch.\n",
    "\n",
    "5 Save the updated model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training loop is a series of steps that's performed to train or update a model.\n",
    "\n",
    "We usually need to perform it several times, for multiple iterations, so that the model can learn from it effectively. If we want to train for 10 iterations, we need to loop 10 times.\n",
    "\n",
    "To prevent the model from getting stuck in a suboptimal solution, we randomly shuffle the data for each iteration. This is a very common strategy when doing stochastic gradient descent.\n",
    "\n",
    "Next, we divide the training data into batches of several examples, also known as minibatching. This makes it easier to make a more accurate estimate of the gradient.\n",
    "\n",
    "Finally, we update the model for each batch, and start the loop again until we've reached the last iteration.\n",
    "\n",
    "We can then save the model to a directory and use it in spaCy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recap: how training works\n",
    "\n",
    "<img src=\"img/trainning1.png\" />\n",
    "\n",
    "* Training data: Examples and their annotations.\n",
    "* Text: The input text the model should predict a label for.\n",
    "* Label: The label the model should predict.\n",
    "* Gradient: How to change the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To recap:\n",
    "\n",
    "The training data are the examples we want to update the model with.\n",
    "\n",
    "The text should be a sentence, paragraph or longer document. For the best results, it should be similar to what the model will see at runtime.\n",
    "\n",
    "The label is what we want the model to predict. This can be a text category, or an entity span and its type.\n",
    "\n",
    "The gradient is how we should change the model to reduce the current error. It's computed when we compare the predicted label to the true label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example loop\n",
    "\n",
    "Here's an example.\n",
    "\n",
    "Let's imagine we have a list of training examples consisting of texts and entity annotations.\n",
    "\n",
    "We want to loop for 10 iterations, so we're iterating over a range of 10.\n",
    "\n",
    "Next, we use the random module to randomly shuffle the training data.\n",
    "\n",
    "We then use spaCy's minibatch utility function to divide the examples into batches.\n",
    "\n",
    "For each batch, we get the texts and annotations and call the nlp dot update method to update the model.\n",
    "\n",
    "Finally, we call the nlp dot to disk method to save the trained model to a directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "TRAINING_DATA = [\n",
    "    (\"How to preorder the iPhone X\", {'entities': [(20, 28, 'GADGET')]})\n",
    "    # And many more examples...\n",
    "]\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n# Loop for 10 iterations\\nfor i in range(10):\\n    # Shuffle the training data\\n    random.shuffle(TRAINING_DATA)\\n    # Create batches and iterate over them\\n    for batch in spacy.util.minibatch(TRAINING_DATA):\\n        # Split the batch in texts and annotations\\n        texts = [text for text, annotation in batch]\\n        annotations = [annotation for text, annotation in batch]\\n        # Update the model\\n        nlp.update(texts, annotations)\\n\\n# Save the model\\nnlp.to_disk(\"example\")\\n'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "# Loop for 10 iterations\n",
    "for i in range(10):\n",
    "    # Shuffle the training data\n",
    "    random.shuffle(TRAINING_DATA)\n",
    "    # Create batches and iterate over them\n",
    "    for batch in spacy.util.minibatch(TRAINING_DATA):\n",
    "        # Split the batch in texts and annotations\n",
    "        texts = [text for text, annotation in batch]\n",
    "        annotations = [annotation for text, annotation in batch]\n",
    "        # Update the model\n",
    "        nlp.update(texts, annotations)\n",
    "\n",
    "# Save the model\n",
    "nlp.to_disk(\"example\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update an existing model\n",
    "\n",
    "* Improve the predictions on new data\n",
    "* Especially useful to improve existing categories, like PERSON\n",
    "* Also possible to add new categories\n",
    "* Be careful and make sure the model doesn't \"forget\" the old ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy lets you update an existing pre-trained model with more data – for example, to improve its predictions on different texts.\n",
    "\n",
    "This is especially useful if you want to improve categories the model already knows, like \"person\" or \"organization\".\n",
    "\n",
    "You can also update a model to add new categories.\n",
    "\n",
    "Just make sure to always update it with examples of the new category and examples of the other categories it previously predicted correctly. Otherwise improving the new category might hurt the other categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up a new pipeline from scratch\n",
    "\n",
    "In this example, we start off with a blank English model using the spacy dot blank method. The blank model doesn't have any pipeline components, only the language data and tokenization rules.\n",
    "\n",
    "We then create a blank entity recognizer and add it to the pipeline.\n",
    "\n",
    "Using the \"add label\" method, we can add new string labels to the model.\n",
    "\n",
    "We can now call nlp dot begin training to initialize the model with random weights.\n",
    "\n",
    "To get better accuracy, we want to loop over the examples more than once and randomly shuffle the data on each iteration.\n",
    "\n",
    "On each iteration, we divide the examples into batches using spaCy's minibatch utility function. Each example consists of a text and its annotations.\n",
    "\n",
    "Finally, we update the model with the texts and annotations and continue the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n# Start with blank English model\\nnlp = spacy.blank('en')\\n# Create blank entity recognizer and add it to the pipeline\\nner = nlp.create_pipe('ner')\\nnlp.add_pipe(ner)\\n# Add a new label\\nner.add_label('GADGET')\\n\\n# Start the training\\nnlp.begin_training()\\n# Train for 10 iterations\\nfor itn in range(10):\\n    random.shuffle(examples)\\n    # Divide examples into batches\\n    for batch in spacy.util.minibatch(examples, size=2):\\n        texts = [text for text, annotation in batch]\\n        annotations = [annotation for text, annotation in batch]\\n        # Update the model\\n        nlp.update(texts, annotations)\\n        \""
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "# Start with blank English model\n",
    "nlp = spacy.blank('en')\n",
    "# Create blank entity recognizer and add it to the pipeline\n",
    "ner = nlp.create_pipe('ner')\n",
    "nlp.add_pipe(ner)\n",
    "# Add a new label\n",
    "ner.add_label('GADGET')\n",
    "\n",
    "# Start the training\n",
    "nlp.begin_training()\n",
    "# Train for 10 iterations\n",
    "for itn in range(10):\n",
    "    random.shuffle(examples)\n",
    "    # Divide examples into batches\n",
    "    for batch in spacy.util.minibatch(examples, size=2):\n",
    "        texts = [text for text, annotation in batch]\n",
    "        annotations = [annotation for text, annotation in batch]\n",
    "        # Update the model\n",
    "        nlp.update(texts, annotations)\n",
    "        '''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to practice! Now that you've seen the training loop, let's use the data created in the previous exercise to update a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the pipeline\n",
    "\n",
    "In this exercise, you’ll prepare a spaCy pipeline to train the entity recognizer to recognize 'GADGET' entities in a text – for example, “iPhone X”.\n",
    "\n",
    "* Create a blank 'en' model, for example using the spacy.blank method.\n",
    "* Create a new entity recognizer using nlp.create_pipe and add it to the pipeline.\n",
    "* Add the new label 'GADGET' to the entity recognizer using the add_label method on the pipeline component.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# Create a blank 'en' model\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "# Create a new entity recognizer and add it to the pipeline\n",
    "ner = nlp.create_pipe(\"ner\")\n",
    "nlp.add_pipe(ner)\n",
    "\n",
    "# Add the label 'GADGET' to the entity recognizer\n",
    "ner.add_label(\"GADGET\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a training loop\n",
    "\n",
    "Let’s write a simple training loop from scratch!\n",
    "\n",
    "The pipeline you’ve created in the previous exercise is available as the nlp object. It already contains the entity recognizer with the added label 'GADGET'.\n",
    "\n",
    "The small set of labelled examples that you’ve created previously is available as TRAINING_DATA. To see the examples, you can print them in your script.\n",
    "\n",
    "Call nlp.begin_training, create a training loop for 10 iterations and shuffle the training data.\n",
    "Create batches of training data using spacy.util.minibatch and iterate over the batches.\n",
    "Convert the (text, annotations) tuples in the batch to lists of texts and annotations.\n",
    "For each batch, use nlp.update to update the model with the texts and annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2457692833\n",
      "3.2140585109\n",
      "3.4722888768\n",
      "0.0000015177\n",
      "0.5200842448\n",
      "0.5200924764\n",
      "0.0000000003\n",
      "0.0000000077\n",
      "0.0000000077\n",
      "0.0000000000\n",
      "0.0000077896\n",
      "0.0000088767\n",
      "0.0000000349\n",
      "0.0000000349\n",
      "0.0000000354\n",
      "0.0000000000\n",
      "0.0000000003\n",
      "0.0000000003\n",
      "0.0000000000\n",
      "0.0000000000\n",
      "0.0000000001\n",
      "0.0000000001\n",
      "0.0000000002\n",
      "0.0000000002\n",
      "0.0000000006\n",
      "0.0000000006\n",
      "0.0000000006\n",
      "0.0000000000\n",
      "0.0000000000\n",
      "0.0000000050\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "import random\n",
    "import json\n",
    "   \n",
    "with open(\"exercises/gadgets.json\") as f:\n",
    "    TRAINING_DATA = json.loads(f.read())\n",
    "\n",
    "nlp = spacy.blank(\"en\")\n",
    "ner = nlp.create_pipe(\"ner\")\n",
    "nlp.add_pipe(ner)\n",
    "ner.add_label(\"GADGET\")\n",
    "\n",
    "nlp.vocab.vectors.name = 'example'\n",
    "\n",
    "# Start the training\n",
    "nlp.begin_training()\n",
    "\n",
    "# Loop for 10 iterations\n",
    "for itn in range(10):\n",
    "    # Shuffle the training data\n",
    "    random.shuffle(TRAINING_DATA)\n",
    "    losses = {}\n",
    "\n",
    "    # Batch the examples and iterate over them\n",
    "    for batch in spacy.util.minibatch(TRAINING_DATA, size=2):\n",
    "        texts = [text for text, entities in batch]\n",
    "        annotations = [entities for text, entities in batch]\n",
    "\n",
    "        # Update the model\n",
    "        nlp.update(texts, annotations, losses=losses)\n",
    "        print(\"{0:.10f}\".format(losses['ner']) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the model\n",
    "\n",
    "Let’s see how the model performs on unseen data! To speed things up a little, we already ran a trained model for the label 'GADGET' over some text. Here are some of the results:\n",
    "\n",
    "<img src=\"img/table.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of all the entities in the texts, how many did the model get correct? \n",
    "\n",
    "Keep in mind that incomplete entity spans count as mistakes, too! Tip: Count the number of entities that the model should have predicted. Then count the number of entities it actually predicted correctly and divide it by the number of total correct entities.\n",
    "\n",
    "( ) 45%\n",
    "\n",
    "( ) 60%\n",
    "\n",
    "(X) 70%\n",
    "\n",
    "( ) 90%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training best practices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you start running your own experiments, you might find that a lot of things just don't work the way you want them to. And that's okay.\n",
    "\n",
    "Training models is an iterative process, and you have to try different things until you find out what works best.\n",
    "\n",
    "In this lesson, I'll be sharing some best practices and things to keep in mind when training your own models.\n",
    "\n",
    "Let's take a look at some of the problems you may come across."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1: Models can \"forget\" things\n",
    "\n",
    "* Existing model can overfit on new data\n",
    "e.g.: if you only update it with WEBSITE, it can \"unlearn\" what a PERSON is\n",
    "\n",
    "* Also known as \"catastrophic forgetting\" problem\n",
    "\n",
    "Statistical models can learn lots of things – but it doesn't mean that they won't unlearn them.\n",
    "\n",
    "If you're updating an existing model with new data, especially new labels, it can overfit and adjust too much to the new examples.\n",
    "\n",
    "For instance, if you're only updating it with examples of \"website\", it may \"forget\" other labels it previously predicted correctly – like \"person\".\n",
    "\n",
    "This is also known as the catastrophic forgetting problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution 1: Mix in previously correct predictions\n",
    "\n",
    "To prevent this, make sure to always mix in examples of what the model previously got correct.\n",
    "\n",
    "If you're training a new category \"website\", also include examples of \"person\".\n",
    "\n",
    "spaCy can help you with this. You can create those additional examples by running the existing model over data and extracting the entity spans you care about.\n",
    "\n",
    "You can then mix those examples in with your existing data and update the model with annotations of all labels.\n",
    "\n",
    "For example, if you're training WEBSITE, also include examples of PERSON\n",
    "\n",
    "Run existing spaCy model over data and extract all other relevant entities\n",
    "\n",
    "**BAD:**\n",
    "\n",
    "TRAINING_DATA = [\n",
    "    ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]})\n",
    "]\n",
    "\n",
    "**GOOD:**\n",
    "\n",
    "TRAINING_DATA = [\n",
    "    ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]}),\n",
    "    ('Obama is a person', {'entities': [(0, 5, 'PERSON')]})\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 2: Models can't learn everything\n",
    "\n",
    "Another common problem is that your model just won't learn what you want it to.\n",
    "\n",
    "spaCy's models make predictions based on the local context – for example, for named entities, the surrounding words are most important.\n",
    "\n",
    "If the decision is difficult to make based on the context, the model can struggle to learn it.\n",
    "\n",
    "The label scheme also needs to be consistent and not too specific.\n",
    "\n",
    "For example, it may be very difficult to teach a model to predict whether something is adult clothing or children's clothing based on the context. However, just predicting the label \"clothing\" may work better.\n",
    "\n",
    "spaCy's models make predictions based on local context:\n",
    "\n",
    "* Model can struggle to learn if decision is difficult to make based on context\n",
    "\n",
    "* Label scheme needs to be consistent and not too specific\n",
    "\n",
    "For example: CLOTHING is better than ADULT_CLOTHING and CHILDRENS_CLOTHING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution 2: Plan your label scheme carefully\n",
    "\n",
    "Before you start training and updating models, it's worth taking a step back and planning your label scheme.\n",
    "\n",
    "Try to pick categories that are reflected in the local context and make them more generic if possible.\n",
    "\n",
    "You can always add a rule-based system later to go from generic to specific.\n",
    "\n",
    "Generic categories like \"clothing\" or \"band\" are both easier to label and easier to learn.\n",
    "\n",
    "Pick categories that are reflected in local context:\n",
    "    \n",
    "* More generic is better than too specific\n",
    "\n",
    "* Use rules to go from generic labels to specific categories\n",
    "\n",
    "**BAD:**\n",
    "\n",
    "LABELS = ['ADULT_SHOES', 'CHILDRENS_SHOES', 'BANDS_I_LIKE]\n",
    "          \n",
    "**GOOD:**\n",
    "\n",
    "LABELS = ['CLOTHING', 'BAND']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Good data x bad data\n",
    "\n",
    "Here’s an excerpt from a training set that labels the entity type TOURIST_DESTINATION in traveler reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA = [\n",
    "    (\n",
    "        \"i went to amsterdem last year and the canals were beautiful\",\n",
    "        {\"entities\": [(10, 19, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"You should visit Paris once in your life, but the Eiffel Tower is kinda boring\",\n",
    "        {\"entities\": [(17, 22, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "    (\"There's also a Paris in Arkansas, lol\", {\"entities\": []}),\n",
    "    (\n",
    "        \"Berlin is perfect for summer holiday: lots of parks, great nightlife, cheap beer!\",\n",
    "        {\"entities\": [(0, 6, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is this data and label scheme problematic? \n",
    "\n",
    "( X ) Whether a place is a tourist destination is a subjective judgement and not a definitive category. It will be very difficult for the entity recognizer to learn.\n",
    "\n",
    "( ) Paris and Arkansas should also be labelled as tourist destinations for consistency. Otherwise, the model will be confused.\n",
    "\n",
    "( ) Rare out-of-vocabulary words like the misspelled 'amsterdem' shouldn't be labelled as entities.\n",
    "\n",
    "That's correct! A much better approach would be to only label GPE (geopolitical entity) or LOCATION and then use a rule-based system to determine whether the entity is a tourist destination in this context. For example, you could resolve the entities types back to a knowledge base or look them up in a travel wiki."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rewrite the TRAINING_DATA to only use the label GPE (cities, states, countries) instead of TOURIST_DESTINATION.\n",
    "\n",
    "Don’t forget to add tuples for the GPE entities that weren’t labeled in the old data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA = [\n",
    "    (\n",
    "        \"i went to amsterdem last year and the canals were beautiful\",\n",
    "        {\"entities\": [(10, 19, \"GPE\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"You should visit Paris once in your life, but the Eiffel Tower is kinda boring\",\n",
    "        {\"entities\": [(17, 22, \"GPE\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"There's also a Paris in Arkansas, lol\",\n",
    "        {\"entities\": [(15, 20, \"GPE\"), (24, 32, \"GPE\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"Berlin is perfect for summer holiday: lots of parks, great nightlife, cheap beer!\",\n",
    "        {\"entities\": [(0, 6, \"GPE\")]},\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Traiing multiple labels\n",
    "\n",
    "Here’s a small sample of a dataset created to train a new entity type WEBSITE. The original dataset contains a few thousand sentences. In this exercise, you’ll be doing the labeling by hand. In real life, you probably want to automate this and use an annotation tool – for example, Brat, a popular open-source solution, or Prodigy, our own annotation tool that integrates with spaCy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the entity offsets for the WEBSITE entities in the data. Feel free to use len() if you don’t want to count the characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA = [\n",
    "    (\n",
    "        \"Reddit partners with Patreon to help creators build communities\",\n",
    "        {\"entities\": [(0, 6, \"WEBSITE\"), (21, 28, \"WEBSITE\")]},\n",
    "    ),\n",
    "    (\"PewDiePie smashes YouTube record\", {\"entities\": [(18, 25, \"WEBSITE\")]}),\n",
    "    (\n",
    "        \"Reddit founder Alexis Ohanian gave away two Metallica tickets to fans\",\n",
    "        {\"entities\": [(0, 6, \"WEBSITE\")]},\n",
    "    ),\n",
    "    # And so on...\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A model was trained with the data you just labelled, plus a few thousand similar examples. After training, it’s doing great on WEBSITE, but doesn’t recognize PERSON anymore. Why could this be happening?\n",
    "\n",
    "( ) It's very difficult for the model to learn about different categories like PERSON and WEBSITE.\n",
    "\n",
    "( X ) The training data included no examples of PERSON, so the model learned that this label is incorrect.\n",
    "\n",
    "( ) The hyperparameters need to be retuned so that both entity types can be recognized.\n",
    "\n",
    "**If PERSON entities occur in the training data but aren’t labelled, the model will learn that they shouldn’t be predicted.** \n",
    "\n",
    "Similarly, if an existing entity type isn’t present in the training data, the model may ”forget” and stop predicting it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update the training data to include annotations for the PERSON entities “PewDiePie” and “Alexis Ohanian”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA = [\n",
    "    (\n",
    "        \"Reddit partners with Patreon to help creators build communities\",\n",
    "        {\"entities\": [(0, 6, \"WEBSITE\"), (21, 28, \"WEBSITE\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"PewDiePie smashes YouTube record\",\n",
    "        {\"entities\": [(0, 9, \"PERSON\"), (18, 25, \"WEBSITE\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"Reddit founder Alexis Ohanian gave away two Metallica tickets to fans\",\n",
    "        {\"entities\": [(0, 6, \"WEBSITE\"), (15, 29, \"PERSON\")]},\n",
    "    ),\n",
    "    # And so on...\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up !\n",
    "\n",
    "Congratulations – you've made it to the end of the course!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your new spaCy skills**\n",
    "\n",
    "* Extract linguistic features: part-of-speech tags, dependencies, named entities\n",
    "* Work with pre-trained statistical models\n",
    "* Find words and phrases using Matcher and PhraseMatcher match rules\n",
    "* Best practices for working with data structures Doc, Token Span, Vocab, Lexeme\n",
    "* Find semantic similarities using word vectors\n",
    "* Write custom pipeline components with extension attributes\n",
    "* Scale up your spaCy pipelines and make them fast\n",
    "* Create training data for spaCy' statistical models\n",
    "* Train and update spaCy's neural network models with new data\n",
    "\n",
    "\n",
    "Here's an overview of all the new skills you learned so far:\n",
    "\n",
    "In the first chapter, you learned how to extract linguistic features like part-of-speech tags, syntactic dependencies and named entities, and how to work with pre-trained statistical models.\n",
    "\n",
    "You also learned to write powerful match patterns to extract words and phrases using spaCy's matcher and phrase matcher.\n",
    "\n",
    "Chapter 2 was all about information extraction, and you learned how to work with the data structures, the Doc, Token and Span, as well as the vocab and lexical entries.\n",
    "\n",
    "You also used spaCy to predict semantic similarities using word vectors.\n",
    "\n",
    "In chapter 3, you got some more insights into spaCy's pipeline, and learned to write your own custom pipeline components that modify the Doc.\n",
    "\n",
    "You also created your own custom extension attributes for Docs, Tokens and Spans, and learned about processing streams and making your pipeline faster.\n",
    "\n",
    "Finally, in chapter 4, you learned about training and updating spaCy's statistical models, specifically the entity recognizer.\n",
    "\n",
    "You learned some useful tricks for how to create training data, and how to design your label scheme to get the best results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**More things to do with spaCy**\n",
    "\n",
    "Of course, there's a lot more that spaCy can do that we didn't get to cover in this course.\n",
    "\n",
    "While we focused mostly on training the entity recognizer, you can also train and update the other statistical pipeline components like the part-of-speech tagger and dependency parser.\n",
    "\n",
    "Another useful pipeline component is the text classifier, which can learn to predict labels that apply to the whole text. It's not part of the pre-trained models, but you can add it to an existing model and train it on your own data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and updating other pipeline components:\n",
    "\n",
    "* Part-of-speech tagger\n",
    "* Dependency parser\n",
    "* Text classifier\n",
    "\n",
    "Visit:\n",
    "\n",
    "https://spacy.io/usage/training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}