Chapter_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Large-scale data analysis with spaCy\n",
    "\n",
    "https://course.spacy.io/chapter2\n",
    "\n",
    "In this chapter, you'll use your new skills to extract specific information from large volumes of text. You''ll learn how to make the most of spaCy's data structures, and how to effectively combine statistical and rule-based approaches for text analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.3\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "print(spacy.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data Structures (1): Vocab, Lexemes and StringStore\n",
    "\n",
    "Now that you've had some real experience using spaCy's objects, it's time for you to learn more about what's actually going on under spaCy's hood.\n",
    "\n",
    "In this lesson, we'll take a look at the shared vocabulary and how spaCy deals with strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shared Vocab and string store\n",
    "\n",
    "SpaCy stores all shared data in a vocabulary, the Vocab.\n",
    "\n",
    "This includes words, but also the labels schemes for tags and entities.\n",
    "\n",
    "To save memory, all strings are encoded to hash IDs. If a word occurs more than once, we don't need to save it every time.\n",
    "\n",
    "Instead, spaCy uses a hash function to generate an ID and stores the string only once in the string store. The string store is available as nlp dot vocab dot strings.\n",
    "\n",
    "It's a lookup table that works in both directions. You can look up a string and get its hash, and look up a hash to get its string value. Internally, spaCy only communicates in hash IDs.\n",
    "\n",
    "Hash IDs can't be reversed, though. If a word in not in the vocabulary, there's no way to get its string. That's why we always need to pass around the shared vocab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "976"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "len(nlp.vocab.strings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3197928453018144401\n"
     ]
    }
   ],
   "source": [
    "coffe_hash= nlp.vocab.strings['coffee']\n",
    "print(coffe_hash)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '3197928453018144401'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-d24d695fda14>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mcoffee_string\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mcoffe_hash\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcoffee_string\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '3197928453018144401'.\""
     ]
    }
   ],
   "source": [
    "coffee_string = nlp.vocab.strings[coffe_hash]\n",
    "print(coffee_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "976"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "len(nlp.vocab.strings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5505"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load(\"pt_core_news_sm\")\n",
    "len(nlp.vocab.strings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10569699879655997926\n"
     ]
    }
   ],
   "source": [
    "coffe_hash= nlp.vocab.strings['cafe']\n",
    "print(coffe_hash)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32833993555699147\n"
     ]
    }
   ],
   "source": [
    "coffe_hash= nlp.vocab.strings[u'café']\n",
    "print(coffe_hash)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '99999999999'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-9-9f3701b5a4f8>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m99999999999\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '99999999999'.\""
     ]
    }
   ],
   "source": [
    "nlp.vocab.strings[99999999999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "IS_ALPHA\n",
      "IS_ASCII\n",
      "IS_DIGIT\n",
      "IS_LOWER\n",
      "IS_PUNCT\n",
      "IS_SPACE\n",
      "IS_TITLE\n",
      "IS_UPPER\n",
      "LIKE_URL\n",
      "LIKE_NUM\n",
      "LIKE_EMAIL\n",
      "IS_STOP\n",
      "IS_OOV\n",
      "IS_BRACKET\n",
      "IS_QUOTE\n",
      "IS_LEFT_PUNCT\n",
      "IS_RIGHT_PUNCT\n",
      "IS_CURRENCY\n",
      "FLAG19\n",
      "FLAG20\n",
      "FLAG21\n",
      "FLAG22\n",
      "FLAG23\n",
      "FLAG24\n",
      "FLAG25\n",
      "FLAG26\n",
      "FLAG27\n",
      "FLAG28\n",
      "FLAG29\n",
      "FLAG30\n",
      "FLAG31\n",
      "FLAG32\n",
      "FLAG33\n",
      "FLAG34\n",
      "FLAG35\n",
      "FLAG36\n",
      "FLAG37\n",
      "FLAG38\n",
      "FLAG39\n",
      "FLAG40\n",
      "FLAG41\n",
      "FLAG42\n",
      "FLAG43\n",
      "FLAG44\n",
      "FLAG45\n",
      "FLAG46\n",
      "FLAG47\n",
      "FLAG48\n",
      "FLAG49\n",
      "FLAG50\n",
      "FLAG51\n",
      "FLAG52\n",
      "FLAG53\n",
      "FLAG54\n",
      "FLAG55\n",
      "FLAG56\n",
      "FLAG57\n",
      "FLAG58\n",
      "FLAG59\n",
      "FLAG60\n",
      "FLAG61\n",
      "FLAG62\n",
      "FLAG63\n",
      "ID\n",
      "ORTH\n",
      "LOWER\n",
      "NORM\n",
      "SHAPE\n",
      "PREFIX\n",
      "SUFFIX\n",
      "LENGTH\n",
      "CLUSTER\n",
      "LEMMA\n",
      "POS\n",
      "TAG\n",
      "DEP\n",
      "ENT_IOB\n",
      "ENT_TYPE\n",
      "HEAD\n",
      "SENT_START\n",
      "SPACY\n",
      "PROB\n",
      "LANG\n",
      "ADJ\n",
      "ADP\n",
      "ADV\n",
      "AUX\n",
      "CONJ\n",
      "CCONJ\n",
      "DET\n",
      "INTJ\n",
      "NOUN\n",
      "NUM\n",
      "PART\n",
      "PRON\n",
      "PROPN\n",
      "PUNCT\n",
      "SCONJ\n",
      "SYM\n",
      "VERB\n",
      "X\n",
      "EOL\n",
      "SPACE\n",
      "Animacy_anim\n",
      "Animacy_inam\n",
      "Animacy_hum\n",
      "Animacy_nhum\n",
      "Aspect_freq\n",
      "Aspect_imp\n",
      "Aspect_mod\n",
      "Aspect_none\n",
      "Aspect_perf\n",
      "Aspect_iter\n",
      "Aspect_hab\n",
      "Case_abe\n",
      "Case_abl\n",
      "Case_abs\n",
      "Case_acc\n",
      "Case_ade\n",
      "Case_all\n",
      "Case_cau\n",
      "Case_com\n",
      "Case_cmp\n",
      "Case_dat\n",
      "Case_del\n",
      "Case_dis\n",
      "Case_ela\n",
      "Case_equ\n",
      "Case_ess\n",
      "Case_gen\n",
      "Case_ill\n",
      "Case_ine\n",
      "Case_ins\n",
      "Case_loc\n",
      "Case_lat\n",
      "Case_nom\n",
      "Case_par\n",
      "Case_sub\n",
      "Case_sup\n",
      "Case_tem\n",
      "Case_ter\n",
      "Case_tra\n",
      "Case_voc\n",
      "Definite_two\n",
      "Definite_def\n",
      "Definite_red\n",
      "Definite_cons\n",
      "Definite_ind\n",
      "Definite_spec\n",
      "Degree_cmp\n",
      "Degree_comp\n",
      "Degree_none\n",
      "Degree_pos\n",
      "Degree_sup\n",
      "Degree_abs\n",
      "Degree_com\n",
      "Degree_dim\n",
      "Degree_equ\n",
      "Evident_nfh\n",
      "Gender_com\n",
      "Gender_fem\n",
      "Gender_masc\n",
      "Gender_neut\n",
      "Mood_cnd\n",
      "Mood_imp\n",
      "Mood_ind\n",
      "Mood_n\n",
      "Mood_pot\n",
      "Mood_sub\n",
      "Mood_opt\n",
      "Mood_prp\n",
      "Mood_adm\n",
      "Negative_neg\n",
      "Negative_pos\n",
      "Negative_yes\n",
      "Polarity_neg\n",
      "Polarity_pos\n",
      "Number_com\n",
      "Number_dual\n",
      "Number_none\n",
      "Number_plur\n",
      "Number_sing\n",
      "Number_ptan\n",
      "Number_count\n",
      "Number_tri\n",
      "NumType_card\n",
      "NumType_dist\n",
      "NumType_frac\n",
      "NumType_gen\n",
      "NumType_mult\n",
      "NumType_none\n",
      "NumType_ord\n",
      "NumType_sets\n",
      "Person_one\n",
      "Person_two\n",
      "Person_three\n",
      "Person_none\n",
      "Poss_yes\n",
      "PronType_advPart\n",
      "PronType_art\n",
      "PronType_default\n",
      "PronType_dem\n",
      "PronType_ind\n",
      "PronType_int\n",
      "PronType_neg\n",
      "PronType_prs\n",
      "PronType_rcp\n",
      "PronType_rel\n",
      "PronType_tot\n",
      "PronType_clit\n",
      "PronType_exc\n",
      "PronType_emp\n",
      "Reflex_yes\n",
      "Tense_fut\n",
      "Tense_imp\n",
      "Tense_past\n",
      "Tense_pres\n",
      "VerbForm_fin\n",
      "VerbForm_ger\n",
      "VerbForm_inf\n",
      "VerbForm_none\n",
      "VerbForm_part\n",
      "VerbForm_partFut\n",
      "VerbForm_partPast\n",
      "VerbForm_partPres\n",
      "VerbForm_sup\n",
      "VerbForm_trans\n",
      "VerbForm_conv\n",
      "VerbForm_gdv\n",
      "VerbForm_vnoun\n",
      "Voice_act\n",
      "Voice_cau\n",
      "Voice_pass\n",
      "Voice_mid\n",
      "Voice_int\n",
      "Voice_antip\n",
      "Voice_dir\n",
      "Voice_inv\n",
      "Abbr_yes\n",
      "AdpType_prep\n",
      "AdpType_post\n",
      "AdpType_voc\n",
      "AdpType_comprep\n",
      "AdpType_circ\n",
      "AdvType_man\n",
      "AdvType_loc\n",
      "AdvType_tim\n",
      "AdvType_deg\n",
      "AdvType_cau\n",
      "AdvType_mod\n",
      "AdvType_sta\n",
      "AdvType_ex\n",
      "AdvType_adadj\n",
      "ConjType_oper\n",
      "ConjType_comp\n",
      "Connegative_yes\n",
      "Derivation_minen\n",
      "Derivation_sti\n",
      "Derivation_inen\n",
      "Derivation_lainen\n",
      "Derivation_ja\n",
      "Derivation_ton\n",
      "Derivation_vs\n",
      "Derivation_ttain\n",
      "Derivation_ttaa\n",
      "Echo_rdp\n",
      "Echo_ech\n",
      "Foreign_foreign\n",
      "Foreign_fscript\n",
      "Foreign_tscript\n",
      "Foreign_yes\n",
      "Gender_dat_masc\n",
      "Gender_dat_fem\n",
      "Gender_erg_masc\n",
      "Gender_erg_fem\n",
      "Gender_psor_masc\n",
      "Gender_psor_fem\n",
      "Gender_psor_neut\n",
      "Hyph_yes\n",
      "InfForm_one\n",
      "InfForm_two\n",
      "InfForm_three\n",
      "NameType_geo\n",
      "NameType_prs\n",
      "NameType_giv\n",
      "NameType_sur\n",
      "NameType_nat\n",
      "NameType_com\n",
      "NameType_pro\n",
      "NameType_oth\n",
      "NounType_com\n",
      "NounType_prop\n",
      "NounType_class\n",
      "Number_abs_sing\n",
      "Number_abs_plur\n",
      "Number_dat_sing\n",
      "Number_dat_plur\n",
      "Number_erg_sing\n",
      "Number_erg_plur\n",
      "Number_psee_sing\n",
      "Number_psee_plur\n",
      "Number_psor_sing\n",
      "Number_psor_plur\n",
      "Number_pauc\n",
      "Number_grpa\n",
      "Number_grpl\n",
      "Number_inv\n",
      "NumForm_digit\n",
      "NumForm_roman\n",
      "NumForm_word\n",
      "NumValue_one\n",
      "NumValue_two\n",
      "NumValue_three\n",
      "PartForm_pres\n",
      "PartForm_past\n",
      "PartForm_agt\n",
      "PartForm_neg\n",
      "PartType_mod\n",
      "PartType_emp\n",
      "PartType_res\n",
      "PartType_inf\n",
      "PartType_vbp\n",
      "Person_abs_one\n",
      "Person_abs_two\n",
      "Person_abs_three\n",
      "Person_dat_one\n",
      "Person_dat_two\n",
      "Person_dat_three\n",
      "Person_erg_one\n",
      "Person_erg_two\n",
      "Person_erg_three\n",
      "Person_psor_one\n",
      "Person_psor_two\n",
      "Person_psor_three\n",
      "Person_zero\n",
      "Person_four\n",
      "Polite_inf\n",
      "Polite_pol\n",
      "Polite_abs_inf\n",
      "Polite_abs_pol\n",
      "Polite_erg_inf\n",
      "Polite_erg_pol\n",
      "Polite_dat_inf\n",
      "Polite_dat_pol\n",
      "Polite_infm\n",
      "Polite_form\n",
      "Polite_form_elev\n",
      "Polite_form_humb\n",
      "Prefix_yes\n",
      "PrepCase_npr\n",
      "PrepCase_pre\n",
      "PunctSide_ini\n",
      "PunctSide_fin\n",
      "PunctType_peri\n",
      "PunctType_qest\n",
      "PunctType_excl\n",
      "PunctType_quot\n",
      "PunctType_brck\n",
      "PunctType_comm\n",
      "PunctType_colo\n",
      "PunctType_semi\n",
      "PunctType_dash\n",
      "Style_arch\n",
      "Style_rare\n",
      "Style_poet\n",
      "Style_norm\n",
      "Style_coll\n",
      "Style_vrnc\n",
      "Style_sing\n",
      "Style_expr\n",
      "Style_derg\n",
      "Style_vulg\n",
      "Style_yes\n",
      "StyleVariant_styleShort\n",
      "StyleVariant_styleBound\n",
      "VerbType_aux\n",
      "VerbType_cop\n",
      "VerbType_mod\n",
      "VerbType_light\n",
      "PERSON\n",
      "NORP\n",
      "FACILITY\n",
      "ORG\n",
      "GPE\n",
      "LOC\n",
      "PRODUCT\n",
      "EVENT\n",
      "WORK_OF_ART\n",
      "LANGUAGE\n",
      "LAW\n",
      "DATE\n",
      "TIME\n",
      "PERCENT\n",
      "MONEY\n",
      "QUANTITY\n",
      "ORDINAL\n",
      "CARDINAL\n",
      "acomp\n",
      "advcl\n",
      "advmod\n",
      "agent\n",
      "amod\n",
      "appos\n",
      "attr\n",
      "aux\n",
      "auxpass\n",
      "cc\n",
      "ccomp\n",
      "complm\n",
      "conj\n",
      "cop\n",
      "csubj\n",
      "csubjpass\n",
      "dep\n",
      "det\n",
      "dobj\n",
      "expl\n",
      "hmod\n",
      "hyph\n",
      "infmod\n",
      "intj\n",
      "iobj\n",
      "mark\n",
      "meta\n",
      "neg\n",
      "nmod\n",
      "nn\n",
      "npadvmod\n",
      "nsubj\n",
      "nsubjpass\n",
      "num\n",
      "number\n",
      "oprd\n",
      "obj\n",
      "obl\n",
      "parataxis\n",
      "partmod\n",
      "pcomp\n",
      "pobj\n",
      "poss\n",
      "possessive\n",
      "preconj\n",
      "prep\n",
      "prt\n",
      "punct\n",
      "quantmod\n",
      "relcl\n",
      "rcmod\n",
      "root\n",
      "xcomp\n",
      "acl\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '452'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-10-dbfe2b15eb5d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '452'.\""
     ]
    }
   ],
   "source": [
    "for i in range (0,len(nlp.vocab.strings)):\n",
    "    print(nlp.vocab.strings[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the hash for a string, we can look it up in nlp dot vocab dot strings.\n",
    "\n",
    "To get the string representation of a hash, we can look up the hash.\n",
    "\n",
    "A Doc object also exposes its vocab and strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hash value: 3197928453018144401\n",
      "string value: coffee\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I love coffee\")\n",
    "print('hash value:', nlp.vocab.strings['coffee'])\n",
    "print('string value:', nlp.vocab.strings[3197928453018144401])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hash value: 3197928453018144401\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I love coffee\")\n",
    "print('hash value:', doc.vocab.strings['coffee'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lexemes: entries in the vocabulary\n",
    "\n",
    "Lexemes are context-independent entries in the vocabulary.\n",
    "\n",
    "You can get a lexeme by looking up a string or a hash ID in the vocab.\n",
    "\n",
    "Lexemes expose attributes, just like tokens.\n",
    "\n",
    "They hold context-independent information about a word, like the text, or whether the the word consists of alphanumeric characters.\n",
    "\n",
    "Lexemes don't have part-of-speech tags, dependencies or entity labels. Those depend on the context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "coffee 3197928453018144401 True\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I love coffee\")\n",
    "lexeme = nlp.vocab['coffee']\n",
    "\n",
    "# Print the lexical attributes\n",
    "print(lexeme.text, lexeme.orth, lexeme.is_alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contains the context-independent information about a word.\n",
    "\n",
    "Word text: lexeme.text and lexeme.orth (the hash).\n",
    "    \n",
    "Lexical attributes like lexeme.is_alpha.\n",
    "\n",
    "Not context-dependent part-of-speech tags, dependencies or entity labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "img src=\"img/data-struct.png\" - removed the tags for compatibility reasons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above one example:\n",
    "\n",
    "The Doc contains words in context – in this case, the tokens \"I\", \"love\" and \"coffee\" with their part-of-speech tags and dependencies.\n",
    "\n",
    "Each token refers to a lexeme, which knows the word's hash ID. To get the string representation of the word, spaCy looks up the hash in the string store."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. String to hashes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1\n",
    "\n",
    "Look up the string “cat” in nlp.vocab.strings to get the hash.\n",
    "\n",
    "Look up the hash to get back the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5439657043933447811\n",
      "cat\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"I have a cat\")\n",
    "\n",
    "# Look up the hash for the word \"cat\"\n",
    "cat_hash = nlp.vocab.strings[\"cat\"]\n",
    "print(cat_hash)\n",
    "\n",
    "# Look up the cat_hash to get the string\n",
    "cat_string = nlp.vocab.strings[cat_hash]\n",
    "print(cat_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 \n",
    "\n",
    "Look up the string label “PERSON” in nlp.vocab.strings to get the hash.\n",
    "\n",
    "Look up the hash to get back the string.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "380\n",
      "PERSON\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"David Bowie is a PERSON\")\n",
    "\n",
    "# Look up the hash for the string label \"PERSON\"\n",
    "person_hash = nlp.vocab.strings[\"PERSON\"]\n",
    "print(person_hash)\n",
    "\n",
    "# Look up the person_hash to get the string\n",
    "person_string = nlp.vocab.strings[person_hash]\n",
    "print(person_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14800503047316267216\n",
      "person\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"David Bowie is a PERSON\")\n",
    "\n",
    "# Look up the hash for the string label \"PERSON\"\n",
    "person_hash = nlp.vocab.strings[\"person\"]\n",
    "print(person_hash)\n",
    "\n",
    "# Look up the person_hash to get the string\n",
    "person_string = nlp.vocab.strings[person_hash]\n",
    "print(person_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2313063860588076218\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '2313063860588076218'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-17-144c347826da>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;31m# Look up the person_hash to get the string\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m \u001b[0mperson_string\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mperson_hash\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mperson_string\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '2313063860588076218'.\""
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"David Bowie is a PERSON\")\n",
    "\n",
    "# Look up the hash for the string label \"PERSON\"\n",
    "person_hash = nlp.vocab.strings[\"Person\"]\n",
    "print(person_hash)\n",
    "\n",
    "# Look up the person_hash to get the string\n",
    "person_string = nlp.vocab.strings[person_hash]\n",
    "print(person_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9822269368127343413\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '9822269368127343413'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-18-8fa39320c534>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;31m# Look up the person_hash to get the string\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m \u001b[0mperson_string\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnlp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mperson_hash\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mperson_string\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '9822269368127343413'.\""
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"David Bowie is a PERSON\")\n",
    "\n",
    "# Look up the hash for the string label \"PERSON\"\n",
    "person_hash = nlp.vocab.strings[\"pErson\"]\n",
    "print(person_hash)\n",
    "\n",
    "# Look up the person_hash to get the string\n",
    "person_string = nlp.vocab.strings[person_hash]\n",
    "print(person_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Vocab, hashes and lexemes\n",
    "\n",
    "Why does this code throw an error?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2644858412616767388\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "\"[E018] Can't retrieve string for hash '2644858412616767388'.\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-19-8f4643000de0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[1;31m# Look up the ID for 'Bowie' in the vocab\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnlp_de\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrings\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mbowie_id\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mstrings.pyx\u001b[0m in \u001b[0;36mspacy.strings.StringStore.__getitem__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: \"[E018] Can't retrieve string for hash '2644858412616767388'.\""
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.lang.de import German\n",
    "\n",
    "# Create an English and German nlp object\n",
    "nlp = English()\n",
    "nlp_de = German()\n",
    "\n",
    "# Get the ID for the string 'Bowie'\n",
    "bowie_id = nlp.vocab.strings['Bowie']\n",
    "print(bowie_id)\n",
    "\n",
    "# Look up the ID for 'Bowie' in the vocab\n",
    "print(nlp_de.vocab.strings[bowie_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The string 'Bowie' isn't in the German vocab, so the hash can't be resolved in the string store.\n",
    "\n",
    "Hashes can’t be reversed. To prevent this problem, add the word to the new vocab by processing a text or looking up the string, or use the same vocab to resolve the hash back to a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Data Structures(2)\n",
    "\n",
    "Now that you know all about the vocabulary and string store, we can take a look at the most important data structure: the Doc, and its views Token and Span."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Doc object\n",
    "\n",
    "The Doc is one of the central data structures in spaCy. It's created automatically when you process a text with the nlp object. But you can also instantiate the class manually.\n",
    "\n",
    "After creating the nlp object, we can import the Doc class from spacy dot tokens.\n",
    "\n",
    "Here we're creating a Doc from three words. The spaces are a list of boolean values indicating whether the word is followed by a space. Every token includes that information – even the last one!\n",
    "\n",
    "The Doc class takes three arguments: the shared vocab, the words and the spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an nlp object\n",
    "from spacy.lang.en import English\n",
    "nlp = English()\n",
    "\n",
    "# Import the Doc class\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "# The words and spaces to create the doc from\n",
    "words = ['Hello', 'world', '!']\n",
    "spaces = [True, False, False]\n",
    "\n",
    "# Create a doc manually\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/span2.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Span is a slice of a Doc consisting of one or more tokens. The Span takes at least three arguments: the doc it refers to, and the start and end index of the span. Remember that the end index is exclusive!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a Span manually, we can also import the class from spacy dot tokens. We can then instantiate it with the doc and the span's start and end index.\n",
    "\n",
    "To add an entity label to the span, we first need to look up the string in the string store. We can then provide it to the span as the label argument.\n",
    "\n",
    "The doc dot ents are writable, so we can add entities manually by overwriting it with a list of spans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Doc and Span classes\n",
    "from spacy.tokens import Doc, Span\n",
    "\n",
    "# The words and spaces to create the doc from\n",
    "words = ['Hello', 'world', '!']\n",
    "spaces = [True, False, False]\n",
    "\n",
    "# Create a doc manually\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "\n",
    "# Create a span manually\n",
    "span = Span(doc, 0, 2)\n",
    "\n",
    "# Create a span with a label\n",
    "span_with_label = Span(doc, 0, 2, label=\"GREETING\")\n",
    "\n",
    "# Add span to the doc.ents\n",
    "doc.ents = [span_with_label]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few tips and tricks before we get started:\n",
    "\n",
    "The Doc and Span are very powerful and optimized for performance. They give you access to all references and relationships of the words and sentences.\n",
    "\n",
    "If your application needs to output strings, make sure to convert the doc as late as possible. If you do it too early, you'll lose all relationships between the tokens.\n",
    "\n",
    "To keep things consistent, try to use built-in token attributes wherever possible. For example, token dot i for the token index.\n",
    "\n",
    "Also, don't forget to always pass in the shared vocab!\n",
    "\n",
    "<b> Best practices: </b>\n",
    "\n",
    "* Doc and Span are very powerful and hold references and relationships of words and sentences\n",
    "\n",
    "* Convert result to strings as late as possible\n",
    "    \n",
    "* Use token attributes if available – for example, token.i for the token index\n",
    "\n",
    "* Don't forget to pass in the shared vocab\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Creating a Doc\n",
    "\n",
    "Let’s create some Doc objects from scratch!\n",
    "\n",
    "### Part 1\n",
    "\n",
    "Import the Doc from spacy.tokens.\n",
    "\n",
    "Create a Doc from the words and spaces. Don’t forget to pass in the vocab!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spaCy is cool!\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Import the Doc class\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "# Desired text: \"spaCy is cool!\"\n",
    "words = [\"spaCy\", \"is\", \"cool\", \"!\"]\n",
    "spaces = [True, True, False, False]\n",
    "\n",
    "# Create a Doc from the words and spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "print(doc.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2\n",
    "\n",
    "Import the Doc from spacy.tokens.\n",
    "\n",
    "Create a Doc from the words and spaces. Don’t forget to pass in the vocab!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go, get started!\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Import the Doc class\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "# Desired text: \"Go, get started!\"\n",
    "words = [\"Go\", \",\", \"get\", \"started\", \"!\"]\n",
    "spaces = [False, True, True, False, False]\n",
    "\n",
    "# Create a Doc from the words and spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "print(doc.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> DOC Docstring:     </b>\n",
    "    \n",
    "A sequence of Token objects. Access sentences and named entities, export\n",
    "annotations to numpy arrays, losslessly serialize to compressed binary\n",
    "strings. The `Doc` object holds an array of `TokenC` structs. The\n",
    "Python-level `Token` and `Span` objects are views of this array, i.e.\n",
    "they don't own the data themselves.\n",
    "\n",
    "EXAMPLE: Construction 1\n",
    "    >>> doc = nlp(u'Some text')\n",
    "\n",
    "    Construction 2\n",
    "    >>> from spacy.tokens import Doc\n",
    "    >>> doc = Doc(nlp.vocab, words=[u'hello', u'world', u'!'],\n",
    "                  spaces=[True, False, False])\n",
    "Init docstring:\n",
    "Create a Doc object.\n",
    "\n",
    "vocab (Vocab): A vocabulary object, which must match any models you\n",
    "    want to use (e.g. tokenizer, parser, entity recognizer).\n",
    "    \n",
    "words (list or None): A list of unicode strings to add to the document\n",
    "    as words. If `None`, defaults to empty list.\n",
    "    \n",
    "spaces (list or None): A list of boolean values, of the same length as\n",
    "    words. True means that the word is followed by a space, False means\n",
    "    it is not. If `None`, defaults to `[True]*len(words)`\n",
    "    \n",
    "user_data (dict or None): Optional extra data to attach to the Doc.\n",
    "\n",
    "RETURNS (Doc): The newly constructed object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3\n",
    "\n",
    "Import the Doc from spacy.tokens.\n",
    "\n",
    "Complete the words and spaces to match the desired text and create a doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Oh, really?!\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Import the Doc class\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "# Desired text: \"Oh, really?!\"\n",
    "words = [\"Oh\", \",\", \"really\", \"?\",\"!\"]\n",
    "spaces = [False, True, False, False, False]\n",
    "\n",
    "# Create a Doc from the words and spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "print(doc.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Docs, spans and entities from scratch\n",
    "\n",
    "In this exercise, you’ll create the Doc and Span objects manually, and update the named entities – just like spaCy does behind the scenes. \n",
    "\n",
    "A shared nlp object has already been created.\n",
    "\n",
    "- Import the Doc and Span classes from spacy.tokens.\n",
    "\n",
    "- Use the Doc class directly to create a doc from the words and spaces.\n",
    "\n",
    "- Create a Span for “David Bowie” from the doc and assign it the label \"PERSON\".\n",
    "\n",
    "- Overwrite the doc.ents with a list of one entity, the “David Bowie” span.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I like David Bowie\n",
      "David Bowie PERSON\n",
      "[('David Bowie', 'PERSON')]\n"
     ]
    }
   ],
   "source": [
    "# from spacy.lang.en import English\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Import the Doc and Span classes\n",
    "from spacy.tokens import Doc, Span\n",
    "\n",
    "words = [\"I\", \"like\", \"David\", \"Bowie\"]\n",
    "spaces = [True, True, True, False]\n",
    "\n",
    "# Create a doc from the words and spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "print(doc.text)\n",
    "\n",
    "# Create a span for \"David Bowie\" from the doc and assign it the label \"PERSON\"\n",
    "span = Span(doc, 2, 4, label=\"PERSON\")\n",
    "print(span.text, span.label_)\n",
    "\n",
    "# Add the span to the doc's entities\n",
    "doc.ents = [span]\n",
    "\n",
    "# Print entities' text and labels\n",
    "print([(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eu amo o David Bowie\n",
      "David Bowie PERSON\n",
      "[('David Bowie', 'PERSON')]\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.pt import Portuguese\n",
    "\n",
    "nlp = Portuguese()\n",
    "\n",
    "# Import the Doc and Span classes\n",
    "from spacy.tokens import Doc, Span\n",
    "\n",
    "words = [\"Eu\", \"amo\", \"o\", \"David\", \"Bowie\"]\n",
    "spaces = [True, True, True, True, False]\n",
    "\n",
    "# Create a doc from the words and spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "print(doc.text)\n",
    "\n",
    "# Create a span for \"David Bowie\" from the doc and assign it the label \"PERSON\"\n",
    "span = Span(doc, 3, 5, label=\"PERSON\")\n",
    "print(span.text, span.label_)\n",
    "\n",
    "# Add the span to the doc's entities\n",
    "doc.ents = [span]\n",
    "\n",
    "# Print entities' text and labels\n",
    "print([(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Data structures and best practices\n",
    "\n",
    "The code in this example is trying to analyze a text and collect all proper nouns that are followed by a verb.\n",
    "\n",
    "### Part 1\n",
    "\n",
    "Why is the code bad?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found proper noun before a verb: Berlin\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Berlin is a nice city\")\n",
    "\n",
    "# Get all tokens and part-of-speech tags\n",
    "token_texts = [token.text for token in doc]\n",
    "pos_tags = [token.pos_ for token in doc]\n",
    "\n",
    "for index, pos in enumerate(pos_tags):\n",
    "    # Check if the current token is a proper noun\n",
    "    if pos == \"PROPN\":\n",
    "        # Check if the next token is a verb\n",
    "        if pos_tags[index + 1] == \"VERB\":\n",
    "            result = token_texts[index]\n",
    "            print(\"Found proper noun before a verb:\", result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Berlin', 'is', 'a', 'nice', 'city']\n",
      "['PROPN', 'VERB', 'DET', 'ADJ', 'NOUN']\n"
     ]
    }
   ],
   "source": [
    "print(token_texts)\n",
    "print(pos_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "proper noun\n",
      "verb\n",
      "determiner\n",
      "adjective\n",
      "noun\n"
     ]
    }
   ],
   "source": [
    "print(spacy.explain('PROPN'))\n",
    "print(spacy.explain('VERB'))\n",
    "print(spacy.explain('DET'))\n",
    "print(spacy.explain('ADJ'))\n",
    "print(spacy.explain('NOUN'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: \n",
    "\n",
    "It only uses lists of strings instead of native token attributes. This is often less efficient, and can't express complex relationships.\n",
    "\n",
    "Always convert the results to strings as late as possible, and try to use native token attributes to keep things consistent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2\n",
    "Rewrite the code to use the native token attributes instead of lists of token_texts and pos_tags.\n",
    "Loop over each token in the doc and check the token.pos_ attribute.\n",
    "Use doc[token.i + 1] to check for the next token and its .pos_ attribute.\n",
    "If a proper noun before a verb is found, print its token.text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2\n",
    "\n",
    "\n",
    "- Rewrite the code to use the native token attributes instead of lists of token_texts and pos_tags.\n",
    "- Loop over each token in the doc and check the token.pos_ attribute.\n",
    "- Use doc[token.i + 1] to check for the next token and its .pos_ attribute.\n",
    "- If a proper noun before a verb is found, print its token.text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found proper noun before a verb: Berlin\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Berlin is a nice city\")\n",
    "\n",
    "# Iterate over the tokens\n",
    "for token in doc:\n",
    "    # Check if the current token is a proper noun\n",
    "    if token.pos_ == \"PROPN\":\n",
    "        # Check if the next token is a verb\n",
    "        if doc[token.i + 1].pos_ == \"VERB\":\n",
    "            print(\"Found proper noun before a verb:\", token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Word vectors and semantic similarities\n",
    "\n",
    "In this lesson, you'll learn how to use spaCy to predict how similar documents, spans or tokens are to each other.\n",
    "\n",
    "You'll also learn about how to use word vectors and how to take advantage of them in your NLP application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy can compare two objects and predict how similar they are – for example, documents, spans or single tokens.\n",
    "\n",
    "The Doc, Token and Span objects have a dot similarity method that takes another object and returns a floating point number between 0 and 1, indicating how similar they are.\n",
    "\n",
    "One thing that's very important: In order to use similarity, you need a larger spaCy model that has word vectors included.\n",
    "\n",
    "<b>For example, the medium or large English model – but not the small one. So if you want to use vectors, always go with a model that ends in \"md\" or \"lg\". You can find more details on this in the models documentation.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- spaCy can compare two objects and predict similarity\n",
    "- Doc.similarity(), Span.similarity() and Token.similarity()\n",
    "- Take another object and return a similarity score (0 to 1)\n",
    "- Important: needs a model that has word vectors included, for example:\n",
    "    - ✅ en_core_web_md (medium model)\n",
    "    - ✅ en_core_web_lg (large model)\n",
    "    - 🚫 NOT en_core_web_sm (small model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example. Let's say we want to find out whether two documents are similar.\n",
    "\n",
    "First, we load the medium English model, \"en_core_web_md\".\n",
    "\n",
    "We can then create two doc objects and use the first doc's similarity method to compare it to the second.\n",
    "\n",
    "Here, a fairly high similarity score of 0.86 is predicted for \"I like fast food\" and \"I like pizza\".\n",
    "\n",
    "The same works for tokens.\n",
    "\n",
    "According to the word vectors, the tokens \"pizza\" and \"pasta\" are kind of similar, and receive a score of 0.7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8627204117787385\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Load a larger model with vectors\n",
    "nlp = spacy.load('en_core_web_md')\n",
    "\n",
    "# Compare two documents\n",
    "doc1 = nlp(\"I like fast food\")\n",
    "doc2 = nlp(\"I like pizza\")\n",
    "print(doc1.similarity(doc2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.73695457\n"
     ]
    }
   ],
   "source": [
    "# Compare two tokens\n",
    "doc = nlp(\"I like pizza and pasta\")\n",
    "token1 = doc[2]\n",
    "token2 = doc[4]\n",
    "print(token1.similarity(token2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the similarity methods to compare different types of objects.\n",
    "\n",
    "For example, a document and a token.\n",
    "\n",
    "Here, the similarity score is pretty low and the two objects are considered fairly dissimilar.\n",
    "\n",
    "Here's another example comparing a span – \"pizza and pasta\" – to a document about McDonalds.\n",
    "\n",
    "The score returned here is 0.61, so it's determined to be kind of similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.32531983166759537\n"
     ]
    }
   ],
   "source": [
    "# Compare a document with a token\n",
    "doc = nlp(\"I like pizza\")\n",
    "token = nlp(\"soap\")[0]\n",
    "\n",
    "print(doc.similarity(token))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6199091710787739\n"
     ]
    }
   ],
   "source": [
    "# Compare a span with a document\n",
    "span = nlp(\"I like pizza and pasta\")[2:5]\n",
    "doc = nlp(\"McDonalds sells burgers\")\n",
    "\n",
    "print(span.similarity(doc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But how does spaCy do this under the hood?\n",
    "\n",
    "Similarity is determined using word vectors, multi-dimensional representations of meanings of words.\n",
    "\n",
    "You might have heard of Word2Vec, which is an algorithm that's often used to train word vectors from raw text.\n",
    "\n",
    "Vectors can be added to spaCy's statistical models.\n",
    "\n",
    "By default, the similarity returned by spaCy is the cosine similarity between two vectors – but this can be adjusted if necessary.\n",
    "\n",
    "Vectors for objects consisting of several tokens, like the Doc and Span, default to the average of their token vectors.\n",
    "\n",
    "That's also why you usually get more value out of shorter phrases with fewer irrelevant words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does spaCy predict similarity?\n",
    "- Similarity is determined using word vectors\n",
    "- Multi-dimensional meaning representations of words\n",
    "- Generated using an algorithm like Word2Vec and lots of text\n",
    "- Can be added to spaCy's statistical models\n",
    "- Default: cosine similarity, but can be adjusted\n",
    "- Doc and Span vectors default to average of token vectors\n",
    "- Short phrases are better than long documents with many irrelevant words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give you an idea of what those vectors look like, here's an example.\n",
    "\n",
    "First, we load the medium model again, which ships with word vectors.\n",
    "\n",
    "Next, we can process a text and look up a token's vector using the dot vector attribute.\n",
    "\n",
    "The result is a 300-dimensional vector of the word \"banana\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2.0228e-01 -7.6618e-02  3.7032e-01  3.2845e-02 -4.1957e-01  7.2069e-02\n",
      " -3.7476e-01  5.7460e-02 -1.2401e-02  5.2949e-01 -5.2380e-01 -1.9771e-01\n",
      " -3.4147e-01  5.3317e-01 -2.5331e-02  1.7380e-01  1.6772e-01  8.3984e-01\n",
      "  5.5107e-02  1.0547e-01  3.7872e-01  2.4275e-01  1.4745e-02  5.5951e-01\n",
      "  1.2521e-01 -6.7596e-01  3.5842e-01 -4.0028e-02  9.5949e-02 -5.0690e-01\n",
      " -8.5318e-02  1.7980e-01  3.3867e-01  1.3230e-01  3.1021e-01  2.1878e-01\n",
      "  1.6853e-01  1.9874e-01 -5.7385e-01 -1.0649e-01  2.6669e-01  1.2838e-01\n",
      " -1.2803e-01 -1.3284e-01  1.2657e-01  8.6723e-01  9.6721e-02  4.8306e-01\n",
      "  2.1271e-01 -5.4990e-02 -8.2425e-02  2.2408e-01  2.3975e-01 -6.2260e-02\n",
      "  6.2194e-01 -5.9900e-01  4.3201e-01  2.8143e-01  3.3842e-02 -4.8815e-01\n",
      " -2.1359e-01  2.7401e-01  2.4095e-01  4.5950e-01 -1.8605e-01 -1.0497e+00\n",
      " -9.7305e-02 -1.8908e-01 -7.0929e-01  4.0195e-01 -1.8768e-01  5.1687e-01\n",
      "  1.2520e-01  8.4150e-01  1.2097e-01  8.8239e-02 -2.9196e-02  1.2151e-03\n",
      "  5.6825e-02 -2.7421e-01  2.5564e-01  6.9793e-02 -2.2258e-01 -3.6006e-01\n",
      " -2.2402e-01 -5.3699e-02  1.2022e+00  5.4535e-01 -5.7998e-01  1.0905e-01\n",
      "  4.2167e-01  2.0662e-01  1.2936e-01 -4.1457e-02 -6.6777e-01  4.0467e-01\n",
      " -1.5218e-02 -2.7640e-01 -1.5611e-01 -7.9198e-02  4.0037e-02 -1.2944e-01\n",
      " -2.4090e-04 -2.6785e-01 -3.8115e-01 -9.7245e-01  3.1726e-01 -4.3951e-01\n",
      "  4.1934e-01  1.8353e-01 -1.5260e-01 -1.0808e-01 -1.0358e+00  7.6217e-02\n",
      "  1.6519e-01  2.6526e-04  1.6616e-01 -1.5281e-01  1.8123e-01  7.0274e-01\n",
      "  5.7956e-03  5.1664e-02 -5.9745e-02 -2.7551e-01 -3.9049e-01  6.1132e-02\n",
      "  5.5430e-01 -8.7997e-02 -4.1681e-01  3.2826e-01 -5.2549e-01 -4.4288e-01\n",
      "  8.2183e-03  2.4486e-01 -2.2982e-01 -3.4981e-01  2.6894e-01  3.9166e-01\n",
      " -4.1904e-01  1.6191e-01 -2.6263e+00  6.4134e-01  3.9743e-01 -1.2868e-01\n",
      " -3.1946e-01 -2.5633e-01 -1.2220e-01  3.2275e-01 -7.9933e-02 -1.5348e-01\n",
      "  3.1505e-01  3.0591e-01  2.6012e-01  1.8553e-01 -2.4043e-01  4.2886e-02\n",
      "  4.0622e-01 -2.4256e-01  6.3870e-01  6.9983e-01 -1.4043e-01  2.5209e-01\n",
      "  4.8984e-01 -6.1067e-02 -3.6766e-01 -5.5089e-01 -3.8265e-01 -2.0843e-01\n",
      "  2.2832e-01  5.1218e-01  2.7868e-01  4.7652e-01  4.7951e-02 -3.4008e-01\n",
      " -3.2873e-01 -4.1967e-01 -7.5499e-02 -3.8954e-01 -2.9622e-02 -3.4070e-01\n",
      "  2.2170e-01 -6.2856e-02 -5.1903e-01 -3.7774e-01 -4.3477e-03 -5.8301e-01\n",
      " -8.7546e-02 -2.3929e-01 -2.4711e-01 -2.5887e-01 -2.9894e-01  1.3715e-01\n",
      "  2.9892e-02  3.6544e-02 -4.9665e-01 -1.8160e-01  5.2939e-01  2.1992e-01\n",
      " -4.4514e-01  3.7798e-01 -5.7062e-01 -4.6946e-02  8.1806e-02  1.9279e-02\n",
      "  3.3246e-01 -1.4620e-01  1.7156e-01  3.9981e-01  3.6217e-01  1.2816e-01\n",
      "  3.1644e-01  3.7569e-01 -7.4690e-02 -4.8480e-02 -3.1401e-01 -1.9286e-01\n",
      " -3.1294e-01 -1.7553e-02 -1.7514e-01 -2.7587e-02 -1.0000e+00  1.8387e-01\n",
      "  8.1434e-01 -1.8913e-01  5.0999e-01 -9.1960e-03 -1.9295e-03  2.8189e-01\n",
      "  2.7247e-02  4.3409e-01 -5.4967e-01 -9.7426e-02 -2.4540e-01 -1.7203e-01\n",
      " -8.8650e-02 -3.0298e-01 -1.3591e-01 -2.7765e-01  3.1286e-03  2.0556e-01\n",
      " -1.5772e-01 -5.2308e-01 -6.4701e-01 -3.7014e-01  6.9393e-02  1.1401e-01\n",
      "  2.7594e-01 -1.3875e-01 -2.7268e-01  6.6891e-01 -5.6454e-02  2.4017e-01\n",
      " -2.6730e-01  2.9860e-01  1.0083e-01  5.5592e-01  3.2849e-01  7.6858e-02\n",
      "  1.5528e-01  2.5636e-01 -1.0772e-01 -1.2359e-01  1.1827e-01 -9.9029e-02\n",
      " -3.4328e-01  1.1502e-01 -3.7808e-01 -3.9012e-02 -3.4593e-01 -1.9404e-01\n",
      " -3.3580e-01 -6.2334e-02  2.8919e-01  2.8032e-01 -5.3741e-01  6.2794e-01\n",
      "  5.6955e-02  6.2147e-01 -2.5282e-01  4.1670e-01 -1.0108e-02 -2.5434e-01\n",
      "  4.0003e-01  4.2432e-01  2.2672e-01  1.7553e-01  2.3049e-01  2.8323e-01\n",
      "  1.3882e-01  3.1218e-03  1.7057e-01  3.6685e-01  2.5247e-03 -6.4009e-01\n",
      " -2.9765e-01  7.8943e-01  3.3168e-01 -1.1966e+00 -4.7156e-02  5.3175e-01]\n"
     ]
    }
   ],
   "source": [
    "# Load a larger model with vectors\n",
    "# nlp = spacy.load('en_core_web_md')\n",
    "\n",
    "doc = nlp(\"I have a banana\")\n",
    "# Access the vector via the token.vector attribute\n",
    "print(doc[3].vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predicting similarity can be useful for many types of applications. For example, to recommend a user similar texts based on the ones they have read. It can also be helpful to flag duplicate content, like posts on an online platform.\n",
    "\n",
    "However, it's important to keep in mind that there's no objective definition of what's similar and what isn't. It always depends on the context and what your application needs to do.\n",
    "\n",
    "Here's an example: spaCy's default word vectors assign a very high similarity score to \"I like cats\" and \"I hate cats\". This makes sense, because both texts express sentiment about cats. But in a different application context, you might want to consider the phrases as very dissimilar, because they talk about opposite sentiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9501446702124066\n"
     ]
    }
   ],
   "source": [
    "doc1 = nlp(\"I like cats\")\n",
    "doc2 = nlp(\"I hate cats\")\n",
    "\n",
    "print(doc1.similarity(doc2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Useful for many applications: recommendation systems, flagging duplicates etc.\n",
    "- There's no objective definition of \"similarity\"\n",
    "- Depends on the context and what application needs to do"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Inspecting word vectors\n",
    "\n",
    "In this exercise, you’ll use a larger English model, which includes around 20.000 word vectors. Because vectors take a little longer to load, we’re using a slightly compressed version of it than the one you can download with spaCy. The model is already pre-installed.\n",
    "\n",
    "- Load the medium 'en_core_web_md' model with word vectors.\n",
    "- Print the vector for \"bananas\" using the token.vector attribute.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-2.2009e-01 -3.0322e-02 -7.9859e-02 -4.6279e-01 -3.8600e-01  3.6962e-01\n",
      " -7.7178e-01 -1.1529e-01  3.3601e-02  5.6573e-01 -2.4001e-01  4.1833e-01\n",
      "  1.5049e-01  3.5621e-01 -2.1508e-01 -4.2743e-01  8.1400e-02  3.3916e-01\n",
      "  2.1637e-01  1.4792e-01  4.5811e-01  2.0966e-01 -3.5706e-01  2.3800e-01\n",
      "  2.7971e-02 -8.4538e-01  4.1917e-01 -3.9181e-01  4.0434e-04 -1.0662e+00\n",
      "  1.4591e-01  1.4643e-03  5.1277e-01  2.6072e-01  8.3785e-02  3.0340e-01\n",
      "  1.8579e-01  5.9999e-02 -4.0270e-01  5.0888e-01 -1.1358e-01 -2.8854e-01\n",
      " -2.7068e-01  1.1017e-02 -2.2217e-01  6.9076e-01  3.6459e-02  3.0394e-01\n",
      "  5.6989e-02  2.2733e-01 -9.9473e-02  1.5165e-01  1.3540e-01 -2.4965e-01\n",
      "  9.8078e-01 -8.0492e-01  1.9326e-01  3.1128e-01  5.5390e-02 -4.2423e-01\n",
      " -1.4082e-02  1.2708e-01  1.8868e-01  5.9777e-02 -2.2215e-01 -8.3950e-01\n",
      "  9.1987e-02  1.0180e-01 -3.1299e-01  5.5083e-01 -3.0717e-01  4.4201e-01\n",
      "  1.2666e-01  3.7643e-01  3.2333e-01  9.5673e-02  2.5083e-01 -6.4049e-02\n",
      "  4.2143e-01 -1.9375e-01  3.8026e-01  7.0883e-03 -2.0371e-01  1.5402e-01\n",
      " -3.7877e-03 -2.9396e-01  9.6518e-01  2.0068e-01 -5.6572e-01 -2.2581e-01\n",
      "  3.2251e-01 -3.4634e-01  2.7064e-01 -2.0687e-01 -4.7229e-01  3.1704e-01\n",
      " -3.4665e-01 -2.5188e-01 -1.1201e-01 -3.3937e-01  3.1518e-01 -3.2221e-01\n",
      " -2.4530e-01 -7.1571e-02 -4.3971e-01 -1.2070e+00  3.3365e-01 -5.8208e-02\n",
      "  8.0899e-01  4.2335e-01  3.8678e-01 -6.0797e-01 -7.3760e-01 -2.0547e-01\n",
      " -1.7499e-01 -3.7842e-03  2.1930e-01 -5.2486e-02  3.4869e-01  4.3852e-01\n",
      " -3.4471e-01  2.8910e-01  7.2554e-02 -4.8625e-01 -3.8390e-01 -4.4760e-01\n",
      "  4.3278e-01 -2.7128e-03 -9.0067e-01 -3.0819e-02 -3.8630e-01 -8.0798e-02\n",
      " -1.6243e-01  2.8830e-01 -2.6349e-01  1.7628e-01  3.5958e-01  5.7672e-01\n",
      " -5.4624e-01  3.8555e-02 -2.0182e+00  3.2916e-01  3.4672e-01  1.5398e-01\n",
      " -4.3446e-01 -4.1428e-02 -6.9588e-02  5.1513e-01 -1.3489e-01 -5.7239e-02\n",
      "  4.9241e-01  1.8643e-01  3.8596e-01 -3.7329e-02 -5.4216e-01 -1.8152e-01\n",
      "  4.3110e-01 -4.6967e-01  6.6801e-02  5.0323e-01 -2.4059e-01  3.6742e-01\n",
      "  2.9300e-01 -8.7883e-02 -4.7940e-01 -4.3431e-02 -2.6137e-01 -6.2658e-01\n",
      "  1.1446e-01  2.7682e-01  3.4800e-01  5.0018e-01  1.4269e-01 -3.3545e-01\n",
      " -3.9712e-01 -3.3121e-01 -3.4434e-01 -4.1627e-01 -3.5707e-03 -6.2350e-01\n",
      "  3.7794e-01 -1.6765e-01 -4.1954e-01 -3.3134e-01  3.1232e-01 -3.9494e-01\n",
      " -4.6921e-03 -4.8884e-01 -2.2059e-02 -2.6174e-01  1.7937e-01  3.6628e-01\n",
      "  5.8971e-02 -3.5991e-01 -4.4393e-01 -1.1890e-01  3.3487e-01  3.6505e-02\n",
      " -3.2788e-01  3.3425e-01 -5.6361e-01 -1.1190e-01  5.3770e-01  2.0311e-01\n",
      "  1.5110e-01  1.0623e-02  3.3401e-01  4.6084e-01  5.6293e-01 -7.5432e-02\n",
      "  5.4813e-01  1.9395e-01 -2.6265e-01 -3.1699e-01 -8.1778e-01  5.8169e-02\n",
      " -5.7866e-02 -1.1781e-01 -5.8742e-02 -1.4092e-01 -9.9394e-01 -9.4532e-02\n",
      "  2.3503e-01 -4.9027e-01  8.5832e-01  1.1540e-01 -1.5049e-01  1.9065e-01\n",
      " -2.6705e-01  2.5326e-01 -6.7579e-01 -1.0633e-02 -5.5158e-02 -3.1004e-01\n",
      " -5.8036e-02 -1.7200e-01  1.3298e-01 -3.2899e-01 -7.5481e-02  2.9425e-02\n",
      " -3.2949e-01 -1.8691e-01 -9.5323e-01 -3.5468e-01 -3.3162e-01  5.6441e-02\n",
      "  2.1790e-02  1.7182e-01 -4.4267e-01  6.9765e-01 -2.6876e-01  1.1659e-01\n",
      " -1.6584e-01  3.8296e-01  2.9109e-01  3.6318e-01  3.6961e-01  1.6305e-01\n",
      "  1.8152e-01  2.2453e-01  3.9866e-02 -3.7607e-02 -3.6089e-01  7.0818e-02\n",
      " -2.1509e-01  3.6551e-01 -5.1603e-01 -5.8102e-03 -4.8320e-01 -2.5068e-01\n",
      " -5.2062e-02 -2.0828e-01  2.9060e-01  2.2084e-02 -6.8123e-01  4.2063e-01\n",
      "  9.5973e-02  8.1720e-01 -1.5241e-01  6.2994e-01  2.6449e-01 -1.3516e-01\n",
      "  3.2450e-01  3.0503e-01  1.2357e-01  1.5107e-01  2.8327e-01 -3.3838e-01\n",
      "  4.6106e-02 -1.2361e-01  1.4516e-01 -2.7947e-02  2.6231e-02 -5.9591e-01\n",
      " -4.4183e-01  7.8440e-01 -3.4375e-02 -1.3928e+00  3.5248e-01  6.5220e-01]\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Load the en_core_web_md model\n",
    "# nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "# Process a text\n",
    "doc = nlp(\"Two bananas in pyjamas\")\n",
    "\n",
    "# Get the vector for the token \"bananas\"\n",
    "bananas_vector = doc[1].vector\n",
    "print(bananas_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Comparing similarities\n",
    "\n",
    "In this exercise, you’ll be using spaCy’s similarity methods to compare Doc, Token and Span objects and get similarity scores.\n",
    "\n",
    "Part 1\n",
    "\n",
    "Use the doc.similarity method to compare doc1 to doc2 and print the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8789265574516525\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc1 = nlp(\"It's a warm summer day\")\n",
    "doc2 = nlp(\"It's sunny outside\")\n",
    "\n",
    "# Get the similarity of doc1 and doc2\n",
    "similarity = doc1.similarity(doc2)\n",
    "print(similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2\n",
    "\n",
    "Use the token.similarity method to compare token1 to token2 and print the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22325331\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc = nlp(\"TV and books\")\n",
    "token1, token2 = doc[0], doc[2]\n",
    "\n",
    "# Get the similarity of the tokens \"TV\" and \"books\"\n",
    "similarity = token1.similarity(token2)\n",
    "print(similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 3\n",
    "\n",
    "Create spans for “great restaurant”/“really nice bar”.\n",
    "\n",
    "Use span.similarity to compare them and print the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7517392\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc = nlp(\"This was a great restaurant. Afterwards, we went to a really nice bar.\")\n",
    "\n",
    "# Create spans for \"great restaurant\" and \"really nice bar\"\n",
    "span1 = doc[3:5]\n",
    "span2 = doc[12:15]\n",
    "\n",
    "# Get the similarity of the spans\n",
    "similarity = span1.similarity(span2)\n",
    "print(similarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.87835974\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I like a hot cup of coffe but I hate a cold bread with butter.\")\n",
    "\n",
    "span1 = doc[0:7]\n",
    "span2 = doc[8:15]\n",
    "\n",
    "# Get the similarity of the spans\n",
    "similarity = span1.similarity(span2)\n",
    "print(similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Combining models and rules\n",
    "\n",
    "Combining statistical models with rule-based systems is one of the most powerful tricks you should have in your NLP toolbox.\n",
    "\n",
    "In this lesson, we'll take a look at how to do it with spaCy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical predictions vs. rules \n",
    "\n",
    "Statistical models are useful if your application needs to be able to generalize based on a few examples.\n",
    "\n",
    "For instance, detecting product or person names usually benefits from a statistical model. Instead of providing a list of all person names ever, your application will be able to predict whether a span of tokens is a person name. Similarly, you can predict dependency labels to find subject/object relationships.\n",
    "\n",
    "To do this, you would use spaCy's entity recognizer, dependency parser or part-of-speech tagger.\n",
    "\n",
    "Rule-based approaches on the other hand come in handy if there's a more or less finite number of instances you want to find. For example, all countries or cities of the world, drug names or even dog breeds.\n",
    "\n",
    "In spaCy, you can achieve this with custom tokenization rules, as well as the matcher and phrase matcher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "img src=\"img/vs.png\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RECAP: Rule-based matching\n",
    "\n",
    "In the last chapter, you learned how to use spaCy's rule-based matcher to find complex patterns in your texts. Here's a quick recap.\n",
    "\n",
    "The matcher is initialized with the shared vocabulary – usually nlp dot vocab.\n",
    "\n",
    "Patterns are lists of dictionaries, and each dictionary describes one token and its attributes. Patterns can be added to the matcher using the matcher dot add method.\n",
    "\n",
    "Operators let you specify how often to match a token. For example, \"+\" will match one or more times.\n",
    "\n",
    "Calling the matcher on a doc object will return a list of the matches. Each match is a tuple consisting of an ID, and the start and end token index in the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize with the shared vocab\n",
    "from spacy.matcher import Matcher\n",
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "# Patterns are lists of dictionaries describing the tokens\n",
    "pattern = [{'LEMMA': 'love', 'POS': 'VERB'}, {'LOWER': 'cats'}]\n",
    "matcher.add('LOVE_CATS', None, pattern)\n",
    "\n",
    "# Operators can specify how often a token should be matched\n",
    "pattern = [{'TEXT': 'very', 'OP': '+'}, {'TEXT': 'happy'}]\n",
    "\n",
    "# Calling matcher on doc returns list of (match_id, start, end) tuples\n",
    "doc = nlp(\"I love cats and I'm very very happy\")\n",
    "matches = matcher(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(9137535031263442622, 1, 3)]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RETURNS (list): A list of `(key, start, end)` tuples,\n",
    "    describing the matches. A match tuple describes a span\n",
    "    `doc[start:end]`. The `label_id` and `key` are both integers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding statistical predictions \n",
    "\n",
    "Here's an example of a matcher rule for \"golden retriever\".\n",
    "\n",
    "If we iterate over the matches returned by the matcher, we can get the match ID and the start and end index of the matched span. We can then find out more about it. Span objects give us access to the original document and all other token attributes and linguistic features predicted by the model.\n",
    "\n",
    "For example, we can get the span's root token. If the span consists of more than one token, this will be the token that decides the category of the phrase. For example, the root of \"Golden Retriever\" is \"Retriever\". We can also find the head token of the root. This is the syntactic \"parent\" that governs the phrase – in this case, the verb \"have\".\n",
    "\n",
    "Finally, we can look at the previous token and its attributes. In this case, it's a determiner, the article \"a\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched span: Golden Retriever\n",
      "Root token: Retriever\n",
      "Root head token: have\n",
      "Previous token: a DET\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add('DOG', None, [{'LOWER': 'golden'}, {'LOWER': 'retriever'}])\n",
    "doc = nlp(\"I have a Golden Retriever\")\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    span = doc[start:end]\n",
    "    print('Matched span:', span.text)\n",
    "    # Get the span's root token and root head token\n",
    "    print('Root token:', span.root.text)\n",
    "    print('Root head token:', span.root.head.text)\n",
    "    # Get the previous token and its POS tag\n",
    "    print('Previous token:', doc[start - 1].text, doc[start - 1].pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Efficient phrase matching (1) \n",
    "\n",
    "The phrase matcher is another helpful tool to find sequences of words in your data.\n",
    "\n",
    "It performs a keyword search on the document, but instead of only finding strings, it gives you direct access to the tokens in context.\n",
    "\n",
    "It takes Doc objects as patterns.\n",
    "\n",
    "It's also really fast.\n",
    "\n",
    "This makes it very useful for matching large dictionaries and word lists on large volumes of text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- PhraseMatcher like regular expressions or keyword search – but with access to the tokens!\n",
    "- Takes Doc object as patterns\n",
    "- More efficient and faster than the Matcher\n",
    "- Great for matching large word lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example.\n",
    "\n",
    "The phrase matcher can be imported from spacy dot matcher and follows the same API as the regular matcher.\n",
    "\n",
    "Instead of a list of dictionaries, we pass in a Doc object as the pattern.\n",
    "\n",
    "We can then iterate over the matches in the text, which gives us the match ID, and the start and end of the match. This lets us create a Span object for the matched tokens \"Golden Retriever\" to analyze it in context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched span: Golden Retriever\n"
     ]
    }
   ],
   "source": [
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "\n",
    "pattern = nlp(\"Golden Retriever\")\n",
    "matcher.add('DOG', None, pattern)\n",
    "doc = nlp(\"I have a Golden Retriever\")\n",
    "\n",
    "# Iterate over the matches\n",
    "for match_id, start, end in matcher(doc):\n",
    "    # Get the matched span\n",
    "    span = doc[start:end]\n",
    "    print('Matched span:', span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Debugging patterns (1)\n",
    "\n",
    "Why does this pattern not match the tokens “Silicon Valley” in the doc?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "matcher.add('Sil', None, [{'LOWER': 'silicon'}, {'TEXT': ' '}, {'LOWER': 'valley'}])\n",
    "\n",
    "doc = nlp(\"Can Silicon Valley workers rein in big tech from within?\")\n",
    "\n",
    "# Iterate over the matches\n",
    "for match_id, start, end in matcher(doc):\n",
    "    # Get the matched span\n",
    "    span = doc[start:end]\n",
    "    print('Matched span:', span.text)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer doesn't create tokens for single spaces, so there's no token with the value ' ' in between. The tokenizer already takes care of splitting off whitespace and each dictionary in the pattern describes one token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Debugging patterns (2)\n",
    "\n",
    "Both patterns in this exercise contain mistakes and won’t match as expected. Can you fix them? If you get stuck, try printing the tokens in the doc to see how the text will be split and adjust the pattern so that each dictionary represents one token.\n",
    "\n",
    "- Edit pattern1 so that it correctly matches all case-insensitive mentions of \"Amazon\" plus a title-cased proper noun.\n",
    "- Edit pattern2 so that it correctly matches all case-insensitive mentions of \"ad-free\", plus the following noun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy.matcher import Matcher\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\n",
    "    \"Twitch Prime, the perks program for Amazon Prime members offering free \"\n",
    "    \"loot, games and other benefits, is ditching one of its best features: \"\n",
    "    \"ad-free viewing. According to an email sent out to Amazon Prime members \"\n",
    "    \"today, ad-free viewing will no longer be included as a part of Twitch \"\n",
    "    \"Prime for new members, beginning on September 14. However, members with \"\n",
    "    \"existing annual subscriptions will be able to continue to enjoy ad-free \"\n",
    "    \"viewing until their subscription comes up for renewal. Those with \"\n",
    "    \"monthly subscriptions will have access to ad-free viewing until October 15.\"\n",
    ")\n",
    "\n",
    "# Create the match patterns\n",
    "pattern1 = [{\"LOWER\": \"Amazon\"}, {\"IS_TITLE\": True, \"POS\": \"PROPN\"}]\n",
    "pattern2 = [{\"LOWER\": \"ad-free\"}, {\"POS\": \"NOUN\"}]\n",
    "\n",
    "# Initialize the Matcher and add the patterns\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"PATTERN1\", None, pattern1)\n",
    "matcher.add(\"PATTERN2\", None, pattern2)\n",
    "\n",
    "# Iterate over the matches\n",
    "for match_id, start, end in matcher(doc):\n",
    "    # Print pattern string name and text of matched span\n",
    "    print(doc.vocab.strings[match_id], doc[start:end].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PATTERN1 Amazon Prime\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN1 Amazon Prime\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN2 ad-free viewing\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy.matcher import Matcher\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\n",
    "    \"Twitch Prime, the perks program for Amazon Prime members offering free \"\n",
    "    \"loot, games and other benefits, is ditching one of its best features: \"\n",
    "    \"ad-free viewing. According to an email sent out to Amazon Prime members \"\n",
    "    \"today, ad-free viewing will no longer be included as a part of Twitch \"\n",
    "    \"Prime for new members, beginning on September 14. However, members with \"\n",
    "    \"existing annual subscriptions will be able to continue to enjoy ad-free \"\n",
    "    \"viewing until their subscription comes up for renewal. Those with \"\n",
    "    \"monthly subscriptions will have access to ad-free viewing until October 15.\"\n",
    ")\n",
    "\n",
    "# Create the match patterns\n",
    "pattern1 = [{\"LOWER\": \"amazon\"}, {\"IS_TITLE\": True, \"POS\": \"PROPN\"}]\n",
    "pattern2 = [{\"LOWER\": \"ad\"}, {\"TEXT\": \"-\"}, {\"LOWER\": \"free\"}, {\"POS\": \"NOUN\"}]\n",
    "\n",
    "# Initialize the Matcher and add the patterns\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"PATTERN1\", None, pattern1)\n",
    "matcher.add(\"PATTERN2\", None, pattern2)\n",
    "\n",
    "# Iterate over the matches\n",
    "for match_id, start, end in matcher(doc):\n",
    "    # Print pattern string name and text of matched span\n",
    "    print(doc.vocab.strings[match_id], doc[start:end].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14. Efficient phrase matching\n",
    "\n",
    "Sometimes it’s more efficient to match exact strings instead of writing patterns describing the individual tokens. This is especially true for finite categories of things – like all countries of the world. We already have a list of countries, so let’s use this as the basis of our information extraction script. A list of string names is available as the variable COUNTRIES.\n",
    "\n",
    "- Import the PhraseMatcher and initialize it with the shared vocab as the variable matcher.\n",
    "- Add the phrase patterns and call the matcher on the doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Czech Republic, Slovakia]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from spacy.lang.en import English\n",
    "\n",
    "with open(\"exercises/countries.json\") as f:\n",
    "    COUNTRIES = json.loads(f.read())\n",
    "\n",
    "nlp = English()\n",
    "doc = nlp(\"Czech Republic may help Slovakia protect its airspace\")\n",
    "\n",
    "# Import the PhraseMatcher and initialize it\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "\n",
    "# Create pattern Doc objects and add them to the matcher\n",
    "# This is the faster version of: [nlp(country) for country in COUNTRIES]\n",
    "patterns = list(nlp.pipe(COUNTRIES))\n",
    "matcher.add(\"COUNTRY\", None, *patterns)\n",
    "\n",
    "# Call the matcher on the test document and print the result\n",
    "matches = matcher(doc)\n",
    "print([doc[start:end] for match_id, start, end in matches])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting countries and relationships ----\n",
    "\n",
    "In the previous exercise, you wrote a script using spaCy’s PhraseMatcher to find country names in text. Let’s use that country matcher on a longer text, analyze the syntax and update the document’s entities with the matched countries.\n",
    "\n",
    "- Iterate over the matches and create a Span with the label \"GPE\" (geopolitical entity).\n",
    "- Overwrite the entities in doc.ents and add the matched span.\n",
    "- Get the matched span’s root head token.\n",
    "- Print the text of the head token and the span."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namibia --> Namibia\n",
      "South --> South Africa\n",
      "Cambodia --> Cambodia\n",
      "Kuwait --> Kuwait\n",
      "Somalia --> Somalia\n",
      "Haiti --> Haiti\n",
      "Mozambique --> Mozambique\n",
      "Somalia --> Somalia\n",
      "Rwanda --> Rwanda\n",
      "Singapore --> Singapore\n",
      "Sierra --> Sierra Leone\n",
      "Afghanistan --> Afghanistan\n",
      "Iraq --> Iraq\n",
      "Sudan --> Sudan\n",
      "Congo --> Congo\n",
      "Haiti --> Haiti\n",
      "[('Namibia', 'GPE'), ('South Africa', 'GPE'), ('Cambodia', 'GPE'), ('Kuwait', 'GPE'), ('Somalia', 'GPE'), ('Haiti', 'GPE'), ('Mozambique', 'GPE'), ('Somalia', 'GPE'), ('Rwanda', 'GPE'), ('Singapore', 'GPE'), ('Sierra Leone', 'GPE'), ('Afghanistan', 'GPE'), ('Iraq', 'GPE'), ('Sudan', 'GPE'), ('Congo', 'GPE'), ('Haiti', 'GPE')]\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.matcher import PhraseMatcher\n",
    "from spacy.tokens import Span\n",
    "import json\n",
    "\n",
    "with open(\"exercises/countries.json\") as f:\n",
    "    COUNTRIES = json.loads(f.read())\n",
    "with open(\"exercises/country_text.txt\") as f:\n",
    "    TEXT = f.read()\n",
    "\n",
    "nlp = English()\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "patterns = list(nlp.pipe(COUNTRIES))\n",
    "matcher.add(\"COUNTRY\", None, *patterns)\n",
    "\n",
    "# Create a doc and find matches in it\n",
    "doc = nlp(TEXT)\n",
    "\n",
    "# Iterate over the matches\n",
    "for match_id, start, end in matcher(doc):\n",
    "    # Create a Span with the label for \"GPE\"\n",
    "    span = Span(doc, start, end, label=\"GPE\")\n",
    "\n",
    "    # Overwrite the doc.ents and add the span\n",
    "    doc.ents = list(doc.ents) + [span]\n",
    "\n",
    "    # Get the span's root head token\n",
    "    span_root_head = span.root.head\n",
    "    # Print the text of the span root's head token and the span text\n",
    "    print(span_root_head.text, \"-->\", span.text)\n",
    "\n",
    "# Print the entities in the document\n",
    "print([(ent.text, ent.label_) for ent in doc.ents if ent.label_ == \"GPE\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contens",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "332.807px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}