123028_Assignment1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cc315cf1-28a7-4bf9-8942-2db41c3c8a6e",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Word2Vec (Assignment)\n",
    "\n",
    "Assignment Instruction\n",
    "\n",
    "1.  Try a real corpus (instead of banana apple, try something real... on the internet....) - not so big!  Just so you have a good taste of real stuff....like 50 documents, each having 50 words...(really up to you)\n",
    "\n",
    "2. Try a window size of 2\n",
    "\n",
    "3. Implement CBOW (instead of skipgrams)\n",
    "\n",
    "4. Compare normal version of skipgrams vs. negative sampling version of skipgrams in terms of time (using real corpus)\n",
    "\n",
    "Point criteria:\n",
    "0: not done/copy directly from your friend (inspired is ok)\n",
    "1: ok\n",
    "2: with comments, and a nice explanation along the notebook (like how Chaky do his tutorial...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06832538-50aa-4cb0-aab7-47674e9aec68",
   "metadata": {},
   "source": [
    "I will use Negative sampling both skipgram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7f8c1012-f7eb-4a04-ba3a-5d6553ea50ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "import spacy\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f2ca7235-30fc-456a-9312-6bbed9530087",
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3689848b-ef94-482f-9dd0-0652e1e60cd2",
   "metadata": {},
   "source": [
    "### 1. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a41db260-1b01-42c1-9a32-29bc65710f00",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load tokeniser from SpaCy. I will use only small version.\n",
    "\n",
    "tokeniser = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5b67071d-6fdc-48d3-b33c-ea4056547d2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#define corpus from real world messages.\n",
    "\n",
    "corpus = [\"The cat (Felis catus) is a domestic species of small carnivorous mammal. It is the only domesticated species in the family Felidae and is commonly referred to as the domestic cat or house cat to distinguish it from the wild members of the family. Cats are commonly kept as house pets but can also be farm cats or feral cats; the feral cat ranges freely and avoids human contact. Domestic cats are valued by humans for companionship and their ability to kill rodents. About 60 cat breeds are recognized by various cat registries.\",\n",
    "         \"The cat is similar in anatomy to the other felid species: it has a strong flexible body, quick reflexes, sharp teeth, and retractable claws adapted to killing small prey. Its night vision and sense of smell are well developed. Cat communication includes vocalizations like meowing, purring, trilling, hissing, growling, and grunting as well as cat-specific body language. Although the cat is a social species, it is a solitary hunter. As a predator, it is crepuscular, i.e. most active at dawn and dusk. It can hear sounds too faint or too high in frequency for human ears, such as those made by mice and other small mammals. It also secretes and perceives pheromones.\",\n",
    "         \"Female domestic cats can have kittens from spring to late autumn, with litter sizes often ranging from two to five kittens. Domestic cats are bred and shown at events as registered pedigreed cats, a hobby known as cat fancy. Population control of cats may be effected by spaying and neutering, but their proliferation and the abandonment of pets has resulted in large numbers of feral cats worldwide, contributing to the extinction of entire bird, mammal, and reptile species.\",\n",
    "         \"It was long thought that cat domestication began in ancient Egypt, where cats were venerated from around 3100 BC, but recent advances in archaeology and genetics have shown that their domestication occurred in Western Asia around 7500 BC.\",\n",
    "         \"As of 2021, there were an estimated 220 million owned and 480 million stray cats in the world. As of 2017, the domestic cat was the second most popular pet in the United States, with 95.6 million cats owned and around 42 million households owning at least one cat. In the United Kingdom, 26% of adults have a cat, with an estimated population of 10.9 million pet cats as of 2020.\",\n",
    "         \"The scientific name Felis catus was proposed by Carl Linnaeus in 1758 for a domestic cat. Felis catus domesticus was proposed by Johann Christian Polycarp Erxleben in 1777. Felis daemon proposed by Konstantin Satunin in 1904 was a black cat from the Transcaucasus, later identified as a domestic cat.\",\n",
    "         \"In 2003, the International Commission on Zoological Nomenclature ruled that the domestic cat is a distinct species, namely Felis catus. In 2007, it was considered a subspecies, F. silvestris catus, of the European wildcat (F. silvestris) following results of phylogenetic research. In 2017, the IUCN Cat Classification Taskforce followed the recommendation of the ICZN in regarding the domestic cat as a distinct species, Felis catus.\",\n",
    "         \"The domestic cat is a member of the Felidae, a family that had a common ancestor about 10–15 million years ago. The genus Felis diverged from other Felidae around 6–7 million years ago. Results of phylogenetic research confirm that the wild Felis species evolved through sympatric or parapatric speciation, whereas the domestic cat evolved through artificial selection. The domesticated cat and its closest wild ancestor are diploid and both possess 38 chromosomes and roughly 20,000 genes. The leopard cat (Prionailurus bengalensis) was tamed independently in China around 5500 BC. This line of partially domesticated cats leaves no trace in the domestic cat populations of today.\",\n",
    "         \"The earliest known indication for the taming of an African wildcat (F. lybica) was excavated close by a human Neolithic grave in Shillourokambos, southern Cyprus, dating to about 7500–7200 BC. Since there is no evidence of native mammalian fauna on Cyprus, the inhabitants of this Neolithic village most likely brought the cat and other wild mammals to the island from the Middle Eastern mainland.[48] Scientists therefore assume that African wildcats were attracted to early human settlements in the Fertile Crescent by rodents, in particular the house mouse (Mus musculus), and were tamed by Neolithic farmers. This mutual relationship between early farmers and tamed cats lasted thousands of years. As agricultural practices spread, so did tame and domesticated cats. Wildcats of Egypt contributed to the maternal gene pool of the domestic cat at a later time.\",\n",
    "         \"The earliest known evidence for the occurrence of the domestic cat in Greece dates to around 1200 BC. Greek, Phoenician, Carthaginian and Etruscan traders introduced domestic cats to southern Europe. During the Roman Empire they were introduced to Corsica and Sardinia before the beginning of the 1st millennium. By the 5th century BC, they were familiar animals around settlements in Magna Graecia and Etruria.[52] By the end of the Western Roman Empire in the 5th century, the Egyptian domestic cat lineage had arrived in a Baltic Sea port in northern Germany.\",\n",
    "         \"During domestication, cats have undergone only minor changes in anatomy and behavior, and they are still capable of surviving in the wild. Several natural behaviors and characteristics of wildcats may have pre-adapted them for domestication as pets. These traits include their small size, social nature, obvious body language, love of play and relatively high intelligence. Captive Leopardus cats may also display affectionate behavior toward humans but were not domesticated. House cats often mate with feral cats. Hybridisation between domestic and other Felinae species is also possible, producing hybrids such as the Kellas cat in Scotland.\",\n",
    "         \"Development of cat breeds started in the mid 19th century. An analysis of the domestic cat genome revealed that the ancestral wildcat genome was significantly altered in the process of domestication, as specific mutations were selected to develop cat breeds. Most breeds are founded on random-bred domestic cats. Genetic diversity of these breeds varies between regions, and is lowest in purebred populations, which show more than 20 deleterious genetic disorders.\",\n",
    "         \"The domestic cat has a smaller skull and shorter bones than the European wildcat. It averages about 46 cm (18 in) in head-to-body length and 23–25 cm (9–10 in) in height, with about 30 cm (12 in) long tails. Males are larger than females. Adult domestic cats typically weigh between 4 and 5 kg (9 and 11 lb).\",\n",
    "         \"Cats have seven cervical vertebrae (as do most mammals); 13 thoracic vertebrae (humans have 12); seven lumbar vertebrae (humans have five); three sacral vertebrae (as do most mammals, but humans have five); and a variable number of caudal vertebrae in the tail (humans have only three to five vestigial caudal vertebrae, fused into an internal coccyx). The extra lumbar and thoracic vertebrae account for the cat's spinal mobility and flexibility. Attached to the spine are 13 ribs, the shoulder, and the pelvis. Unlike human arms, cat forelimbs are attached to the shoulder by free-floating clavicle bones which allow them to pass their body through any space into which they can fit their head.\",\n",
    "         \"The cat skull is unusual among mammals in having very large eye sockets and a powerful specialized jaw.  Within the jaw, cats have teeth adapted for killing prey and tearing meat. When it overpowers its prey, a cat delivers a lethal neck bite with its two long canine teeth, inserting them between two of the prey's vertebrae and severing its spinal cord, causing irreversible paralysis and death. Compared to other felines, domestic cats have narrowly spaced canine teeth relative to the size of their jaw, which is an adaptation to their preferred prey of small rodents, which have small vertebrae.\",\n",
    "         \"The premolar and first molar together compose the carnassial pair on each side of the mouth, which efficiently shears meat into small pieces, like a pair of scissors. These are vital in feeding, since cats' small molars cannot chew food effectively, and cats are largely incapable of mastication.  Although cats tend to have better teeth than most humans, with decay generally less likely because of a thicker protective layer of enamel, a less damaging saliva, less retention of food particles between teeth, and a diet mostly devoid of sugar, they are nonetheless subject to occasional tooth loss and infection.\",\n",
    "         \"Cats have protractible and retractable claws. In their normal, relaxed position, the claws are sheathed with the skin and fur around the paw's toe pads. This keeps the claws sharp by preventing wear from contact with the ground and allows for the silent stalking of prey. The claws on the forefeet are typically sharper than those on the hindfeet. Cats can voluntarily extend their claws on one or more paws. They may extend their claws in hunting or self-defense, climbing, kneading, or for extra traction on soft surfaces. Cats shed the outside layer of their claw sheaths when scratching rough surfaces.\",\n",
    "         \"Most cats have five claws on their front paws and four on their rear paws. The dewclaw is proximal to the other claws. More proximally is a protrusion which appears to be a sixth \\\"finger\\\". This special feature of the front paws on the inside of the wrists has no function in normal walking but is thought to be an antiskidding device used while jumping. Some cat breeds are prone to having extra digits (\\\"polydactyly\\\"). Polydactylous cats occur along North America's northeast coast and in Great Britain.\",\n",
    "         \"The cat is digitigrade. It walks on the toes, with the bones of the feet making up the lower part of the visible leg. Unlike most mammals, it uses a \\\"pacing\\\" gait and moves both legs on one side of the body before the legs on the other side. It registers directly by placing each hind paw close to the track of the corresponding fore paw, minimizing noise and visible tracks. This also provides sure footing for hind paws when navigating rough terrain. As it speeds up from walking to trotting, its gait changes to a \\\"diagonal\\\" gait: The diagonally opposite hind and fore legs move simultaneously.\",\n",
    "         \"Most breeds of cat are notably fond of sitting in high places, or perching. A higher place may serve as a concealed site from which to hunt; domestic cats strike prey by pouncing from a perch such as a tree branch. Another possible explanation is that height gives the cat a better observation point, allowing it to survey its territory. A cat falling from heights of up to 3 meters (9.8 ft) can right itself and land on its paws.\",\n",
    "         \"During a fall from a high place, a cat reflexively twists its body and rights itself to land on its feet using its acute sense of balance and flexibility. This reflex is known as the cat righting reflex. A cat always rights itself in the same way during a fall, if it has enough time to do so, which is the case in falls of 90 cm (2 ft 11 in) or more. How cats are able to right themselves when falling has been investigated as the \\\"falling cat problem\\\".\",\n",
    "         \"The cat family (Felidae) can pass down many colors and patterns to their offsprings. The domestic cat genes MC1R and ASIP allow for the variety of color in coats. The feline ASIP gene consists of three coding exons. Three novel microsatellite markers linked to ASIP were isolated from a domestic cat BAC clone containing this gene and were used to perform linkage analysis in a pedigree of 89 domestic cats that segregated for melanism.\",\n",
    "         \"Cats have excellent night vision and can see at only one-sixth the light level required for human vision. This is partly the result of cat eyes having a tapetum lucidum, which reflects any light that passes through the retina back into the eye, thereby increasing the eye's sensitivity to dim light. Large pupils are an adaptation to dim light. The domestic cat has slit pupils, which allow it to focus bright light without chromatic aberration. At low light, a cat's pupils expand to cover most of the exposed surface of its eyes. The domestic cat has rather poor color vision and only two types of cone cells, optimized for sensitivity to blue and yellowish green; its ability to distinguish between red and green is limited. A response to middle wavelengths from a system other than the rod cells might be due to a third type of cone. This appears to be an adaptation to low light levels rather than representing true trichromatic vision. Cats also have a nictitating membrane, allowing them to blink without hindering their vision.\",\n",
    "         \"The domestic cat's hearing is most acute in the range of 500 Hz to 32 kHz. It can detect an extremely broad range of frequencies ranging from 55 Hz to 79 kHz. It can hear a range of 10.5 octaves, while humans and dogs can hear ranges of about 9 octaves. Its hearing sensitivity is enhanced by its large movable outer ears, the pinnae, which amplify sounds and help detect the location of a noise. It can detect ultrasound, which enables it to detect ultrasonic calls made by rodent prey. Recent research has shown that cats have socio-spatial cognitive abilities to create mental maps of owners' locations based on hearing owners' voices.\",\n",
    "         \"Cats have an acute sense of smell, due in part to their well-developed olfactory bulb and a large surface of olfactory mucosa, about 5.8 square centimetres (29⁄32 square inch) in area, which is about twice that of humans. Cats and many other animals have a Jacobson's organ in their mouths that is used in the behavioral process of flehmening. It allows them to sense certain aromas in a way that humans cannot. Cats are sensitive to pheromones such as 3-mercapto-3-methylbutan-1-ol, which they use to communicate through urine spraying and marking with scent glands. Many cats also respond strongly to plants that contain nepetalactone, especially catnip, as they can detect that substance at less than one part per billion. About 70–80% of cats are affected by nepetalactone. This response is also produced by other plants, such as silver vine (Actinidia polygama) and the herb valerian; it may be caused by the smell of these plants mimicking a pheromone and stimulating cats' social or sexual behaviors.\",\n",
    "         \"Cats have relatively few taste buds compared to humans (470 or so versus more than 9,000 on the human tongue).[97] Domestic and wild cats share a taste receptor gene mutation that keeps their sweet taste buds from binding to sugary molecules, leaving them with no ability to taste sweetness. Their taste buds instead respond to acids, amino acids like protein, and bitter tastes. Cats also have a distinct temperature preference for their food, preferring food with a temperature around 38 °C (100 °F) which is similar to that of a fresh kill and routinely rejecting food presented cold or refrigerated (which would signal to the cat that the \\\"prey\\\" item is long dead and therefore possibly toxic or decomposing).\",\n",
    "         \"To aid with navigation and sensation, cats have dozens of movable whiskers (vibrissae) over their body, especially their faces. These provide information on the width of gaps and on the location of objects in the dark, both by touching objects directly and by sensing air currents; they also trigger protective blink reflexes to protect the eyes from damage.\"]\n",
    "\n",
    "corpus_tokenised = []\n",
    "for i in range(len(corpus)):\n",
    "    corpus_tokenised.append([str(token) for token in tokeniser([c for c in corpus][i])])\n",
    "# corpus_tokenised"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "00f7f809-e735-4e1c-b2ba-28b1444b2518",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "109"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#delete duplicate word define as vocabs, also include unknow as <UNK>.\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocabs  = list(set(flatten(corpus_tokenised)))\n",
    "vocabs.append('<UNK>') # to let '<UNK>' become the last.\n",
    "word2index = {v: idx+1 for idx, v in enumerate(vocabs)} #plus one to shift value in word index to assign <UNK>'s value to 0 without duplication with others.\n",
    "word2index['<UNK>'] = 0\n",
    "\n",
    "#Let check word to index's value\n",
    "word2index[\"cat\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4934c2dc-6e6d-49fc-a38b-e5fea886ed4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'coast'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create index to word\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "index2word[50]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19715136-4470-4bff-8035-9a9a84206679",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2. Prepare train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "839cb11d-4e10-49d4-9a79-7a98014545e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['(', 'The'],\n",
       " ['(', 'cat'],\n",
       " ['(', 'Felis'],\n",
       " ['(', 'catus'],\n",
       " ['Felis', 'cat'],\n",
       " ['Felis', '('],\n",
       " ['Felis', 'catus'],\n",
       " ['Felis', ')'],\n",
       " ['catus', '('],\n",
       " ['catus', 'Felis'],\n",
       " ['catus', ')'],\n",
       " ['catus', 'is'],\n",
       " [')', 'Felis'],\n",
       " [')', 'catus'],\n",
       " [')', 'is'],\n",
       " [')', 'a'],\n",
       " ['is', 'catus'],\n",
       " ['is', ')'],\n",
       " ['is', 'a'],\n",
       " ['is', 'domestic'],\n",
       " ['a', ')'],\n",
       " ['a', 'is'],\n",
       " ['a', 'domestic'],\n",
       " ['a', 'species'],\n",
       " ['domestic', 'is'],\n",
       " ['domestic', 'a'],\n",
       " ['domestic', 'species'],\n",
       " ['domestic', 'of'],\n",
       " ['species', 'a'],\n",
       " ['species', 'domestic'],\n",
       " ['species', 'of'],\n",
       " ['species', 'small'],\n",
       " ['of', 'domestic'],\n",
       " ['of', 'species'],\n",
       " ['of', 'small'],\n",
       " ['of', 'carnivorous'],\n",
       " ['small', 'species'],\n",
       " ['small', 'of'],\n",
       " ['small', 'carnivorous'],\n",
       " ['small', 'mammal'],\n",
       " ['carnivorous', 'of'],\n",
       " ['carnivorous', 'small'],\n",
       " ['carnivorous', 'mammal'],\n",
       " ['carnivorous', '.'],\n",
       " ['mammal', 'small'],\n",
       " ['mammal', 'carnivorous'],\n",
       " ['mammal', '.'],\n",
       " ['mammal', 'It'],\n",
       " ['.', 'carnivorous'],\n",
       " ['.', 'mammal']]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skipgrams = []\n",
    "\n",
    "#for each corpus\n",
    "for sent in corpus_tokenised:\n",
    "    #for each sent [\"apple\", \"banana\", \"fruit\"]\n",
    "    for i in range(2, len(sent) - 2): # start from 1 to second last but we use window size of 2, then first centre word should be at index 2\n",
    "        center_word = sent[i]\n",
    "        outside_words = np.ravel([sent[i - 2:i],sent[i+1: i + 3]])  # window_size = 2\n",
    "        for o in outside_words:\n",
    "            skipgrams.append([center_word, o])\n",
    "\n",
    "skipgrams[:50] #shows only sample 50 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0e442e68-4612-4bce-ac3d-cd52b65bb082",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size, corpus):\n",
    "    \n",
    "    skipgrams = []\n",
    "\n",
    "    for sent in corpus:\n",
    "        for i in range(2, len(sent) - 2): #start from 2 to third last for window size = 2\n",
    "            centre_word = word2index[sent[i]]\n",
    "            outside_words = []\n",
    "            for j in np.ravel([sent[i - 2:i],sent[i + 1: i + 3]]):\n",
    "                outside_words.append(word2index[j])  #window_size = 2\n",
    "            for o in outside_words:\n",
    "                skipgrams.append([centre_word, o])\n",
    "                \n",
    "    #only get a batch, not the entire list\n",
    "    random_index = np.random.choice(range(len(skipgrams)), batch_size, replace=False)\n",
    "    #appending some list of inputs and labels\n",
    "    random_inputs, random_labels = [], []   \n",
    "    for index in random_index:\n",
    "        random_inputs.append([skipgrams[index][0]])  #center words, this will be a shape of (1, ) --> (1, 1) for modeling\n",
    "        random_labels.append([skipgrams[index][1]])\n",
    "        \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "91df28c8-4579-4a75-95ac-e355ba35052b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 1)\n",
      "label=array([[  89],\n",
      "       [1007],\n",
      "       [1007],\n",
      "       [ 684],\n",
      "       [ 363],\n",
      "       [ 927],\n",
      "       [ 335],\n",
      "       [1043],\n",
      "       [ 320],\n",
      "       [  60]])\n"
     ]
    }
   ],
   "source": [
    "input, label = random_batch(10, corpus_tokenised)\n",
    "\n",
    "print(f\"{input.shape}\")\n",
    "print(f\"{label=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "117023ba-2646-4604-9fe1-2d66d96b60bb",
   "metadata": {},
   "source": [
    "### 3. Unigram distribution\n",
    "\n",
    "$$P(w)=U(w)^{3/4}/Z$$\n",
    "\n",
    "Defining the probability of sampling negative words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ae427673-9360-4a92-a388-c349c5b79f73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_count = Counter(flatten(corpus_tokenised))\n",
    "# word_count # Showing word count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9eb29e54-17f9-47e5-974d-47ef6a700f1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2898"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_total_words = sum([c for w, c in word_count.items()])\n",
    "num_total_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d36deb1a-d578-4439-9279-ad8f9a2c8fd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "z = 0.001  #scaling up low frequency terms\n",
    "unigram_table = []\n",
    "\n",
    "# create loop for the P(w) formula\n",
    "for v in vocabs:\n",
    "    uw = word_count[v]/num_total_words\n",
    "    uw_alpha = uw ** (3/4)\n",
    "    uw_alpha_dividebyz = int(uw_alpha / z)\n",
    "    unigram_table.extend([v] * uw_alpha_dividebyz)\n",
    "    \n",
    "# Counter(unigram_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0648a416-f885-46cb-bbbf-4464f3e738e9",
   "metadata": {},
   "source": [
    "### 4. Negative sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cc3179db-a5e4-4e09-aef2-b35dac97ee17",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1b3cb074-fd05-4223-a780-9adfc01470b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def negative_sampling(targets, unigram_table, k):\n",
    "    \n",
    "    batch_size = targets.shape[0]\n",
    "    neg_samples = []\n",
    "    \n",
    "    for i in range(batch_size):\n",
    "        #randomly pick k negative words from unigram_table\n",
    "        target_index = targets[i].item()  #looping each of the batch....\n",
    "        nsample = []\n",
    "        while len(nsample) < k:\n",
    "            neg = random.choice(unigram_table)\n",
    "            #if this word == target, skip this word\n",
    "            if word2index[neg] == target_index:\n",
    "                continue\n",
    "            nsample.append(neg)\n",
    "        #append this word to some list\n",
    "        neg_samples.append(prepare_sequence(nsample, word2index).reshape(1, -1))  #tensor[], tensor[]\n",
    "    return torch.cat(neg_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6149da4a-00db-49f4-ba35-494e3e8aee72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[431],\n",
       "        [205]]),\n",
       " array([[272],\n",
       "        [912]]))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2\n",
    "input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "\n",
    "input_batch, label_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d70845bb-5dcf-40a6-9ba6-78c57f395f89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Tensor\n",
    "\n",
    "input_batch = torch.LongTensor(input_batch)\n",
    "label_batch = torch.LongTensor(label_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "10459369-7b8d-4553-97d6-0f90cf452bab",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_neg = 10  # I will try 10 as in the paper.\n",
    "neg_samples = negative_sampling(label_batch, unigram_table, num_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0b451334-4325-4eca-9bb1-520ec01bdc68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 10])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neg_samples.shape # shape is (batch_size, num_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39fc2fd-44bc-43a9-adc7-c1f9e79d50de",
   "metadata": {},
   "source": [
    "### 5. Model\n",
    "\n",
    "$$\\mathbf{J}_{\\text{neg-sample}}(\\mathbf{v}_c,o,\\mathbf{U})=-\\log(\\sigma(\\mathbf{u}_o^T\\mathbf{v}_c))-\\sum_{k=1}^K\\log(\\sigma(-\\mathbf{u}_k^T\\mathbf{v}_c))$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "517ecde4-fb1b-4337-a7a2-90202532ca39",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SkipgramNeg(nn.Module):\n",
    "    \n",
    "    def __init__(self, voc_size, emb_size):\n",
    "        super(SkipgramNeg, self).__init__()\n",
    "        self.embedding_centre_word  = nn.Embedding(voc_size, emb_size)\n",
    "        self.embedding_outside_word = nn.Embedding(voc_size, emb_size)\n",
    "        self.logsigmoid = nn.LogSigmoid()\n",
    "        \n",
    "    def forward(self, centre_words, outside_words, negative_words):\n",
    "        #center_words, outside_words: (batch_size, 1)\n",
    "        #negative_words:  (batch_size, k)\n",
    "        \n",
    "        centre_embed  = self.embedding_centre_word(centre_words)    #(batch_size, 1, emb_size)\n",
    "        outside_embed = self.embedding_outside_word(outside_words)  #(batch_size, 1, emb_size)\n",
    "        neg_embed     = self.embedding_outside_word(negative_words) #(batch_size, k, emb_size)\n",
    "        \n",
    "        uovc          =  outside_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, 1)\n",
    "        ukvc          = -neg_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, k)\n",
    "        ukvc_sum      =  torch.sum(ukvc, 1).view(-1, 1) #(batch_size, 1)\n",
    "        \n",
    "        loss = self.logsigmoid(uovc) + self.logsigmoid(ukvc_sum)  #(batch_size, 1) + (batch_size, 1)\n",
    "                \n",
    "        return -torch.mean(loss)  #scalar, loss should be scalar, to call backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f71998dc-c5ae-4e10-bf7f-970e85907c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "input, label = random_batch(batch_size, corpus_tokenised)\n",
    "input_tensor = torch.LongTensor(input)  \n",
    "label_tensor = torch.LongTensor(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f31d5d8f-7207-484c-88bc-5725c46ea151",
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_size = 10 # I will try on 10.\n",
    "voc_size = len(vocabs)\n",
    "model = SkipgramNeg(voc_size, emb_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fa980b9f-c95b-493a-88a6-8824d1f1e0c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_tensor = negative_sampling(label_tensor, unigram_table, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "50be8f75-c0c5-41cf-b919-0641699bb5e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 1]), torch.Size([2, 1]))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_tensor.shape, label_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "af86775f-5adf-4a5d-bc34-f88bc71f6199",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(16.9753, grad_fn=<NegBackward0>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = model(input_tensor, label_tensor, neg_tensor)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af276ca9-f6ff-4c76-91c3-a23d13070342",
   "metadata": {},
   "source": [
    "### 4. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "32c112ad-5625-43e7-8514-6f882e59b106",
   "metadata": {},
   "outputs": [],
   "source": [
    "voc_size   = len(vocabs)\n",
    "batch_size = 2\n",
    "emb_size   = 10\n",
    "model      = SkipgramNeg(voc_size, emb_size)\n",
    "\n",
    "optimizer  = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a7508771-94d9-4185-821d-366ea8e98c24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1000 | Loss: 3.679335 | Time: 0m 0s\n",
      "Epoch 2000 | Loss: 7.086188 | Time: 0m 0s\n",
      "Epoch 3000 | Loss: 0.686530 | Time: 0m 0s\n",
      "Epoch 4000 | Loss: 0.000000 | Time: 0m 0s\n",
      "Epoch 5000 | Loss: 2.601206 | Time: 0m 0s\n",
      "Process Time: 3m 51s\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 5000\n",
    "startp = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    #get random batch\n",
    "    start = time.time()\n",
    "    input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "    input_batch = torch.LongTensor(input_batch)\n",
    "    label_batch = torch.LongTensor(label_batch)\n",
    "    neg_batch   = negative_sampling(label_batch, unigram_table, 5)    \n",
    "    \n",
    "    #loss = model\n",
    "    loss = model(input_batch, label_batch, neg_batch)\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "    #backpropagate\n",
    "    loss.backward()\n",
    "    \n",
    "    #update alpha\n",
    "    optimizer.step()\n",
    "    \n",
    "    #print epoch loss\n",
    "    if (epoch + 1) % 1000 == 0:\n",
    "        print(f\"Epoch {epoch+1} | Loss: {loss:.6f} | Time: {epoch_mins}m {epoch_secs}s\")\n",
    "endp = time.time()\n",
    "mins, secs = epoch_time(startp, endp)\n",
    "print(f\"Process Time: {mins}m {secs}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52971237-11bd-47a0-b6be-7bf53a5ddb66",
   "metadata": {},
   "source": [
    "### 5. Plot the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b6980cec-2e17-45a3-973f-12e099b0b0bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([109])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat = torch.LongTensor([word2index['cat']])\n",
    "cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1bbb495d-9ab1-40bf-9a2f-8a3ca80afb6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.0557,  0.0826, -1.5152, -0.1629, -1.0396, -0.8252,  0.7407,  0.0850,\n",
       "          0.1388, -0.0639]], grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_centre_embed = model.embedding_centre_word(cat)\n",
    "cat_outside_embed = model.embedding_outside_word(cat)\n",
    "\n",
    "cat_embed = (cat_centre_embed + cat_outside_embed) / 2\n",
    "cat_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "715018f5-7c04-464b-b49c-f35f44f182ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embed(word):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except:\n",
    "        index = word2index['<UNK>']\n",
    "    \n",
    "    word = torch.LongTensor([index])\n",
    "\n",
    "    center_embed  = model.embedding_centre_word(word)\n",
    "    outside_embed = model.embedding_outside_word(word)\n",
    "    \n",
    "    embed = (center_embed + outside_embed) / 2\n",
    "    \n",
    "    return  embed[0][0].item(), embed[0][1].item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "64b8c148-8200-4752-b48d-aba4de3d0d53",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-1.0556931495666504, 0.08260565996170044)\n"
     ]
    }
   ],
   "source": [
    "print(get_embed('cat'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ba766811-2269-4c01-a3df-872c7056ae41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAADFCAYAAAB3ldR7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7fElEQVR4nO3de3zOdf/A8ddnBzsyNIcRZo7teJlNDm1kMWWOTRRluSncQt0J6bC7+HGHSDrcuotCt7EQbiFDxlQ2ZuaUaNJMDGPY2OHz+2N2ZWyYXdu1w/v5eOxh+3w/1+d6f7/Y+/p+PyeltUYIIYQwFQtzByCEEKJykcQihBDCpCSxCCGEMClJLEIIIUxKEosQQgiTksQihBDCpCSxCFGBKKXClVKv3sfr3lFKPVYaMQlxKytzByCEKH1a67fMHYOoOuSORYhyTik1RSl1RCm1GWh1o6yZUmqDUipOKRWtlGqtlHJSSiUppSxu1LFXSp1USlkrpRYppUJvlPsrpWKUUvuUUj8rpaorpSyVUjOVUruVUglKqRfNeMqiglPmmHnv7OysXV1dy/x9hahorly5QlJSEg899BBaaw4dOoSzszOXLl2icePG2NracuXKFZKTk2nZsiW//vor9erVo3r16pw/f55Lly7h6upKUlISTk5OODk5ceDAAdzc3HBwcCAnJwcLCwtSU1PJzs7GxcWF3Nxcjhw5gpubGzY2Nua+BOImcXFxqVrrOuaO427M8ijM1dWV2NhYc7y1EBXK3LlzOX/+PO+88w4Ar7zyCrVr12batGlYWVmRnZ2NjY0NDRs2JDY2lq+//prt27fz6aef0q9fP0aPHk23bt0ICwsjJCSEVq1aMXLkSHbu3FngfUJDQ0lISCA7OxuAunXrMnv2bLp3717m5yyKppQ6UYy6llrrnNKMpyjSxyJEOaeUKvBzbm4uNWvWJD4+/ra6vXv3ZvLkyZw/f564uDi6du1a4LjW+rb28ss//PBDgoODTRq7uHd9+/bl5MmTZGZmMm7cOHJycvjtt9947733AFi0aBFAIwCl1BBgLFAN+AkYrbXOUUpdBt4HgoF/KKW6Ar0AOyAGeFFrrZVS/sDnwBVgB/C41tpTKWUJzAC6ADbAR1rrfyulXIAIoAZ5eWOU1jq6qHORPhYhyrHAwEBWrVpFRkYG6enprF27Fnt7e5o2bcqKFSuAvKSwb98+ABwdHWnXrh3jxo0jJCQES0vLAu21bt2aU6dOsXv3bgDS09PJzs4mODiYTz75hKysLAB++eUXrly5UoZnKr744gvi4uKIjY1l3rx59O/fn5UrVxqPR0REAFxQSj0EDAQ6aa0NQA4w+EY1ByBRa/2w1noHMF9r7a+19iQvuYTcqLcQGKm17nDj9fn+BlzUWvsD/sAIpVRT4Blg44338wHi73QucscihJn97/j/+GDPB5y+cpr6DvUZ5zuOnm49AfD19WXgwIEYDAaaNGlCQEAAAEuXLmXUqFFMnTqVrKwsBg0ahI+PDwADBw5kwIABbNu27bb3qlatGhEREbz00ktkZGRgZ2fH5s2bGT58OElJSfj6+qK1pk6dOqxevbqsLoEA5s2bx6pVqwA4efIkv/32G25ubvz444+0aNGCI0eOAFwGgoC2wO4bd592wJkbzeQA39zU7KNKqdcAe6A2cEApFQ1U11rH3KjzNX8lnO6Ad/5AD8AJaAHsBr5QSlkDq7XW8Xc6lxJ33iulbIHt5N02WQGRWuu37/QaPz8/LX0sQuQllfCYcDJzMo1ltpa2hHcMNyYXUflt27aNN954g02bNmFvb0+XLl0IDw/n2LFjHDhwgNatW3PkyBHef//9OOBLoIHWevKt7SilLmutHW98bwucAPy01ieVUuE3qn0AxGutm9yo5w18feNR2DfAAq31xkLabgD0JO8R3Eyt9VdFnY8pHoVdA7pqrX0AA9BDKdXeBO0KUel9sOeDAkkFIDMnkw/2fGCmiIQ5XLx4kVq1amFvb8/hw4f58ccfAejfvz+rV6/mv//9LwMHDsyvHgWEKqXqAiilaiulmhTSrO2NP1OVUo5AKIDW+gKQftPv6UE3vWYjMOrGnQlKqZZKKYcb7Z/RWn9GXt+M753Op8SPwnTeLc/lGz9a3/iS3cOEuAenr5wuVrmouBISEoiKiuLixYs4OTkRFBSEt7c3AD169ODTTz/F29ubVq1a0b593u/8WrVq4e7uzsGDB2nXrh0AWuuDSqk3gE035ixlAX8n7+7ESGudppT6DNgPJJH3OCvf34DPlFJXgG3AxRvl/wFcgT0q7znbWaAveZ35E5RSWeT9vn/uTudqknksN0YSxAHNyRtFMPFO9eVRmBB5ukd2J+VKym3lLg4ubArdZIaIRGlISEhg7dq1xsERANbW1vTq1cuYXO6FUipOa+1X0niUUo5a68s3vp8EuGitx5W03XwmGRWmtc65MVrgQaCdUsrz1jpKqReUUrFKqdizZ8+a4m2FqPDG+Y7D1tK2QJmtpS3jfE32f1yUA1FRUQWSCkBWVhZRUVFmioieSql4pVQiEABMNWXjJh0VduPWaxvQA0i85dgCYAHk3bGY8n2FqKjyO+iLGhUmKoeLFy8Wq7y0aa0jyJuXUipKnFiUUnWArBtJxQ54DPhXiSMTooro6dZTEkkl5+TkVGgScXJyMkM0pc8Uj8JcgK1KqQTyOoe+11qvM0G7QghRKQQFBWFtbV2gzNramqCgIDNFVLpMMSosAWhjgliEEKJSyu+gL2pUWGUjM++FEKIMeHt7V9pEcitZK0wIIYRJSWIRQghhUpJYhBBCmJQkFiGEECYliUUIIYRJSWIRQghhUpJYhBBCmJQkFiGEECYliUUIIYRJSWIpx06dOkVoaOjdKwohRDkiiaUca9CgAZGRkeYOQwghikUSSzkxceJEPv74Y+PP4eHhzJ49G0/PvD3TFi1aRP/+/enRowctWrTgtddeM9bdtGkTHTp0wNfXlwEDBnD58uXb2hdCiLIiiaWcGDRoEBERf+27s3z5cvz9/QvUiY+PJyIigv379xMREcHJkydJTU1l6tSpbN68mT179uDn58f7779f1uELIYSRKTb6agR8BdQHcoEFWusPStpuVdOmTRvOnDnDqVOnOHv2LLVq1aJx48YF6gQFBRk3BnJ3d+fEiROkpaVx8OBBOnXqBMD169fp0KFDmccvhBD5TLFsfjbwD631HqVUdSBOKfW91vqgCdquUkJDQ4mMjOT06dMMGjTotuM2NjbG7y0tLcnOzkZrTbdu3fjvf/9blqEKIUSRSvwoTGudorXec+P7dOAQ0LCk7VZFgwYNYtmyZURGRt7zaLD27duzc+dOfv31VwCuXr3KL7/8UpphCiHEHZl0oy+llCt5u0n+ZMp2K4sre89waWMSOWnXsKxpQ41gVxza1DUe9/DwID09nYYNG+Li4kJSUtJd26xTpw6LFi3i6aef5tq1awBMnTqVli1bltZpCCHEHSmttWkaUsoR+AGYprVeWcjxF4AXABo3btz2xIkTJnnfiuLK3jOkrTyKzso1lilrC2r2b1EguQghRFGUUnFaaz9zx3E3JhkVppSyBr4BlhaWVAC01gu01n5aa786deqY4m0rlEsbkwokFQCdlculjUnmCUgIIUpJiROLUkoBnwOHtNYyzrUIOWnXilUuhBAVlSnuWDoBzwJdlVLxN76eMEG7lYplTZtilQshREVV4s57rfUOQJkglkqtRrBroX0sNYJdzReUEEKUApl5X0Yc2tSlZv8WxjsUy5o20nFfRlxdXUlNTTV3GEJUGSYdbizuzKFNXUkkQohKT+5YxH376quv8Pb2xsfHh2effZYTJ04QFBSEt7c3QUFB/P777wCEhYUxatQoHn30Udzc3Pjhhx8YNmwYDz30EGFhYcb2HB0dmThxIm3btuWxxx7j559/pkuXLri5ubFmzRoAMjMzef755/Hy8qJNmzZs3boVuPMinfnefPNNPvjgr9WGpkyZwrx580rxCglRRWmty/yrbdu2WlRsiYmJumXLlvrs2bNaa63PnTunQ0JC9KJFi7TWWn/++ee6T58+Wmuthw4dqgcOHKhzc3P16tWrdfXq1XVCQoLOycnRvr6+eu/evVprrQG9fv16rbXWffv21d26ddPXr1/X8fHx2sfHR2ut9axZs3RYWJjWWutDhw7pRo0a6YyMDL1w4ULdtGlTnZaWpjMyMnTjxo3177//rrXWukmTJvrs2bP6t99+023atNFaa52Tk6Pd3Nx0ampqWVwuIUwCiNVm+J1d3C+5YxH3ZcuWLYSGhuLs7AxA7dq12bVrF8888wwAzz77LDt27DDW79WrF0opvLy8qFevHl5eXlhYWODh4WFcYaBatWr06NEDAC8vLzp37oy1tTVeXl7GOjt27ODZZ58FoHXr1jRp0sS4hE3+Ip22trbGRTpv5urqygMPPMDevXvZtGkTbdq04YEHHii1ayREVSV9LOK+aK3Jm8JUtJuP5y+gaWFhUWAxTQsLC7KzswGwtrY2vubmejfX0XdYKaKwRTpvNXz4cBYtWsTp06cZNmzYHeMXQtwfuWMR9yUoKIjly5dz7tw5AM6fP0/Hjh1ZtmwZAEuXLuWRRx4x+fsGBgaydOlSAH755Rd+//13WrVqdc+v79evHxs2bGD37t0EBwebPD5R/uSvFt6sWTPc3d154oknWLBgASEhIff0+rfeeovNmzcXeXz16tUcPHjwnutXBXLHIgq1em8yMzce4VRaBg1q2jEhuBV92/y1aLWHhwdTpkyhc+fOWFpa0qZNG+bNm8ewYcOYOXMmderUYeHChSaPa/To0YwcORIvLy+srKxYtGhRgTuVu6lWrRqPPvooNWvWxNLS0uTxifJFa02/fv0YOnSo8UNPfHw8a9euvafX5+Tk8M4779yxzurVqwkJCcHd3R3grvWrApMtQlkcfn5+OjY2tszfV9yb1XuTmbxyPxlZOcYyO2tLpvf3KpBcKqLc3Fx8fX1ZsWIFLVq0MHc4opRt2bKF8PBwtm/fXqB827ZthIeH4+zsTGJiIm3btmXJkiUopXB1dWXYsGFs2rSJMWPGsGHDBkJCQggNDWXSpEmsWbMGKysrunfvTv/+/QkJCcHJyQknJye++eYb3n33XWP9d955h7Vr15KRkUHHjh3597//jVKKLl268PDDD7N161bS0tL4/PPPCQgIuOv5VKlFKEXlMnPjkQJJBSAjK4eZG4+YKSLTOHjwIM2bNycoKEiSShWRnzQKs3fvXubOncvBgwc5fvw4O3fuNB6ztbVlx44dBTbcO3/+PKtWreLAgQMkJCTwxhtv0LFjR3r37s3MmTOJj4+nWbNmBd5jzJgx7N69m8TERDIyMli3bp3xWHZ2Nj///DNz587ln//8p4nP3LwksYjbnErLKFZ5ReHu7s7x48eZPXu2uUMR5UC7du148MEHsbCwwGAwFNj/aODAgbfVr1GjBra2tgwfPpyVK1dib29/1/fYunUrDz/8MF5eXmzZsoUDBw4Yj/Xv3x+Atm3b3tPeSxVJpUksjo6OJm1v0aJFnDp1yqRtVhQNatoVq1yI8srDw4O4uLhCj91pFKGDg8Nt9a2srPj555958sknWb16tXFofFEyMzMZPXo0kZGR7N+/nxEjRpCZmXnb+xc1grEiqzSJxdSqcmKZENwKO+uCHdt21pZMCL730VdClAddu3bl2rVrfPbZZ8ay3bt388MPPxS7rcuXL3Px4kWeeOIJ5s6dS3x8PADVq1cnPT39tvr5ScTZ2ZnLly8TGRl5fydRAZlqo68vlFJnlFKJpmivpGbOnIm/vz/e3t68/fbbxvK+ffvStm1bPDw8WLBgAZA36iMsLAxPT0+8vLyYM2cOkZGRxMbGMnjwYAwGAxkZFfsRUHH1bdOQ6f29aFjTDgU0rGlXJh33hQ0LzZ/8eKu0tDQ+/vjjUo1HVAyHorey4O/PM3tQLxb8/XkORW81HlNKsWrVKr7//nuaNWuGh4cH4eHhNGjQoNjvk56eTkhICN7e3nTu3Jk5c+YAMGjQIGbOnEmbNm04duyYsX7NmjUZMWIEXl5e9O3bF39//5KfbEVhiun7QCDgCyTeS/3SWNLFwcFBa631xo0b9YgRI3Rubq7OycnRPXv21D/88IPWOm/ZEa21vnr1qvbw8NCpqak6NjZWP/bYY8Z2Lly4oLXWunPnznr37t0mj1MULjc3V7dv315/8sknxrK9e/fq7du3F1r/t99+0x4eHmUVniinDm7foucO6a9nPdXT+DV3SH99cPsWc4dWKqhKS7porbcD503RVklt2rTJuFyHr68vhw8f5ujRowDMmzcPHx8f2rdvz8mTJzl69Chubm4cP36cl156iQ0bNlCjRg0zn0HVtHXrVqytrRk5cqSxzGAw0KZNG4KCgvD19cXLy4tvv/0WgEmTJnHs2DEMBgMTJkwgJSWFwMBADAYDnp6eREdHm+tURBmKXvYV2dcL7sKaff0a0cu+MlNEAirhBEmtNZMnT+bFF18sUL5t2zY2b97Mrl27sLe3p0uXLmRmZlKrVi327dvHxo0b+eijj1i+fDlffPGFmaKvuooaFmpra8uqVauoUaMGqamptG/fnt69ezNjxgwSExONz7lnz55NcHAwU6ZMIScnh6tXr5bxGQhzSD9X+D47RZWLslFmiUUp9QLwAkDjxo2L/fr/Hf8fH+z5gNNXTlPfoT7jfMfR063nbfWCg4N58803GTx4MI6OjiQnJ2Ntbc3FixepVasW9vb2HD58mB9//BGA1NRUqlWrxpNPPkmzZs2My7gX1SEnypbWmtdff53t27djYWFBcnIyf/755231/P39GTZsGFlZWfTt2xeDwVD2wYoyV/0BZ9JTzxZaLsynzEaFaa0XaK39tNZ+derUKdZr/3f8f4THhJNyJQWNJuVKCuEx4fzv+P9uq9u9e3eeeeYZOnTogJeXF6GhoaSnp9OjRw+ys7Px9vbmzTffpH379gAkJyfTpUsXDAYDYWFhTJ8+HcjbQ2TkyJFVsvPeHIoaFrp06VLOnj1LXFwc8fHx1KtXr8CQzXyBgYFs376dhg0b8uyzz/LVV/IoJF9SUhKenp7mDqNUBAx6DqtqBZf0sapmQ8Cg58wUkQATLumilHIF1mmt7/ovuLhLunSP7E7KlZTbyl0cXNgUuqk4YYpySmtN+/btGT58OCNGjADyhoWuX7+e1NRUPvzwQ7Zu3UrXrl357bffqF69Or6+vsal8U+cOEHDhg2xsrJi7ty5JCUlMXfuXDOeUfmRlJRESEgIiYnlYtCmyR2K3kr0sq9IP5dK9QecCRj0HA8FPGrusEpFRVnSxSSPwpRS/wW6AM5KqT+At7XWn5uibYDTV04Xq1yUP9+cPs/04ykkX8uioY01k91ceLJ+bePx/GGh48ePZ8aMGdja2uLq6kp4eDhjx47Fz88Pg8FA69atAXjggQfo1KkTnp6ePP7443h6ejJz5kysra1xdHSUO5Zb5OTkMGLECGJiYmjYsCHffvstjz/+OLNmzcLPz4/U1FT8/PxISkpi0aJFrF69mpycHBITE/nHP/7B9evXWbx4MTY2Nqxfv57atWvz2WefsWDBAq5fv07z5s1ZvHgx9vb2hIWFUaNGDWJjYzl9+jTvvfceoaGhpKSkMHDgQC5dukR2djaffPLJPa2PdTcPBTxaaRNJhWWOoWjFHW7cbUU37bnI87avbiu6FasdUTwXLlzQH330kdZa661bt+qePXveVzuRKee067Z4XW/LXuOX67Z4HZmSN/y7Kg7tfvvtt/XMmTOLPH7mzBndrl07bTAYihxyfa9+++03bWlpadypc8CAAXrx4sUFrvvZs2d1kyZNtNZaL1y4UDdr1kxfunRJnzlzRteoUcM4DHz8+PF6zpw5WmtdYPfNKVOm6Hnz5mmt83YMDQ0N1Tk5OfrAgQO6WbNmWuu83T+nTp2qtdY6OztbX7p0qUTnVRVRlYYbl7ZxvuOwtbQtUGZracs433FmiqhqMNUkxOnHU7ialVWgLCNXM/347Y83RZ6oqChat27N3r17TfKpvmnTpsYBDfeyNtWjjz5K9erVqVOnDk5OTvTq1QugwG6eiYmJBAQE4OXlxdKlSwusg9W3b18sLCxwd3c3Drbw9/dn4cKFhIeHs3//fqpXr17i8xLlU4VILD3dehLeMRwXBxcUChcHF8I7hhc6KkyYzq1zRS5fvkxoaCitW7dm8ODB+ZNjiYuLo3PnzrRt25bg4GBSUvISRpcuXXj99ddJGPUcV7/5mqxfDnJ+/N849+IzXHhtNL/ftGTOkiVL6NixI56envz888/k5ubSokULzp7NG/GTm5tL8+bNSU2tuMNIp02bRqtWrXjsscc4ciRvpehjx47Ro0cP2rZtS0BAAIcPHyY+Pp7XXnuN9evXGwePbNq0iQ4dOuDr68uAAQO4fPkykLfd8ttvv22c53P48GEAfvjhBwwGAwaDgZ49e2JllffUe+bMmcyfP58PP/yQ33//ndzcXIDbBkTcustnYbt5hoWFMX/+fPbv38/bb79d6DpY8NeunzLAouqoEIkF8pLLptBNJAxNYFPoJkkqZWDGjBk0a9aM+Ph4Zs6cWegy41lZWbz00ktERkYSFxfHsGHDmDJlirGNtLQ0vD/5Cvv+T5M+71/UDJ/JA//+GrvH+5C76K+7oStXrhATE8PHH3/MsGHDsLCwYMiQIcbdIjdv3oyPjw/OzhVzGGlcXBzLli1j7969rFy5kt27dwPwwgsv8OGHHxIXF8esWbMYPXo0BoOBd955h4EDBxIfH8+VK1eYOnUqmzdvZs+ePfj5+fH+++8b23Z2dmbPnj2MGjWKWbNmATBr1iw++ugj4uPjWb58ORYWFmzatImjR48yduxYRo4cSXZ2tnHzq/tZxyo9PR0XFxeysrKMf093cuLECerWrcuIESP429/+xp49e4r9nqJiqHQTJEXpyV9mHDAuM16zZk0SExPp1q0bkNdJ7OLiYnzNwIEDCXJzYex3W8lOOsaFCaMAULm5PNToQWO9p59+Gsj7VHvp0iXS0tIYNmwYffr0Yfz48XzxxRc8//zzZXWqJhcdHU2/fv2MS6337t2bzMxMYmJiGDBggLHetWvXbnvtjz/+yMGDB+nUqRMA169fp0OHDsbj/VtbwxxP2h5IYuV2IKE7nTp14pVXXmHw4MH4+/ujlDKuSrFu3Tpyc3OxsbFh+fLl/Pjjj3Tt2rXY5/Tuu+/y8MMP06RJE7y8vO4672vbtm0ywKKKkMQi7llhy4xrrfHw8GDXrl2FvsbBwYHO9WtzwrUebzVtTvUPFxU5KuxmSikaNWpEvXr12LJlCz/99NM9fSourpdffpkmTZowfvx4IG+CbaNGjfjPf/4DwD/+8Q+cnJyoVq0akyZNYvXq1bRs2dK4DW2XLl2MI6tuFhsby1dffcW8efOKPMfc3Fxq1qxpXD2gKFprunXrxn//+9/bD2ZdxWbrW1DtGpYWkH09E9aOZVKvefTs+R/Wr1/PgAED2Lx5M5999lmhq1Lkmzp1KpD3iCt/ojBQoD/m5mOjRo1i1KhRt7WzaNGiAj/nP7YbOnQoQ4cOveO5isqhwjwKE2XvXlYfaNWqFWfPnjUmlqysrAKduPnGdGpH/czLrLTNILajB70fqF6gXkREBAA7duwwbvMKMHz4cIYMGcJTTz1VKnvUd+zYkZiYGCDvF31qamqBuGJiYggODmbSpElA3v7mBw8evGu7fn5+BZJKYGAgq1atIiMjg/T0dNauXYu9vT1NmzZlxYoVQF4C2bdv321ttW/fnp07d/Lrr78CcPXq1b9Wfc5Ig+xbJvBmZXAs4g28vLyYOHEifn5+HD58mODgYL744gvjL/rk5GTOnDlzbxdKiGKQxFKBnTp1itDQ0DvW6dixY5HHfvnpNF++vpOPRm7hy9d38stPBecF3TxXZMKECYW2Ua1aNSIjI5k4cSI+Pj4YDAbjL+ri1KtVqxYdO3Zk5MiRfP75X1OgevfuzeXLl0vtMVinTp2McRw4cABPT0+qV6/OhQsXuHbtGocOHWLfvn2MGTOGmJgY1qxZw4QJEzAYDMYl0lesWIG3Rxvq127Ey33m8uXrO1n88UpCQkIACA8PZ/78+aSnp+Pk5IS/v79xpNfSpUv5/PPP8fHxwcPDw7jI5s3q1KnDokWLePrpp/H29qZ9+/bGTnpyC98gau73J/D09MTHxwc7Ozsef/zxIlelEMLUTDbzvjiKO/NemN4vP51m69LDZF/PNZZZVbPg0cGtaflwfTNGVlBsbCwvv/xyqa5W7Orqyvbt2/nuu+/QWpOcnEyHDh1wcnJi8uTJDBs2jNjYWObPn09YWBghISHGhN6lSxfcGrTmYedB7Pt1F1sSInkpZCbHziSw98z/2Lrje8LDw9m0aRNbt24lPT2dVq1acfr0aaytrUse/BxPuHjy9nKnRvBy5ZxpX5VVqZn3ovRNnDiRJk2aMHr0aCDvU3D16tVZuHAhiYmJHDhwgOeff57r16+Tm5vLN998Q4sWLXB0dOTy5ctorXnttdf47rvvUEoR0GwAHvUf4ZdT8ayP/QpH2xqkXEhiUVRrYhI33tYfYA4zZszgk08+KZW+lZvl37XExMTwyiuvkJycTExMDE5OTne848vXwMqH7Ou5NHJuwbn0vLu+nOxcLpz+a4Xlnj17YmNjg42NDXXr1uXPP/80DoQokaC3YO1YyLrpcZi1XV65EGYij8IqiEGDBhn7IQCWL19eYEe6Tz/9lHHjxhEfH09sbOxtv7RWrlxJfHw8+/btY/PmzSyL+oiLV84B8Me5X3my49+Z8tQXnE79g507d5bNSd3FpEmTOHHiBI888kipvk9+P8v+/fvx9PSkffv27Nq1i5iYGONIrDu5fiXvTwtlSW5ujrE8O+uv7++0v3qJeD8Fvebl3aGg8v7sNS+vXAgzkcRSQbRp04YzZ85w6tQp9u3bR61atQpsP9ChQwf+7//+j3/961+cOHECOzu7Aq/fsWMHTz/9NJaWltSrV4/WTdpw4mzeJL0mdVpRy7EOFsqCpg1b3XVWdmXTqVMn1q1bR+3atbG0tKR27dqkpaWxa9euAsN6ofABDfY1qhXarpW16QcbFMr7qbzHXuFpeX9KUhFmJomlAgkNDSUyMpKIiAgGDRpU4NgzzzzDmjVrsLOzIzg4mC1bthQ4fmtfWp1Gjlha5T3usrLMe9ZvVc2CBs1qme7TdDlwp/3Q83l5eRk3Ebu5zMnJ6bYJmYXtb+7VuSFW1Qr+V7K0sqBWfftSOKOytW3bNuMgBCHulfSxVCCDBg1ixIgRpKam8sMPPxSYTHf8+HHc3NwYO3Ysx48fJyEhocCkt8DAQP79738zdOhQzp8/T/zB3bz2yVusXfoDAI61bejQpxn7F1f8X4b5DkVvZdOC+cata9NTz7JpwXyAAqvhWlpacunSpQKvvXkuxs1zNzp16lRguPG2bdsA+OWh0+z69hjvDP4ax9o2jHx+EC0fHg/k9YfdrLIuXy9EPkks5UhCQgJRUVFcvHgRJycngoKC8Pb2Nh738PAgPT2dhg0b4uLiUuCRVUREBEuWLMHa2pr69evz1lsFO2/79evHrl278PHxQSnFe++9R6cnvMmyP0/iRWeG/t+NvoTFZXGmZeNO+6Gbepn1lg/XL5XRdFeuXOGpp57ijz/+ICcnhzfffJPmzZvzyiuvcPnyZZydnVm0aBEuLi78+uuvjBw5krNnz2JpacmKFStwc3MrMGjjjTfeYODAgWzbto3w8HCcnZ2N20IvWbIEpRQbNmxg/PjxODs74+vra/JzEpWfSYYbK6V6AB8AlsB/tNYz7lRfhhvfLiEhgbVr15J10yrA1tbW9OrVq0ByEfdu9qBeUNi/b6X4x7K1ZR/Qffjmm2/YsGEDn332GQAXL17k8ccf59tvv6VOnTpERESwceNGvvjiCx5++GEmTZpEv379yMzMJDc3l++++45PP/2UDRs2kJqair+/Pz/99BNHjhyhT58+HDhwgAYNGtCpUydmzpyJn58fLVq0YMuWLTRv3pyBAwdy9epV1q1bZ+YrIaDiDDcucR+LUsoS+Ah4HHAHnlZKuZe03aomKiqqQFKBvFnsUVFRZoqo4itq3/OKtB+6l5cXmzdvZuLEiURHR3Py5Enj2mwGg4GpU6fyxx9/kJ6eTnJyMv369QPA1tYWe3v72wZtdO7c2bgAZv7abxYWFsa13w4fPkzTpk1p0aIFSimGDBliztMXFZQpHoW1A37VWh8HUEotA/oAd1/3QhhdvHixWOXi7gIGPVegjwUq3n7oLVu2JC4ujvXr1zN58mS6detW6Npst/YR5bvTE4mihkCXhzlMomIzxaiwhsDNU3//uFEmiiF/bax7LRd391DAo3R/YQzVneuAUlR3rkP3F8ZUqG1sT506hb29PUOGDOHVV1/lp59+KnRttho1avDggw+yevVqIG+V5KtXrxIYGEhERAQ5OTmcPXuW7du3065duyLfr3Xr1vz222/GEW+FLnwpxF2Y4o6lsI83t31MUkq9ALwAFJh/IfIEBQUV2scSFBRkxqgqvnK/H3rCcoh6By7+AU4P5s2Yv2keyv79+5kwYQIWFhZYW1vzySefYGVlxdixY7l48SLZ2dmMHz8eDw8PFi9ezIsvvshbb72FtbU1K1asKHTQRv369f9aa+wWtra2LFiwgJ49e+Ls7Mwjjzwio9hEsZW4814p1QEI11oH3/h5MoDWenpRr5HO+8LdbVSYqGQSlhe+HIvMnBdFqCid96a4Y9kNtFBKNQWSgUHAMyZot8rx9vaWRFKVRL1TMKlA3s9R70hiERVaiROL1jpbKTUG2EjecOMvtNa3b8ghhCjo4h/FKxeigjDJBEmt9XpgvSnaEqLKcHqwiCXvTbDqsRBmJGuFCWEuQW/l9ancTJa8F5WAJBYhzEWWvBeVlKwVJoQ5eT8liURUOnLHIoQQwqQksQghhDApSSxCCCFMShKLEEIIk5LEIoQQwqQksQghhDApSSxCCCFMShKLEEIIk5LEIoQQwqQksQghhDCpEiUWpdQApdQBpVSuUqrcbz4jhBCi9JX0jiUR6A9sN0EsQgghKoESLUKptT4EoFRh294LUX6Eh4fj6OjIq6++au5QhKj0pI9FCCGESd01sSilNiulEgv56lOcN1JKvaCUilVKxZ49e/b+IxbiHk2bNo1WrVrx2GOPceTIEQDi4+Np37493t7e9OvXjwsXLnDmzBnatm0LwL59+1BK8fvvvwPQrFkzrl69SlhYGGPHjqVjx464ubkRGRlptvMSory7a2LRWj+mtfYs5Ovb4ryR1nqB1tpPa+1Xp06d+49YiHsQFxfHsmXL2Lt3LytXrmT37t0APPfcc/zrX/8iISEBLy8v/vnPf1K3bl0yMzO5dOkS0dHR+Pn5ER0dzYkTJ6hbty729vYApKSksGPHDtatW8ekSZPMeXpClGvyKExUStHR0fTr1w97e3tq1KhB7969uXLlCmlpaXTu3BmAoUOHsn173riTjh07snPnTrZv387rr7/O9u3biY6OJiAgwNhm3759sbCwwN3dnT///NMs51VePPHEE6SlpQHg6OhYaJ2wsLD7urNLSkri66+/Lkl4wsxKOty4n1LqD6AD8D+l1EbThCVEyRVnUElAQIDxLqVPnz7s27ePHTt2EBgYaKxjY2Nj/F5rbdJYK5r169dTs2bNUmlbEkvFV6LEorVepbV+UGtto7Wup7UONlVgouq4+dOvqQQGBrJq1SoyMjJIT09n7dq1ODg4UKtWLaKjowFYvHix8e4lMDCQJUuW0KJFCywsLKhduzbr16+nU6dOJo2ronjvvfeYN28eAC+//DJdu3YFICoqiiFDhuDq6kpqamqB12itGTNmDO7u7vTs2ZMzZ84Yj+3evZuOHTvi4+NDu3btSE9PJykpiYCAAHx9ffH19SUmJgaASZMmER0djcFgYM6cORw4cIB27dphMBjw9vbm6NGjZXQVxP2SPe+F2a1fv/6+Xndx7VrOzJlLdkoKVi4u1H15PE69egHg6+vLwIEDMRgMNGnSxPhI68svv2TkyJFcvXoVNzc3Fi5cCICrqyuA8Q7lkUce4Y8//qBWrVolPLuKKTAwkNmzZzN27FhiY2O5du0aWVlZ7Nixg4CAAHbs2HHba1atWsWRI0fYv38/f/75J+7u7gwbNozr168zcOBAIiIi8Pf359KlS9jZ2VG3bl2+//57bG1tOXr0KE8//TSxsbHMmDGDWbNmsW7dOgBeeuklxo0bx+DBg7l+/To5OTllfTlEcWmty/yrbdu2WlQsly9f1k888YT29vbWHh4eetmyZbpJkyb6tdde0/7+/trf318fPXpUa631mTNndP/+/bWfn5/28/PTO3bs0FprnZ6ersPCwrSnp6f28vLSkZGRWmutmzRpos+ePau11nrx4sXa399f+/j46BdeeEFnZ2fr7OxsPXToUO3h4aE9PT31+++/r9PWrNGHfAz6YKvWxq9DPgadtmaNeS5QJXP9+nXdtGlTfenSJR0UFKTHjh2rY2JidFBQkD5w4ECBvzMHBwettdbjxo3Tn3/+ubGNfv366RUrVuiEhATdsWPH294jLS1NDxkyRHt6emofHx9tZ2entdZ669atumfPnsZ6S5cu1e7u7nrGjBn6l19+Kc3TLveAWG2G39nF/ZLOe3FPNmzYQIMGDdi3bx+JiYn06NEDgBo1avDzzz8zZswYxo8fD8C4ceN4+eWX2b17N9988w3Dhw8H4N1338XJyYn9+/eTkJBgfLyS79ChQ0RERLBz507i4+OxtLRk6dKlxMfHk5ycTGJiIvv37+f555/nzJy56MzMAq/XmZmcmTO31K9FVWBtbY2rqysLFy6kY8eOBAQEsHXrVo4dO8ZDDz1U5OsK69fSWhdaPmfOHOrVq8e+ffuIjY3l+vXrhbb5zDPPsGbNGuzs7AgODmbLli33f2KiTEhiEffEy8uLzZs3M3HiRNatW0fnzp05deoUc+fOpWHDhsyePZsNGzZw/fp1Nm/ezJgxYzAYDPTu3ZtLly6Rnp7O5s2b+fvf/25s89bHTFFRUcTFxeHv74/BYCAqKorjx4/j5ubG8ePHeemll9iwYQM1atQgOyWl0DiLKhfFFxgYyKxZswgMDCQgIIBPP/0Ug8FQ5KCIwMBAli1bRk5ODikpKWzduhWA1q1bc+rUKeOQ7/T0dLKzs7l48SIuLi5YWFiwePFi4yOu6tWrk56ebmw3/9/A2LFj6d27NwkJCaV85qKkpI9F3JOWLVsSFxfH+vXrmTFjBv379yctLY0+ffrQqFEjxo0bh4uLC9WqVSM3N5ddu3ZhZ2dXoI2iPrnefHzo0KFMnz79tmP79u1j48aNfPTRRyxfvpzJLi5knzp1Wz0rF5eSn2wVcWXvGS5tTCIn7RqWNW2oEeyKQ5u6xuMBAQFMmzaNDh064ODggK2tbYHh17fq168fW7ZswcvLi5YtWxoHRlSrVo2IiAheeuklMjIysLOzY/PmzYwePZonn3ySFStW8Oijj+Lg4ACAt7c3VlZW+Pj4EBYWRmZmJkuWLMHa2pr69evz1ltvle6FESWmtBmGTfr5+enY2Ngyf19x/06dOkXt2rWxtbVl9erVLFq0iPj4eJo3b06PHj04d+4cH3/8MY0bN+bKlSuMGDGCyZMnA3mz3Q0GA5MmTSIzM5O5c+cCcOHCBWrVqoWrqyuxsbGcOXOGPn36sHPnTurWrcv58+dJT0/HwcGBatWqUaNGDeLj4wkLC+OHd98l5c23CjwOU7a2uLz7jrEDXxTtyt4zpK08is7KNZYpawtq9m9RILmI8kUpFae1LvcrycujMAFAyulv2bkzgKgtzdm5M4CU0wUXVti/f79xyOe0adN44403AMjOzmbu3LnMnj2byMhI9u/fj7+/P8uXL8fb2xt3d3c+/fRTAN544w0uXLiAp6cnPj4+xkcl+dzd3Zk6dSrdu3fH29ubbt26kZKSQnJyMl26dMFgMBAWFsb06dNx6tULl3ffwapBA1AKqwYNJKkUw6WNSQWSCoDOyuXSxiTzBCQqFbljEaSc/pbDh6eQm5thLLOwsKN162m41C96SThXV1eeeuopHBwciIqKMs5ij4qK4qOPPmLlypWlHru4P39Mii7y2IMzin7cJcxL7lhEhXH82KwCSQUgNzeD48dm3dPr85+Nl7W0tDQ+/vhjALZt20ZISIhZ4qiILGvaFKtciOKQxCLIvFb4SKqiyvMlJSVhb29PZmYmSUlJ/Prrr0DBGe2l6ebEIoqnRrAryrrgf39lbUGNYFfzBCQqFUksAlubokZSWRTZ51Lg9ba2LFy4kAEDBuDl5YWFhQUjR44snWBvMmnSJI4dO4bBYGDChAlcvnyZ0NBQWrduzeDBg43reUVFRdGmTRu8vLwYNmwY165dM77e3d0db29v4wZgZ8+e5cknn8Tf3x9/f3927txZ6udhDg5t6lKzfwvjHYplTRvpuBcmI30sotA+llvdS59LWUtKSiIkJITExES2bdtGnz59OHDgAA0aNKBTp07MnDkTPz8/WrRoQVRUFC1btuS5557D19eX5557jg4dOnD48GGUUqSlpVGzZk2eeeYZRo8ezSOPPMLvv/9OcHAwhw4dMvepCgFIH4uoQFzq96F162nY2jQAFGB5W53i9LmYS7t27XjwwQexsLDAYDCQlJTEkSNHaNq0KS1btgT+Wiq/Ro0a2NraMnz4cFauXGncc6WoyZ1CiHsnEyQFkJdc8u9GorY0L7TO3fpczO3mZe0tLS3Jzs4ucnl7Kysrfv75Z6Kioli2bBnz589ny5YtRU7uFELcu5LuxzJTKXVYKZWglFqllKpporiEGRXV51J0X4x53Lr0R2Fat25d6MCCy5cvc/HiRZ544gnmzp1LfHw8AN27d2f+/PnG1+eXCyHuXUkfhX0PeGqtvYFfgMklD0mYm1uzV7GwKPiJ3cLCDrdmr5ZpHN+cPo9fzAFctsbjF3OAb06fL3D8gQceoFOnTnh6ejJhwoRC2yhqYEF6ejohISF4e3vTuXNn5syZA8C8efOIjY29bXKnEOLemazzXinVDwjVWg++W13pvC//Uk5/y/Fjs8i8loKtjQtuzV4t0477b06f59UjJ8nI/evfp52FYlarRjxZv3aZxSFEeVJROu9N2ccyDIgwYXvCjG7uczGH6cdTCiQVgIxczfTjKZJYhCjn7ppYlFKbgfqFHJqitf72Rp0pQDaw9A7tvAC8ANC4ceP7ClZUHcnXsopVLoQoP+6aWLTWj93puFJqKBACBOk7PFfTWi8AFkDeo7BiximqmIY21vxRSBJpaGNthmiEEMVR0lFhPYCJQG+t9VXThCQETHZzwc6i4N4tdhaKyW7la2SaEOJ2Je1jmQ/YAN/f2MDpR6116a/lISq9/H6U6cdTSL6WRUMbaya7uUj/ihAVQIkSi9a68Jl0QpjAk/VrSyIRogKSJV2EEEKYlCQWIYQQJiWJRQghhElJYhFCCGFSkliEEEKYlCQWIYQQJiWJRQghhElJYhFCCGFSkliEEEKYlCQWIYQQJiWJRQghhElJYhFCCGFSkliEEEKYVEn3Y3lXKZWglIpXSm1SSjUwVWBCCCEqppLesczUWntrrQ3AOuCtkockhBCiIitRYtFaX7rpRwdAthwWQogqrsR9LEqpaUqpk8Bg5I5F3Idp06bh4eGBt7c3BoOBn376ieHDh3Pw4EFzhyaEuA9K6zvfZCilNgP1Czk0RWv97U31JgO2Wuu3i2jnBeAFgMaNG7c9ceLEfQctKo9du3bxyiuvsG3bNmxsbEhNTeX69es0aCDddULcSikVp7X2M3ccd3PXOxat9WNaa89Cvr69perXwJN3aGeB1tpPa+1Xp06dksYtKomUlBScnZ2xsbEBwNnZmQYNGtClSxdiY2MBcHR0ZMqUKfj4+NC+fXv+/PNPAP7880/69euHj48PPj4+xMTEALBkyRLatWuHwWDgxRdfJCcnxzwndxdpaWl8/PHH5g5DCJMr6aiwFjf92Bs4XLJwRFXTvXt3Tp48ScuWLRk9ejQ//PDDbXWuXLlC+/bt2bdvH4GBgXz22WcAjB07ls6dO7Nv3z727NmDh4cHhw4dIiIigp07dxIfH4+lpSVLly4t69O6J5JYRGVV0j6WGUqpRKVUAtAdGGeCmEQV4ujoSFxcHAsWLKBOnToMHDiQRYsWFahTrVo1QkJCAGjbti1JSUkAbNmyhVGjRgFgaWmJk5MTUVFRxMXF4e/vj8FgICoqiuPHj5flKd2zSZMmcezYMQwGA88//zxr1qwBoF+/fgwbNgyAzz//nDfeeAOA999/H09PTzw9PZk7d665whbirqxK8mKtdZGPvoS4V5aWlnTp0oUuXbrg5eXFl19+WeC4tbU1Silj3ezs7CLb0lozdOhQpk+fXqoxm8KMGTNITEwkPj6eZcuWER0dTe/evUlOTiYlJQWAHTt2MGjQIOLi4li4cCE//fQTWmsefvhhOnfuTJs2bcx8FkLcTmbei1J1ce1ajnYN4tBD7hztGsTFtWsLHD9y5AhHjx41/hwfH0+TJk3uqe2goCA++eQTAHJycrh06RJBQUFERkZy5swZAM6fP09FGCgSEBBAdHQ0Bw8exN3dnXr16pGSksKuXbvo2LEjO3bsoF+/fjg4OODo6Ej//v2Jjo42d9hCFEoSiyg1F9euJeXNt8g+dQq0JvvUKVLefKtAcrl8+TJDhw7F3d0db29vDh48SHh4+D21/8EHH7B161a8vLxo27YtBw4cwN3dnalTp9K9e3e8vb3p1q2b8dN/edawYUMuXLjAhg0bCAwMJCAggOXLl+Po6Ej16tW52+hNIcqTuw43Lg1+fn46f8SPqLyOdg3KSyq3sGrQgBZboswQUfly7tw5fH19jXdUYWFhbNmyhS1btnDu3DlCQ0MJDQ1lzpw57Nmzh7CwMH788Ufjo7DFixfLo7AqpqIMNy5RH4sQd5JdxJ1CUeWV0eq9yczceIRTaRk0qGnHhOBW9G3TEIAHHniATp064enpyeOPP05AQACbNm2iefPmNGnShPPnzxMQEACAr68vYWFhtGvXDoDhw4dLUhHlltyxiFJT1e9YVu9NZvLK/WRk/TWPxs7akun9vYzJRYjiqCh3LNLHIkpN3ZfHo2xtC5QpW1vqvjzePAGVsZkbjxRIKgAZWTnM3HjETBEJUTbkUZgoNU69egFwZs5cslNSsHJxoe7L443lld2ptIxilQtRWUhiEaXKqVevKpNIbtWgph3JhSSRBjXtzBCNEGVHHoUJUUomBLfCztqyQJmdtSUTgluZKSIhyobcsQhRSvI76IsaFSZEZSWJRYhS1LdNQ0kkosqRR2FCCCFMShKLEEIIk5LEIoQQwqTMMvNeKXUWKM0lZ52B1FJsv6KQ6yDXAOQa5KsM16GJ1rrcb8FrlsRS2pRSsRVh2YPSJtdBrgHINcgn16HsyKMwIYQQJiWJRQghhElV1sSywNwBlBNyHeQagFyDfHIdykil7GMRQghhPpX1jkUIIYSZVPrEopR6VSmllVLO5o6lrCmlZiqlDiulEpRSq5RSNc0dU1lRSvVQSh1RSv2qlJpk7njMQSnVSCm1VSl1SCl1QCk1ztwxmYtSylIptVcptc7csVQFlTqxKKUaAd2A380di5l8D3hqrb2BX4DJZo6nTCilLIGPgMcBd+BppZS7eaMyi2zgH1rrh4D2wN+r6HUAGAccMncQVUWlTizAHOA1oEp2JGmtN2mts2/8+CPwoDnjKUPtgF+11se11teBZUAfM8dU5rTWKVrrPTe+TyfvF2uVWxFTKfUg0BP4j7ljqSoqbWJRSvUGkrXW+8wdSzkxDPjO3EGUkYbAyZt+/oMq+Av1ZkopV6AN8JOZQzGHueR9wMw1cxxVRoVeNl8ptRmoX8ihKcDrQPeyjajs3ekaaK2/vVFnCnmPRZaWZWxmpAopq5J3rQBKKUfgG2C81vqSueMpS0qpEOCM1jpOKdXFzOFUGRU6sWitHyusXCnlBTQF9imlIO8R0B6lVDut9ekyDLHUFXUN8imlhgIhQJCuOmPL/wAa3fTzg8ApM8ViVkopa/KSylKt9Upzx2MGnYDeSqknAFughlJqidZ6iJnjqtSqxDwWpVQS4Ke1rugL0BWLUqoH8D7QWWt91tzxlBWllBV5gxWCgGRgN/CM1vqAWQMrYyrvU9WXwHmt9Xgzh2N2N+5YXtVah5g5lEqv0vaxCADmA9WB75VS8UqpT80dUFm4MWBhDLCRvA7r5VUtqdzQCXgW6Hrj7z/+xid3IUpVlbhjEUIIUXbkjkUIIYRJSWIRQghhUpJYhBBCmJQkFiGEECYliUUIIYRJSWIRQghhUpJYhBBCmJQkFiGEECb1/0Xa58ja3laMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,3))\n",
    "for i, word in enumerate(vocabs[:20]): #loop each unique vocab\n",
    "    x, y = get_embed(word)\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(word, xy=(x, y), xytext=(5, 2), textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c350eda7-ffac-4f9b-a52c-5ca0c25724a1",
   "metadata": {},
   "source": [
    "### 6. Try on CBOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "82df03b3-051f-4cc6-a411-e9e5c3c3ff12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "109"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#delete duplicate word define as vocabs, also include unknow as <UNK>.\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocabs  = list(set(flatten(corpus_tokenised)))\n",
    "vocabs.append('<UNK>') # to let '<UNK>' become the last.\n",
    "word2index = {v: idx+1 for idx, v in enumerate(vocabs)} #plus one to shift value in word index to assign <UNK>'s value to 0 without duplication with others.\n",
    "word2index['<UNK>'] = 0\n",
    "\n",
    "#Let check word to index's value\n",
    "word2index[\"cat\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b78fa9b0-c725-40ac-a799-e1ee533b4123",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'coast'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create index to word\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "index2word[50]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "116b4bae-9afc-4cde-b332-799b6e092cda",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Prepare train data\n",
    "\n",
    "I use same way but change the variable to not making myself get confused."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "eaf4b7b8-cd5b-4bf0-a60d-82f5a893acb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['(', 'The'],\n",
       " ['(', 'cat'],\n",
       " ['(', 'Felis'],\n",
       " ['(', 'catus'],\n",
       " ['Felis', 'cat'],\n",
       " ['Felis', '('],\n",
       " ['Felis', 'catus'],\n",
       " ['Felis', ')'],\n",
       " ['catus', '('],\n",
       " ['catus', 'Felis'],\n",
       " ['catus', ')'],\n",
       " ['catus', 'is'],\n",
       " [')', 'Felis'],\n",
       " [')', 'catus'],\n",
       " [')', 'is'],\n",
       " [')', 'a'],\n",
       " ['is', 'catus'],\n",
       " ['is', ')'],\n",
       " ['is', 'a'],\n",
       " ['is', 'domestic'],\n",
       " ['a', ')'],\n",
       " ['a', 'is'],\n",
       " ['a', 'domestic'],\n",
       " ['a', 'species'],\n",
       " ['domestic', 'is'],\n",
       " ['domestic', 'a'],\n",
       " ['domestic', 'species'],\n",
       " ['domestic', 'of'],\n",
       " ['species', 'a'],\n",
       " ['species', 'domestic'],\n",
       " ['species', 'of'],\n",
       " ['species', 'small'],\n",
       " ['of', 'domestic'],\n",
       " ['of', 'species'],\n",
       " ['of', 'small'],\n",
       " ['of', 'carnivorous'],\n",
       " ['small', 'species'],\n",
       " ['small', 'of'],\n",
       " ['small', 'carnivorous'],\n",
       " ['small', 'mammal'],\n",
       " ['carnivorous', 'of'],\n",
       " ['carnivorous', 'small'],\n",
       " ['carnivorous', 'mammal'],\n",
       " ['carnivorous', '.'],\n",
       " ['mammal', 'small'],\n",
       " ['mammal', 'carnivorous'],\n",
       " ['mammal', '.'],\n",
       " ['mammal', 'It'],\n",
       " ['.', 'carnivorous'],\n",
       " ['.', 'mammal']]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CBOW = []\n",
    "\n",
    "#for each corpus\n",
    "for sent in corpus_tokenised:\n",
    "    #for each sent [\"apple\", \"banana\", \"fruit\"]\n",
    "    for i in range(2, len(sent) - 2): # start from 1 to second last but we use window size of 2, then first centre word should be at index 2\n",
    "        center_word = sent[i]\n",
    "        outside_words = np.ravel([sent[i - 2:i],sent[i+1: i + 3]])  # window_size = 2\n",
    "        for o in outside_words:\n",
    "            CBOW.append([center_word, o])\n",
    "\n",
    "CBOW[:50] #shows only sample 50 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "34dd843a-d1a9-48d2-b249-a646f0814e70",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size, corpus):\n",
    "    \n",
    "    skipgrams = []\n",
    "\n",
    "    for sent in corpus:\n",
    "        for i in range(2, len(sent) - 2): #start from 2 to third last for window size = 2\n",
    "            centre_word = word2index[sent[i]]\n",
    "            outside_words = []\n",
    "            for j in np.ravel([sent[i - 2:i],sent[i + 1: i + 3]]):\n",
    "                outside_words.append(word2index[j])  #window_size = 2\n",
    "            for o in outside_words:\n",
    "                skipgrams.append([centre_word, o])\n",
    "                \n",
    "    #only get a batch, not the entire list\n",
    "    random_index = np.random.choice(range(len(skipgrams)), batch_size, replace=False)\n",
    "    #appending some list of inputs and labels\n",
    "    random_inputs, random_labels = [], []   \n",
    "    for index in random_index:\n",
    "        random_inputs.append([skipgrams[index][0]])  #center words, this will be a shape of (1, ) --> (1, 1) for modeling\n",
    "        random_labels.append([skipgrams[index][1]])\n",
    "        \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "39b5fc25-f9ec-422e-bc77-772ea20806a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 1)\n",
      "label=array([[ 278],\n",
      "       [ 171],\n",
      "       [ 799],\n",
      "       [1007],\n",
      "       [ 781],\n",
      "       [1019],\n",
      "       [ 589],\n",
      "       [ 405],\n",
      "       [ 272],\n",
      "       [ 721]])\n"
     ]
    }
   ],
   "source": [
    "input, label = random_batch(10, corpus_tokenised)\n",
    "\n",
    "print(f\"{input.shape}\")\n",
    "print(f\"{label=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53a9fdea-4d32-4044-960e-55526a7b564e",
   "metadata": {},
   "source": [
    "#### Unigram distribution\n",
    "\n",
    "$$P(w)=U(w)^{3/4}/Z$$\n",
    "\n",
    "Defining the probability of sampling negative words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "75c05bec-8848-4a2e-95c6-d7a45811d6c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_count = Counter(flatten(corpus_tokenised))\n",
    "# word_count # Showing word count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8a63b387-4320-4cbc-9001-2f9fb88d8ce8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2898"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_total_words = sum([c for w, c in word_count.items()])\n",
    "num_total_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "16ee2ada-d0d4-4c84-a0cd-c3f13a40e34e",
   "metadata": {},
   "outputs": [],
   "source": [
    "z = 0.001  #scaling up low frequency terms\n",
    "unigram_table = []\n",
    "\n",
    "# create loop for the P(w) formula\n",
    "for v in vocabs:\n",
    "    uw = word_count[v]/num_total_words\n",
    "    uw_alpha = uw ** (3/4)\n",
    "    uw_alpha_dividebyz = int(uw_alpha / z)\n",
    "    unigram_table.extend([v] * uw_alpha_dividebyz)\n",
    "    \n",
    "# Counter(unigram_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a88d713-ba21-4be3-a1e0-a613630fc982",
   "metadata": {},
   "source": [
    "#### Negative sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "fce0f1fd-2624-4501-90e1-b85dc3cea509",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "78441c1a-0eab-426c-b174-6fc0dadac3ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def negative_sampling(targets, unigram_table, k):\n",
    "    \n",
    "    batch_size = targets.shape[0]\n",
    "    neg_samples = []\n",
    "    \n",
    "    for i in range(batch_size):\n",
    "        #randomly pick k negative words from unigram_table\n",
    "        target_index = targets[i].item()  #looping each of the batch....\n",
    "        nsample = []\n",
    "        while len(nsample) < k:\n",
    "            neg = random.choice(unigram_table)\n",
    "            #if this word == target, skip this word\n",
    "            if word2index[neg] == target_index:\n",
    "                continue\n",
    "            nsample.append(neg)\n",
    "        #append this word to some list\n",
    "        neg_samples.append(prepare_sequence(nsample, word2index).reshape(1, -1))  #tensor[], tensor[]\n",
    "    return torch.cat(neg_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0293445e-a0b0-4230-b8d9-c022c364a332",
   "metadata": {},
   "source": [
    "According to quiz, Skipgram use centre to predict context but CBOW use context to predict centre, I just flip label and input on this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c33afbc4-6113-4a36-96ff-863cff27a99e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[873],\n",
       "        [956]]),\n",
       " array([[645],\n",
       "        [967]]))"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2\n",
    "label_batch, input_batch = random_batch(batch_size, corpus_tokenised)\n",
    "\n",
    "input_batch, label_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ddc16910-6a7d-4f0e-8915-2ec80aa36c97",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Tensor\n",
    "\n",
    "input_batch = torch.LongTensor(input_batch)\n",
    "label_batch = torch.LongTensor(label_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "b56a60e6-5b6a-428b-b9ef-861af47a61b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_neg = 10  # I will try 10 as in the paper.\n",
    "neg_samples = negative_sampling(label_batch, unigram_table, num_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "8436ba74-2a96-4cda-87a3-48b7773d2930",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 10])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neg_samples.shape # shape is (batch_size, num_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd90cc2-54e1-44c6-ae57-d17e1c2bf8e5",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "aaf3c4cf-7bce-4f0f-a0a2-d6c00e12a09a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get inspired from https://github.com/jeffchy/pytorch-word-embedding/blob/master/CBOW.py\n",
    "\n",
    "class CBOW(nn.Module):\n",
    "    \n",
    "    def __init__(self, voc_size, emb_size):\n",
    "        super(CBOW, self).__init__()\n",
    "        self.embedding_centre_word  = nn.Embedding(voc_size, emb_size)\n",
    "        self.embedding_outside_word = nn.Embedding(voc_size, emb_size)\n",
    "        self.logsigmoid = nn.LogSigmoid()\n",
    "        \n",
    "    def forward(self, centre_words, outside_words, negative_words):\n",
    "        \n",
    "        centre_embed  = self.embedding_centre_word(centre_words)    #(batch_size, 1, emb_size)\n",
    "        outside_embed = self.embedding_outside_word(outside_words)  #(batch_size, 1, emb_size)\n",
    "        neg_embed     = self.embedding_outside_word(negative_words) #(batch_size, k, emb_size)\n",
    "        \n",
    "        uovc          =  outside_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, 1)\n",
    "        ukvc          = -neg_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, k)\n",
    "        ukvc_sum      =  torch.sum(ukvc, 1).view(-1, 1) #(batch_size, 1)\n",
    "        \n",
    "        loss = self.logsigmoid(uovc) + self.logsigmoid(ukvc_sum)  #(batch_size, 1) + (batch_size, 1)\n",
    "                \n",
    "        return -torch.mean(loss)  #scalar, loss should be scalar, to call backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a38113-0b9f-4960-82af-c3dceef3a6cc",
   "metadata": {},
   "source": [
    "I flip label and input on this step also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "cf5c8c24-2957-4129-8b57-7adf607b78b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "label, input = random_batch(batch_size, corpus_tokenised)\n",
    "input_tensor = torch.LongTensor(input)  \n",
    "label_tensor = torch.LongTensor(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "023182d0-a3d4-44bd-be17-8ce250b2f065",
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_size = 10 # I will try on 10.\n",
    "voc_size = len(vocabs)\n",
    "model = CBOW(voc_size, emb_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "796b685e-8a35-4365-8695-11a6305b1e6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_tensor = negative_sampling(label_tensor, unigram_table, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "da3857c1-2e46-4f65-aba6-5acc60ffed49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 1]), torch.Size([2, 1]))"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_tensor.shape, label_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "967c6ae4-1e50-4ba3-8692-a87ac38d3f88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0043, grad_fn=<NegBackward0>)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = model(input_tensor, label_tensor, neg_tensor)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f3ea3b-5555-40d4-817b-151bb6df1f97",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "23c8c9c4-9db0-4dc0-8783-954d6b3a8005",
   "metadata": {},
   "outputs": [],
   "source": [
    "voc_size   = len(vocabs)\n",
    "batch_size = 2\n",
    "emb_size   = 10\n",
    "model      = CBOW(voc_size, emb_size)\n",
    "\n",
    "optimizer  = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "6002557e-879b-4f21-95fd-2f6c57a24a65",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1000 | Loss: 12.842160 | Time: 0m 0s\n",
      "Epoch 2000 | Loss: 1.380744 | Time: 0m 0s\n",
      "Epoch 3000 | Loss: 0.778933 | Time: 0m 0s\n",
      "Epoch 4000 | Loss: 9.280187 | Time: 0m 0s\n",
      "Epoch 5000 | Loss: 33.798153 | Time: 0m 0s\n",
      "Process Time: 3m 42s\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 5000\n",
    "startp = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "    start = time.time()\n",
    "    #get random batch\n",
    "    input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "    input_batch = torch.LongTensor(input_batch)\n",
    "    label_batch = torch.LongTensor(label_batch)\n",
    "    neg_batch   = negative_sampling(label_batch, unigram_table, 5)    \n",
    "    \n",
    "    #loss = model\n",
    "    loss = model(input_batch, label_batch, neg_batch)\n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "    #backpropagate\n",
    "    loss.backward()\n",
    "    \n",
    "    #update alpha\n",
    "    optimizer.step()\n",
    "    \n",
    "    #print epoch loss\n",
    "    if (epoch + 1) % 1000 == 0:\n",
    "        print(f\"Epoch {epoch+1} | Loss: {loss:.6f} | Time: {epoch_mins}m {epoch_secs}s\")\n",
    "endp = time.time()\n",
    "mins, secs = epoch_time(startp, endp)\n",
    "print(f\"Process Time: {mins}m {secs}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4cb8914-d820-411a-8c2d-ffd33ff8183e",
   "metadata": {},
   "source": [
    "#### Plot the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "aae2adce-f016-4ac0-943a-e3df52ee5f0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([109])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat = torch.LongTensor([word2index['cat']])\n",
    "cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "21feb9e6-dfb1-44a0-af9c-87e339611bc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2046, -1.2338, -0.2762,  1.8821,  1.3368,  0.1162,  0.4612, -0.3049,\n",
       "          0.0875, -0.7649]], grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_centre_embed = model.embedding_centre_word(cat)\n",
    "cat_outside_embed = model.embedding_outside_word(cat)\n",
    "\n",
    "cat_embed = (cat_centre_embed + cat_outside_embed) / 2\n",
    "cat_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7875886d-4889-47cc-a54e-fe4f1e9961a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embed(word):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except:\n",
    "        index = word2index['<UNK>']\n",
    "    \n",
    "    word = torch.LongTensor([index])\n",
    "\n",
    "    center_embed  = model.embedding_centre_word(word)\n",
    "    outside_embed = model.embedding_outside_word(word)\n",
    "    \n",
    "    embed = (center_embed + outside_embed) / 2\n",
    "    \n",
    "    return  embed[0][0].item(), embed[0][1].item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "740268de-b405-4884-af20-27c507df71a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-0.20457780361175537, -1.2338156700134277)\n"
     ]
    }
   ],
   "source": [
    "print(get_embed('cat'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "fd114924-2e93-4871-80fc-8ed7a9ccfc91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAADFCAYAAAChMA4eAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA5iklEQVR4nO3deXxNd/748dcniwSJoHZRRC3NcnOTSCRIpGKrWGobTBdh1KBazGhptaqqrSnfUtVpfzrKtDUjqqq1TMeIqJAUiQSxb7FEVEhDRBKSfH5/RG6FBJHlZnk/H4883Hvuuee8z8V953yW90dprRFCCCEKY2HuAIQQQlRckiSEEEIUSZKEEEKIIkmSEEIIUSRJEkIIIYokSUIIIUSRJEkIUUxKqbpKqYm3HwcqpTaU0Xm2KaU6lsWxhXhYkiSEKL66wMTSOJBSyqo0jiNEWZEkIUTxzQPaKKXigPmAnVJqjVLqiFJqpVJKASilvJRSPyulYpRS/1VKNb29fZtS6n2l1M/A5KL2u+05pVSkUipeKeWjlLJQSh1XSjW8fSwLpdQJpVSD8v0IRHWhzDHjukGDBrpVq1blfl4hSkNWVhYnTpzAxcWFtLQ002Nra2uOHj2Ko6MjtWvX5ujRo7Rp0wZra2tSUlK4du0arVq14ujRo9SsWZPHH38crfV997O1taVly5akpaVx9uxZXFxcuHDhApaWljRu3Jhr166RnJxMmzZtzP2xiHIQExNzWWvdsFxPqrUu9x8vLy8tRGV1+vRp7eLiorXWOjw8XPfo0cP02vjx4/XXX3+tDxw4oO3t7bW7u7t2d3fXrq6uumfPnlprrbt166a3bdumtdYP3C8sLMx07BYtWujffvtNnz17Vnt4eGittR4+fLhev359uVy3MD8gWpfz97W0hwpRQjY2NqbHlpaWZGdno7XGxcWFqKioQt9Tu3ZtgAfud7vlqsDzFi1a0LhxY7Zu3cquXbtYuXJlKV2JEPeSPgkhisne3p60tLT77tO+fXuSk5NNX/63bt3i4MGDxd4vNDQUgB07duDg4ICDgwMAY8eO5bnnnuMPf/gDlpaWpXJdQhRG7iSEKMSxXReJ+uEk11OysKtvg9/ANrTr1ASAxx57jC5duuDq6krNmjVp3LjxPe+vUaMGa9as4ZVXXuHq1atkZ2czZcoUXFxcirVfvXr16Ny5M9euXePLL780vW/AgAGMHj2a0aNHl+GnIISZOq47duyoo6Ojy/28QjyMY7suEr7yCNk3c03brGpY8NSzHUyJwtyio6OZOnUqERER5g5FlCOlVIzWulznzkhzkxB3ifrhZIEEAZB9M5eoH06aKaKC5s2bx5AhQ/jggw/MHYqoBiRJCHGX6ylZxdpe3mbMmMGZM2fo2rWruUMR1YAkCSHuYlffpljbhajKJEkIcRe/gW2wqlHwv4ZVDQv8BsqENVH9yOgmIe6S3zld1OgmIaoTSRJCFKJdpyaSFIRAmpuEEELchyQJIYQQRZIkIYQQokiSJIQQQhRJkoQQQogiSZIQQghRJEkSQgghilRqSUIpZamUilVKbSitYwohhDCv0ryTmAwcLsXjCSGEMLNSSRJKKUcgGPhHaRxPCCFExVBadxKLgNeA3AfsJ4QQohIpcZJQSvUDLmmtYx6w3zilVLRSKjo5ObmkpxVCCFEOSuNOogswQCmVAKwCuiulvrl7J631Uq11R611x4YNG5bCaYUQQpS1EicJrfXrWmtHrXUrYASwVWv9XIkjE0IIYXYyT0IIIUSRSnU9Ca31NmBbaR5TCCGE+cidhKhQ+vbtS2pqqrnDEELcJivTiQpl06ZN5g5BCHEHuZMQxZaenk5wcDDu7u64uroSGhpKq1atmD59Oj4+Pvj4+HDixAkAkpOTGTJkCN7e3nh7e7Nz504Arl+/zujRo3Fzc8NgMPDdd98B0KpVKy5fvgzAN998g4+PD0ajkT//+c/k5OSQk5NDSEgIrq6uuLm5sXDhQvN8CEJUE3InIYrtp59+olmzZmzcuBGAq1evMn36dOrUqcPu3bv56quvmDJlChs2bGDy5MlMnTqVrl27cvbsWXr37s3hw4d59913cXBw4MCBAwD89ttvBc5x+PBhQkND2blzJ9bW1kycOJGVK1fi4uJCYmIi8fHxANI0JUQZkyQhis3NzY1p06Yxffp0+vXrh7+/PwAjR440/Tl16lQAtmzZwqFDh0zvvXbtGmlpaWzZsoVVq1aZtterV6/AOcLCwoiJicHb2xuAjIwMGjVqRP/+/Tl16hQvv/wywcHB9OrVq0yvVYjqTpqbRLG1a9eOmJgY3NzceP3115kzZw4ASinTPvmPc3NziYqKIi4ujri4OBITE7G3t0drXWD/u2mtGTVqlOl9R48eZfbs2dSrV499+/YRGBjIp59+ytixY8v2Ykvozo54Ozu7QvcJCQlhzZo1xT52QkIC//rXv0oSnhAPJElCFNuFCxeoVasWzz33HNOmTWPv3r0AhIaGmv708/MDoFevXixZssT03ri4uEK3393cFBQUxJo1a7h06RIAKSkpnDlzhsuXL5Obm8uQIUN49913TeeuqDZt2kTdunXL5NiSJER5kCRRDcyePZsFCxYU6z1JF39g505/wrY+wc6d/iRd/MH02oEDB0wdyu+99x5vvvkmAFlZWXTq1ImPP/7Y1KG8ePFioqOjMRgMODs78/nnnwPw5ptv8ttvv+Hq6oq7uzvh4eEFzu/s7MzcuXPp1asXBoOBnj17kpSURGJiIoGBgRiNRkJCQvjggw9K8tGU2IcffsjixYsBmDp1Kt27dwfymsuee+65Ah3x+bTWTJo0CWdnZ4KDg02JEGDPnj107twZd3d3fHx8SEtLIyEhAX9/fzw9PfH09CQyMhKAGTNmEBERgdFoZOHChRw8eND092IwGDh+/Hg5fQqiStNal/uPl5eXFuXn7bff1vPnz3/o/S8krdNbw130ljAn08/WcBd9IWldke9p2bKlTk5OLo1wK5WoqCg9dOhQrbXWXbt21d7e3vrmzZt69uzZ+vPPPy/wudSuXVtrrfV3332ne/ToobOzs3ViYqJ2cHDQ3377rc7KytKtW7fWu3fv1lprffXqVX3r1i2dnp6uMzIytNZaHzt2TOf//wkPD9fBwcGmWCZNmqS/+eYbrbXWWVlZ+saNG+XzIYhyA0Trcv6+ljuJKuq9996jffv29OjRg6NHjwJ5TT2+vr4YDAYGDRrEb7/9xqVLl/Dy8gJg3759KKWI3DmX3NwMnn/uLJmZuXz4t0ssXnyOXj2fx8nJ6ZHaz6sqLy8vYmJiSEtLw8bGBj8/P6Kjo4mIiDB16N9t+/btjBw5EktLS5o1a2a6+zh69ChNmzY1ddbXqVMHKysrbt26xYsvvoibmxvDhg0rMBDgTn5+frz//vv87W9/48yZM9SsWbNsLlpUK5IkqqCYmBhWrVpFbGwsa9euZc+ePQC88MIL/O1vf2P//v24ubnxzjvv0KhRIzIzM7l27RoRERF07NiRmJiz/PrrLerWtcTWNu+fSMqVHD5a2JANGzYwY8aMe86ZkJBAgwYNyvU6KwJra2tatWrF8uXL6dy5M/7+/oSHh3Py5EmefPLJIt9XWKe9LqIzf+HChTRu3Jh9+/YRHR3NzZs3Cz3mH//4R3788Udq1qxJ79692bp1K6mpqfz9738HYNu2bfTr1+8Rr1RUV5IkqqCIiAgGDRpErVq1qFOnDgMGDCA9PZ3U1FS6desGwKhRo9i+fTsAnTt3ZufOnWzfvp033niDgwctOLA/Ezc3W9Mxu3SpRa2azXB2dubXX381y3VVVAEBASxYsICAgAD8/f35/PPPMRqNRY7eCggIYNWqVeTk5JCUlGTqj+nQoQMXLlwwJfW0tDSys7O5evUqTZs2xcLCgq+//pqcnBwA7O3tSUtLMx331KlTODk58corrzBgwAD2799fIEkI8SgkSVRR9xteejd/f38iIiI4c+YMAwcO5Pz5Ohw8mI3B8HuSqGFjg1ObaUDeb7zVSXrsJZLm7eb8jAiS5u0mPfZSgdf9/f1JSkrCz8+Pxo0bY2trW2RTE8CgQYNo27Ytbm5uTJgwwZS4a9SoQWhoKC+//DLu7u707NmTzMxMJk6cyD//+U98fX05duwYtWvXBsBgMGBlZYW7uzsLFy4kNDQUV1dXjEYjR44c4YUXXmDGjBmcPHkSo9HIq6++yvXr1xk6dCgdOnTg2WefNf1dhoWF4eHhgZubG2PGjCErKwvI6xx3dnbGYDAwbVre339Rs+hFFVXenSBaOq7LXExMjHZzc9M3btzQ165d00888YSeP3++NhgMevv27VrrvM7sKVOmaK21Pn36tG7RooV+9tlntdZaP/3007pZswZ60yZfvSWsjX766UZ66RevmY6f3wFbHVzf+6s+/+YOfW76dtPP+Td36Ot7fzV3aA/l9OnT2sXFRWud19Fdp04dfe7cOZ2Tk6N9fX11RESEzsjI0I6Ojvro0aNaa62ff/55vXDhQn3lyhXdrl07nZubq7XW+rffftNaaz1y5EgdERGhtdb6zJkzukOHDuV/YdUUZui4lhnXldDV9eu5tHAR2UlJWDVtSqOpU3Do39/0uqenJ8OHD8doNNKyZUvTb7X//Oc/GT9+PDdu3MDJyYnly5cDefWSIK8ZBKBr166cP3+ep5+OAuDrr0KoV9e7HK+w4rj23wT0rYJLt+tbuVz7bwK1PRqZKapH5+Pjg6OjIwBGo5GEhATs7e1p3bo17dq1A/KaIj/99FMmTZqEra0tY8eOJTg42NSfUdQsent7+/K/IFHmJElUMlfXryfprVnozEwAsi9cIOmtWQAFEsXMmTOZOXPmPe//5ZdfCj3u2bNnTY/feOMN3njjDdPzFStWFNj3+vXrjxx/ZZOTmlWs7RWdjY2N6bGlpSXZ2dlFNh9aWVmxe/duwsLCWLVqFUuWLGHr1q2mWfQyeqp6kD6JSubSwkWmBJFPZ2ZyaeEi8wRUxVnWtSnW9orm7s7twnTo0IGEhART5d6vv/6abt26cf36da5evUrfvn1ZtGhRkbPl87eLqknuJCqZ7KSkYm0XJVOndytS1x4v0OSkrC2o07uV+YK6y3cXU/jgVBKJWbdobmPN605NGdKkPgCPPfYYXbp0wdXVlZo1a9K4ceN73m9ra8vy5csZNmwY2dnZeHt7M378eFJSUhg4cCCZmZlorQvMon/ppZcwGAxkZ2cTEBBgmkkvqh5V1K1mWerYsaOOjo4u9/NWBce7B5F94cI9262aNaPt1jAzRFT1pcde4tp/E8hJzcKyrg11ereqMP0R311MYdrRc2Tk/v7/uKaFYkH7FqZEIaoOpVSM1rpjeZ5T7iQqmUZTpxTokwBQtrY0mjrFfEFVcbU9GlWYpHC3D04lFUgQABm5mg9OJUmSEKVCkkQlk985fb/RTaL6SMy6VaztQhSXJIlKyKF/f0kKAoDmNtacLyQhNLexNkM0oiqS0U2iSqou5Shed2pKTYuCs+trWihed2pqpohEVVPiJKGUaqGUCldKHVZKHVRKTS6NwIQoieqSJIY0qc+C9i1wtLFGAY421tJpLUpVadxJZAN/1Vo/CfgCLymlnEvhuEI8sjtrFo0ePZoff/wRyKubNGbMGACWLVtmWjDpo48+wtXVFVdXVxYtWmSusB/JkCb1ie7sQtJTRqI7u0iCEKWqxElCa52ktd57+3EacBhoXtLjClES8+bNo02bNsTFxdG7d28iIiIASExMNJWU2LFjB/7+/sTExLB8+XJ27drFL7/8whdffEFsbKw5wxeiwijVPgmlVCvAA9hVyGvjlFLRSqno5OTk0jytEPeVX+X20KFDODs707hxY5KSkoiKiqJz587s2LGDQYMGUbt2bezs7Bg8eLApqQhR3ZXa6CallB3wHTBFa33t7te11kuBpZA3ma60zivEgzRv3pzffvuNn376iYCAAFJSUli9ejV2dnbY29tXu9LnQhRHqdxJKKWsyUsQK7XWa0vjmEKUxN01i/z8/Fi0aJFpYaAFCxaYquMGBASwbt06bty4QXp6Ot9///1914MQojop8Z2EylvdZhlwWGv9UclDEuLB1sUmMv+/R7mQmkGzujV5tXd7nvH4vSvszppFTz/9NP7+/mzevJknnniCli1bkpKSYkoEnp6ehISE4OPjA8DYsWPx8PAwy3UJUdGUuHaTUqorEAEcAPKroL2htd5U1HukdpMoiXWxiby+9gAZt3JM22paW/LBYLcCiUKIqqZS1m7SWu8AHn6tTCFKaP5/jxZIEAAZt3KY/9+jkiSEKGUy41pUOhdSM4q1XQjx6CRJiEqnWd3CV0QrarsQ4tFJkhCVzqu921PT2rLAtprWlrzau72ZIhKi6pIqsKLSye93uN/oJiFE6ZAkISqlZzyaS1IQohxIc5MQQogiSZIQQghRJEkSQohqo1WrVly+fNncYVQqkiSEEEIUSZKEEKLEvvrqKwwGA+7u7jz//POcOXOGoKAgDAYDQUFBnD17FoCQkBAmTJjAU089hZOTEz///DNjxozhySefJCQkxHQ8Ozs7pk+fjpeXFz169GD37t0EBgbi5ORkWkAqMzOT0aNH4+bmhoeHB+Hh4QCsWLGCwYMH06dPH9q2bctrr712T7xvvfUWH3/8sen5zJkzWbx4cRl+QpWY1rrcf7y8vLQQomqIj4/X7dq108nJyVprra9cuaL79eunV6xYobXWetmyZXrgwIFaa61HjRqlhw8frnNzc/W6deu0vb293r9/v87JydGenp46NjZWa601oDdt2qS11vqZZ57RPXv21Ddv3tRxcXHa3d1da631ggULdEhIiNZa68OHD+sWLVrojIwMvXz5ct26dWudmpqqMzIy9OOPP67Pnj2rtda6efPmul+/fvr06dPaw8NDa611Tk6OdnJy0pcvX9Zaa+3n51fmn9mjAqJ1OX9fy52EEKJEtm7dytChQ2nQoAEA9evXJyoqij/+8Y8APP/88+zYscO0f//+/VFK4ebmRuPGjXFzc8PCwgIXFxcSEhIAqFGjBn369AHAzc2Nbt26YW1tjZubm2mfHTt28PzzzwPQoUMHWrZsybFjxwAICgrCwcEBW1tbnJ2dOXPmDABWVlYsX76cVq1a8dhjjxEbG8vmzZvx8PDgscceAyAyMrJsP7BKRpKEEKJEtNbkrRhQtDtft7GxAcDCwsL0OP95dnY2ANbW1qb33Lnfnfvo+1SwtrGxYfr06fz973/H0tKS7OxsZs+ezbVr10wl4nv37s3TTz/NyJEjiYmJ4fjx40BeU1f+8V999VVcXV1xc3MjNDQUgG3bthEYGMjQoUPp0KEDzz77bJVeuEqShBCiRIKCgli9ejVXrlwBICUlhc6dO7Nq1SoAVq5cSdeuXUv9vAEBAaxcuRKAY8eOcfbsWdq3/700y4gRI0xf7ACrV6+mRo0apuenTp1CKUW9evU4dOgQjo6OBY6/du1a4uLi2LdvH1u2bOHVV18lKSkJgNjYWBYtWsShQ4c4deoUO3fuLPXrqygkSQgh7mtdbCJd5m2l9YyNdJm3lXWxiQVed3FxYebMmXTr1g13d3f+8pe/sHjxYpYvX47BYODrr78u0ElcWiZOnEhOTg5ubm4MHz6cFStWFLgz8fDw4NKlS2RmZnLixAnq1auHldXvRSa6du1KVlYWjz/+OOfPn6dmzYIFInfs2MHIkSOxtLSkcePGdOvWjT179gDg4+ODo6MjFhYWGI1GUxNYVVTiRYcehSw6JETlUNkXeHrrrbdo2LAhFy9epGnTpvTv359+/foRHx9Pbm4uLi4uDB06lJUrV/KPf/yD7t27Y2dnx/Xr15kyZQoGg4ExY8YAeX0rw4YNo06dOixYsIANGzYAMGnSJDp27FhgdFZZMceiQ3InIYQo0v0WeKoMRowYwapVq1izZg1Dhw41bT906BAtW7bk6aef5t1332XAgAHs37+/wHsDAgIIDQ0lJyeH5ORktm/fblritjqRAn9CiCJV5AWe9u/fT1hYGFevXsXBwcE0L+NOLi4upKWl0bx5c5o2bWpqFnJ2dmbixIl88803GI1GmjRpwqxZswq8d9CgQURFReHu7o5Sig8//JAmTZpw5MiR8rrECkGam4QQReoybyuJhSSE5nVrsnNGdzNElGf//v2sX7+eW7dumbZZW1vTv3//exJFVSLNTUKICqWiLvAUFhZWIEEA3Lp1i7CwMDNFVHVJc5MQokgVdYGnq1evFmu7eHSSJIQQ91URF3hycHAoNCE4ODiYIZqqrVSam5RSfZRSR5VSJ5RSM0rjmEIIUZSgoCCsra0LbLO2tiYoKMhMEVVdJb6TUEpZAp8CPYHzwB6l1I9a60MlPbYQQhQmv3P6QaObRMmVRnOTD3BCa30KQCm1ChgISJIQQpQZg8EgSaEclEZzU3Pg3B3Pz9/eJoQQopIrjSRRWPnHeyZfKKXGKaWilVLRycnJpXBaIYQQZa00ksR5oMUdzx2BC3fvpLVeqrXuqLXu2LBhw1I4rRBCiLJWGkliD9BWKdVaKVUDGAH8WArHFUIIYWYlThJa62xgEvBf4DCwWmt9sKTHFaIiuXDhQoECcUJUF6UyT0JrvUlr3U5r3UZr/V5pHFOIiqRZs2asWbPG3GE8tKlTp7Jo0SLT8969ezN27FjT87/+9a/MmTOHefPmAbBu3ToOHfp9QGJgYCCF1VeLjo7mlVdeKbvARYUjtZuEuEv+spf5Zs+ezf/93//h6uoKwIoVKxg8eDB9+vShbdu2vPbaa6Z9N2/ejJ+fH56engwbNozr16+Xe/wAnTt3Nq3VnJuby+XLlzl48Pcb/MjISHr37s2MGXlzX+9OEkXp2LEjixcvLpugRYUkSUKIuxS27KW3t3eBfeLi4ggNDeXAgQOEhoZy7tw5Ll++zNy5c9myZQt79+6lY8eOfPTRR+UdPgBdunQxJYmDBw/i6uqKvb09v/32G1lZWRw+fJh9+/YxadIkIiMj+fHHH3n11VcxGo2cPHkSgG+//RYfHx/atWtHREQEkLe+c79+/YC85DlmzBgCAwNxcnKS5FFFSe0mIe6Sv+zlhQsXSE5Opl69ejz++OMF9gkKCjLVCXJ2dubMmTOkpqZy6NAhunTpAsDNmzfx8/Mr9/ghr3nMysqKs2fPEhkZiZ+fH4mJiURFReHg4IDBYDCt99y5c2cGDBhAv379CvS7ZGdns3v3bjZt2sQ777zDli1b7jnPkSNHCA8PJy0tjfbt2zNhwoR7ymWIyk2ShBCFGDp0KGvWrOHixYuMGDHintfvXEvZ0tKS7OxstNb07NmTf//73+UZapHy7yYiIyP5y1/+QmJiIpGRkTg4ONC5c+cHvn/w4MEAeHl5FbmGc3BwMDY2NtjY2NCoUSN+/fVXHB0dS/MyhJlJc5MQhShq2cv78fX1ZefOnZw4cQKAGzducOzYsbIM877y+yUOHDiAq6srvr6+REVFERkZabrbuZ/8RJifBO+3z4P2E8Uze/ZsFixYUOz3KaXmKKV6lGYskiREtZMee4mkebs5PyOCpHm7SY+9dM8+dy97+TAaNmzIihUrGDlyJAaDAV9fX7MuddmlSxc2bNhA/fr1sbS0pH79+qSmphIVFXVPM5i9vT1paWlmilSUFq31LK31ve2CJSDNTaJaSY+9ROra4+hbuQDkpGaRuvY4ALU9GhXY98CBA6bHrVq1Ij4+HoCQkBBCQkJMr23YsMH0uHv37uzZs6eswjc5HBFOxKqvSLtyGfvHGuA/4gWe9H+qwD5ubm5cvnyZP/7xjwW2Xb9+nQYNGhTYd8SIEbz44ossXry4Ug31rUree+89vvrqK1q0aEHDhg3x8vLi5MmTvPTSSyQnJ1OrVi0AW6WUA7APcNJa5yqlagFHASfgC2CD1nqNUsob+BioDWQBQcANYB4QCNgAn2qt/9/94pI1rkW1kjRvNzmpWfdst6xrQ9MZPmaIqPgOR4SzeekSsm/+fh1WNWzoNW7SPYlCVA4xMTGEhISwa9cusrOz8fT0ZPz48fznP//h888/p23btuzatQtfX980rXUdpdQPwCKtdbhSajjQU2s9Vim1AthAXtWLI8BwrfUepVQd8hLEGKCR1nquUsoG2AkM01qfLio2uZMQ1UphCeJ+2yuiiFVfFUgQANk3s4hY9ZUkiUoqIiKCQYMG5d8tMGDAADIzM4mMjGTYsGF37po/dCwUGA6Ek1cK6e8U1B5I0lrvAdBaXwNQSvUCDEqp/I42B6AtUGSSkD4JUa1Y1rUp1vaKKO3K5WJtN4f8UWFt2rTB2dmZvn37snTpUtMciweZNWtWoUNu8909+e9B+1cGShUsqJ2bm0vdunWJi4sz/QD5MyJ/BJ5WStUHvICtdx+OQqpx397+stbaePuntdZ68/3ikiQhqpU6vVuhrAv+s1fWFtTp3co8AT0C+8caFGt7edNaM2jQIAIDAzl58iSHDh3i/fff59dff32o9+fk5DBnzhx69Ch6kM7dSeJB+1d0AQEBfP/992RkZJCWlsb69eupVasWrVu35ttvvwXyPleg5u3H14Hd5PU5bNBa59x1yCNAs9v9Eiil7JVSVuTV2JuglLK+vb2dUqr2/WKTJCGqldoejag7uK3pzsGyrg11B7e9p9O6IvMf8QJWNQre+VjVsMF/xAtmiqig8PBwrK2tGT9+vGmb0WjE39+f69evM3ToUDp06MCzzz6b/8VHq1atmDNnDl27duXbb78lJCTE1IE+Y8YMnJ2dMRgMTJs2rdAZ4nfuP2fOHLy9vXF1dWXcuHGmcwQGBjJ9+vR7ZpGXl42nNtJrTS8M/zTQa00vNp7aaHrN09OT4cOHYzQaGTJkCP7+/gCsXLmSZcuW4e7ujouLC0DdOw4ZCjx3+88CtNY3yWuO+kQptQ/4H2AL/IO8VUP3KqXigf/HA7odpE9CVDu1PRpVqqRwt/x+hweNbjKX+Ph4vLy8Cn0tNjaWgwcP0qxZM7p06cLOnTvp2rUrALa2tuzYsQOAn376CYCUlBS+//57jhw5glKK1NRU6tatW+gM8XyTJk1i1qxZADz//PNs2LCB/v37Aw83i7wsbDy1kdmRs8nMyQQgKT2J2ZGzAQh2CgZg5syZzJw585735n8WAEqppPzHWus13LXom9Y65I7HewDfQsJ54/bPQ5EkIUQl9KT/UxUmKRSHj4+PaUa20WgkISHBlCSGDx9+z/516tTB1taWsWPHEhwc/FB9GuHh4Xz44YfcuHGDlJQUXFxcTEniYWaRl4WP935sShD5MnMy+Xjvx6YkUVFJc5MQolS5uLgQExNT6Gv3m6Fdu/a9TeNWVlbs3r2bIUOGsG7dOvr06XPfc2dmZjJx4kTWrFnDgQMHePHFF8nM/P3L+WFmkZeFi+kXi7W9IpEkIYQoVd27dycrK4svvvjCtG3Pnj38/PPPxT7W9evXuXr1Kn379mXRokX5I3yKnCGenxAaNGjA9evXK8zEwCa1mxRre0UiSUIIUWyHI8JZ+tJo/m9Ef5a+NJrDEeGm15RSfP/99/zvf/+jTZs2uLi4MHv2bJo1a1bs86SlpdGvXz8MBgPdunVj4cKFQN4M8fnz5+Ph4WEqbQ5Qt25dXnzxRdzc3HjmmWfuKfFuLpM9J2NraVtgm62lLZM9J5spoocnM66FEMUiM74fzcZTG/l478dcTL9Ik9pNmOw5udj9EUqpGK11xzIKsVDScV3N5eTkYGlpae4wxG0JCQn069fPVCeqIpIZ348m2Cm4wndSF0aamyqZZ555Bi8vL1xcXFi6dCmfffZZgeUzV6xYwcsvvwzAN998g4+PD0ajkT//+c/k5OTNt7Gzs2PWrFl06tSJqKioIseV79mzB4PBgJ+fH6+++qpp+c6cnBxeffVVvL29MRgM/L//l1cfLCkpiYCAAIxGI66uruU+Dr0yy5+h3K1bN06cOEHfvn2LLDOemppaYHnV8lYZZnyL0lNlk8SD6rEnJyfTqVMnPDw8KtWX2ZdffklMTAzR0dEsXryYwYMHs3btWtProaGhDB8+nMOHDxMaGsrOnTuJi4vD0tKSlStXApCeno6rqyu7du2ia9euTJo0iT179hAfH09GRoapquno0aP5/PPPiYqKKnC3sWzZMhwcHNizZw979uzhiy++4PTp0/zrX/+id+/exMXFsW/fPoxGY7l+NpXVnTOUf/75Z1q3bo2trS29evWiV69eZGRkEBgYSH4T7alTp5gyZQqQ90vBM888Q//+/WndujVLlizho48+wsPDA19fX1JSUgD44osv8Pb2xt3dnSFDhnDjxg0gr6LtK6+8QufOnXFycjJ19N4v4Vf0Gd+idFXZJPEgYWFhdOjQgdjYWNPsxspg8eLFuLu74+vry7lz5zh9+jROTk788ssvXLlyhaNHj9KlSxfCwsKIiYnB29sbo9FIWFgYp06dAvKG/w0ZMsR0zPDwcDp16oSbmxtbt27l4MGDpKamkpaWZlrB7M5y05s3b+arr77CaDTSqVMnrly5wvHjx/H29mb58uXMnj2bAwcOYG9vX74fTiV19wzl48ePM2vWLOLj49m3bx9PPvkk0dHRptFB7777LtnZ2RiNRkJDQ4mLiyMlJYXatWszefJkEhMTiY2Nxc/Pj6+++grImx+wZ88e0/GWLVtmOn9SUhI7duxgw4YNzJgxA+C+Cb+iz/gWpatKJYn33nuP9u3b06NHD44ePQrAyZMn6dOnD15eXvj7+3PkyBHi4uJ47bXX2LRpE0ajkYyMDDZv3oyfnx+enp4MGzaM69evA3nlAt5++208PT1xc3MzLSLz888/YzQaMRqNeHh4mIbjzZ8/39QM8/bbb5fq9W3bto0tW7YQFRXFvn378PDwIDMzk+HDh7N69Wq+++47Bg0ahFIKrTWjRo0yFQY7evQos2fPBvJmtubfGRQ1rvx+Axq01nzyySemY58+fZpevXoREBDA9u3bad68Oc8//7zpC0rc390zlFu3bo3RaMTW1paJEycyduxY3N3dWbRoEVpr3nrrLaysrIiLi2P48OE0btyYvn37Eh8fT7NmzRg3bhyQt3ZE/oSx+Ph4/P39cXNzY+XKlRw8eNB0vmeeeQYLCwucnZ1N9ZXul/Cf9H+KXuMmYd+gISiFfYOG0mldhZUoSSil5iuljiil9iulvldK1S2luIotJiaGVatWERsby9q1a00Lv4wbN45PPvmEmJgYFixYwMSJEzEajcyZM4fhw4cTFxdHeno6c+fOZcuWLezdu5eOHTvy0UcfmY7doEED9u7dy4QJE0xNWAsWLODTTz8lLi6OiIgIatasyebNmzl+/Di7d+8mLi6OmJgYtm/fXmrXePXqVerVq0etWrU4cuQIv/zyC5D3W+K6dev497//bZq1GhQUxJo1a7h0KW/VtZSUFM6cOXPPMYsaV16vXj3s7e1N51i1apXpPb179+azzz7j1q1bABw7doz09HTOnDlDo0aNePHFF/nTn/7E3r17S+3aq5P8CV9aazZv3swnn3zC/v37SU5O5tdffyUrq2CncfPmzU1f6Dk5OTz22GMAWFhYmCaMhYSEsGTJEg4cOMDbb79d6ASz/HMCD0z4T/o/xbhPl/PXVesZ9+lySRBVWElHN/0PeF1rna2U+hvwOjC95GEV38PWY7/7PxjAL7/8wqFDh0zr/t68ebPA8o53TuXPb//v0qULf/nLX3j22WcZPHgwjo6ObN68mc2bN+Ph4QHkTQQ6fvw4AQEBD30d+/fvJywsjKtXr+Lg4EBQUBAGgwGAPn368Pnnn2MwGGjfvj2+vnllWerVq4ezszOHDh3Cxydv4RxnZ2fmzp1Lr169yM3Nxdramk8//ZSWLVsWON+d48pbtWpVYFz5smXLePHFF6lduzaBgYE4ODgAMHbsWBISEvD09ERrTcOGDVm3bh3btm1j/vz5WFtbY2dnJ3cSD8nFxaXQSV8rV64kPT2dcePGkZSUxPfff09mZib/+c9/CuzXrFkzlixZwsaNG7l8+TKhoaG89NJLBfZJS0ujadOm3Lp1i5UrV9K8efP7xnTmzBmaN2/Oiy++SHp6Onv37uWFF6Q5qToqUZK4qw75L8DDrRhfRu5Xj/1+tNb07NmTf//734W+XthU/hkzZhAcHMymTZvw9fVly5YtaK15/fXX+fOf//xI8e/fv5/169ebfkO/evUq69evB8BgMGBjY3PPF0S+O5fQzDd8+PBC6+HkN6Xlmzt3LnPnzr1nPxcXF/bv3w/AvHnz6Ngxb3i2hYUF77//Pu+//36B/UeNGsWoUaMedJnV2/7VEDYHrp4HB0cImkX37sN44403+OKLL+jZsyeQN7LszJkz2NnZYWlpib+/P8uWLWPw4MEEBgaSm5trOuS1a9dMd3DTp083/Z3d6d1336VTp060bNkSNze3B65nLQlf5CvNPokxQOHfYOXgYeux79u37573+vr6snPnTk6cOAHAjRs3ihx+mO/kyZO4ubkxffp0OnbsyJEjR+jduzdffvml6Us4MTHR1NzzMMLCwkwJIt+tW7cICwt76GOUpo0bNxYY3fLmm2+aJY7K7ruLKXSMPEjT8Fg6JjrwXY12gIar52D9K6gD35pmKAcFBaG1Zvbs2fTt25ecnBw2bNhAREQEHTp0YO3atXz00Uf84Q9/wNXVlYMHDxIUFGTqG3NxcTF1Puc3MQFMmDCB06dPs23bNj755BNWrFgB5I2OurOSav6/3VGjRhEfH09sbCwRERG0bt26XD8zUXE88E5CKbUFKKzAyEyt9Q+395kJZAMr73OcccA4gMcff7zYgR7bdZGoH05yPSULu/o2+A1sQ7tOv4d1Zz32li1bFqjHPmHCBObOncutW7cYMWIE7u7uBY7dsGFDVqxYwciRI03NUXPnzqVdu3ZFxrNo0SLCw8OxtLTE2dmZp59+GhsbGw4fPmxqqrKzs+Obb76hUaOHK0t99erVYm0va0XdiYiH993FFKYdPUdGrgYU520bM6193ryWIclhcCsDwubQbGo8q1evvuf9UVFRhR73X//6V4HncgcnykqJy3IopUYB44EgrfWNh3lPcctyHNt1kfCVR8i++fsttlUNC556tkOBRFHZLVy4sNCE4ODgwNSpU80QkSipjpEHOZ91657tjpkXid6Vn4AVzE4t17hE5WSOshwlHd3Uh7yO6gEPmyAeRdQPJwskCIDsm7lE/XCyiHdUTkFBQVhbWxfYZm1tTVBQkJkiEiWVWEiCAEi0uePu0sGxnKIRovhKOrppCWAD/O92p/EvWuvx939L8V1PuXdE0v22V1b5o5iKGt0kKp/mNtaF3kk0z7rdV2VdE4JmlXNUQjy8ko5ueqK0Arkfu/o2hSYEu/o2hexduRkMBkkKldi2bdtYsGCBabTZ605N7+iTyFMzN4vXT30BDi3yEoThD+YKV4gHqhRVYP0Gtim0T8JvYBszRiXEgw1pUh+AD04lkZh1i+Y21rzu9DhDgr4zc2RCPJxKUZajXacmPPVsB9Odg119myrXaV2dvffee7i4uGAwGDAajezatYuxY8dy6NChYh8rPT2d4OBg3N3dcXV1JTQ0lJiYGLp164aXlxe9e/cmKSlvLfkTJ07Qo0cP3N3d8fT05OTJk2itTRVv3dzcCA0NBfLuEAIDAxk6dCgdOnTg2WefNc1O/umnn+jQoQNdu3YtUGwx35Am9Ynu7ELSU0aiO7uYEocQlYLWutx/vLy8tBBaax0ZGal9fX11Zmam1lrr5ORknZiY+MjHW7NmjR47dqzpeWpqqvbz89OXLl3SWmu9atUqPXr0aK211j4+Pnrt2rVaa60zMjJ0enq6XrNmje7Ro4fOzs7WFy9e1C1atNAXLlzQ4eHhuk6dOvrcuXM6JydH+/r66oiICJ2RkaEdHR31sWPHdG5urh42bJgODg5+5PiFuB8gWpfz93WluJMQVVdSUhINGjQwzWpv0KABzZo1K1Aa287OjpkzZ5qq3+YXofv1118ZNGgQ7u7uuLu7ExkZiZubG+vWraNp06Y88cQTjB07lvj4eHr27InRaGTu3LmcP3+etLQ0EhMTGTRoEJBX9LBWrVrs2LGDkSNHYmlpSePGjenWrZupDpiPjw+Ojo5YWFhgNBpJSEjgyJEjtG7dmrZt26KU4rnnnjPDpyhE2ZEkIcyqV69enDt3jnbt2jFx4kRTOew7paen4+vry759+wgICOCLL74A4JVXXqFbt27s27ePvXv34uLiQk5ODp6ennzwwQc0adKEhIQEGjVqZKpYe+DAATZv3lxklduitkPBQnh3lmi5uxyMEFWJJAlhVnZ2dsTExLB06VIaNmzI8OHDTSUj8tWoUYN+/foBeUUW88tfb926lQkTJgB5X9oODg6sXbuW/fv3s2jRIhITEzl06BApKSmmmcu3bt3i4MGD1KlTB0dHR9atWwfkFX68ceMGAQEBhIaGkpOTQ3JyMtu3bzcVTSxMhw4dOH36NCdP5s3ZKar+V0ViZ2dXqsdbsWIFFy5cKNVjiopDkoQwO0tLSwIDA3nnnXdYsmQJ331XcOSPtbW16bd1S0tLspNPwEJXuHEFFnvkFc27LTEx0bRMa4MGDfj555/ZunUr06dPx93dHaPRSGRkJABff/01ixcvxmAw0LlzZy5evMigQYMwGAy4u7vTvXt3PvzwQ5o0KXqAhK2tLUuXLiU4OJiuXbveU2W3OpAkUbWVuCzHoyhuWQ5RdR09ehQLCwvatm0LwJtvvklqairx8fEsWLCAjh07YmdnZyo8t+b//sKGlZ+xYkANRqy5ga+jJVP865LTdxHpbfpy/vx5Bg4cyM6dO2nUqBEpKSmkpaVVyy/votz5ec6fP5/Vq1eTlZXFoEGDeOedd4C8hYjOnTtHZmYmkydPZty4ceTk5PCnP/2J6OholFKMGTOGFi1aEBISQvPmzalZsyZRUVHUrFnTnJdXpZmjLEelmCchKq+r69dzaeEispOSsGralEZTp+DQv7/p9evXr/Pyyy+TmpqKlZUVTzzxBEuXLi1QmbSAA99Cbt6dwsd9bBm3IZNlsclY/n00n63Zip+f30OtoyEosEiW1poBAwawfft2AgIC+PLLL6lfvz4ZGRl4e3szZMgQEhISSExMJD4+HoDU1FTq1q3LkiVLTAldVD2SJESZubp+PUlvzULfXgUt+8IFkt7KK0GRnyi8vLxMzT932rZtm+nxnetfDG2VxtBWeb+pNraz4IcRtW6/ouB29V2pXvtw7rdI1uLFi/n+++8BOHfuHMePH6d9+/acOnWKl19+meDgYHr16mXO8EU5kT4JUWYuLVxkShD5dGYmlxYuevSDFlUMT4rkFZu+vUhW/sivEydO8Kc//anItdTr1avHvn37CAwM5NNPP2Xs2LHmvgRRDiRJiDKTfXtm88NufyhBs/KK4t1JiuQ9kqIWySpqLfXLly+Tm5vLkCFDePfdd01rmNvb2z9wpTtReUlzkygzVk2bkl3IqBerpk0f/aD5xfDuWgJUiuT9buOpjXy892Mupl+kSe0mTPacTLBT8D379erVq9BFsopaSz0xMZHRo0eblk794IMPgLwV8MaPHy8d11WUjG4SZebuPgkAZWtL03fnFOi8FqVn46mNzI6cTWbO75+5raUtszvPLjRRiMql0i06JMT9OPTvT9N352DVrBkohVWzZpIgytjHez8ukCAAMnMy+Xjvx2aKSFR20twkypRD//6SFMrRxfSLxdouxIPInYQQVUiT2oXPDi9quxAPIklCiCpksudkbC1tC2yztbRlsudkM0UkKjtpbhKiCsnvnH6Y0U1CPAxJEkJUMcFOwZIURKmR5iYhhBBFkiQhhBCiSNLcJMRdrly5QlBQEAAXL17E0tKShg0bArB7925q1KhhzvCEKFelkiSUUtOA+UBDrfXl0jimEOby2GOPERcXB8Ds2bOxs7Nj2rRp5g1KCDMpcXOTUqoF0BM4W/JwhKiYwsLC8PDwwM3NjTFjxpCVlWXukIQoF6XRJ7EQeA0o/yJQQpSDzMxMQkJCCA0N5cCBA2RnZ/PZZ5+ZOywhykWJkoRSagCQqLXeV0rxCFHh5OTk0Lp1a9q1awfAqFGj2L59u5mjEqJ8PLBPQim1BShsTv9M4A3goZanUkqNA8YBPP7448UIUQjzql27trlDEMJsHpgktNY9CtuulHIDWgP7lFIAjsBepZSP1vqeamJa66XAUsgrFV6SoIUoT5mZmSQkJHDixAmeeOIJvv76a7p162busIQoF488uklrfQBolP9cKZUAdJTRTaIySLr4A6dOLiAzKwlbm6Y4tZlG0yYDC93X1taW5cuXM2zYMLKzs/H29mb8+PHlHLEQ5lFqiw4VJ0nIokPCnJIu/sCRIzPJzc0wbbOwqEmHDu8VmSiEqAgq9aJDWutWchchKoNTJxcUSBAAubkZnDq5wEwRCVFxSVkOUe1kZiUVa7sQ1ZkkCVHt2No0LdZ2IaozSRKi2nFqMw0Li5oFtllY1MSpjZTeEOJuUuBPVDv5ndMPO7pJiOpMkoSolpo2GShJQYiHIM1NQgghiiRJQgghRJEkSQghhChSqc24LtZJlUoGzjzCWxsA1WXCXnW51upynVB9rrW6XCeU/7W21Fo3LMfzmSdJPCqlVHR5T0k3l+pyrdXlOqH6XGt1uU6oHtcqzU1CCCGKJElCCCFEkSpbklhq7gDKUXW51upynVB9rrW6XCdUg2utVH0SQgghyldlu5MQQghRjiplklBKvayUOqqUOqiU+tDc8ZQ1pdQ0pZRWSjUwdyxlQSk1Xyl1RCm1Xyn1vVKqrrljKk1KqT63/72eUErNMHc8ZUUp1UIpFa6UOnz7/+Zkc8dUlpRSlkqpWKXUBnPHUpYqXZJQSj0FDAQMWmsXoEqvFKOUagH0BM6aO5Yy9D/AVWttAI4Br5s5nlKjlLIEPgWeBpyBkUopZ/NGVWaygb9qrZ8EfIGXqvC1AkwGDps7iLJW6ZIEMAGYp7XOAtBaXzJzPGVtIfAaUGU7j7TWm7XW2bef/gI4mjOeUuYDnNBan9Ja3wRWkfdLTpWjtU7SWu+9/TiNvC/Q5uaNqmwopRyBYOAf5o6lrFXGJNEO8FdK7VJK/ayU8jZ3QGVFKTUASNRa7zN3LOVoDPAfcwdRipoD5+54fp4q+sV5J6VUK8AD2GXmUMrKIvJ+ecs1cxxlrkKWCldKbQGaFPLSTPJirkfe7aw3sFop5aQr6TCtB1zrG0Cv8o2obNzvOrXWP9zeZyZ5TRYryzO2MqYK2VYp/60+LKWUHfAdMEVrfc3c8ZQ2pVQ/4JLWOkYpFWjmcMpchUwSWuseRb2mlJoArL2dFHYrpXLJq5+SXF7xlaairlUp5Qa0BvYppSCvCWavUspHa32xHEMsFff7OwVQSo0C+gFBlTXhF+E80OKO547ABTPFUuaUUtbkJYiVWuu15o6njHQBBiil+gK2QB2l1Dda6+fMHFeZqHTzJJRS44FmWutZSql2QBjweBX7YrmHUioB6Ki1rnKF05RSfYCPgG5a60qZ7IuilLIirzM+CEgE9gB/1FofNGtgZUDl/TbzTyBFaz3FzOGUi9t3EtO01v3MHEqZqYx9El8CTkqpePI6AUdV9QRRDSwB7IH/KaXilFKfmzug0nK7Q34S8F/yOnJXV8UEcVsX4Hmg++2/x7jbv22LSqzS3UkIIYQoP5XxTkIIIUQ5kSQhhBCiSJIkhBBCFEmShBBCiCJJkhBCCFEkSRJCCCGKJElCCCFEkSRJCCGEKNL/B4vXPTUqNKNYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,3))\n",
    "for i, word in enumerate(vocabs[:20]): #loop each unique vocab\n",
    "    x, y = get_embed(word)\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(word, xy=(x, y), xytext=(5, 2), textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db93c73b-f7d4-4f70-b375-e8b5d021d45f",
   "metadata": {},
   "source": [
    "## 7. Normal Skipgram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "2ee51d87-112a-4e35-b2d2-c719ae4f3373",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "109"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#delete duplicate word define as vocabs, also include unknow as <UNK>.\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocabs  = list(set(flatten(corpus_tokenised)))\n",
    "vocabs.append('<UNK>') # to let '<UNK>' become the last.\n",
    "word2index = {v: idx+1 for idx, v in enumerate(vocabs)} #plus one to shift value in word index to assign <UNK>'s value to 0 without duplication with others.\n",
    "word2index['<UNK>'] = 0\n",
    "\n",
    "#Let check word to index's value\n",
    "word2index[\"cat\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "3001e6a0-4fcb-44d0-a38f-46debfcdeacf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'coast'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create index to word\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "index2word[50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "28dc7e9d-334d-4aa9-bbc4-c258ca57f428",
   "metadata": {},
   "outputs": [],
   "source": [
    "skipgrams = []\n",
    "\n",
    "#for each corpus\n",
    "for sent in corpus_tokenised:\n",
    "    #for each sent [\"apple\", \"banana\", \"fruit\"]\n",
    "    for i in range(2, len(sent) - 2): # start from 1 to second last but we use window size of 2, then first centre word should be at index 2\n",
    "        center_word = sent[i]\n",
    "        outside_words = np.ravel([sent[i - 2:i],sent[i+1: i + 3]])  # window_size = 2\n",
    "        for o in outside_words:\n",
    "            skipgrams.append([center_word, o])\n",
    "\n",
    "# skipgrams[:50] #shows only sample 50 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "c9fd6fca-3c9d-4c49-9593-0365755f3659",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size, corpus):\n",
    "    \n",
    "    skipgrams = []\n",
    "\n",
    "    for sent in corpus:\n",
    "        for i in range(2, len(sent) - 2): #start from 2 to third last for window size = 2\n",
    "            centre_word = word2index[sent[i]]\n",
    "            outside_words = []\n",
    "            for j in np.ravel([sent[i - 2:i],sent[i + 1: i + 3]]):\n",
    "                outside_words.append(word2index[j])  #window_size = 2\n",
    "            for o in outside_words:\n",
    "                skipgrams.append([centre_word, o])\n",
    "                \n",
    "    #only get a batch, not the entire list\n",
    "    random_index = np.random.choice(range(len(skipgrams)), batch_size, replace=False)\n",
    "    #appending some list of inputs and labels\n",
    "    random_inputs, random_labels = [], []   \n",
    "    for index in random_index:\n",
    "        random_inputs.append([skipgrams[index][0]])  #center words, this will be a shape of (1, ) --> (1, 1) for modeling\n",
    "        random_labels.append([skipgrams[index][1]])\n",
    "        \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "ee839e55-b4b7-49ad-9cf7-b65f8c0cf9f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Skipgram(nn.Module):\n",
    "    \n",
    "    def __init__(self, voc_size, emb_size):\n",
    "        super(Skipgram, self).__init__()\n",
    "        self.embedding_centre_word  = nn.Embedding(voc_size, emb_size)  #is a lookup table mapping all ids in voc_size, into some vector of size emb_size\n",
    "        self.embedding_outside_word = nn.Embedding(voc_size, emb_size)\n",
    "    \n",
    "    def forward(self, centre_word, outside_word, all_vocabs):\n",
    "        #center_word, outside_word: (batch_size, 1)\n",
    "        #all_vocabs: (batch_size, voc_size)\n",
    "        \n",
    "        #convert them into embedding\n",
    "        centre_word_embed  = self.embedding_centre_word(centre_word)     #(batch_size, 1, emb_size)\n",
    "        outside_word_embed = self.embedding_outside_word(outside_word)   #(batch_size, 1, emb_size)\n",
    "        all_vocabs_embed   = self.embedding_outside_word(all_vocabs)     #(batch_size, voc_size, emb_size)\n",
    "        \n",
    "        #bmm is basically @ or .dot , but across batches (i.e., ignore the batch dimension)\n",
    "        top_term = outside_word_embed.bmm(centre_word_embed.transpose(1, 2)).squeeze(2)\n",
    "        #(batch_size, 1, emb_size) @ (batch_size, emb_size, 1) = (batch_size, 1, 1) ===> (batch_size, 1)\n",
    "        \n",
    "        top_term_exp = torch.exp(top_term)  #exp(uo vc)\n",
    "        #(batch_size, 1)\n",
    "        \n",
    "        lower_term = all_vocabs_embed.bmm(centre_word_embed.transpose(1, 2)).squeeze(2)\n",
    "         #(batch_size, voc_size, emb_size) @ (batch_size, emb_size, 1) = (batch_size, voc_size, 1) = (batch_size, voc_size)\n",
    "         \n",
    "        lower_term_sum = torch.sum(torch.exp(lower_term), 1) #sum exp(uw vc)\n",
    "        #(batch_size, 1)\n",
    "        \n",
    "        loss_fn = -torch.mean(torch.log(top_term_exp / lower_term_sum))\n",
    "        #(batch_size, 1) / (batch_size, 1) ==mean==> scalar\n",
    "        \n",
    "        return loss_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "c85847a9-1e24-46be-8ddd-99ad1eb7ed7c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 1)\n",
      "label=array([[ 999],\n",
      "       [ 596],\n",
      "       [ 921],\n",
      "       [ 419],\n",
      "       [ 445],\n",
      "       [1007],\n",
      "       [ 206],\n",
      "       [ 912],\n",
      "       [ 109],\n",
      "       [ 570]])\n"
     ]
    }
   ],
   "source": [
    "input, label = random_batch(10, corpus_tokenised)\n",
    "\n",
    "print(f\"{input.shape}\")\n",
    "print(f\"{label=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "111adfee-41aa-4639-87d3-6bd14b04e52b",
   "metadata": {},
   "source": [
    "#### Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "2c274654-b519-42da-a888-57e57a4c0b98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1059])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2\n",
    "\n",
    "def prepare_sequence(seq, word2index):\n",
    "    #map(function, list of something)\n",
    "    #map will look at each of element in this list, and apply this function\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)\n",
    "\n",
    "all_vocabs = prepare_sequence(list(vocabs), word2index).expand(batch_size, voc_size)\n",
    "all_vocabs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "ff82449c-4b19-4430-a96f-8b3daaa403ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "input, label = random_batch(batch_size, corpus_tokenised)\n",
    "input_tensor = torch.LongTensor(input)  \n",
    "label_tensor = torch.LongTensor(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "88ce7e54-6003-46e1-b163-ff0f6b08f8bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_size = 10 # I will try on 10.\n",
    "voc_size = len(vocabs)\n",
    "model = Skipgram(voc_size, emb_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "e90176b5-3876-406c-9b9e-3a01443894eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 1]), torch.Size([2, 1]))"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_tensor.shape, label_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "a30ae1a5-067c-459e-94ab-35c8a8430b39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(8.7218, grad_fn=<NegBackward0>)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = model(input_tensor, label_tensor, all_vocabs)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973884f0-7145-4bf6-ac2d-32e5c72fd16e",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "edb69961-e926-49b3-9a33-22c9497643b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "voc_size   = len(vocabs)\n",
    "batch_size = 2\n",
    "emb_size   = 10\n",
    "model      = Skipgram(voc_size, emb_size)\n",
    "\n",
    "optimizer  = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "09b08267-54fb-43af-b33f-5d75c7cd25a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1000 | Loss: 5.950226 | Time: 0m 0s\n",
      "Epoch 2000 | Loss: 10.029776 | Time: 0m 0s\n",
      "Epoch 3000 | Loss: 12.667464 | Time: 0m 0s\n",
      "Epoch 4000 | Loss: 26.544285 | Time: 0m 0s\n",
      "Process Time: 3m 49s\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 4000\n",
    "startp = time.time()\n",
    "#for epoch\n",
    "for epoch in range(num_epochs):\n",
    "    start = time.time()\n",
    "    #get random batch\n",
    "    input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "    input_batch = torch.LongTensor(input_batch)\n",
    "    label_batch = torch.LongTensor(label_batch)\n",
    "    \n",
    "    # print(input_batch.shape, label_batch.shape, all_vocabs.shape)\n",
    "    \n",
    "    #loss = model\n",
    "    loss = model(input_batch, label_batch, all_vocabs)\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "    \n",
    "    #backpropagate\n",
    "    loss.backward()\n",
    "    \n",
    "    #update alpha\n",
    "    optimizer.step()\n",
    "    \n",
    "    #print epoch loss\n",
    "    if (epoch + 1) % 1000 == 0:\n",
    "        print(f\"Epoch {epoch+1} | Loss: {loss:.6f} | Time: {epoch_mins}m {epoch_secs}s\")\n",
    "\n",
    "endp = time.time()\n",
    "mins, secs = epoch_time(startp, endp)\n",
    "print(f\"Process Time: {mins}m {secs}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1ab4f14-d73f-4fbf-903e-dc79c3baceba",
   "metadata": {},
   "source": [
    "#### Plot the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "55a5a7b7-a08c-432a-b968-45f1b854804a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([109])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat = torch.LongTensor([word2index['cat']])\n",
    "cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "7e9bb4c9-7100-4b47-b562-28d4806476d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.2993, -0.1524,  0.7747, -0.4467,  0.3981,  0.3952, -1.1624,  1.2481,\n",
       "          0.5322,  1.3042]], grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_centre_embed = model.embedding_centre_word(cat)\n",
    "cat_outside_embed = model.embedding_outside_word(cat)\n",
    "\n",
    "cat_embed = (cat_centre_embed + cat_outside_embed) / 2\n",
    "cat_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "3496d9af-afeb-4ab9-bd63-02953ac00c9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embed(word):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except:\n",
    "        index = word2index['<UNK>']\n",
    "    \n",
    "    word = torch.LongTensor([index])\n",
    "\n",
    "    center_embed  = model.embedding_centre_word(word)\n",
    "    outside_embed = model.embedding_outside_word(word)\n",
    "    \n",
    "    embed = (center_embed + outside_embed) / 2\n",
    "    \n",
    "    return  embed[0][0].item(), embed[0][1].item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "8e5b05d8-77de-40ac-91a8-2c6224916774",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-1.2992620468139648, -0.1523895263671875)\n"
     ]
    }
   ],
   "source": [
    "print(get_embed('cat'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "158ab7dc-d2b9-4dd7-929c-7abdacabb0c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAADFCAYAAAChMA4eAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6x0lEQVR4nO3deVyVZfr48c/NIqAiuCYuiTuxHkEQVNDCxNTMddRJ00wdMzObn462TDmWaWmTOZqm39QWC4rU3EpH1MQlFRQV9yCcUkjUQEBAlvv3B3gSBQU5cgCv9+vFi3PuZznXc2fn4nnuTWmtEUIIIYpjYe4AhBBCVF6SJIQQQpRIkoQQQogSSZIQQghRIkkSQgghSiRJQgghRIkkSVQySqkEpVQDc8chhBAgSUIIIcQdPDBJQin1jFLqqFLqiFLqc6VUC6VURGFZhFLq4cL9Vimlliildiil4pVS3ZRSK5RSJ5VSq246X7pS6l2lVLRSaptSyk8ptbPwmH6F+9gqpVYqpY4ppQ4rpR4tLB+tlFqjlPpBKXVWKfVeMfG+pZR66ab3s5VSk+97RQkhxE2UOUZcN2jQQDs7O1fY52VmZhIXF4eLiwtWVlbk5uaSkJBA3bp1qV+/PpcuXSIlJYU2bdqQkJBAfn4+LVu2JDU1lV9++QUXFxdsbW05deoULVq0oGbNmkRHR9OmTRscHByIi4sjLy+Ptm3bkpmZSUJCAq6urvz+++9kZmbi7OxMVlYWZ86cwd3dnStXrpCYmIirqytKKY4fP0779u2pUaMGx44d45FHHiEvL4+4uDhcXV3RWhMbG8sjjzyClZVVhdWbEKJyiY6OvqS1blihH6q1rvAfHx8fXZEWLlyoX3311SJl9evX19evX9daa339+nVdv359rbXWo0aN0l988YXWWuu4uDjdpk0b4zEjR47Ua9eu1VprXaNGDZ2fn6+11vqf//ynfvvtt7XWWufl5WkHBwettdb9+/fXERERxuO7du2qjxw5oleuXKnHjh1rLO/Vq5eOjIzUWmvdokULnZycrLXWukePHvrQoUP6+++/14MGDTJJXQghqi4gSlfw9/UD8Wep1hql1B33uXm7jY0NABYWFsbXN97n5uYCYG1tbTzm5v1u3kff4S7t5vNaWloaj7nZ2LFjWbVqFUlJSYwZM+aO8QshxP3wQLRJBAcH8/XXX3P58mUArly5QufOnQkNDQVg9erVdO3a1eSfGxQUxOrVqwE4c+YM//vf/2jfvn2pjx8wYAA//PADBw8eJCQkxOTxCSHE3VSbO4l1h88zb8tpLqRk0sTRjmkh7enfoSkAbm5uvPbaa3Tr1g1LS0s6dOjAwoULGTNmDPPmzaNhw4asXLnS5DFNnDiRCRMm4OHhgZWVFatWrSpyB3E3NWrU4NFHH8XR0RFLS0uTxyeEEHdjlobrjh076qioKJOdb93h87yy5hiZOXnGMjtrS+YM9DAmiqooPz8fb29vvvnmG9q2bWvucIQQZqaUitZad6zIz6wWj5vmbTldJEEAZObkMW/LaTNFVH4nTpygTZs2BAcHS4IQQphNtXjcdCEls0zlVYGrqyvx8fHmDkMI8YCrFncSTRztylQuhBCidKpFkpgW0h4766INu3bWlkwLKX1PIiGEELerFo+bbjROl9S7SQghxL2pFkkCChKFJAUhhDCtavG4SQghxP0hSUIIIUSJyp0klFLNC6fVPqmUOn7z9NZCCCGqNlO0SeQC/09rfUgpZQ9EK6X+q7U+YYJzCyGEMKNy30lorRO11ocKX6cBJwFpQRZCiGrApG0SSilnoAOwv5ht45VSUUqpqOTkZFN+rBBCiPvEZElCKVUb+BaYorW+eut2rfUyrXVHrXXHhg0rdmElIYQQ98YkSUIpZU1BglittV5jinMKIYQwP1P0blLAJ8BJrfW/yx+SEEKIysIUdxJdgJHAY0qpmMKf3iY4rxBCCDMrdxdYrfVu4M4LSAshhKiSZMS1EEKIEkmSMJPevXuTkpICQO3atYvdZ/To0YSHh5f53AkJCXz55ZflCU8IIQBJEmazefNmHB0d78u5JUkIIUxFksR98t5777Fw4UIAXn75ZR577DEAIiIiGDFiBM7Ozly6dKnIMVprJk2ahKurK3369OHixYvGbQcPHqRz5854eXnh5+dHWloaCQkJBAYG4u3tjbe3N3v37gVgxowZREZGYjAY+OCDDzh+/Dh+fn4YDAY8PT05e/ZsBdWCEKLK01pX+I+Pj4+u7vbt26cHDx6stda6a9eu2tfXV1+/fl3PnDlTL126VLdo0UInJydrrbWuVauW1lrrb7/9Vvfo0UPn5ubq8+fPawcHB/3NN9/o7Oxs3bJlS33gwAGttdapqak6JydHZ2Rk6MzMTK211mfOnNE36nXHjh26T58+xlgmTZqkv/jiC6211tnZ2fratWsVUwlCCJMConQFf1/LncR94uPjQ3R0NGlpadjY2BAQEEBUVBSRkZEEBgYWe8yuXbsYPnw4lpaWNGnSxHj3cfr0aZycnPD19QWgTp06WFlZkZOTw7hx4/Dw8GDIkCGcOFH8nIoBAQG88847vPvuu5w7dw47O1n7WwhROpIk7hNra2ucnZ1ZuXIlnTt3JjAwkB07dhAXF8cjjzxS4nEFYxOL0loXW/7BBx/w0EMPceTIEaKiorh+/Xqx5/zrX//K+vXrsbOzIyQkhO3bt9/7hQkhHiiSJO6joKAg5s+fT1BQEIGBgSxduhSDwVDsF/6N/UNDQ8nLyyMxMZEdO3YA4OLiwoULFzh48CAAaWlp5ObmkpqaipOTExYWFnz++efk5eUBYG9vT1pamvG88fHxtGrVismTJ9OvXz+OHj16n69cCFFdVJs1rs0h4/BFrm5JIC8lG0tHG+qEOFOrQyPj9sDAQGbPnk1AQAC1atXC1ta2xEdNAAMGDGD79u14eHjQrl07unXrBkCNGjUICwvjxRdfJDMzEzs7O7Zt28bEiRMZNGgQ33zzDY8++ii1atUCwNPTEysrK7y8vBg9ejRZWVl88cUXWFtb07hxY9544437WzFCiGpDFbSFVKyOHTvqqKioCv9cU8o4fJGUNWfROfnGMmVtgePAtkUShRBCmIpSKlpr3bEiP1MeN92jq1sSiiQIAJ2Tz9UtCeYJSAgh7gNJEvcoLyW7TOXmNHPmTObPn1/i9uTkZDp16kSHDh2IjIyswMiEEJWdtEncI0tHm2ITgqWjjRmiKZ+IiAhcXFz49NNPzR2KEKKSkTuJe1QnxBllXbT6lLUFdUKczRPQLWbPnk379u3p0aMHp0+fBiAuLo5evXrh4+NDYGAgp06dIiYmhn/84x9s3rwZg8FAZmYmW7duJSAgAG9vb4YMGUJ6ejoAzs7OvPnmm3h7e+Ph4cGpU6cA+PHHHzEYDBgMBjp06GDsWTVv3jx8fX3x9PTkzTffNE9FCCHKRZLEParVoRGOA9sa7xwsHW0qTaN1dHQ0oaGhHD58mDVr1hi7zo4fP57//Oc/REdHM3/+fCZOnIjBYGDWrFkMHTqUmJgYMjIyePvtt9m2bRuHDh2iY8eO/Pvff64l1aBBAw4dOsTzzz9vfIQ1f/58Fi9eTExMDJGRkdjZ2bF161bOnj3LgQMHiImJITo6ml27dpmlPoQQ904eN5VDrQ6NKkVSuFVkZCQDBgygZs2aAPTr14+srCz27t3LkCFDjPtlZ9/+uOynn37ixIkTdOnSBYDr168TEBBg3D5w4ECgYET5mjUFK9V26dKFv//97zz99NMMHDiQZs2asXXrVrZu3UqHDh0ASE9P5+zZswQFBd2fiwYuXLjA5MmT7zhzbufOnY1zXAkh7s4kSUIptQLoC1zUWrub4pyifG4dsJefn4+joyMxMTF3PE5rzeOPP85XX31V7HYbm8I7J0tLcnNzgYIJBfv06cPmzZvx9/dn27ZtaK155ZVX+Nvf/lb+iymlJk2a3HVqdUkQQpSNqR43rQJ6mehcopyCgoJYu3YtmZmZpKWlsWHDBmrWrEnLli355ptvgIJkcOTIkduO9ff3Z8+ePfz8888AXLt2jTNnztzx8+Li4vDw8GD69Ol07NiRU6dOERISwooVK4ztGefPny8yq215TZ8+nY8++sj4fubMmbz//vu4uxf8jVLSzLc31u7QWjNt2jTc3d3x8PAgLCwMgJ07d9K9e3cGDx6Mi4sLTz/9NOYYSyREZWGSJKG13gVcMcW5xN2d2Z/Ep6/uYfGE7Xz66h7O7E8qst3b25uhQ4diMBgYNGiQcZT36tWr+eSTT/Dy8sLNzY3vvvvutnM3bNiQVatWMXz4cDw9PfH39zc2UJdkwYIFuLu74+XlhZ2dHU888QQ9e/bkr3/9KwEBAXh4eDB48OAiU4WU17Bhw4xf7ABff/21cQJEgKVLl/LSSy8RExNDVFQUzZo1K3L8mjVriImJ4ciRI2zbto1p06aRmJgIwOHDh1mwYAEnTpwgPj6ePXv2mCxuIaoak424Vko5AxtL87ipOoy4Npcz+5PYsfoUudf/HMhnVcOCR592oV2nxmaMrOI98sgjREREkJyczMSJE1m9ejV9+/YlNjaWL7/8ktmzZ/PMM88wcOBA2rZtCxTcSaSnp/Pyyy/j4eHBmDFjABg5ciRDhgyhTp06zJ49m//+978APP/883Tp0oURI0aY7TqFuKFaj7hWSo1XSkUppaKSk5Mr6mOrnX3fxRVJEAC51/PZ912cmSIyn8GDBxMeHk5YWBjDhg0rsu1uM9/e6Y+jG+0uULTtRYgHUYUlCa31Mq11R611x4YNG1bUx1Y76VeKH9FdUnl1NmzYMEJDQwkPD2fw4MFFtt1t5tugoCDCwsLIy8sjOTmZXbt24efnV5HhC1ElSBfYKqZ2PZtiE0LtelVvpPfdHD16lIiICFJTU3FwcCA4OBhPT0/jdjc3N9LS0mjatClOTk4kJCQYt4WFhd1x5tsBAwawb98+vLy8UErx3nvv0bhx47u2vwjxoDFJm4RS6iugO9AA+B14U2v9SUn7S5vEvXtQ2iSOHj3Khg0byMnJMZZZW1vz5JNPFkkUQjxIqmybhNZ6uNbaSWttrbVudqcEIcqnXafGPPq0i/HOoXY9m2qXIKBgPqmbEwRATk4OERERZoroT7179yYlJaVMx1y+fNk4dUnjxo1p2rSp8X1JKwpWlBvdgk1l1apVXLhwwaTnFOYjj5uqoHadGle7pHCr1NTUMpVXpM2bN5f5mPr16xsHMs6cOZPatWszdepUE0dWOaxatQp3d3eaNGli7lCECcjcTaJScnBwKFP5rTIyMujTpw9eXl64u7sTFhaGs7Mz06dPx8/PDz8/P+OAweTkZAYNGoSvry++vr7GcRHp6ek8++yzeHh44OnpybfffgsUTHR46dIlAL744gvjoL2//e1v5OXlkZeXx+jRo40D9T744INiY4yIiKBDhw7GrrjFTZNS0UqalLF///74+Pjg5ubGsmXLAIq9zvDwcKKionj66aeNE0aKqk3uJESlFBwcXGybRHBwcKmO/+GHH2jSpAmbNm0CCu5Apk+fTp06dThw4ACfffYZU6ZMYePGjbz00ku8/PLLdO3alf/973+EhIRw8uRJ3nrrLRwcHDh27BgAf/zxR5HPOHnyJGFhYezZswdra2vjWA03NzfOnz9PbGwsQLGPprKyshg9ejQRERG0a9eOZ555hiVLljBlypR7qC3TuHlSRq01/fr1Y9euXQQFBbFixQrq1atHZmYmvr6+DBo0iISEhNuu09HRkUWLFjF//nw6dqzQR+fiPpE7CVEpeXp68uSTTxrvHBwcHMrUaO3h4cG2bduYPn06kZGRxvMMHz7c+Hvfvn0AbNu2jUmTJmEwGOjXrx9Xr14lLS2Nbdu28cILLxjPWbdu3SKfERERQXR0NL6+vhgMBiIiIoxdb+Pj43nxxRf54YcfqFOnzm3x5eXl0bJlS9q1awfAqFGjzD5L7s2TMnp7e3Pq1CnjdCYLFy7Ey8sLf39/fv31V86ePVuq6xRVn9xJiErL09PznnsytWvXjujoaDZv3swrr7xCz549gaITH954nZ+fz759+7CzsytyDq31bRMl3rp91KhRzJkz57ZtR44cYcuWLSxevJivv/6aFStWFNleq1ate7qu+6mkSRl37tzJtm3b2LdvHzVr1qR79+5kZWVRt27du16nqPrkTkJUSxcuXKBmzZqMGDGCqVOncujQIQDjfE9hYWHGKdB79uzJokWLjMfeaGC+tfzWx03BwcGEh4cbJy68cuUK586d49KlS+Tn5zNo0CDeeust42ffLCsri4SEBGO7yOeff063bt1MdPX3pqRJGVNTU6lbty41a9bk1KlT/PTTTwAlXqe9vb1J5+kS5iV3EqJKSkz6jvi4+WRlJ2Jr40Sr1lNxavyUcfuxY8eYNm0aFhYWWFtbs2TJEgYPHkx2djadOnUiPz/fOB36woULeeGFF/D09CQ3N5egoCCWLl3K66+/zgsvvIC7uzuWlpa8+eabxvU0AFxdXXn77bfp2bMn+fn5QAYvvFAbZXGF9+f/gZV1Q6ytHIq907C1tWXlypUMGTKE3NxcfH19mTBhwn2pq03xm/jw0IckZSTRuFZjXvJ+iT6t+ty2X8+ePTl58qQxedauXZsvvviCXr16sXTpUjw9PWnfvj3+/v5AQRJ59tlnC68d43WOHj2aCRMmYGdnV+wdmqhaTDbBX1nIYDpRHolJ33Hq1Gvk5//Zc8bCwg4Xl9lFEsWtnJ2diYqKokGDBpUmpvttU/wmZu6dSVZelrHM1tKWmZ1nFpsoROVWZQfTibtLSEgwrnUg4OWXX2bBggXG9yEhIYwdO9b4/v/9v//HrFmzmDt3LgDr1q3jxIkTAMTHzWfKlDhOn/6zy2h+fibxcfOJiopi8uTJFXMRN4mPm18kQdwckzl9eOjDIgkCICsviw8PfWimiERVI0lCmMXNy4jm5+dz6dIljh8/bty+d+9eQkJCmDFjBlA0SWRlJxZ7zqzsRDp27MjChQuL3Z6QkHBf7iLuFpM5JWUklalciFtJkqhAeXl5jBs3Djc3N3r27ElmZibdu3fnxqO3S5cu4ezsDBSMWu3fvz9PPvkkLVu2ZNGiRfz73/+mQ4cO+Pv7c+VKwRpPy5cvx9fXFy8vLwYNGsS1a9eAgufCkydPpnPnzrRq1cq4rGdiYiJBQUEYDAbc3d2JjIys+IqgYF3sG0ni+PHjuLu7Y29vzx9//EF2djYnT57kyJEjTJo0ib1797J+/XqmTZuGwWDg8iVHAHb9mM4LE88z6plfOXY0E1sbJ3bu3Enfvn2BgpHNY8aMoXv37rRq1arE5GEKtjZOZSqvKI1rFT8yv6RyIW4lSaICnT17lhdeeIHjx4/j6OhoHMFbkhuL5xw4cIDXXnuNmjVrcvjwYQICAvjss88AGDhwIAcPHuTIkSM88sgjfPLJn9NmJSYmsnv3bjZu3Gj8i/zLL78kJCTEuCqbwWC4b9d7J02aNMHKyor//e9/7N27l4CAADp16sS+ffuIiorC09OTGjVqAAV3Hf369WPevHnExMQQGPRPFBbk5cHij5oy8YX6fP75VVq1vn2ai1OnTrFlyxYOHDjAv/71r9vmgzKVVq2nYmFRtIHWwsKu2Jgq0kveL2FraVukzNbSlpe8XzJTRKKqkd5NFahly5bGL2UfH58iU1sX59FHH8Xe3h57e3vjYDIoGCh2Y32E2NhYXn/9dVJSUkhPTyckJMR4fP/+/bGwsMDV1ZXff/8dAF9fX8aMGUNOTg79+/c3W5KAP+8m9u7dy9///nfOnz/P3r17cXBwoHPnziUe59T4KWrWbMljj1kCGXi4P8zyZf/DqfFTnD61s8i+ffr0wcbGBhsbGxo1asTvv/9+21KmpnCjcfpOPa7M4UbjdGl6NwlRHEkSFejWFc8yMzOxsrIydiHMysoqcX8LCwvjewsLC+NqaaNHj2bdunV4eXmxatUqdu7cWezxN3qxBQUFsWvXLjZt2sTIkSOZNm0azzzzjGkvtJRutEscO3YMd3d3mjdvzvvvv0+dOnUYM2YMly9fLvHYGjUa4OtbMPXDpUuXKKnDR0WuMufU+CmzJ4Xi9GnVp8KTwoULF5g8ebLxMaeouuRxk5k5OzsTHR0NcE//Q6WlpeHk5EROTg6rV6++6/7nzp2jUaNGjBs3jueee67YgV4VpUuXLmzcuJF69ephaWlJvXr1SElJYd++fca++jfIAK2qpUmTJpIgqglJEqZy9Gv4wB1mOhb8Pvp1qQ6bOnUqS5YsoXPnzsaZRcvirbfeolOnTjz++OO4uLjcdf+dO3diMBjo0KED3377LS+9dH+eTZ+M3MGyF57l/WFPsuyFZzkZueO2fTw8PLh06ZJxcNaNMgcHh9t6IQ0bNox58+bRoUMH4uIevPW8K7Pp06fz0UcfGd/PnDmT999/39jle9WqVQwcOJBevXrRtm1b/vGPfxj33bp1KwEBAXh7ezNkyBDjaG9RechgOlM4+jVsmAw5N/WTt7aDJxeC51/MF5eZnIzcwdZli8i9/uc4BqsaNvQcP4lHAh81Y2Tifjh8+DBTpkzhxx9/BApGoi9dupSJEycSGxvLqlWrmDVrFocPH8bGxob27duze/du7OzsGDhwIN9//z21atXi3XffJTs7+7alZsWfzDGYziRtEkqpXsCHgCXwf1rruaY4b5URMatogoCC9xGzHsgkERn6WZEEAZB7PZvI0M8kSVRDHTp04OLFi1y4cIHk5GTq1q3Lww8/XGSf4OBg40y8rq6unDt3jpSUFE6cOEGXLl0AuH79+m2PGYX5lTtJKKUsgcXA48BvwEGl1Hqt9YnynrvKSP2tbOXVXNrl4h+blVQuqr7BgwcTHh5OUlISw4YNu217cR0ItNY8/vjjxjm0ROVkijYJP+BnrXW81vo6EApUvi4e95NDCV0qSyqv5uzrFz+quaRyUfUNGzaM0NBQwsPDGTx4cKmO8ff3Z8+ePcaZcK9du8aZM2fuZ5jiHpgiSTQFfr3p/W+FZQ+O4DcK2iBuZm1XUP4AChz2DFY1bIqUWdWwIXCYebraivLLOHyRxLkH+G1GJIlzD5Bx+GKR7W5ubqSlpdG0aVOcnEo3yrxhw4asWrWK4cOH4+npib+/P6dOnbof4YtyKHfDtVJqCBCitR5b+H4k4Ke1fvGW/cYD4wEefvhhn3PnzpXrcyuLpKQkpkyZwsE9O7HJScHZPo8Fg51pN3z2be0RKSkpfPnll0ycONFM0Vack5E7iAz9jLTLl7Cv34DAYc9Ie0QVlXH4IilrzqJz8o1lytoCx4FtqdWhkRkje/BU1Ybr34DmN71vBly4dSet9TJgGRT0bjLB55qd1poBAwYwatQoQkNDgYIFa35PS6OdZ+Bt+6ekpPDRRx89EEnikcBHJSlUE1e3JBRJEAA6J5+rWxIkSTwATPG46SDQVinVUilVAxgGrDfBeSu9HTt2YG1tXWSxmBtjEIKDg/H29sbDw4PvvvsOgBkzZhAXF4fBYGDatGmVZrI9Ie4kLyW7TOWiein3nYTWOlcpNQnYQkEX2BVa6+N3OaxaiI2NxcfH57ZyW1tb1q5dS506dYyDxfr168fcuXOJjY01Lo/5/vvvExISwmuvvUZeXp5xBlchKhNLR5tiE4Klo00xe4vqxiQjrrXWm7XW7bTWrbXWs01xzqpMa82rr76Kp6cnPXr04Pz588YJ9m7m6+vLypUrmTlzJseOHcPe3t4M0VYPN08RLkyrTogzyrroV4WytqBOiLN5AhIVSqblKAc3NzfjvEs3W716NcnJyURHRxMTE8NDDz102+R98Odke02bNmXkyJHG6b/Fn2bPno2bmxuenp4YDAb279/P2LFjjQsQifuvVodGOA5sa7xzsHS0kUbrB4gkiXJ47LHHyM7OZvny5caygwcPGifRs7a2ZseOHdzoyXXrJHWVabK98srIyKBPnz54eXnh7u5OWFgY0dHRdOvWDR8fH0JCQkhMLFil7eeff6ZHjx54eXnh7e1NXFwcWmumTZuGu7s7Hh4ehIWFsW/fPlavXk39+vVp164dGRkZvPvuuyxfvhxXV1d++OEHXFxc6Nq1K2vWrDFzDVRvtTo0wmmGH83mBuI0w08SxINEa13hPz4+PrqqCE+8rH32xOrG2w9rnz2xOjzxcpHt58+f10OGDNGtWrXSrq6uunfv3vrAgQPa399f+/j46Oeee067uLjoX375RWut9fDhw7Wbm5ueOnWqXrVqlXZzc9MGg0F37dpVx8fHm+EKTSM8PFyPHTvW+D4lJUUHBAToixcvaq21Dg0N1c8++6zWWms/Pz+9Zs0arbXWmZmZOiMjQ4eHh+sePXro3NxcnZSUpJs3b67/7//+T/v7++s6deroX3/9Vefl5Wl/f39tMBj07t27dbNmzbSdnZ1+5ZVXtIODg3Z0dNRJSUlaa62TkpJ0//79taenp/b09NR79uzRWmv9+eefa19fX+3l5aXHjx+vc3NzK7KahCgXIEpX8Pe1rCdxB98mXWHq6V/JzC/osftbdg5TTxeMGxzUuB5QMCXy11/fPuPrvn37ij3nl19+WeT9qFGjTBmy2Xh4eDB16lSmT59O3759qVu3LrGxsTz++ONAwdKtTk5OpKWlcf78eQYMGAAUNPID7N69m+HDh2NpaclDDz1Et27dqF27NsnJyeTk5PDOO+8wdOhQDAYDO3fuJCEhgZYtW/Lbb78REBCAv78/L7/8MsuXL+f1119n8uTJdOvWjbVr15KXl0d6ejonT54kLCyMPXv2YG1tzcSJE1m9erXZ1tMQoiqQJHEHc+ITjQnihsx8zZz4RGOSEAXatWtHdHQ0mzdv5pVXXuHxxx/Hzc3ttmR59erVYo/XxQzqtLOz4+OPP+b111+nYcOGDB06FDc3N+O+Silq1KhB37592bBhAw4ODsbV/rZv325s47G0tMTBwYHPP/+c6OhofH19AcjMzKRRI3lsIsSdSJvEHZzPLn495JLKH2QXLlygZs2ajBgxgqlTp7J//36Sk5ONSSInJ4fjx49Tp04dmjVrxrp16wDIzs7m2rVrBAUFERYWRl5eHsnJyezatQs/Pz8sLS2pX78+//rXv1i0aJFxLQlnZ2d++eUXLC0tUUrx1VdfoZS648pzWmtGjRpFTEwMMTExnD59mpkzZ97vqhGiSpMkcQdNbazLVF7t3WFhpWPHjuHn54fBYGD27NnMmjWL8PBwpk+fjpeXFwaDgb179wLw+eefs3DhQjw9PencuTNJSUkMGDAAT09PvLy8eOyxx3jvvfdITU3lt9/+nEk3JiaGOnXqAAWzii5btoysrCy6du1KixYtioQaHBzMkiVLgIJHXVevXiU4OJjw8HAuXiyYd+jKlStUl+lhhLhfZNGhO7i1TQLAzkIxv33zB+9x031aWCl1wwYufrCA3MRErJycaPTyFByefBKA6OhoXnzxRVJSUrCysqJNmzYsW7aMwYMHM39+wfrWtWvXNq5mFh4ezsaNG1m1ahW///4748ePJz4+HktLS5YsWUJAQABhYWHMmTOH/Px8rK2tWbx4cZGV8YSozMwxd5Mkibv4NukKc+ITOZ+dQ1Mba15p5fTgJQgouHNI/fX2cofm8HLsPZ0ydcMGEv/5BvqmMSTK1hant2YZE4UQ4k/mSBLyuOkuBjWuR1RnNxIfNRDV2e3BTBBgsoWVbkxyCLDhjTeZ8PPZItt1VhYXP1hwLxEW0b17d6rKHyJCVGaSJETpmGhhpZuTRN7ly8Xuk1s46O5u7tRILYQwDUkSonRMtLDSzTPhzk/5g2v5+fT9JZ6uP59l2oULBYN3nJxKHK3dvXt3Xn31Vbp168aHH35Y4n4AX3zxBZ07d8bd3Z0DBw6Qn59P27ZtSU5OBiA/P582bdpw6ZIsqypESSRJiNLx/EtBI7VDc0AV/L6HRuu5c+fSunVrYmJimDNjBiezs+lasybP1q3LbznXOZyXR90XJ/Hiiy8SHh5OdHQ0Y8aM4bXXXjOeIyUlhR9//JHJkyffcb+MjAz27t3LRx99xJgxY7CwsGDEiBGsXr0agG3btuHl5UWDBrKsqhAlkcF0ovQ8/1Kunkw3mz17NkuXLsWqZk0u16hBIw1O9vb8MyOdf02fzi+//MJjjz2G1pq4uDgCAwM5cuQIP/74o3HRplatWpGamkr79u2xtLQkIyMDrTW9e/cGYPjw4UDBRIpXr14lJSWFMWPG8NRTTzFlyhRWrFjBs88+a5LrEaK6kjsJUeEyMzMJDQ1l+fLldOrShVP29jz0j2kcyMvjL6NG8e233/LQQw8RHBxMbGwsrVq1Ijw8nMjISGrXrs3p06c5d+4c9erVw83NjX79+tG9e3fS09OJjo5mxowZQMGI7JsppWjevDkPPfQQ27dvZ//+/TzxxBPmqAIhqgxJEqJC2dvb88cffzBgwABsbW2xsrKiX79+ZGRkcP36dVxcXIx3Bt9//z0A/v7+fPnll+zatYsWLVpw+PBhIiMj6dGjB8nJyVy8eJH+/fuTl5eH1tq4dkdYWBhQMC+Ug4MDDg4OAIwdO5YRI0bwl7/8BUtLS/NURBnc3NgvREWTx03C5M7sT2Lfd3GkX8mmdj0bAp5qTbtOjQGoX78+LVu2ZOnSpXz//fc89NBDtx1fo0YNFi1axPDhw/Hy8uLSpUskJiZy+fJlGjRowJkzZ9i9ezd9+/Zl5MiRhISEcOLECd555x2mTJlinNupbt26dO7cmatXr7JixQrj+fv168ezzz5bZR41PUhro4vKp1x3EkqpIUqp40qpfKVUhQ7wEJXTmf1J7Fh9ivQrBctdpl/JZsfqU5zZn2TcZ/ny5TRu3Jhdu3bx1VdfsWHDBmrVqkWrVq1o3bo1UDDaety4cRw5coQ9e/YQGxtL27Zt2blzJw8//DCbN2+mS5cuGAwGnnjiCebPn8/x48cZN24cULBS3Zw5c9i7dy+xsbH4+fkZP//IkSN4eXnh4uJSgTVz727uEfbss8+yfn3BEvIDBgxgzJgxAHzyySe8/vrrAPz73//G3d0dd3d3FixYYK6wRTVR3juJWGAg8LEJYhHVwL7v4si9nl+kLPd6PntDY413E97e3sZpv1u0aEFgYCAAn376KRMmTODatWu0atWKlStXAgWT+UFBAzRA165d+e2336hbt26Z45s7dy5Lliwx9nCqCm5eGz00NJTIyEj69evH+fPnjV1+d+/ezbBhw4iOjmblypXs378frTWdOnWiW7dudOjQwcxXIaoqk0zLoZTaCUzVWpdqiGtVmpZDlM3iCduL36A1I/pek+k27kFCQgJ9+/YlNjaW8+fPM2jQIFasWMF7773HH3/8wdKlS3n00Uc5ePAgK1as4PLly8yaNQuAf/7znzRs2JDJkyeb+SqEKVTraTmUUuOVUlFKqagbg5lE9VO7nk2x5TbZV0wy3caDrmnTpvzxxx/88MMPBAUFERgYyNdff03t2rWxt7cvdl0OIcrjrklCKbVNKRVbzM9TZfkgrfUyrXVHrXXHhg0b3nvEolILeKo1FnnZRcos8rJpHb++1NNtiKJuXRs9ICCABQsWGJPE/PnzjY/sgoKCWLduHdeuXSMjI4O1a9catwlxL+7aJqG17lERgYjqoV2nxiS+s5mzdYPItqmHTfYVWsevp/HFKKyaNDF3eJXWusPnmbflNBdSMmniaMe0kPb079AUKOgR1qVLF9zd3XniiScIDAxk69attGnThhYtWnDlyhVjIvD29mb06NHGhvqxY8dKe4QoF2mTECYnU4CXzbrD53llzTEyc/KMZXbWlswZ6GFMFEJAFWyTUEoNUEr9BgQAm5RSW0wTlqjKHJ58Eqe3ZhXcOSiFVZMmkiDuYN6W00USBEBmTh7ztpw2U0RC/KlcSUJrvVZr3UxrbaO1fkhrHWKqwETV5vDkk7TdHsEjJ0/QdnuEJIg7uJCSWaZyUbKbR6fv3LmTvn37mjmiqk+m5RDCzJo42pWpXJRMpjAxPUkSQpjZtJD22FkXnUPKztqSaSHtzRRR1XXz6PRp06aRnp7O4MGDcXFx4emnnzZ2EY6IiKBDhw54eHgwZswYsrOzjce7urri6enJ1KlTAUhOTmbQoEH4+vri6+vLnj17zHZ9ZqG1rvAfHx8fLYT409pDv+nOcyK08/SNuvOcCL320G/mDqlK+uWXX7Sbm5vWWusdO3boOnXq6F9//VXn5eVpf39/HRkZqTMzM3WzZs306dOntdZajxw5Un/wwQf68uXLul27djo/P19rrfUff/yhtdZ6+PDhOjIyUmut9blz57SLi0vFX1ghIEpX8Pe1TPAnRCXQv0NT6cl0H/j5+dGsWcESuwaDgYSEBOzt7WnZsiXt2rUDYNSoUSxevJhJkyZha2vL2LFj6dOnj7E9Y9u2bZw4ccJ4zqtXr5KWloa9vX3FX5AZSJIQQlRbNjZ/zgBgaWlJbm5uiaPSraysOHDgABEREYSGhrJo0SK2b99Ofn4++/btw87uwWwjkjYJIUS1cevo9OK4uLiQkJDAzz//DMDnn39Ot27dSE9PJzU1ld69e7NgwQJiYmIA6NmzJ4sWLTIef6P8QSF3EkKIKuXbpCvMiU/kfHYOTW2seaWVE4Ma1wOKjk63s7Mrdr0SW1tbVq5cyZAhQ8jNzcXX15cJEyZw5coVnnrqKbKystBa88EHHwCwcOFCXnjhBTw9PcnNzSUoKIilS5dW6DWbk0lGXJeVjLgWQtyLb5OuMPX0r2Tm//m9ZWehmN++uTFRVGdVbsS1EEJUpDnxiUUSBEBmvmZOvEweeb9IkhBCVBnns3PKVC7KT5KEEKLKaGpjXaZyUX6SJIQQVcYrrZyws1BFyuwsFK+0cjJTRNWf9G4SQlQZNxqnS+rdJExPkoQQokoZ1LieJIUKJI+bhBBClEiShBBCiBKVd2W6eUqpU0qpo0qptUopRxPFJYQQohIo753EfwF3rbUncAZ4pfwhCSGEqCzKu3zpVq11buHbn4Bm5Q9JiNLJy8u7+05CiHIxZZvEGOB7E55PVDP9+/fHx8cHNzc3li1bxpIlS/jHP/5h3L5q1SpefPFFAL744gv8/PwwGAz87W9/MyaE2rVr88Ybb9CpUyf27dvHrFmz8PX1xd3dnfHjxxungT548CCenp4EBAQwbdo03N3dgYLEMm3aNHx9ffH09OTjjz8GIDExkaCgIAwGA+7u7kRGRlZk1QhRed1tVSJgGxBbzM9TN+3zGrCWwgkDSzjPeCAKiHr44YdNulqTqBouX76stdb62rVr2s3NTSclJenWrVsbt/fq1UtHRkbqEydO6L59++rr169rrbV+/vnn9aeffqq11hrQYWFht51Ta61HjBih169fr7XW2s3NTe/Zs0drrfX06dONq5V9/PHH+q233tJaa52VlaV9fHx0fHy8nj9/vn777be11lrn5ubqq1ev3pc6EKI8qIwr02mte9xpu1JqFNAXCC68iJLOswxYBgWzwN7tc0X1s3DhQtauXQvAr7/+yi+//EKrVq346aefaNu2LadPn6ZLly4sXryY6OhofH19AcjMzKRRo0ZAwcIxgwYNMp5zx44dvPfee1y7do0rV67g5uZGYGAgaWlpdO7cGYC//vWvbNy4EYCtW7dy9OhRwsPDAUhNTeXs2bP4+voyZswYcnJy6N+/PwaDoaKqRYhKrVyD6ZRSvYDpQDet9TXThCSqo507d7Jt2zb27dtHzZo16d69O1lZWQwdOpSvv/4aFxcXBgwYgFIKrTWjRo1izpw5t53H1tYWS0tLALKyspg4cSJRUVE0b96cmTNnGtcCKInWmv/85z+EhITctm3Xrl1s2rSJkSNHMm3aNJ555hnTVYAQVVR52yQWAfbAf5VSMUqpB2clDlEmqamp1K1bl5o1a3Lq1Cl++uknAAYOHMi6dev46quvGDp0KADBwcGEh4dz8eJFAK5cucK5c+duO2dWVhYADRo0ID093Xh3ULduXezt7Y2fERoaajwmJCSEJUuWkJNTMGvomTNnyMjI4Ny5czRq1Ihx48bx3HPPcejQoftUE0JULeW6k9BatzFVIKLqO3r0KBEREaSmpuLg4EBwcDCenp4A9OrVi6VLl+Lp6Un79u3x9/cHCr7QXV1dOXHiBH5+fgC4urry9ttv07NnT/Lz87G2tmbx4sW0aNGiyOc5Ojoybtw4PDw8cHZ2Nj6eAvjkk08YN24ctWrVonv37jg4OAAwduxYEhIS8Pb2RmtNw4YNWbduHTt37mTevHlYW1tTu3ZtPvvss4qoMiEqPVmZTpjE0aNH2bBhg/EvdABra2uefPJJY6KoSOnp6dSuXRuAuXPnkpiYyIcffljhcQhhSrIynaiyIiIiiiQIgJycHCIiIswSz6ZNm4p0Z3399dfNEocQVZ3MAitMIjU1tUzl99vQoUONbRxCiHsndxLCJG488y9tuRCiapAkIUwiODgYa+uiS0haW1sTHBxspoiEEKYgj5uESdxonC6pd5MQomqSJCFMxtPTU5KCENWMPG4SQghRIkkSQgghSiRJQgghRIkkSQghhCiRJAkhhKhkZs6cyfz588t8nFJqllLqjss7lJX0bhJCiGpCa/2Gqc8pdxJCCFEJzJ49m/bt29OjRw9Onz4NQFxcHL169cLHx4fAwEAAW6WUg1IqQSllAaCUqqmU+lUpZa2UWqWUGlxY7quU2quUOqKUOqCUsldKWSql5imlDiqljiql/na3uOROQgghzCw6OprQ0FAOHz5Mbm4u3t7e+Pj4MH78eJYuXUrbtm3Zv38//v7+D2utU5VSR4BuwA7gSWCL1jpHKQWAUqoGEAYM1VofVErVATKB54BUrbWvUsoG2KOU2qq1/qWk2CRJCCGEmUVGRjJgwABq1qwJQL9+/cjKymLv3r0MGTLk5l1vzH0TBgylIEkMAz665ZTtgUSt9UEArfVVAKVUT8Dzxt0G4AC0BSRJCCFEZXbjLuCG/Px8HB0diYmJuXmf44Uv1wNzlFL1AB9g+62nA4pbLEgBL2qtt5Q2rnK1SSil3ip8rhWjlNqqlGpSnvMJIcSDKCgoiLVr15KZmUlaWhobNmygZs2atGzZkm+++QbgxtrtdoWv04EDwIfARq113i2nPAU0UUr5AhS2R1gBW4DnlVLWheXtlFK17hRbeRuu52mtPbXWBmAjYPKWdSGEqA42xW+iZ3hPPD/1pGd4TzbFbzJu8/b2ZujQoRgMBgYNGnSjkZrVq1fzySef4OXlhZubG4DjTacMA0YU/i5Ca32dgsdR/ylsv/gvYAv8H3ACOKSUigU+5i5PlEy2fKlS6hXgYa3183fbV5YvFUI8SDbFb2Lm3plk5WUZy2wtbZnZeSZ9WvUp9Xmq5PKlSqnZSqlfgacx851EUlISw4YNo3Xr1ri6utK7d2+WLVtG3759S3X8G2+8wbZt20rcvm7dOk6cOFHq/YUQAuDDQx8WSRAAWXlZfHio8q+7ftckoZTappSKLebnKQCt9Wta6+bAamDSHc4zXikVpZSKSk5ONt0VFNJaM2DAALp3705cXBwnTpzgnXfe4ffffy/V8Xl5ecyaNYsePUoerHhrkrjb/kIIAZCUkVSm8srkrklCa91Da+1ezM93t+z6JTDoDudZprXuqLXu2LBhw/LGfZsdO3ZgbW3NhAkTjGUGg4HAwEDS09MZPHgwLi4uPP300zcagHB2dmbWrFl07dqVb775htGjRxMeHg7AjBkzcHV1xdPTk6lTp7J3717Wr1/PtGnTMBgMxMXFFdl/1qxZ+Pr64u7uzvjx442f0b17d6ZPn46fnx/t2rUjMjLS5NcuhKjcGtdqXKbyyqS8vZva3vS2HwUt6mYRGxuLj49PsdsOHz7MggULOHHiBPHx8ezZs8e4zdbWlt27dzNs2DBj2ZUrV1i7di3Hjx/n6NGjvP7663Tu3Jl+/foxb948YmJiaN26dZHPmDRpEgcPHiQ2NpbMzEw2btxo3Jabm8uBAwdYsGAB//rXv0x85UKIyu4l75ewtbQtUmZractL3i+ZKaLSK2+bxNzCR09HgZ5ApbxiPz8/mjVrhoWFBQaDgYSEBOO2oUOH3rZ/nTp1sLW1ZezYsaxZs8Y4wOVOduzYQadOnfDw8GD79u0cP37cuG3gwIEA+Pj4FPlsIcSDoU+rPszsPBOnWk4oFE61nMrcaG0u5RpMp7Uu8fFSRXNzczM++rmVjY2N8bWlpSW5ubnG97Vq3d5F2MrKigMHDhAREUFoaCiLFi1i+/Zbx6r8KSsri4kTJxIVFUXz5s2ZOXMmWVl/NlLd+PxbP1sI8eDo06pPlUgKt6o2E/w99thjZGdns3z5cmPZwYMH+fHHH8t8rvT0dFJTU+nduzcLFiwwjni0t7cnLS3ttv1vJIQGDRqQnp5eYrISQoiqpspMy3EycgeRoZ+RdvkS9vUbEDjsGR4JfNS4XSnF2rVrmTJlCnPnzsXW1hZnZ2f69+9f5s9KS0vjqaeeIisrC601H3zwAQDDhg1j3LhxLFy4sEgicHR0ZNy4cXh4eODs7Iyvr2+5r1cIISoDkw2mK4uyDqY7GbmDrcsWkXs921hmVcOGnuMnFUkUQghRnVXJwXQVITL0syIJAiD3ejaRoZ+ZKSIhhHgwVIkkkXb5UpnKhRBCmEaVSBL29RuUqVwIIYRpVIkkETjsGaxq2BQps6phQ+CwZ8wUkRBCPBiqRO+mG43Td+rdJIQQwvSqRJKAgkQhSUEIISpWlXjcJIQQwjwkSQghhCiRJAkhhBAlMsuIa6VUMnCujIc1AKriwAiJu+JUxZhB4q5IVTFm+DPuFlpr0y/IcwdmSRL3QikVVdHD0U1B4q44VTFmkLgrUlWMGcwbtzxuEkIIUSJJEkIIIUpUlZLEMnMHcI8k7opTFWMGibsiVcWYwYxxV5k2CSGEEBWvKt1JCCGEqGCVNkkopYYopY4rpfKVUiW26iulEpRSx5RSMUqp0q9kdJ+UIe5eSqnTSqmflVIzKjLGEuKpp5T6r1LqbOHvuiXsZ/b6vlvdqQILC7cfVUp5myPOW5Ui7u5KqdTCuo1RSr1hjjhviWmFUuqiUiq2hO2Vta7vFndlrOvmSqkdSqmThd8hLxWzT8XXt9a6Uv4AjwDtgZ1AxzvslwA0MHe8ZYkbsATigFZADeAI4GrmuN8DZhS+ngG8WxnruzR1B/QGvgcU4A/srwT/LkoTd3dgo7ljvSWmIMAbiC1he6Wr61LGXRnr2gnwLnxtD5ypDP+2K+2dhNb6pNb6tLnjKKtSxu0H/Ky1jtdaXwdCgafuf3R39BTwaeHrT4H+5gvljkpTd08Bn+kCPwGOSimnig70FpXxv/ldaa13AVfusEtlrOvSxF3paK0TtdaHCl+nASeBprfsVuH1XWmTRBloYKtSKlopNd7cwZRSU+DXm97/xu3/GCraQ1rrRCj4xwo0KmE/c9d3aequMtZvaWMKUEodUUp9r5Ryq5jQyqUy1nVpVdq6Vko5Ax2A/bdsqvD6NutU4UqpbUDjYja9prX+rpSn6aK1vqCUagT8Vyl1qvCviPvGBHGrYsruezezO8VdhtNUeH3fojR1Z5b6vYvSxHSIgmkX0pVSvYF1QNv7HVg5Vca6Lo1KW9dKqdrAt8AUrfXVWzcXc8h9rW+zJgmtdQ8TnONC4e+LSqm1FNzW39cvLRPE/RvQ/Kb3zYAL5TznXd0pbqXU70opJ611YuHt68USzlHh9X2L0tSdWer3Lu4a081fCFrrzUqpj5RSDbTWlXmuocpY13dVWetaKWVNQYJYrbVeU8wuFV7fVfpxk1KqllLK/sZroCdQbG+GSuYg0FYp1VIpVQMYBqw3c0zrgVGFr0cBt90RVZL6Lk3drQeeKewJ4g+k3niUZkZ3jVsp1VgppQpf+1Hw/+flCo+0bCpjXd9VZazrwng+AU5qrf9dwm4VX9/mbtG/Q0v/AAqyZjbwO7ClsLwJsLnwdSsKeokcAY5T8Lin0set/+ylcIaCHi+VIe76QARwtvB3vcpa38XVHTABmFD4WgGLC7cf4w694ypZ3JMK6/UI8BPQuRLE/BWQCOQU/rt+rorU9d3irox13ZWCR0dHgZjCn97mrm8ZcS2EEKJEVfpxkxBCiPtLkoQQQogSSZIQQghRIkkSQgghSiRJQgghRIkkSQghhCiRJAkhhBAlkiQhhBCiRP8flIoy55UiGbcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,3))\n",
    "for i, word in enumerate(vocabs[:20]): #loop each unique vocab\n",
    "    x, y = get_embed(word)\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(word, xy=(x, y), xytext=(5, 2), textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d9bc342-efb9-4656-8563-2f0850497a41",
   "metadata": {},
   "source": [
    "Comparing both negative and normal skipgram with real corpus, normal uses 3:58 mins but negative uses 3:51 mins, but normal cannot reach epoch of 5000 I am not sure why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9e0ccbd-1852-463b-950d-ed5695ad609e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50853349-3712-45ba-a329-56b81b65773f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}