paddlenlp_updated_notes_English/WordEmbedding/4-training word embeddings using skip-gram with negative sampling in paddle.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"PaddlePaddle 2.1.0 (Python 3.5)","language":"python","name":"py35-paddle1.2.0"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"},"colab":{"name":"4-training word embeddings using skip-gram with negative sampling in paddle.ipynb","provenance":[{"file_id":"1zBf08Z6uzf4RY2FNRohitImWmhg0dkQ0","timestamp":1627545204374}],"collapsed_sections":[]}},"cells":[{"cell_type":"code","metadata":{"id":"hbkykuy9F-vg"},"source":["# Author: Zhengxiang (Jack) Wang \n","# Date: 2021-07-28\n","# GitHub: https://github.com/jaaack-wang \n","# About: Training word embeddings using skip-gram with negative sampling in paddle"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"SxC-PLJLF-vj"},"source":["# Overview\n","\n","In this notebook, we will train word embeddings employing Skip-gram with negative sampling algorithms as described by [Mikolov et. al. (2013a)](https://arxiv.org/pdf/1301.3781.pdf) and [Mikolov et. al. (2013b)](https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf). More concretely, we will learn the steps (e.g., text normalization, subsampling, negative sampling, and mini-batches) needed to preprocess a corpus and how to use `paddle` to construct a skip-gram neural network model to train word embeddings out of the preprocessed corpus.\n","\n","<br>\n","\n","Related contents: <br>\n","\n","- [loading pre-trained word embedding in paddlenp](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing)\n","- [calculating text cosine similarity in paddlenlp](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing)\n","- [embedding visualization using paddlepaddle tool](https://colab.research.google.com/drive/1B9pcYR9fVvmB1pPWiIqb0u_WmxlY--T8?usp=sharing)\n","- [embedding visualization using tensorflow tool](https://colab.research.google.com/drive/1HZdDA_TzdJhGo_uIUSa84rP6yQjHvMT3?usp=sharing)\n","\n","\n","<br>\n","\n","\n","<table align=\"right\">\n","  <td>\n","    <a target=\"_blank\" href=\"https://colab.research.google.com/drive/16xghJX6WGEdlY3f2MamI5W7SJ6cniW4U?usp=sharing\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab </a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://github.com/jaaack-wang\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> Author's GitHub </a>\n","  </td>\n","  <td>\n","    <a href=\"https://docs.google.com/uc?export=download&id=16xghJX6WGEdlY3f2MamI5W7SJ6cniW4U\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /> Download this notebook </a>\n","  </td>\n","</table> \n","\n","\n","<br>\n"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"_eglsVagF-vk"},"source":["# Table of Contents\n","- [1. Word representation and word embedding](#1)\n","  - [1.1 Word representation](#1-1)\n","  - [1.2 General model to learn word embeddings](#1-2)\n","  - [1.3 Skip-gram and negative sampling](#1-3)\n","- [2. Word embedding training using skip-gram with negative sampling algorithms](#2)\n","  - [2.1 General procedure](#2-1)\n","  - [2.2 Data loading and preprocessing](#2-2)\n","  - [2.3 Model configuration](#2-3)\n","  - [2.4 Training](#2-4)\n","- [3. Things to do next](#3)\n","  - [3.1 Save the trained embeddings](#3-1)\n","  - [3.2 Visualization](#3-2)\n","  - [3.3 Word analogy test](#3-3)\n","- [4. References](#4)"]},{"cell_type":"code","metadata":{"id":"EsIKwKm9F-vl","outputId":"b0c0f32d-ac71-4aaf-a4b6-22ff8911fe71"},"source":["# first, make sure that you have paddlepaddle installed\n","!pip install --upgrade paddlepaddle"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Looking in indexes: https://mirror.baidu.com/pypi/simple/\n","Collecting paddlepaddle\n","\u001b[?25l  Downloading https://mirror.baidu.com/pypi/packages/91/e9/b391472d83a8c740f8de247d3856c19a8db01051765f4965bcaf9d03689c/paddlepaddle-2.1.1-cp37-cp37m-manylinux1_x86_64.whl (108.9MB)\n","\u001b[K     |███████████████████▋            | 66.7MB 8.9MB/s eta 0:00:05\u001b[?25hRequirement already satisfied, skipping upgrade: astor in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (0.8.1)\n","Requirement already satisfied, skipping upgrade: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (1.15.0)\n","Requirement already satisfied, skipping upgrade: decorator==4.4.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (4.4.2)\n","Requirement already satisfied, skipping upgrade: Pillow in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (7.1.2)\n","Requirement already satisfied, skipping upgrade: requests>=2.20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (2.22.0)\n","Requirement already satisfied, skipping upgrade: gast>=0.3.3; platform_system != \"Windows\" in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (0.3.3)\n","Requirement already satisfied, skipping upgrade: numpy>=1.13; python_version >= \"3.5\" and platform_system != \"Windows\" in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (1.20.3)\n","Requirement already satisfied, skipping upgrade: protobuf>=3.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle) (3.14.0)\n","Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (3.0.4)\n","Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (1.25.6)\n","Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (2019.9.11)\n","Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (2.8)\n","Installing collected packages: paddlepaddle\n","Successfully installed paddlepaddle-2.1.1\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"faQ8lBYdF-vm"},"source":["**Dependencies**"]},{"cell_type":"code","metadata":{"id":"yabsCBtOF-vm"},"source":["import os\n","import requests\n","from collections import Counter \n","import math\n","import random\n","import numpy as np\n","import paddle\n","from paddle.nn import Embedding\n","import paddle.nn.functional as F"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"pQDWB9dPF-vn"},"source":["<a name='1'></a>\n","# 1. Word representation and word embedding\n","\n","<a name='1-1'></a>\n","### 1.1 Word representation\n","\n","One-hot encoding and word embedding are one of two commonest ways to represent words in machine(/deep)-learning-based natural language processing. In both approches, a word is represented by a vector that has certain dimensions $N$(number of elements in the vector) and a vocabulary list can thus be represented as a vocabulary size $V$ by dimensions $N$ matrix (i.e., $R^{V*N}$).\n","\n","<br>\n","\n","For one-hot encoding, a word vector is always made up of one 1 and many 0s such that the sum of each vector all equals 1, as exemplified by the figure below. This approach is very easy to implement, but does not consider any potential relations among words as the relation between any two one-hot encoded words is nothing but identical (e.g., the distance between two word vectors is always $\\sqrt{2}$). Moreover, it can also be computationally expensive to perform matrix multiplication as one-hot encoding requires the word matrix to be $V$ by $V$ (i.e., $N = V$) dimensions, provided that a real-world NLP problem usaully has tens of thousands or even larger vocabulary size. \n","\n","<img src='https://i.gzn.jp/img/2018/10/12/language-translator-deep-learning/008.png' height='400' width='600'>\n","\n","In contrast, word embedding is a much more economic and efficient way to represent words as it uses much smaller $N$ to be the dimension of each vector, which typically ranges from 50 to 300. Instead of using only 1 and 0, word embedding uses floating numbers to fill in each word vector such that each word vector has different relations with other word vectors. Ideally, similar words should have closer relations and dissimilar words should have more distant relations (usaully in terms of their **cosine similarity** score that measures the angel between two vectors). It has also been shown by [previous research](https://arxiv.org/pdf/1301.3781.pdf) that two word pairs that has similar internal relation (e.g., semantic, syntactic) should have large cosine similarity score (or visually, parallel to each other), such as the classic example: $women \\rightarrow queen \\approx man \\rightarrow king$. The figure below is a visual illustration of what ideal word embeddings should look like in a 2-D dimension-reduced plot. \n"," \n","\n","![](https://ai-studio-static-online.cdn.bcebos.com/00ba55f7304e4f97942165cf1deb946ced404a19325d40969b7c220e30cf527e)\n","\n","<br>\n","\n","Similar to tokenization to text processing, word embedding has also been recgonized as one of the most fundamental technique in natuural language processing and is essential for many downstream machine(/deep)-learning-based natural language processing tasks, such as Named Entity Recognition, Information Retrieval, and Machine Translation etc."]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"mcDcYnrRF-vn"},"source":["<a name='1-2'></a>\n","### 1.2 General models to learn word embeddings\n","\n","To train word embeddings, three models are widely used, namely [Continuous Bag-of-Words (CBOW) model, Skip-gram model](), and [Global Vectors for Word Representation (GloVe) model](https://nlp.stanford.edu/pubs/glove.pdf). According to Google Scholar, the papers where these models were origianlly proposed all have over 20,000 citations. The seminal work by [Bengio et. al., 2003](https://www.jmlr.org/papers/volume3/tmp/bengio03a.pdf), which proposes a Nerual-Network-based probabilistic Language Model (NNLM), can be seen as a prototype for CBOW and Skip-gram models as they all rely on local context and shallow neural network to train word embeddings. GloVe is different than these models in that it additionally leverages global word-word co-occurrence counts to make up for the constrains of the previous models only looking at local contexts. "]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"apggiEsaF-vo"},"source":["<a name='1-3'></a>\n","### 1.3 Skip-gram model architecture and negative sampling\n","\n","As mentioned above, Skip-gram model is trained on the local contexts of target words, just like CBOW model. But like the latter, which uses the nearby words (contexts) of target words to predict the target words, the Skip-gram model uses the target words to predict their nearby words. Nearby words are usually defined by a window size that spans around the target words. For example, if the window size is 3, then the contexts are words that are in both the left side and the right side of the target words up to 3 words. The model architectures of these two models can be illustrated by the following figure, which shows that both models are made up of an input layer, a hidden layer and an output layer, a simple shallow neural network structure. \n","\n","<img src='https://drive.google.com/uc?export=view&id=1jw9Tae0avHcWY7RKrxigOMuq8f-kbWsk'>\n","\n","<br>\n","\n","For Skip-gram model, the training process goes like this: the input layer is a 1 by $V$ (vocabulary size) **one-hot** row vector, which are then multiplied by a $V$ by $N$ (i.e., number of vector dimensions or features) weight matrix that would result in a 1 by $N$ row vector in the hidden layer. This row vector is then multiplied by another $N$ by $V$ weight matrix followed by a softmax function that will return a 1 by $V$ row vector. This row vector stores the probabilities of the context wors for the target word. To optimize our model, we want to make sure that the top the top $C$ candidates $C$ (i.e., number of context words to predict) match with the real context words as much as possible by minimizing the loss of our model during backpropagation. The architecture of the Skip-gram model is shown in the figure below. \n","\n","<img src='http://www.claudiobellei.com/2018/01/06/backprop-word2vec/Skipgram.png' width='400' height='400'>\n","\n","<br>\n","\n","Please note that the operation between the input layer and the hidden layer is just a **simple linear activation** that takes the form of $A^{(1)} = Z^{(1)} = xW^{(1)}$. <ins>**This $V$ by $N$ weight matrix is actually the word embeddings we aim to optimize and obtain for the entire vocab!**</ins> The linear activtaion can be seen as a transformation that extracts a target word embedding from the embedding matrix because the multiplication between the input one-hot vector and the embedding weight matrix will locate and return a row vector of the embedding matrix whose position matches with the postion of 1 in the one-hot vector. \n","\n","<br>\n","\n","Moreover, since the vocabulary size for a corpus which we can train word embeddings on is usually very large, the last step of the calculation between the hidden layer and the output is very computational costly. To solve this problem, multiple methods have been proposed, such as [hierarchical softmax](https://ruder.io/word-embeddings-softmax/) and negative sampling. For example, the negatve sampling method will randomly select $K$ non-context words from the vocabulary for every positive context word of the target word to form comparisons. As a result, we can change the Skip-gram model from predicting context words from a large vocabulary into one that only determines whether the context word is a real context word or a just negative sample for a limited amount of times (number of context words plus $K$ times of the context words). Therefore, the last weight matrix can be reduced to a $N$ and 1 vector and the last activation function in the output can be a sigmoid function. This will significantly speed up the entire training process. "]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"d41aLjswF-vp"},"source":["<a name='2'></a>\n","# 2. Word embedding training using skip-gram and negative sampling algorithms\n","\n","<a name='2-1'></a>\n","### 2.1 General procedure\n","\n","There are three steps we need to take to train word embeddings: \n","- First, we need to load a corpus that we train word embedding on and preprocess the corpus so that it can be trained. \n","- Then, we need to build the Skip-gram model to train the preprocessed corpus. In this notebook, we will use `paddle` to construct the model. \n","- Lastl, we need to train word embeddings using the corpus and the model by minimizing the prediction loss for the context words. \n","\n","<br>\n","\n","In the following parts of the notebook, we will mostly spend time preprocessing the corpus because utilizing `paddle` package frees us from writing algorithms for the Skip-gram model  from bottom up. Training the model only takes time, so we will not discuss it too much either. \n","\n","<br>\n","\n","Moreover, it should also be noted that, with various free deep learning frameworks available that can save us from building deep learning models with lengthy code, preprocessing a corpus is often actually more important than building those models. "]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"-QHAmrpbF-vp"},"source":["<a name='2-2'></a>\n","### 2.2 Data loading and preprocessing\n","\n","<a name='2-2-1'></a>\n","#### 2.2.1 Data loading\n","- We will use a corpus of 100 MB collected from wikipedia to train word embeddings. \n","- This corpus is in a way classic as it was also used by Mikolov as demo text in his original [word2vec project](https://github.com/tmikolov/word2vec)\n","- We will build a simple function to download the file from a given url. "]},{"cell_type":"code","metadata":{"id":"6gz2QodpF-vp"},"source":["def get_file_from_url(url, file_name, file_dir='./'):\n","  '''A simple function that downloads file from url and returns \n","  the saved file path.\n","  '''\n","  file_path = file_dir + file_name\n","\n","  if os.path.exists(file_path):\n","    return file_path\n","  \n","  r = requests.get(url)\n","  try:\n","    content = r.content\n","  except:\n","    raise FileNotFoundError('File not found or cannot be downloaded.')\n","\n","  with open(file_path, 'wb') as f:\n","    f.write(content)\n","  f.close()\n","  \n","  print(file_name + ' saved in ' + file_path)\n","  return file_path"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"_yjJ5dLXF-vq","outputId":"c7c17b43-00cc-42e3-a7d2-c71ecd391a49"},"source":["file_path = get_file_from_url(\n","    url='https://dataset.bj.bcebos.com/word2vec/text8.txt',\n","    file_name = 'text8.txt')\n","\n","# read the corpus\n","text8 = open(file_path, 'r').read()\n","# print the first 100 characters \n","# the corpus has already been unpunctuated,\n","# so we can tokenize it by simply splitting it by spaces.\n","print('\\nThe first 300 characters of text8: \\n\\n', text8[:300])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["\n","The first 300 characters of text8: \n","\n","  anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organiz\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"NQwguIsDF-vq"},"source":["<a name='2-2-2'></a>\n","#### 2.2.2 Data preprocessing\n","**First steps:**\n","- We will need to first tokenize the corpus and get the vocabulary out of it.\n","- Then, we will create the following three dictionaries:\n","  - `word2id_dict`, which maps words to their ids, will be used to convert the corpus into integers for ease of computer processing;\n","  - `id2word_dict`, which maps word ids to words, will be used as lookup table to convert word ids back to words;\n","  - `word2id_fdist`, which stores the frequency distribution of words (represented by theird ids), will be used for subsumpling.\n","- In theory, the word ids can be assigned according to the occurrence of new words in the corpus ('first come first serve'), but in practice they are usually in the descending order of word frequency. In other words, the more frequently a word occurs, the lower its id will be. This arrangement makes the word ids more meaningful and easier to manage. \n"]},{"cell_type":"code","metadata":{"id":"aZQlDd76F-vq","outputId":"858ac4e7-002a-4919-f878-fa133f4ea1fc"},"source":["# tokenize the corpus \n","# the corpus is already preprocessed, but it \n","# never hurts to strip extra spaces and lower case\n","# the corpus before tokenization\n","tokens = text8.strip().lower().split()\n","print('The first 50 words of the corpus:')\n","print(tokens[:50])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["The first 50 words of the corpus:\n","['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst', 'the', 'term', 'is', 'still', 'used', 'in', 'a', 'pejorative', 'way', 'to', 'describe', 'any', 'act', 'that', 'used', 'violent', 'means', 'to', 'destroy', 'the']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"6z1VFhx9F-vr","outputId":"61413333-7853-4e4e-898e-f8a851d3ce27"},"source":["# create the dictionaries\n","\n","# we will first create a word_fdist that maps \n","# words to their frequency from highest to lowest\n","word_fdist = dict(Counter(tokens).most_common())\n","\n","# create the three dictionaries\n","word_to_id = {}\n","id_to_word = {}\n","word2id_fdist = {}\n","\n","# we use integer indices (ids) to represent words\n","id = 0\n","for word, freq in word_fdist.items():\n","  word_to_id[word] = id\n","  id_to_word[id] = word\n","  word2id_fdist[id] = freq\n","  id += 1\n","\n","print('vocab size of the corpus: ', len(word_to_id))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["vocab size of the corpus:  253854\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"vFS1ukl9F-vs","outputId":"ac098af1-7af9-4754-c5b5-d98217c59328"},"source":["# let's randomly check 10 words along with their ids and frequency\n","\n","random_ids = [23, 768, 456, 5677, 6879, 987, 234435, 234, 4522, 78633]\n","tmp = '{0}\\t\\t{1:10}\\t{2}'\n","print(tmp.format('id', 'word', 'frequency'))\n","for id in sorted(random_ids):\n","  print(tmp.format(id, id_to_word[id], word2id_fdist[id]))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["id\t\tword      \tfrequency\n","23\t\twith      \t95603\n","234\t\tpopular   \t5967\n","456\t\tchange    \t3483\n","768\t\teffects   \t2194\n","987\t\tconditions\t1805\n","4522\t\temotional \t362\n","5677\t\tpossess   \t273\n","6879\t\tpaths     \t211\n","78633\t\tdocu      \t4\n","234435\t\trecrystallizes\t1\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"Ygehk-yUF-vt"},"source":["**Further steps:**\n","- Now, with the dictionaries ready, we want to convert the entire corpus into integers using the `word_to_id`. \n","- As we can see above, the frequency between frequently used words and infrequently used is very imbalanced. The [Zipf's law](https://en.wikipedia.org/wiki/Zipf%27s_law) states that the rank-frequency distribution is an inverse relation on a log-log scale for natural languages, something like the following. \n","\n","<img src='https://upload.wikimedia.org/wikipedia/commons/a/ac/Zipf_30wiki_en_labels.png' alt='A plot of the rank versus frequency for the first 10 million words in 30 Wikipedias in a log-log scale.' width='600' height='400'>\n","\n","- Compared to infrequent words, highly frequent words usually contain less contextual information. Scholars (e.g., [Mikolov et. al., 2013b](https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf)) have proposed to reduce the occurence of highly used words by means of subsampling so that a target word can learn much more meaningful word embedding during training. The subsampling formula is given by a probability function that filters frequent words:\n","\n","$$P(w_i) = 1 - \\sqrt{\\frac{t}{f(w_i)}}$$\n","\n","where $f(w_i)$ is the frequency of word $w_i$ (divided by the size of the corpus) and t is a chosen threshold, typically around $10^{-5}$. For example, for $t=10^{-5}$, if a word occurs 1 per 1,000 words ($f=10^{-3}$), then the possibility of it being filtered equals \n","\n","$$1 - \\sqrt{\\frac{10^{-5}}{10^{-3}}} = 1 - \\sqrt{0.01} = 0.9$$\n","\n","whereas a word with frequency of 1 per $10^5$ words ($f=10^{-5}$) will never be discarded as $P(w_i)$ will equal 0 in this case. Therefore, the greater $f(w_i)$ is than the threshold $t$, the higher probability it will get reduced.  \n","\n","<br>\n","\n","- Subsampling will aggressively subsample words whose frequency is greater than $t$. Let's say the most frequently used word occurs 5 per 100 words, then it will be reduced to 1.4% of its original size with $t=10^{-5}$. In a big corpus (e.g., billion words), even with such great reduction, this 1.4% will still be much larger than those low frequency words whose frequencies do not pass $t=10^{-5}$, so that the original ranking of the frequencies can be preserved. However, for a small corpus, such as one of 100,000 words, such reduction will change the original ranking and severely damage the original linguistic information of the corpus. Therefore, a good threshold is not an unimportant decision to make.\n","\n","- As subsampling reduces the corpus size, it also speeds up the computation for training word embeddings. \n","\n","- Subsampling might not be entirely necessary as the imbalanced nature of word usage in natural languages can also be something a word embedding algorith should learn. "]},{"cell_type":"code","metadata":{"id":"kT1pYWBJF-vt","outputId":"c0f22d36-ee7e-4fc6-88b3-a6ece2e01fa2"},"source":["# convert the corpus into integer indices using word2id_dict\n","text8 = [word_to_id[word] for word in tokens]\n","print('First 50 \"words\" of the converted corpus:')\n","print(text8[:50])\n","# also check the corpus size before subsampling\n","print('\\nCorpus size before subsampling: ', len(text8))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["First 50 \"words\" of the converted corpus:\n","[5233, 3080, 11, 5, 194, 1, 3133, 45, 58, 155, 127, 741, 476, 10571, 133, 0, 27349, 1, 0, 102, 854, 2, 0, 15067, 58112, 1, 0, 150, 854, 3580, 0, 194, 10, 190, 58, 4, 5, 10712, 214, 6, 1324, 104, 454, 19, 58, 2731, 362, 6, 3672, 0]\n","\n","Corpus size before subsampling:  17005207\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"OU4giEddF-vu"},"source":["# define the subsampling function\n","# we use random and math because they run faster than the related numpy functions\n","\n","def subsample(converted_text, word2id_fdist, threshold=1e-5, seed=323):\n","  '''Subsamples the converted corpus. \n","\n","  Args:\n","    converted_text (list): text converted into a list of integer indices.\n","    word2id_fdist(dict): frequency distribution disctionary of words represented by ids.\n","    threshold (float): a threshold float for the discard possibility function.\n","\n","  Returns: subsampled text (a list of integer indices.)\n","  '''\n","  def to_discard(word):\n","    # a / (b / c) = a / b * c\n","    p = 1 - math.sqrt(threshold / word2id_fdist[word] * size)\n","    return random.random() < p\n","  \n","  # the seed is just for the replicability of this notebook \n","  random.seed(seed)\n","  size = len(converted_text)\n","  # word here means word_id\n","  return [word for word in converted_text if not to_discard(word)]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"3NDZLdhqF-vv","outputId":"c09062c2-7ab6-41f6-c5ca-52c0261d3c21"},"source":["# our corpus only has 17 million words, \n","# so we will choose a larger threshold value (1e-4 instead of 1e-5)\n","# meaning we will only subsample words that occurs over 1 time per 10000 words\n","# this reduces our corpus by 1/2 of the original size!\n","text8_sub = subsample(text8, word2id_fdist, 1e-4)\n","print('Subsampled corpus size:, ', len(text8_sub))\n","print('\\nFirst 50 words of the subsampled corpus:')\n","print([id_to_word[id] for id in text8_sub[:50]])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Subsampled corpus size:,  8743856\n","\n","First 50 words of the subsampled corpus:\n","['anarchism', 'originated', 'term', 'abuse', 'first', 'against', 'working', 'radicals', 'including', 'diggers', 'revolution', 'sans', 'culottes', 'of', 'the', 'revolution', 'whilst', 'pejorative', 'way', 'describe', 'any', 'act', 'violent', 'means', 'destroy', 'organization', 'has', 'taken', 'positive', 'label', 'by', 'self', 'anarchists', 'anarchism', 'derived', 'greek', 'archons', 'ruler', 'chief', 'king', 'anarchism', 'as', 'political', 'philosophy', 'belief', 'rulers', 'unnecessary', 'and', 'should', 'abolished']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"hoiO_KuRF-vw"},"source":["**Final steps:** <br>\n","\n","- With the original corpus converted into integer indices and subsampled, we can now build a dataset for training our word embedding model. As we use negative sampling to obtain non-context words (labelled as 0) for a particular word in a sequence (other words in the sequence are the context words to be predicted), for every training example, we need a `target word`, a `context word` and a `label` to indicate whether the `context` appears in the same sequence of text with the `target word`.\n","- To speed up our training process, we will use mini-batch gradient descent instead of batch gradient descent to learn word embeddings. We will build a function to split the corpus into mini bacthes. "]},{"cell_type":"code","metadata":{"id":"oNOwSSapF-vw"},"source":["# define a function to generate the training dataset that contains\n","# a list of (target word, context word, label) tuples\n","\n","def get_training_dataset(corpus, window_size, num_ns, vocab=None, seed=654): \n","  '''Generates a dataset for word embedding training using skip gram \n","  with negative sampling. \n","\n","  Args:\n","    - corpus (list): a list of integer indices that represent words.\n","    - window_size (int): the max distance for a context word\n","    - num_ns (int): number of negative examples to sample\n","    - vocab (list, set): a list or set of integer incides that denote the vocab of \n","        given corpus. If not given, vocab = set(corpus).\n","    - seed(int): defaults to 654, used only for the replicability of this notebook \n","\n","  Returns:\n","    A dataset that contains a list of ((target word, context word), label) tuples.\n","  '''\n","\n","  random.seed(seed)\n","\n","  if vocab is None:\n","    vocab = set(corpus)\n","\n","  dataset = []\n","  # we will scan every word in the corpus to get context words and negative samples\n","  for idx, word in enumerate(corpus):\n","    left_context = corpus[idx-window_size if idx-window_size>= 0 else 0: idx]\n","    right_context = corpus[idx+1: idx + window_size + 1]\n","    contexts = left_context + right_context\n","    for context in contexts:\n","      dataset.append((word, context, 1))\n","\n","      # for every positive example, we need num_ns negative examples\n","      \n","      # although we do not want to sample two or more identical negative examples, \n","      # as the vocab size is big enough, the ns_chosen part of code is only optional \n","      ns_chonsen = []\n","      for _ in range(num_ns):\n","        neg_example = random.choice(vocab)\n","        while neg_example in contexts or neg_example in ns_chonsen:\n","          neg_example = random.choice(vocab)\n","        else:\n","          dataset.append((word, neg_example, 0))\n","          ns_chonsen.append(neg_example)\n","\n","  return dataset\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GSgW3O2yN5Bz"},"source":["<a name='reminder'></a>\n","<font color='red'>\n","<b>Reminder:</b>\n","<font color='black'>\n","If you are not runing this notebook on GPU, it is recommended that you use a much smaller training dataset by only selecting 10,000 words from the text8_sub below. "]},{"cell_type":"code","metadata":{"id":"T2TuEApDF-vw","outputId":"d7879524-b542-4a3e-d5b0-c163cfc79163"},"source":["# If you are not running on a GPU that supports paddle, \n","# you can only select 10,000 words from the text8_sub for practice\n","# Just so you know: it took me about 50 minutes to train the current model\n","# on GPU (Tesla V100) provided by Baidu's AI studio. \n","# please also note that paddle gpu version cannot run on Google Colab\n","dataset = get_training_dataset(text8_sub[:4000000], \n","                               window_size=3, \n","                               num_ns=4, \n","                               vocab=list(id_to_word.keys()))\n","tmp = '{0:20}{1:20}{2:15}'\n","\n","# let's check the first 50 examples of our dataset\n","print(tmp.format('target word', 'context word', 'true context (1: yes; 0: no)'))\n","for i in range(50):\n","  print(tmp.format(id_to_word[dataset[i][0]],\n","                   id_to_word[dataset[i][1]], \n","                   dataset[i][2]))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["target word         context word        true context (1: yes; 0: no)\n","anarchism           originated                        1\n","anarchism           comsume                           0\n","anarchism           auerochse                         0\n","anarchism           behalten                          0\n","anarchism           channelised                       0\n","anarchism           term                              1\n","anarchism           militarev                         0\n","anarchism           grotzer                           0\n","anarchism           illite                            0\n","anarchism           ckis                              0\n","anarchism           abuse                             1\n","anarchism           nykyrka                           0\n","anarchism           walnut                            0\n","anarchism           tyseley                           0\n","anarchism           urbanski                          0\n","originated          anarchism                         1\n","originated          bohra                             0\n","originated          myodbc                            0\n","originated          riddance                          0\n","originated          psy                               0\n","originated          term                              1\n","originated          vello                             0\n","originated          lcs                               0\n","originated          chludov                           0\n","originated          opar                              0\n","originated          abuse                             1\n","originated          biosciences                       0\n","originated          habington                         0\n","originated          toeppen                           0\n","originated          mantik                            0\n","originated          first                             1\n","originated          disputations                      0\n","originated          commitments                       0\n","originated          darkstep                          0\n","originated          agr                               0\n","term                anarchism                         1\n","term                plani                             0\n","term                langen                            0\n","term                downplayed                        0\n","term                safajah                           0\n","term                originated                        1\n","term                buckintoopa                       0\n","term                guilino                           0\n","term                homoi                             0\n","term                networking                        0\n","term                abuse                             1\n","term                syncs                             0\n","term                thorvaldsen                       0\n","term                misjudgment                       0\n","term                neurosecretory                    0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"DrY7JJKWF-vx"},"source":["**Make batches**"]},{"cell_type":"code","metadata":{"id":"kpxZSf4PF-vx"},"source":["def make_batches(dataset, batch_size=64, keep_reminder=True, seed=None):\n","  '''Convert a dataset into mini batches. \n","\n","  Args:\n","    dataset (list): should contains a list of tuples (target, context, label)\n","    batch_size (int): the maximum size for a batch. \n","    keep_reminder (bool): Defaults to True. When set false, the last batch, if \n","      smaller than the given batch_size, will not be included. \n","    seed (int): defaults to None, used only for the replicability of this notebook  \n","  '''\n","  def read_dataset(start_idx, end_idx):\n","    targets, contexts, labels = [], [], []\n","    for t, c, l in dataset[start_idx: end_idx]:\n","      targets.append(t)\n","      contexts.append(c)\n","      labels.append(l)\n","    \n","    targets = paddle.to_tensor(targets, dtype=\"int64\")\n","    contexts = paddle.to_tensor(contexts, dtype=\"int64\")\n","    labels = paddle.to_tensor(labels, dtype=\"float32\")\n","    return targets, contexts, labels\n","  \n","  if seed is not None:\n","    random.seed(seed)\n","\n","  random.shuffle(dataset)\n","  data_size = len(dataset) \n","  \n","  num_complete_batches = data_size // batch_size\n","  for n in range(num_complete_batches):\n","    yield read_dataset(n * batch_size, (n + 1) * batch_size)\n","  \n","  if keep_reminder and data_size % batch_size:\n","    yield read_dataset(num_complete_batches * batch_size, data_size)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"yuY9M0iVF-vy","outputId":"6615a9c9-1641-4f05-d96e-3842280cfb77"},"source":["# check the batches\n","for batch in make_batches(dataset[:100], batch_size=30, seed=76):\n","  print(batch)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [3133, 3133, 45  , 5233, 5233, 3080, 3133, 3080, 3080, 194 , 3080, 3133, 194 , 3133, 194 , 3133, 194 , 194 , 45  , 3080, 3133, 3080, 3080, 5233, 5233, 194 , 194 , 45  , 3133, 3133]), Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [55198 , 23149 , 114538, 194   , 116274, 177719, 23147 , 178101, 247675, 3080  , 5233  , 155   , 119722, 71781 , 112970, 5233  , 45    , 70000 , 201734, 194   , 136228, 41520 , 128407, 148229, 164179, 5233  , 72512 , 5658  , 98287 , 37569 ]), Tensor(shape=[30], dtype=float32, place=CUDAPlace(0), stop_gradient=True,\n","       [0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.]))\n","(Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [3080, 3133, 194 , 194 , 5233, 194 , 3080, 3133, 3133, 3080, 3133, 194 , 45  , 3133, 3133, 3080, 3080, 3080, 3080, 3133, 194 , 3133, 45  , 3133, 3133, 3133, 3133, 5233, 3133, 45  ]), Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [245617, 194   , 72955 , 6370  , 165218, 3133  , 220124, 181597, 33391 , 120381, 156666, 173598, 1804  , 121603, 203733, 187281, 53878 , 232727, 74142 , 77382 , 185031, 3080  , 123839, 43782 , 30429 , 103507, 105647, 172974, 151242, 216572]), Tensor(shape=[30], dtype=float32, place=CUDAPlace(0), stop_gradient=True,\n","       [0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]))\n","(Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [3080, 194 , 194 , 5233, 3133, 5233, 3080, 5233, 3080, 194 , 5233, 5233, 194 , 3133, 3133, 45  , 45  , 3133, 194 , 5233, 5233, 3133, 3133, 3133, 45  , 194 , 5233, 3080, 3133, 194 ]), Tensor(shape=[30], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [17602 , 138496, 155   , 158792, 129284, 242840, 74313 , 26106 , 65599 , 70159 , 166799, 100647, 242912, 205002, 54212 , 154711, 190620, 45    , 192407, 3080  , 191304, 253655, 741   , 55328 , 3080  , 162733, 3133  , 227912, 89967 , 251654]), Tensor(shape=[30], dtype=float32, place=CUDAPlace(0), stop_gradient=True,\n","       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0.]))\n","(Tensor(shape=[10], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [45  , 194 , 3080, 194 , 5233, 194 , 3080, 194 , 194 , 194 ]), Tensor(shape=[10], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n","       [194   , 216328, 45    , 16924 , 82532 , 36532 , 3133  , 119656, 100622, 84643 ]), Tensor(shape=[10], dtype=float32, place=CUDAPlace(0), stop_gradient=True,\n","       [1., 0., 1., 0., 0., 0., 1., 0., 0., 0.]))\n"],"name":"stdout"},{"output_type":"stream","text":["/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. \n","Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n","  if data.dtype == np.object:\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"3MwUJHWqF-vy"},"source":["<a name='2-3'></a>\n","### 2.3 Model configuration\n","\n","The Skip-gram model (with negative sampling) configuration has already been explained in [1.3 Skip-gram model architecture](#1-3) and here is the quick recap:\n","\n","<br>\n","\n","- Input layer, which is a 1 by $V$ one-hot vector ($x$); \n","- Hidden layer, a 1 by $N$ vector (i.e., $A^{(1)}$, the embedding for the input word, $N$ is the vector dimension) gotten through linear activitation: $A^{(1)} = Z^{(1)}  = xW^{(1)}$ where $W$ is a $V$ by $N$ matrix. \n","- Output layer, a scalar (i.e., $y$) ranging from 0 to 1, gotten through sigmoid activitation: $y = \\sigma(A^{(1)}W^{(2)})$ where $N$ is a $N$ by 1 vector. \n","\n","<br>\n","\n","The reason why the last layer of the model uses sigmoid function instead of softmax function is due to the use of negative sampling, which turns the task of the model from predicting the most likely context words into deciding whether a given word is the context word or not. "]},{"cell_type":"code","metadata":{"id":"Ip_k12oGF-vy"},"source":["#construct the Skip-gram model\n","\n","#we need to inherit the paddle.nn.Layer for our model\n","class SkipGram(paddle.nn.Layer):\n","\n","    def __init__(self, vocab_size, embedding_size, init_scale=0.1):\n","      '''\n","      Args: \n","        vocab_size (int): the size of the vocabulary. \n","        embedding_size (int): the size of the word embedding dimension. \n","        init_scale (float): this defines the initial range of the word \n","          embedding weights, i.e., [-init_scale, init_scale].  \n","      '''\n","      super(SkipGram, self).__init__()\n","      self.vocab_size = vocab_size\n","      self.embedding_size = embedding_size\n","\n","      # we use Embedding inherited from paddle.nn.Layer to construct \n","      # the word embeddings between the input layer and the hidden layer. \n","      self.embedding = Embedding(\n","          num_embeddings = self.vocab_size,\n","          embedding_dim = self.embedding_size,\n","          weight_attr=paddle.ParamAttr(\n","              initializer=paddle.nn.initializer.Uniform(\n","                  low=-init_scale, high=init_scale)))\n","        \n","      # we use Embedding inherited from paddle.nn.Layer to construct \n","      # the word embeddings between the hidden layer and the output layer. \n","      self.embedding_out = Embedding(\n","          num_embeddings = self.vocab_size,\n","          embedding_dim = self.embedding_size,\n","          weight_attr=paddle.ParamAttr(\n","              initializer=paddle.nn.initializer.Uniform(\n","                  low=-init_scale, high=init_scale)))\n","\n","    def forward(self, target, context, label):\n","      '''A callable function to do forward propagation. Simply call it by \n","      SkipGram()(target, context, label).\n","\n","      Args:\n","        target(tensor): integer, the target word used to predict context words.\n","        context(tensor): integer, the context word to be predicted. \n","        label(int): 0 --> false ontext word. 1 --> true context word. \n","\n","      Returns:\n","        pred(float): the sigmoid output used to predict the label. \n","        loss(float): the loss score. \n","      '''\n","      # convert input into a word embedding: input layer --> hidden layer\n","      target_embd = self.embedding(target)\n","      context_embd = self.embedding_out(context)\n","\n","      # hidden layer --> output layer followed by a sigmoid function\n","      word_sim = paddle.multiply(target_embd, context_embd)\n","      word_sim = paddle.sum(word_sim, axis=-1)\n","      word_sim = paddle.reshape(word_sim, shape=[-1])\n","      pred = F.sigmoid(word_sim)\n","\n","      # we use binary_cross_entropy_with_logits to define the loss function\n","      loss = F.binary_cross_entropy_with_logits(word_sim, label) # word_sim but not pred here!\n","      loss = paddle.mean(loss)\n","\n","      return pred, loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"88lPyqR5F-vz"},"source":["<a name='2-4'></a>\n","### 2.4 Training\n"]},{"cell_type":"code","metadata":{"id":"RRdCQG_0F-vz"},"source":["# we can first define a get_similar_words function we can use during training\n","# so that we can monitor how the embeddings improve during training\n","def get_similar_words(query_word, k, embd, print_words=True):\n","  '''Returns the k most similar tokens to the query_token based on cosine similarity. \n","\n","  Args:\n","    query_word('str'): the query_word should be in the training dataset. \n","    k (int): the number of most similar tokens to the query token. \n","    embd (array-like): the embedding matrix.\n","    print_tokens(bool): defaults to True. If False, the most similar tokens will \n","      not be printed out. \n","\n","  Returns:\n","    k most most similar tokens to the query_token. \n","  '''\n","  W = embd.numpy() # embedding matrix as a numpy array\n","  x = W[word_to_id[query_word]] # the query token's embedding\n","  # 1e-9 is used to avoid zero division \n","  cos = np.dot(W, x) / np.sqrt(np.sum(W * W, axis=1) * np.sum(x * x) + 1e-9)\n","  flat = cos.flatten()\n","  # we will not include the query token itself \n","  k = k + 1\n","  indices = np.argpartition(flat, -k)[-k:]\n","  indices = indices[np.argsort(-flat[indices])]\n","  sim_words = [id_to_word[idx] for idx in indices][1:]\n","  if print_words:\n","    print('for word %s, the similar word is %s' % (query_word, sim_words))\n","\n","  return sim_words"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"f9fAEsRKPIeD"},"source":["<font color='red'>\n","<b>Reminder:</b>\n","<font color='black'>\n","\n","If you are not runing this notebook on GPU, it is recommended that you use a much smaller training dataset as noted in the previous [reminder](#reminder). You may also want to use a smaller batch size, a smaller embedding size (e.g., 20) to speed up the training process. Training word embeddings takes a lot of resources, so it is more important to know the process than training a good model yourself. If you are interested in training a larger model, you can run this notebook in [Baidu's AI studio](https://aistudio.baidu.com/aistudio/index).\n","\n","<br>\n","\n","For your convenience, you can download the trained embeddings by this notebook [in this zip file](https://drive.google.com/file/d/1KA14tazUiIEYQZ1BUjV15qg_FRY11lYW/view?usp=sharing). "]},{"cell_type":"code","metadata":{"id":"3HbX-8_wF-v0","outputId":"fcd515ce-8939-4cd6-c586-54d86b87d4e6"},"source":["# trainning \n","\n","# Hyperparameters\n","# if you run the following code, your results might be different \n","# because of the randomization during th optimization process\n","batch_size = 256\n","epoch_num = 2\n","embedding_size = 50\n","step = 0\n","learning_rate = 0.001\n","vocab_size = len(id_to_word)\n","\n","# Google Colab does not support this for some reason\n","# uncomment the following code if you run this on GPU\n","paddle.set_device('gpu:0')\n","\n","# Skip gram model\n","skip_gram_model = SkipGram(vocab_size, embedding_size)\n","# We will use adam to optimize during the backpropagation \n","adam = paddle.optimizer.Adam(learning_rate=learning_rate, parameters = skip_gram_model.parameters())\n","\n","for epoch in range(epoch_num):\n","  for target, context, label in make_batches(dataset, batch_size):    \n","    # foward propagation \n","      pred, loss = skip_gram_model(target, context, label)\n","    # paddle will automatically do the backpropagation for us\n","      loss.backward()\n","    # adam optimizer \n","      adam.step()\n","    # clear the grads we already used.\n","      adam.clear_grad()\n","\n","    # we will print the loss of our model every 10000 steps\n","      step += 1\n","      if step % 10000 == 0:\n","          print(\"epoch %d step %d, loss %.3f\" % (epoch+1, step, loss.numpy()[0]))\n","\n","    # we will manually check the performance of the model every 50000 steps by\n","    # seeing whether the similiar tokens to a query token make sense to us.  \n","      if step % 50000 ==0:\n","        embd_weights = skip_gram_model.embedding.weight\n","        print()\n","        get_similar_words('china', 5, embd_weights)\n","        get_similar_words('one', 5, embd_weights)\n","        get_similar_words('computer', 5, embd_weights)\n","        print()\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. \n","Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n","  if data.dtype == np.object:\n"],"name":"stderr"},{"output_type":"stream","text":["epoch 1 step 10000, loss 0.426\n","epoch 1 step 20000, loss 0.189\n","epoch 1 step 30000, loss 0.221\n","epoch 1 step 40000, loss 0.285\n","epoch 1 step 50000, loss 0.255\n","\n","for word china, the similar word is ['spirits', 'responses', 'fairly', 'makers', 'flowers']\n","for word one, the similar word is ['zero', 'nine', 'american', 'eight', 'two']\n","for word computer, the similar word is ['existed', 'creation', 'may', 'prayer', 'accepted']\n","\n","epoch 1 step 60000, loss 0.271\n","epoch 1 step 70000, loss 0.162\n","epoch 1 step 80000, loss 0.247\n","epoch 1 step 90000, loss 0.296\n","epoch 1 step 100000, loss 0.342\n","\n","for word china, the similar word is ['states', 'austria', 'baltic', 'denver', 'bulgarian']\n","for word one, the similar word is ['eight', 'nine', 'four', 'three', 'six']\n","for word computer, the similar word is ['application', 'color', 'does', 'user', 'models']\n","\n","epoch 1 step 110000, loss 0.265\n","epoch 1 step 120000, loss 0.191\n","epoch 1 step 130000, loss 0.186\n","epoch 1 step 140000, loss 0.211\n","epoch 1 step 150000, loss 0.165\n","\n","for word china, the similar word is ['iran', 'colonial', 'taiwan', 'austria', 'phoenix']\n","for word one, the similar word is ['seven', 'eight', 'zero', 'four', 'five']\n","for word computer, the similar word is ['memory', 'user', 'mac', 'programming', 'serial']\n","\n","epoch 1 step 160000, loss 0.204\n","epoch 1 step 170000, loss 0.323\n","epoch 1 step 180000, loss 0.341\n","epoch 1 step 190000, loss 0.205\n","epoch 1 step 200000, loss 0.301\n","\n","for word china, the similar word is ['iran', 'northeast', 'taiwan', 'province', 'kong']\n","for word one, the similar word is ['three', 'seven', 'four', 'eight', 'five']\n","for word computer, the similar word is ['available', 'serial', 'interface', 'software', 'user']\n","\n","epoch 1 step 210000, loss 0.329\n","epoch 1 step 220000, loss 0.234\n","epoch 1 step 230000, loss 0.186\n","epoch 1 step 240000, loss 0.221\n","epoch 1 step 250000, loss 0.343\n","\n","for word china, the similar word is ['azerbaijan', 'turkey', 'africa', 'mainland', 'ethnic']\n","for word one, the similar word is ['eight', 'four', 'three', 'six', 'seven']\n","for word computer, the similar word is ['download', 'software', 'serial', 'api', 'memory']\n","\n","epoch 1 step 260000, loss 0.331\n","epoch 1 step 270000, loss 0.208\n","epoch 1 step 280000, loss 0.184\n","epoch 1 step 290000, loss 0.336\n","epoch 1 step 300000, loss 0.210\n","\n","for word china, the similar word is ['azerbaijan', 'mainland', 'madagascar', 'malay', 'bengal']\n","for word one, the similar word is ['five', 'seven', 'nine', 'eight', 'three']\n","for word computer, the similar word is ['download', 'serial', 'server', 'kb', 'format']\n","\n","epoch 1 step 310000, loss 0.157\n","epoch 1 step 320000, loss 0.118\n","epoch 1 step 330000, loss 0.217\n","epoch 1 step 340000, loss 0.235\n","epoch 1 step 350000, loss 0.230\n","\n","for word china, the similar word is ['lesser', 'malay', 'mongolia', 'occidental', 'turkey']\n","for word one, the similar word is ['seven', 'eight', 'nine', 'six', 'zero']\n","for word computer, the similar word is ['programmers', 'api', 'powerpc', 'aes', 'hardware']\n","\n","epoch 1 step 360000, loss 0.203\n","epoch 1 step 370000, loss 0.278\n","epoch 1 step 380000, loss 0.173\n","epoch 1 step 390000, loss 0.270\n","epoch 1 step 400000, loss 0.260\n","\n","for word china, the similar word is ['occidental', 'lesser', 'mainland', 'malay', 'sahara']\n","for word one, the similar word is ['six', 'eight', 'four', 'three', 'nine']\n","for word computer, the similar word is ['api', 'proprietary', 'programmers', 'specifications', 'user']\n","\n","epoch 1 step 410000, loss 0.200\n","epoch 1 step 420000, loss 0.373\n","epoch 1 step 430000, loss 0.134\n","epoch 1 step 440000, loss 0.316\n","epoch 1 step 450000, loss 0.101\n","\n","for word china, the similar word is ['afghanistan', 'nanjing', 'outlying', 'peacekeeping', 'turkey']\n","for word one, the similar word is ['seven', 'eight', 'four', 'two', 'three']\n","for word computer, the similar word is ['programmers', 'proprietary', 'memory', 'api', 'user']\n","\n","epoch 1 step 460000, loss 0.248\n","epoch 2 step 470000, loss 0.177\n","epoch 2 step 480000, loss 0.202\n","epoch 2 step 490000, loss 0.212\n","epoch 2 step 500000, loss 0.483\n","\n","for word china, the similar word is ['occidental', 'asia', 'guangdong', 'nanjing', 'caucasus']\n","for word one, the similar word is ['three', 'eight', 'five', 'seven', 'two']\n","for word computer, the similar word is ['specifications', 'aes', 'interface', 'programmers', 'api']\n","\n","epoch 2 step 510000, loss 0.354\n","epoch 2 step 520000, loss 0.289\n","epoch 2 step 530000, loss 0.163\n","epoch 2 step 540000, loss 0.188\n","epoch 2 step 550000, loss 0.118\n","\n","for word china, the similar word is ['guangdong', 'outlying', 'influence', 'ming', 'buddhism']\n","for word one, the similar word is ['eight', 'two', 'nine', 'five', 'four']\n","for word computer, the similar word is ['specifications', 'computers', 'programmers', 'bbs', 'aes']\n","\n","epoch 2 step 560000, loss 0.192\n","epoch 2 step 570000, loss 0.345\n","epoch 2 step 580000, loss 0.181\n","epoch 2 step 590000, loss 0.403\n","epoch 2 step 600000, loss 0.300\n","\n","for word china, the similar word is ['buddhism', 'guangdong', 'ming', 'khmer', 'outlying']\n","for word one, the similar word is ['eight', 'three', 'zero', 'two', 'six']\n","for word computer, the similar word is ['specifications', 'programmers', 'identifier', 'aes', 'interface']\n","\n","epoch 2 step 610000, loss 0.292\n","epoch 2 step 620000, loss 0.189\n","epoch 2 step 630000, loss 0.284\n","epoch 2 step 640000, loss 0.151\n","epoch 2 step 650000, loss 0.345\n","\n","for word china, the similar word is ['guangdong', 'outlying', 'buddhism', 'occidental', 'turkey']\n","for word one, the similar word is ['five', 'eight', 'zero', 'seven', 'two']\n","for word computer, the similar word is ['aes', 'calculators', 'specifications', 'identifier', 'proprietary']\n","\n","epoch 2 step 660000, loss 0.448\n","epoch 2 step 670000, loss 0.185\n","epoch 2 step 680000, loss 0.268\n","epoch 2 step 690000, loss 0.183\n","epoch 2 step 700000, loss 0.295\n","\n","for word china, the similar word is ['buddhism', 'taiwanese', 'andhra', 'indo', 'guangdong']\n","for word one, the similar word is ['eight', 'zero', 'three', 'five', 'seven']\n","for word computer, the similar word is ['identifier', 'aes', 'specifications', 'peripherals', 'chipset']\n","\n","epoch 2 step 710000, loss 0.255\n","epoch 2 step 720000, loss 0.361\n","epoch 2 step 730000, loss 0.302\n","epoch 2 step 740000, loss 0.243\n","epoch 2 step 750000, loss 0.191\n","\n","for word china, the similar word is ['guangdong', 'andhra', 'tibet', 'buddhism', 'singapore']\n","for word one, the similar word is ['zero', 'nine', 'five', 'eight', 'three']\n","for word computer, the similar word is ['peripherals', 'specifications', 'gui', 'imac', 'aes']\n","\n","epoch 2 step 760000, loss 0.402\n","epoch 2 step 770000, loss 0.592\n","epoch 2 step 780000, loss 0.141\n","epoch 2 step 790000, loss 0.277\n","epoch 2 step 800000, loss 0.383\n","\n","for word china, the similar word is ['guangdong', 'buddhism', 'singapore', 'hunan', 'cuon']\n","for word one, the similar word is ['three', 'four', 'eight', 'nine', 'seven']\n","for word computer, the similar word is ['peripherals', 'cassette', 'identifier', 'desktop', 'chipset']\n","\n","epoch 2 step 810000, loss 0.287\n","epoch 2 step 820000, loss 0.268\n","epoch 2 step 830000, loss 0.386\n","epoch 2 step 840000, loss 0.379\n","epoch 2 step 850000, loss 0.339\n","\n","for word china, the similar word is ['guangdong', 'truman', 'mainland', 'buddhism', 'cambodia']\n","for word one, the similar word is ['nine', 'five', 'three', 'seven', 'zero']\n","for word computer, the similar word is ['peripherals', 'hardware', 'software', 'imac', 'specifications']\n","\n","epoch 2 step 860000, loss 0.115\n","epoch 2 step 870000, loss 0.229\n","epoch 2 step 880000, loss 0.298\n","epoch 2 step 890000, loss 0.177\n","epoch 2 step 900000, loss 0.187\n","\n","for word china, the similar word is ['mainland', 'guangdong', 'truman', 'liang', 'promoted']\n","for word one, the similar word is ['five', 'seven', 'eight', 'four', 'nine']\n","for word computer, the similar word is ['eniac', 'peripherals', 'bbs', 'aes', 'gcc']\n","\n","epoch 2 step 910000, loss 0.140\n","epoch 2 step 920000, loss 0.231\n","epoch 2 step 930000, loss 0.103\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"FDB2oRujF-v4"},"source":["<a name='3'></a>\n","# 3. Things to do next\n","\n","<a name='3-1'></a>\n","### 3.1 Save the trained embeddings\n"]},{"cell_type":"code","metadata":{"id":"NfUPcsOcHZcy"},"source":["# let's save the words and their embeddings as tsv files\n","\n","words = list(word_to_id.keys())\n","embeddings = skip_gram_model.embedding.weight\n","\n","with open('text8_words.tsv', 'w') as f:\n","    f.write('\\n'.join(words))\n","    f.close()\n","\n","with open('text8_embds.tsv', 'w') as f:\n","    for embd in embeddings.numpy():\n","        f.write('\\t'.join([str(em) for em in embd]))\n","        f.write('\\n')\n","    f.close()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"YAn9b07VF-v6"},"source":["<a name='3-2'></a>\n","### 3.2 Visualization\n","\n","You can refer to [embedding visualization using paddlepaddle tool](https://colab.research.google.com/drive/1B9pcYR9fVvmB1pPWiIqb0u_WmxlY--T8?usp=sharing)\n","and [embedding visualization using tensorflow tool](https://colab.research.google.com/drive/1HZdDA_TzdJhGo_uIUSa84rP6yQjHvMT3?usp=sharing) two files to learn how to visualize word embeddings. Below is the visualization of the word embeddings trained above projected by [TensorBoard's Embedding Projector](http://projector.tensorflow.org).\n","\n","<img src='https://drive.google.com/uc?export=view&id=1N9z3_lTWgwhDZt-tzpI5fGs7SbiK-19D'>"]},{"cell_type":"markdown","metadata":{"id":"3vYS5t3LJybE"},"source":["<a name='3-3'></a>\n","### 3.3 Word analogy test\n","\n","Word analogy is one of the most interesting application of word embeddings. Suppose we have three words, King, Queen, Women, and we want to find another word $W_4$ that makes the relations between Women and $W_4$ most similar to that between King and Queen. Word analysis is like finding the most similar word to another word, but at the word-pair level. \n","\n","<br>\n","\n","As the trained word embeddings are not human readable, word analogy test has thus become a way to evaluate the performance of the trained embeddings. If you are interested, try to write a function to do just this. You may find this [questions-words.txt](https://github.com/tmikolov/word2vec/blob/master/questions-words.txt) corpus helpful for the evaluation. "]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"rUi71MoTF-v7"},"source":["<a name='4'></a>\n","# 4. References\n","- [Mikolov et. al., 2013a. Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf)\n","- [Mikolov et. al., 2013b. Distributed representation of words and phrases and their compositionality](https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf)\n","- [Source code in C by Mikolov](https://github.com/tmikolov/word2vec)\n","- [Pennington et.al., 2014. GloVe- Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf)\n","- [On word embeddings - Part 2: Approximating the Softmax](https://ruder.io/word-embeddings-softmax/)\n","- [Sequence Models (Course 5) in Deep Learning Specialization taught by Andrew Ng on Coursera](https://www.coursera.org/specializations/deep-learning?).\n","- [Marginalia, 2018. The backpropagation algorithm for Word2Vec](http://www.claudiobellei.com/2018/01/06/backprop-word2vec/)\n","- [Rong, 2014. Word2vec Parameter Learning Explained](https://arxiv.org/pdf/1411.2738.pdf)"]}]}