paddlenlp_updated_notes_English/WordEmbedding/3-embedding visualization using paddlepaddle tool.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"3-embedding visualization using paddlepaddle tool.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyPWZhPk8E+XX6mkPQte3NTM"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"code","metadata":{"id":"tRtY07vYT7QN"},"source":["# Author: Zhengxiang (Jack) Wang \n","# Date: 2021-07-20\n","# GitHub: https://github.com/jaaack-wang \n","# About: embedding visualization using paddlepaddle tool"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"sfgIFgWSUNUw"},"source":["# Overview\n","\n","In previous notebooks, we learned how to [load pre-trained word embedding in paddlenp](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing) and how to [calculate text cosine similarity in paddlenlp](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing). In this notebook, we will learn how to visualze word and text embeddings using a paddlepaddle Deep Learning Visualization Toolkit, called [VisualDL](https://github.com/PaddlePaddle/VisualDL). More concretely, we will visualize high-dimensional word embeddings in a 3-D coordinate system. \n","\n","<br>\n","\n","**Please note that, for this notebook, all the code should be executed on your local machine (e.g., terminal) if you do not use Baidu's [AI Studio](https://aistudio.baidu.com/aistudio/index). Colab as well as Jupyter notebook seems not to work well with visualdl. <ins>If you do not want to download `paddlepaddle` and `paddlenlp`, that is fine too</ins>. This is notebook will show you a general way to visualize word/text embeddings.**\n","\n","<br>\n","\n","\n","<table align=\"right\">\n","  <td>\n","    <a target=\"_blank\" href=\"https://colab.research.google.com/drive/1B9pcYR9fVvmB1pPWiIqb0u_WmxlY--T8?usp=sharing\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab </a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://github.com/jaaack-wang\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> Author's GitHub </a>\n","  </td>\n","  <td>\n","    <a href=\"https://docs.google.com/uc?export=download&id=1B9pcYR9fVvmB1pPWiIqb0u_WmxlY\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /> Download this notebook </a>\n","  </td>\n","</table> \n","\n","\n","<br>\n"]},{"cell_type":"markdown","metadata":{"id":"ONmPv040UCoS"},"source":["# Table of Contents\n","- [1. General use of VisualDL for embedding visualization](#1)\n","- [2. Visualizing word embeddings](#2)\n","  - [2.1 With `paddlepaddle` and `paddlenlp` installed](#2-1)\n","  - [2.2 Without `paddlepaddle` and `paddlenlp` installed](#2-2)\n","- [3. Visualizing sentence embeedings](#3)\n","  - [3.1 With `paddlepaddle` and `paddlenlp` installed](#3-1)\n","  - [3.2 Without `paddlepaddle` and `paddlenlp` installed](#3-2)\n","- [4. References](#4)"]},{"cell_type":"markdown","metadata":{"id":"D5hQlb3cjgB8"},"source":["<a name=\"1\"></a>\n","# 1. General use of VisualDL for embedding visualization\n","\n","The general use of VisualDL can be found on its [GitHub project page](https://github.com/PaddlePaddle/VisualDL). \n","\n","<br>\n","\n","For visualizing embeddings, \n","\n","  1. you first need to get the embeddings as `\"embeddings\"` as well as the corresponding words or texts as `\"labels\"`;\n","  2. then, you need to create a log file that stores the embeddings as follows:\n","\n","\n","```python\n","from visualdl import LogWriter\n","\n",">>> labels = \"words or texts whose embeddings we will get\"\n",">>> embeddings = \"the embeddings for the labels\"\n",">>> with LogWriter(logdir='./embds_vdl') as writer:\n",">>>      writer.add_embeddings(tag='A_Example', mat=[em for em in embeddings], metadata=labels)\n","```\n","\n","   3. finally, you need to launch the visualDL panel to see the visualized embeddings as illustrated [here](https://github.com/PaddlePaddle/VisualDL#2-launch-panel). \n","    - If you are using Baidu's [AI Studio](https://aistudio.baidu.com/aistudio/index) and running a notebook there, we can launch the VisualDL panel by clicking the `\"可视化\"` button on the left bar. \n","    - On your command line, use ```visualdl --logdir dir``` (this might not work if you do not get `visualdl` to your `$PATH`.) where `dir` = 'the path to the directory where the log file created above is stored`.\n","    - Run python on your shell or terminal, and do the following:\n","    \n","```python\n","from visualdl.server import app\n","# dir = 'the path to the directory where the log file created above is stored`\n","# the log file within the dir\n",">>> app.run(logdir=\"dir\", model='file_name') \n","```\n","\n","<br>\n","\n","**Of course, we need to install `visualdl` first! (you will also need to have it installed for step 3)**\n","\n","```\n","pip3 install --upgrade visualdl\n","```"]},{"cell_type":"markdown","metadata":{"id":"LMKs0SzaiOvR"},"source":["<a name=\"2\"></a>\n","# 2. Visualizing word embeddings\n","\n","First, we need to load the pre-trained word embeddings available in `padddlenlp.embeddings.TokenEmbedding` and get the embeddings of a list of words or texts whose embeddings you want to visualize. You can refer to [loading pre-trained word embedding in paddlenp.ipynb](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing) and [calculating text cosine similarity in paddlenlp.ipynb](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing) to see how these can be done for words and for sentences respectively. \n","\n","<br>\n","\n","For visualizing pre-trained word embeddings not natively available in `paddlenlp`, you can refer to [loading pre-trained word embedding in paddlenp.ipynb](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing) to learn how to manually load a pre-trained word embedding model if you want to stick to `paddlenlp`.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"uBDAdevp8_WI"},"source":["<a name=\"2-1\"></a>\n","### 2.1 With paddlepaddle and paddlenlp installed\n","\n","  - First, make sure that you have `paddlepaddle` and `paddlenlp` installed by running the following commands:\n","  ```\n","  pip3 install --upgrade paddlepaddle\n","  pip3 install --upgrade paddlenlp\n","  ```\n","\n","  <br>\n","\n","  - In python, load a word embedding you want to visualize. Here, we will visualize the first 10,000 words of `glove.wiki2014-gigaword.target.word-word.dim50.en`. Only 10,000 word embeddings are visualized for the ease of display.\n","  ```python\n","from paddlenlp.embeddings import TokenEmbedding\n","# load the model: glove.wiki2014-gigaword.target.word-word.dim50.en\n","# model size: 73.45 MB; vocab size: 400002\n","tk_embedding = TokenEmbedding(embedding_name=\"glove.wiki2014-gigaword.target.word-word.dim50.en\")\n","'''Output:\n","[2021-07-20 19:55:56,510] [    INFO] - Loading token embedding...\n","[2021-07-20 19:55:57,498] [    INFO] - Finish loading embedding vector.\n","[2021-07-20 19:55:57,498] [    INFO] - Token Embedding info:             \n","Unknown index: 400000             \n","Unknown token: [UNK]             \n","Padding index: 400001             \n","Padding token: [PAD]             \n","Shape :[400002, 50]\n","'''\n","```\n","\n","<br>\n","\n","  - Then, let's get the first 10,000 words and their embeddings.\n","  ```python\n","words = tk_embedding.vocab.to_tokens(list(range(10000)))\n","words_ems = tk_embedding.search(words)\n","  ```\n","\n","<br>\n","\n","  - Then, create a log file to record the word embeddings using `visualdl.LogWriter`.\n","  ```python\n","from visualdl import LogWriter\n","# you can define the logdir and add the tag name in your way\n","with LogWriter(logdir='./embds_vdl') as writer:\n","writer.add_embeddings(tag='glove_word_embds', mat=[em for em in words_ems], metadata=words)\n","  ```\n","\n","<br>\n","\n","  - Finally, let's visualize the word embeddings we just restored.\n","  ```python\n","from visualdl.server import app\n","app.run(logdir='./embds_vdl')\n","'''\n","output:\n","11324\n","VisualDL 2.2.0\n","* Running on http://localhost:8040/ (Press CTRL+C to quit)\n","127.0.0.1 - - [20/Jul/2021 20:05:02] \"GET /alive HTTP/1.1\" 204 -\n","Running VisualDL at http://localhost:8040/ (Press CTRL+C to quit)\n","Serving VisualDL on localhost; to expose to the network, use a proxy or pass --host 0.0.0.0'''\n","  ```\n","\n","<br>\n","\n","You will see something like the following when you copy and paste http://localhost:8040/ into your browser and click \"A\" in the loaded webpage. Successful!\n","\n","<img src='https://drive.google.com/uc?export=view&id=19MaoquRipqV237ODbEVdvlyTzK5OnY8J' width=\"1000\" height=\"600\">"]},{"cell_type":"markdown","metadata":{"id":"E_-CubpGUqyY"},"source":["\n","<a name=\"2-2\"></a>\n","\n","### 2.2 Without `paddlepaddle` and `paddlenlp` installed\n","\n","  - For your convience, I have already extract 10,000 examples from `glove.6B.50d.txt` downloaded from [glove website](https://nlp.stanford.edu/projects/glove/). You can use the reduced version directly by clicking [here](https://drive.google.com/file/d/1wU0LLC3KcZleSsT-eRrYQ0puC1_ykXjJ/view?usp=sharing).\n","\n","<br>\n","\n","  - In python, run the following code to load the words and their embeddings stored in the [glove.6B.50d_reduced.txt](https://drive.google.com/file/d/1wU0LLC3KcZleSsT-eRrYQ0puC1_ykXjJ/view?usp=sharing) file that you just downloaded. \n","  ```python\n","  file_path = 'path/to/glove.6B.50d_reduced.txt'\n","  # we will need numpy to convert the embeddings (stored as str) into numpy array\n","  import numpy as np\n","  glove = open(file_path, 'r')\n","  words, words_ems = [], []\n","  for line in glove:\n","      line = line.split(maxsplit=1)\n","      words.append(line[0])\n","      words_ems.append(np.array(line[1].strip().split(), dtype=np.float32))\n","  ```\n","\n","  <br>\n","\n","  - Then, create a log file to record the word embeddings using `visualdl.LogWriter`.\n","  ```python\n","from visualdl import LogWriter\n","# you can define the logdir and add the tag name in your way\n","with LogWriter(logdir='./embds_vdl') as writer:\n","writer.add_embeddings(tag='glove_word_embds', mat=[em for em in words_ems], metadata=words)\n","  ```\n","\n","<br>\n","\n","  - Finally, let's visualize the word embeddings we just restored.\n","  ```python\n","from visualdl.server import app\n","app.run(logdir='./embds_vdl')\n","'''\n","output:\n","11324\n","VisualDL 2.2.0\n","* Running on http://localhost:8040/ (Press CTRL+C to quit)\n","127.0.0.1 - - [20/Jul/2021 20:05:02] \"GET /alive HTTP/1.1\" 204 -\n","Running VisualDL at http://localhost:8040/ (Press CTRL+C to quit)\n","Serving VisualDL on localhost; to expose to the network, use a proxy or pass --host 0.0.0.0'''\n","  ```\n","\n","<br>\n","\n","You will see the same picture as above when you copy and paste http://localhost:8040/ into your browser and click \"A\" in the loaded webpage.  "]},{"cell_type":"markdown","metadata":{"id":"3bQMdyj5B3cg"},"source":["<a name=\"3\"></a>\n","# 3. Visualizing sentence embeddings\n","\n","To visualize sentence embeddings, you need to have sentences embeddings to visualize first. For how to generate sentence embeddings based on word embeddings, you can use refer to [calculating text cosine similarity in paddlenlp.ipynb](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing) to learn how to apply Bag-of-Words model. \n","\n","<br>\n","\n","**Once you get the sentence embeddings ready, the rest is identical to how you visualize word embeddings above.** "]},{"cell_type":"markdown","metadata":{"id":"gTMKd1YMC4fP"},"source":["<a name=\"3-1\"></a>\n","### 3.1 With paddlepaddle and paddlenlp installed\n","\n","  - First, make sure that you have `paddlepaddle` and `paddlenlp` installed by running the following commands:\n","  ```\n","  pip3 install --upgrade paddlepaddle\n","  pip3 install --upgrade paddlenlp\n","  ```\n","\n","  <br>\n","\n","  - In python, load a word embedding you want to calculate the sentence embeddings. Here, we will load `glove.wiki2014-gigaword.target.word-word.dim50.en`. \n","  ```python\n","from paddlenlp.embeddings import TokenEmbedding\n","# load the model: glove.wiki2014-gigaword.target.word-word.dim50.en\n","# model size: 73.45 MB; vocab size: 400002\n","tk_embedding = TokenEmbedding(embedding_name=\"glove.wiki2014-gigaword.target.word-word.dim50.en\")\n","'''Output:\n","[2021-07-20 19:55:56,510] [    INFO] - Loading token embedding...\n","[2021-07-20 19:55:57,498] [    INFO] - Finish loading embedding vector.\n","[2021-07-20 19:55:57,498] [    INFO] - Token Embedding info:             \n","Unknown index: 400000             \n","Unknown token: [UNK]             \n","Padding index: 400001             \n","Padding token: [PAD]             \n","Shape :[400002, 50]\n","'''\n","```\n","\n","<br>\n","\n","  - Then, we will continue to use [sts_sample.tsv](https://drive.google.com/file/d/1TJgl4WtKY4JlcCVZtd9CQK-1w26wNo9D/view?usp=sharing) which contains 50 English sentence pairs, namely, 100 sentences. You are also welcome to use any English sentences whose embeddings you want to visualize.\n","\n","  ```python\n","  # First load the sents\n","  file_path = 'file/path/to/sts_sample.tsv'\n","  sts = open(file_path, 'r')\n","  sents = []\n","  for line in sts:\n","      line = line.split('\\t')\n","      sents.append(line[0])\n","      sents.append(line[1])\n","  # check the first 5 sentences\n","  print('The first 5 sentences:\\n', sents[:5])\n","  '''output:\n","  The first 5 sentences:  \n","  ['A multi-colored bird clings to a wire fence.', 'A bird holding on to a metal gate.', 'A woman is mixing meat.', 'A woman is feeding a man.', 'Two men sailing in a small sailboat.']\n","  '''\n","  ```\n","\n","<br>\n","\n","  - Then, use the functions we built in third section of [calculating text cosine similarity in paddlenlp.ipynb](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing) to use the simple Bag-of-Words model to calculate the sentence embeddings. \n","  ```python\n","  def sentence_embedding(sentence, tokenizer, embedder):\n","      def embedding(sent):\n","        tokens = tokenizer(sent)\n","        return np.sum(embedder.search(tokens), axis=0) # sum vertically\n","\n","      if isinstance(sentence, str):\n","        return embedding(sentence)\n","      elif isinstance(sentence, list):\n","        return [embedding(sent) for sent in sentence]\n","      else:\n","        raise TypeError(f'sentence should be either a str or a list.' \n","    '{type(sentence)} not supported. ')\n","  ```\n","\n","  <br>\n","\n","  - Then, let's get the sentence embeddings for the 100 we just loaded. \n","  ```python\n","  # here, we simply use space to tokenize English text\n","  en_tokenizer = lambda x: x.split() \n","  sent_ems = sentence_embedding(sents, en_tokenizer, tk_embedding)\n","  ```\n","\n","  <br>\n","\n","  - Finally, let's log the sentence embeddings and visualize them!\n","    ```python\n","from visualdl import LogWriter\n","# you can define the logdir and add the tag name in your way\n","with LogWriter(logdir='./sent_embds_vdl') as writer:\n","      writer.add_embeddings(tag='glove_sent_embds', mat=[em for em in sent_ems], metadata=sents)\n","# visualize them\n","from visualdl.server import app\n","app.run(logdir='./sent_embds_vdl')\n","'''output:\n","11760\n"," * Running on http://localhost:8040/ (Press CTRL+C to quit)\n","127.0.0.1 - - [20/Jul/2021 21:43:12] \"GET /alive HTTP/1.1\" 204 -\n","Running VisualDL at http://localhost:8040/ (Press CTRL+C to quit)\n","Serving VisualDL on localhost; to expose to the network, use a proxy or pass --host 0.0.0.0\n","'''\n","  ```\n","\n","<br>\n","\n","You will see something like the following when you copy and paste http://localhost:8040/ into your browser and click \"A\" in the loaded webpage. Successful!\n","\n","<img src='https://drive.google.com/uc?export=view&id=1ALsLToHkOfKU3gOpf2hXxWTh4rrfRsh2' width=\"1000\" height=\"600\">"]},{"cell_type":"markdown","metadata":{"id":"mWKWf2PhC5Vh"},"source":["<a name=\"3-2\"></a>\n","### 3.2 Without paddlepaddle and paddlenlp installed\n","\n","  - For generating sentence embeddings, you may want to load the entire `glove.6B.50d.txt` so that there is less chance of running into unseen words. You will also want to set a token to represent all the unseen words. For your convience, you can download the `glove.6B.50d.txt`  by clicking [here](https://drive.google.com/file/d/1o1fUeoAt260P90FeP_L5eICiQowHIcvY/view?usp=sharing).\n","\n","\n","<br>\n","\n","  - In python, run the following code to load the words and their embeddings stored in the [glove.6B.50d.txt](https://drive.google.com/file/d/1o1fUeoAt260P90FeP_L5eICiQowHIcvY/view?usp=sharing) file that you just downloaded. You also want to create a `tk_em_dict` that you can look up later to get word embeddings.  \n","  ```python\n","  file_path = 'path/to/glove.6B.50d.txt'\n","  # we will need numpy to convert the embeddings (stored as str) into numpy array\n","  import numpy as np\n","  glove = open(file_path, 'r')\n","  words, words_ems = [], []\n","  for line in glove:\n","      line = line.split(maxsplit=1)\n","      words.append(line[0])\n","      words_ems.append(np.array(line[1].strip().split(), dtype=np.float32))\n","  # create the tk_em_dict\n","  tk_em_dict = dict(zip(words, words_ems))\n","  # let's set np.zeros(50) for all unseen tokens\n","  from collections import defaultdict\n","  tk_em_dict = defaultdict(lambda: np.zeros(50, dtype=np.float32), tk_em_dict)\n","  ```\n","\n"," <br>\n","\n","  - Then, we will continue to use [sts_sample.tsv](https://drive.google.com/file/d/1TJgl4WtKY4JlcCVZtd9CQK-1w26wNo9D/view?usp=sharing) which contains 50 English sentence pairs, namely, 100 sentences. You are also welcome to use any English sentences whose embeddings you want to visualize.\n","\n","  ```python\n","  # First load the sents\n","  file_path = 'file/path/to/sts_sample.tsv'\n","  sts = open(file_path, 'r')\n","  sents = []\n","  for line in sts:\n","      line = line.split('\\t')\n","      sents.append(line[0])\n","      sents.append(line[1])\n","  # check the first 5 sentences\n","  print('The first 5 sentences:\\n', sents[:5])\n","  '''output:\n","  The first 5 sentences:  \n","  ['A multi-colored bird clings to a wire fence.', 'A bird holding on to a metal gate.', 'A woman is mixing meat.', 'A woman is feeding a man.', 'Two men sailing in a small sailboat.']\n","  '''\n","  ```\n","\n","<br>\n","\n","  - Then, use the functions we built in third section of [calculating text cosine similarity in paddlenlp.ipynb](https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing) with small modifications to use the simple Bag-of-Words model to calculate the sentence embeddings. \n","  ```python\n","  def sentence_embedding(sentence, tokenizer, tk_em_dict):\n","      def embedding(sent):\n","        tokens = tokenizer(sent)\n","        return np.sum([tk_em_dict[tk.lower()] for tk in tokens], axis=0) # sum vertically\n","\n","      if isinstance(sentence, str):\n","        return embedding(sentence)\n","      elif isinstance(sentence, list):\n","        return [embedding(sent) for sent in sentence]\n","      else:\n","        raise TypeError(f'sentence should be either a str or a list.' \n","    '{type(sentence)} not supported. ')\n","  ```\n","\n","  <br>\n","\n","  - Then, let's get the sentence embeddings for the 100 we just loaded. \n","  ```python\n","  # here, we simply use space to tokenize English text for convience\n","  # you can build or import a more sophisticated tokenizer if you wish\n","  en_tokenizer = lambda x: x.split() \n","  sent_ems = sentence_embedding(sents, en_tokenizer, tk_em_dict)\n","  ```\n","\n","  <br>\n","\n","  - Finally, let's log the sentence embeddings and visualize them!\n","    ```python\n","from visualdl import LogWriter\n","# you can define the logdir and add the tag name in your way\n","with LogWriter(logdir='./sent_embds_vdl') as writer:\n","      writer.add_embeddings(tag='glove_sent_embds', mat=[em for em in sent_ems], metadata=sents)\n","# visualize them\n","from visualdl.server import app\n","app.run(logdir='./sent_embds_vdl')\n","'''output:\n","11857\n"," * Running on http://localhost:8040/ (Press CTRL+C to quit)\n","127.0.0.1 - - [20/Jul/2021 21:43:12] \"GET /alive HTTP/1.1\" 204 -\n","Running VisualDL at http://localhost:8040/ (Press CTRL+C to quit)\n","Serving VisualDL on localhost; to expose to the network, use a proxy or pass --host 0.0.0.0\n","'''\n","  ```\n","\n","<br>\n","\n","You will see a similar picture as above when you copy and paste http://localhost:8040/ into your browser and click \"A\" in the loaded webpage.  "]},{"cell_type":"markdown","metadata":{"id":"cMiMktCSUuNB"},"source":["<a name=\"4\"></a>\n","# 4. References\n","\n","- [VisualDL](https://github.com/PaddlePaddle/VisualDL)"]}]}