paddlenlp_updated_notes_English/WordEmbedding/1-loading pre-trained word embedding in paddlenp.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"1-loading pre-trained word embedding in paddlenp.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"code","metadata":{"id":"1UGfBdj-rVY7"},"source":["# Author: Zhengxiang (Jack) Wang\n","# Date: 2021-07-18\n","# GitHub: https://github.com/jaaack-wang\n","# About: Loading pre-trained word embedding in paddlenlp."],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-uPa9buBVK8a"},"source":["# Overview\n","\n","`paddlenlp` provides an [Embedding API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/embeddings.html) that can easily loads various open-source pre-trained word embedding models in just one line of code when calling `padddlenlp.embeddings.TokenEmbedding`. All you need to do is to correctly specify the name of the provided word embedding model inside the `TokenEmbedding` class. Besides, this notebook will also serve as a brief introudction to these word embedding models that can be natively loaded in `paddlenlp`.\n","\n","\n","<table align=\"right\">\n","  <td>\n","    <a target=\"_blank\" href=\"https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab </a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://github.com/jaaack-wang\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> Author's GitHub </a>\n","  </td>\n","  <td>\n","    <a href=\"https://docs.google.com/uc?export=download&id=1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /> Download this notebook </a>\n","  </td>\n","</table> \n","\n","\n","<br>\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"-NoQZ3vOaA22"},"source":["# Table of Contents\n","\n","- [1. Loading pre-trained word embedding models](#1)\n","  - [1.1 Make sure `paddlepaddle` and `paddlenlp` have been installed](#1-1)\n","  - [1.2 Two ways of loading a word embedding model](#1-2)\n","    - [1.2.1 First way: Specifying the model's name inside padddlenlp.embeddings.TokenEmbedding](#1-2-1)\n","    - [1.2.2 Second way: Manually load a model](#1-2-2)\n","- [2. Pre-trained word embedding available in `paddlenlp`](#2)\n","  - [2.1 Chinese word embedding models](#2-1)\n","  - [2.2 English word embedding models](#2-2)\n","  - [2.3 General info of these models](#2-3)\n","- [3. References](#3)"]},{"cell_type":"markdown","metadata":{"id":"vIaZqLPSc4qI"},"source":["<a name=\"1\"></a>\n","# 1. Loading pre-trained word embedding models"]},{"cell_type":"markdown","metadata":{"id":"qegXjYMtdTWs"},"source":["<a name=\"1-1\"></a>\n","### 1.1 Make sure `paddlepaddle` and `paddlenlp` have been installed"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QwqT2z67YHDr","executionInfo":{"status":"ok","timestamp":1626646676571,"user_tz":360,"elapsed":3186,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"04fd70aa-50ac-45c2-83ef-ff8fb2bbfa2b"},"source":["# First make sure that you have paddlepaddle installed before using paddlenlp\n","!pip3 install --upgrade paddlepaddle"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Requirement already up-to-date: paddlepaddle in /usr/local/lib/python3.7/dist-packages (2.1.1)\n","Requirement already satisfied, skipping upgrade: protobuf>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (3.17.3)\n","Requirement already satisfied, skipping upgrade: gast>=0.3.3; platform_system != \"Windows\" in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (0.4.0)\n","Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (1.15.0)\n","Requirement already satisfied, skipping upgrade: Pillow in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (7.1.2)\n","Requirement already satisfied, skipping upgrade: numpy>=1.13; python_version >= \"3.5\" and platform_system != \"Windows\" in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (1.19.5)\n","Requirement already satisfied, skipping upgrade: decorator==4.4.2 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (4.4.2)\n","Requirement already satisfied, skipping upgrade: astor in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (0.8.1)\n","Requirement already satisfied, skipping upgrade: requests>=2.20.0 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (2.23.0)\n","Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (3.0.4)\n","Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (2021.5.30)\n","Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (1.24.3)\n","Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (2.10)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ThoyriSIdo5t","executionInfo":{"status":"ok","timestamp":1626646679702,"user_tz":360,"elapsed":3137,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"9f9a0c36-18c2-4d7e-fede-6d14e5605a26"},"source":["# install the lastest paddlenlp\n","!pip3 install --upgrade paddlenlp"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Requirement already up-to-date: paddlenlp in /usr/local/lib/python3.7/dist-packages (2.0.5)\n","Requirement already satisfied, skipping upgrade: visualdl in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (2.2.0)\n","Requirement already satisfied, skipping upgrade: jieba in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (0.42.1)\n","Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (0.4.4)\n","Requirement already satisfied, skipping upgrade: h5py in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (3.1.0)\n","Requirement already satisfied, skipping upgrade: seqeval in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (1.2.2)\n","Requirement already satisfied, skipping upgrade: multiprocess in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (0.70.12.2)\n","Requirement already satisfied, skipping upgrade: colorlog in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (5.0.1)\n","Requirement already satisfied, skipping upgrade: bce-python-sdk in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (0.8.61)\n","Requirement already satisfied, skipping upgrade: flask>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.1.4)\n","Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.19.5)\n","Requirement already satisfied, skipping upgrade: shellcheck-py in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (0.7.2.1)\n","Requirement already satisfied, skipping upgrade: six>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.15.0)\n","Requirement already satisfied, skipping upgrade: protobuf>=3.11.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (3.17.3)\n","Requirement already satisfied, skipping upgrade: Flask-Babel>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (2.0.0)\n","Requirement already satisfied, skipping upgrade: pandas in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.1.5)\n","Requirement already satisfied, skipping upgrade: pre-commit in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (2.13.0)\n","Requirement already satisfied, skipping upgrade: matplotlib in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (3.2.2)\n","Requirement already satisfied, skipping upgrade: flake8>=3.7.9 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (3.9.2)\n","Requirement already satisfied, skipping upgrade: Pillow>=7.0.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (7.1.2)\n","Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (2.23.0)\n","Requirement already satisfied, skipping upgrade: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py->paddlenlp) (1.5.2)\n","Requirement already satisfied, skipping upgrade: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval->paddlenlp) (0.22.2.post1)\n","Requirement already satisfied, skipping upgrade: dill>=0.3.4 in /usr/local/lib/python3.7/dist-packages (from multiprocess->paddlenlp) (0.3.4)\n","Requirement already satisfied, skipping upgrade: future>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from bce-python-sdk->visualdl->paddlenlp) (0.16.0)\n","Requirement already satisfied, skipping upgrade: pycryptodome>=3.8.0 in /usr/local/lib/python3.7/dist-packages (from bce-python-sdk->visualdl->paddlenlp) (3.10.1)\n","Requirement already satisfied, skipping upgrade: Werkzeug<2.0,>=0.15 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.0.1)\n","Requirement already satisfied, skipping upgrade: Jinja2<3.0,>=2.10.1 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (2.11.3)\n","Requirement already satisfied, skipping upgrade: click<8.0,>=5.1 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (7.1.2)\n","Requirement already satisfied, skipping upgrade: itsdangerous<2.0,>=0.24 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.1.0)\n","Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.7/dist-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2018.9)\n","Requirement already satisfied, skipping upgrade: Babel>=2.3 in /usr/local/lib/python3.7/dist-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2.9.1)\n","Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->visualdl->paddlenlp) (2.8.1)\n","Requirement already satisfied, skipping upgrade: identify>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (2.2.11)\n","Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (4.6.1)\n","Requirement already satisfied, skipping upgrade: nodeenv>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (1.6.0)\n","Requirement already satisfied, skipping upgrade: cfgv>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (3.3.0)\n","Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (0.10.2)\n","Requirement already satisfied, skipping upgrade: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (5.4.1)\n","Requirement already satisfied, skipping upgrade: virtualenv>=20.0.8 in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (20.6.0)\n","Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (2.4.7)\n","Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (0.10.0)\n","Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (1.3.1)\n","Requirement already satisfied, skipping upgrade: pycodestyle<2.8.0,>=2.7.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.7.9->visualdl->paddlenlp) (2.7.0)\n","Requirement already satisfied, skipping upgrade: mccabe<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.7.9->visualdl->paddlenlp) (0.6.1)\n","Requirement already satisfied, skipping upgrade: pyflakes<2.4.0,>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.7.9->visualdl->paddlenlp) (2.3.1)\n","Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (3.0.4)\n","Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (2021.5.30)\n","Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (1.24.3)\n","Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (2.10)\n","Requirement already satisfied, skipping upgrade: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.4.1)\n","Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.0.1)\n","Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2<3.0,>=2.10.1->flask>=1.1.1->visualdl->paddlenlp) (2.0.1)\n","Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->pre-commit->visualdl->paddlenlp) (3.7.4.3)\n","Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->pre-commit->visualdl->paddlenlp) (3.5.0)\n","Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->visualdl->paddlenlp) (3.0.12)\n","Requirement already satisfied, skipping upgrade: platformdirs<3,>=2 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->visualdl->paddlenlp) (2.0.2)\n","Requirement already satisfied, skipping upgrade: backports.entry-points-selectable>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->visualdl->paddlenlp) (1.1.0)\n","Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.1 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->visualdl->paddlenlp) (0.3.2)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"uPatJ_-qd6vC"},"source":["<a name=\"1-2\"></a>\n","### 1.2 Two ways of loading a word embedding model\n","- Specifying the model's name inside `padddlenlp.embeddings.TokenEmbedding`\n","- Manually download a model\n","<br><br>\n","\n","First, we need to import `padddlenlp.embeddings.TokenEmbedding`.\n","\n","\n","**Reference**: [TokenEmbedding](https://paddlenlp.readthedocs.io/zh/latest/source/paddlenlp.embeddings.token_embedding.html) \n","\n","**Parameters or functions of interest**:\n","\n","- **embedding_name (str, optional)** -- The pre-trained embedding model name. Use **paddlenlp.embeddings.list_embedding_name()** to list the names of all embedding models available. Defaults to <ins>w2v.baidu_encyclopedia.target.word-word.dim300</ins>.\n","  - **`vocab.to_tokens (indices)`**: indices: `list` or `int`. Rerturns: corresponding tokens in the vocabulary of the model.\n","  - **`search(words)`**: words: `list` or `str` or `int`. Retruns: the vectors of specified words."]},{"cell_type":"code","metadata":{"id":"yLdA8Wyldy8j"},"source":["# import padddlenlp.embeddings.TokenEmbedding\n","from paddlenlp.embeddings import TokenEmbedding"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vOIcmmNGfO-b"},"source":["<a name=\"1-2-1\"></a>\n","##### **1.2.1 First way: Specifying the model's name inside padddlenlp.embeddings.TokenEmbedding**\n","\n","Even if you do not have the embedding model installed beforehand, `paddlenlp` will automatically download it for you."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"wi5CGyNAfOLN","executionInfo":{"status":"ok","timestamp":1626646681522,"user_tz":360,"elapsed":10,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"7a04c103-30a9-4c97-fcc3-b6751fc28274"},"source":["# w2v.sikuquanshu.target.word-word.dim300 is the smallest model, only 20.7 MB\n","token_embedding = TokenEmbedding(embedding_name=\"w2v.sikuquanshu.target.word-word.dim300\")\n","\n","# print the loaded model info\n","print(token_embedding)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["\u001b[32m[2021-07-18 22:18:01,280] [    INFO]\u001b[0m - Loading token embedding...\u001b[0m\n","\u001b[32m[2021-07-18 22:18:01,484] [    INFO]\u001b[0m - Finish loading embedding vector.\u001b[0m\n","\u001b[32m[2021-07-18 22:18:01,492] [    INFO]\u001b[0m - Token Embedding info:             \n","Unknown index: 19527             \n","Unknown token: [UNK]             \n","Padding index: 19528             \n","Padding token: [PAD]             \n","Shape :[19529, 300]\u001b[0m\n"],"name":"stderr"},{"output_type":"stream","text":["Object   type: TokenEmbedding(19529, 300, padding_idx=19528, sparse=False)             \n","Unknown index: 19527             \n","Unknown token: [UNK]             \n","Padding index: 19528             \n","Padding token: [PAD]             \n","Parameter containing:\n","Tensor(shape=[19529, 300], dtype=float32, place=CPUPlace, stop_gradient=False,\n","       [[ 0.10636300, -0.15333299, -0.00901000, ...,  0.08603100, -0.21248300, -0.11166500],\n","        [ 0.03748500, -0.02521100,  0.13308799, ...,  0.10142900, -0.23215801,  0.02003300],\n","        [ 0.12701400, -0.05975900,  0.09185100, ...,  0.06476500, -0.24700700, -0.10331500],\n","        ...,\n","        [-0.08905500,  0.03231100,  0.10511100, ..., -0.01329400,  0.01188400, -0.02801500],\n","        [ 0.01407126, -0.02133216, -0.00614622, ...,  0.05757869, -0.02248066,  0.01433324],\n","        [ 0.        ,  0.        ,  0.        , ...,  0.        ,  0.        ,  0.        ]])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5EdOYBq1ebpB","executionInfo":{"status":"ok","timestamp":1626646681748,"user_tz":360,"elapsed":17,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"3b85d1cf-902f-48ba-cdda-c3d6e9678db3"},"source":["# We can check the vocabulary in the model as follows:\n","# As we can see above, the vocabulary size is 19529\n","vocab = token_embedding.vocab.to_tokens(list(range(19529)))\n","\n","# check the first 20 vocabularies\n","print('This is the first 20 words in the model\\' vocabulary:\\n\\n', vocab[:20])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["This is the first 20 words in the model' vocabulary:\n","\n"," ['之', '不', '以', '】', '【', '为', '而', '也', '其', '于', '曰', '丨', '子', '有', '人', '防', '云', '无', '中', '一']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ivR9x-ocjLIj","executionInfo":{"status":"ok","timestamp":1626646681749,"user_tz":360,"elapsed":16,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"5da521d5-e4d4-43b9-c890-b0a51e799b29"},"source":["# To check the corresponding embedding of a given word/token\n","# using TokenEmbedding().search(word)\n","w_e = token_embedding.search('之')\n","\n","print('This is the pre-trained word embedding (300 dims) for 之 in the model: \\n\\n', w_e)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["This is the pre-trained word embedding (300 dims) for 之 in the model: \n","\n"," [[ 1.06363e-01 -1.53333e-01 -9.01000e-03  1.62510e-01 -9.58900e-03\n","  -1.20850e-01  1.02605e-01  3.34600e-03  1.35963e-01  2.41380e-02\n","  -4.15040e-02  4.03810e-02  1.36350e-02 -7.29750e-02 -8.56830e-02\n","  -1.13738e-01  4.78800e-03 -1.21901e-01  5.74850e-02 -7.23350e-02\n","  -2.18830e-01  1.52990e-02  9.93080e-02  5.20510e-02  2.95950e-02\n","   1.19608e-01 -1.05340e-02 -1.18171e-01  2.17480e-02 -1.03129e-01\n","   1.92324e-01  5.96840e-02 -8.32330e-02 -2.02900e-02  9.60600e-03\n","  -7.17610e-02 -3.91730e-02 -1.14774e-01 -2.03384e-01  9.58800e-03\n","   7.48400e-02  3.13860e-02  3.51280e-02  3.80000e-05 -6.72880e-02\n","   5.08650e-02 -8.90020e-02  4.20170e-02 -1.87957e-01  1.58577e-01\n","   4.19750e-02  1.68900e-01 -1.11841e-01  3.50260e-02  3.53730e-02\n","  -5.59850e-02 -4.40030e-02  1.55936e-01  3.93170e-02 -5.75030e-02\n","   1.07756e-01 -8.60630e-02 -2.31200e-02 -6.34500e-03  1.56109e-01\n","  -1.43128e-01 -2.75030e-02  5.26310e-02 -6.49380e-02 -5.95500e-02\n","   1.18940e-02 -1.27600e-02  4.45020e-02 -2.81580e-02  9.77200e-03\n","   1.14870e-01 -9.62100e-03 -1.46720e-02  7.13530e-02 -3.33840e-02\n","  -4.82290e-02  8.50870e-02  3.82010e-02 -1.84780e-02 -1.23120e-01\n","   1.12867e-01  6.13580e-02  4.65490e-02 -9.46180e-02 -3.05080e-02\n","  -5.24650e-02 -1.70357e-01 -6.10660e-02 -3.99200e-03  1.74493e-01\n","   1.39420e-02 -3.64770e-02  3.28940e-02  5.19000e-04 -7.54000e-03\n","  -1.00379e-01  8.98000e-03  1.27680e-01  6.70580e-02  7.75070e-02\n","   2.45010e-02  4.97910e-02  6.29880e-02  4.40410e-02 -3.91540e-02\n","  -8.10840e-02  1.16985e-01 -6.33890e-02 -2.78080e-02  2.11550e-02\n","  -6.76680e-02  2.67200e-03  1.23416e-01  2.75110e-02 -1.46857e-01\n","  -5.75650e-02 -3.68020e-02  1.72990e-02  3.57210e-02  1.40310e-01\n","   2.10250e-02  1.17190e-01  7.74300e-02 -7.72080e-02 -3.54640e-02\n","  -8.27420e-02  8.79120e-02  9.87600e-03  1.78216e-01  1.05112e-01\n","   7.74030e-02 -8.26880e-02  2.27880e-02  3.92520e-02 -1.08174e-01\n","   5.04800e-02 -1.55930e-01 -8.00400e-02  1.15855e-01  7.57800e-02\n","   1.39190e-02 -6.68810e-02 -4.40160e-02 -1.22727e-01  1.71470e-01\n","  -2.32910e-02  1.45799e-01 -1.40552e-01 -1.91700e-01 -7.59260e-02\n","  -1.93803e-01 -4.98100e-03 -5.19030e-02 -1.09141e-01  1.21824e-01\n","   2.46580e-02 -6.11690e-02  9.13610e-02  9.96830e-02 -1.35947e-01\n","   3.10310e-02 -1.77313e-01 -1.63034e-01  9.54200e-02 -1.85308e-01\n","   8.66900e-02 -4.66020e-02  8.75840e-02 -7.27900e-02 -9.97510e-02\n","  -8.97070e-02  1.09540e-02  1.19610e-01 -4.87390e-02 -1.65680e-02\n","  -6.22160e-02  4.64600e-03 -2.50350e-02 -8.35010e-02 -2.08369e-01\n","  -2.77970e-02 -1.90495e-01 -7.27210e-02 -1.13147e-01 -2.72730e-02\n","   1.13924e-01  1.48356e-01  5.93700e-02 -2.63761e-01  5.67800e-03\n","  -6.18500e-02  4.78650e-02 -1.00666e-01 -7.16990e-02 -9.69490e-02\n","   7.90770e-02  1.80410e-02 -7.24970e-02 -1.12320e-01  1.95360e-02\n","   7.96180e-02  1.16354e-01  6.60160e-02 -1.52620e-02  5.41550e-02\n","  -3.31590e-02 -1.17317e-01 -1.86910e-02 -1.39360e-02 -4.58330e-02\n","  -1.17503e-01 -2.02318e-01 -5.48050e-02 -1.35268e-01  1.80870e-02\n","   8.43930e-02  1.12903e-01  6.19500e-03 -3.10130e-02 -2.91050e-02\n","   3.50430e-02 -9.76380e-02 -3.61650e-02 -3.05190e-02 -1.52231e-01\n","   1.25414e-01 -1.07045e-01  4.39380e-02 -3.17570e-02 -9.46590e-02\n","  -5.38900e-02 -1.40460e-02  6.39180e-02  6.08040e-02  3.99130e-02\n","  -1.92557e-01 -1.27382e-01 -1.24932e-01 -1.33961e-01 -9.80000e-04\n","   3.69010e-02 -1.91448e-01 -2.42900e-03  5.79160e-02 -2.55936e-01\n","  -9.95450e-02 -2.86910e-02 -4.94020e-02  1.34884e-01  8.87010e-02\n","  -8.04300e-02 -6.44840e-02  8.20050e-02  3.46260e-02 -8.37020e-02\n","   1.79600e-02 -2.71250e-02 -1.30220e-02 -3.59580e-02 -7.19600e-03\n","   1.24198e-01  7.47590e-02  7.83700e-02  8.78310e-02 -9.94710e-02\n","  -7.53110e-02 -1.11116e-01 -7.93190e-02  1.92650e-02 -1.07024e-01\n","   5.32800e-03  1.35023e-01 -6.07900e-03  4.18260e-02 -1.80959e-01\n","  -7.85650e-02  8.37910e-02 -5.40100e-02 -9.47020e-02 -7.81780e-02\n","   3.26160e-02  6.83600e-02 -1.34954e-01  1.06028e-01 -3.88640e-02\n","  -2.99640e-02  1.06011e-01 -2.03233e-01 -1.35940e-02  4.58320e-02\n","  -1.22418e-01 -1.03397e-01  8.60310e-02 -2.12483e-01 -1.11665e-01]]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Oag3t3OXl17j"},"source":["<a name=\"1-2-2\"></a>\n","##### **1.2.2 Second way: Manually load a model**\n","\n","- First, you need to download the embedding model from the internet and uncompress it; \n","- Then, you should move the uncompressed model file to <ins>`HOME_DIRECRORY/.paddlenlp/models/embeddings/`</ins>;\n","- Then, convert the model file into `\".npz\"` format. \n","- Finally call the model file by its name using `TokenEmbedding(embedding_name=...)` as shown above. \n","\n","<br>\n","\n","---\n","\n","Please note that: \n","- 1. `HOME_DIRECRORY` refers to the one you see when you first open you terminal/command shell on your computer; \n","- 2. `\".paddlenlp\"` is a hidden directory that normally be cannot be seen on your computer. To access it, on mac, you can make them visible by pressing `command` + `shift` + `.` and then move the model file inside `~/models/embeddings/` directory. You can also do the same thing by programming.  \n","- 3. The `.npz` should have two arrays named as ***vocab***, ***embedding***. You can see how this can be done by checking [`numpy.savez`](https://numpy.org/doc/stable/reference/generated/numpy.savez.html).\n","\n","\n","---\n","\n","<br>\n","\n","\n","Below, let's take \"glove.6B.50d\" downloaded from [Stanford GloVe](https://nlp.stanford.edu/projects/glove/) as an example, although this model is already available in `padddlenlp.embeddings.TokenEmbedding` by using the `embedding_name=glove.wiki2014-gigaword.target.word-word.dim50.en`. The original \"glove.6B.50d\" file is in \".txt\" format. \n","\n","If yo run the following codes on Colab, please save [this file](https://drive.google.com/file/d/1o1fUeoAt260P90FeP_L5eICiQowHIcvY/view?usp=sharing) to \"My Drive\" (you can either download the file and then upload or simply add a shortcut of this file to \"My Drive\" by clicking \"Add shortcut to Drive\"). \n","\n","<br>\n","\n","**More on how to handle external files in Colab can be found [here](https://colab.research.google.com/notebooks/io.ipynb).**"]},{"cell_type":"code","metadata":{"id":"hnbWoMa_kKy_","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1626646681750,"user_tz":360,"elapsed":13,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"7a127401-f41a-49d5-933c-8bdc36b242e0"},"source":["from google.colab import drive\n","drive.mount('/drive')\n","file_path = '/drive/My Drive/glove.6B.50d.txt'"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Drive already mounted at /drive; to attempt to forcibly remount, call drive.mount(\"/drive\", force_remount=True).\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"YSxf8_tzOoS-"},"source":["import numpy as np\n","\n","\n","glove = open(file_path, 'r')\n","vocab, em = [], []\n","for line in glove:\n","  line = line.split(maxsplit=1)\n","  vocab.append(line[0])\n","  em.append(np.array(line[1].strip().split(), dtype=np.float32))\n","\n","\n","len(vocab), len(em)\n","assert len(vocab)==len(em), 'vocab size does not match the size of word embeddings'"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"z1yRtjc2PNEK"},"source":["# change the type of vocab and em from list to numpy.array\n","# before saving them as a .npz file\n","\n","vocab = np.asarray(vocab)\n","em = np.array(em)\n","\n","\n","# output file path \n","out_fp = '/root/.paddlenlp/models/embeddings/glove.6B.50d.npz'\n","np.savez(out_fp, vocab=vocab, embedding=em)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"oilGpUOpQ6dK","executionInfo":{"status":"ok","timestamp":1626646690489,"user_tz":360,"elapsed":1344,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"18794a09-c4c2-4eb6-eb51-d1a2d03d32f5"},"source":["from paddlenlp.embeddings import TokenEmbedding\n","\n","tk_em = TokenEmbedding(embedding_name='glove.6B.50d')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["\u001b[32m[2021-07-18 22:18:09,016] [    INFO]\u001b[0m - Loading token embedding...\u001b[0m\n","\u001b[32m[2021-07-18 22:18:10,382] [    INFO]\u001b[0m - Finish loading embedding vector.\u001b[0m\n","\u001b[32m[2021-07-18 22:18:10,384] [    INFO]\u001b[0m - Token Embedding info:             \n","Unknown index: 400000             \n","Unknown token: [UNK]             \n","Padding index: 400001             \n","Padding token: [PAD]             \n","Shape :[400002, 50]\u001b[0m\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zCjuLODiRrmG","executionInfo":{"status":"ok","timestamp":1626646738809,"user_tz":360,"elapsed":48325,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"4e4a2873-e630-4613-fa55-aaaebccf4cb7"},"source":["# let's de bug!\n","\n","def de_bug():\n","  errs = []\n","  tk_em_dict = dict(zip(vocab, em))\n","  for v in vocab:\n","    if np.all(tk_em_dict[v] != tk_em.search(v)):\n","      errs.append(v)\n","  \n","  if len(errs) == 0:\n","    print('No bug identified, congratulations!')\n","  \n","  return errs\n","\n","\n","de_bug()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["No bug identified, congratulations!\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["[]"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"markdown","metadata":{"id":"vQNz8WVXUqgv"},"source":["<a name=\"2\"></a>\n","# 2. Pre-trained word embedding available in `paddlenlp`\n","\n","All the pre-trained word embedding models can be found [here](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/embeddings/constant.py). You can also run the following code to access the names of these models. "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"MU9WaWj9YONB","executionInfo":{"status":"ok","timestamp":1626646738810,"user_tz":360,"elapsed":18,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"22cebda0-9c21-48b2-a984-04b5e2e9359c"},"source":["from paddlenlp.embeddings.constant import *\n","\n","counter = 0\n","tmp = '{0:30}{1:20}{2:30}'\n","for name in EMBEDDING_NAME_LIST:\n","  counter += 1\n","  print(tmp.format(f'embedding model {counter}', '------>', name))\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["embedding model 1             ------>             w2v.baidu_encyclopedia.target.word-word.dim300\n","embedding model 2             ------>             w2v.baidu_encyclopedia.target.word-character.char1-1.dim300\n","embedding model 3             ------>             w2v.baidu_encyclopedia.target.word-character.char1-2.dim300\n","embedding model 4             ------>             w2v.baidu_encyclopedia.target.word-character.char1-4.dim300\n","embedding model 5             ------>             w2v.baidu_encyclopedia.target.word-ngram.1-2.dim300\n","embedding model 6             ------>             w2v.baidu_encyclopedia.target.word-ngram.1-3.dim300\n","embedding model 7             ------>             w2v.baidu_encyclopedia.target.word-ngram.2-2.dim300\n","embedding model 8             ------>             w2v.baidu_encyclopedia.target.word-wordLR.dim300\n","embedding model 9             ------>             w2v.baidu_encyclopedia.target.word-wordPosition.dim300\n","embedding model 10            ------>             w2v.baidu_encyclopedia.target.bigram-char.dim300\n","embedding model 11            ------>             w2v.baidu_encyclopedia.context.word-word.dim300\n","embedding model 12            ------>             w2v.baidu_encyclopedia.context.word-character.char1-1.dim300\n","embedding model 13            ------>             w2v.baidu_encyclopedia.context.word-character.char1-2.dim300\n","embedding model 14            ------>             w2v.baidu_encyclopedia.context.word-character.char1-4.dim300\n","embedding model 15            ------>             w2v.baidu_encyclopedia.context.word-ngram.1-2.dim300\n","embedding model 16            ------>             w2v.baidu_encyclopedia.context.word-ngram.1-3.dim300\n","embedding model 17            ------>             w2v.baidu_encyclopedia.context.word-ngram.2-2.dim300\n","embedding model 18            ------>             w2v.baidu_encyclopedia.context.word-wordLR.dim300\n","embedding model 19            ------>             w2v.baidu_encyclopedia.context.word-wordPosition.dim300\n","embedding model 20            ------>             w2v.wiki.target.bigram-char.dim300\n","embedding model 21            ------>             w2v.wiki.target.word-char.dim300\n","embedding model 22            ------>             w2v.wiki.target.word-word.dim300\n","embedding model 23            ------>             w2v.wiki.target.word-bigram.dim300\n","embedding model 24            ------>             w2v.people_daily.target.bigram-char.dim300\n","embedding model 25            ------>             w2v.people_daily.target.word-char.dim300\n","embedding model 26            ------>             w2v.people_daily.target.word-word.dim300\n","embedding model 27            ------>             w2v.people_daily.target.word-bigram.dim300\n","embedding model 28            ------>             w2v.weibo.target.bigram-char.dim300\n","embedding model 29            ------>             w2v.weibo.target.word-char.dim300\n","embedding model 30            ------>             w2v.weibo.target.word-word.dim300\n","embedding model 31            ------>             w2v.weibo.target.word-bigram.dim300\n","embedding model 32            ------>             w2v.sogou.target.bigram-char.dim300\n","embedding model 33            ------>             w2v.sogou.target.word-char.dim300\n","embedding model 34            ------>             w2v.sogou.target.word-word.dim300\n","embedding model 35            ------>             w2v.sogou.target.word-bigram.dim300\n","embedding model 36            ------>             w2v.zhihu.target.bigram-char.dim300\n","embedding model 37            ------>             w2v.zhihu.target.word-char.dim300\n","embedding model 38            ------>             w2v.zhihu.target.word-word.dim300\n","embedding model 39            ------>             w2v.zhihu.target.word-bigram.dim300\n","embedding model 40            ------>             w2v.financial.target.bigram-char.dim300\n","embedding model 41            ------>             w2v.financial.target.word-char.dim300\n","embedding model 42            ------>             w2v.financial.target.word-word.dim300\n","embedding model 43            ------>             w2v.financial.target.word-bigram.dim300\n","embedding model 44            ------>             w2v.literature.target.bigram-char.dim300\n","embedding model 45            ------>             w2v.literature.target.word-char.dim300\n","embedding model 46            ------>             w2v.literature.target.word-word.dim300\n","embedding model 47            ------>             w2v.literature.target.word-bigram.dim300\n","embedding model 48            ------>             w2v.sikuquanshu.target.word-word.dim300\n","embedding model 49            ------>             w2v.sikuquanshu.target.word-bigram.dim300\n","embedding model 50            ------>             w2v.mixed-large.target.word-char.dim300\n","embedding model 51            ------>             w2v.mixed-large.target.word-word.dim300\n","embedding model 52            ------>             w2v.google_news.target.word-word.dim300.en\n","embedding model 53            ------>             glove.wiki2014-gigaword.target.word-word.dim50.en\n","embedding model 54            ------>             glove.wiki2014-gigaword.target.word-word.dim100.en\n","embedding model 55            ------>             glove.wiki2014-gigaword.target.word-word.dim200.en\n","embedding model 56            ------>             glove.wiki2014-gigaword.target.word-word.dim300.en\n","embedding model 57            ------>             glove.twitter.target.word-word.dim25.en\n","embedding model 58            ------>             glove.twitter.target.word-word.dim50.en\n","embedding model 59            ------>             glove.twitter.target.word-word.dim100.en\n","embedding model 60            ------>             glove.twitter.target.word-word.dim200.en\n","embedding model 61            ------>             fasttext.wiki-news.target.word-word.dim300.en\n","embedding model 62            ------>             fasttext.crawl.target.word-word.dim300.en\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"WQXSD-KVVE18"},"source":["<a name=\"2-1\"></a>\n","### 2.1 Chinese word embedding models\n","\n","The following pre-trained Chinese word embedding models come from [Chinese-Word-Vectors](https://github.com/Embedding/Chinese-Word-Vectors)\n","\n","<br>\n","\n","Some explanations (for the following headers): \n","\n","- Corpus: The corpus based on which a corresponding word embedding is trained. \n","- Word: During the pre-training process, the predicted target is at word-level. \n","- Word + N-gram: the predicted target is eitehr a word or a n-gram. \n","- Word + Character: the predicted target is eitehr a word or a character. \"word-character.char1-2\" means the length of the predicted character is either 1 or 2. \n","- Word + Character + Ngram: the predicted target is either a word, or a character, or a ngram. \"ngram-char\" means ngram at the level of characters. \n"]},{"cell_type":"markdown","metadata":{"id":"zRwVJcaQbB3R"},"source":["| Corpus | Word | Word + N-gram | Word + Character | Word + Character + N-gram |\n","| ------------------------------------------- | ----   | ---- | ----   | ---- |\n","| Baidu Encyclopedia 百度百科                 | w2v.baidu_encyclopedia.target.word-word.dim300 | w2v.baidu_encyclopedia.target.word-ngram.1-2.dim300 | w2v.baidu_encyclopedia.target.word-character.char1-2.dim300 | w2v.baidu_encyclopedia.target.bigram-char.dim300 |\n","| Wikipedia_zh 中文维基百科                   | w2v.wiki.target.word-word.dim300 | w2v.wiki.target.word-bigram.dim300 | w2v.wiki.target.word-char.dim300 | w2v.wiki.target.bigram-char.dim300 |\n","| People's Daily News 人民日报                | w2v.people_daily.target.word-word.dim300 | w2v.people_daily.target.word-bigram.dim300 | w2v.people_daily.target.word-char.dim300 | w2v.people_daily.target.bigram-char.dim300 |\n","| Sogou News 搜狗新闻                         | w2v.sogou.target.word-word.dim300 | w2v.sogou.target.word-bigram.dim300 | w2v.sogou.target.word-char.dim300 | w2v.sogou.target.bigram-char.dim300 |\n","| Financial News 金融新闻                     | w2v.financial.target.word-word.dim300 | w2v.financial.target.word-bigram.dim300 | w2v.financial.target.word-char.dim300 | w2v.financial.target.bigram-char.dim300 |\n","| Zhihu_QA 知乎问答                           | w2v.zhihu.target.word-word.dim300 | w2v.zhihu.target.word-bigram.dim300 | w2v.zhihu.target.word-char.dim300 | w2v.zhihu.target.bigram-char.dim300 |\n","| Weibo 微博                                  | w2v.weibo.target.word-word.dim300 | w2v.weibo.target.word-bigram.dim300 | w2v.weibo.target.word-char.dim300 | w2v.weibo.target.bigram-char.dim300 |\n","| Literature 文学作品                         | w2v.literature.target.word-word.dim300 | w2v.literature.target.word-bigram.dim300 | w2v.literature.target.word-char.dim300 | w2v.literature.target.bigram-char.dim300 |\n","| Complete Library in Four Sections 四库全书  | w2v.sikuquanshu.target.word-word.dim300 | w2v.sikuquanshu.target.word-bigram.dim300 | 无 | 无 |\n","| Mixed-large 综合                            | w2v.mixed-large.target.word-word.dim300 | 暂无 | w2v.mixed-large.target.word-word.dim300 | 暂无 |"]},{"cell_type":"markdown","metadata":{"id":"3nIL961ubVQi"},"source":["<br>\n","\n","In particular, for the Baidu Encyclopedia, target vectors and context vectors are also provided based on different co-occurence types. \n","\n","<br>\n","\n","\n","| Co-occurrence type          | Target vector | Context vector  |\n","| --------------------------- | ------   | ---- |\n","|    Word → Word              | w2v.baidu_encyclopedia.target.word-word.dim300     |   w2v.baidu_encyclopedia.context.word-word.dim300    |\n","|    Word → Ngram (1-2)       |  w2v.baidu_encyclopedia.target.word-ngram.1-2.dim300    |   w2v.baidu_encyclopedia.context.word-ngram.1-2.dim300    |\n","|    Word → Ngram (1-3)       |  w2v.baidu_encyclopedia.target.word-ngram.1-3.dim300    |   w2v.baidu_encyclopedia.context.word-ngram.1-3.dim300    |\n","|    Ngram (1-2) → Ngram (1-2)|  w2v.baidu_encyclopedia.target.word-ngram.2-2.dim300   |   w2v.baidu_encyclopedia.target.word-ngram.2-2.dim300    |\n","|    Word → Character (1)     |  w2v.baidu_encyclopedia.target.word-character.char1-1.dim300    |  w2v.baidu_encyclopedia.context.word-character.char1-1.dim300     |\n","|    Word → Character (1-2)   |  w2v.baidu_encyclopedia.target.word-character.char1-2.dim300    |  w2v.baidu_encyclopedia.context.word-character.char1-2.dim300     |\n","|    Word → Character (1-4)   |  w2v.baidu_encyclopedia.target.word-character.char1-4.dim300    |  w2v.baidu_encyclopedia.context.word-character.char1-4.dim300     |\n","|    Word → Word (left/right) |   w2v.baidu_encyclopedia.target.word-wordLR.dim300   |   w2v.baidu_encyclopedia.context.word-wordLR.dim300    |\n","|    Word → Word (distance)   |   w2v.baidu_encyclopedia.target.word-wordPosition.dim300   |   w2v.baidu_encyclopedia.context.word-wordPosition.dim300    |"]},{"cell_type":"markdown","metadata":{"id":"7ZmMm0cpVPGq"},"source":["<a name=\"2-2\"></a>\n","### 2.2 English word embedding models\n","\n","<br>\n","\n","- Word2Vec\n","  - Google News: w2v.google_news.target.word-word.dim300.en\n","- [FastText](https://fasttext.cc/docs/en/english-vectors.html)\n","  - Wiki2017: fasttext.wiki-news.target.word-word.dim300.en\n","  - Crawl: fasttext.crawl.target.word-word.dim300.en \n"]},{"cell_type":"markdown","metadata":{"id":"sc3WcrO3bgdd"},"source":["- [GloVe](https://nlp.stanford.edu/projects/glove/)\n","\n","\n","| Corpus | Wiki2014 + GigaWord | Twitter |\n","| ---------------  | -------------  | ------  |\n","| 25 dims   | NA  | glove.twitter.target.word-word.dim25.en  |\n","| 50 dims   | glove.wiki2014-gigaword.target.word-word.dim50.en  | glove.twitter.target.word-word.dim50.en  |\n","| 100 dims   | glove.wiki2014-gigaword.target.word-word.dim100.en | glove.twitter.target.word-word.dim100.en  |\n","| 200 dims   | glove.wiki2014-gigaword.target.word-word.dim200.en  | glove.twitter.target.word-word.dim200.en  |\n","| 300 dims   | glove.wiki2014-gigaword.target.word-word.dim300.en  | NA  |\n","\n","\n","<br>"]},{"cell_type":"markdown","metadata":{"id":"J2j-12GCVWcc"},"source":["<a name=\"2-3\"></a>\n","### 2.3 General info of these models\n"]},{"cell_type":"markdown","metadata":{"id":"1tykiWL6bydP"},"source":["| Model | File size | Vocab size |\n","|-----|---------|---------|\n","| w2v.baidu_encyclopedia.target.word-word.dim300                         | 678.21 MB  | 635965 |\n","| w2v.baidu_encyclopedia.target.word-character.char1-1.dim300            | 679.15 MB  | 636038 |\n","| w2v.baidu_encyclopedia.target.word-character.char1-2.dim300            | 679.30 MB  | 636038 |\n","| w2v.baidu_encyclopedia.target.word-character.char1-4.dim300            | 679.51 MB  | 636038 |\n","| w2v.baidu_encyclopedia.target.word-ngram.1-2.dim300                    | 679.48 MB  | 635977 |\n","| w2v.baidu_encyclopedia.target.word-ngram.1-3.dim300                    | 671.27 MB  | 628669 |\n","| w2v.baidu_encyclopedia.target.word-ngram.2-2.dim300                    | 7.28 GB    | 6969069 |\n","| w2v.baidu_encyclopedia.target.word-wordLR.dim300                       | 678.22 MB  | 635958 |\n","| w2v.baidu_encyclopedia.target.word-wordPosition.dim300                 | 679.32 MB  | 636038 |\n","| w2v.baidu_encyclopedia.target.bigram-char.dim300                       | 679.29 MB  | 635976 |\n","| w2v.baidu_encyclopedia.context.word-word.dim300                        | 677.74 MB  | 635952 |\n","| w2v.baidu_encyclopedia.context.word-character.char1-1.dim300           | 678.65 MB  | 636200 |\n","| w2v.baidu_encyclopedia.context.word-character.char1-2.dim300           | 844.23 MB  | 792631 |\n","| w2v.baidu_encyclopedia.context.word-character.char1-4.dim300           | 1.16 GB    | 1117461 |\n","| w2v.baidu_encyclopedia.context.word-ngram.1-2.dim300                   | 7.25 GB    | 6967598 |\n","| w2v.baidu_encyclopedia.context.word-ngram.1-3.dim300                   | 5.21 GB    | 5000001 |\n","| w2v.baidu_encyclopedia.context.word-ngram.2-2.dim300                   | 7.26 GB    | 6968998 |\n","| w2v.baidu_encyclopedia.context.word-wordLR.dim300                      | 1.32 GB    | 1271031 |\n","| w2v.baidu_encyclopedia.context.word-wordPosition.dim300                | 6.47 GB    | 6293920 |\n","| w2v.wiki.target.bigram-char.dim300                                     | 375.98 MB  | 352274 |\n","| w2v.wiki.target.word-char.dim300                                       | 375.52 MB  | 352223 |\n","| w2v.wiki.target.word-word.dim300                                       | 374.95 MB  | 352219 |\n","| w2v.wiki.target.word-bigram.dim300                                     | 375.72 MB  | 352219 |\n","| w2v.people_daily.target.bigram-char.dim300                             | 379.96 MB  | 356055 |\n","| w2v.people_daily.target.word-char.dim300                               | 379.45 MB  | 355998 |\n","| w2v.people_daily.target.word-word.dim300                               | 378.93 MB  | 355989 |\n","| w2v.people_daily.target.word-bigram.dim300                             | 379.68 MB  | 355991 |\n","| w2v.weibo.target.bigram-char.dim300                                    | 208.24 MB  | 195199 |\n","| w2v.weibo.target.word-char.dim300                                      | 208.03 MB  | 195204 |\n","| w2v.weibo.target.word-word.dim300                                      | 207.94 MB  | 195204 |\n","| w2v.weibo.target.word-bigram.dim300                                    | 208.19 MB  | 195204 |\n","| w2v.sogou.target.bigram-char.dim300                                    | 389.81 MB  | 365112 |\n","| w2v.sogou.target.word-char.dim300                                      | 389.89 MB  | 365078 |\n","| w2v.sogou.target.word-word.dim300                                      | 388.66 MB  | 364992 |\n","| w2v.sogou.target.word-bigram.dim300                                    | 388.66 MB  | 364994 |\n","| w2v.zhihu.target.bigram-char.dim300                                    | 277.35 MB  | 259755 |\n","| w2v.zhihu.target.word-char.dim300                                      | 277.40 MB  | 259940 |\n","| w2v.zhihu.target.word-word.dim300                                      | 276.98 MB  | 259871 |\n","| w2v.zhihu.target.word-bigram.dim300                                    | 277.53 MB  | 259885 |\n","| w2v.financial.target.bigram-char.dim300                                | 499.52 MB  | 467163 |\n","| w2v.financial.target.word-char.dim300                                  | 499.17 MB  | 467343 |\n","| w2v.financial.target.word-word.dim300                                  | 498.94 MB  | 467324 |\n","| w2v.financial.target.word-bigram.dim300                                | 499.54 MB  | 467331 |\n","| w2v.literature.target.bigram-char.dim300                               | 200.69 MB  | 187975 |\n","| w2v.literature.target.word-char.dim300                                 | 200.44 MB  | 187980 |\n","| w2v.literature.target.word-word.dim300                                 | 200.28 MB  | 187961 |\n","| w2v.literature.target.word-bigram.dim300                               | 200.59 MB  | 187962 |\n","| w2v.sikuquanshu.target.word-word.dim300                                | 20.70 MB   | 19529 |\n","| w2v.sikuquanshu.target.word-bigram.dim300                              | 20.77 MB   | 19529 |\n","| w2v.mixed-large.target.word-char.dim300                                | 1.35 GB    | 1292552 |\n","| w2v.mixed-large.target.word-word.dim300                                | 1.35 GB    | 1292483 |\n","| w2v.google_news.target.word-word.dim300.en                             | 1.61 GB    | 3000000 |\n","| glove.wiki2014-gigaword.target.word-word.dim50.en                      | 73.45 MB   | 400002 |\n","| glove.wiki2014-gigaword.target.word-word.dim100.en                     | 143.30 MB  | 400002 |\n","| glove.wiki2014-gigaword.target.word-word.dim200.en                     | 282.97 MB  | 400002 |\n","| glove.wiki2014-gigaword.target.word-word.dim300.en                     | 422.83 MB  | 400002 |\n","| glove.twitter.target.word-word.dim25.en                                | 116.92 MB  | 1193516 |\n","| glove.twitter.target.word-word.dim50.en                                | 221.64 MB  | 1193516 |\n","| glove.twitter.target.word-word.dim100.en                               | 431.08 MB  | 1193516 |\n","| glove.twitter.target.word-word.dim200.en                               | 848.56 MB  | 1193516 |\n","| fasttext.wiki-news.target.word-word.dim300.en                          | 541.63 MB  | 999996 |\n","| fasttext.crawl.target.word-word.dim300.en                              | 1.19 GB    | 2000002 |"]},{"cell_type":"markdown","metadata":{"id":"AKKFat4fVj4p"},"source":["<a name=\"3\"></a>\n","# 3. References\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"mlBx-cgIb_-o"},"source":["- [PaddleNLP Embedding API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/embeddings.html)\n","- [paddlenlp.embeddings docs](https://paddlenlp.readthedocs.io/zh/latest/source/paddlenlp.embeddings.html)\n","- [paddlenlp.embeddings source code](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/embeddings)"]}]}