docs/tutorials/word_embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IZBRUaiBBEpa"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "YS3NA-i6nAFC"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7SN5USFEIIK3"
      },
      "source": [
        "# Word embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aojnnc7sXrab"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/text/tutorials/word_embeddings\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/word_embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/text/blob/master/docs/tutorials/word_embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/text/docs/tutorials/word_embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6mJg1g3apaz"
      },
      "source": [
        "This tutorial contains an introduction to word embeddings. You will train your own word embeddings using a simple Keras model for a sentiment classification task, and then visualize them in the [Embedding Projector](http://projector.tensorflow.org) (shown in the image below). \n",
        "\n",
        "\u003cimg src=\"images/embedding.jpg\" alt=\"Screenshot of the embedding projector\" width=\"400\"/\u003e\n",
        "\n",
        "## Representing text as numbers\n",
        "\n",
        "Machine learning models take vectors (arrays of numbers) as input. When working with text, the first thing you must do is come up with a strategy to convert strings to numbers (or to \"vectorize\" the text) before feeding it to the model. In this section, you will look at three strategies for doing so.\n",
        "\n",
        "### One-hot encodings\n",
        "\n",
        "As a first idea, you might \"one-hot\" encode each word in your vocabulary. Consider the sentence \"The cat sat on the mat\". The vocabulary (or unique words) in this sentence is (cat, mat, on, sat, the). To represent each word, you will create a zero vector with length equal to the vocabulary, then place a one in the index that corresponds to the word. This approach is shown in the following diagram.\n",
        "\n",
        "\u003cimg src=\"images/one-hot.png\" alt=\"Diagram of one-hot encodings\" width=\"400\" /\u003e\n",
        "\n",
        "To create a vector that contains the encoding of the sentence, you could then concatenate the one-hot vectors for each word.\n",
        "\n",
        "Key point: This approach is inefficient. A one-hot encoded vector is sparse (meaning, most indices are zero). Imagine you have 10,000 words in the vocabulary. To one-hot encode each word, you would create a vector where 99.99% of the elements are zero.\n",
        "\n",
        "### Encode each word with a unique number\n",
        "\n",
        "A second approach you might try is to encode each word using a unique number. Continuing the example above, you could assign 1 to \"cat\", 2 to \"mat\", and so on. You could then encode the sentence \"The cat sat on the mat\" as a dense vector like [5, 1, 4, 3, 5, 2]. This approach is efficient. Instead of a sparse vector, you now have a dense one (where all elements are full).\n",
        "\n",
        "There are two downsides to this approach, however:\n",
        "\n",
        "* The integer-encoding is arbitrary (it does not capture any relationship between words).\n",
        "\n",
        "* An integer-encoding can be challenging for a model to interpret. A linear classifier, for example, learns a single weight for each feature. Because there is no relationship between the similarity of any two words and the similarity of their encodings, this feature-weight combination is not meaningful.\n",
        "\n",
        "### Word embeddings\n",
        "\n",
        "Word embeddings give us a way to use an efficient, dense representation in which similar words have a similar encoding. Importantly, you do not have to specify this encoding by hand. An embedding is a dense vector of floating point values (the length of the vector is a parameter you specify). Instead of specifying the values for the embedding manually, they are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer). It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn.\n",
        "\n",
        "\u003cimg src=\"images/embedding2.png\" alt=\"Diagram of an embedding\" width=\"400\"/\u003e\n",
        "\n",
        "Above is a diagram for a word embedding. Each word is represented as a 4-dimensional vector of floating point values. Another way to think of an embedding is as \"lookup table\". After these weights have been learned, you can encode each word by looking up the dense vector it corresponds to in the table."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SZUQErGewZxE"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RutaI-Tpev3T"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "import os\n",
        "import re\n",
        "import shutil\n",
        "import string\n",
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D\n",
        "from tensorflow.keras.layers import TextVectorization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBFctV8-JZOc"
      },
      "source": [
        "### Download the IMDb Dataset\n",
        "You will use the [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/) through the tutorial. You will train a sentiment classifier model on this dataset and in the process learn embeddings from scratch. To read more about loading a dataset from scratch, see the [Loading text tutorial](https://www.tensorflow.org/tutorials/load_data/text).  \n",
        "\n",
        "Download the dataset using Keras file utility and take a look at the directories."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aPO4_UmfF0KH"
      },
      "outputs": [],
      "source": [
        "url = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n",
        "\n",
        "dataset = tf.keras.utils.get_file(\"aclImdb_v1.tar.gz\", url,\n",
        "                                  untar=True, cache_dir='.',\n",
        "                                  cache_subdir='')\n",
        "\n",
        "dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')\n",
        "os.listdir(dataset_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eY6yROZNKvbd"
      },
      "source": [
        "Take a look at the `train/` directory. It has `pos` and `neg` folders with movie reviews labelled as positive and negative respectively. You will use reviews from `pos` and `neg` folders to train a binary classification model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9-iOHJGN6SDu"
      },
      "outputs": [],
      "source": [
        "train_dir = os.path.join(dataset_dir, 'train')\n",
        "os.listdir(train_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9O59BdioK8jY"
      },
      "source": [
        "The `train` directory also has additional folders which should be removed before creating training dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1_Vfi9oWMSh-"
      },
      "outputs": [],
      "source": [
        "remove_dir = os.path.join(train_dir, 'unsup')\n",
        "shutil.rmtree(remove_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFoJjiEyJz9u"
      },
      "source": [
        "Next, create a `tf.data.Dataset` using `tf.keras.utils.text_dataset_from_directory`. You can read more about using this utility in this [text classification tutorial](https://www.tensorflow.org/tutorials/keras/text_classification). \n",
        "\n",
        "Use the `train` directory to create both train and validation datasets with a split of 20% for validation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ItYD3TLkCOP1"
      },
      "outputs": [],
      "source": [
        "batch_size = 1024\n",
        "seed = 123\n",
        "train_ds = tf.keras.utils.text_dataset_from_directory(\n",
        "    'aclImdb/train', batch_size=batch_size, validation_split=0.2,\n",
        "    subset='training', seed=seed)\n",
        "val_ds = tf.keras.utils.text_dataset_from_directory(\n",
        "    'aclImdb/train', batch_size=batch_size, validation_split=0.2,\n",
        "    subset='validation', seed=seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHa6cq0-Ym0g"
      },
      "source": [
        "Take a look at a few movie reviews and their labels `(1: positive, 0: negative)` from the train dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aTCbSkvkYmTT"
      },
      "outputs": [],
      "source": [
        "for text_batch, label_batch in train_ds.take(1):\n",
        "  for i in range(5):\n",
        "    print(label_batch[i].numpy(), text_batch.numpy()[i])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHV2pchDhzDn"
      },
      "source": [
        "### Configure the dataset for performance\n",
        "\n",
        "These are two important methods you should use when loading data to make sure that I/O does not become blocking.\n",
        "\n",
        "`.cache()` keeps data in memory after it's loaded off disk. This will ensure the dataset does not become a bottleneck while training your model. If your dataset is too large to fit into memory, you can also use this method to create a performant on-disk cache, which is more efficient to read than many small files.\n",
        "\n",
        "`.prefetch()` overlaps data preprocessing and model execution while training. \n",
        "\n",
        "You can learn more about both methods, as well as how to cache data to disk in the [data performance guide](https://www.tensorflow.org/guide/data_performance)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Oz6k1IW7h1TO"
      },
      "outputs": [],
      "source": [
        "AUTOTUNE = tf.data.AUTOTUNE\n",
        "\n",
        "train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)\n",
        "val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqBazMiVQkj1"
      },
      "source": [
        "## Using the Embedding layer\n",
        "\n",
        "Keras makes it easy to use word embeddings. Take a look at the [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding) layer.\n",
        "\n",
        "The Embedding layer can be understood as a lookup table that maps from integer indices (which stand for specific words) to dense vectors (their embeddings). The dimensionality (or width) of the embedding is a parameter you can experiment with to see what works well for your problem, much in the same way you would experiment with the number of neurons in a Dense layer.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-OjxLVrMvWUE"
      },
      "outputs": [],
      "source": [
        "# Embed a 1,000 word vocabulary into 5 dimensions.\n",
        "embedding_layer = tf.keras.layers.Embedding(1000, 5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dKKV1L2Rk7e"
      },
      "source": [
        "When you create an Embedding layer, the weights for the embedding are randomly initialized (just like any other layer). During training, they are gradually adjusted via backpropagation. Once trained, the learned word embeddings will roughly encode similarities between words (as they were learned for the specific problem your model is trained on).\n",
        "\n",
        "If you pass an integer to an embedding layer, the result replaces each integer with the vector from the embedding table:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0YUjPgP7w0PO"
      },
      "outputs": [],
      "source": [
        "result = embedding_layer(tf.constant([1, 2, 3]))\n",
        "result.numpy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O4PC4QzsxTGx"
      },
      "source": [
        "For text or sequence problems, the Embedding layer takes a 2D tensor of integers, of shape `(samples, sequence_length)`, where each entry is a sequence of integers. It can embed sequences of variable lengths. You could feed into the embedding layer above batches with shapes `(32, 10)` (batch of 32 sequences of length 10) or `(64, 15)` (batch of 64 sequences of length 15).\n",
        "\n",
        "The returned tensor has one more axis than the input, the embedding vectors are aligned along the new last axis. Pass it a `(2, 3)` input batch and the output is `(2, 3, N)`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vwSYepRjyRGy"
      },
      "outputs": [],
      "source": [
        "result = embedding_layer(tf.constant([[0, 1, 2], [3, 4, 5]]))\n",
        "result.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGQp2N92yOyB"
      },
      "source": [
        "When given a batch of sequences as input, an embedding layer returns a 3D floating point tensor, of shape `(samples, sequence_length, embedding_dimensionality)`. To convert from this sequence of variable length to a fixed representation there are a variety of standard approaches. You could use an RNN, Attention, or pooling layer before passing it to a Dense layer. This tutorial uses pooling because it's the simplest. The [Text Classification with an RNN](https://www.tensorflow.org/text/tutorials/text_classification_rnn) tutorial is a good next step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aGicgV5qT0wh"
      },
      "source": [
        "## Text preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6NZSqIIoU0Y"
      },
      "source": [
        "Next, define the dataset preprocessing steps required for your sentiment classification model. Initialize a TextVectorization layer with the desired parameters to vectorize movie reviews. You can learn more about using this layer in the [Text Classification](https://www.tensorflow.org/tutorials/keras/text_classification) tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2MlsXzo-ZlfK"
      },
      "outputs": [],
      "source": [
        "# Create a custom standardization function to strip HTML break tags '\u003cbr /\u003e'.\n",
        "def custom_standardization(input_data):\n",
        "  lowercase = tf.strings.lower(input_data)\n",
        "  stripped_html = tf.strings.regex_replace(lowercase, '\u003cbr /\u003e', ' ')\n",
        "  return tf.strings.regex_replace(stripped_html,\n",
        "                                  '[%s]' % re.escape(string.punctuation), '')\n",
        "\n",
        "\n",
        "# Vocabulary size and number of words in a sequence.\n",
        "vocab_size = 10000\n",
        "sequence_length = 100\n",
        "\n",
        "# Use the text vectorization layer to normalize, split, and map strings to\n",
        "# integers. Note that the layer uses the custom standardization defined above.\n",
        "# Set maximum_sequence length as all samples are not of the same length.\n",
        "vectorize_layer = TextVectorization(\n",
        "    standardize=custom_standardization,\n",
        "    max_tokens=vocab_size,\n",
        "    output_mode='int',\n",
        "    output_sequence_length=sequence_length)\n",
        "\n",
        "# Make a text-only dataset (no labels) and call adapt to build the vocabulary.\n",
        "text_ds = train_ds.map(lambda x, y: x)\n",
        "vectorize_layer.adapt(text_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zI9_wLIiWO8Z"
      },
      "source": [
        "## Create a classification model\n",
        "\n",
        "Use the [Keras Sequential API](https://www.tensorflow.org/guide/keras/sequential_model) to define the sentiment classification model. In this case it is a \"Continuous bag of words\" style model.\n",
        "* The [`TextVectorization`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization) layer transforms strings into vocabulary indices. You have already initialized `vectorize_layer` as a TextVectorization layer and built its vocabulary by calling `adapt` on `text_ds`. Now vectorize_layer can be used as the first layer of your end-to-end classification model, feeding transformed strings into the Embedding layer.\n",
        "* The [`Embedding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding) layer takes the integer-encoded vocabulary and looks up the embedding vector for each word-index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are: `(batch, sequence, embedding)`.\n",
        "\n",
        "* The [`GlobalAveragePooling1D`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAveragePooling1D) layer returns a fixed-length output vector for each example by averaging over the sequence dimension. This allows the model to handle input of variable length, in the simplest way possible.\n",
        "\n",
        "* The fixed-length output vector is piped through a fully-connected ([`Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense)) layer with 16 hidden units.\n",
        "\n",
        "* The last layer is densely connected with a single output node. \n",
        "\n",
        "Caution: This model doesn't use masking, so the zero-padding is used as part of the input and hence the padding length may affect the output.  To fix this, see the [masking and padding guide](https://www.tensorflow.org/guide/keras/masking_and_padding)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pHLcFtn5Wsqj"
      },
      "outputs": [],
      "source": [
        "embedding_dim=16\n",
        "\n",
        "model = Sequential([\n",
        "  vectorize_layer,\n",
        "  Embedding(vocab_size, embedding_dim, name=\"embedding\"),\n",
        "  GlobalAveragePooling1D(),\n",
        "  Dense(16, activation='relu'),\n",
        "  Dense(1)\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjLNgKO7W2fe"
      },
      "source": [
        "## Compile and train the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpX9etB6IOQd"
      },
      "source": [
        "You will use [TensorBoard](https://www.tensorflow.org/tensorboard) to visualize metrics including loss and accuracy. Create a `tf.keras.callbacks.TensorBoard`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W4Hg3IHFt4Px"
      },
      "outputs": [],
      "source": [
        "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7OrKAKAKIbuH"
      },
      "source": [
        "Compile and train the model using the `Adam` optimizer and `BinaryCrossentropy` loss. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lCUgdP69Wzix"
      },
      "outputs": [],
      "source": [
        "model.compile(optimizer='adam',\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5mQehiQyv8rP"
      },
      "outputs": [],
      "source": [
        "model.fit(\n",
        "    train_ds,\n",
        "    validation_data=val_ds,\n",
        "    epochs=15,\n",
        "    callbacks=[tensorboard_callback])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wYnVedSPfmX"
      },
      "source": [
        "With this approach the model reaches a validation accuracy of around 78% (note that the model is overfitting since training accuracy is higher).\n",
        "\n",
        "Note: Your results may be a bit different, depending on how weights were randomly initialized before training the embedding layer. \n",
        "\n",
        "You can look into the model summary to learn more about each layer of the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mDCgjWyq_0dc"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiQbOJZ2WBFY"
      },
      "source": [
        "Visualize the model metrics in TensorBoard."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Uanp2YH8RzU"
      },
      "outputs": [],
      "source": [
        "#docs_infra: no_execute\n",
        "%load_ext tensorboard\n",
        "%tensorboard --logdir logs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvURkGVpXDOy"
      },
      "source": [
        "![embeddings_classifier_accuracy.png](images/embeddings_classifier_accuracy.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCoA6qwqP836"
      },
      "source": [
        "## Retrieve the trained word embeddings and save them to disk\n",
        "\n",
        "Next, retrieve the word embeddings learned during training. The embeddings are weights of the Embedding layer in the model. The weights matrix is of shape `(vocab_size, embedding_dimension)`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zp5rv01WG2YA"
      },
      "source": [
        "Obtain the weights from the model using `get_layer()` and `get_weights()`. The `get_vocabulary()` function provides the vocabulary to build a metadata file with one token per line. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Uamp1YH8RzU"
      },
      "outputs": [],
      "source": [
        "weights = model.get_layer('embedding').get_weights()[0]\n",
        "vocab = vectorize_layer.get_vocabulary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8MiCA77X8B8"
      },
      "source": [
        "Write the weights to disk. To use the [Embedding Projector](http://projector.tensorflow.org), you will upload two files in tab separated format: a file of vectors (containing the embedding), and a file of meta data (containing the words)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VLIahl9s53XT"
      },
      "outputs": [],
      "source": [
        "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
        "\n",
        "for index, word in enumerate(vocab):\n",
        "  if index == 0:\n",
        "    continue  # skip 0, it's padding.\n",
        "  vec = weights[index]\n",
        "  out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
        "  out_m.write(word + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JQyMZWyxYjMr"
      },
      "source": [
        "If you are running this tutorial in [Colaboratory](https://colab.research.google.com), you can use the following snippet to download these files to your local machine (or use the file browser, *View -\u003e Table of contents -\u003e File browser*)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lUsjQOKMIV2z"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  from google.colab import files\n",
        "  files.download('vectors.tsv')\n",
        "  files.download('metadata.tsv')\n",
        "except Exception:\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXLfFA54Yz-o"
      },
      "source": [
        "## Visualize the embeddings\n",
        "\n",
        "To visualize the embeddings, upload them to the embedding projector.\n",
        "\n",
        "Open the [Embedding Projector](http://projector.tensorflow.org/) (this can also run in a local TensorBoard instance).\n",
        "\n",
        "* Click on \"Load data\".\n",
        "\n",
        "* Upload the two files you created above: `vecs.tsv` and `meta.tsv`.\n",
        "\n",
        "The embeddings you have trained will now be displayed. You can search for words to find their closest neighbors. For example, try searching for \"beautiful\". You may see neighbors like \"wonderful\". \n",
        "\n",
        "Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. Try deleting the `Dense(16)` layer, retraining the model, and visualizing the embeddings again.\n",
        "\n",
        "Note: Typically, a much larger dataset is needed to train more interpretable word embeddings. This tutorial uses a small IMDb dataset for the purpose of demonstration.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvKiEHjramNh"
      },
      "source": [
        "## Next Steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSgAZpwF5xF_"
      },
      "source": [
        "This tutorial has shown you how to train and visualize word embeddings from scratch on a small dataset.\n",
        "\n",
        "* To train word embeddings using Word2Vec algorithm, try the [Word2Vec](https://www.tensorflow.org/tutorials/text/word2vec) tutorial. \n",
        "\n",
        "* To learn more about advanced text processing, read the [Transformer model for language understanding](https://www.tensorflow.org/text/tutorials/transformer)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "word_embeddings.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}