ML-Explainability/NLP/investigating_word_embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QJ4_kuYrcxZb"
      },
      "source": [
        "# Exploring Language Model's Embeddings\n",
        "\n",
        "<a href=\"https://colab.research.google.com/drive/1he7aM1o8b8TtCWRStXFHpXVO7LYbQ5Mm\" target=\"_blank\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\">\n",
        "</a>\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle).\n",
        "\n",
        "In language modeling, embeddings refer to dense vector representations of words or tokens in a continuous vector space. These embeddings are learned during training a language model, such as [Word2Vec](https://paperswithcode.com/method/skip-gram-word2vec), [GloVe](https://paperswithcode.com/method/glove), or in the case of neural language models like GPT and BERT, the embedding layer is treated as any other layer of a neural network, i.e., collections of learned parameters discovered via gradient descent and backpropagation. Embeddings are crucial in language modeling because they provide a way to represent words meaningfully and computationally efficiently, enabling the model to learn complex patterns and relationships within language data.\n",
        "\n",
        "Since embeddings are the foundation for various natural language processing tasks, it would be nice to explore their inner workings more.\n",
        "\n",
        "![word-embeddings](https://lena-voita.github.io/resources/lectures/word_emb/lookup_table.gif)\n",
        "\n",
        "[Source](https://lena-voita.github.io/nlp_course/word_embeddings.html).\n",
        "\n",
        "Again, you can think of an embedding layer as a lookup table. Imagine a large table with rows and columns, much like a spreadsheet. Each row represents a unique item, and each column represents some attribute or feature of that item. For example, if you're dealing with words, each row might represent a word, and each column could represent a linguistic feature. In this lookup table, each row is assigned a unique index. This index serves as the \"address\" to access the corresponding row. So, if you want to retrieve information about a specific word, you look up that word's index in the table, and it will lead you to the row containing the relevant information about that word.\n",
        "\n",
        "In a nutshell, this is an embedding layer. An embedding layer maps each word (or token) from a high-dimensional one-hot encoded space to a lower-dimensional dense vector space. Each word is represented by a unique vector in this space. And just like in the lookup table, each word is associated with a unique index.\n",
        "\n",
        "Below, we have a minimalistic example demonstrating this relationship between words (or tokens), their integer representations (their index on the embedding matrix), and their vector representation (the actual vector that represents that specific token)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uGLLJLIbm6YY",
        "outputId": "b19db3d8-d85f-448d-a057-618b74a02cb3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Word: 'banana', Integer representation: 1, Vector representation: [0.2 0.3 0.4]\n",
            "Word: 'pineapple', Integer representation: 4, Vector representation: [0.5 0.6 0.7]\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "\n",
        "# Define a small vocabulary.\n",
        "vocab = [\"apple\", \"banana\", \"orange\", \"grape\", \"pineapple\"]\n",
        "\n",
        "# Define corresponding vectors for each word (for demonstration purposes).\n",
        "word_to_vec = {\n",
        "    \"apple\": np.array([0.1, 0.2, 0.3]), # Index 0\n",
        "    \"banana\": np.array([0.2, 0.3, 0.4]), # Index 1\n",
        "    \"orange\": np.array([0.3, 0.4, 0.5]), # Index 2 ...\n",
        "    \"grape\": np.array([0.4, 0.5, 0.6]),\n",
        "    \"pineapple\": np.array([0.5, 0.6, 0.7])\n",
        "}\n",
        "\n",
        "# Function to retrieve vector representation of a word from the vocabulary\n",
        "def get_vector(word):\n",
        "    return word_to_vec.get(word, np.zeros(3))  # Return zero vector if word not found\n",
        "\n",
        "# Test the vocabulary and lookup table\n",
        "word = \"banana\"\n",
        "print(f\"Word: '{word}', Integer representation: {list(word_to_vec.keys()).index(word)}, Vector representation: {get_vector(word)}\")\n",
        "\n",
        "word = \"pineapple\"  # Word not in vocabulary\n",
        "print(f\"Word: '{word}', Integer representation: {list(word_to_vec.keys()).index(word)}, Vector representation: {get_vector(word)}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBBbyaT7l8G5"
      },
      "source": [
        "However, instead of handcrafting these vectors, let a neural network find them for us. In this notebook, just as [done before](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Explainability/NLP/model_maker.ipynb), we will create a language model for sentiment analysis. However, we only want to train the embedding layer, so do not pay attention to the rest of the architecture (it is merely a husk). As our learning signal, we will use the same dataset we used to train our first language models ([`sentiment-analysis-dataset`](https://huggingface.co/datasets/AiresPucrs/sentiment-analysis)), available in English and Portuguese on the Hugging Face Hub! 🤗"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 387,
          "referenced_widgets": [
            "3a683fa31a49407ab3288abf350841d6",
            "37e73407aa344803b67601735e2f3347",
            "f8816357acc747f4877a2be83271f8ba",
            "84966d894109493fa9e48ac14bb5021a",
            "cf212eb006f1475aaf260f5c5713fe73",
            "ed12e960d58244e08523c9108068effe",
            "962e80e5b1ad4d9da37791592a67cb13",
            "cf99c56f37b84cff94ea2b94134ae465",
            "1937b30314d7458dbdfa0febd814a650",
            "6b6b1ed4c1174d5a9edabc4f0cd6095e",
            "c09609c8c1b647fdb4af5119d6ace4d2",
            "4afa55d091a84d9dacab7f83925ec504",
            "005ae0cbe84f436fb0bbd7b8b79301d3",
            "ef8af826e5054af09cf7e3085bc4b6fe",
            "a2610366d6d040c0a0b189a72822f28a",
            "8a4c4211ce974f89b0ef49089057da5c",
            "e3b83bf785c841079b0eb8da03d2280a",
            "f85debf1d10e439492e3c9a2ab63ab25",
            "7699d5f31503460f9b23866342a04c90",
            "52c85bdd458e4be3a1ce7ca19f8c167a",
            "58c29a56ecec4552833d012eceb91869",
            "e99fb7502fe947b5a53781d09251afd6",
            "9aa1604cba644cc2bd67f3d85e6795b5",
            "e744ba11e2074ebc9997edea58a6ca73",
            "c5126ed32897487b9153c5b7e3b7337e",
            "f57b3512068b4be7bcb85e3f280a2150",
            "e6970614afd04dc39b89490b4435d8d7",
            "c0d7d534cfeb4244bbd692758dee1937",
            "8289e55cd9024772b455cd284734f401",
            "a8f393d5e45f4c6e8d2ee8e0d7f49dbb",
            "3eb89ad6e7164435b45d62bcfcc26bf8",
            "a78b63316ef04a8eab2f5728bedd3c7a",
            "87033f75d6ce4862bda300407514397f"
          ]
        },
        "id": "3NIsiFd8ghIl",
        "outputId": "ac92107b-ca9d-4be2-9d41-9e21d37e0a22"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m510.5/510.5 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3a683fa31a49407ab3288abf350841d6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/912 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4afa55d091a84d9dacab7f83925ec504",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/44.5M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9aa1604cba644cc2bd67f3d85e6795b5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split:   0%|          | 0/85089 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['text', 'label'],\n",
              "    num_rows: 85089\n",
              "})"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "!pip install datasets -q\n",
        "\n",
        "from datasets import load_dataset\n",
        "\n",
        "dataset = load_dataset('AiresPucrs/sentiment-analysis', split = 'train')\n",
        "\n",
        "display(dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PQIbRXpArowJ"
      },
      "source": [
        "Now, before training our embeddings, we need to extract a vocabulary of our corpus. And we will do that by using the [`TextVectorization`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization) layer from the Keras API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5o4820HIrn7h"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "vocab_size = 10000 # Size of the vocabulary\n",
        "sequence_length = 100 # Maximum sequence length to consider\n",
        "\n",
        "# Create and instance of the `TextVectorization` layer\n",
        "vectorization_layer = tf.keras.layers.TextVectorization(\n",
        "    max_tokens=vocab_size,\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=sequence_length\n",
        "    )\n",
        "\n",
        "# Fit the `TextVectorization` layer (which is basically a tokenizer) to the text\n",
        "vectorization_layer.adapt(dataset['text'])\n",
        "\n",
        "# Get the vocabulary out of the `TextVectorization` layer\n",
        "embedding_vocabulary = vectorization_layer.get_vocabulary()\n",
        "\n",
        "# Save the vocabulary for further use\n",
        "with open(r'embedding-vocabulary.txt', 'w', encoding='utf-8') as fp:\n",
        "    for word in embedding_vocabulary:\n",
        "        fp.write(\"%s\\n\" % word)\n",
        "    fp.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TRaK151LcxZg"
      },
      "source": [
        "Now that we have a dataset and a tokenizer that can translate words to integers, we can begin training our model. To create an ML split, we will use the [`train_test_split`](https://huggingface.co/docs/datasets/v2.18.0/en/package_reference/main_classes#datasets.Dataset.train_test_split) method to break down our corpus into training and test sets.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N4i7frifuJHH",
        "outputId": "cfadf0bc-7752-4bfe-e301-3ca1d71ba872"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Inputs:  (76580, 100)\n",
            "Validation Inputs:  (8509, 100)\n"
          ]
        }
      ],
      "source": [
        "# Split the dataset\n",
        "dataset = dataset.train_test_split(test_size=0.1)\n",
        "\n",
        "# Vectorize the text samples and turn the labels into floats.\n",
        "x_train = vectorization_layer(dataset['train']['text'])\n",
        "y_train = np.array(dataset['train']['label']).astype(float)\n",
        "x_val = vectorization_layer(dataset['test']['text'])\n",
        "y_val = np.array(dataset['test']['label']).astype(float)\n",
        "\n",
        "print('Training Inputs: ', x_train.shape)\n",
        "print('Validation Inputs: ', x_val.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zFJ37ks-cxZg"
      },
      "source": [
        "For the purposes of this notebook, we will create a simple model with a 16-embedding dimension. Thismeans that every word will be assosiated with a vector of size 16. Since we have a cocabulary with 10000 words, we have an embedding layer (matrix) of shape (10000, 16)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 416
        },
        "id": "Q2PMsropcxZg",
        "outputId": "07a585f2-d662-4332-f85a-7e80a4095b12"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"Embedding-16\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input (InputLayer)          [(None, None)]            0         \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, None, 16)          160000    \n",
            "                                                                 \n",
            " globa_average (GlobalAvera  (None, 16)                0         \n",
            " gePooling1D)                                                    \n",
            "                                                                 \n",
            " dense (Dense)               (None, 16)                272       \n",
            "                                                                 \n",
            " output (Dense)              (None, 1)                 17        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 160289 (626.13 KB)\n",
            "Trainable params: 160289 (626.13 KB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "None"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the Embedding matrix: (10000, 16)\n"
          ]
        }
      ],
      "source": [
        "# Dimensionality of the embedding layer\n",
        "embed_size = 16\n",
        "\n",
        "# Layers of this neural network are defined as funtions via the The Functional API\n",
        "inputs = tf.keras.Input(shape=(None,), dtype=\"int32\", name='input')\n",
        "x = tf.keras.layers.Embedding(input_dim=vocab_size,\n",
        "                              output_dim=embed_size,\n",
        "                              input_length=sequence_length,\n",
        "                              name='embedding')(inputs)\n",
        "\n",
        "x = tf.keras.layers.GlobalAveragePooling1D(name='globa_average')(x)\n",
        "x = tf.keras.layers.Dense(embed_size, activation=\"relu\", name='dense')(x)\n",
        "\n",
        "# Output layers is a single unbouded neuron\n",
        "outputs = tf.keras.layers.Dense(1, name='output')(x)\n",
        "model = tf.keras.Model(inputs, outputs)\n",
        "\n",
        "model._name=\"Embedding-16\"\n",
        "\n",
        "model.compile(loss=tf.losses.BinaryCrossentropy(from_logits = True),\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "display(model.summary())\n",
        "\n",
        "print(f\"Shape of the Embedding matrix: {model.layers[1].weights[0].shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GUQSMUpFcxZh"
      },
      "source": [
        "Since our model only has a small number of units to form the latent dimension, it will quickly overfit. Using [`EarlyStopping`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping), even with a max of 20 epochs, should stop the training under 10 epochs if we set the patience to something like 5 or 6.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "miuDET-IcxZi"
      },
      "outputs": [],
      "source": [
        "callbacks = [\n",
        "    # Define how to save the model\n",
        "    tf.keras.callbacks.ModelCheckpoint(\"embedding-model-16.keras\",\n",
        "                                                save_best_only=True),\n",
        "    # Define when to stop the training\n",
        "    tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",\n",
        "                                                patience=5,\n",
        "                                                verbose=1,\n",
        "                                                mode=\"auto\",\n",
        "                                                baseline=None,\n",
        "                                                restore_best_weights=True)\n",
        "    ]\n",
        "\n",
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
        "\n",
        "model.fit(x_train,\n",
        "          y_train,\n",
        "          epochs=20,\n",
        "          validation_data=(x_val, y_val),\n",
        "          callbacks=callbacks,\n",
        "          verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BpdU5yohy3PP"
      },
      "source": [
        "Next cell downloads the already trained language model. You can skip it if you trained the model on the cell above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CEnRIllWy0FD"
      },
      "outputs": [],
      "source": [
        "!pip install huggingface_hub -q\n",
        "\n",
        "from huggingface_hub import hf_hub_download\n",
        "\n",
        "# Download the model\n",
        "hf_hub_download(repo_id=\"AiresPucrs/embedding-model-16\",\n",
        "                filename=\"embedding-model-16.keras\",\n",
        "                local_dir=\"./\",\n",
        "                repo_type=\"model\"\n",
        "                )\n",
        "\n",
        "# Download the embedding vocabulary txt file\n",
        "hf_hub_download(repo_id=\"AiresPucrs/embedding-model-16\",\n",
        "                filename=\"embedding-vocabulary.txt\",\n",
        "                local_dir=\"./\",\n",
        "                repo_type=\"model\"\n",
        "                )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8MrQiv80Apc"
      },
      "source": [
        "Now, let us retrieve our embedding layer. We will also use our vocabulary to create a lookup table (a dictionary) where keys will be the words in our vocabulary and the values of their vector representation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F5T-0qjK0LRZ",
        "outputId": "b5b5f31e-515b-4208-f27c-50f5c5ce1afb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Embeddings Dimensions:  (9999, 16)\n",
            "Vocabulary Size:  9999\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "# Load the model\n",
        "model = tf.keras.models.load_model('embedding-model-16.keras')\n",
        "\n",
        "# Load the vocabulary\n",
        "with open('embedding-vocabulary.txt', encoding='utf-8') as fp:\n",
        "    embedding_vocabulary = [line.strip() for line in fp]\n",
        "    fp.close()\n",
        "\n",
        "# Get the embedding matrix back\n",
        "embeddings = model.get_layer('embedding').get_weights()[0]\n",
        "\n",
        "# Create a lookup table for the words and their respective vector representations\n",
        "words_embeddings = {}\n",
        "\n",
        "# iterating through the elements of list\n",
        "for i, word in enumerate(embedding_vocabulary):\n",
        "    # here we skip the embedding/token 0 (\"\"), because is just the PAD token.\n",
        "    if i == 0:\n",
        "        continue\n",
        "    words_embeddings[word] = embeddings[i]\n",
        "\n",
        "print(\"Embeddings Dimensions: \", np.array(list(words_embeddings.values())).shape)\n",
        "print(\"Vocabulary Size: \", len(words_embeddings.keys()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsCXWcj2cxZk"
      },
      "source": [
        "Now, one way to explore the relationship of these embedding vectors is by how similar they are. **Cosine similarity** is a similarity measure between two non-zero vectors of an inner product space. It measures the cosine of the angle between the two vectors and returns a value between -1 and 1, where 1 means identical, 0 means orthogonal, and -1 means opposite.\n",
        "\n",
        "Here, we can use cosine similarity to compare word embeddings in natural language processing. By calculating the cosine similarity between two-word embeddings, we can measure how similar the two words are in meaning and context.\n",
        "\n",
        "The equation for cosine similarity is:\n",
        "\n",
        "$$\\text{Cosine Similarity}(u, v) = \\frac{u \\cdot v}{\\lVert u \\rVert \\lVert v \\rVert} = \\cos(\\theta)$$\n",
        "\n",
        "Where:\n",
        "\n",
        "- $u$ and $v$ are the two vectors being compared.\n",
        "- $\\cdot$ represents the dot product operation.\n",
        "- $\\lVert u \\rVert$ and $\\lVert v \\rVert$ are the magnitudes of the two vectors.\n",
        "- $\\theta$ is the angle between them.\n",
        "\n",
        "To calculate the cosine similarity between two-word embeddings, we plug the embeddings as the vectors $u$ and $v$ in the above equation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h0svHUs0cxZk",
        "outputId": "1ebf4d6d-0a37-4ffb-e193-308b33bf0d2d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cosine Similarity between 'wonderful' with 'horrible': -0.9910120368003845\n",
            "Cosine Similarity between 'good' with 'bad': -0.7805653810501099\n"
          ]
        }
      ],
      "source": [
        "from numpy.linalg import norm\n",
        "\n",
        "def cosine_similarity(word1, word2, dictionary):\n",
        "    \"\"\"\n",
        "    Computes the cosine similarity between a two given words.\n",
        "\n",
        "    Parameters:\n",
        "    2 strings : str\n",
        "        Two words to be compared.\n",
        "    dictionary : dict\n",
        "        A dictionary with words as keys and their corresponding embeddings as values.\n",
        "\n",
        "    Returns:\n",
        "    --------\n",
        "    The Cosine Similarity Score (float).\n",
        "    -----------\n",
        "    \"\"\"\n",
        "    return np.dot(dictionary[word1], dictionary[word2])/(norm(dictionary[word1])*norm(dictionary[word2]))\n",
        "\n",
        "cos = cosine_similarity(\"wonderful\", \"horrible\",words_embeddings)\n",
        "print(f\"\"\"Cosine Similarity between 'wonderful' with 'horrible': {cos}\"\"\")\n",
        "\n",
        "cos = cosine_similarity(\"good\", \"bad\",words_embeddings)\n",
        "print(f\"\"\"Cosine Similarity between 'good' with 'bad': {cos}\"\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32kBpn70cxZl"
      },
      "source": [
        "As we can see, the word embeddings for **\"wonderful\"** and **\"horrible\"**, just as the embeddings for **\"good\"** and **\"bad\"**, are almost opposite words (as they should be for this sentiment-focused embedding layer). Meanwhile, similar adjectives should show a high positive value (and they do!)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CVsB2bFpcxZl",
        "outputId": "981dae80-0647-483e-9019-14d97558128d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cosine Similarity between 'good' with 'beautiful': 0.8989092707633972\n"
          ]
        }
      ],
      "source": [
        "cos = cosine_similarity(\"good\", \"beautiful\",words_embeddings)\n",
        "print(f\"\"\"Cosine Similarity between 'good' with 'beautiful': {cos}\"\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qd2zzWBCcxZm"
      },
      "source": [
        "Now, let us create a function to get the most similar word embeddings according to the cosine similarity measure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 431
        },
        "id": "Q_cDTbFncxZm",
        "outputId": "ad130803-cc87-4072-9d42-a5e7b9477659"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cosine Similarity (most different word embeddings:)\n"
          ]
        },
        {
          "data": {
            "text/markdown": [
              "| Closest Match   |   Similarity Score |\n",
              "|:----------------|-------------------:|\n",
              "| devito          |          -0.996515 |\n",
              "| fears           |          -0.995742 |\n",
              "| funniest        |          -0.995613 |\n",
              "| unique          |          -0.995567 |\n",
              "| appreciate      |          -0.995236 |"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cosine Similarity (most similar word embeddings:)\n"
          ]
        },
        {
          "data": {
            "text/markdown": [
              "| Closest Match   |   Similarity Score |\n",
              "|:----------------|-------------------:|\n",
              "| stinker         |           0.997925 |\n",
              "| talentless      |           0.998165 |\n",
              "| unwatchable     |           0.998458 |\n",
              "| worst           |           0.998905 |\n",
              "| terrible        |           0.998943 |"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from numpy.linalg import norm\n",
        "from IPython.display import Markdown\n",
        "\n",
        "def compute_cosine_table(string, dictionary,\n",
        "                         vocabulary):\n",
        "    \"\"\"\n",
        "    Computes the cosine similarity between a given word and all other words in a dictionary.\n",
        "\n",
        "    Parameters:\n",
        "    -----------\n",
        "    string : str\n",
        "        The word to compare against.\n",
        "    dictionary : dict\n",
        "        A dictionary with words as keys and their corresponding embeddings as values.\n",
        "    vocabulary : list\n",
        "        A list of words in the dictionary.\n",
        "\n",
        "    Returns:\n",
        "    --------\n",
        "    A pandas DataFrame with the closest matches to the input word and their\n",
        "    corresponding similarity scores. The index of the DataFrame is set\n",
        "    to the closest matches.\n",
        "    \"\"\"\n",
        "\n",
        "    l = vocabulary.copy()\n",
        "    l.remove(string)\n",
        "\n",
        "    cos = []\n",
        "    for word in l[1::]:\n",
        "\n",
        "        cosine = np.dot(dictionary[string],\n",
        "                dictionary[word])/(norm(dictionary[string])*norm(dictionary[word]))\n",
        "        cos.append(cosine)\n",
        "\n",
        "    return pd.DataFrame({\"Closest Match\": l[1::],f\"Similarity Score\": cos})\\\n",
        "        .sort_values(f\"Similarity Score\", ascending=True)\\\n",
        "        .set_index('Closest Match')\n",
        "\n",
        "df = compute_cosine_table(\"horrible\",\n",
        "        words_embeddings,\n",
        "        embedding_vocabulary)\n",
        "\n",
        "print(\"Cosine Similarity (most different word embeddings:)\")\n",
        "display(Markdown(df.head(5).to_markdown()))\n",
        "\n",
        "print(\"Cosine Similarity (most similar word embeddings:)\")\n",
        "display(Markdown(df.tail(5).to_markdown()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ktv-fdmWcxZn"
      },
      "source": [
        "**\"Terrible\"** is the word embedding most similar to **\"horrible\"**, which makes sense, given that this model learned embeddings were parameterized by an optimizer trying to find parameters that could differentiate negative and positive comments.\n",
        "\n",
        "However, the reason why **\"devito\"** is as antagonistic to **\"horrible\"** as **\"funniest\"** is open to interpretation (e.g., maybe the IMDB portion of our dataset is biased toward liking Danny DeVito).\n",
        "\n",
        "Another way to explore these similarities is through visual representations. For this, we will create a 3D projection of our embedding space. This can allow us to visualize the embedding space and the proximity between word embeddings in a geometrical sense. Even though cosine similarity is a measure of (as the name says) similarity and not of distance, we can bring our 16-dimensional space to a 3D embedding space where distances can be calculated visually. We can achieve this by using [`t-SNE`](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html), something we used before in [this notebook](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Intro-Course/8_Fashion_MNIST.ipynb).\n",
        "\n",
        "Let us first turn our word embeddings and vocabulary into a data frame.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 383
        },
        "id": "gfwsXtc3cxZo",
        "outputId": "83f8c50a-5969-4544-f2af-6cb9641505d8"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"display(df\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"embedding_0\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.031511709094047546,\n          -0.009815451689064503,\n          -0.018663670867681503\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_1\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          0.04096127301454544,\n          -0.03848133608698845,\n          -0.019801408052444458\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_2\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.04626849666237831,\n          -0.03804704174399376,\n          -0.13476280868053436\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_3\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.0267439354211092,\n          0.009836188517510891,\n          0.057085536420345306\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_4\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.024981815367937088,\n          -0.0023566074669361115,\n          0.02488354779779911\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_5\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.02575201541185379,\n          1.605578654562123e-05,\n          0.0003837060939986259\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_6\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          0.015337735414505005,\n          -0.007962805218994617,\n          -0.026022881269454956\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_7\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.03261030465364456,\n          -0.010237554088234901,\n          0.06467614322900772\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_8\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          0.07152129709720612,\n          0.1428583860397339,\n          0.10481075942516327\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_9\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.0007117536151781678,\n          -0.06376317143440247,\n          -0.057229794561862946\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_10\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.293674498796463,\n          -0.3941189646720886,\n          -0.29177144169807434\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_11\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.026487717404961586,\n          -0.05874023586511612,\n          0.006383208092302084\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_12\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.07052382826805115,\n          -0.11145113408565521,\n          -0.028979184105992317\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_13\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          0.002184261567890644,\n          -0.04137515276670456,\n          0.004829245153814554\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_14\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -0.018232738599181175,\n          -0.03741811588406563,\n          -0.049015410244464874\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding_15\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          0.0740012675523758,\n          0.12404362112283707,\n          0.04295467585325241\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-57674df0-0c01-41d5-8acf-6f60a263a6cc\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>embedding_0</th>\n",
              "      <th>embedding_1</th>\n",
              "      <th>embedding_2</th>\n",
              "      <th>embedding_3</th>\n",
              "      <th>embedding_4</th>\n",
              "      <th>embedding_5</th>\n",
              "      <th>embedding_6</th>\n",
              "      <th>embedding_7</th>\n",
              "      <th>embedding_8</th>\n",
              "      <th>embedding_9</th>\n",
              "      <th>embedding_10</th>\n",
              "      <th>embedding_11</th>\n",
              "      <th>embedding_12</th>\n",
              "      <th>embedding_13</th>\n",
              "      <th>embedding_14</th>\n",
              "      <th>embedding_15</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>[UNK]</th>\n",
              "      <td>-0.010852</td>\n",
              "      <td>-0.016661</td>\n",
              "      <td>-0.101499</td>\n",
              "      <td>-0.035356</td>\n",
              "      <td>0.036173</td>\n",
              "      <td>-0.036556</td>\n",
              "      <td>0.034414</td>\n",
              "      <td>0.019626</td>\n",
              "      <td>0.111396</td>\n",
              "      <td>-0.062554</td>\n",
              "      <td>-0.268549</td>\n",
              "      <td>-0.037651</td>\n",
              "      <td>-0.110640</td>\n",
              "      <td>0.057772</td>\n",
              "      <td>0.019258</td>\n",
              "      <td>0.092910</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>the</th>\n",
              "      <td>-0.009815</td>\n",
              "      <td>-0.038481</td>\n",
              "      <td>-0.038047</td>\n",
              "      <td>0.009836</td>\n",
              "      <td>-0.002357</td>\n",
              "      <td>0.000016</td>\n",
              "      <td>-0.007963</td>\n",
              "      <td>-0.010238</td>\n",
              "      <td>0.142858</td>\n",
              "      <td>-0.063763</td>\n",
              "      <td>-0.394119</td>\n",
              "      <td>-0.058740</td>\n",
              "      <td>-0.111451</td>\n",
              "      <td>-0.041375</td>\n",
              "      <td>-0.037418</td>\n",
              "      <td>0.124044</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>and</th>\n",
              "      <td>-0.057627</td>\n",
              "      <td>-0.042199</td>\n",
              "      <td>-0.088547</td>\n",
              "      <td>-0.002199</td>\n",
              "      <td>-0.106559</td>\n",
              "      <td>-0.047287</td>\n",
              "      <td>-0.068991</td>\n",
              "      <td>0.004988</td>\n",
              "      <td>0.060553</td>\n",
              "      <td>-0.043085</td>\n",
              "      <td>-0.480912</td>\n",
              "      <td>-0.090603</td>\n",
              "      <td>-0.103486</td>\n",
              "      <td>0.019498</td>\n",
              "      <td>0.031873</td>\n",
              "      <td>0.182327</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>a</th>\n",
              "      <td>0.053509</td>\n",
              "      <td>0.013593</td>\n",
              "      <td>-0.087281</td>\n",
              "      <td>0.007861</td>\n",
              "      <td>0.022452</td>\n",
              "      <td>-0.090360</td>\n",
              "      <td>-0.029521</td>\n",
              "      <td>0.023927</td>\n",
              "      <td>0.089998</td>\n",
              "      <td>-0.066138</td>\n",
              "      <td>-0.382833</td>\n",
              "      <td>-0.065598</td>\n",
              "      <td>-0.052045</td>\n",
              "      <td>0.027191</td>\n",
              "      <td>-0.026402</td>\n",
              "      <td>0.079612</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>to</th>\n",
              "      <td>0.045605</td>\n",
              "      <td>-0.036270</td>\n",
              "      <td>-0.022603</td>\n",
              "      <td>-0.025769</td>\n",
              "      <td>-0.011813</td>\n",
              "      <td>-0.070337</td>\n",
              "      <td>-0.019752</td>\n",
              "      <td>-0.022252</td>\n",
              "      <td>0.101837</td>\n",
              "      <td>-0.021529</td>\n",
              "      <td>-0.382733</td>\n",
              "      <td>0.003982</td>\n",
              "      <td>-0.027381</td>\n",
              "      <td>0.017267</td>\n",
              "      <td>-0.026032</td>\n",
              "      <td>0.102821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>of</th>\n",
              "      <td>-0.018664</td>\n",
              "      <td>-0.019801</td>\n",
              "      <td>-0.134763</td>\n",
              "      <td>0.057086</td>\n",
              "      <td>0.024884</td>\n",
              "      <td>0.000384</td>\n",
              "      <td>-0.026023</td>\n",
              "      <td>0.064676</td>\n",
              "      <td>0.104811</td>\n",
              "      <td>-0.057230</td>\n",
              "      <td>-0.291771</td>\n",
              "      <td>0.006383</td>\n",
              "      <td>-0.028979</td>\n",
              "      <td>0.004829</td>\n",
              "      <td>-0.049015</td>\n",
              "      <td>0.042955</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>is</th>\n",
              "      <td>-0.002275</td>\n",
              "      <td>-0.007564</td>\n",
              "      <td>-0.092545</td>\n",
              "      <td>0.009008</td>\n",
              "      <td>-0.032594</td>\n",
              "      <td>0.023053</td>\n",
              "      <td>0.003390</td>\n",
              "      <td>0.006978</td>\n",
              "      <td>0.073580</td>\n",
              "      <td>-0.063714</td>\n",
              "      <td>-0.384378</td>\n",
              "      <td>-0.004860</td>\n",
              "      <td>-0.094561</td>\n",
              "      <td>-0.004869</td>\n",
              "      <td>-0.004495</td>\n",
              "      <td>0.087992</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>in</th>\n",
              "      <td>0.001678</td>\n",
              "      <td>-0.032698</td>\n",
              "      <td>-0.144504</td>\n",
              "      <td>-0.044736</td>\n",
              "      <td>-0.049127</td>\n",
              "      <td>0.012565</td>\n",
              "      <td>-0.032565</td>\n",
              "      <td>0.021187</td>\n",
              "      <td>0.057413</td>\n",
              "      <td>-0.060246</td>\n",
              "      <td>-0.321496</td>\n",
              "      <td>-0.036165</td>\n",
              "      <td>-0.048146</td>\n",
              "      <td>-0.011753</td>\n",
              "      <td>0.016123</td>\n",
              "      <td>0.036803</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>i</th>\n",
              "      <td>-0.031512</td>\n",
              "      <td>0.040961</td>\n",
              "      <td>-0.046268</td>\n",
              "      <td>-0.026744</td>\n",
              "      <td>-0.024982</td>\n",
              "      <td>-0.025752</td>\n",
              "      <td>0.015338</td>\n",
              "      <td>-0.032610</td>\n",
              "      <td>0.071521</td>\n",
              "      <td>-0.000712</td>\n",
              "      <td>-0.293674</td>\n",
              "      <td>-0.026488</td>\n",
              "      <td>-0.070524</td>\n",
              "      <td>0.002184</td>\n",
              "      <td>-0.018233</td>\n",
              "      <td>0.074001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>it</th>\n",
              "      <td>-0.023752</td>\n",
              "      <td>-0.023493</td>\n",
              "      <td>-0.016742</td>\n",
              "      <td>-0.050926</td>\n",
              "      <td>-0.067329</td>\n",
              "      <td>0.049322</td>\n",
              "      <td>-0.063472</td>\n",
              "      <td>0.095920</td>\n",
              "      <td>0.020589</td>\n",
              "      <td>-0.025841</td>\n",
              "      <td>-0.454201</td>\n",
              "      <td>-0.064458</td>\n",
              "      <td>-0.087016</td>\n",
              "      <td>-0.045967</td>\n",
              "      <td>-0.009504</td>\n",
              "      <td>0.136230</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-57674df0-0c01-41d5-8acf-6f60a263a6cc')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-57674df0-0c01-41d5-8acf-6f60a263a6cc button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-57674df0-0c01-41d5-8acf-6f60a263a6cc');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-63b8cc94-7aa1-4688-b295-19f6b0535423\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-63b8cc94-7aa1-4688-b295-19f6b0535423')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-63b8cc94-7aa1-4688-b295-19f6b0535423 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       embedding_0  embedding_1  embedding_2  embedding_3  embedding_4  \\\n",
              "[UNK]    -0.010852    -0.016661    -0.101499    -0.035356     0.036173   \n",
              "the      -0.009815    -0.038481    -0.038047     0.009836    -0.002357   \n",
              "and      -0.057627    -0.042199    -0.088547    -0.002199    -0.106559   \n",
              "a         0.053509     0.013593    -0.087281     0.007861     0.022452   \n",
              "to        0.045605    -0.036270    -0.022603    -0.025769    -0.011813   \n",
              "of       -0.018664    -0.019801    -0.134763     0.057086     0.024884   \n",
              "is       -0.002275    -0.007564    -0.092545     0.009008    -0.032594   \n",
              "in        0.001678    -0.032698    -0.144504    -0.044736    -0.049127   \n",
              "i        -0.031512     0.040961    -0.046268    -0.026744    -0.024982   \n",
              "it       -0.023752    -0.023493    -0.016742    -0.050926    -0.067329   \n",
              "\n",
              "       embedding_5  embedding_6  embedding_7  embedding_8  embedding_9  \\\n",
              "[UNK]    -0.036556     0.034414     0.019626     0.111396    -0.062554   \n",
              "the       0.000016    -0.007963    -0.010238     0.142858    -0.063763   \n",
              "and      -0.047287    -0.068991     0.004988     0.060553    -0.043085   \n",
              "a        -0.090360    -0.029521     0.023927     0.089998    -0.066138   \n",
              "to       -0.070337    -0.019752    -0.022252     0.101837    -0.021529   \n",
              "of        0.000384    -0.026023     0.064676     0.104811    -0.057230   \n",
              "is        0.023053     0.003390     0.006978     0.073580    -0.063714   \n",
              "in        0.012565    -0.032565     0.021187     0.057413    -0.060246   \n",
              "i        -0.025752     0.015338    -0.032610     0.071521    -0.000712   \n",
              "it        0.049322    -0.063472     0.095920     0.020589    -0.025841   \n",
              "\n",
              "       embedding_10  embedding_11  embedding_12  embedding_13  embedding_14  \\\n",
              "[UNK]     -0.268549     -0.037651     -0.110640      0.057772      0.019258   \n",
              "the       -0.394119     -0.058740     -0.111451     -0.041375     -0.037418   \n",
              "and       -0.480912     -0.090603     -0.103486      0.019498      0.031873   \n",
              "a         -0.382833     -0.065598     -0.052045      0.027191     -0.026402   \n",
              "to        -0.382733      0.003982     -0.027381      0.017267     -0.026032   \n",
              "of        -0.291771      0.006383     -0.028979      0.004829     -0.049015   \n",
              "is        -0.384378     -0.004860     -0.094561     -0.004869     -0.004495   \n",
              "in        -0.321496     -0.036165     -0.048146     -0.011753      0.016123   \n",
              "i         -0.293674     -0.026488     -0.070524      0.002184     -0.018233   \n",
              "it        -0.454201     -0.064458     -0.087016     -0.045967     -0.009504   \n",
              "\n",
              "       embedding_15  \n",
              "[UNK]      0.092910  \n",
              "the        0.124044  \n",
              "and        0.182327  \n",
              "a          0.079612  \n",
              "to         0.102821  \n",
              "of         0.042955  \n",
              "is         0.087992  \n",
              "in         0.036803  \n",
              "i          0.074001  \n",
              "it         0.136230  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# we are starting from index position 1 because the first element is the padding token.\n",
        "df = pd.DataFrame(np.array(list(words_embeddings.values())),\n",
        "                  columns=[f'embedding_{i}' for i in range(embeddings.shape[1])],\n",
        "                  index=list(words_embeddings.keys()))\n",
        "\n",
        "display(df.head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xYh7nvfGcxZo"
      },
      "source": [
        "Now, let us implement **t-SNE** for dimensionality reduction. But before doing that, we need to keep some things in mind.\n",
        "\n",
        "T-SNE ([t-distributed Stochastic Neighbor Embedding](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding)) is optimized using gradient descent, being also sensitive to the choice of hyperparameters. For example, the choice of perplexity and number of iterations in `t-SNE` depends on the specific dataset and the goals of the analysis. However, here are some general guidelines that can help you choose appropriate values:\n",
        "\n",
        "- **Perplexity** controls the balance between local and global aspects of the data. A perplexity value of 5 to 50 is often used for small to medium datasets. A higher perplexity value may be needed for larger datasets to capture global structure. However, a perplexity value that is too high may result in losing local structure. Generally, a perplexity value of 50 to 100 is often used for larger datasets.\n",
        "\n",
        "- The number of **iterations** determines the amount of computation and time needed to optimize the t-SNE algorithm. Several iterations between 1000 and 5000 are often sufficient for small to medium datasets to obtain a stable embedding. A higher number of iterations may be needed for larger datasets to obtain a stable embedding. However, many iterations may result in overfitting, where the embedding captures noise in the data instead of the underlying structure.\n",
        "\n",
        "> Note: To learn how to \"[Use t-SNE Effectively](https://distill.pub/2016/misread-tsne/)\", we recommend this publication.\n",
        "\n",
        "Get ready for this computation, which might take a while..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ELlNlM_Q_Icb",
        "outputId": "67e7c929-6dda-40a6-87ad-3a1f2770eb14"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] Computing 121 nearest neighbors...\n",
            "[t-SNE] Indexed 9999 samples in 0.001s...\n",
            "[t-SNE] Computed neighbors for 9999 samples in 0.802s...\n",
            "[t-SNE] Computed conditional probabilities for sample 1000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 2000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 3000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 4000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 5000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 6000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 7000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 8000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 9000 / 9999\n",
            "[t-SNE] Computed conditional probabilities for sample 9999 / 9999\n",
            "[t-SNE] Mean sigma: 0.047531\n",
            "[t-SNE] KL divergence after 250 iterations with early exaggeration: 78.496521\n",
            "[t-SNE] KL divergence after 1750 iterations: 2.128672\n"
          ]
        }
      ],
      "source": [
        "from sklearn.manifold import TSNE\n",
        "\n",
        "tsne = TSNE(n_components=3, verbose=1, perplexity=40, n_iter=2000)\n",
        "tsne_results = tsne.fit_transform(df.values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSK_hUt2_Lo4"
      },
      "source": [
        "You can use the cell below to plot these results as a 3D scatter plot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zYkrUdYRcxZp"
      },
      "outputs": [],
      "source": [
        "import plotly.express as px\n",
        "\n",
        "fig = px.scatter_3d(\n",
        "    tsne_results, x=0, y=1, z=2, color=df.index,\n",
        "    labels={'0': 't-SNE 1', '1': 't-SNE 2', '2': 't-SNE 3'}\n",
        ")\n",
        "\n",
        "fig.update_layout(\n",
        "    template='ggplot2',\n",
        "    title=f'Word Embeddings in 3D'\n",
        "                  )\n",
        "\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6eIu19c1cxZp"
      },
      "source": [
        "To measure the distance between two vectors, you can use the [`numpy.linalg.norm`](https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html) function, which calculates the Euclidean distance between two vectors. This will output the distance between the vectors $a$ and $b$.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IRRTJQ76cxZq",
        "outputId": "83a72c1e-f7f2-4549-c9a3-4f579f907bfc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Distance from \"good\" to \"bad\": 98.20780181884766\n",
            "Distance from \"wonderfully\" to \"horrible\": 117.50151062011719\n",
            "Distance from \"terrible\" to \"horrible\": 4.233308792114258\n",
            "Distance from \"terrible\" to \"bad\": 29.911943435668945\n"
          ]
        }
      ],
      "source": [
        "from numpy.linalg import norm\n",
        "\n",
        "tsne_df = pd.DataFrame(tsne_results, index=df.index)\n",
        "\n",
        "def calculate_embedding_distance(word1, word2, df):\n",
        "    \"\"\"\n",
        "    Computes the Euclidean distance between two given vectors.\n",
        "\n",
        "    Parameters:\n",
        "    2 strings : str\n",
        "        Two words to be compared.\n",
        "    DataFrame : pandas.DataFrame\n",
        "        A pandas.DataFrame where the index are words and the\n",
        "        columns are the unitary vector values.\n",
        "    Returns:\n",
        "    --------\n",
        "    The Euclidean distance (float).\n",
        "    -----------\n",
        "    \"\"\"\n",
        "    return norm(df.loc[word1].values - df.loc[word2].values)\n",
        "\n",
        "distance = calculate_embedding_distance(\"good\", \"bad\", tsne_df)\n",
        "print(f\"\"\"Distance from \"good\" to \"bad\": {distance}\"\"\")\n",
        "\n",
        "distance = calculate_embedding_distance(\"wonderfully\", \"horrible\", tsne_df)\n",
        "print(f\"\"\"Distance from \"wonderfully\" to \"horrible\": {distance}\"\"\")\n",
        "\n",
        "distance = calculate_embedding_distance(\"terrible\", \"horrible\", tsne_df)\n",
        "print(f\"\"\"Distance from \"terrible\" to \"horrible\": {distance}\"\"\")\n",
        "\n",
        "distance = calculate_embedding_distance(\"terrible\", \"bad\", tsne_df)\n",
        "print(f\"\"\"Distance from \"terrible\" to \"bad\": {distance}\"\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gpB1AbnrcxZq"
      },
      "source": [
        "Putting together everything we learned thus far, we can start creating similar sub-clusters. For example, we can use cosine similarity to compute what words are more similar to \"terrible\".\n",
        "\n",
        "Let us create a list with the 30 most similar (and different) word embeddings of the \"terrible\" word embedding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2lOfwEKVcxZr"
      },
      "outputs": [],
      "source": [
        "df_cosine_similarity = compute_cosine_table(\"terrible\", words_embeddings, embedding_vocabulary)\n",
        "\n",
        "word_list = []\n",
        "\n",
        "for word in list(df_cosine_similarity.head(30).index):\n",
        "    word_list.append(word)\n",
        "\n",
        "for word in list(df_cosine_similarity.tail(30).index):\n",
        "    word_list.append(word)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsiS0GVFcxZr"
      },
      "source": [
        "Now, we will select only the 3D projections of these words (a.k.a. the \"terrible\" similarity cluster)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 708
        },
        "id": "nonAhpetI4Bo",
        "outputId": "1661e901-ab2c-4c8c-8121-a1174a26d2f2"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"display(cluster\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"X\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2.019906974525944,\n        \"min\": -76.8989486694336,\n        \"max\": -70.6644515991211,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -75.21376037597656,\n          -76.8989486694336,\n          -73.32228088378906\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Y\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 4.421251789412615,\n        \"min\": -5.869705677032471,\n        \"max\": 6.471135139465332,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          3.367316484451294,\n          -5.869705677032471,\n          1.3817412853240967\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Z\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9.19207486573181,\n        \"min\": -7.617326736450195,\n        \"max\": 13.592978477478027,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -6.043014049530029,\n          4.459920883178711,\n          -6.733865737915039\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1430718d-dcd9-4149-aa76-84db5c8460df\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>Y</th>\n",
              "      <th>Z</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>thank</th>\n",
              "      <td>-70.664452</td>\n",
              "      <td>-2.004013</td>\n",
              "      <td>13.265845</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>devito</th>\n",
              "      <td>-76.898949</td>\n",
              "      <td>-5.869706</td>\n",
              "      <td>4.459921</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>funniest</th>\n",
              "      <td>-76.293297</td>\n",
              "      <td>-4.913204</td>\n",
              "      <td>13.592978</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>fears</th>\n",
              "      <td>-76.167618</td>\n",
              "      <td>6.471135</td>\n",
              "      <td>-6.288749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>superbly</th>\n",
              "      <td>-75.346542</td>\n",
              "      <td>-4.158202</td>\n",
              "      <td>6.171668</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unforgettable</th>\n",
              "      <td>-73.322281</td>\n",
              "      <td>1.381741</td>\n",
              "      <td>-6.733866</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>fortunately</th>\n",
              "      <td>-73.753799</td>\n",
              "      <td>5.193653</td>\n",
              "      <td>-7.617327</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>thanks</th>\n",
              "      <td>-72.030792</td>\n",
              "      <td>-1.305513</td>\n",
              "      <td>13.535428</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>-75.213760</td>\n",
              "      <td>3.367316</td>\n",
              "      <td>-6.043014</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>appreciate</th>\n",
              "      <td>-75.496796</td>\n",
              "      <td>3.537618</td>\n",
              "      <td>-4.359625</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1430718d-dcd9-4149-aa76-84db5c8460df')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1430718d-dcd9-4149-aa76-84db5c8460df button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1430718d-dcd9-4149-aa76-84db5c8460df');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c34df07c-fdbe-4367-a3be-7b67d8a519e6\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c34df07c-fdbe-4367-a3be-7b67d8a519e6')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c34df07c-fdbe-4367-a3be-7b67d8a519e6 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                       X         Y          Z\n",
              "thank         -70.664452 -2.004013  13.265845\n",
              "devito        -76.898949 -5.869706   4.459921\n",
              "funniest      -76.293297 -4.913204  13.592978\n",
              "fears         -76.167618  6.471135  -6.288749\n",
              "superbly      -75.346542 -4.158202   6.171668\n",
              "unforgettable -73.322281  1.381741  -6.733866\n",
              "fortunately   -73.753799  5.193653  -7.617327\n",
              "thanks        -72.030792 -1.305513  13.535428\n",
              "unique        -75.213760  3.367316  -6.043014\n",
              "appreciate    -75.496796  3.537618  -4.359625"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"display(cluster\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"X\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0448117420409018,\n        \"min\": 31.373411178588867,\n        \"max\": 34.35503387451172,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          31.988035202026367,\n          31.51063346862793,\n          32.371578216552734\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Y\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2.1285900751739333,\n        \"min\": 37.856327056884766,\n        \"max\": 44.451507568359375,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          42.31867980957031,\n          41.974788665771484,\n          42.836238861083984\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Z\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 7.354145215337991,\n        \"min\": -11.087099075317383,\n        \"max\": 10.894530296325684,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          -9.424381256103516,\n          10.894530296325684,\n          -5.633058547973633\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-20bfb6af-41d5-4214-b639-c3eeaead3e04\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>Y</th>\n",
              "      <th>Z</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>uninteresting</th>\n",
              "      <td>32.075939</td>\n",
              "      <td>44.290012</td>\n",
              "      <td>-4.929207</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>insulting</th>\n",
              "      <td>31.510633</td>\n",
              "      <td>41.974789</td>\n",
              "      <td>10.894530</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unwatchable</th>\n",
              "      <td>32.308258</td>\n",
              "      <td>44.451508</td>\n",
              "      <td>-6.943865</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>trite</th>\n",
              "      <td>31.373411</td>\n",
              "      <td>41.739697</td>\n",
              "      <td>6.890880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uninspired</th>\n",
              "      <td>33.901012</td>\n",
              "      <td>43.864555</td>\n",
              "      <td>-5.942081</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>stinker</th>\n",
              "      <td>32.371578</td>\n",
              "      <td>42.836239</td>\n",
              "      <td>-5.633059</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>waste</th>\n",
              "      <td>33.274792</td>\n",
              "      <td>40.719261</td>\n",
              "      <td>-11.087099</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>worst</th>\n",
              "      <td>34.355034</td>\n",
              "      <td>37.856327</td>\n",
              "      <td>-8.249153</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>horrible</th>\n",
              "      <td>31.988035</td>\n",
              "      <td>42.318680</td>\n",
              "      <td>-9.424381</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>awful</th>\n",
              "      <td>33.691402</td>\n",
              "      <td>39.472031</td>\n",
              "      <td>-9.870434</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-20bfb6af-41d5-4214-b639-c3eeaead3e04')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-20bfb6af-41d5-4214-b639-c3eeaead3e04 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-20bfb6af-41d5-4214-b639-c3eeaead3e04');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-500120be-0615-4f6d-899c-09fa18b3ccc4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-500120be-0615-4f6d-899c-09fa18b3ccc4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-500120be-0615-4f6d-899c-09fa18b3ccc4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                       X          Y          Z\n",
              "uninteresting  32.075939  44.290012  -4.929207\n",
              "insulting      31.510633  41.974789  10.894530\n",
              "unwatchable    32.308258  44.451508  -6.943865\n",
              "trite          31.373411  41.739697   6.890880\n",
              "uninspired     33.901012  43.864555  -5.942081\n",
              "stinker        32.371578  42.836239  -5.633059\n",
              "waste          33.274792  40.719261 -11.087099\n",
              "worst          34.355034  37.856327  -8.249153\n",
              "horrible       31.988035  42.318680  -9.424381\n",
              "awful          33.691402  39.472031  -9.870434"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def select_word_projections(word_list, df):\n",
        "    \"\"\"\n",
        "    Select only the vectors associated with a specific word in a\n",
        "    pandas.DataFrame.\n",
        "\n",
        "    Parameters:\n",
        "    list : list\n",
        "        A list of words to be selected.\n",
        "    DataFrame : pandas.DataFrame\n",
        "        A pandas.DataFrame where the index are words and the\n",
        "        columns are the unitary vector values.\n",
        "    Returns:\n",
        "    --------\n",
        "    A pandas.DataFrame with all the \"X\", \"Y\", \"Z\"\n",
        "    components of the selected words.\n",
        "    -----------\n",
        "    \"\"\"\n",
        "    tsne_df = df.copy().reset_index()\n",
        "    arr = np.array([0, 0, 0])\n",
        "\n",
        "    for word in word_list:\n",
        "        index = tsne_df.loc[tsne_df['index'] == word].index[0]\n",
        "        arr = np.vstack((arr, tsne_df.drop('index', axis=1).values[index, :]))\n",
        "\n",
        "    arr = np.delete(arr, (0), axis=0)\n",
        "\n",
        "    result_df = pd.DataFrame(arr, index=word_list, columns=['X', 'Y', 'Z'])\n",
        "\n",
        "    return result_df\n",
        "\n",
        "cluster = select_word_projections(word_list, tsne_df)\n",
        "\n",
        "display(cluster.head(10))\n",
        "display(cluster.tail(10))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bx4gGjvScxZs"
      },
      "source": [
        "Now, let us visualize these word embedding projections in 3D space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "mNq4R7clcxZs",
        "outputId": "f11a9e75-df7d-4f55-ecdb-e512e780a5e8"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script charset=\"utf-8\" src=\"https://cdn.plot.ly/plotly-2.24.1.min.js\"></script>                <div id=\"7b1393e2-2193-4c52-bef7-39876a875ff1\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"7b1393e2-2193-4c52-bef7-39876a875ff1\")) {                    Plotly.newPlot(                        \"7b1393e2-2193-4c52-bef7-39876a875ff1\",                        [{\"hovertemplate\":\"index=thank\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"thank\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"thank\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-70.6644515991211],\"y\":[-2.0040132999420166],\"z\":[13.26584529876709],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=devito\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"devito\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"devito\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.8989486694336],\"y\":[-5.869705677032471],\"z\":[4.459920883178711],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=funniest\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"funniest\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"funniest\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.29329681396484],\"y\":[-4.913204193115234],\"z\":[13.592978477478027],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=fears\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"fears\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"fears\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.16761779785156],\"y\":[6.471135139465332],\"z\":[-6.288748741149902],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=superbly\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"superbly\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"superbly\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.34654235839844],\"y\":[-4.158202171325684],\"z\":[6.171667575836182],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unforgettable\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unforgettable\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unforgettable\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-73.32228088378906],\"y\":[1.3817412853240967],\"z\":[-6.733865737915039],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=fortunately\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"fortunately\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"fortunately\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-73.75379943847656],\"y\":[5.193652629852295],\"z\":[-7.617326736450195],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=thanks\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"thanks\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"thanks\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-72.03079223632812],\"y\":[-1.3055126667022705],\"z\":[13.535428047180176],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unique\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unique\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unique\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.21376037597656],\"y\":[3.367316484451294],\"z\":[-6.043014049530029],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=appreciate\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"appreciate\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"appreciate\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.49679565429688],\"y\":[3.5376181602478027],\"z\":[-4.359625339508057],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=superb\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"superb\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"superb\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.52119445800781],\"y\":[-3.969752311706543],\"z\":[14.784163475036621],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=pleasantly\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"pleasantly\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"pleasantly\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.81461334228516],\"y\":[-6.602167129516602],\"z\":[12.162758827209473],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=wax\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"wax\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"wax\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.18111419677734],\"y\":[3.663055896759033],\"z\":[-7.275920391082764],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=parker\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"parker\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"parker\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-67.21411895751953],\"y\":[11.457247734069824],\"z\":[-10.373174667358398],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=thx\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"thx\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"thx\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-73.03471374511719],\"y\":[1.4285554885864258],\"z\":[-5.159421443939209],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=improved\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"improved\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"improved\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-73.99702453613281],\"y\":[-4.069666862487793],\"z\":[9.08714485168457],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=underrated\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"underrated\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"underrated\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.9238052368164],\"y\":[-7.363073825836182],\"z\":[12.293558120727539],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=shared\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"shared\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"shared\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-71.12113189697266],\"y\":[4.378503799438477],\"z\":[-7.914693355560303],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=louis\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"louis\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"louis\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-71.22212982177734],\"y\":[0.8131169676780701],\"z\":[-4.645025730133057],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=edie\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"edie\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"edie\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.67280578613281],\"y\":[0.5257472991943359],\"z\":[-7.367892742156982],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=scoop\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"scoop\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"scoop\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.16246795654297],\"y\":[5.603267669677734],\"z\":[-5.9717302322387695],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=surprised\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"surprised\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"surprised\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.99591064453125],\"y\":[-5.411294937133789],\"z\":[9.936249732971191],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=awesome\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"awesome\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"awesome\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-70.86615753173828],\"y\":[-1.2652968168258667],\"z\":[14.617732048034668],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=rewarded\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"rewarded\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"rewarded\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.99559783935547],\"y\":[-0.2534556984901428],\"z\":[-5.502534866333008],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=suggestion\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"suggestion\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"suggestion\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.25018310546875],\"y\":[-1.4468708038330078],\"z\":[-0.5847004652023315],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=paramount\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"paramount\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"paramount\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-72.06446838378906],\"y\":[9.513653755187988],\"z\":[-5.487974643707275],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=pleasant\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"pleasant\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"pleasant\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-76.04872131347656],\"y\":[-3.97763729095459],\"z\":[2.2802021503448486],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=worries\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"worries\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"worries\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.1656494140625],\"y\":[-5.890462875366211],\"z\":[9.755501747131348],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=perfect\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"perfect\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"perfect\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-74.38725280761719],\"y\":[-5.497030735015869],\"z\":[13.179402351379395],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=outstanding\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"outstanding\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"outstanding\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-75.67615509033203],\"y\":[-6.754671096801758],\"z\":[13.842896461486816],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unlikeable\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unlikeable\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unlikeable\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.13069534301758],\"y\":[41.456642150878906],\"z\":[2.3355519771575928],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=dreadful\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"dreadful\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"dreadful\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.9150447845459],\"y\":[41.52838134765625],\"z\":[-4.135792255401611],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=southwestair\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"southwestair\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"southwestair\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[35.648048400878906],\"y\":[39.69951629638672],\"z\":[-9.788666725158691],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=luggage\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"luggage\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"luggage\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[34.221458435058594],\"y\":[45.168190002441406],\"z\":[-7.649970531463623],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=useless\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"useless\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"useless\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[35.33056640625],\"y\":[41.92044448852539],\"z\":[-10.74499797821045],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=dull\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"dull\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"dull\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.95893478393555],\"y\":[43.3472785949707],\"z\":[-10.688202857971191],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=promising\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"promising\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"promising\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.18068504333496],\"y\":[41.77363204956055],\"z\":[1.378912091255188],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=usairways\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"usairways\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"usairways\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[36.12630081176758],\"y\":[37.99903106689453],\"z\":[-6.977839469909668],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=buggy\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"buggy\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"buggy\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.42617416381836],\"y\":[44.599159240722656],\"z\":[-3.5288162231445312],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=talentless\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"talentless\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"talentless\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.45330047607422],\"y\":[35.377384185791016],\"z\":[11.737597465515137],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=miscast\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"miscast\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"miscast\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.02061462402344],\"y\":[41.986873626708984],\"z\":[-2.3099212646484375],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=lame\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"lame\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"lame\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[30.98431396484375],\"y\":[41.851322174072266],\"z\":[-1.6646262407302856],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unconvincing\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unconvincing\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unconvincing\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.58768081665039],\"y\":[40.39582061767578],\"z\":[10.920236587524414],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=disappointment\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"disappointment\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"disappointment\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.88656997680664],\"y\":[42.081077575683594],\"z\":[-9.3167085647583],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=disappointing\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"disappointing\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"disappointing\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[32.352447509765625],\"y\":[41.62972640991211],\"z\":[-7.347678184509277],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unfortunately\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unfortunately\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unfortunately\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[32.42972183227539],\"y\":[41.72054672241211],\"z\":[-0.6767988204956055],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=americanair\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"americanair\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"americanair\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[35.140838623046875],\"y\":[37.096248626708984],\"z\":[-7.1335344314575195],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=badly\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"badly\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"badly\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.52713394165039],\"y\":[41.650909423828125],\"z\":[9.080424308776855],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=garbage\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"garbage\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"garbage\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.266809463500977],\"y\":[42.34782791137695],\"z\":[5.201821804046631],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=laughable\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"laughable\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"laughable\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[29.522497177124023],\"y\":[43.12339401245117],\"z\":[1.9352858066558838],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=uninteresting\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"uninteresting\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"uninteresting\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[32.0759391784668],\"y\":[44.29001235961914],\"z\":[-4.9292073249816895],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=insulting\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"insulting\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"insulting\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.51063346862793],\"y\":[41.974788665771484],\"z\":[10.894530296325684],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=unwatchable\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"unwatchable\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"unwatchable\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[32.308258056640625],\"y\":[44.451507568359375],\"z\":[-6.943864822387695],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=trite\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"trite\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"trite\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.373411178588867],\"y\":[41.73969650268555],\"z\":[6.8908796310424805],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=uninspired\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"uninspired\",\"marker\":{\"color\":\"#FFA15A\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"uninspired\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.9010124206543],\"y\":[43.86455535888672],\"z\":[-5.942080974578857],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=stinker\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"stinker\",\"marker\":{\"color\":\"#19d3f3\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"stinker\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[32.371578216552734],\"y\":[42.836238861083984],\"z\":[-5.633058547973633],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=waste\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"waste\",\"marker\":{\"color\":\"#FF6692\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"waste\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.2747917175293],\"y\":[40.719261169433594],\"z\":[-11.087099075317383],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=worst\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"worst\",\"marker\":{\"color\":\"#B6E880\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"worst\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[34.35503387451172],\"y\":[37.856327056884766],\"z\":[-8.249153137207031],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=horrible\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"horrible\",\"marker\":{\"color\":\"#FF97FF\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"horrible\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[31.988035202026367],\"y\":[42.31867980957031],\"z\":[-9.424381256103516],\"type\":\"scatter3d\"},{\"hovertemplate\":\"index=awful\\u003cbr\\u003eX=%{x}\\u003cbr\\u003eY=%{y}\\u003cbr\\u003eZ=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"awful\",\"marker\":{\"color\":\"#FECB52\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"awful\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[33.691402435302734],\"y\":[39.47203063964844],\"z\":[-9.870433807373047],\"type\":\"scatter3d\"}],                        {\"template\":{\"data\":{\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"rgb(237,237,237)\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"bar\":[{\"error_x\":{\"color\":\"rgb(51,51,51)\"},\"error_y\":{\"color\":\"rgb(51,51,51)\"},\"marker\":{\"line\":{\"color\":\"rgb(237,237,237)\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(51,51,51)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(51,51,51)\"},\"baxis\":{\"endlinecolor\":\"rgb(51,51,51)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(51,51,51)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"choropleth\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"contourcarpet\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"contour\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"heatmapgl\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"heatmap\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"histogram2dcontour\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"histogram2d\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattermapbox\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterpolargl\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterpolar\"}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"colorscale\":{\"sequential\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"sequentialminus\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]]},\"colorway\":[\"#F8766D\",\"#A3A500\",\"#00BF7D\",\"#00B0F6\",\"#E76BF3\"],\"font\":{\"color\":\"rgb(51,51,51)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"rgb(237,237,237)\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"rgb(237,237,237)\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"bgcolor\":\"rgb(237,237,237)\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"bgcolor\":\"rgb(237,237,237)\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"}},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\"}}},\"scene\":{\"domain\":{\"x\":[0.0,1.0],\"y\":[0.0,1.0]},\"xaxis\":{\"title\":{\"text\":\"X\"}},\"yaxis\":{\"title\":{\"text\":\"Y\"}},\"zaxis\":{\"title\":{\"text\":\"Z\"}}},\"legend\":{\"title\":{\"text\":\"index\"},\"tracegroupgap\":0},\"margin\":{\"t\":60},\"title\":{\"text\":\"\\u003cb\\u003eThe \\\"\\u003ci\\u003eterrible\\u003c\\u002fi\\u003e\\\" cluster\\u003c\\u002fb\\u003e\"}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('7b1393e2-2193-4c52-bef7-39876a875ff1');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import plotly.express as px\n",
        "\n",
        "fig = px.scatter_3d(\n",
        "    cluster, x=\"X\", y=\"Y\", z=\"Z\", color=cluster.index,\n",
        "    labels={0: 'X', 1: 'Y', 2: 'Z'}\n",
        ")\n",
        "\n",
        "fig.update_layout(\n",
        "    template='ggplot2',\n",
        "    title=f'<b>The \"<i>terrible</i>\" cluster</b>',\n",
        "    )\n",
        "\n",
        "fig.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQLKNjNKSvui"
      },
      "source": [
        "We now understand what tokens will push a clarification more to one side than the other. In general, exploring embeddings and embedding layers can provide valuable insights into how language models represent language, which can then be used to improve the interpretability and explainability of such models.\n",
        "\n",
        "This same type of analysis can be used to explore models whose outputs are embedding themselves, like [`SentenceTransformers`](https://www.sbert.net/), wich are models that are used to generate raw embeddings that serve in different types of downstream applications (e.g., semantic search). Let us perform a test using all the tools we learned and some open-source models and datasets.\n",
        "\n",
        "> **Note: To learn more about the use of text embeddings and the SentenceTransformer framework, we recommend \"[Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084).\"**\n",
        "\n",
        "First, let us download samples of text that can be easily separated into classes: the AG dataset, a collection of more than 1 million news articles."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "YEjo8jmeT06G",
        "outputId": "16c1c3c1-5cee-432e-b4c2-509f9089898d"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 400,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 400,\n        \"samples\": [\n          \"Wall St. Bears Claw Back Into the Black  NEW YORK (Reuters) - Short-sellers, Wall Street's dwindling  band of ultra-cynics, are seeing green again.\",\n          \"Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\\\about the economy and the outlook for earnings are expected to\\\\hang over the stock market this week during the depth of the\\\\summer doldrums.\",\n          \"Options running out as Najaf talks collapse BAGHDAD -- After more than a week of fighting between the Mahdi Army and the combined US and Iraqi forces in Najaf, Baghdad's Sadr City, and a half-dozen other cities, analysts say neither interim Prime Minister Iyad Allawi nor radical cleric Moqtada al-Sadr had any good options left other than to talk.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"Sports News\",\n          \"Science News\",\n          \"World News\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-cd0e3f87-0dc3-452d-b849-50c7bb432004\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Venezuelans Vote Early in Referendum on Chavez...</td>\n",
              "      <td>World News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>S.Koreans Clash with Police on Iraq Troop Disp...</td>\n",
              "      <td>World News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Palestinians in Israeli Jails Start Hunger Str...</td>\n",
              "      <td>World News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Seven Georgian soldiers wounded as South Osset...</td>\n",
              "      <td>World News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Rwandan Troops Arrive in Darfur (AP) AP - Doze...</td>\n",
              "      <td>World News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>395</th>\n",
              "      <td>Saudis: Bin Laden associate surrenders \\\\\"(CNN...</td>\n",
              "      <td>Science News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>396</th>\n",
              "      <td>Mozilla Exceptions (mexception) \\\\For some rea...</td>\n",
              "      <td>Science News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>397</th>\n",
              "      <td>Ron Regan Jr is My Kinda Guy \\\\\"Now that the c...</td>\n",
              "      <td>Science News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>398</th>\n",
              "      <td>Al Qaeda member surrenders \\\\\"RIYADH, Saudi Ar...</td>\n",
              "      <td>Science News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>399</th>\n",
              "      <td>Mission Accomplished! \\\\\"BAGHDAD, Iraq (CNN) -...</td>\n",
              "      <td>Science News</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>400 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cd0e3f87-0dc3-452d-b849-50c7bb432004')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-cd0e3f87-0dc3-452d-b849-50c7bb432004 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-cd0e3f87-0dc3-452d-b849-50c7bb432004');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-80b37ab7-3c8f-4ce5-950f-442185fe16f4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-80b37ab7-3c8f-4ce5-950f-442185fe16f4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-80b37ab7-3c8f-4ce5-950f-442185fe16f4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                  text         label\n",
              "0    Venezuelans Vote Early in Referendum on Chavez...    World News\n",
              "1    S.Koreans Clash with Police on Iraq Troop Disp...    World News\n",
              "2    Palestinians in Israeli Jails Start Hunger Str...    World News\n",
              "3    Seven Georgian soldiers wounded as South Osset...    World News\n",
              "4    Rwandan Troops Arrive in Darfur (AP) AP - Doze...    World News\n",
              "..                                                 ...           ...\n",
              "395  Saudis: Bin Laden associate surrenders \\\\\"(CNN...  Science News\n",
              "396  Mozilla Exceptions (mexception) \\\\For some rea...  Science News\n",
              "397  Ron Regan Jr is My Kinda Guy \\\\\"Now that the c...  Science News\n",
              "398  Al Qaeda member surrenders \\\\\"RIYADH, Saudi Ar...  Science News\n",
              "399  Mission Accomplished! \\\\\"BAGHDAD, Iraq (CNN) -...  Science News\n",
              "\n",
              "[400 rows x 2 columns]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "import pandas as pd\n",
        "\n",
        "dataset = load_dataset(\"ag_news\", split=\"train\")\n",
        "\n",
        "# Convert the dataset into a pandas.dataframe for easy manipulation\n",
        "dataset = dataset.to_pandas()\n",
        "\n",
        "# Let us select only 100 samples from each class and change the integer\n",
        "# value of the labels by their string counterparts.\n",
        "world_news = dataset[dataset['label'] == 0].head(100).reset_index(drop=True)\n",
        "world_news['label'] = \"World News\"\n",
        "sports_news = dataset[dataset['label'] == 1].head(100).reset_index(drop=True)\n",
        "sports_news['label'] = \"Sports News\"\n",
        "business_news = dataset[dataset['label'] == 2].head(100).reset_index(drop=True)\n",
        "business_news['label'] = \"Business News\"\n",
        "sci_news = dataset[dataset['label'] == 3].head(100).reset_index(drop=True)\n",
        "sci_news['label'] = \"Science News\"\n",
        "\n",
        "df = pd.concat([world_news, sports_news, business_news, sci_news]).reset_index(drop=True)\n",
        "\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JM0fMNUoeKIc"
      },
      "source": [
        "Now, let us embed all of our samples into a 384-dimensional dense vector using the [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) SentenceTransformer, available on the Hub! 🤗"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "metadata": {
        "id": "cUtpv_V3Vgwc"
      },
      "outputs": [],
      "source": [
        "!pip install -U sentence-transformers -q\n",
        "\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "# Load the model\n",
        "model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\n",
        "\n",
        "# Embed every sample and append them to this list\n",
        "embeddings = []\n",
        "\n",
        "for sentence in df.text:\n",
        "  embedding = model.encode(sentence)\n",
        "  embeddings.append(embedding)\n",
        "\n",
        "df['embeddings'] = embeddings\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSva0UZifG6t"
      },
      "source": [
        "Now that we have embeddings to represent our sentences, all previous techniques are applied, like the cosine similarity calculation, to find the most similar instances:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VoWmfKt1V9oV",
        "outputId": "c84ea369-58ca-4966-8bfd-00a310105137"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Similarity from a random sample of class 'World News' and 'World News': 0.02805277518928051\n",
            "Similarity from a random sample of class 'World News' and 'Sports News': -0.011308781802654266\n",
            "Similarity from a random sample of class 'World News' and 'Business News': 0.11066465079784393\n",
            "Similarity from a random sample of class 'World News' and 'Science News': -0.0026667313650250435\n"
          ]
        }
      ],
      "source": [
        "def compare_embeddings(df, class1, class2):\n",
        "    \"\"\"\n",
        "    Calculate the cosine similarity between random embeddings of two classes.\n",
        "\n",
        "    Parameters:\n",
        "    df (DataFrame): DataFrame containing 'label' and 'embeddings' columns.\n",
        "    class1 (str): Name of the first class.\n",
        "    class2 (str): Name of the second class.\n",
        "\n",
        "    Returns:\n",
        "    float: Cosine similarity between random embeddings of the two classes.\n",
        "    \"\"\"\n",
        "\n",
        "    # Get random embeddings for class1 and class2\n",
        "    A = df.loc[df['label'] == class1, 'embeddings'].sample(n=1).iloc[0]\n",
        "    B = df.loc[df['label'] == class2, 'embeddings'].sample(n=1).iloc[0]\n",
        "\n",
        "    # Calculate cosine similarity\n",
        "    cos_similarity = np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B))\n",
        "\n",
        "    # Print the result\n",
        "    print(f\"Similarity from a random sample of class '{class1}' and '{class2}': {cos_similarity}\")\n",
        "\n",
        "compare_embeddings(df, \"World News\", \"World News\")\n",
        "compare_embeddings(df, \"World News\", \"Sports News\")\n",
        "compare_embeddings(df, \"World News\", \"Business News\")\n",
        "compare_embeddings(df, \"World News\", \"Science News\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zXQI3mHUfoeX"
      },
      "source": [
        "Or the 3D projection of this embedding space, which helps visualize how these vectors can help us group/separate data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "id": "Aa34HBOCW6gt",
        "outputId": "e95a3fb8-e40a-4d96-c4f5-8f962fff583f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] Computing 301 nearest neighbors...\n",
            "[t-SNE] Indexed 400 samples in 0.000s...\n",
            "[t-SNE] Computed neighbors for 400 samples in 0.024s...\n",
            "[t-SNE] Computed conditional probabilities for sample 400 / 400\n",
            "[t-SNE] Mean sigma: 0.419129\n",
            "[t-SNE] KL divergence after 100 iterations with early exaggeration: 45.606083\n",
            "[t-SNE] KL divergence after 500 iterations: 0.709099\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script charset=\"utf-8\" src=\"https://cdn.plot.ly/plotly-2.24.1.min.js\"></script>                <div id=\"1d4b1e95-5592-4c94-a816-4027135924b7\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"1d4b1e95-5592-4c94-a816-4027135924b7\")) {                    Plotly.newPlot(                        \"1d4b1e95-5592-4c94-a816-4027135924b7\",                        [{\"hovertemplate\":\"color=World News\\u003cbr\\u003e0=%{x}\\u003cbr\\u003e1=%{y}\\u003cbr\\u003e2=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"World News\",\"marker\":{\"color\":\"#636efa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"World News\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[14.851349830627441,-0.4494503438472748,-18.295042037963867,-8.317859649658203,1.2918637990951538,1.9524338245391846,-2.0971221923828125,1.2509080171585083,-14.247369766235352,-4.38153076171875,-3.1512253284454346,-0.20466671884059906,4.449758529663086,1.0348502397537231,-22.95966339111328,-8.647599220275879,24.04866600036621,-14.662789344787598,-5.900322914123535,6.276693344116211,-16.645204544067383,-2.1464176177978516,-31.078889846801758,-14.307295799255371,0.9167388677597046,7.392427444458008,-12.401911735534668,-7.4642252922058105,-1.7561326026916504,-10.162678718566895,18.885311126708984,43.00531768798828,-0.06369822472333908,-7.111968517303467,-7.0251569747924805,-5.537142276763916,-4.797414779663086,2.3205299377441406,-23.2121639251709,-17.12272834777832,-23.335006713867188,-28.52300453186035,14.614542961120605,3.372549295425415,-18.134714126586914,-30.354290008544922,-13.776876449584961,-2.4164583683013916,2.450815200805664,6.4372758865356445,0.4511706531047821,4.593705654144287,-3.6221635341644287,46.59565353393555,-1.5640389919281006,11.107181549072266,-10.60376262664795,8.111292839050293,-8.414453506469727,-24.985191345214844,-20.116907119750977,-0.15708371996879578,-14.128486633300781,-24.859628677368164,-8.708986282348633,-13.179619789123535,14.570615768432617,10.43133544921875,-7.296465873718262,-18.790525436401367,-11.057378768920898,-20.58102798461914,-5.036354064941406,-20.43958282470703,-16.879384994506836,10.178651809692383,-8.17172908782959,-17.538564682006836,8.433480262756348,14.427213668823242,-9.587438583374023,11.628219604492188,-17.409887313842773,15.996779441833496,-1.7744429111480713,-5.538687705993652,1.4344427585601807,5.532360553741455,9.440223693847656,-21.295841217041016,-0.10356356203556061,7.154561996459961,3.518900156021118,-4.728616237640381,4.334668159484863,5.182398319244385,8.698486328125,25.41961669921875,-4.170196056365967,10.09554386138916],\"y\":[-23.039493560791016,-30.65717315673828,-30.769210815429688,-12.099910736083984,-28.5572509765625,-32.986358642578125,16.892322540283203,-16.70941162109375,11.79236888885498,14.039752006530762,10.53545093536377,28.718673706054688,27.11310577392578,-19.47777557373047,-14.141716003417969,3.4509470462799072,-23.626766204833984,-36.90925598144531,-13.106978416442871,-23.99074935913086,-0.7134338021278381,-35.20429992675781,-15.668548583984375,-36.551727294921875,7.687464714050293,-8.867330551147461,-12.8939208984375,-23.595212936401367,24.51349449157715,32.047157287597656,-19.632122039794922,22.57445526123047,7.14204740524292,-10.896649360656738,21.8912353515625,-22.099775314331055,-39.184391021728516,-23.203691482543945,13.171520233154297,26.479700088500977,14.546634674072266,-26.75040054321289,-29.503732681274414,19.1152400970459,-12.75678539276123,-19.346189498901367,-7.536530494689941,-3.4190003871917725,-3.5437991619110107,5.924500465393066,-14.306596755981445,-1.8974469900131226,-6.697824001312256,7.89823579788208,3.9452414512634277,-21.522106170654297,-23.027847290039062,8.117981910705566,-33.821624755859375,-6.270880222320557,0.10744774341583252,-30.297122955322266,19.11916160583496,-21.05546760559082,-0.26124146580696106,28.567317962646484,-11.635087966918945,18.620317459106445,32.81642532348633,19.622943878173828,-16.59991455078125,-25.4251708984375,-6.181875705718994,-29.285886764526367,-5.388030052185059,-16.630598068237305,0.2185177057981491,10.69266128540039,8.622978210449219,8.002092361450195,-4.744480609893799,-14.018831253051758,-25.625391006469727,-20.53297233581543,-6.341274261474609,7.931460380554199,-10.975284576416016,-32.678890228271484,-22.16535186767578,-3.173175811767578,-23.995113372802734,-33.996829986572266,-39.71698760986328,0.24468745291233063,4.3978753089904785,-14.90866756439209,-24.429101943969727,16.74311637878418,-4.9495344161987305,-11.846867561340332],\"z\":[39.38364028930664,10.44075870513916,23.83829689025879,5.81724214553833,-2.574755907058716,-10.730060577392578,25.129119873046875,-0.40708020329475403,-10.074734687805176,-7.755985736846924,1.8085602521896362,-14.212276458740234,-23.125072479248047,14.433587074279785,31.832212448120117,26.513154983520508,34.298702239990234,-5.564160346984863,19.754165649414062,39.57061004638672,42.54488754272461,-2.2296204566955566,-24.549339294433594,7.680132865905762,-49.01848220825195,8.035333633422852,10.838616371154785,22.467403411865234,17.792375564575195,22.52866554260254,27.72511863708496,15.318778991699219,32.542762756347656,28.049320220947266,33.68373107910156,5.63753604888916,16.269392013549805,31.63558006286621,-13.880946159362793,-21.387128829956055,-21.08916664123535,-14.779151916503906,31.440629959106445,24.84697723388672,22.81771469116211,-14.566500663757324,40.48271560668945,27.229450225830078,19.469093322753906,25.57015037536621,11.462329864501953,-1.3900953531265259,-7.155097961425781,-20.87925910949707,3.400505781173706,32.10195541381836,16.260221481323242,30.878400802612305,32.39999771118164,34.93895721435547,24.72601318359375,16.735755920410156,-23.29747200012207,-20.090557098388672,39.64750671386719,10.340472221374512,32.80015563964844,13.018505096435547,-27.73050880432129,-16.64861297607422,38.55832290649414,23.223474502563477,43.04853439331055,-10.41889476776123,1.6265968084335327,25.68752670288086,46.27104568481445,-18.09786033630371,45.81625747680664,14.191398620605469,21.61635971069336,39.53687286376953,32.27245330810547,18.24578094482422,4.3756937980651855,-7.46132755279541,22.844453811645508,30.827041625976562,-18.41960906982422,-3.956031560897827,-10.940056800842285,-5.237870693206787,-7.334000110626221,19.989736557006836,-6.474249839782715,33.99072265625,24.94208335876465,-31.732179641723633,14.881508827209473,3.27870774269104],\"type\":\"scatter3d\"},{\"hovertemplate\":\"color=Sports News\\u003cbr\\u003e0=%{x}\\u003cbr\\u003e1=%{y}\\u003cbr\\u003e2=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"Sports News\",\"marker\":{\"color\":\"#EF553B\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"Sports News\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-32.27085876464844,-44.921600341796875,-13.053045272827148,-39.7685432434082,-29.833742141723633,-4.400146007537842,-36.79161834716797,-24.812652587890625,-20.264612197875977,-17.56483268737793,-26.2911376953125,-37.15853500366211,-41.43257141113281,-23.21306037902832,-23.942066192626953,-16.421875,-35.9924430847168,-29.598548889160156,-46.78053283691406,-50.760597229003906,-39.7693977355957,-17.444799423217773,-15.550275802612305,-30.695629119873047,-36.66217803955078,-38.838279724121094,-26.93861961364746,-30.281129837036133,-28.524463653564453,-3.4075868129730225,-26.18147850036621,-33.58112335205078,-29.04154396057129,-39.14352798461914,-36.99308395385742,-33.2257194519043,-26.057861328125,-6.126796245574951,-22.894548416137695,-42.69519805908203,-27.222009658813477,-48.9719352722168,-19.69448471069336,-22.15169906616211,-35.70076370239258,-33.8316650390625,-39.85841369628906,-41.42987823486328,-21.9499454498291,-22.3398380279541,-14.512460708618164,-17.65098762512207,-12.911017417907715,-44.24354553222656,-11.030413627624512,17.265064239501953,-15.651721954345703,-11.916211128234863,-14.735610008239746,-29.65213966369629,16.918556213378906,-33.69859313964844,-24.876914978027344,-13.365962028503418,-47.847537994384766,-26.7559757232666,-14.142200469970703,-14.784585952758789,-40.09101867675781,-36.15340042114258,-12.112476348876953,-30.41069221496582,-38.2806510925293,-12.247681617736816,-33.86552810668945,-46.1823844909668,-40.069740295410156,-47.465065002441406,-36.74091339111328,-35.354251861572266,-30.904067993164062,-17.71195411682129,-17.95790672302246,4.375357151031494,-31.557350158691406,-47.409934997558594,-31.251001358032227,-23.317928314208984,-17.098247528076172,-21.040428161621094,-19.80371856689453,-32.35628128051758,-23.911426544189453,-30.643321990966797,-23.81680679321289,-26.31976890563965,-43.46316146850586,-20.577354431152344,-35.85853958129883,-31.987573623657227],\"y\":[-24.0767822265625,9.663642883300781,4.712051868438721,-1.376815915107727,11.819246292114258,28.37314224243164,-17.33838653564453,-22.160831451416016,-23.798629760742188,30.153059005737305,-37.73059844970703,4.375187873840332,7.052773475646973,-32.4421272277832,-10.710949897766113,19.717559814453125,4.4658732414245605,1.8588531017303467,22.503986358642578,5.109516620635986,16.856273651123047,8.075572967529297,37.72163391113281,-30.2070255279541,18.742828369140625,7.971606731414795,14.137394905090332,-5.323047637939453,-11.308667182922363,31.398622512817383,32.8521614074707,-20.695308685302734,-22.75653076171875,-25.0210018157959,-19.199480056762695,-16.31110954284668,21.446256637573242,34.431819915771484,26.491270065307617,-2.9958066940307617,3.1983964443206787,11.692025184631348,-10.124882698059082,-29.62252426147461,17.78798484802246,24.478883743286133,16.404752731323242,-9.85356616973877,6.0913238525390625,-12.254936218261719,22.045551300048828,-29.756046295166016,-12.097258567810059,-15.716573715209961,0.8338536620140076,28.200376510620117,-8.753716468811035,-24.816265106201172,-28.313974380493164,-30.797006607055664,22.753707885742188,-10.64408016204834,-13.36949348449707,-1.0325103998184204,-16.903779983520508,33.99483108520508,-19.40825080871582,-23.573144912719727,7.743034362792969,9.479887008666992,-30.246315002441406,4.257551193237305,28.771394729614258,17.901586532592773,31.850727081298828,-0.6641857028007507,-10.645657539367676,3.0682857036590576,-1.7817610502243042,-0.05154131352901459,-0.6220179200172424,33.23876953125,-1.1143821477890015,21.153879165649414,-4.6073479652404785,2.887742042541504,2.8248019218444824,24.524641036987305,24.458742141723633,16.435016632080078,-20.12311553955078,15.467558860778809,23.565038681030273,28.130823135375977,7.528209209442139,31.988954544067383,-23.052337646484375,-2.7325868606567383,24.844058990478516,13.837584495544434],\"z\":[-21.284564971923828,9.921710968017578,3.3696281909942627,5.067778587341309,3.2915663719177246,-3.0830256938934326,-4.931541919708252,-6.777738094329834,4.67484712600708,-4.154628753662109,-11.168218612670898,-5.399458408355713,18.29486656188965,-1.516144037246704,-12.442060470581055,10.678729057312012,-11.82205867767334,33.591556549072266,1.061905026435852,16.431482315063477,3.7498674392700195,15.53082275390625,5.469494342803955,-4.606789588928223,13.340458869934082,29.681007385253906,12.818482398986816,-19.80269432067871,19.011821746826172,2.8478212356567383,2.3507204055786133,4.449862003326416,14.689623832702637,12.405908584594727,16.78093910217285,9.622994422912598,18.183324813842773,-4.585176944732666,24.995742797851562,-18.00176239013672,-17.936080932617188,-3.2284817695617676,-18.45672035217285,-21.25605010986328,-22.971424102783203,-2.6221137046813965,-10.90378189086914,-3.8175711631774902,7.648508548736572,7.00363302230835,-3.962358236312866,8.004515647888184,-15.08301067352295,6.8646464347839355,7.648642063140869,30.282283782958984,-5.561641216278076,-10.881038665771484,1.5025585889816284,5.967782497406006,32.390830993652344,-6.717737197875977,-2.2946815490722656,-4.957944393157959,1.6596510410308838,-14.277679443359375,-1.1170204877853394,-20.827045440673828,2.4402098655700684,11.43819522857666,-14.777541160583496,16.847644805908203,8.140873908996582,2.8200767040252686,12.068643569946289,12.077552795410156,-14.498862266540527,4.0435590744018555,16.31923484802246,22.06277084350586,0.3415714204311371,12.069853782653809,-10.536965370178223,39.67674255371094,6.891934871673584,-8.006065368652344,5.7035346031188965,6.364495754241943,4.001214981079102,1.093253493309021,-13.138951301574707,-4.864374160766602,-5.633208274841309,-16.875471115112305,-3.526109457015991,24.83130645751953,3.209414482116699,13.835844039916992,18.622098922729492,21.35709571838379],\"type\":\"scatter3d\"},{\"hovertemplate\":\"color=Business News\\u003cbr\\u003e0=%{x}\\u003cbr\\u003e1=%{y}\\u003cbr\\u003e2=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"Business News\",\"marker\":{\"color\":\"#00cc96\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"Business News\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[22.586721420288086,21.03462028503418,22.231348037719727,11.454485893249512,16.448881149291992,29.845077514648438,21.009296417236328,23.172821044921875,22.37908172607422,21.805370330810547,23.2165584564209,20.923580169677734,22.96179962158203,38.89272689819336,25.570112228393555,26.575368881225586,44.4856071472168,21.344350814819336,22.825132369995117,19.27326202392578,32.93518829345703,33.87615203857422,37.8011589050293,26.033531188964844,34.67338943481445,31.833141326904297,45.098175048828125,32.00505065917969,8.709808349609375,4.263030052185059,9.194351196289062,26.7830867767334,15.261666297912598,11.38105583190918,15.399541854858398,14.07942008972168,34.558349609375,41.088748931884766,32.861148834228516,27.0635929107666,-5.262726306915283,15.93550968170166,13.28964900970459,17.13258934020996,16.308536529541016,7.404776573181152,11.864821434020996,37.973079681396484,31.783884048461914,36.6573600769043,-12.762791633605957,34.366127014160156,32.6773567199707,-2.2331149578094482,32.7579345703125,42.306365966796875,29.257097244262695,25.192962646484375,41.001373291015625,7.977536201477051,37.81194305419922,14.294278144836426,-18.721601486206055,-15.350939750671387,11.85093879699707,9.79886245727539,-18.037580490112305,23.50682830810547,35.246742248535156,23.019752502441406,26.233369827270508,34.464691162109375,37.16781997680664,44.54964065551758,44.37367630004883,44.75457763671875,29.061887741088867,37.60343933105469,26.106760025024414,22.97564125061035,28.79549789428711,20.87925910949707,19.048786163330078,27.19162368774414,15.217822074890137,20.09588050842285,14.313364028930664,28.451976776123047,27.890235900878906,20.333316802978516,25.763059616088867,28.072134017944336,17.76294708251953,8.358107566833496,16.33696746826172,24.379404067993164,20.782928466796875,14.258492469787598,27.86727523803711,17.364030838012695],\"y\":[-33.94035339355469,-8.522775650024414,-8.45034408569336,-30.61002540588379,-10.178162574768066,-2.5006728172302246,2.4524264335632324,6.590967655181885,30.016578674316406,-36.782737731933594,-16.07670021057129,-26.272789001464844,-31.672542572021484,-20.410293579101562,-1.860210657119751,40.417755126953125,4.464654445648193,8.41275405883789,-9.158495903015137,-24.562009811401367,-20.54971694946289,-10.232535362243652,-4.883948802947998,-7.073763370513916,7.330511569976807,-8.00218677520752,-21.123579025268555,5.931060791015625,26.268062591552734,-40.54911422729492,30.906099319458008,35.56884765625,-24.322179794311523,-19.761642456054688,34.2401008605957,-5.922678470611572,-26.924697875976562,-5.3308258056640625,11.890508651733398,17.367149353027344,11.111664772033691,-0.3848872780799866,38.69770431518555,42.390869140625,43.249839782714844,38.76151657104492,-18.85567855834961,23.977754592895508,-1.0615984201431274,14.128229141235352,30.377164840698242,19.267114639282227,20.343223571777344,-29.290019989013672,24.7702579498291,-11.460147857666016,28.66389274597168,32.1880989074707,0.7119131684303284,-3.7237601280212402,15.640071868896484,28.009557723999023,10.17222785949707,12.372812271118164,16.03631019592285,12.595552444458008,-16.10988426208496,12.967529296875,5.800457000732422,6.0537004470825195,13.752182006835938,-27.337665557861328,-20.055253982543945,1.7547228336334229,10.864439010620117,22.904420852661133,-18.274877548217773,-9.49548625946045,-10.754616737365723,0.38354557752609253,-11.142891883850098,-13.454364776611328,4.2012128829956055,-12.246875762939453,1.0597412586212158,-28.93602752685547,-31.187030792236328,0.5716927647590637,7.7551140785217285,19.95472526550293,16.27035903930664,22.797513961791992,19.671199798583984,24.97711753845215,15.804278373718262,10.9143705368042,5.1034016609191895,26.703487396240234,-22.361141204833984,25.96300506591797],\"z\":[-15.152987480163574,-11.628274917602539,7.334498882293701,9.68648624420166,14.891940116882324,9.538448333740234,8.34216022491455,17.881223678588867,9.883583068847656,-10.08588695526123,7.94402551651001,6.271447658538818,12.000669479370117,-8.984137535095215,19.359464645385742,2.6513023376464844,16.153085708618164,3.7945823669433594,21.57711410522461,-7.962772846221924,-2.505463123321533,23.755035400390625,27.774362564086914,37.229923248291016,23.786039352416992,35.60227966308594,-5.914170265197754,11.144341468811035,9.526805877685547,2.8943610191345215,1.7043375968933105,-11.184419631958008,1.8051648139953613,-4.4251484870910645,-19.65937614440918,-4.828625202178955,-6.543100833892822,13.046196937561035,-10.333873748779297,-3.829526662826538,14.944662094116211,17.616147994995117,-6.3206400871276855,14.443675994873047,8.719209671020508,-9.736685752868652,9.73900032043457,3.120812177658081,-0.7466244697570801,9.214940071105957,-17.56398582458496,-1.2192424535751343,21.593210220336914,-24.88503074645996,-7.83984375,4.567620277404785,22.571081161499023,2.0335938930511475,6.090366840362549,36.151939392089844,20.868202209472656,18.699993133544922,29.11677360534668,33.730796813964844,-3.4054312705993652,0.8447346091270447,-37.73942947387695,-10.553893089294434,-28.503414154052734,-16.225996017456055,-24.70464515686035,2.0592172145843506,3.3541200160980225,-4.940162181854248,1.5303447246551514,-12.408884048461914,-11.347677230834961,-2.7384519577026367,-4.4197998046875,-3.1194980144500732,5.5031280517578125,-3.3822877407073975,-5.128672122955322,12.70980167388916,2.3070192337036133,21.274959564208984,19.002241134643555,24.496715545654297,34.68619918823242,17.388280868530273,8.423201560974121,9.458313941955566,6.659567356109619,-33.70820617675781,20.41009521484375,30.502168655395508,34.12709045410156,-11.752482414245605,17.285907745361328,-6.885786056518555],\"type\":\"scatter3d\"},{\"hovertemplate\":\"color=Science News\\u003cbr\\u003e0=%{x}\\u003cbr\\u003e1=%{y}\\u003cbr\\u003e2=%{z}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"Science News\",\"marker\":{\"color\":\"#ab63fa\",\"symbol\":\"circle\"},\"mode\":\"markers\",\"name\":\"Science News\",\"scene\":\"scene\",\"showlegend\":true,\"x\":[-28.216171264648438,41.24378967285156,43.930755615234375,38.8336296081543,32.0819091796875,9.991097450256348,45.23860549926758,30.65899658203125,33.71002197265625,29.286407470703125,50.89205551147461,-9.384392738342285,-0.005731950979679823,0.4058060646057129,1.68915855884552,4.793918132781982,-9.361648559570312,19.10898208618164,-12.907402992248535,1.3686308860778809,20.31664276123047,13.86343765258789,18.843795776367188,11.214550971984863,20.45491600036621,15.282966613769531,22.88162612915039,13.7271089553833,-20.55745506286621,25.59332847595215,-24.600826263427734,24.70465660095215,-17.47600555419922,7.273338317871094,-17.219091415405273,-23.309900283813477,8.283601760864258,-6.683192253112793,7.312082290649414,21.7240047454834,0.554929792881012,28.101741790771484,14.476930618286133,9.10632038116455,25.502592086791992,-15.137263298034668,17.449216842651367,17.329429626464844,10.082060813903809,-6.638853549957275,22.93016815185547,-1.1402562856674194,13.9584379196167,7.45583963394165,-0.5478649735450745,-5.73060941696167,-10.356893539428711,16.214994430541992,-3.2695562839508057,-9.863292694091797,2.7741949558258057,1.8239866495132446,2.754068374633789,-34.35438537597656,-23.5819149017334,10.982108116149902,-7.557679176330566,-29.736724853515625,-4.035582065582275,-3.735888957977295,7.532730579376221,-11.212380409240723,-5.4888014793396,-0.01226925291121006,8.70445728302002,-15.012773513793945,-9.558597564697266,-21.719181060791016,4.828854560852051,11.367727279663086,-7.213024616241455,45.67938232421875,37.89380645751953,50.94611740112305,21.55977439880371,31.875568389892578,-3.978919744491577,15.518296241760254,-12.125113487243652,-5.804487228393555,30.982688903808594,36.05878829956055,-1.0118412971496582,-4.519086837768555,-1.659528374671936,2.6700174808502197,3.457690477371216,-0.8825507760047913,3.711272954940796,5.405750751495361],\"y\":[21.428945541381836,-2.6338508129119873,13.194235801696777,-9.492565155029297,8.863542556762695,9.881064414978027,-4.670666694641113,-1.4440665245056152,-9.876845359802246,21.382152557373047,-1.2426670789718628,20.193889617919922,10.470069885253906,-5.27143669128418,-12.8035249710083,20.272083282470703,-18.183950424194336,22.300241470336914,-4.6088175773620605,32.964481353759766,23.7203311920166,-15.359101295471191,-14.370622634887695,0.8712998628616333,-4.476613998413086,19.171218872070312,4.25894021987915,-7.318455219268799,14.548585891723633,4.192678928375244,12.100197792053223,-1.7183820009231567,-13.648207664489746,-3.56527042388916,0.3097197115421295,-6.06726598739624,-20.80162811279297,14.432755470275879,-0.6958110928535461,-21.919912338256836,4.322749137878418,-12.914888381958008,5.311083793640137,-14.091941833496094,-7.4504547119140625,15.493834495544434,-9.603075981140137,19.940784454345703,-13.590520858764648,-21.738916397094727,-11.217842102050781,-12.154531478881836,-17.246662139892578,5.908260822296143,16.35420036315918,23.76105499267578,9.922799110412598,5.373397350311279,-7.995233535766602,4.3525590896606445,17.6915340423584,0.7878743410110474,21.446531295776367,13.406988143920898,4.231443881988525,15.128681182861328,-6.813974380493164,5.322704792022705,0.03138578683137894,24.539936065673828,12.911033630371094,-0.3953452408313751,15.815900802612305,9.220388412475586,-7.455686092376709,10.041614532470703,6.433861255645752,-4.435220718383789,-6.1759538650512695,0.1770937591791153,-0.42612215876579285,17.56423568725586,4.933964252471924,12.222700119018555,-21.594844818115234,32.40995407104492,-14.592242240905762,-2.254678726196289,22.083538055419922,44.854759216308594,3.5057082176208496,16.73758316040039,46.03804397583008,47.86436080932617,42.60956954956055,7.065440654754639,38.1425895690918,18.84610366821289,9.183211326599121,0.21607379615306854],\"z\":[30.93760108947754,-17.523269653320312,-7.111269950866699,-20.60723876953125,0.5511781573295593,-12.15383529663086,-11.07687759399414,-12.1367769241333,-14.775638580322266,-14.775745391845703,0.300069659948349,-16.454391479492188,-16.510149002075195,-22.67158317565918,-24.785457611083984,-2.967221975326538,-40.648345947265625,-39.11324691772461,-44.29424285888672,27.790430068969727,-26.08123207092285,-24.83807945251465,-33.897342681884766,-42.79399490356445,-27.72654914855957,-43.633548736572266,-34.16889572143555,-20.140287399291992,-33.30084991455078,-40.217037200927734,-42.503841400146484,-21.9334659576416,-29.02153205871582,-42.887264251708984,-23.930265426635742,-33.46905517578125,-29.847206115722656,-22.79026985168457,-29.173595428466797,-32.942298889160156,-35.36274719238281,-31.32048225402832,-22.437314987182617,-44.9464111328125,-35.49496078491211,-42.09771728515625,-42.6005973815918,-22.37079620361328,-16.455873489379883,-36.83404541015625,-23.942842483520508,-15.519389152526855,-39.53483581542969,-22.645111083984375,-39.39221954345703,-37.762176513671875,-44.81718063354492,-31.972820281982422,-36.18663024902344,-40.15516662597656,-28.087650299072266,-14.991100311279297,-14.820839881896973,-27.821369171142578,-32.082359313964844,-20.44886589050293,-29.2033634185791,-40.32106399536133,-31.292335510253906,-26.419118881225586,-33.57657241821289,-31.821640014648438,-32.79838943481445,-28.406158447265625,-33.19746017456055,-31.71914291381836,-23.455299377441406,-40.74605941772461,-14.652228355407715,-14.498477935791016,-15.820809364318848,-16.209056854248047,-11.096611022949219,-12.33639144897461,-20.915475845336914,-16.505117416381836,-4.894654273986816,24.544384002685547,22.680583953857422,17.585311889648438,-20.180145263671875,-20.36176109313965,1.748802900314331,-3.0197434425354004,12.603119850158691,15.991859436035156,13.118636131286621,6.941479682922363,10.443632125854492,10.409668922424316],\"type\":\"scatter3d\"}],                        {\"template\":{\"data\":{\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"rgb(237,237,237)\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"bar\":[{\"error_x\":{\"color\":\"rgb(51,51,51)\"},\"error_y\":{\"color\":\"rgb(51,51,51)\"},\"marker\":{\"line\":{\"color\":\"rgb(237,237,237)\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(51,51,51)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(51,51,51)\"},\"baxis\":{\"endlinecolor\":\"rgb(51,51,51)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(51,51,51)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"choropleth\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"contourcarpet\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"contour\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"heatmapgl\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"heatmap\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"histogram2dcontour\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"histogram2d\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scattermapbox\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterpolargl\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterpolar\"}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"},\"colorscale\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"tickcolor\":\"rgb(237,237,237)\",\"ticklen\":6,\"ticks\":\"inside\"}},\"colorscale\":{\"sequential\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]],\"sequentialminus\":[[0,\"rgb(20,44,66)\"],[1,\"rgb(90,179,244)\"]]},\"colorway\":[\"#F8766D\",\"#A3A500\",\"#00BF7D\",\"#00B0F6\",\"#E76BF3\"],\"font\":{\"color\":\"rgb(51,51,51)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"rgb(237,237,237)\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"rgb(237,237,237)\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"bgcolor\":\"rgb(237,237,237)\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"rgb(237,237,237)\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"},\"bgcolor\":\"rgb(237,237,237)\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\"}},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showgrid\":true,\"tickcolor\":\"rgb(51,51,51)\",\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\"}}},\"scene\":{\"domain\":{\"x\":[0.0,1.0],\"y\":[0.0,1.0]},\"xaxis\":{\"title\":{\"text\":\"0\"}},\"yaxis\":{\"title\":{\"text\":\"1\"}},\"zaxis\":{\"title\":{\"text\":\"2\"}}},\"legend\":{\"title\":{\"text\":\"color\"},\"tracegroupgap\":0},\"margin\":{\"t\":60},\"title\":{\"text\":\"\\u003cb\\u003eThe \\\"\\u003ci\\u003eAG News\\u003c\\u002fi\\u003e\\\" cluster\\u003c\\u002fb\\u003e\"}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('1d4b1e95-5592-4c94-a816-4027135924b7');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tsne = TSNE(n_components=3, verbose=1, perplexity=100, n_iter=2000)\n",
        "tsne_results = tsne.fit_transform(np.vstack(df.embeddings)) # `np.vstack` stacks the arrays vertically to create a 2D array\n",
        "\n",
        "fig = px.scatter_3d(\n",
        "    tsne_results, x=0, y=1, z=2, color=df.label,\n",
        "    labels={0: 'X', 1: 'Y', 2: 'Z'}\n",
        ")\n",
        "\n",
        "fig.update_layout(\n",
        "    template='ggplot2',\n",
        "    title=f'<b>The \"<i>AG News</i>\" cluster</b>',\n",
        "    )\n",
        "\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWOIrdX4j7Jc"
      },
      "source": [
        "We could now restart our investigation by examining the embeddings of our SentenceTransformer (i.e., not the output embeddings, but the embeddings of the model itself).\n",
        "\n",
        "```python\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "\n",
        "# Load model from HuggingFace Hub\n",
        "tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')\n",
        "model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')\n",
        "\n",
        "vocabulary = tokenizer.get_vocab() # 30522 tokens\n",
        "embeddings = model.embeddings.word_embeddings.weight.shape # torch.Size([30522, 384])\n",
        "\n",
        "# Explore the embeddings of all-MiniLM-L6-v2 ...\n",
        "\n",
        "```\n",
        "\n",
        "You can definitely do this as an exercise (explore the embeddings of open-source models). This type of research can help improve our understanding of such models, and it was on a similar basis that \"[Glitch Tokens](https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation)\" (like \"SolidGoldMagikarp\") were discovered.\n",
        "\n",
        "---\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/teeny-tiny_castle).\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "aca09746cf57686f00a55ae76e987247ecfb5dd0b3b2e2474d8dbbf0c5e3377e"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "005ae0cbe84f436fb0bbd7b8b79301d3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e3b83bf785c841079b0eb8da03d2280a",
            "placeholder": "​",
            "style": "IPY_MODEL_f85debf1d10e439492e3c9a2ab63ab25",
            "value": "Downloading data: 100%"
          }
        },
        "1937b30314d7458dbdfa0febd814a650": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "37e73407aa344803b67601735e2f3347": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ed12e960d58244e08523c9108068effe",
            "placeholder": "​",
            "style": "IPY_MODEL_962e80e5b1ad4d9da37791592a67cb13",
            "value": "Downloading readme: 100%"
          }
        },
        "3a683fa31a49407ab3288abf350841d6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_37e73407aa344803b67601735e2f3347",
              "IPY_MODEL_f8816357acc747f4877a2be83271f8ba",
              "IPY_MODEL_84966d894109493fa9e48ac14bb5021a"
            ],
            "layout": "IPY_MODEL_cf212eb006f1475aaf260f5c5713fe73"
          }
        },
        "3eb89ad6e7164435b45d62bcfcc26bf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4afa55d091a84d9dacab7f83925ec504": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_005ae0cbe84f436fb0bbd7b8b79301d3",
              "IPY_MODEL_ef8af826e5054af09cf7e3085bc4b6fe",
              "IPY_MODEL_a2610366d6d040c0a0b189a72822f28a"
            ],
            "layout": "IPY_MODEL_8a4c4211ce974f89b0ef49089057da5c"
          }
        },
        "52c85bdd458e4be3a1ce7ca19f8c167a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "58c29a56ecec4552833d012eceb91869": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6b6b1ed4c1174d5a9edabc4f0cd6095e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7699d5f31503460f9b23866342a04c90": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8289e55cd9024772b455cd284734f401": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "84966d894109493fa9e48ac14bb5021a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6b6b1ed4c1174d5a9edabc4f0cd6095e",
            "placeholder": "​",
            "style": "IPY_MODEL_c09609c8c1b647fdb4af5119d6ace4d2",
            "value": " 912/912 [00:00&lt;00:00, 54.6kB/s]"
          }
        },
        "87033f75d6ce4862bda300407514397f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8a4c4211ce974f89b0ef49089057da5c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "962e80e5b1ad4d9da37791592a67cb13": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9aa1604cba644cc2bd67f3d85e6795b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e744ba11e2074ebc9997edea58a6ca73",
              "IPY_MODEL_c5126ed32897487b9153c5b7e3b7337e",
              "IPY_MODEL_f57b3512068b4be7bcb85e3f280a2150"
            ],
            "layout": "IPY_MODEL_e6970614afd04dc39b89490b4435d8d7"
          }
        },
        "a2610366d6d040c0a0b189a72822f28a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_58c29a56ecec4552833d012eceb91869",
            "placeholder": "​",
            "style": "IPY_MODEL_e99fb7502fe947b5a53781d09251afd6",
            "value": " 44.5M/44.5M [00:03&lt;00:00, 14.1MB/s]"
          }
        },
        "a78b63316ef04a8eab2f5728bedd3c7a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a8f393d5e45f4c6e8d2ee8e0d7f49dbb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c09609c8c1b647fdb4af5119d6ace4d2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c0d7d534cfeb4244bbd692758dee1937": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c5126ed32897487b9153c5b7e3b7337e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a8f393d5e45f4c6e8d2ee8e0d7f49dbb",
            "max": 85089,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3eb89ad6e7164435b45d62bcfcc26bf8",
            "value": 85089
          }
        },
        "cf212eb006f1475aaf260f5c5713fe73": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cf99c56f37b84cff94ea2b94134ae465": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e3b83bf785c841079b0eb8da03d2280a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e6970614afd04dc39b89490b4435d8d7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e744ba11e2074ebc9997edea58a6ca73": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c0d7d534cfeb4244bbd692758dee1937",
            "placeholder": "​",
            "style": "IPY_MODEL_8289e55cd9024772b455cd284734f401",
            "value": "Generating train split: 100%"
          }
        },
        "e99fb7502fe947b5a53781d09251afd6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ed12e960d58244e08523c9108068effe": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ef8af826e5054af09cf7e3085bc4b6fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7699d5f31503460f9b23866342a04c90",
            "max": 44486982,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_52c85bdd458e4be3a1ce7ca19f8c167a",
            "value": 44486982
          }
        },
        "f57b3512068b4be7bcb85e3f280a2150": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a78b63316ef04a8eab2f5728bedd3c7a",
            "placeholder": "​",
            "style": "IPY_MODEL_87033f75d6ce4862bda300407514397f",
            "value": " 85089/85089 [00:00&lt;00:00, 216312.27 examples/s]"
          }
        },
        "f85debf1d10e439492e3c9a2ab63ab25": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f8816357acc747f4877a2be83271f8ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cf99c56f37b84cff94ea2b94134ae465",
            "max": 912,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1937b30314d7458dbdfa0febd814a650",
            "value": 912
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}