deep-learning/nlp-sequence-models/week2/Operations_on_word_vectors_v2a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YTNn4zzAJWhA"
   },
   "source": [
    "# Operations on Word Vectors\n",
    "\n",
    "Welcome to your first assignment of Week 2, Course 5 of the Deep Learning Specialization! \n",
    "\n",
    "Because word embeddings are very computationally expensive to train, most ML practitioners will load a pre-trained set of embeddings. In this notebook you'll try your hand at loading, measuring similarity between, and modifying pre-trained embeddings. \n",
    "\n",
    "**After this assignment you'll be able to**:\n",
    "\n",
    "* Explain how word embeddings capture relationships between words\n",
    "* Load pre-trained word vectors\n",
    "* Measure similarity between word vectors using cosine similarity\n",
    "* Use word embeddings to solve word analogy problems such as Man is to Woman as King is to ______.  \n",
    "\n",
    "At the end of this notebook you'll have a chance to try an optional exercise, where you'll modify word embeddings to reduce their gender bias. Reducing bias is an important consideration in ML, so you're encouraged to take this challenge!  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [Packages](#0)\n",
    "- [1 - Load the Word Vectors](#1)\n",
    "- [2 - Embedding Vectors Versus One-Hot Vectors](#2)\n",
    "- [3 - Cosine Similarity](#3)\n",
    "    - [Exercise 1 - cosine_similarity](#ex-1)\n",
    "- [4 - Word Analogy Task](#4)\n",
    "    - [Exercise 2 - complete_analogy](#ex-2)\n",
    "- [5 - Debiasing Word Vectors (OPTIONAL/UNGRADED)](#5)\n",
    "    - [5.1 - Neutralize Bias for Non-Gender Specific Words](#5-1)\n",
    "        - [Exercise 3 - neutralize](#ex-3)\n",
    "    - [5.2 - Equalization Algorithm for Gender-Specific Words](#5-2)\n",
    "        - [Exercise 4 - equalize](#ex-4)\n",
    "- [6 - References](#6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZNlGgfN1JWhL"
   },
   "source": [
    "<a name='0'></a>\n",
    "## Packages\n",
    "\n",
    "Let's get started! Run the following cell to load the packages you'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "cWEywjgpJWhM",
    "outputId": "6b0b856a-d05d-4a6d-ba94-01d247254f20"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from w2v_utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D-CfRQ86JWhN"
   },
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Load the Word Vectors\n",
    "\n",
    "For this assignment, you'll use 50-dimensional GloVe vectors to represent words. \n",
    "Run the following cell to load the `word_to_vec_map`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "xCsAgH3dJWhO"
   },
   "outputs": [],
   "source": [
    "words, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cF9bim0eJWhO"
   },
   "source": [
    "You've loaded:\n",
    "- `words`: set of words in the vocabulary.\n",
    "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.\n",
    "\n",
    "<a name='2'></a>\n",
    "## 2 - Embedding Vectors Versus One-Hot Vectors\n",
    "Recall from the lesson videos that one-hot vectors don't do a good job of capturing the level of similarity between words. This is because every one-hot vector has the same Euclidean distance from any other one-hot vector.\n",
    "\n",
    "Embedding vectors, such as GloVe vectors, provide much more useful information about the meaning of individual words.  \n",
    "Now, see how you can use GloVe vectors to measure the similarity between two words! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xZBg2QwSJWhO"
   },
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Cosine Similarity\n",
    "\n",
    "To measure the similarity between two words, you need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows: \n",
    "\n",
    "$$\\text{CosineSimilarity(u, v)} = \\frac {u \\cdot v} {||u||_2 ||v||_2} = cos(\\theta) \\tag{1}$$\n",
    "\n",
    "* $u \\cdot v$ is the dot product (or inner product) of two vectors\n",
    "* $||u||_2$ is the norm (or length) of the vector $u$\n",
    "* $\\theta$ is the angle between $u$ and $v$. \n",
    "* The cosine similarity depends on the angle between $u$ and $v$. \n",
    "    * If $u$ and $v$ are very similar, their cosine similarity will be close to 1.\n",
    "    * If they are dissimilar, the cosine similarity will take a smaller value. \n",
    "\n",
    "<img src=\"images/cosine_sim.png\" style=\"width:800px;height:250px;\">\n",
    "<caption><center><font color='purple'><b>Figure 1</b>: The cosine of the angle between two vectors is a measure of their similarity.</font></center></caption>\n",
    "\n",
    "<a name='ex-1'></a>\n",
    "### Exercise 1 - cosine_similarity\n",
    "\n",
    "Implement the function `cosine_similarity()` to evaluate the similarity between word vectors.\n",
    "\n",
    "**Reminder**: The norm of $u$ is defined as $ ||u||_2 = \\sqrt{\\sum_{i=1}^{n} u_i^2}$\n",
    "\n",
    "#### Additional Hints\n",
    "* You may find [np.dot](https://numpy.org/doc/stable/reference/generated/numpy.dot.html), [np.sum](https://numpy.org/doc/stable/reference/generated/numpy.sum.html), or [np.sqrt](https://numpy.org/doc/stable/reference/generated/numpy.sqrt.html) useful depending upon the implementation that you choose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "mNjuTQ5JJWhP"
   },
   "outputs": [],
   "source": [
    "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION: cosine_similarity\n",
    "\n",
    "def cosine_similarity(u, v):\n",
    "    \"\"\"\n",
    "    Cosine similarity reflects the degree of similarity between u and v\n",
    "        \n",
    "    Arguments:\n",
    "        u -- a word vector of shape (n,)          \n",
    "        v -- a word vector of shape (n,)\n",
    "\n",
    "    Returns:\n",
    "        cosine_similarity -- the cosine similarity between u and v defined by the formula above.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Special case. Consider the case u = [0, 0], v=[0, 0]\n",
    "    if np.all(u == v):\n",
    "        return 1\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Compute the dot product between u and v (≈1 line)\n",
    "    dot = np.sum(u * v) \n",
    "    # Compute the L2 norm of u (≈1 line)\n",
    "    norm_u = np.sqrt(np.sum(u**2))\n",
    "    \n",
    "    # Compute the L2 norm of v (≈1 line)\n",
    "    norm_v = np.sqrt(np.sum(v**2))\n",
    "    \n",
    "    # Avoid division by 0\n",
    "    if np.isclose(norm_u * norm_v, 0, atol=1e-32):\n",
    "        return 0\n",
    "    \n",
    "    # Compute the cosine similarity defined by formula (1) (≈1 line)\n",
    "    cosine_similarity = dot / (norm_u * norm_v)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nqH8_-3BJWhP",
    "outputId": "9454fe42-80ca-4b40-ea23-69ebac038fae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine_similarity(father, mother) =  0.8909038442893616\n",
      "cosine_similarity(ball, crocodile) =  0.2743924626137942\n",
      "cosine_similarity(france - paris, rome - italy) =  -0.6751479308174201\n",
      "\u001b[92mAll test passed!\n"
     ]
    }
   ],
   "source": [
    "# START SKIP FOR GRADING\n",
    "father = word_to_vec_map[\"father\"]\n",
    "mother = word_to_vec_map[\"mother\"]\n",
    "ball = word_to_vec_map[\"ball\"]\n",
    "crocodile = word_to_vec_map[\"crocodile\"]\n",
    "france = word_to_vec_map[\"france\"]\n",
    "italy = word_to_vec_map[\"italy\"]\n",
    "paris = word_to_vec_map[\"paris\"]\n",
    "rome = word_to_vec_map[\"rome\"]\n",
    "\n",
    "print(\"cosine_similarity(father, mother) = \", cosine_similarity(father, mother))\n",
    "print(\"cosine_similarity(ball, crocodile) = \",cosine_similarity(ball, crocodile))\n",
    "print(\"cosine_similarity(france - paris, rome - italy) = \",cosine_similarity(france - paris, rome - italy))\n",
    "# END SKIP FOR GRADING\n",
    "\n",
    "# PUBLIC TESTS\n",
    "def cosine_similarity_test(target):\n",
    "    a = np.random.uniform(-10, 10, 10)\n",
    "    b = np.random.uniform(-10, 10, 10)\n",
    "    c = np.random.uniform(-1, 1, 23)\n",
    "        \n",
    "    assert np.isclose(cosine_similarity(a, a), 1), \"cosine_similarity(a, a) must be 1\"\n",
    "    assert np.isclose(cosine_similarity((c >= 0) * 1, (c < 0) * 1), 0), \"cosine_similarity(a, not(a)) must be 0\"\n",
    "    assert np.isclose(cosine_similarity(a, -a), -1), \"cosine_similarity(a, -a) must be -1\"\n",
    "    assert np.isclose(cosine_similarity(a, b), cosine_similarity(a * 2, b * 4)), \"cosine_similarity must be scale-independent. You must divide by the product of the norms of each input\"\n",
    "\n",
    "    print(\"\\033[92mAll test passed!\")\n",
    "    \n",
    "cosine_similarity_test(cosine_similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FX8fHEBxJWhQ"
   },
   "source": [
    "#### Try different words!\n",
    "\n",
    "After you get the correct expected output, please feel free to modify the inputs and measure the cosine similarity between other pairs of words! Playing around with the cosine similarity of other inputs will give you a better sense of how word vectors behave."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "63Pjp_QSJWhQ"
   },
   "source": [
    "<a name='4'></a>\n",
    "## 4 - Word Analogy Task\n",
    "\n",
    "* In the word analogy task, complete this sentence:  \n",
    "    <font color='brown'>\"*a* is to *b* as *c* is to **____**\"</font>. \n",
    "\n",
    "* An example is:  \n",
    "    <font color='brown'> '*man* is to *woman* as *king* is to *queen*' </font>. \n",
    "\n",
    "* You're trying to find a word *d*, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner:   \n",
    "    $e_b - e_a \\approx e_d - e_c$\n",
    "* Measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity. \n",
    "\n",
    "<a name='ex-2'></a>\n",
    "### Exercise 2 - complete_analogy\n",
    "\n",
    "Complete the code below to perform word analogies!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "kGBV3yoQJWhS"
   },
   "outputs": [],
   "source": [
    "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION: complete_analogy\n",
    "\n",
    "def complete_analogy(word_a, word_b, word_c, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Performs the word analogy task as explained above: a is to b as c is to ____. \n",
    "    \n",
    "    Arguments:\n",
    "    word_a -- a word, string\n",
    "    word_b -- a word, string\n",
    "    word_c -- a word, string\n",
    "    word_to_vec_map -- dictionary that maps words to their corresponding vectors. \n",
    "    \n",
    "    Returns:\n",
    "    best_word --  the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity\n",
    "    \"\"\"\n",
    "    \n",
    "    # convert words to lowercase\n",
    "    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Get the word embeddings e_a, e_b and e_c (≈1-3 lines)\n",
    "    e_a, e_b, e_c = word_to_vec_map[word_a], word_to_vec_map[word_b], word_to_vec_map[word_c]\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    words = word_to_vec_map.keys()\n",
    "    max_cosine_sim = -100              # Initialize max_cosine_sim to a large negative number\n",
    "    best_word = None                   # Initialize best_word with None, it will help keep track of the word to output\n",
    "    \n",
    "    # loop over the whole word vector set\n",
    "    for w in words:   \n",
    "        # to avoid best_word being one the input words, skip the input word_c\n",
    "        # skip word_c from query\n",
    "        if w == word_c:\n",
    "            continue\n",
    "        \n",
    "        ### START CODE HERE ###\n",
    "        # Compute cosine similarity between the vector (e_b - e_a) and the vector ((w's vector representation) - e_c)  (≈1 line)\n",
    "        cosine_sim = cosine_similarity(e_b - e_a, word_to_vec_map[w] - e_c)\n",
    "        \n",
    "        # If the cosine_sim is more than the max_cosine_sim seen so far,\n",
    "            # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)\n",
    "        if cosine_sim > max_cosine_sim:\n",
    "            max_cosine_sim = cosine_sim\n",
    "            best_word = w\n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "    return best_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# PUBLIC TEST\n",
    "def complete_analogy_test(target):\n",
    "    a = [3, 3] # Center at a\n",
    "    a_nw = [2, 4] # North-West oriented vector from a\n",
    "    a_s = [3, 2] # South oriented vector from a\n",
    "    \n",
    "    c = [-2, 1] # Center at c\n",
    "    # Create a controlled word to vec map\n",
    "    word_to_vec_map = {'a': a,\n",
    "                       'synonym_of_a': a,\n",
    "                       'a_nw': a_nw, \n",
    "                       'a_s': a_s, \n",
    "                       'c': c, \n",
    "                       'c_n': [-2, 2], # N\n",
    "                       'c_ne': [-1, 2], # NE\n",
    "                       'c_e': [-1, 1], # E\n",
    "                       'c_se': [-1, 0], # SE\n",
    "                       'c_s': [-2, 0], # S\n",
    "                       'c_sw': [-3, 0], # SW\n",
    "                       'c_w': [-3, 1], # W\n",
    "                       'c_nw': [-3, 2] # NW\n",
    "                      }\n",
    "    \n",
    "    # Convert lists to np.arrays\n",
    "    for key in word_to_vec_map.keys():\n",
    "        word_to_vec_map[key] = np.array(word_to_vec_map[key])\n",
    "            \n",
    "    assert(target('a', 'a_nw', 'c', word_to_vec_map) == 'c_nw')\n",
    "    assert(target('a', 'a_s', 'c', word_to_vec_map) == 'c_s')\n",
    "    assert(target('a', 'synonym_of_a', 'c', word_to_vec_map) != 'c'), \"Best word cannot be input query\"\n",
    "    assert(target('a', 'c', 'a', word_to_vec_map) == 'c')\n",
    "\n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "complete_analogy_test(complete_analogy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vnJJ_2sQJWhT"
   },
   "source": [
    "Run the cell below to test your code. Patience, young grasshopper...this may take 1-2 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "small -> smaller :: big -> competitors\n"
     ]
    }
   ],
   "source": [
    "# START SKIP FOR GRADING\n",
    "triads_to_try = [('small', 'smaller', 'big')]\n",
    "for triad in triads_to_try:\n",
    "    print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad, word_to_vec_map)))\n",
    "\n",
    "# END SKIP FOR GRADING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "41Ozkp_-JWhV"
   },
   "source": [
    "Once you get the output, try modifying the input cells above to test your own analogies. \n",
    "\n",
    "**Hint**: Try to find some other analogy pairs that will work, along with some others where the algorithm doesn't give the right answer:\n",
    "    * For example, you can try small->smaller as big->?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "\n",
    "You've come to the end of the graded portion of the assignment. By now, you've: \n",
    "\n",
    "* Loaded some pre-trained word vectors\n",
    "* Measured the similarity between word vectors using cosine similarity\n",
    "* Used word embeddings to solve word analogy problems such as Man is to Woman as King is to __.\n",
    "\n",
    "Cosine similarity is a relatively simple and intuitive, yet powerful, method you can use to capture nuanced relationships between words. These exercises should be helpful to you in explaining how it works, and applying it to your own projects!  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3nK_lD7mJWhV"
   },
   "source": [
    "<font color='blue'>\n",
    "    <b>What you should remember</b>:\n",
    "\n",
    "- Cosine similarity is a good way to compare the similarity between pairs of word vectors.\n",
    "    - Note that L2 (Euclidean) distance also works.\n",
    "- For NLP applications, using a pre-trained set of word vectors is often a great way to get started. </font>\n",
    "\n",
    "Even though you've finished the graded portion, please take a look at the rest of this notebook to learn about debiasing word vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HMRD25MuJWhW"
   },
   "source": [
    "<a name='5'></a>\n",
    "## 5 - Debiasing Word Vectors (OPTIONAL/UNGRADED) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Q1w3ZpEJWhW"
   },
   "source": [
    "In the following exercise, you'll examine gender biases that can be reflected in a word embedding, and explore algorithms for reducing the bias. In addition to learning about the topic of debiasing, this exercise will also help hone your intuition about what word vectors are doing. This section involves a bit of linear algebra, though you can certainly complete it without being an expert! Go ahead and give it a shot. This portion of the notebook is optional and is not graded...so just have fun and explore.  \n",
    "\n",
    "First, see how the GloVe word embeddings relate to gender. You'll begin by computing a vector $g = e_{woman}-e_{man}$, where $e_{woman}$ represents the word vector corresponding to the word *woman*, and $e_{man}$ corresponds to the word vector corresponding to the word *man*. The resulting vector $g$ roughly encodes the concept of \"gender\". \n",
    "\n",
    "You might get a more accurate representation if you compute $g_1 = e_{mother}-e_{father}$, $g_2 = e_{girl}-e_{boy}$, etc. and average over them, but just using $e_{woman}-e_{man}$ will give good enough results for now.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "_qpU-C3KJWhW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.087144    0.2182     -0.40986    -0.03922    -0.1032      0.94165\n",
      " -0.06042     0.32988     0.46144    -0.35962     0.31102    -0.86824\n",
      "  0.96006     0.01073     0.24337     0.08193    -1.02722    -0.21122\n",
      "  0.695044   -0.00222     0.29106     0.5053     -0.099454    0.40445\n",
      "  0.30181     0.1355     -0.0606     -0.07131    -0.19245    -0.06115\n",
      " -0.3204      0.07165    -0.13337    -0.25068714 -0.14293    -0.224957\n",
      " -0.149       0.048882    0.12191    -0.27362    -0.165476   -0.20426\n",
      "  0.54376    -0.271425   -0.10245    -0.32108     0.2516     -0.33455\n",
      " -0.04371     0.01258   ]\n"
     ]
    }
   ],
   "source": [
    "g = word_to_vec_map['woman'] - word_to_vec_map['man']\n",
    "print(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ORIvi0s1JWhX"
   },
   "source": [
    "Now, consider the cosine similarity of different words with $g$. What does a positive value of similarity mean, versus a negative cosine similarity? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "TgqV6pDxJWhX",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of names and their similarities with constructed vector:\n",
      "john -0.23163356145973724\n",
      "marie 0.31559793539607295\n",
      "sophie 0.31868789859418784\n",
      "ronaldo -0.31244796850329437\n",
      "priya 0.17632041839009405\n",
      "rahul -0.16915471039231722\n",
      "danielle 0.24393299216283895\n",
      "reza -0.07930429672199553\n",
      "katy 0.2831068659572615\n",
      "yasmin 0.23313857767928758\n"
     ]
    }
   ],
   "source": [
    "print ('List of names and their similarities with constructed vector:')\n",
    "\n",
    "# girls and boys name\n",
    "name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']\n",
    "\n",
    "for w in name_list:\n",
    "    print (w, cosine_similarity(word_to_vec_map[w], g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YLELp0LJJWhY"
   },
   "source": [
    "As you can see, female first names tend to have a positive cosine similarity with our constructed vector $g$, while male first names tend to have a negative cosine similarity. This is not surprising, and the result seems acceptable. \n",
    "\n",
    "Now try with some other words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "wgadfCaGJWhY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Other words and their similarities:\n",
      "lipstick 0.2769191625638267\n",
      "guns -0.1888485567898898\n",
      "science -0.060829065409296994\n",
      "arts 0.008189312385880347\n",
      "literature 0.06472504433459929\n",
      "warrior -0.20920164641125288\n",
      "doctor 0.11895289410935038\n",
      "tree -0.0708939917547809\n",
      "receptionist 0.3307794175059375\n",
      "technology -0.13193732447554293\n",
      "fashion 0.03563894625772699\n",
      "teacher 0.1792092343182567\n",
      "engineer -0.0803928049452407\n",
      "pilot 0.001076449899191721\n",
      "computer -0.10330358873850495\n",
      "singer 0.1850051813649629\n"
     ]
    }
   ],
   "source": [
    "print('Other words and their similarities:')\n",
    "word_list = ['lipstick', 'guns', 'science', 'arts', 'literature', 'warrior','doctor', 'tree', 'receptionist', \n",
    "             'technology',  'fashion', 'teacher', 'engineer', 'pilot', 'computer', 'singer']\n",
    "for w in word_list:\n",
    "    print (w, cosine_similarity(word_to_vec_map[w], g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YUzfNtq4JWhY"
   },
   "source": [
    "Do you notice anything surprising? It is astonishing how these results reflect certain unhealthy gender stereotypes. For example, we see “computer” is negative and is closer in value to male first names, while “literature” is positive and is closer to female first names. Ouch! \n",
    "\n",
    "You'll see below how to reduce the bias of these vectors, using an algorithm due to [Boliukbasi et al., 2016](https://arxiv.org/abs/1607.06520). Note that some word pairs such as \"actor\"/\"actress\" or \"grandmother\"/\"grandfather\" should remain gender-specific, while other words such as \"receptionist\" or \"technology\" should be neutralized, i.e. not be gender-related. You'll have to treat these two types of words differently when debiasing.\n",
    "\n",
    "<a name='5-1'></a>\n",
    "### 5.1 - Neutralize Bias for Non-Gender Specific Words \n",
    "\n",
    "The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which is called $g_{\\perp}$ here. In linear algebra, we say that the 49-dimensional $g_{\\perp}$ is perpendicular (or \"orthogonal\") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$. \n",
    "\n",
    "Even though $g_{\\perp}$ is 49-dimensional, given the limitations of what you can draw on a 2D screen, it's illustrated using a 1-dimensional axis below. \n",
    "\n",
    "<img src=\"images/neutral.png\" style=\"width:800px;height:300px;\">\n",
    "<caption><center><font color='purple'><b>Figure 2</b>: The word vector for \"receptionist\" represented before and after applying the neutralize operation.</font> </center></caption>\n",
    "\n",
    "<a name='ex-3'></a>\n",
    "### Exercise 3 - neutralize\n",
    "\n",
    "Implement `neutralize()` to remove the bias of words such as \"receptionist\" or \"scientist.\"\n",
    "\n",
    "Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$: \n",
    "\n",
    "$$e^{bias\\_component} = \\frac{e \\cdot g}{||g||_2^2} * g\\tag{2}$$\n",
    "$$e^{debiased} = e - e^{bias\\_component}\\tag{3}$$\n",
    "\n",
    "If you are an expert in linear algebra, you may recognize $e^{bias\\_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert in linear algebra, don't worry about this. ;) \n",
    "\n",
    "<!-- \n",
    "**Reminder**: a vector $u$ can be split into two parts: its projection over a vector-axis $v_B$ and its projection over the axis orthogonal to $v$:\n",
    "$$u = u_B + u_{\\perp}$$\n",
    "where : $u_B = $ and $ u_{\\perp} = u - u_B $\n",
    "!--> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "79Pk0QDhJWhZ"
   },
   "outputs": [],
   "source": [
    "def neutralize(word, g, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Removes the bias of \"word\" by projecting it on the space orthogonal to the bias axis. \n",
    "    This function ensures that gender neutral words are zero in the gender subspace.\n",
    "    \n",
    "    Arguments:\n",
    "        word -- string indicating the word to debias\n",
    "        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)\n",
    "        word_to_vec_map -- dictionary mapping words to their corresponding vectors.\n",
    "    \n",
    "    Returns:\n",
    "        e_debiased -- neutralized word vector representation of the input \"word\"\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Select word vector representation of \"word\". Use word_to_vec_map. (≈ 1 line)\n",
    "    e = word_to_vec_map[word]\n",
    "    \n",
    "    # Compute e_biascomponent using the formula given above. (≈ 1 line)\n",
    "    dot = e * g\n",
    "    e_biascomponent = np.sum(dot) / (np.sum(g**2)) * g\n",
    "\n",
    "    # Neutralize e by subtracting e_biascomponent from it \n",
    "    # e_debiased should be equal to its orthogonal projection. (≈ 1 line)\n",
    "    e_debiased = e - e_biascomponent\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return e_debiased"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "6PgkxwxXJWhZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine similarity between receptionist and g, before neutralizing:  0.3307794175059375\n",
      "cosine similarity between receptionist and g, after neutralizing:  -9.692463026316581e-17\n"
     ]
    }
   ],
   "source": [
    "e = \"receptionist\"\n",
    "print(\"cosine similarity between \" + e + \" and g, before neutralizing: \", cosine_similarity(word_to_vec_map[\"receptionist\"], g))\n",
    "\n",
    "e_debiased = neutralize(\"receptionist\", g, word_to_vec_map)\n",
    "print(\"cosine similarity between \" + e + \" and g, after neutralizing: \", cosine_similarity(e_debiased, g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Da--RZGIJWha"
   },
   "source": [
    "**Expected Output**: The second result is essentially 0, up to numerical rounding (on the order of $10^{-17}$).\n",
    "\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <b>cosine similarity between receptionist and g, before neutralizing:</b> :\n",
    "        </td>\n",
    "        <td>\n",
    "         0.330779417506\n",
    "        </td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>\n",
    "            <b>cosine similarity between receptionist and g, after neutralizing</b> :\n",
    "        </td>\n",
    "        <td>\n",
    "         -4.442232511624783e-17\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ciATgxwLJWha"
   },
   "source": [
    "<a name='5-2'></a>\n",
    "### 5.2 - Equalization Algorithm for Gender-Specific Words\n",
    "\n",
    "Next, let's see how debiasing can also be applied to word pairs such as \"actress\" and \"actor.\" Equalization is applied to pairs of words that you might want to have differ only through the gender property. As a concrete example, suppose that \"actress\" is closer to \"babysit\" than \"actor.\" By applying neutralization to \"babysit,\" you can reduce the gender stereotype associated with babysitting. But this still does not guarantee that \"actor\" and \"actress\" are equidistant from \"babysit.\" The equalization algorithm takes care of this. \n",
    "\n",
    "The key idea behind equalization is to make sure that a particular pair of words are equidistant from the 49-dimensional $g_\\perp$. The equalization step also ensures that the two equalized steps are now the same distance from $e_{receptionist}^{debiased}$, or from any other work that has been neutralized. Visually, this is how equalization works: \n",
    "\n",
    "<img src=\"images/equalize10.png\" style=\"width:800px;height:400px;\">\n",
    "\n",
    "\n",
    "The derivation of the linear algebra to do this is a bit more complex. (See Bolukbasi et al., 2016 in the References for details.) Here are the key equations: \n",
    "\n",
    "\n",
    "$$ \\mu = \\frac{e_{w1} + e_{w2}}{2}\\tag{4}$$ \n",
    "\n",
    "$$ \\mu_{B} = \\frac {\\mu \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{5}$$ \n",
    "\n",
    "$$\\mu_{\\perp} = \\mu - \\mu_{B} \\tag{6}$$\n",
    "\n",
    "$$ e_{w1B} = \\frac {e_{w1} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{7}$$ \n",
    "$$ e_{w2B} = \\frac {e_{w2} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{8}$$\n",
    "\n",
    "\n",
    "$$e_{w1B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w1B}} - \\mu_B} {||(e_{w1} - \\mu_{\\perp}) - \\mu_B||_2} \\tag{9}$$\n",
    "\n",
    "\n",
    "$$e_{w2B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w2B}} - \\mu_B} {||(e_{w2} - \\mu_{\\perp}) - \\mu_B||_2} \\tag{10}$$\n",
    "\n",
    "$$e_1 = e_{w1B}^{corrected} + \\mu_{\\perp} \\tag{11}$$\n",
    "$$e_2 = e_{w2B}^{corrected} + \\mu_{\\perp} \\tag{12}$$\n",
    "\n",
    "\n",
    "<a name='ex-4'></a>\n",
    "### Exercise 4 - equalize\n",
    "\n",
    "Implement the `equalize()` function below. \n",
    "\n",
    "Use the equations above to get the final equalized version of the pair of words. Good luck!\n",
    "\n",
    "**Hint**\n",
    "- Use [np.linalg.norm](https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "aBhxJtGIJWha"
   },
   "outputs": [],
   "source": [
    "def equalize(pair, bias_axis, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Debias gender specific words by following the equalize method described in the figure above.\n",
    "    \n",
    "    Arguments:\n",
    "    pair -- pair of strings of gender specific words to debias, e.g. (\"actress\", \"actor\") \n",
    "    bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender\n",
    "    word_to_vec_map -- dictionary mapping words to their corresponding vectors\n",
    "    \n",
    "    Returns\n",
    "    e_1 -- word vector corresponding to the first word\n",
    "    e_2 -- word vector corresponding to the second word\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Step 1: Select word vector representation of \"word\". Use word_to_vec_map. (≈ 2 lines)\n",
    "    w1, w2 = pair\n",
    "    e_w1, e_w2 = word_to_vec_map[w1], word_to_vec_map[w2]\n",
    "    \n",
    "    # Step 2: Compute the mean of e_w1 and e_w2 (≈ 1 line)\n",
    "    mu = (e_w1 + e_w2) / 2\n",
    "    \n",
    "    # Step 3: Compute the projections of mu over the bias axis and the orthogonal axis (≈ 2 lines)\n",
    "    mu_B = np.sum(mu * bias_axis) / (np.sum(bias_axis**2)) * bias_axis\n",
    "    mu_orth = mu - mu_B\n",
    "    \n",
    "    # Step 4: Use equations (7) and (8) to compute e_w1B and e_w2B (≈2 lines)\n",
    "    e_w1B = np.sum(e_w1 * bias_axis) / (np.sum(bias_axis**2)) * bias_axis\n",
    "    e_w2B = np.sum(e_w2 * bias_axis) / (np.sum(bias_axis**2)) * bias_axis\n",
    "    \n",
    "    # Step 5: Adjust the Bias part of e_w1B and e_w2B using the formulas (9) and (10) given above (≈2 lines)\n",
    "    corrected_e_w1B = np.sqrt(np.abs(1 - np.sum(mu_orth**2))) * (e_w1B - mu_B) / np.sqrt(np.sum((e_w1 - mu_orth - mu_B)**2))\n",
    "    corrected_e_w2B = np.sqrt(np.abs(1 - np.sum(mu_orth**2))) * (e_w2B - mu_B) / np.sqrt(np.sum((e_w2 - mu_orth - mu_B)**2))\n",
    "    \n",
    "    # Step 6: Debias by equalizing e1 and e2 to the sum of their corrected projections (≈2 lines)\n",
    "    e1 = corrected_e_w1B + mu_orth\n",
    "    e2 = corrected_e_w2B + mu_orth\n",
    "                                                                \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return e1, e2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "P405J5ZSJWhb",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine similarities before equalizing:\n",
      "cosine_similarity(word_to_vec_map[\"man\"], gender) =  -0.11711095765336832\n",
      "cosine_similarity(word_to_vec_map[\"woman\"], gender) =  0.35666618846270376\n",
      "\n",
      "cosine similarities after equalizing:\n",
      "cosine_similarity(e1, gender) =  -0.7004364289309388\n",
      "cosine_similarity(e2, gender) =  0.7004364289309387\n"
     ]
    }
   ],
   "source": [
    "print(\"cosine similarities before equalizing:\")\n",
    "print(\"cosine_similarity(word_to_vec_map[\\\"man\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"man\"], g))\n",
    "print(\"cosine_similarity(word_to_vec_map[\\\"woman\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"woman\"], g))\n",
    "print()\n",
    "e1, e2 = equalize((\"man\", \"woman\"), g, word_to_vec_map)\n",
    "print(\"cosine similarities after equalizing:\")\n",
    "print(\"cosine_similarity(e1, gender) = \", cosine_similarity(e1, g))\n",
    "print(\"cosine_similarity(e2, gender) = \", cosine_similarity(e2, g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5cNhiOEaJWhb"
   },
   "source": [
    "**Expected Output**:\n",
    "\n",
    "cosine similarities before equalizing:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <b>cosine_similarity(word_to_vec_map[\"man\"], gender)</b> =\n",
    "        </td>\n",
    "        <td>\n",
    "         -0.117110957653\n",
    "        </td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>\n",
    "            <b>cosine_similarity(word_to_vec_map[\"woman\"], gender)</b> =\n",
    "        </td>\n",
    "        <td>\n",
    "         0.356666188463\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "cosine similarities after equalizing:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <b>cosine_similarity(e1, gender)</b> =\n",
    "        </td>\n",
    "        <td>\n",
    "         -0.7004364289309388\n",
    "        </td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>\n",
    "            <b>cosine_similarity(e2, gender)</b> =\n",
    "        </td>\n",
    "        <td>\n",
    "         0.7004364289309387\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mXi96ToSJWhc"
   },
   "source": [
    "Go ahead and play with the input words in the cell above, to apply equalization to other pairs of words. \n",
    "\n",
    "Hint: Try...\n",
    "\n",
    "These debiasing algorithms are very helpful for reducing bias, but aren't perfect and don't eliminate all traces of bias. For example, one weakness of this implementation was that the bias direction $g$ was defined using only the pair of words _woman_ and _man_. As discussed earlier, if $g$ were defined by computing $g_1 = e_{woman} - e_{man}$; $g_2 = e_{mother} - e_{father}$; $g_3 = e_{girl} - e_{boy}$; and so on and averaging over them, you would obtain a better estimate of the \"gender\" dimension in the 50 dimensional word embedding space. Feel free to play with these types of variants as well! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DDjEdbzuJWhd"
   },
   "source": [
    "### Congratulations!\n",
    "\n",
    "You have come to the end of both graded and ungraded portions of this notebook, and have seen several of the ways that word vectors can be applied and  modified. Great work pushing your knowledge in the areas of neutralizing and equalizing word vectors! See you next time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FY2ZhorNJWhd"
   },
   "source": [
    "<a name='6'></a>\n",
    "## 6 - References\n",
    "\n",
    "- The debiasing algorithm is from Bolukbasi et al., 2016, [Man is to Computer Programmer as Woman is to\n",
    "Homemaker? Debiasing Word Embeddings](https://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)\n",
    "- The GloVe word embeddings were due to Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (https://nlp.stanford.edu/projects/glove/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "schema_names": [
    "DLSC5W2-A1"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}