experiments/vq-layer/003-embedding-space-training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VQ Layer Embedding Space $\\alpha$-Training Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding space of a VQ layer can be trained. This notebook evaluates whether (1) vectors in the embeddings pace are moving into the direction of the points that are being projected onto them and (2) vectors remain in-place if they are not being used at all (opposed to $k$-means clustering).\n",
    "\n",
    "We call this kind of training $\\alpha$-training because vectors are being affected by three kinds of losses and this one has the coefficient $\\alpha$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vector_quantization(x: tf.Tensor, n, alpha=0.1, lookup_ord='euclidean', embedding_initializer=tf.random_normal_initializer) -> tf.Tensor:\n",
    "    \"\"\"Simplified version of the function in vq-layer-function, without losses beta and gamma and gradient skipping.\"\"\"\n",
    "    # shape of x is [batch, , q], where this function quantizes along dimension q\n",
    "    vec_size = x.shape[2]\n",
    "    with tf.variable_scope('vq'):\n",
    "        # embedding space\n",
    "        emb_space = tf.get_variable('emb_space', shape=[n, vec_size], dtype=x.dtype, initializer=embedding_initializer, trainable=True)\n",
    "        \n",
    "        # map x to y, where y is the vector from emb_space that is closest to x\n",
    "        diff = tf.expand_dims(x, axis=2) - emb_space  # distance of x from all vectors in the embedding space\n",
    "        dist = tf.norm(diff, lookup_ord, axis=3)  # distance between x and all vectors in emb\n",
    "        emb_index = tf.argmin(dist, axis=2)\n",
    "        y = tf.gather(emb_space, emb_index, axis=0)\n",
    "\n",
    "        # closest embedding update loss\n",
    "        nearest_loss = tf.reduce_mean(alpha * tf.norm(y - x, lookup_ord, axis=2), axis=[0, 1])\n",
    "        tf.add_to_collection(tf.GraphKeys.LOSSES, nearest_loss)\n",
    "        \n",
    "        return y, emb_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x10ecaf630>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEMlJREFUeJzt3X+s3XV9x/Hna23RZjqa2LsM2mJZVpsxZIPdIIvJZEHTaiYl0biSTcWgJC5Ml5lG2BZ0+Jdr4hImjjVK/JFN7Bxprq6kM8pCsg3GxQoIrOYO5+jFhCtSdPFOWvLeH+dUb6+3Pee2595zz6fPR3LT7/fzfXO+7w/f9nW/9/v9nnNTVUiS2vJzw25AkjR4hrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQauHteP169fX5s2bh7V7SRpJDz300PeqaqxX3dDCffPmzUxOTg5r95I0kpJ8p586L8tIUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSgns+5J7kT+F3gmaq6eIHtvw98EAjwQ+C9VfXwoBtt0b6D0+w+cIinj8xy/rq17Nq2lWsu3TDstiQ1oJ8z908D20+x/dvA66rq1cBHgD0D6Kt5+w5Oc/PdjzJ9ZJYCpo/McvPdj7Lv4PSwW5PUgJ7hXlX3Ad8/xfZ/q6rnuqv3AxsH1FvTdh84xOzRF08Ymz36IrsPHBpSR5JaMuhr7tcD95xsY5IbkkwmmZyZmRnwrkfL00dmFzUuSYsxsHBP8jt0wv2DJ6upqj1VNV5V42NjPT/3pmnnr1u7qHFJWoyBhHuSS4BPAjuq6tlBvGbrdm3byto1q04YW7tmFbu2bR1SR5JacsafCpnkAuBu4O1V9a0zb+nscPypGJ+WkbQU+nkU8vPAlcD6JIeBDwFrAKrqDuAW4BXAJ5IAHKuq8aVquCXXXLrBMJe0JHqGe1Vd22P7u4F3D6wjSdIZ8x2qktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDWoZ7gnuTPJM0m+eZLtSXJbkqkkjyS5bPBtSpIWY3UfNZ8GPg589iTb3whs6X69Bvib7p86S+w7OM3uA4d4+sgs569by65tW7nm0g3Dbks6q/U8c6+q+4Dvn6JkB/DZ6rgfWJfkvEE1qJVt38Fpbr77UaaPzFLA9JFZbr77UfYdnB52a9JZbRDX3DcAT81ZP9wd01lg94FDzB598YSx2aMvsvvAoSF1JAmW+YZqkhuSTCaZnJmZWc5da4k8fWR2UeOSlscgwn0a2DRnfWN37GdU1Z6qGq+q8bGxsQHsWsN2/rq1ixqXtDwGEe4TwDu6T81cATxfVd8dwOtqBOzatpW1a1adMLZ2zSp2bds6pI4kQR9PyyT5PHAlsD7JYeBDwBqAqroD2A+8CZgCfgS8a6ma1cpz/KkYn5aRVpZU1VB2PD4+XpOTk0PZtySNqiQPVdV4rzrfoSpJDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg/oK9yTbkxxKMpXkpgW2X5Dk3iQHkzyS5E2Db1WS1K+e4Z5kFXA78EbgIuDaJBfNK/tzYG9VXQrsBD4x6EYlSf3r58z9cmCqqp6sqheAu4Ad82oK+IXu8rnA04NrUZK0WKv7qNkAPDVn/TDwmnk1Hwb+OckfAT8PvH4g3UmSTsugbqheC3y6qjYCbwI+l+RnXjvJDUkmk0zOzMwMaNeSpPn6CfdpYNOc9Y3dsbmuB/YCVNW/Ay8F1s9/oaraU1XjVTU+NjZ2eh1LknrqJ9wfBLYkuTDJOXRumE7Mq/kf4CqAJL9KJ9w9NZekIekZ7lV1DLgROAA8QeepmMeS3Jrk6m7ZB4D3JHkY+DxwXVXVUjUtSTq1fm6oUlX7gf3zxm6Zs/w48NrBtiZJOl2+Q1WSGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBvUV7km2JzmUZCrJTSepeVuSx5M8luTvB9umJGkxVvcqSLIKuB14A3AYeDDJRFU9PqdmC3Az8Nqqei7JLy5Vw5Kk3vo5c78cmKqqJ6vqBeAuYMe8mvcAt1fVcwBV9cxg25QkLUY/4b4BeGrO+uHu2FyvAl6V5F+T3J9k+6AalCQtXs/LMot4nS3AlcBG4L4kr66qI3OLktwA3ABwwQUXDGjXkqT5+jlznwY2zVnf2B2b6zAwUVVHq+rbwLfohP0JqmpPVY1X1fjY2Njp9ixJ6qGfcH8Q2JLkwiTnADuBiXk1++ictZNkPZ3LNE8OsE9J0iL0DPeqOgbcCBwAngD2VtVjSW5NcnW37ADwbJLHgXuBXVX17FI1LUk6tVTVUHY8Pj5ek5OTQ9m3JI2qJA9V1XivOt+hKkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S616JG98FcXw4fXdf58ZO+wO9IyH5NBfSqkpJXikb3wpffB0dnO+vNPddYBLnnb8Po6mw3hmHjmLrXmq7f+NESOOzrbGddwDOGYGO5Sa54/vLhxLb0hHBPDXWrNuRsXN66lN4RjYrhLrbnqFliz9sSxNWs74xqOIRwTw11qzSVvgzffBuduAtL58823eTN1mIZwTPw8d0kaIX6euySdxQx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIa1Fe4J9me5FCSqSQ3naLuLUkqSc8H7CVJS6dnuCdZBdwOvBG4CLg2yUUL1L0ceD/wwKCblCQtTj9n7pcDU1X1ZFW9ANwF7Fig7iPAR4H/G2B/kqTT0E+4bwCemrN+uDv2E0kuAzZV1T+d6oWS3JBkMsnkzMzMopuVJPXnjG+oJvk54GPAB3rVVtWeqhqvqvGxsbEz3bUk6ST6CfdpYNOc9Y3dseNeDlwM/EuS/wauACa8qSpJw9NPuD8IbElyYZJzgJ3AxPGNVfV8Va2vqs1VtRm4H7i6qvw8X0kakp7hXlXHgBuBA8ATwN6qeizJrUmuXuoGJUmLt7qfoqraD+yfN7bg74eqqivPvC1J0pnwHaqS1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNaivcE+yPcmhJFNJblpg+58keTzJI0m+muSVg29VktSvnuGeZBVwO/BG4CLg2iQXzSs7CIxX1SXAF4G/HHSjkqT+9XPmfjkwVVVPVtULwF3AjrkFVXVvVf2ou3o/sHGwbUqSFqOfcN8APDVn/XB37GSuB+5ZaEOSG5JMJpmcmZnpv0tJ0qIM9IZqkj8AxoHdC22vqj1VNV5V42NjY4PctSRpjtV91EwDm+asb+yOnSDJ64E/A15XVT8eTHuSpNPRz5n7g8CWJBcmOQfYCUzMLUhyKfC3wNVV9czg25QkLUbPcK+qY8CNwAHgCWBvVT2W5NYkV3fLdgMvA/4hyTeSTJzk5SRJy6CfyzJU1X5g/7yxW+Ysv37AfUmSzoDvUJWkBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQX2Fe5LtSQ4lmUpy0wLbX5LkC93tDyTZPOhGJUn9W92rIMkq4HbgDcBh4MEkE1X1+Jyy64HnqupXkuwEPgr83lI0vO/gNLsPHOLpI7Ocv24tu7Zt5ZpLNyzFriRpZPVz5n45MFVVT1bVC8BdwI55NTuAz3SXvwhclSSDa7Nj38Fpbr77UaaPzFLA9JFZbr77UfYdnB70riRppPUT7huAp+asH+6OLVhTVceA54FXDKLBuXYfOMTs0RdPGJs9+iK7Dxwa9K4kaaQt6w3VJDckmUwyOTMzs+j//ukjs4sal6SzVT/hPg1smrO+sTu2YE2S1cC5wLPzX6iq9lTVeFWNj42NLbrZ89etXdS4JJ2t+gn3B4EtSS5Mcg6wE5iYVzMBvLO7/Fbga1VVg2uzY9e2raxds+qEsbVrVrFr29ZB70qSRlrPp2Wq6liSG4EDwCrgzqp6LMmtwGRVTQCfAj6XZAr4Pp1vAAN3/KkYn5aRpFPLEpxg92V8fLwmJyeHsm9JGlVJHqqq8V51vkNVkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGDe059yQzwHfO4CXWA98bUDvD1spcWpkHtDMX57HynOlcXllVPT+/ZWjhfqaSTPbzIP8oaGUurcwD2pmL81h5lmsuXpaRpAYZ7pLUoFEO9z3DbmCAWplLK/OAdubiPFaeZZnLyF5zlySd3CifuUuSTmLFh3uS7UkOJZlKctMC21+S5Avd7Q8k2bz8XfbWxzyuSzKT5Bvdr3cPo89ektyZ5Jkk3zzJ9iS5rTvPR5Jcttw99quPuVyZ5Pk5x+SW5e6xH0k2Jbk3yeNJHkvy/gVqVvxx6XMeo3JMXprkP5I83J3LXyxQs7TZVVUr9ovOLwf5L+CXgXOAh4GL5tX8IXBHd3kn8IVh932a87gO+Piwe+1jLr8NXAZ88yTb3wTcAwS4Anhg2D2fwVyuBL487D77mMd5wGXd5ZcD31rg79eKPy59zmNUjkmAl3WX1wAPAFfMq1nS7FrpZ+6XA1NV9WRVvQDcBeyYV7MD+Ex3+YvAVUmyjD32o595jISquo/Ob9s6mR3AZ6vjfmBdkvOWp7vF6WMuI6GqvltVX+8u/xB4Apj/68lW/HHpcx4jofv/+X+7q2u6X/NvcC5pdq30cN8APDVn/TA/e7B/UlNVx4DngVcsS3f962ceAG/p/sj8xSSbFtg+Cvqd66j4re6P1vck+bVhN9NL90f7S+mcKc41UsflFPOAETkmSVYl+QbwDPCVqjrpMVmK7Frp4X42+RKwuaouAb7CT7+ja3i+Tuet3r8O/DWwb8j9nFKSlwH/CPxxVf1g2P2crh7zGJljUlUvVtVvABuBy5NcvJz7X+nhPg3MPYPd2B1bsCbJauBc4Nll6a5/PedRVc9W1Y+7q58EfnOZehu0fo7ZSKiqHxz/0bqq9gNrkqwfclsLSrKGTiD+XVXdvUDJSByXXvMYpWNyXFUdAe4Fts/btKTZtdLD/UFgS5ILk5xD56bDxLyaCeCd3eW3Al+r7h2KFaTnPOZd/7yazvXGUTQBvKP7dMYVwPNV9d1hN3U6kvzS8WugSS6n8+9lpZ040O3xU8ATVfWxk5St+OPSzzxG6JiMJVnXXV4LvAH4z3llS5pdqwf1Qkuhqo4luRE4QOeJkzur6rEktwKTVTVB5y/D55JM0bk5tnN4HS+sz3m8L8nVwDE687huaA2fQpLP03liYX2Sw8CH6NwsoqruAPbTeTJjCvgR8K7hdNpbH3N5K/DeJMeAWWDnCjxxAHgt8Hbg0e41XoA/BS6AkTou/cxjVI7JecBnkqyi8w1ob1V9eTmzy3eoSlKDVvplGUnSaTDcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lq0P8DDCvTL+/in8YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "points_val = np.array([[0, 0], [1, 1], [0.5, 1.2]], dtype=np.float32)\n",
    "emb_space_val = np.array([[2, 0.5], [3, 0.5]], dtype=np.float32)\n",
    "\n",
    "plt.scatter(points_val[:, 0], points_val[:, 1])\n",
    "plt.scatter(emb_space_val[:, 0], emb_space_val[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The step size $\\alpha$ determines how quickly the embedding space is being adjusted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "points = tf.placeholder(tf.float32, shape=[None, 2])\n",
    "vq_in = tf.expand_dims(points, axis=1)\n",
    "vq_out, emb_space = vector_quantization(vq_in, len(emb_space_val), alpha, lookup_ord=2, embedding_initializer=tf.constant_initializer(emb_space_val))\n",
    "points_out = tf.squeeze(vq_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "with tf.control_dependencies(update_ops):\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=1)\n",
    "    loss = tf.add_n(tf.get_collection(tf.GraphKeys.LOSSES))\n",
    "    train_op = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGwRJREFUeJzt3X90VeWd7/H3Nz9IMoDJgsSVEGAARQq1ucWbBf643MrQCtWLuKrj4Nix7dLLXXOXFTvKjFynlKGrq52hrWLrjEOrbe1ypFZzMQxonIv06vUHGkSDiFFEWgnJIkATAZOQH9/7xzmJOeEkOYHzc+fzWovF2c9+OPv7uOOHzbOfs4+5OyIiEixZqS5ARETiT+EuIhJACncRkQBSuIuIBJDCXUQkgBTuIiIBpHAXEQkghbuISAAp3EVEAignVQcuLi72adOmperwIiIZadeuXUfdvWS4fikL92nTplFbW5uqw4uIZCQz+30s/TQtIyISQAp3EZEAUriLiASQwl1EJIAU7iIiAaRwFxEJoGGXQprZI8B/A464+8VR9t8M/B1gwAngr939rXgXGlSbdzewvqaewy1tTCoqYNXiWVw3tzzVZYlIhovlyv2XwJIh9n8IfMHdPwd8F9gYh7pGhc27G1hdtYeGljYcaGhpY3XVHjbvbkh1aSKS4YYNd3d/ATg+xP6X3f2P4c1Xgclxqi3w1tfU09bZHdHW1tnN+pr6FFUkIkER7zn3W4FnBttpZivMrNbMapubm+N86MxzuKVtRO0iIrGKW7ib2UJC4f53g/Vx943uXunulSUlwz4aIfAmFRWMqF1EJFZxCXczqwB+Dixz92PxeM/RYNXiWRTkZke0FeRms2rxrBRVJCJBcc4PDjOzqUAV8Ffu/t65lzR69K6K0WoZEYm3WJZCPg5cCRSb2SHgO0AugLs/BKwBJgL/bGYAXe5emaiCg+a6ueUKcxGJu2HD3d1vGmb/bcBtcatIRETOmT6hKiISQAp3EZEAUriLiASQwl1EJIAU7iIiAaRwFxEJIIW7iEgAKdxFRAJI4S4iEkAKdxGRAFK4i4gEkMJdRCSAFO4iIgGkcBcRCSCFu4hIACncRUQCSOEuIhJACncRkQBSuIuIBJDCXUQkgBTuIiIBpHAXEQkghbuISAANG+5m9oiZHTGztwfZb2b2gJntN7M6M7sk/mWKiMhI5MTQ55fAT4FHB9n/ZWBm+Nd84F/Cv8sosnl3A+tr6jnc0sakogJWLZ7FdXPLU12WyKg17JW7u78AHB+iyzLgUQ95FSgys7J4FSjpb/PuBlZX7aGhpQ0HGlraWF21h827G1JdmsioFY8593Lgo37bh8JtMkqsr6mnrbM7oq2ts5v1NfUpqkhEknpD1cxWmFmtmdU2Nzcn89CSQIdb2kbULiKJF49wbwCm9NueHG47g7tvdPdKd68sKSmJw6ElHUwqKhhRu4gkXjzCvRq4Jbxq5lKg1d0b4/C+MgJPNR3nc797ldLnd/PZ559jTc3/YMdDf8+p3UcSfuxVi2dRkJsd0VaQm82qxbMSfmwRiW7Y1TJm9jhwJVBsZoeA7wC5AO7+ELANuBrYD3wCfCNRxUp0TzUd52/2fUgH+WBwjPP5Ze43yLrwX+l54Uec/OeLKDryJp+/7SoKly6N+/F7V8VotYxI+jB3T8mBKysrvba2NiXHDprKl/dyqKPzjPZiP8L69lXM2P49ao5+wOzf15JbsYsrb1sLFTcmvU4ROXdmtsvdK4frp0+oBkBDlGAHOEoxuXkn+ZOc8+g8/Sq/n3Yt2a86Vc/exR1r7+XiJ3byVNNQq1xFJFMp3AOgPC83ansxR+nsGMcnXR9Dzwk68iYw4WN4aMI4/rZnEyfePMa3/k8d33/hThqbnk5y1SKSSAr3AFg9o4w8IteZj/F2buh5nDHvL6Huj/8XssaT13GcY+dBU042k+wY1uOwv53Huq7m7bf/ltde+0mKRiAi8aZwD4DrSyfw49nTKbF2cGeiH+Frnb/g8j0TOf5WD3849T65Yy5l6sFq/u1Ko7Srm8M+EQBr7+YoxWRldXH06M+oq6tL8WhEJB5iebaMZIDrSydwfemlAOx7cQc7/q2NPScawMZTkHsZ5U21VM3bxa7ZxurmE/xT11cB8PxsijkKwJi8UzxXtY0LuksZO/f8lI1FRM6dwj2AZi9YyOwFCwHYemArG179Pk2nWyjtcr7Z3MXvWr5Kdc9/wbMMLsznRn4BQEfHWE56Oy1V7wMo4EUymMI94K6ZcQ3XzLgGCD3g6x+2vcMfe07Tk59FwYVw86RfcAX/j+7ubA5++HnGeT7e2UPD47voPpnf95eEiGQWhfsoct3c8r4PFr322k84evRnjPFTtHeM5eCHn+f4kQtZ0DUDgAIbS9XGDQAKeJEMpHAfpebN+yZ1dV/guaptnPR2xnk+C7pmcGFP6GnNn3R9TNfpDl7c9KjCXSQDKdxHsYqKCi7oLqWl6n28s6evvaunM7R8Ejhx7GiqyhORc6ClkKPc2LnnU/SVmXzScxJ351RnK68ffYY/nNoHQFb2eN7b2ZTiKkVkpHTlLoydez7dJ/Op2riBrtMd/fbkkDXmCnY89i4AF80vTU2BIjJiunIXIHTT9KoVt5OVc16oIWs8OX/yJXLyZtN1uodXnv4gtQWKyIjoyl36zF6wkOcfi/6U0JPHO6K2i0h60pW7RBg3IS9qe35nC/tmz+H9P1tE65YtSa5KREZK4S4RLlt2ATljIn8ssrpPM+P9/w3udB0+TOO31yjgRdKcwl0iXDS/lIU3f6bvCj6/s4XP1D9G6ZFPv1jF29s5ct/9qSpRRGKgOXc5w0XzS/tWxuybPQeifFtXV6O+JlcknenKXYaUU1Y2onYRSQ8KdxnS+d+6E8vPj2jrycvlkcvbqfhVBVc9eRVbD2xNUXUiMhhNy8iQCpcuBeDIfffT1dhIZ0khP7+8jR0zPwag8VQja19eC9D39EkRST3zKPOpyVBZWem1tbXDd5S0ctWTV9F46sz59rKxZTx3w3MpqEhkdDGzXe5eOVw/TcvIiDSdiv6cmcHaRSQ1FO4yIqVjoz9fZrB2EUmNmMLdzJaYWb2Z7Teze6Lsn2pmO8xst5nVmdnV8S9V0sHKS1aSnx15gzU/O5+VxfPhvothbVHo97onUlShiEAMN1TNLBt4EPgScAh43cyq3f2dft3+HnjC3f/FzOYA24BpCahXUqz3pumGNzbQdKqJ0rGlrCyezzUv/Qw620KdWj+CLXeEXlfcmKJKRUa3WFbLzAP2u/sBADPbBCwD+oe7A+HHCVIIHI5nkZJe+n8vKxC6Uu8N9l6dbbB9ncJdJEViCfdy4KN+24eA+QP6rAWeM7NvAmOBL8alOskMrYdG1i4iCRevG6o3Ab9098nA1cCvzeyM9zazFWZWa2a1zc3NcTq0pFzh5JG1i0jCxRLuDcCUftuTw2393Qo8AeDurwD5QPHAN3L3je5e6e6VJSUlZ1expJ9FayC3ILIttwAWrWHz7gau+MHzTL9nK1f84Hk27x74oyMiiRBLuL8OzDSz6WY2BlgOVA/o8wdgEYCZzSYU7ro0Hy0qboSlD0DhFMBCvy99gM3dV7C6ag8NLW040NDSxuqqPQp4kSQYds7d3bvM7HagBsgGHnH3vWa2Dqh192rgLuBnZvYtQjdXv+6p+uirpEbFjWfcPF3/g+dp6+yOaGvr7GZ9TT3XzS1PZnUio05Mz5Zx922Eljf2b1vT7/U7wBXxLU0y3eGWthG1i0j86BOqkjCTigpG1C4i8aNwl4RZtXgWBbnZEW0FudmsWjwLgKeajlP58l7KdrxJ5ct7earpeCrKFAkkPfJXEqZ3Xn19TT2HW9qYVFTAqsWzuG5uOU81Hefu+o9o6wndmjnU0cnd9aGPU1xfOiFlNYsEhR75KylR+fJeDnV0ntE+OS+X2ss/m4KKRDKDHvkraa0hSrAP1S4iI6Nwl5Qoz8sdUbuIjIzCXVJi9YwyCrIsoq0gy1g949Mv3m5sepqXXlrA9ucv5KWXFtDY9HSyyxTJWLqhKinRe9P0+wcaaejopDwvl9UzyvraG5ue5t1376WnJ7Qmvr3jMO++ey8AZaXLUlO0SAZRuEvKXF86YdCVMQc++GFfsPfq6WnjwAc/VLiLxEDTMpKW2jvO/BLuodpFJJLCXdJSfl7ZiNpFJJKmZSQtzbjg7og5d4CsrAJmXHB333ZdXR3bt2+ntbWVwsJCFi1aREVFRSrKFUk7CndJS73z6gc++CHtHY3k55Ux44K7+9rr6urYsmULnZ2hdfGtra1s2bIFQAEvgsJd0lhZ6bJBb55u3769L9h7dXZ2sn37doW7CJpzlwzV2to6onaR0UZX7pKRCgsLowZ5YWFh3+tTu4/wcc1Buls6yC7K47zF0xg79/xklimSMrpyl4y0aNEicnMjH1WQm5vLokWLgFCwt1S9T3dLBwDdLR20VL3Pqd1Hkl6rSCoo3CUjVVRUsHTp0r4r9cLCQpYuXdo33/5xzUG8syfiz3hnDx/XHEx2qSIpoWkZyVgVFRWD3jztvWKPtV0kaBTuEkjZRXlRgzy7KC9ie9+LO3hx06OcOHaU8ROLWbD8FmYvWJisMkUSRtMyEkjnLZ6G5Ub+eFtuFuctnta3ve/FHTy38aecONoM7pw42sxzG3/Kvhd3JLlakfhTuEsgjZ17PkVfmdl3pZ5dlEfRV2ZGrJZ5cdOjdJ2OvLrvOt3Bi5seTWqtIomgaRkJrLFzzx9y6eOJY0dH1C6SSWIKdzNbAmwAsoGfu/sPovS5EVgLOPCWu/9lHOsUibvxE4tDUzJR2nu9t7OJV57+gJPHOxg3IY/Lll3ARfNLk1mmyFkZdlrGzLKBB4EvA3OAm8xszoA+M4HVwBXu/lngzgTUKhJXC5bfQs6YyBusOWPyWLD8FiAU7Dsee5eTx0NTNyePd7DjsXd5b2dT0msVGalY5tznAfvd/YC7nwY2AQMf+PHfgQfd/Y8A7q5Pikjam71gIVetuJ3xxSVgxvjiEq5acXvfaplXnv6ArtORa+W7TvfwytMfpKJckRGJZVqmHPio3/YhYP6APhcBmNlLhKZu1rr7s3GpUCSBZi9YOOjSx94r9uHaW7ds4ch999PV2EhOWRnnf+tOCpcujXutIiMRrxuqOcBM4EpgMvCCmX3O3Vv6dzKzFcAKgKlTp8bp0CKJMW5CXtSAHzfh06mc1i1baPz2Gry9HYCuw4dp/PYaAAW8pFQs0zINwJR+25PDbf0dAqrdvdPdPwTeIxT2Edx9o7tXuntlSUnJ2dYskhSXLbuAnDGR/4vkjMnismUX9G0fue/+vmDv5e3tHLnv/qTUKDKYWML9dWCmmU03szHAcqB6QJ/NhK7aMbNiQtM0B+JYp0jSXTS/lIU3f6bvSn3chDwW3vyZiNUyXY3Rv9N1sHaRZBl2Wsbdu8zsdqCG0Hz6I+6+18zWAbXuXh3ed5WZvQN0A6vc/VgiCxdJhovmlw659DGnrIyuw4ejtve39cBWNryxgaZTTZSOLWXlJSu5ZsY1ca9XpJe5e0oOXFlZ6bW1tSk5tki8DJxzB7D8fMq+u65vzn3rga2sfXkt7d2f9snPzmft5WsV8DJiZrbL3SuH66fHD4icg8KlSyn77jpyJk0CM3ImTYoIdoANb2yICHaA9u52NryxIdnlyiiixw+InKPCpUuHXBnTdCr6h57OaK97Aravg9ZDUDgZFq2BihvjWaqMIrpyF0mw0rHR5+wj2uuegC13QOtHgId+33JHqF3kLCjcRRJs5SUryc/Oj2jLz85n5SUrP23Yvg462yL/YGdbqF3kLGhaRiTBem+aDrlapvVQ9D88WLvIMBTuIklwzYxrhl4ZUzg5PCUTpb2fzbsbWF9Tz+GWNiYVFbBq8Syum1se52olCDQtI5IOFq2B3ILIttyCUHvY5t0NrK7aQ0NLGw40tLSxumoPm3cP/MC4iMJdJD1U3AhLH4DCKYCFfl/6QMRqmfU19bR1dkf8sbbObtbX1Ce5WMkEmpYRSRcVNw659PFwS9uI2iU93PXw93h22nyO2QQm+nGWHNzJj269N+HH1ZW7SIaYVFQwonZJvbse/h6/nb6IY1nFYFkcyyrmt9MXcdfD30v4sRXuIhli1eJZFORmR7QV5GazavGsFFUkw3l22nxOW+Qy2NOWz7PTBn4lRvxpWkYkQ/SuitFqmcxxzCaMqD2eFO4iGeS6ueUK8wwy0Y9zzIqjtieapmVERBJkycGdjPHIh8aN8XaWHNyZ8GMr3EVEEuRHt97Ln3+4nYk9R8F7mNhzlD//cHtSVsvoee4iIhlEz3MXERnFFO4iIgGkcBcRCSCFu4hIACncRUQCSOEuIhJACncRkQBSuIuIBFBM4W5mS8ys3sz2m9k9Q/S73szczIZdYC8iIokzbLibWTbwIPBlYA5wk5nNidJvPLASSPxDE0REZEixXLnPA/a7+wF3Pw1sApZF6fdd4B+B9ij7REQkiWIJ93Kg/9eyHwq39TGzS4Ap7r41jrWJiMhZOucbqmaWBfwYuCuGvivMrNbMapubm8/10CIiMohYwr0BmNJve3K4rdd44GLgd2Z2ELgUqI52U9XdN7p7pbtXlpSUnH3VIiIypFjC/XVgpplNN7MxwHKgunenu7e6e7G7T3P3acCrwLXuruf5ioikyLDh7u5dwO1ADbAPeMLd95rZOjO7NtEFiojIyMX0Haruvg3YNqBtzSB9rzz3skRE5FzoE6oiIgGkcBcRCSCFu4hIACncRUQCSOEuIhJACncRkQBSuIuIBJDCXUQkgBTuIiIBpHAXEQkghbuISAAp3EVEAkjhLiISQAp3EZEAUriLiASQwl1EJIAU7iIiAaRwFxEJIIW7iEgAKdxFRAJI4S4iEkAKdxGRAFK4i4gEUEzhbmZLzKzezPab2T1R9v+Nmb1jZnVmtt3M/jT+pYqISKyGDXczywYeBL4MzAFuMrM5A7rtBirdvQJ4EvineBcqIiKxi+XKfR6w390PuPtpYBOwrH8Hd9/h7p+EN18FJse3TBERGYlYwr0c+Kjf9qFw22BuBZ45l6JEROTc5MTzzczsq0Al8IVB9q8AVgBMnTo1nocWEZF+YrlybwCm9NueHG6LYGZfBO4FrnX3jmhv5O4b3b3S3StLSkrOpl4REYlBLOH+OjDTzKab2RhgOVDdv4OZzQX+lVCwH4l/mSIiMhLDhru7dwG3AzXAPuAJd99rZuvM7Npwt/XAOOC3ZvammVUP8nYiIpIEMc25u/s2YNuAtjX9Xn8xznWJiMg50CdURUQCSOEuIhJACncRkQBSuIuIBJDCXUQkgBTuIiIBpHAXEQkghbuISAAp3EVEAkjhLiISQAp3EZEAUriLiASQwl1EJIAU7iIiAaRwFxEJIIW7iEgAKdxFRAJI4S4iEkAKdxGRAFK4i4gEkMJdRCSAFO4iIgGkcBcRCSCFu4hIAMUU7ma2xMzqzWy/md0TZX+emf0mvH+nmU2Ld6EiIhK7nOE6mFk28CDwJeAQ8LqZVbv7O/263Qr80d0vNLPlwD8Cf5GIgntt3t3A+pp6Dre0MamogFWLZ3Hd3PJEHlJEJGPEcuU+D9jv7gfc/TSwCVg2oM8y4Ffh108Ci8zM4ldmpM27G1hdtYeGljYcaGhpY3XVHjbvbkjUIUVEMkos4V4OfNRv+1C4LWofd+8CWoGJ8SgwmvU19bR1dke0tXV2s76mPlGHFBHJKEm9oWpmK8ys1sxqm5ubz/p9Dre0jahdRGS0iSXcG4Ap/bYnh9ui9jGzHKAQODbwjdx9o7tXuntlSUnJ2VUMTCoqGFG7iMhoE0u4vw7MNLPpZjYGWA5UD+hTDXwt/PoG4Hl39/iVGWnV4lkU5GZHtBXkZrNq8axEHVJEJKMMu1rG3bvM7HagBsgGHnH3vWa2Dqh192rgYeDXZrYfOE7oL4CE6V0Vo9UyIiLRWQIvsIdUWVnptbW1KTm2iEimMrNd7l45XD99QlVEJIAU7iIiAaRwFxEJIIW7iEgAKdxFRAJI4S4iEkApWwppZs3A7+PwVsXA0Ti8T6ppHOlF40gvQRhHvMbwp+4+7Ef8Uxbu8WJmtbGs+Ux3Gkd60TjSSxDGkewxaFpGRCSAFO4iIgEUhHDfmOoC4kTjSC8aR3oJwjiSOoaMn3MXEZEzBeHKXUREBsiIcDezJWZWb2b7zeyeKPvzzOw34f07zWxa8qscXgzj+LqZNZvZm+Fft6WizuGY2SNmdsTM3h5kv5nZA+Fx1pnZJcmuMRYxjONKM2vtdz7WJLvG4ZjZFDPbYWbvmNleM1sZpU/an48Yx5EJ5yPfzF4zs7fC4/iHKH2Sk1funta/CD1D/gNgBjAGeAuYM6DP/wQeCr9eDvwm1XWf5Ti+Dvw01bXGMJb/ClwCvD3I/quBZwADLgV2prrmsxzHlcC/p7rOYcZQBlwSfj0eeC/Kz1Xan48Yx5EJ58OAceHXucBO4NIBfZKSV5lw5T4P2O/uB9z9NLAJWDagzzLgV+HXTwKLzMySWGMsYhlHRnD3Fwh9KctglgGPesirQJGZlSWnutjFMI605+6N7v5G+PUJYB9nfoF92p+PGMeR9sL/jU+GN3PDvwbe2ExKXmVCuJcDH/XbPsSZJ72vj7t3Aa3AxKRUF7tYxgFwffifzk+a2ZQo+zNBrGPNBJeF/4n9jJl9NtXFDCX8z/u5hK4W+8uo8zHEOCADzoeZZZvZm8AR4D/cfdDzkci8yoRwH022ANPcvQL4Dz79211S4w1CH/X+T8BPgM0prmdQZjYOeAq4090/TnU9Z2uYcWTE+XD3bnf/PDAZmGdmF6eijkwI9wag/xXs5HBb1D5mlgMUAseSUl3shh2Hux9z947w5s+B/5yk2uItlnOW9tz9495/Yrv7NiDXzIpTXNYZzCyXUCA+5u5VUbpkxPkYbhyZcj56uXsLsANYMmBXUvIqE8L9dWCmmU03szGEbkBUD+hTDXwt/PoG4HkP361II8OOY8A86LWE5h0zUTVwS3iVxqVAq7s3prqokTKz0t65UDObR+j/l7S6aAjX9zCwz91/PEi3tD8fsYwjQ85HiZkVhV8XAF8C3h3QLSl5lRPvN4w3d+8ys9uBGkIrTh5x971mtg6odfdqQj8Uvzaz/YRukC1PXcXRxTiOO8zsWqCL0Di+nrKCh2BmjxNauVBsZoeA7xC6cYS7PwRsI7RCYz/wCfCN1FQ6tBjGcQPw12bWBbQBy9PwouEK4K+APeF5XoD/BUyFjDofsYwjE85HGfArM8sm9JfPE+7+76nIK31CVUQkgDJhWkZEREZI4S4iEkAKdxGRAFK4i4gEkMJdRCSAFO4iIgGkcBcRCSCFu4hIAP1/QEWhOFCJ4X4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "with sess.as_default():\n",
    "    sess.run(tf.group(tf.global_variables_initializer(), tf.local_variables_initializer()))\n",
    "    for _ in range(30):\n",
    "        emb_space_learned, _ = sess.run([emb_space, train_op], feed_dict={points:points_val})\n",
    "        plt.scatter(emb_space_learned[:, 0], emb_space_learned[:, 1])\n",
    "    plt.scatter(points_val[:, 0], points_val[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot shows how the embedding vector in them middle is moving towards the _cluster_ center, following the gradient. The chosen hyperparameters are\n",
    "* learning rate $\\eta=1$\n",
    "* update weighting $\\alpha=0.1$\n",
    "* iterations $n=50$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}