RankNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RankNet\n",
    "\n",
    "This is the generic implementation of RankNet on a synthetic dataset.\n",
    "\n",
    "This model is applied to hotel imags in the other notebook.\n",
    "\n",
    "![alt text](assets/siamese-net.png \"Title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "import keras\n",
    "from keras import backend\n",
    "from keras.layers import Activation, Add, Dense, Input, Lambda, Dropout, Subtract\n",
    "from keras.models import Model, Sequential\n",
    "from keras.utils import plot_model\n",
    "%pylab inline\n",
    "\n",
    "INPUT_DIM = 1536"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Base Network\n",
    "\n",
    "This is a scoring function (regression) that takes an images vector as input and outputs a scalar number. The Siamese model create two of these base networks to for higher and lower ranking images respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_base_network(input_dim):\n",
    "    '''Base network to be shared (eq. to feature extraction).\n",
    "    '''\n",
    "    seq = Sequential()\n",
    "    seq.add(Dense(INPUT_DIM, input_shape=(input_dim,), activation='relu'))\n",
    "    seq.add(Dropout(0.1))\n",
    "    seq.add(Dense(64, activation='relu'))\n",
    "    seq.add(Dropout(0.1))\n",
    "    seq.add(Dense(32, activation='relu'))\n",
    "    seq.add(Dense(1))\n",
    "    return seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Siamse Archecture\n",
    "\n",
    "We create two instances of the base archecture and take the difference between their scalar outputs. We then pass the difference between a sigmoid function so that we can use binary-cross-entropy for the differntiable loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def create_meta_network(input_dim, base_network):\n",
    "    input_a = Input(shape=(input_dim,))\n",
    "    input_b = Input(shape=(input_dim,))\n",
    "\n",
    "    rel_score = base_network(input_a)\n",
    "    irr_score = base_network(input_b)\n",
    "\n",
    "    # subtract scores\n",
    "    diff = Subtract()([rel_score, irr_score])\n",
    "\n",
    "    # Pass difference through sigmoid function.\n",
    "    prob = Activation(\"sigmoid\")(diff)\n",
    "\n",
    "    # Build model.\n",
    "    model = Model(inputs = [input_a, input_b], outputs = prob)\n",
    "    model.compile(optimizer = \"adam\", loss = \"binary_crossentropy\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_7 (InputLayer)            (None, 1536)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_8 (InputLayer)            (None, 1536)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "sequential_4 (Sequential)       (None, 1)            2461313     input_7[0][0]                    \n",
      "                                                                 input_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "subtract_4 (Subtract)           (None, 1)            0           sequential_4[1][0]               \n",
      "                                                                 sequential_4[2][0]               \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 1)            0           subtract_4[0][0]                 \n",
      "==================================================================================================\n",
      "Total params: 2,461,313\n",
      "Trainable params: 2,461,313\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "base_network = create_base_network(INPUT_DIM)\n",
    "model = create_meta_network(INPUT_DIM, base_network)\n",
    "model.summary()\n",
    "\n",
    "plot_model(base_network, to_file='base.png')\n",
    "plot_model(model, to_file='meta.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base Network\n",
    "\n",
    "This can be any (scoring) function which outputs a scalar:\n",
    "![base](base.png)\n",
    "\n",
    "### Meta Network\n",
    "\n",
    "This takes an instance of the base network and evalutes the pair on it:\n",
    "![meta](meta.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synthetic Dataset\n",
    "\n",
    "We generate two N dimensional datasets from an uniform distribution where one distribution is shift by a constant. This can be thought of as a good image and a less good image that we want to learn to differentiate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0023731553517547066, -0.00052067296140614276)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fake data.\n",
    "N = 1000\n",
    "SPLIT=750\n",
    "X_1 = 2* np.random.randn(N, INPUT_DIM) # notice this data is shifted by 2\n",
    "X_2 = np.random.randn(N, INPUT_DIM)\n",
    "\n",
    "X_1_train = X_1[0:SPLIT,:]\n",
    "X_1_test = X_1[SPLIT:N,:]\n",
    "\n",
    "X_2_train = X_2[0:SPLIT,:]\n",
    "X_2_test =  X_2[SPLIT:N,:]\n",
    "\n",
    "\n",
    "y = np.ones((X_1.shape[0], 1))\n",
    "\n",
    "y_train = y[0:SPLIT,:]\n",
    "y_test = y[SPLIT:N,:]\n",
    "\n",
    "np.mean(X_1_train), np.mean(X_2_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 750 samples, validate on 250 samples\n",
      "Epoch 1/10\n",
      "750/750 [==============================] - 7s 9ms/step - loss: 0.0460 - val_loss: 1.1323e-04\n",
      "Epoch 2/10\n",
      "750/750 [==============================] - 5s 7ms/step - loss: 0.0045 - val_loss: 3.0593e-04\n",
      "Epoch 3/10\n",
      "750/750 [==============================] - 6s 8ms/step - loss: 8.4689e-04 - val_loss: 8.8093e-04\n",
      "Epoch 00003: early stopping\n"
     ]
    }
   ],
   "source": [
    "# Train model.\n",
    "\n",
    "es=keras.callbacks.EarlyStopping(monitor='val_loss',\n",
    "                              min_delta=0,\n",
    "                              patience=2,\n",
    "                              verbose=1, mode='auto')\n",
    "\n",
    "NUM_EPOCHS = 10\n",
    "BATCH_SIZE = 10\n",
    "history = model.fit([X_1_train, X_2_train], y_train, \n",
    "                    validation_data=([X_1_test, X_2_test], y_test), \n",
    "                    batch_size = BATCH_SIZE, epochs = NUM_EPOCHS, verbose = 1, callbacks=[es])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXd//H3NzsJSSAL+5IgWGRRkIBrW1u1olbRahUV\nBaVF29rapz5UfdRWbftUf+2j1qUqCgWXiora0qp1qdrNjYQdQQkEJCBLAiSQkJDl/v0xJziEJDOB\nTE4y+byuay7OnHOfmU8Ok/PNue8z55hzDhERkZbE+B1AREQ6PhULEREJScVCRERCUrEQEZGQVCxE\nRCQkFQsREQlJxULkCJnZXDP7ZZhtN5jZGUf6OiLtTcVCRERCUrEQEZGQVCykS/C6f2aa2XIzqzCz\n2WbW28xeM7M9ZvaWmfUMan++ma0ys91m9q6ZHRO0bKyZLfbWew5IavRe3zSzpd6675nZsYeZ+btm\nVmhmO81soZn18+abmd1nZtvNrNzMVpjZKG/ZOWb2sZdts5n992FtMJFGVCykK7kIOBM4GjgPeA34\nHyCbwO/CjwDM7GjgWeDH3rJXgb+YWYKZJQB/Ap4CMoAXvNfFW3csMAe4FsgEHgMWmllia4Ka2deB\nXwOXAH2BjcB8b/E3gK94P0e616bUWzYbuNY5lwqMAt5uzfuKNEfFQrqSB51z25xzm4F/AR8655Y4\n56qAl4GxXrtLgVecc28652qA3wLdgJOBE4F44H7nXI1zbgGwKOg9ZgCPOec+dM7VOefmAdXeeq1x\nBTDHObfYOVcN3AKcZGY5QA2QCgwHzDm32jn3ubdeDTDCzNKcc7ucc4tb+b4iTVKxkK5kW9D0viae\nd/em+xH4Sx4A51w9sAno7y3b7A6+AufGoOnBwI1eF9RuM9sNDPTWa43GGfYSOHro75x7G3gIeBjY\nbmazzCzNa3oRcA6w0cz+YWYntfJ9RZqkYiFyqC0EdvpAYIyAwA5/M/A50N+b12BQ0PQm4FfOuR5B\nj2Tn3LNHmCGFQLfWZgDn3APOuXHACALdUTO9+Yucc5OAXgS6y55v5fuKNEnFQuRQzwPnmtnpZhYP\n3EigK+k94H2gFviRmcWb2beACUHrPg5cZ2YneAPRKWZ2rpmltjLDs8DVZjbGG+/4XwLdZhvMbLz3\n+vFABVAF1HtjKleYWbrXfVYO1B/BdhA5QMVCpBHn3CfAFOBBoITAYPh5zrn9zrn9wLeAacBOAuMb\nLwWtmw98l0A30S6g0Gvb2gxvAbcDLxI4mjkKmOwtTiNQlHYR6KoqBX7jLbsS2GBm5cB1BMY+RI6Y\n6eZHIiISio4sREQkJBULEREJScVCRERCUrEQEZGQ4vwO0FaysrJcTk6O3zFERDqVgoKCEudcdqh2\nUVMscnJyyM/P9zuGiEinYmYbQ7dSN5SIiIRBxUJEREJSsRARkZCiZsxCRORw1NTUUFxcTFVVld9R\nIiopKYkBAwYQHx9/WOurWIhIl1ZcXExqaio5OTkcfDHh6OGco7S0lOLiYnJzcw/rNdQNJSJdWlVV\nFZmZmVFbKADMjMzMzCM6elKxEJEuL5oLRYMj/Rm7fLGoqK7ljoWrKNtX43cUEZEOq8sXizVby3n6\ng4384JnF1NTpPjEi0r52797N73//+1avd84557B79+4IJGpaly8W4wZn8L/fGs2/C0v4+cJV6P4e\nItKemisWtbW1La736quv0qNHj0jFOoTOhgIuyRtIUUkFj7y7jiFZKXzny0P8jiQiXcTNN9/MunXr\nGDNmDPHx8SQlJdGzZ0/WrFnDp59+ygUXXMCmTZuoqqrihhtuYMaMGcAXlzjau3cvZ599Nqeeeirv\nvfce/fv3589//jPdunVr05wqFp6Z3/gSG0oq+NWrqxmUkcw3RvbxO5KItLM7/7KKj7eUt+lrjuiX\nxs/PG9ns8rvvvpuVK1eydOlS3n33Xc4991xWrlx54BTXOXPmkJGRwb59+xg/fjwXXXQRmZmZB73G\n2rVrefbZZ3n88ce55JJLePHFF5kyZUqb/hxdvhuqQUyMce8lYzi2fzo3zF/Kys1lfkcSkS5owoQJ\nB30X4oEHHuC4447jxBNPZNOmTaxdu/aQdXJzcxkzZgwA48aNY8OGDW2eS0cWQbolxPL41DwufPg9\nps9bxJ9+cAp909v2UE5EOq6WjgDaS0pKyoHpd999l7feeov333+f5ORkTjvttCa/K5GYmHhgOjY2\nln379rV5Lh1ZNNIrNYnZ0/KoqK5j+tx8KqpbHmQSETkSqamp7Nmzp8llZWVl9OzZk+TkZNasWcMH\nH3zQzum+oGLRhOF90njw8rGs2VrODfOXUFevM6REJDIyMzM55ZRTGDVqFDNnzjxo2cSJE6mtreWY\nY47h5ptv5sQTT/QpJVi0nCqal5fn2vrmR0++v4Gf/XkV00/N5fZvjmjT1xaRjmH16tUcc8wxfsdo\nF039rGZW4JzLC7WuxixacNVJOazfUcHsfxeRk5XClScO9juSiIgvVCxCuP2bI9hYWsEdC1cxKCOZ\nrx4d8la1IiJRR2MWIcTGGA9efjzDenXn+mcW8+m2pgeiRESimYpFGLonxjF72niSEmK5+g+L2LGn\n2u9IIiLtSsUiTP17dOOJq/IorahmxlP5VNXU+R1JRKTdqFi0wnEDe3D/pWNY8tlu/vuFZdTrlFoR\n6SJULFpp4qi+3Hz2cP66/HPue+tTv+OISCd3uJcoB7j//vuprKxs40RNU7E4DNd+ZQiX5g3kwbcL\nebGg2O84ItKJdZZioVNnD4OZ8YsLRrFpVyU3v7ScAT27ccKQzNAriog0EnyJ8jPPPJNevXrx/PPP\nU11dzYUXXsidd95JRUUFl1xyCcXFxdTV1XH77bezbds2tmzZwte+9jWysrJ45513IppTxeIwJcTF\n8MgV47jwkf9w7dMFvPz9U8jNSgm9ooh0XK/dDFtXtO1r9hkNZ9/d7OLgS5S/8cYbLFiwgI8++gjn\nHOeffz7//Oc/2bFjB/369eOVV14BAteMSk9P59577+Wdd94hKyurbTM3Qd1QRyA9OZ4/TBuPAdfM\nXcTuyv1+RxKRTuyNN97gjTfeYOzYsRx//PGsWbOGtWvXMnr0aN58801uuukm/vWvf5Gent7u2XRk\ncYQGZ6Yw66o8rnj8Q659qoCnpp9AQpxqsEin1MIRQHtwznHLLbdw7bXXHrJs8eLFvPrqq9x2222c\nfvrp/OxnP2vXbNqrtYHxORncc/FoPizaya0vr9B9vEUkbMGXKD/rrLOYM2cOe/fuBWDz5s1s376d\nLVu2kJyczJQpU5g5cyaLFy8+ZN1I05FFG7lw7ACKSip54O9ryc1O4funDfU7koh0AsGXKD/77LO5\n/PLLOemkkwDo3r07Tz/9NIWFhcycOZOYmBji4+N55JFHAJgxYwYTJ06kX79+ER/gjuglys1sIvA7\nIBZ4wjl3d6PlicCTwDigFLjUObchaPkg4GPgDufcb1t6r0hcory1nHPcMH8pC5dt4fdXHM85o/v6\nmkdEQtMlysO7RHnEuqHMLBZ4GDgbGAFcZmaNbwoxHdjlnBsK3Afc02j5vcBrkcrY1syM/3fxsRw/\nqAf/9dxSlm7a7XckEZE2EckxiwlAoXNuvXNuPzAfmNSozSRgnje9ADjdzAzAzC4AioBVEczY5pLi\nY3n8qjx6pSXynXn5FO9qny/MiIhEUiSLRX9gU9DzYm9ek22cc7VAGZBpZt2Bm4A7W3oDM5thZvlm\nlr9jx442C36kMrsnMmfqeKprA/fx3lNV43ckEWlBVzgp5Uh/xo56NtQdwH3Oub0tNXLOzXLO5Tnn\n8rKzO9ZNiYb1TuWRK8ZRuGMv1/9xCbV19X5HEpEmJCUlUVpaGtUFwzlHaWkpSUlJh/0akTwbajMw\nMOj5AG9eU22KzSwOSCcw0H0CcLGZ/T+gB1BvZlXOuYcimLfNnTosi19eMIpbXlrBnX/5mLsmjcTr\nZRORDmLAgAEUFxfTkXonIiEpKYkBAwYc9vqRLBaLgGFmlkugKEwGLm/UZiEwFXgfuBh42wXK+5cb\nGpjZHcDezlYoGlw2YRBFJRXM+ud6hmSncPUpuX5HEpEg8fHx5Obq9zKUiBUL51ytmV0PvE7g1Nk5\nzrlVZnYXkO+cWwjMBp4ys0JgJ4GCEnVumjicopIKfvHXjxmcmczXh/f2O5KISKtE9HsW7akjfM+i\nJZX7a/n2o++zoaSCF647mRH90vyOJCLi//cs5GDJCXHMnjqe1KR4ps9bxPbyKr8jiYiETcWiHfVJ\nT+KJqXmU7ath+rx8KvfX+h1JRCQsKhbtbFT/dB6YPJaVW8r4r+eW6j7eItIpqFj44IwRvbnt3BG8\nvmob97y+xu84IiIh6aqzPrnmlByKSvby2D/Wk5uZwuQJg/yOJCLSLBULn5gZd5w3ko2lldz2p5UM\nzEjmlKGRvzWiiMjhUDeUj+JiY3j4iuMZkp3CdU8XULi9fW5iIiLSWioWPktLimf21PEkxsVwzdx8\nSvdW+x1JROQQKhYdwMCMZGZdlcfW8iqufaqA6to6vyOJiBxExaKDOH5QT/7v28eRv3EXNy1YHtVX\nwBSRzkcD3B3Iecf1Y2NpBb9941Nys7pzwxnD/I4kIgKoWHQ4P/jaUNaXVHDfW5+Sk5XMpDGN7xcl\nItL+1A3VwZgZv/7WaCbkZjDzheXkb9jpdyQRERWLjigxLpbHpoyjX48kZjxVwGeluo+3iPhLxaKD\n6pmSwJxp46mrd1w99yPK9uk+3iLiHxWLDmxIdnceu3Icn+2s5PvPFFCj+3iLiE9ULDq4E4dk8r8X\njuY/haXc/qeVOqVWRHyhs6E6gW/nDWRDaQUPv7OOIdkpzPjKUX5HEpEuRsWik7jxzC9RVFLBr19b\nw+DMFM4a2cfvSCLShagbqpOIiTHuvWQMxw7owY/nL2VFcZnfkUSkC1Gx6ESS4mN5/KpxZKQkMH3e\nIj4v2+d3JBHpIlQsOpleqUnMnpZH5f46rpmbT0W17uMtIpGnYtEJDe+TxkOXj+WTreX86Nkl1Ok+\n3iISYSoWndRpX+rFneeP5O9rtvOrV1b7HUdEopzOhurErjwph/UlFcz5TxG5WclceVKO35FEJEqp\nWHRyt507go2lldzxl48ZmJHMaV/q5XckEYlC6obq5GJjjAcuG8vRvVO5/o9L+GSr7uMtIm1PxSIK\ndE+MY/bUPJITYrlm7iK276nyO5KIRBkViyjRr0c3Zk8dT2lFNTOeLKCqRvfxFpG2o2IRRUYPSOf+\nS8eyrHg3Nz6/jHqdUisibUTFIspMHNWHmycO55UVn3Pvm5/6HUdEooTOhopCM74yhKKSCh56p5Cc\nrBQuHjfA70gi0snpyCIKmRm/uGAUJx+VyS0vLeeD9aV+RxKRTk7FIkrFx8bwyBXjGJSRzLVPFbB+\nx16/I4lIJxbRYmFmE83sEzMrNLObm1ieaGbPecs/NLMcb/4EM1vqPZaZ2YWRzBmt0pPjmTNtPLEx\nxvR5+eyq2O93JBHppCJWLMwsFngYOBsYAVxmZiMaNZsO7HLODQXuA+7x5q8E8pxzY4CJwGNmpvGV\nwzA4M4VZV45j8659XPt0AftrdR9vEWm9SB5ZTAAKnXPrnXP7gfnApEZtJgHzvOkFwOlmZs65Sudc\nw7W3kwCdA3oE8nIy+M23j+Wjop3c8tIK3cdbRFotksWiP7Ap6HmxN6/JNl5xKAMyAczsBDNbBawA\nrgsqHgeY2Qwzyzez/B07dkTgR4gek8b058dnDOPFxcX8/t11fscRkU6mww5wO+c+dM6NBMYDt5hZ\nUhNtZjnn8pxzednZ2e0fspO54fRhTBrTj9+8/gl/Xb7F7zgi0olEslhsBgYGPR/gzWuyjTcmkQ4c\ndJ6nc241sBcYFbGkXYSZcc9FxzJucE9ufH4ZSz7b5XckEekkIlksFgHDzCzXzBKAycDCRm0WAlO9\n6YuBt51zzlsnDsDMBgPDgQ0RzNplJMXHMuvKcfRKS+S7T+azaWel35FEpBOIWLHwxhiuB14HVgPP\nO+dWmdldZna+12w2kGlmhcBPgIbTa08FlpnZUuBl4PvOuZJIZe1qMrsn8odp46murWf6vEWUV9X4\nHUlEOjiLljNj8vLyXH5+vt8xOpX/FJYwdc5HnDw0izlT84iL7bBDWCISIWZW4JzLC9VOe4cu7JSh\nWfziglH889Md3PGXVTqlVkSapS+6dXGXTRjEhpIKHvvneoZkdeeaU3P9jiQiHZCKhXDTxOEUlVTw\ni1c+ZlBGMmeM6O13JBHpYNQNJcTEGPdPHsOofun8aP4SVm0p8zuSiHQwKhYCQHJCHE9MzSO9WzzT\n5+azrVz38RaRL6hYyAG905J4Ymoe5VU1TJ+3iMr9h1xhRUS6KBULOcjIfuk8eNlYPt5Szo/nL9V9\nvEUEULGQJpx+TG9uO3cEb3y8jXv+tsbvOCLSAehsKGnS1afkUOSdUpuTlcJlEwb5HUlEfKRiIU0y\nM35+3gg+21nJ7X9aycCeyZw6LMvvWCLiE3VDSbPiYmN46PKxHJXdne89U0Dh9j1+RxIRn6hYSItS\nk+KZPS2PxLgYrp67iNK91X5HEhEfqFhISAN6JvP4VXlsL69mxlMFVNXU+R1JRNqZioWEZeygntx7\nyRgKNu7ipwuW66KDIl2MioWE7dxj+zLzrC+xcNkW7n9rrd9xRKQd6WwoaZXvn3YU63dU8Lu/ryU3\nK4ULxvb3O5KItAMdWUirmBm//tZoTsjN4KcLlpO/YaffkUSkHahYSKslxMXw6JRx9O/ZjRlPFbCx\ntMLvSCISYWEVCzO7wczSLGC2mS02s29EOpx0XD1TEpgzbTz1znHN3EWUVeo+3iLRLNwji2ucc+XA\nN4CewJXA3RFLJZ1CblYKj04Zx2c7K/neMwXU1NX7HUlEIiTcYmHev+cATznnVgXNky7sxCGZ/Ppb\nx/LeulJue3mlTqkViVLhng1VYGZvALnALWaWCujPSAHg4nED2FBSwUPvFDIkO4Vrv3qU35FEpI2F\nWyymA2OA9c65SjPLAK6OXCzpbH5y5tEUlVZw99/WMDgzmYmj+vodSUTaULjdUCcBnzjndpvZFOA2\nQDdqlgNiYoz/+/ZxHDegBz9+binLi3f7HUlE2lC4xeIRoNLMjgNuBNYBT0YslXRKSfGxPH5VHpkp\niUyfl8+W3fv8jiQibSTcYlHrAiOXk4CHnHMPA6mRiyWdVXZqInOmjWff/jqumbuIvdW6j7dINAi3\nWOwxs1sInDL7ipnFAPGRiyWd2Zf6pPLwFcezdvtefvTsEup0H2+RTi/cYnEpUE3g+xZbgQHAbyKW\nSjq9rx6dzR3nj+TtNdv55Ssf+x1HRI5QWMXCKxDPAOlm9k2gyjmnMQtp0ZUnDuaaU3L5w3828OT7\nG/yOIyJHINzLfVwCfAR8G7gE+NDMLo5kMIkOt557DKcP78UdC1fx7ifb/Y4jIocp3G6oW4Hxzrmp\nzrmrgAnA7ZGLJdEiNsZ44LKxDO+TxvV/XMKareV+RxKRwxBusYhxzgX/WVjainWli0tJjGP2tDyS\nE2KZPjef7Xuq/I4kIq0U7g7/b2b2uplNM7NpwCvAq5GLJdGmb3o3Zk8dz86K/Xz3yQL27dd9vEU6\nk3AHuGcCs4Bjvccs59xNkQwm0Wf0gHTunzyG5cW7ufGFpdTrlFqRTiPsriTn3IvOuZ94j5fDWcfM\nJprZJ2ZWaGY3N7E80cye85Z/aGY53vwzzazAzFZ4/3493JzSsZ01sg+3nD2cV1ds5bdvfOJ3HBEJ\nU4sXEjSzPUBTf/4Z4JxzaS2sGws8DJwJFAOLzGyhcy74pPvpwC7n3FAzmwzcQ+A7HSXAec65LWY2\nCngd0M2eo8R3vzyEopIKfv/uOnKyUrgkb6DfkUQkhBaLhXPuSC7pMQEodM6tBzCz+QQuFxJcLCYB\nd3jTC4CHzMycc0uC2qwCuplZonOu+gjySAdhZtw1aRSf7azk1pdXMLBnMicdlel3LBFpQSTPaOoP\nbAp6XsyhRwcH2jjnaglcybbxXuMiYHFThcLMZphZvpnl79ixo82CS+TFx8bw+yvGMSgjmeueLmD9\njr1+RxKRFnTo01/NbCSBrqlrm1runJvlnMtzzuVlZ2e3bzg5Yund4vnDtAnExhjXzF3Eror9fkcS\nkWZEslhsBoI7owd485psY2ZxQDqB73BgZgOAl4GrnHPrIphTfDQoM5nHrxrHlrIqrn2qgOpanVIr\n0hFFslgsAoaZWa6ZJQCTgYWN2iwEpnrTFwNvO+ecmfUg8F2Om51z/4lgRukAxg3O4DcXH8tHG3Zy\ny0srdB9vkQ4oYsXCG4O4nsCZTKuB551zq8zsLjM732s2G8g0s0LgJ0DD6bXXA0OBn5nZUu/RK1JZ\nxX+TxvTnv844mpcWb+bhdwr9jiMijVi0/BWXl5fn8vPz/Y4hR8A5x0+eX8bLSzbz4GVjOe+4fn5H\nEol6ZlbgnMsL1a5DD3BL12Jm3H3RaMbn9OTGF5ZRsHGX35FExKNiIR1KYlwsj12ZR5+0JGY8mc+m\nnZV+RxIRVCykA8pISWDOtPHsr6vnmrmLKK+q8TuSSJenYiEd0tBe3Xl0yjiKSir4wTOLqa2r9zuS\nSJemYiEd1ilDs/jlBaP419oSfr5wlU6pFfFRi9eGEvHb5AmDKCqt4LF/rGdIdnemn5rrdySRLknF\nQjq8m84azsaSSn75yscMzkjmjBG9/Y4k0uWoG0o6vJgY475LxzC6fzo/mr+ElZvL/I4k0uWoWEin\n0C0hlieuyiO9WzzfmZfP1jLdx1ukPalYSKfRKy2J2VPHs6eqhunzFlFRXet3JJEuQ8VCOpUR/dJ4\n8PKxrP68nBvmL6VO9/EWaRcqFtLpfH14b27/5gjeWr2Nu19b7XcckS5BZ0NJpzTt5ByKSip4/F9F\n5GZ15/ITBvkdSSSqqVhIp2Rm/OybI9hYWsntf17JoIxkTh2W5XcskailbijptOJiY3jo8rEMze7O\n954pYO22PX5HEolaKhbSqaUmxTN7Wh6JcbFcM28RJXur/Y4kEpVULKTTG9AzmSem5rG9vJoZT+ZT\nVaP7eIu0NRULiQpjBvbgvkvHsPiz3cxcsFwXHRRpYyoWEjXOGd2Xn078En9ZtoX73lrrdxyRqKKz\noSSqfO+rR1G0o4IH/r6W3KxkLhw7wO9IIlFBRxYSVcyMX104mhOHZHDTghV8VLTT70giUUHFQqJO\nQlwMj04Zx4Ce3bj2qXw2lFT4HUmk01OxkKjUIzmB2dPG44Br5i6irFL38RY5EioWErVys1J4bMo4\nNu2q5LqnC9hfq/t4ixwuFQuJaicMyeTubx3L++tLuf1PK3VKrchh0tlQEvUuGjeADaUVPPh2IbnZ\nKVz31aP8jiTS6ahYSJfwX2ccTVFJBXe/toaczGQmjurrdySRTkXdUNIlxMQYv/32cYwd1IMfP7eU\nZZt2+x1JpFNRsZAuIyk+lllX5pHVPZHvPJnP5t37/I4k0mmoWEiXkp2ayJxp46naX8f0uYvYU6VT\nakXCoWIhXc7RvVN5+IrjWbt9Lz98dgm1dTqlViQUFQvpkr5ydDZ3nD+Sdz/ZwS9f0X28RULR2VDS\nZV154mA2lFQw+99F5GalMPXkHL8jiXRYKhbSpf3POcewsbSCO/+yikEZyXxteC+/I4l0SBHthjKz\niWb2iZkVmtnNTSxPNLPnvOUfmlmONz/TzN4xs71m9lAkM0rXFhtj/G7yWIb3SeP6Py5m9eflfkcS\n6ZAiVizMLBZ4GDgbGAFcZmYjGjWbDuxyzg0F7gPu8eZXAbcD/x2pfCINUhLjmD0tj+5JcUyfu4jt\ne6r8jiTS4UTyyGICUOicW++c2w/MByY1ajMJmOdNLwBONzNzzlU45/5NoGiIRFzf9G7MnjqeXZU1\nfHdePvv26z7eIsEiWSz6A5uCnhd785ps45yrBcqAzHDfwMxmmFm+meXv2LHjCONKVzeqfzq/mzyG\n5ZvL+MnzS6mv10UHRRp06lNnnXOznHN5zrm87Oxsv+NIFPjGyD7ces4xvLZyK7954xO/44h0GJE8\nG2ozMDDo+QBvXlNtis0sDkgHSiOYSSSk6afmsr6kgkfeXUduZgqXjB8YeiWRKBfJI4tFwDAzyzWz\nBGAysLBRm4XAVG/6YuBtpxsOiM/MjDvPH8mXh2XxPy+v4L11JX5HEvFdxIqFNwZxPfA6sBp43jm3\nyszuMrPzvWazgUwzKwR+Ahw4vdbMNgD3AtPMrLiJM6lEIiY+NoaHLj+enKwUrnuqgHU79vodScRX\nFi1/yOfl5bn8/Hy/Y0iU2bSzkgse/g/dk+J4+funkJGS4HckkTZlZgXOubxQ7Tr1ALdIpA3MSGbW\nVeP4vKyK654qoLpWp9RK16RiIRLCuMEZ/ObiY/low05ueXGF7uMtXZKuDSUShklj+rOxtJJ73/yU\n3KwUfnj6ML8jibQrFQuRMP3w60MpKqng/978lMFZKZx/XD+/I4m0G3VDiYTJzLj7otGMz+nJf7+w\njIKNu/yOJNJuVCxEWiExLpbHrsyjb3oSM57MZ9POSr8jibQLFQuRVspISWDOtPHU1NVz9dxFlO3T\nfbwl+qlYiByGo7K78+iV49hQUsH1f1xMje7jLVFOxULkMJ18VBa/unAU/1pbws8XrtIptRLVdDaU\nyBG4dPwgikoqefQf6xiSlcJ3vjzE70giEaFiIXKEfnrWl9hQUsGvXl3N4MwUzhzR2+9IIm1O3VAi\nRygmxrjv0jGM7p/Oj55dwsrNZX5HEmlzKhYibaBbQixPXJVHz+R4ps9bxNYy3RFYoouKhUgb6ZWW\nxOxp49lbVcv0eYuoqK71O5JIm1GxEGlDx/RN46HLj2f15+XcMH8pdbqPt0QJFQuRNva14b34+Xkj\neWv1Nn796mq/44i0CZ0NJRIBU0/Ooaikgif+XUROVgpTThzsdySRI6JiIRIht517DBtKK/j5wlV8\nVLSTvulJ9E4LPPqkJ9IrNYleaYkkxsX6HVUkJBULkQiJi43hwcvG8tMFy1myaRd/W1XN/tpDLwuS\nkZJAr9SE85PjAAANzUlEQVRE+qQn0Ts1id7pSfROS6RP2hfFJTMlgZgY8+GnEAlQsRCJoNSkeB6Z\nMg4A5xy7K2vYWl7FtvIqtpdXH5gOPKpZtaWckr3VNL5ySFyMkZ2a6BWPQCHplZYUVFAS6Z2eRGpi\nHGYqKtL2VCxE2omZ0TMlgZ4pCRzTN63ZdrV19ezYW8228mq2llWxfU+gmGwtq2b7nirW76jg/XWl\nlFcdemput/hY+qQnfXGkElxMvOKSnZpIUry6vqR1VCxEOpi42Bj6pnejb3o3GNh8u8r9tQcdnTQ+\nUlny2W62llc12fXVMzn+oELS1JFKZvdEYtX1JR4VC5FOKjkhjpysOHKyUppt45yjbF9D11d1oJCU\nVbFtzxdHKqs/D3R9Nf5KSGyMkd090L3VO+hIpfFRS1qSur66AhULkShmZvRITqBHcgLD+zTfrrau\nntKK/WwtO3gMpeFIZUNpBR8W7WzyRk9J8TGNjkwSg45akrxl6vrq7FQsRIS42JgDO/eW7Ntf542h\nBArJ9vKqQIHZU822siqWFe9ma1kV1U10ffVIjv/ibK+GwfpGRy1Z6vrqsFQsRCRs3RJiGZyZwuDM\nlru+yvfVel1dhx6pbC+v4tOte9i+p+qQrq8Yg+zUL8ZQmhpP6ZOWRFo3dX21NxULEWlTZkZ6cjzp\nyfEc3Tu12XZ19Y7SvdUHxlMaH6ls2lnJog072V3ZdNdX77TA91J6BX8npdGRirq+2o6KhYj4IjbG\n6OUdNbSkqqaO7eXVTR6pbCuvYuXmMt5avY2qmkO7vtK7xTcaQzn0SCWrewJxsbpMXigqFiLSoSXF\nxzIoM5lBmcnNtnHOUV5VGzgyCT7zK+hIpXB7Cdv3VB9yJeAYg6zuhxaUA0cq3vP0bvFduutLxUJE\nOj0zI71bPOnd4hkWquuropptZYFi0tD11dANVryrkoKNO9nVRNdXQlxME0cmiQed9dU7LYluCdHZ\n9aViISJdRmyMBS7gmJrEaNKbbVdVU8eOPV8UlG3l1UFHLVV8vKWct1dvZ19N3SHrpibFHXRdr95p\nid636r+YzuqeSHwn6/pSsRARaSQpPpaBGckMzGi562tPtdf15R2pbNvjfenRO1JZvy7Q9VXbqOvL\nDnR9HXqkEjye0jO543R9qViIiBwGMyMtKZ60pHiG9mq+66u+3lFasf+LMZRGRyqbd1ex+LPd7KzY\nf8i6CbEx9E6Np39aHH1T4+mTGk/v7rH0TomjV/c4spNjyE6JpVtyKqT1jeSPq2IhIp2Ec1BfB/W1\nLTzqoK7m4OcHpmuamFcLdU28Rlu1r6shpr6O7PpasutrGXVgeaOMybW4pFrqa2sOeu0YV4dVOagC\ntje/aQpSv864G1+O6OaPaLEws4nA74BY4Ann3N2NlicCTwLjgFLgUufcBm/ZLcB0oA74kXPu9Uhm\nFelU6uvBhdoxHulOsYnXaGpH12L7Zt6zrpn3bKm9O3R8oF3FxLXwiA38Gxt/8POGR1wixKS02N5i\n4oht6fW9tlV1Rvl+R3m1o6zasavKkdxrSMR//IgVCzOLBR4GzgSKgUVmttA593FQs+nALufcUDOb\nDNwDXGpmI4DJwEigH/CWmR3tnN+flghwDlx94F+Cpl299zx4uqEdYbYLfj0XZruGZYTZrvF0Z/85\nCLNdwzLCe+1wd4zh7kQbsvrCgnZyjXeM8YfuKIN2dMTEQXw3SExtecfb5Os3syOObWbHHXb7hjZN\nvGdDDosJDDR0AEneo1c7v28kjywmAIXOufUAZjYfmAQEF4tJwB3e9ALgIQuM5kwC5jvnqoEiMyv0\nXu/9Nk+5bRW8cDW+7JxwzYSSiLIYwAK//AemG3YGQdMHnodq17CMZtpZMztRb8cZ7k631TvFZnbc\nTe0Um23f1KNzncUjbSOSxaI/sCnoeTFwQnNtnHO1ZlYGZHrzP2i0bv/Gb2BmM4AZAIMGDTq8lHFJ\n0Gt4mDsQWrGjiWkIGeaOxg5d1my7FtYJma+p12jLHWYH/TkOeh8Raa1OPcDtnJsFzALIy8s7vD/T\nM4+CS55sy1giIlEnkseTmzn4Pl8DvHlNtjGzOCCdwEB3OOuKiEg7iWSxWAQMM7NcM0sgMGC9sFGb\nhcBUb/pi4G3nnPPmTzazRDPLBYYBH0Uwq4iItCBi3VDeGMT1wOsETp2d45xbZWZ3AfnOuYXAbOAp\nbwB7J4GCgtfueQKD4bXAD6LyTCgRkU7CnIuOM3Ly8vJcfn6+3zFERDoVMytwzuWFaqdz4EREJCQV\nCxERCUnFQkREQlKxEBGRkKJmgNvMdgAbj+AlsoCSNorTlpSrdZSrdZSrdaIx12DnXHaoRlFTLI6U\nmeWHc0ZAe1Ou1lGu1lGu1unKudQNJSIiIalYiIhISCoWX5jld4BmKFfrKFfrKFfrdNlcGrMQEZGQ\ndGQhIiIhqViIiEhIUV8szGyimX1iZoVmdnMTyxPN7Dlv+YdmlhO07BZv/idmdlY75/qJmX1sZsvN\n7O9mNjhoWZ2ZLfUejS/7Hulc08xsR9D7fydo2VQzW+s9pjZeN8K57gvK9KmZ7Q5aFsntNcfMtpvZ\nymaWm5k94OVebmbHBy2L5PYKlesKL88KM3vPzI4LWrbBm7/UzNr06pxh5DrNzMqC/r9+FrSsxc9A\nhHPNDMq00vtMZXjLIrm9BprZO96+YJWZ3dBEm/b5jDnnovZB4NLo64AhQAKwDBjRqM33gUe96cnA\nc970CK99IpDrvU5sO+b6GpDsTX+vIZf3fK+P22sa8FAT62YA671/e3rTPdsrV6P2PyRwSfyIbi/v\ntb8CHA+sbGb5OcBrgAEnAh9GenuFmevkhvcDzm7I5T3fAGT5tL1OA/56pJ+Bts7VqO15BO690x7b\nqy9wvDedCnzaxO9ku3zGov3IYgJQ6Jxb75zbD8wHJjVqMwmY500vAE43M/Pmz3fOVTvnioBC7/Xa\nJZdz7h3nXKX39AMCdwuMtHC2V3POAt50zu10zu0C3gQm+pTrMuDZNnrvFjnn/kngXizNmQQ86QI+\nAHqYWV8iu71C5nLOvee9L7Tf5yuc7dWcI/lstnWu9vx8fe6cW+xN7wFWA/0bNWuXz1i0F4v+wKag\n58UcuqEPtHHO1QJlQGaY60YyV7DpBP5yaJBkZvlm9oGZXdBGmVqT6yLvcHeBmTXc/rZDbC+vuy4X\neDtodqS2Vziayx7J7dVajT9fDnjDzArMbIYPeU4ys2Vm9pqZjfTmdYjtZWbJBHa4LwbNbpftZYEu\n8rHAh40WtctnLGJ3ypO2YWZTgDzgq0GzBzvnNpvZEOBtM1vhnFvXTpH+AjzrnKs2s2sJHJV9vZ3e\nOxyTgQXu4Dsr+rm9OjQz+xqBYnFq0OxTve3VC3jTzNZ4f3m3h8UE/r/2mtk5wJ8I3Fa5ozgP+I9z\nLvgoJOLby8y6EyhQP3bOlbfla4cr2o8sNgMDg54P8OY12cbM4oB0oDTMdSOZCzM7A7gVON85V90w\n3zm32ft3PfAugb822iWXc640KMsTwLhw141kriCTadRFEMHtFY7mskdye4XFzI4l8H84yTlX2jA/\naHttB16m7bpfQ3LOlTvn9nrTrwLxZpZFB9henpY+XxHZXmYWT6BQPOOce6mJJu3zGYvEoExHeRA4\nclpPoFuiYVBsZKM2P+DgAe7nvemRHDzAvZ62G+AOJ9dYAgN6wxrN7wkketNZwFraaKAvzFx9g6Yv\nBD5wXwymFXn5enrTGe2Vy2s3nMBgo7XH9gp6jxyaH7A9l4MHHz+K9PYKM9cgAuNwJzeanwKkBk2/\nB0xsx1x9Gv7/COx0P/O2XVifgUjl8panExjXSGmv7eX97E8C97fQpl0+Y222oTvqg8CZAp8S2PHe\n6s27i8Bf6wBJwAveL85HwJCgdW/11vsEOLudc70FbAOWeo+F3vyTgRXeL8sKYHo75/o1sMp7/3eA\n4UHrXuNtx0Lg6vbM5T2/A7i70XqR3l7PAp8DNQT6hKcD1wHXecsNeNjLvQLIa6ftFSrXE8CuoM9X\nvjd/iLetlnn/z7e2c67rgz5fHxBUzJr6DLRXLq/NNAInvQSvF+ntdSqBMZHlQf9X5/jxGdPlPkRE\nJKRoH7MQEZE2oGIhIiIhqViIiEhIKhYiIhKSioWIiISkYiHSAXhXW/2r3zlEmqNiISIiIalYiLSC\nmU0xs4+8exc8ZmaxZrbXu5/GKgvceyTbazvGu3jhcjN72cx6evOHmtlb3sXyFpvZUd7Ld/cuzrjG\nzJ7xrn4s0iGoWIiEycyOAS4FTnHOjQHqgCsIXOYh3zk3EvgH8HNvlSeBm5xzxxL4Zm3D/GeAh51z\nxxH4hvnn3vyxwI8J3EtlCHBKxH8okTDpqrMi4TudwIUTF3l/9HcDtgP1wHNem6eBl8wsHejhnPuH\nN38e8IKZpQL9nXMvAzjnqgC81/vIOVfsPV9K4FpF/478jyUSmoqFSPgMmOecu+WgmWa3N2p3uNfQ\nqQ6arkO/n9KBqBtKJHx/By727luAmWV4N1uKAS722lwO/Ns5VwbsMrMve/OvBP7hAnc7K264CZMF\n7gGf3K4/hchh0F8uImFyzn1sZrcRuCtaDIErlP4AqAAmeMu2ExjXAJgKPOoVg/XA1d78K4HHzOwu\n7zW+3Y4/hshh0VVnRY6Qme11znX3O4dIJKkbSkREQtKRhYiIhKQjCxERCUnFQkREQlKxEBGRkFQs\nREQkJBULEREJ6f8Ds6UvfTZ1pJkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1251d2a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46.19725, 22.895014)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#the mean score of the high-rank distribution should be higher\n",
    "np.mean(base_network.predict(X_1_test)), np.mean(base_network.predict(X_2_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#AUC\n",
    "(np.sum(base_network.predict(X_1_test) > base_network.predict(X_2_test))+0.0) / X_1_test.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}