Deep Semi-NMF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep Semi-NMF demo the CMU PIE Pose dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sklearn\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from dsnmf import DSNMF, appr_seminmf\n",
    "from scipy.io import loadmat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mat = loadmat('PIE_pose27.mat', struct_as_record=False, squeeze_me=True)\n",
    "\n",
    "data, gnd = mat['fea'].astype('float32'), mat['gnd']\n",
    "\n",
    "# Normalise each feature to have an l2-norm equal to one.\n",
    "data /= np.linalg.norm(data, 2, 1)[:, None] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In  order to evaluate the different features we will use a simple k-means clustering with the only assumption of knowing the true number of classes existing in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_classes = np.unique(gnd).shape[0]\n",
    "kmeans = KMeans(n_classes, precompute_distances=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the cluster indicators for each data sample we then use the normalised mutual information score to evalutate the similarity between the predicted labels and the ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_nmi(X):\n",
    "    pred = kmeans.fit_predict(X)\n",
    "    score = sklearn.metrics.normalized_mutual_info_score(gnd, pred)\n",
    "    \n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will perform k-means clustering on the raw feature space.\n",
    "\n",
    "It will take some time, depending on your setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K-means on the raw pixels has an NMI of 39.62%\n"
     ]
    }
   ],
   "source": [
    "print(\"K-means on the raw pixels has an NMI of {:.2f}%\".format(100 * evaluate_nmi(data)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K-means clustering using the top 100 eigenvectors has an NMI of 6.10%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "fea = PCA(100).fit_transform(data)\n",
    "score = evaluate_nmi(fea)\n",
    "\n",
    "print(\"K-means clustering using the top 100 eigenvectors has an NMI of {:.2f}%\".format(100 * score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use a single layer DSNMF model -- i.e. Semi-NMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Semi-NMF factorisation decomposes the original data-matrix \n",
    "\n",
    "$$\\mathbf X \\approx \\mathbf Z \\mathbf H$$\n",
    "\n",
    "subject to the elements of H being non-negative. The objective function of Semi-NMF is closely related \n",
    "to the one of K-means clustering.  In fact, if we had a matrix ${\\mathbf H}$ that was comprised only by zeros and ones (i.e. a binary matrix) then this would be exactly equivalent to K-means clustering. Instead, Semi-NMF only forces the elements to be non-negative and thus can be seen as a soft clustering method where the features matrix describes the compatibility of each component with a cluster centroid, a base in $\\mathbf Z$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Z, H = appr_seminmf(data.T, 100) # seminmf expects a num_features x num_samples matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K-means clustering using the Semi-NMF features has an NMI of 82.18%\n"
     ]
    }
   ],
   "source": [
    "print(\"K-means clustering using the Semi-NMF features has an NMI of {:.2f}%\".format(100 * evaluate_nmi(H.T)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad! That's a huge improvement over using k-means\n",
    "on the raw pixels!\n",
    "\n",
    "Let's try doing the same with a Deep Semi-NMF model with more than one \n",
    "layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### Initialize a Deep Semi-NMF model with 2 layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Semi-NMF the goal is to construct a low-dimensional representation $\\mathbf H^+$ of our original data $\\mathbf X^\\pm$, with the bases matrix $\\mathbf Z^\\pm$ serving as the mapping between our original data and its lower-dimensional representation.\n",
    "\n",
    "In many cases the data we wish to analyse is often rather complex and has a collection of distinct, often unknown, attributes. In this example, we deal with datasets of human faces where the variability in the data does not only stem from the difference in the appearance of the subjects, but also from other attributes, such as the pose of the head in relation to the camera, or the facial expression of the subject. The multi-attribute nature of our data calls for a hierarchical framework that is better at representing it than a shallow Semi-NMF.\n",
    "\n",
    "$$ \\mathbf X^{\\pm} \\approx {\\mathbf Z}_1^{\\pm}{\\mathbf Z}_2^{\\pm}\\cdots{\\mathbf Z}_m^{\\pm}{\\mathbf H}^+_m $$\n",
    "\n",
    "In this example we have a 2-layer network ($m=2$), with $\\mathbf Z_1 \\in \\mathbb{R}^{1024\\times 400}$, $\\mathbf Z_2 \\in \\mathbb{R}^{400 \\times 100}$, and $\\mathbf H_2 \\in \\mathbb{R}^{100 \\times 2856}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "dsnmf = DSNMF(data, layers=(400, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "for epoch in range(1000):\n",
    "    residual = float(dsnmf.train_fun())\n",
    "    \n",
    "    print(\"Epoch {}. Residual [{:.2f}]\".format(epoch, residual), end=\"\\r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate it in terms of clustering performance using \n",
    "the normalised mutual information score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NMI: 98.25%\n"
     ]
    }
   ],
   "source": [
    "fea = dsnmf.get_features().T # this is the last layers features i.e. h_2\n",
    "pred = kmeans.fit_predict(fea)\n",
    "score = sklearn.metrics.normalized_mutual_info_score(gnd, pred)\n",
    "\n",
    "print(\"NMI: {:.2f}%\".format(100 * score))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}