docs/notebooks/input_matrices_and_tensors.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy.sparse as sp\n",
    "import smurff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input to SMURFF\n",
    "\n",
    "In this notebook we will look at how to provide input to SMURFF with dense and sparse matrices;\n",
    "\n",
    "SMURFF accepts the following matrix files for train, test and side-info data:\n",
    "\n",
    "* for dense matrix or tensor input:\n",
    "  [numpy.ndarrays](https://docs.scipy.org/doc/numpy-.14.0/reference/generated/numpy.ndarray.html)\n",
    "* for sparse matrices input:\n",
    "  [scipy Sparse matrices](https://docs.scipy.org/doc/scipy/reference/sparse.html) in COO, CSR or CSC format\n",
    "* for sparse tensors: a wrapper around a\n",
    "  [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)\n",
    "\n",
    "Let's have a look on how this could work.\n",
    "\n",
    "#### Dense Train Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dense input\n",
    "Ydense  = np.random.rand(10, 20)\n",
    "session = smurff.TrainSession(burnin = 5, nsamples = 5)\n",
    "session.addTrainAndTest(Ydense)\n",
    "session.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse Matrix Input\n",
    "\n",
    "The so-called *zero* elements in sparse matrices can either represent\n",
    "\n",
    "1. missing values, also called 'unknown' or 'not-available' (NA) values.\n",
    "2. actual zero values, to optimize the space that stores the matrix\n",
    "\n",
    "**Important**:\n",
    "* when calling `addTrainAndTest(Ytrain, Ytest, is_scarce)` the `is_scarce`\n",
    "    refers to the `Ytrain` matrix. `Ytest` is *always* scarce.\n",
    "* when calling `addSideInfo(mode, sideinfoMatrix)` with a sparse `sideinfoMatrix`,\n",
    "    this matrix is always fully known."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sparse matrix input with 20% zeros (fully known)\n",
    "Ysparse = sp.rand(15, 10, 0.2)\n",
    "session = smurff.TrainSession(burnin = 5, nsamples = 5)\n",
    "session.addTrainAndTest(Ysparse, is_scarce = False)\n",
    "session.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sparse matrix input with unknowns (the default)\n",
    "Yscarce = sp.rand(15, 10, 0.2)\n",
    "session = smurff.TrainSession(burnin = 5, nsamples = 5)\n",
    "session.addTrainAndTest(Yscarce, is_scarce = True)\n",
    "session.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Tensor input\n",
    "\n",
    "SMURFF also supports tensor factorization with and without side information on any of the modes. Tensor can be thought as generalization of matrix to relations with more than two items. For example 3-tensor of `drug x cell x gene` could express the effect of a drug on the given cell and gene. In this case the prediction for the element `Yhat[i,j,k]`* is given by\n",
    "\n",
    "$$ \\hat{Y}_{ijk} = \\sum_{d=1}^{D}u^{(1)}_{d,i}u^{(2)}_{d,j}u^{(3)}_{d,k} + mean $$\n",
    "\n",
    "Visually the model can be represented as follows:\n",
    "\n",
    "![Tensor Model Visualization](tensor-model.png)\n",
    "\n",
    "Tensor model predicts `Yhat[i,j,k]` by multiplying all latent vectors together element-wise and then taking the sum along the latent dimension (figure omits the global mean).\n",
    "\n",
    "For tensors SMURFF implements a `SparseTensor` class. `SparseTensor` is a wrapper around a pandas `DataFrame` where each row stores the coordinate and the value of a known cell in the tensor. Specifically, the integer columns in the DataFrame give the coordinate of the cell and `float` (or double) column stores the value in the cell (the order of the columns does not matter). The coordinates are 0-based. The shape of the `SparseTensor` can be provided, otherwise it is inferred from the maximum index in each mode.\n",
    "\n",
    "Here is a simple toy example with factorizing a 3-tensor with side information on the first mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.sparse\n",
    "import smurff\n",
    "import itertools\n",
    "\n",
    "## generating toy data\n",
    "A = np.random.randn(15, 2)\n",
    "B = np.random.randn(3, 2)\n",
    "C = np.random.randn(2, 2)\n",
    "\n",
    "idx = list( itertools.product(np.arange(A.shape[0]),\n",
    "                              np.arange(B.shape[0]),\n",
    "                              np.arange(C.shape[0])) )\n",
    "df  = pd.DataFrame( np.asarray(idx), columns=[\"A\", \"B\", \"C\"])\n",
    "df[\"value\"] = np.array([ np.sum(A[i[0], :] * B[i[1], :] * C[i[2], :]) for i in idx ])\n",
    "\n",
    "## assigning 20% of the cells to test set\n",
    "Ytrain, Ytest = smurff.make_train_test_df(df, 0.2)\n",
    "\n",
    "print(\"Ytrain = \", Ytrain)\n",
    "\n",
    "## for artificial dataset using small values for burnin, nsamples and num_latents is fine\n",
    "predictions = smurff.BPMFSession(\n",
    "                        Ytrain=Ytrain,\n",
    "                        Ytest=Ytest,\n",
    "                        num_latent=4,\n",
    "                        burnin=20,\n",
    "                        nsamples=20).run()\n",
    "\n",
    "print(\"First prediction of Ytest tensor: \", predictions[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}