[en]User-based Collaborative Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User-based Collaborative filtering\n",
    "\n",
    "This notebook demonstrates a basic Collaborative filtering technuques implemented with MovieLens Datasets.\n",
    "\n",
    "* Predict unknown rating with a sparse user-item matrix\n",
    "* Measure similarity with 'Cosine similarity' and 'Pearson correlation coefficient'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.spatial.distance import cosine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and preprocess the datasets\n",
    "\n",
    "We use 100K MovieLens datasets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kwoncheol/deep/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/kwoncheol/deep/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    }
   ],
   "source": [
    "path = './datasets/'\n",
    "ratings = np.array(pd.read_csv(path + \"ratings.csv\"))#.astype(int)\n",
    "movies = np.array(pd.read_csv(path + \"movies.csv\"))\n",
    "\n",
    "u_count = int(max(ratings[:,0])+1) # number of users\n",
    "m_count = len(movies) # number of movies\n",
    "\n",
    "# There are 9125 movies in this Datasets.\n",
    "# But the range of Movie ID is between 1 and 164979\n",
    "# We are going to make a movie ID dictionary\n",
    "\n",
    "mv_idx = {}\n",
    "\n",
    "for i in range(m_count):\n",
    "    mv_idx[movies.T[0][i]] = i\n",
    "    \n",
    "# Create user-item rating matrix \n",
    "r_mat = np.zeros([u_count,m_count])\n",
    "\n",
    "for i in range(ratings.shape[0]):\n",
    "    r_mat[int(ratings[i][0]),mv_idx[ratings[i][1]]] = ratings[i][2]\n",
    "    \n",
    "# Mean of each user's rating\n",
    "r_mean = np.zeros([u_count])\n",
    "\n",
    "for i in range(u_count):\n",
    "    r_mean[i] = np.mean(r_mat[i,np.nonzero(r_mat[i])]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize technique 1\n",
    "\n",
    "Extract mean of each user from each user's rating values.\n",
    "\n",
    "It prevents handling un-rated item as a negative value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(u_count):\n",
    "    r_mat[i,r_mat[i] != 0] = r_mat[i,r_mat[i] != 0] - r_mean[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize technuque 2\n",
    "\n",
    "Set values bigger than 3 as 1\n",
    "    values smaller than 3 or not rated as 0.\n",
    "In this notebook we are not going to use this technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#for i in range(self.ratings.shape[0]):\n",
    "#    self.r_mat[self.ratings[i][0],self.ratings[i][1]] = 0 if self.ratings[i][2]>=3 else 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Pearson Correlation measure function\n",
    "\n",
    "There is a PCC function in Scipy but we will define our own PCC function.\n",
    "It receives indices of two users and returns their PCC value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_pears(a, b): \n",
    "    '''\n",
    "    Pearson Correlation Coefficient function\n",
    "        \n",
    "    Arguments:\n",
    "        a - Index of user A\n",
    "        b - Index of user B\n",
    "\n",
    "    '''\n",
    "    # Transpose the rating matrix for easy to find the items that users(a,b) have rated.\n",
    "    \n",
    "    aidx = ratings.T[0] == a\n",
    "    bidx = ratings.T[0] == b\n",
    "\n",
    "    indices = np.intersect1d(ratings[aidx][:,1],ratings[bidx][:,1]).astype(int)\n",
    "        \n",
    "    # Return 0 if no intersect\n",
    "\n",
    "    if(len(indices) == 0):\n",
    "        return 0\n",
    "    \n",
    "    br_idx = []\n",
    "    for i in range(len(indices)):\n",
    "        br_idx.append(mv_idx[indices[i]])\n",
    "        \n",
    "    br_idx = np.array(br_idx)\n",
    "\n",
    "    sim = np.sum((r_mat[a,br_idx]-r_mean[a])*(r_mat[b,br_idx]-r_mean[b])) \\\n",
    "        / (np.sqrt(np.sum(np.square(r_mat[a,br_idx]-r_mean[a]))) \\\n",
    "        * np.sqrt(np.sum(np.square(r_mat[b,br_idx]-r_mean[b]))))\n",
    "\n",
    "    return sim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict users' unknown ratings\n",
    "\n",
    "Define function that predicts rating of an user and a movie.\n",
    "1. Measure similarities of users who have rated movie what we want to predict. Then select K most similar users to our user.\n",
    "2. Predict our user's rating based on K most similar users' ratings.\n",
    "\n",
    "### There are two predicting methods\n",
    "1) Divide Sum of ratings of K most similar user by K\n",
    "\n",
    "2) Divide Sum of product of each user's similiarties and ratings by sum of similarities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_score(u, mov, k=10, sim_met='pearson'):\n",
    "    '''\n",
    "    Prediction for item 'mov' of user 'u'\n",
    "        \n",
    "    Arguments:\n",
    "        u - user\n",
    "        mov - Movie's index     \n",
    "        k - Number of similar users for prediction (Default = 10)         \n",
    "        sim_met - similarity measure method (Default = 'pearson')\n",
    "  \n",
    "    '''\n",
    "    \n",
    "    # Find users who have rated 'mov'\n",
    "    rated_u = np.array(np.nonzero(r_mat[:,mov]))\n",
    "        \n",
    "    if rated_u.shape[1] == 0 or (rated_u.shape[1] == 1 and rated_u[0,0] == u):\n",
    "        #print(\"No users who have rated this movie\")\n",
    "        return (0,0)\n",
    "        \n",
    "    rated_u = rated_u.flatten()\n",
    "        \n",
    "    sims = {}\n",
    "    for i in range(rated_u.shape[0]):\n",
    "        if rated_u[i] == u:\n",
    "            continue\n",
    "            \n",
    "        if sim_met is 'pearson':\n",
    "            sim = my_pears(u,rated_u[i])\n",
    "        else:\n",
    "            sim = cosine(r_mat[u,:],r_mat[rated_u[i],:])\n",
    "\n",
    "        if len(sims) < k:\n",
    "            sims[rated_u[i]] = sim\n",
    "\n",
    "        # If the similarity of a current user is smaller than the minimum similarity of 'Sims',\n",
    "        # then replace minimum user with current user.\n",
    "        elif sim > sims[min_val]:\n",
    "            sims.pop(min_val)\n",
    "            sims[rated_u[i]] = sim\n",
    "\n",
    "        min_val = min(sims, key=lambda k:sims[k])\n",
    "\n",
    "\n",
    "    p_ver_1 = np.sum(r_mat[[*sims],mov]) / (rated_u.shape[0] if rated_u.shape[0] < k else k)\n",
    "    p_ver_2 = np.sum(r_mat[[*sims],mov] * list(sims.values())) / np.sum(list(sims.values()))\n",
    "\n",
    "    return p_ver_1, p_ver_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "Let's predict known ratings of 'user 3'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  56,  100,  219,  239,  266,  284,  320,  321,  341,  472,  521,\n",
       "        524,  525,  527,  617,  642,  699,  954,  966,  990, 1025, 1118,\n",
       "       1253, 1359, 1455, 1590, 1834, 2010, 2156, 2162, 2173, 2212, 2273,\n",
       "       2288, 2374, 2599, 2804, 3157, 4085, 4259, 4610, 5026, 5127, 5480,\n",
       "       5485, 5904, 6383, 6557, 6601, 6916, 7733])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure items user 3 have rated out\n",
    "rated_list = np.array(np.nonzero(r_mat[3,:])).flatten()\n",
    "rated_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict rating of Movie 56 with a 'pearson' methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting values : \t (-0.3416791353532814, -0.3458529191646994)\n",
      "Real rating value :\t -0.5686274509803924\n"
     ]
    }
   ],
   "source": [
    "# pearson\n",
    "print('Predicting values : \\t',predict_score(3,56,sim_met='pearson'))\n",
    "print('Real rating value :\\t', r_mat[3,56])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict rating of Movie 56 with a 'cosine' methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting values : \t (-1.3883074678646374, -1.383321162066603)\n",
      "Real rating value :\t -0.5686274509803924\n"
     ]
    }
   ],
   "source": [
    "# cosine\n",
    "print('Predicting values : \\t',predict_score(3,56,sim_met='cosine'))\n",
    "print('Real rating value :\\t', r_mat[3,56])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pearson method returns a bit more close result.  Let's do on movie 266."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting values : \t (0.6374409153924766, 0.6373943411987764)\n",
      "Real rating value :\t 0.9313725490196076\n"
     ]
    }
   ],
   "source": [
    "# pearson\n",
    "print('Predicting values : \\t',predict_score(3,266,sim_met='pearson'))\n",
    "print('Real rating value :\\t', r_mat[3,266])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting values : \t (-0.5139090122865949, -0.5227894524126228)\n",
      "Real rating value :\t 0.9313725490196076\n"
     ]
    }
   ],
   "source": [
    "#cosine\n",
    "print('Predicting values : \\t',predict_score(3,266,sim_met='cosine'))\n",
    "print('Real rating value :\\t', r_mat[3,266])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PCC made a better prediction again.\n",
    "#### Now we are going to measure its error more precisely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pearson_ver1_errors = 0\n",
    "pearson_ver2_errors = 0\n",
    "cosine_ver1_errors = 0\n",
    "cosine_ver2_errors = 0\n",
    "\n",
    "for i in range(u_count):\n",
    "    rated_mov = np.array(np.nonzero(r_mat[i,:])).flatten()\n",
    "    \n",
    "    for j in rated_mov:\n",
    "        pred_val = predict_score(i,j,sim_met='pearson')\n",
    "        \n",
    "        pearson_ver1_errors = pearson_ver1_errors + np.sqrt(np.square(pred_val[0] - r_mat[i,j]))\n",
    "        pearson_ver2_errors = pearson_ver2_errors + np.sqrt(np.square(pred_val[1] - r_mat[i,j]))\n",
    "        \n",
    "        pred_val = predict_score(i,j,sim_met='cosine')\n",
    "        \n",
    "        cosine_ver1_errors = cosine_ver1_errors + np.sqrt(np.square(pred_val[0] - r_mat[i,j]))\n",
    "        cosine_ver2_errors = cosine_ver2_errors + np.sqrt(np.square(pred_val[1] - r_mat[i,j]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine: \n",
      "\tver 1: 86580\n",
      "\tver 2: 87551\n"
     ]
    }
   ],
   "source": [
    "print('cosine: \\n\\tver 1: %d\\n\\tver 2: %d'%(cosine_ver1_errors, cosine_ver2_errors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pearson: \n",
      "\tver 1: 63984\n",
      "\tver 2: 64757\n"
     ]
    }
   ],
   "source": [
    "print('pearson: \\n\\tver 1: %d\\n\\tver 2: %d'%(pearson_ver1_errors,pearson_ver2_errors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pearson method shows a far better results.\n",
    "#### And It seems there is no that significant difference between two predict methods we used ."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deep",
   "language": "python",
   "name": "deep"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}