Chapter-14/14.2 MovieLens 1M Dataset（MovieLens 1M数据集）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.2 MovieLens 1M Dataset（MovieLens 1M数据集）\n",
    "\n",
    "这个数据集是电影评分数据：包括电影评分，电影元数据（风格类型，年代）以及关于用户的人口统计学数据（年龄，邮编，性别，职业等）。\n",
    "\n",
    "MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。分为三个表：评分，用户信息，电影信息。这些数据都是dat文件格式，可以通过pandas.read_table将各个表分别读到一个pandas DataFrame对象中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Make display smaller\n",
    "pd.options.display.max_rows = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/xu/anaconda/envs/py35/lib/python3.5/site-packages/ipykernel/__main__.py:3: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.\n",
      "  app.launch_new_instance()\n"
     ]
    }
   ],
   "source": [
    "unames = ['user_id', 'gender', 'age', 'occupation', 'zip']\n",
    "users = pd.read_table('../datasets/movielens/users.dat', sep='::', \n",
    "                      header=None, names=unames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为sep='::'有点像是正则表达式，于是有了上面的错误。在这个[帖子](https://stackoverflow.com/questions/27301477/python-file-path-failing-in-pycharm-regex-confusion)找到了解决方法，设置engine为python即可。\n",
    "\n",
    "Looks like on Python 2.7 Pandas just doesn't handle separators that look regexish. The initial \"error\" can be worked around by adding engine='python' as a named parameter in the call, as suggested in the warning.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "users = pd.read_table('../datasets/movielens/users.dat', sep='::', \n",
    "                      header=None, names=unames, engine='python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rnames = ['user_id', 'movie_id', 'rating', 'timestamp']\n",
    "ratings = pd.read_table('../datasets/movielens/ratings.dat', sep='::', header=None, names=rnames, engine='python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnames = ['movie_id', 'title', 'genres']\n",
    "movies = pd.read_table('../datasets/movielens/movies.dat', sep='::', header=None, names=mnames, engine='python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载前几行验证一下数据加载工作是否顺利"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id gender  age  occupation    zip\n",
       "0        1      F    1          10  48067\n",
       "1        2      M   56          16  70072\n",
       "2        3      M   25          15  55117\n",
       "3        4      M   45           7  02460\n",
       "4        5      M   25          20  55455"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp\n",
       "0        1      1193       5  978300760\n",
       "1        1       661       3  978302109\n",
       "2        1       914       3  978301968\n",
       "3        1      3408       4  978300275\n",
       "4        1      2355       5  978824291"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                               title                        genres\n",
       "0         1                    Toy Story (1995)   Animation|Children's|Comedy\n",
       "1         2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2         3             Grumpier Old Men (1995)                Comedy|Romance\n",
       "3         4            Waiting to Exhale (1995)                  Comedy|Drama\n",
       "4         5  Father of the Bride Part II (1995)                        Comedy"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，年龄和职业是以编码形式给出的，它们的具体含义请参考改数据集的REAMDE文件。分析散布在三个表中的数据不是一件轻松的事情。假设我们想要根据性别和年龄来计算某部电影的平均得分，如果将所有的数据都合并到一个表中的话，问题就简单多了。我们先用pandas的merge函数将ratings和users合并到一起，然后再将movies也合并进去。pandas会根据列名的重叠情况推断出哪些列是合并（或连接）键："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.merge(pd.merge(ratings, users), movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978298413</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978220179</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>12</td>\n",
       "      <td>32793</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978199279</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>22903</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978158471</td>\n",
       "      <td>M</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>95350</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp gender  age  occupation    zip  \\\n",
       "0        1      1193       5  978300760      F    1          10  48067   \n",
       "1        2      1193       5  978298413      M   56          16  70072   \n",
       "2       12      1193       4  978220179      M   25          12  32793   \n",
       "3       15      1193       4  978199279      M   25           7  22903   \n",
       "4       17      1193       5  978158471      M   50           1  95350   \n",
       "\n",
       "                                    title genres  \n",
       "0  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "1  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "2  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "3  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "4  One Flew Over the Cuckoo's Nest (1975)  Drama  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id                                            1\n",
       "movie_id                                        1193\n",
       "rating                                             5\n",
       "timestamp                                  978300760\n",
       "gender                                             F\n",
       "age                                                1\n",
       "occupation                                        10\n",
       "zip                                            48067\n",
       "title         One Flew Over the Cuckoo's Nest (1975)\n",
       "genres                                         Drama\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，只要稍微熟悉一下pandas，就能轻松地根据任意个用户或电影属性对评分数据进行聚合操作了。为了按性别计算每部电影的平均得分，我们可以使用pivot_table方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_ratings = data.pivot_table('rating', index='title',\n",
    "                                columns='gender', aggfunc='mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>$1,000,000 Duck (1971)</th>\n",
       "      <td>3.375000</td>\n",
       "      <td>2.761905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'Night Mother (1986)</th>\n",
       "      <td>3.388889</td>\n",
       "      <td>3.352941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'Til There Was You (1997)</th>\n",
       "      <td>2.675676</td>\n",
       "      <td>2.733333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'burbs, The (1989)</th>\n",
       "      <td>2.793478</td>\n",
       "      <td>2.962085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...And Justice for All (1979)</th>\n",
       "      <td>3.828571</td>\n",
       "      <td>3.689024</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gender                                F         M\n",
       "title                                            \n",
       "$1,000,000 Duck (1971)         3.375000  2.761905\n",
       "'Night Mother (1986)           3.388889  3.352941\n",
       "'Til There Was You (1997)      2.675676  2.733333\n",
       "'burbs, The (1989)             2.793478  2.962085\n",
       "...And Justice for All (1979)  3.828571  3.689024"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_ratings[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该操作产生了另一个DataFrame，其内容为电影平均得分，行标为电影名称，列表为性别。现在，我们打算过滤掉评分数据不够250条的电影（这个数字可以自己设定）。为了达到这个目的，我们先对title进行分组，然后利用size()得到一个含有各电影分组大小的Series对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ratings_by_title = data.groupby('title').size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "title\n",
       "$1,000,000 Duck (1971)                37\n",
       "'Night Mother (1986)                  70\n",
       "'Til There Was You (1997)             52\n",
       "'burbs, The (1989)                   303\n",
       "...And Justice for All (1979)        199\n",
       "1-900 (1994)                           2\n",
       "10 Things I Hate About You (1999)    700\n",
       "101 Dalmatians (1961)                565\n",
       "101 Dalmatians (1996)                364\n",
       "12 Angry Men (1957)                  616\n",
       "dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_by_title[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "active_titles = ratings_by_title.index[ratings_by_title >= 250]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index([''burbs, The (1989)', '10 Things I Hate About You (1999)',\n",
      "       '101 Dalmatians (1961)', '101 Dalmatians (1996)', '12 Angry Men (1957)',\n",
      "       '13th Warrior, The (1999)', '2 Days in the Valley (1996)',\n",
      "       '20,000 Leagues Under the Sea (1954)', '2001: A Space Odyssey (1968)',\n",
      "       '2010 (1984)',\n",
      "       ...\n",
      "       'X-Men (2000)', 'Year of Living Dangerously (1982)',\n",
      "       'Yellow Submarine (1968)', 'You've Got Mail (1998)',\n",
      "       'Young Frankenstein (1974)', 'Young Guns (1988)',\n",
      "       'Young Guns II (1990)', 'Young Sherlock Holmes (1985)',\n",
      "       'Zero Effect (1998)', 'eXistenZ (1999)'],\n",
      "      dtype='object', name='title', length=1216)\n"
     ]
    }
   ],
   "source": [
    "print(active_titles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的active_titles中的电影，都是评论是大于250条以上的。我们可以用这些标题作为索引，从mean_ratings中选出这些评论大于250条的电影："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>'burbs, The (1989)</th>\n",
       "      <td>2.793478</td>\n",
       "      <td>2.962085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10 Things I Hate About You (1999)</th>\n",
       "      <td>3.646552</td>\n",
       "      <td>3.311966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101 Dalmatians (1961)</th>\n",
       "      <td>3.791444</td>\n",
       "      <td>3.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101 Dalmatians (1996)</th>\n",
       "      <td>3.240000</td>\n",
       "      <td>2.911215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12 Angry Men (1957)</th>\n",
       "      <td>4.184397</td>\n",
       "      <td>4.328421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Young Guns (1988)</th>\n",
       "      <td>3.371795</td>\n",
       "      <td>3.425620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Young Guns II (1990)</th>\n",
       "      <td>2.934783</td>\n",
       "      <td>2.904025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Young Sherlock Holmes (1985)</th>\n",
       "      <td>3.514706</td>\n",
       "      <td>3.363344</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zero Effect (1998)</th>\n",
       "      <td>3.864407</td>\n",
       "      <td>3.723140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eXistenZ (1999)</th>\n",
       "      <td>3.098592</td>\n",
       "      <td>3.289086</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1216 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "gender                                    F         M\n",
       "title                                                \n",
       "'burbs, The (1989)                 2.793478  2.962085\n",
       "10 Things I Hate About You (1999)  3.646552  3.311966\n",
       "101 Dalmatians (1961)              3.791444  3.500000\n",
       "101 Dalmatians (1996)              3.240000  2.911215\n",
       "12 Angry Men (1957)                4.184397  4.328421\n",
       "...                                     ...       ...\n",
       "Young Guns (1988)                  3.371795  3.425620\n",
       "Young Guns II (1990)               2.934783  2.904025\n",
       "Young Sherlock Holmes (1985)       3.514706  3.363344\n",
       "Zero Effect (1998)                 3.864407  3.723140\n",
       "eXistenZ (1999)                    3.098592  3.289086\n",
       "\n",
       "[1216 rows x 2 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_ratings = mean_ratings.loc[active_titles]\n",
    "mean_ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "想要查看女性观众喜欢的电影，可以按F列进行降序操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Close Shave, A (1995)</th>\n",
       "      <td>4.644444</td>\n",
       "      <td>4.473795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wrong Trousers, The (1993)</th>\n",
       "      <td>4.588235</td>\n",
       "      <td>4.478261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)</th>\n",
       "      <td>4.572650</td>\n",
       "      <td>4.464589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wallace &amp; Gromit: The Best of Aardman Animation (1996)</th>\n",
       "      <td>4.563107</td>\n",
       "      <td>4.385075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Schindler's List (1993)</th>\n",
       "      <td>4.562602</td>\n",
       "      <td>4.491415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shawshank Redemption, The (1994)</th>\n",
       "      <td>4.539075</td>\n",
       "      <td>4.560625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grand Day Out, A (1992)</th>\n",
       "      <td>4.537879</td>\n",
       "      <td>4.293255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>To Kill a Mockingbird (1962)</th>\n",
       "      <td>4.536667</td>\n",
       "      <td>4.372611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Creature Comforts (1990)</th>\n",
       "      <td>4.513889</td>\n",
       "      <td>4.272277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Usual Suspects, The (1995)</th>\n",
       "      <td>4.513317</td>\n",
       "      <td>4.518248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gender                                                     F         M\n",
       "title                                                                 \n",
       "Close Shave, A (1995)                               4.644444  4.473795\n",
       "Wrong Trousers, The (1993)                          4.588235  4.478261\n",
       "Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)       4.572650  4.464589\n",
       "Wallace & Gromit: The Best of Aardman Animation...  4.563107  4.385075\n",
       "Schindler's List (1993)                             4.562602  4.491415\n",
       "Shawshank Redemption, The (1994)                    4.539075  4.560625\n",
       "Grand Day Out, A (1992)                             4.537879  4.293255\n",
       "To Kill a Mockingbird (1962)                        4.536667  4.372611\n",
       "Creature Comforts (1990)                            4.513889  4.272277\n",
       "Usual Suspects, The (1995)                          4.513317  4.518248"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)\n",
    "top_female_ratings[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Measuring Rating Disagreement（计算评分分歧）\n",
    "\n",
    "假设我们想要找出男性和女性观众分歧最大的电影。一个办法是给mean_ratings加上一个用于存放平均得分之差的列，并对其进行排序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "按‘diff’排序即可得到分歧最大且女性观众更喜欢的电影："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "      <th>diff</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dirty Dancing (1987)</th>\n",
       "      <td>3.790378</td>\n",
       "      <td>2.959596</td>\n",
       "      <td>-0.830782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jumpin' Jack Flash (1986)</th>\n",
       "      <td>3.254717</td>\n",
       "      <td>2.578358</td>\n",
       "      <td>-0.676359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grease (1978)</th>\n",
       "      <td>3.975265</td>\n",
       "      <td>3.367041</td>\n",
       "      <td>-0.608224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Little Women (1994)</th>\n",
       "      <td>3.870588</td>\n",
       "      <td>3.321739</td>\n",
       "      <td>-0.548849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steel Magnolias (1989)</th>\n",
       "      <td>3.901734</td>\n",
       "      <td>3.365957</td>\n",
       "      <td>-0.535777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French Kiss (1995)</th>\n",
       "      <td>3.535714</td>\n",
       "      <td>3.056962</td>\n",
       "      <td>-0.478752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Little Shop of Horrors, The (1960)</th>\n",
       "      <td>3.650000</td>\n",
       "      <td>3.179688</td>\n",
       "      <td>-0.470312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Guys and Dolls (1955)</th>\n",
       "      <td>4.051724</td>\n",
       "      <td>3.583333</td>\n",
       "      <td>-0.468391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mary Poppins (1964)</th>\n",
       "      <td>4.197740</td>\n",
       "      <td>3.730594</td>\n",
       "      <td>-0.467147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Patch Adams (1998)</th>\n",
       "      <td>3.473282</td>\n",
       "      <td>3.008746</td>\n",
       "      <td>-0.464536</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>15 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "gender                                     F         M      diff\n",
       "title                                                           \n",
       "Dirty Dancing (1987)                3.790378  2.959596 -0.830782\n",
       "Jumpin' Jack Flash (1986)           3.254717  2.578358 -0.676359\n",
       "Grease (1978)                       3.975265  3.367041 -0.608224\n",
       "Little Women (1994)                 3.870588  3.321739 -0.548849\n",
       "Steel Magnolias (1989)              3.901734  3.365957 -0.535777\n",
       "...                                      ...       ...       ...\n",
       "French Kiss (1995)                  3.535714  3.056962 -0.478752\n",
       "Little Shop of Horrors, The (1960)  3.650000  3.179688 -0.470312\n",
       "Guys and Dolls (1955)               4.051724  3.583333 -0.468391\n",
       "Mary Poppins (1964)                 4.197740  3.730594 -0.467147\n",
       "Patch Adams (1998)                  3.473282  3.008746 -0.464536\n",
       "\n",
       "[15 rows x 3 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_by_diff = mean_ratings.sort_values(by='diff')\n",
    "sorted_by_diff[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对行进行反序操作，并取出前15行，得到的则是男性更喜欢，而女性评价较低的电影："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "      <th>diff</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Good, The Bad and The Ugly, The (1966)</th>\n",
       "      <td>3.494949</td>\n",
       "      <td>4.221300</td>\n",
       "      <td>0.726351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Kentucky Fried Movie, The (1977)</th>\n",
       "      <td>2.878788</td>\n",
       "      <td>3.555147</td>\n",
       "      <td>0.676359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dumb &amp; Dumber (1994)</th>\n",
       "      <td>2.697987</td>\n",
       "      <td>3.336595</td>\n",
       "      <td>0.638608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Longest Day, The (1962)</th>\n",
       "      <td>3.411765</td>\n",
       "      <td>4.031447</td>\n",
       "      <td>0.619682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cable Guy, The (1996)</th>\n",
       "      <td>2.250000</td>\n",
       "      <td>2.863787</td>\n",
       "      <td>0.613787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Evil Dead II (Dead By Dawn) (1987)</th>\n",
       "      <td>3.297297</td>\n",
       "      <td>3.909283</td>\n",
       "      <td>0.611985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hidden, The (1987)</th>\n",
       "      <td>3.137931</td>\n",
       "      <td>3.745098</td>\n",
       "      <td>0.607167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rocky III (1982)</th>\n",
       "      <td>2.361702</td>\n",
       "      <td>2.943503</td>\n",
       "      <td>0.581801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Caddyshack (1980)</th>\n",
       "      <td>3.396135</td>\n",
       "      <td>3.969737</td>\n",
       "      <td>0.573602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For a Few Dollars More (1965)</th>\n",
       "      <td>3.409091</td>\n",
       "      <td>3.953795</td>\n",
       "      <td>0.544704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gender                                         F         M      diff\n",
       "title                                                               \n",
       "Good, The Bad and The Ugly, The (1966)  3.494949  4.221300  0.726351\n",
       "Kentucky Fried Movie, The (1977)        2.878788  3.555147  0.676359\n",
       "Dumb & Dumber (1994)                    2.697987  3.336595  0.638608\n",
       "Longest Day, The (1962)                 3.411765  4.031447  0.619682\n",
       "Cable Guy, The (1996)                   2.250000  2.863787  0.613787\n",
       "Evil Dead II (Dead By Dawn) (1987)      3.297297  3.909283  0.611985\n",
       "Hidden, The (1987)                      3.137931  3.745098  0.607167\n",
       "Rocky III (1982)                        2.361702  2.943503  0.581801\n",
       "Caddyshack (1980)                       3.396135  3.969737  0.573602\n",
       "For a Few Dollars More (1965)           3.409091  3.953795  0.544704"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reverse order of rows, take first 10 rows\n",
    "sorted_by_diff[::-1][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果只是想要找出分歧最大的电影（不考虑性别因素），则可以计算得分数据的方差或标准差："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 根据电影名称分组的得分数据的标准差\n",
    "rating_std_by_title = data.groupby('title')['rating'].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 根据active_titles进行过滤\n",
    "rating_std_by_title = rating_std_by_title.loc[active_titles]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "title\n",
       "Dumb & Dumber (1994)                     1.321333\n",
       "Blair Witch Project, The (1999)          1.316368\n",
       "Natural Born Killers (1994)              1.307198\n",
       "Tank Girl (1995)                         1.277695\n",
       "Rocky Horror Picture Show, The (1975)    1.260177\n",
       "Eyes Wide Shut (1999)                    1.259624\n",
       "Evita (1996)                             1.253631\n",
       "Billy Madison (1995)                     1.249970\n",
       "Fear and Loathing in Las Vegas (1998)    1.246408\n",
       "Bicentennial Man (1999)                  1.245533\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Order Series by value in descending order\n",
    "rating_std_by_title.sort_values(ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们注意到，电影分类是以竖线`|`分割的字符串形式给出的。如果想对不同的电影分类进行分析的话，就需要先将其转换成更有用的形式才行。"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [py35]",
   "language": "python",
   "name": "Python [py35]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}