Weekly/Week_6/Comments-prediction-skeleton.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting NYT Picks Comments\n",
    "The New York Times has a feature on its website called \"NYT Picks\" which is an effort by the 14 moderators on staff to select and highlight the most interesting and insightful comments that are submitted by readers. The selected comments are put in a separate user interface tab so that they're easier to find, and they get a little yellow badge which serves to highlight them further. \n",
    "\n",
    "There are various editorial criteria that the Times' staff uses to decide which comments to publish and to select as NYT Picks. Some of the criteria used are described in this [article](https://www.google.com/#safe=off&q=http:%2F%2Fwww.nytimes.com%2Ftimes-insider%2F2014%2F04%2F17%2Fa-comments-path-to-publication%2F) (top link). Criteria used to decide to publish include: incoherency, political insults, profanity / obscenity, and insults or sterotyped condemnation. For NYT Picks they look for: high quality, broad representation, back and forth conversation, the unexpected, personal stories. Other criteria for editorial selection of comments that have been [developed from the research literature](http://www.nickdiakopoulos.com/wp-content/uploads/2011/07/ISOJ_Journal_V5_N1_2015_Spring_Diakopoulos_Picking-NYT-Picks.pdf) include argument quality, criticality, emotionality, entertainment value, readability, personal experience, internal coherence, thoughtfulness, length, relevance, fairness, and novelty. \n",
    "\n",
    "In this exercise we're goint to try to predict whether a comment should be a NYT Pick based on other scores that we use to make that prediction. We'll first develop a prediction framework, and then for the majority of class you'll work in pairs to code a new score based on the text of the comment that can be used in that prediction framework to improve the prediction. Toward the end, we'll combine all of our scores together and see if the combination of all our scores leads to an improvement in prediction accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The Data**  \n",
    "The data includes about 25,084 comments (half of which, 12,542, are NYT Picks) collected in 2014 from the NYT Community API. You can download that [here](https://www.dropbox.com/s/dqkgewvtxtfocy4/comments-sampled.csv?dl=0).\n",
    "\n",
    "The dataset includes several variables including:\n",
    "- `commentID`: the unique identifier for the comment\n",
    "- `commentBody`: test text of the comment\n",
    "- `approveDate`: the date and time when the comment was approved and published\n",
    "- `recommendationCount`: the number of times the comment was up-voted (i.e. recommended) by the community\n",
    "- `display_name`: the screen name of the user who made the comment\n",
    "- `articleURL`: the link to the article to which this comment was posted\n",
    "- `NYTPicks`: 0 or 1 to indicate whether the comment was selected as a Times Pick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Makes it so that you can scroll horizontally to see all columns of an output DataFrame\n",
    "pd.set_option('display.max_columns', None)\n",
    "# Make it so urls and tweets won't get truncated when we print them out\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "\n",
    "# This magic function allows you to see the charts directly within the notebook. \n",
    "%matplotlib inline\n",
    "\n",
    "# This command will make the plots more attractive by adopting the commone style of ggplot\n",
    "matplotlib.style.use(\"ggplot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cdf = pd.read_csv(\"Data/comments-sampled.csv\", parse_dates=[\"approveDate\"])\n",
    "# cdf_nonpicks = cdf[cdf.NYTPicks==0].sample(12542)\n",
    "# cdf_picks = cdf[cdf.NYTPicks==1]\n",
    "# cdf_new = cdf_nonpicks.append(cdf_picks, ignore_index=True)\n",
    "# cdf_new.to_csv(\"Data/comments-sampled.csv\", index=False)\n",
    "\n",
    "#cdf.columns\n",
    "#cdf.rename(columns={\"editorsSelection\": \"NYTPicks\"}, inplace=True)\n",
    "#cdf = cdf.iloc[np.random.permutation(len(cdf))]\n",
    "#cdf.to_csv(\"Data/comments-1.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print cdf.shape[0]\n",
    "print cdf[cdf.NYTPicks==0].shape[0]\n",
    "print cdf[cdf.NYTPicks==1].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Supervised Learning**  \n",
    "The prediction problem in this case is one of classification. We want to predict for any input comment whether it should be classified as an NYT Pick (noted as \"1\" in the NYTPicks) column. \n",
    "\n",
    "For the dataset we have at hand we actually know the answer already since for every comment we can see whether it was actually picked by editors or not. This allows us to develop a predictive model that *learns* the relationship between the input comment and the \"target\" or \"label\" which in this case is the NYT Pick status. This is called **supervised learning** -- the learning process is supervised in the sense that we're already given it the answer. The power of this of course is that once the learning process is complete we can use the model that was learned to apply it to *new comments* for which we don't yet know if it should be an NYT Pick or not. \n",
    "\n",
    "**Features**  \n",
    "For classification to work we need to have some attributes, often called **features** which are used as predictors for the classification. Many of the editorial criteria described above may make good candidates, but let's start simple. To find possible features that may have some predictive power (not all features will), we might start by doing a bit of exploratory analysis. Lets see how the mean and median recommendation count varies between comments that were picked and those that weren't.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cdf.groupby(\"NYTPicks\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cdf.groupby(\"NYTPicks\").median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This would appear to suggest that recommendationCount might be a good predictor since NYT Picks have much higher means and medians than non picks. \n",
    "\n",
    "Ok, so let's set up our classification model to use recommendationCount as a feature used to predict NYTPicks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Logistic Regression**  \n",
    "There are many different types of machine learning algorithms that can be used to learn from input data in a supervised fashion. You can read up more on [machine learning with scikit-learn](http://scikit-learn.org/stable/tutorial/basic/tutorial.html), a popular Python library. We'll just use a very basic algorithm in this case called **logistic regression**. You may have encountered regression before in stats class. Logistic regression is a form of regression that is used for classification problems in which the variable you're predicting isn't continuous but is binary (i.e. NYTPicks is either 0 or 1).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()\n",
    "\n",
    "# Create an array in the proper format from all of the recommendation Count values\n",
    "X = cdf.recommendationCount.values.reshape(-1,1)\n",
    "\n",
    "# Create an array in the proper format for the NYTPicks outcomes that we want to learn\n",
    "y = cdf.NYTPicks.values.ravel()\n",
    "\n",
    "# Train the model (or \"fit\" it to the data)\n",
    "model = model.fit(X, y)\n",
    "\n",
    "# Now score the model\n",
    "model.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "73.9% accuracy. That's not too bad. \n",
    "\n",
    "But, wait a minute. That's not really fair, since we just tested it on the exact same data that it was trained on ... that's basically cheating. To get an accurate evaluation it's important to have a **training dataset** and a **testing dataset** when doing supervised learning. In order to really know whether your learned model is successful you need to test it on data that it's never seen before. That's because if you just test it on the examples it was trained on you won't know if it generalizes to new examples it's unfamiliar with. Let's create training and testing sets so we can properly evaluate the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "model2 = LogisticRegression()\n",
    "\n",
    "# Create a train-test split of 50-50\n",
    "X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, y, test_size=0.5)\n",
    "# Train the model just on the training data\n",
    "model2 = model.fit(X_train, y_train)\n",
    "\n",
    "# And test the model on the test data\n",
    "#print model2.score(X_train, y_train)\n",
    "#print model2.score(X_test, y_test)\n",
    "\n",
    "# Use the model to predict NYTPicks values for the test dataset\n",
    "predicted = model2.predict(X_test)\n",
    "print \"Accuracy\"\n",
    "print metrics.accuracy_score(y_test, predicted)\n",
    "print \"\\nConfusion Matrix\"\n",
    "print metrics.confusion_matrix(y_test, predicted)\n",
    "#print sklearn.metrics.classification_report(y_test, predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The way to read the confusion matrix is:\n",
    "\n",
    "True Positives (TP) | False Positive (FP)\n",
    "----|----\n",
    "**False Negatives (FN)** | **True Positives (TP)**\n",
    "\n",
    "The sum of the two True Positive (TP) cells divided by the total number of test cases (i.e. all four cells) yields the accuracy. \n",
    "\n",
    "A False Positive (FP) is a comment that is actually not a NYT Pick, but was predicted by the classifier to be a NYT Pick. \n",
    "\n",
    "A False Negative (FN) is a comment that is actually a NYT Picks, but was predicted by the classifier to *not* be a NYT Pick.\n",
    "\n",
    "You can see from the result above that the errors are imbalanced. There are many more FNs (2576) than FPs (645). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "**New Features**  \n",
    "Now let's try to improve our classifier, increasing the accuracy as well as decreasing the FP and FN rates. To do that we need to find more features that are predictive of NYT Picks status. \n",
    "\n",
    "Let's break into teams of two. Each team should spend about 40 minutes writing code to compute a feature or score based on comment text analysis. The previous [class tutorial on text analysis](https://github.com/comp-journalism/UMD-J479V-J779V-Spring2016/blob/master/Weekly/Week_3/text-analysis.ipynb) will come in handy. What can you count / measure from the text itself that might be predictive? The score should ideally help predict NYT Picks status. A template is provided below so that you can test your new score's predictive power. \n",
    "\n",
    "Then you'll send me your .ipynb files and we'll combine everyone's features to see if we can make the predictive power of the classifier ever greater.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "import string\n",
    "from nltk.tokenize import WhitespaceTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stopword_list = stopwords.words('english')\n",
    "\n",
    "tokenizer = WhitespaceTokenizer()\n",
    "\n",
    "def remove_punctuation(text):\n",
    "    # Grab the list of standard punctuation symbols that are provided in the string library\n",
    "    punctuations = string.punctuation # includes following characters: !\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~\n",
    "\n",
    "    # But don't strip out apostrophes, as we want to preserve possessives and contractions, an alternative would be to expand contractions\n",
    "    excluded_punctuations = [\"'\"]\n",
    "    for p in punctuations:\n",
    "        if p not in excluded_punctuations:\n",
    "            # replace each punctuation symbol by a space\n",
    "            text = text.replace(p, \" \") \n",
    "\n",
    "    return text\n",
    "\n",
    "# Takes a tokenized list and returns a list minus any of the words in the stopword list\n",
    "def remove_stopwords(tokens):\n",
    "     return [w for w in tokens if w not in stopword_list]\n",
    "\n",
    "# There are a bunch of commented out pieces that may or may not be useful in the next function. Use as you see fit.\n",
    "def calculate_score(text):\n",
    "    #text = text.lower()\n",
    "    #text = remove_punctuation(text)\n",
    "    #text = \" \".join(text.split())\n",
    "    text_tokens = tokenizer.tokenize(text)\n",
    "    #text_tokens = remove_stopwords(text_tokens)\n",
    "    #text_tokens = [porter.stem(w) for w in text_tokens if w not in stopword_list]\n",
    "    # Very simple (silly?) new score counts the number of times a period was used in the comment\n",
    "    period_count = text.count(\".\")\n",
    "    # This function should return a numerical score\n",
    "    return float(period_count) / len(text_tokens)\n",
    "\n",
    "# This will create a new_score column by applying the above function to the text column\n",
    "cdf[\"new_score\"] = cdf[\"commentBody\"].apply(calculate_score)\n",
    "cdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# After calculating your score you might do a quick eyeball to see if the aggregate score is different between the two classes\n",
    "cdf.groupby(\"NYTPicks\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Repeat predictive train test\n",
    "X = cdf[[\"new_score\", \"recommendationCount\"]].values.reshape(-1,2)\n",
    "\n",
    "# Create an array in the proper format for the NYTPicks outcomes that we want to learn\n",
    "y = cdf.NYTPicks.values.ravel()\n",
    "\n",
    "X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, y, test_size=0.5)\n",
    "\n",
    "# Train the model just on the training data\n",
    "model3 = model.fit(X_train, y_train)\n",
    "\n",
    "# Use the model to predict NYTPicks values for the test dataset\n",
    "predicted = model2.predict(X_test)\n",
    "print \"Accuracy\"\n",
    "print metrics.accuracy_score(y_test, predicted)\n",
    "print \"\\nConfusion Matrix\"\n",
    "print metrics.confusion_matrix(y_test, predicted)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}