sklearn_tutorial_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# scikit-learn for NLP -- Part 2 Feature Engineering\n",
    "\n",
    "\n",
    "If you are reading this, that means you are still alive. Welcome back to the reality of learning scikit-learn. \n",
    "\n",
    "This tutorial focuses on feature engineering and covers more advanced topics in scikit-learn like feature extraction, building a pipeline, creating custom transformers, feature union, dimensionality reduction, and grid search. Feature engineering is a very important step in NLP and ML; it is not a trival task to select good features. Therefore, we are spending a lot of time on it here. \n",
    "\n",
    "Without further ado, let's start with loading a dataset again. This time, we will use a CSV file that has more than two columns, i.e. one column for labels and mutiple columns for raw data/features. This time, we are using a subset of the Yelp Review Data Challenge dataset. Just like the 20 News Group dataset, I converted this dataset to CSV. \n",
    "\n",
    "There are 5 star ratings (shocking), and I extracted 500 reviews for each rating. This dataset is small, because it's only intended for some quick demo; therefore, the performance of any classifier won't be too good (and this should be a regression problem instead of classification, but to make things easier, let's stick with classification). Other than extracing the text of each review, I also included other users' votes for each review, i.e funny, useful and cool. For this tutorial, our task is to predict the star rating of each review. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "dataset = pd.read_csv('yelp-review-subset.csv', header=0, delimiter=',', names=['stars', 'text', 'funny', 'useful', 'cool'])\n",
    "\n",
    "# just checking the dataset\n",
    "print('There are {0} star ratings, and {1} reviews'.format(len(dataset.stars.unique()), len(dataset)))\n",
    "print(dataset.stars.value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Pandas dataframe, it is very easy to select a subset of a dataframe by column names, simply pass in a list of column names. So we are going to split our data just like the previous tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(dataset[['text', 'funny', 'useful', 'cool']], dataset['stars'], train_size=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference now is that we have four columns in our raw data. Three of them are ``funny``, ``useful`` and ``cool`` which contain numeric values, which is perfect for scikit-learn as it expects values of features to be numeric or indexes. What we need to do is to extract features from the ``text`` column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(X_train.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can first do something very similar to the previous tutorial, we initialize a ``CountVectorizer`` object and then pass raw text dat into its ``fit_transform()`` function to index the count of each word. Please note that you should not pass in the whole ``X_train`` dataframe into the function, but only the ``text`` column, i.e. the ``X_train.text`` dataframe (more or less like an array). Otherwise, it does not extract features from your ``text`` column, but simly indexes all the values in each column. Please see the shape of the output of two different dataframes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# initialize a CountVectorizer\n",
    "cv = CountVectorizer()\n",
    "# fit the raw data into the vectorizer and tranform it into a series of arrays\n",
    "X_train_counts = cv.fit_transform(X_train.text)\n",
    "print(X_train_counts.shape)\n",
    "\n",
    "# this is not what you want.\n",
    "cv_test = CountVectorizer()\n",
    "X_train_counts_test = cv.fit_transform(X_train)\n",
    "print(X_train_counts_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a problem, the ``X_train_counts`` vector only contains features from ``text``, but now what should we do if we want to include ``funny``, ``useful`` and ``cool`` vote counts as feature as well? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.  ``Pipeline`` and ``FeatureUnion`` \n",
    "\n",
    "To deal with that problem, we need to talk about [``Pipeline``](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) and [``FeatureUnion``](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.html). ``Pipeline`` lets us define a list of steps which consists of a list of transformers to extract features from data (including ``FeatureUnion``) and a final estimator (aka classifier). ``FeatureUnion`` basically concatenates results of multiple transformer objects. The following is a complete example of how to use these two together. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.base import TransformerMixin, BaseEstimator\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.pipeline import FeatureUnion, Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "\n",
    "class ItemSelector(TransformerMixin, BaseEstimator):\n",
    "    \"\"\"This class allows you to select a subset of a dataframe based on given column name(s).\"\"\"\n",
    "    def __init__(self, keys):\n",
    "        self.keys = keys\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, dataframe):\n",
    "        return dataframe[self.keys]\n",
    "\n",
    "\n",
    "class VotesToDictTransformer(TransformerMixin, BaseEstimator):\n",
    "    \"\"\"This tranformer converts the vote counts of each row into a dictionary.\"\"\"\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, votes):\n",
    "        funny, useful, cool = votes['funny'], votes['useful'], votes['cool']\n",
    "        return [{'funny': binarize_number(f, 1), 'useful': binarize_number(u, 1), 'cool': binarize_number(c, 1)} \n",
    "                for f, u, c in zip(funny, useful, cool)]\n",
    "\n",
    "    \n",
    "def binarize_number(num, threshold):\n",
    "    return 0 if num < threshold else 1\n",
    "\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    # Use FeatureUnion to combine the features from text and votes\n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "\n",
    "            # Pipeline for getting BOW features from the texts\n",
    "            ('bag-of-words', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('counts', CountVectorizer()),\n",
    "            ])),\n",
    "\n",
    "            # Pipeline for getting vote counts as features\n",
    "            # the DictVecotrizer object there transform indexes the values of the dictionaries\n",
    "            # passed down from the VotesToDictTransformer object.\n",
    "            ('votes', Pipeline([\n",
    "                ('selector', ItemSelector(keys=['funny', 'useful', 'cool'])),\n",
    "                ('votes_to_dict', VotesToDictTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "\n",
    "        ],\n",
    "\n",
    "        # weight components in FeatureUnion\n",
    "        transformer_weights={\n",
    "            'bag-of-words': 1.0,\n",
    "            'votes': 0.5\n",
    "        },\n",
    "    )),\n",
    "\n",
    "    # Use a naive bayes classifier on the combined features\n",
    "    ('clf', LogisticRegression()),\n",
    "])\n",
    "\n",
    "\n",
    "pipeline.fit(X_train, y_train)\n",
    "predicted = pipeline.predict(X_test)\n",
    "print(classification_report(predicted, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Custom Transformers\n",
    "In the previous section, I defined two classes ``ItemSelector`` and ``VotesToDictTransformer``, and the commonality of these two is that they inherited the [``TransformerMixin``](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html) class. ``TransformerMixin`` is the base class of many built-in transformers and vectorizers in scikit-learn, e.g. ``CountVectorizer``, ``TfidfVectorizer``, ``TfidfTransformer``, ``DictVectorizer``, etc. We define the ``transform()`` function to manipulate the data in a more custom way. For example, ``ItemSelector`` returns a subset of dataframe based on given column names, and ``VotesToDictTransformer`` transforms a dataframe into a list of dictionaries.  \n",
    "\n",
    "To demonstrate how useful customer transformers are, let's define another one. Say, we hypothesize that the sentiment of each review can be a strong feature for predict the star rating. Then we would need a ``SentimentTransformer`` class. \n",
    "\n",
    "To avoid spending time to train our own sentiment classifier, we use the ``TextBlob`` package for its built-in [sentiment analysis feature](https://textblob.readthedocs.io/en/dev/quickstart.html#sentiment-analysis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "\n",
    "class SentimentTransformer(TransformerMixin, BaseEstimator):\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, texts):\n",
    "        features = []\n",
    "        for text in texts:\n",
    "            blob = TextBlob(text.decode('utf-8'))\n",
    "            features.append({'polarity': binarize_number(blob.sentiment.polarity, 0.5),\n",
    "                             'subjectivity': binarize_number(blob.sentiment.subjectivity, 0.5)})\n",
    "        return features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add that transformer to our existing pipeline, and see if the additional features help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "        \n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "                    \n",
    "            ('bag-of-words', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('counts', CountVectorizer()),\n",
    "            ])),\n",
    "\n",
    "            ('votes', Pipeline([\n",
    "                ('selector', ItemSelector(keys=['funny', 'useful', 'cool'])),\n",
    "                ('votes_to_dict', VotesToDictTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "                    \n",
    "            ('sentiments', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('sentiment_transform', SentimentTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "        ],\n",
    "\n",
    "        # weight components in FeatureUnion\n",
    "        transformer_weights={\n",
    "            'bag-of-words': 1.0,\n",
    "            'votes': 0.5,\n",
    "            'sentiments': 1.0,\n",
    "        },\n",
    "    )),\n",
    "\n",
    "    # Use a naive bayes classifier on the combined features\n",
    "    ('clf', LogisticRegression()),\n",
    "])\n",
    "\n",
    "\n",
    "pipeline.fit(X_train, y_train)\n",
    "predicted = pipeline.predict(X_test)\n",
    "print(classification_report(predicted, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 3. Feature Reduction/Selection\n",
    "\n",
    "Two major problems of using bag-of-words as features are (1) that it introduces noise; and (2) that it increases the dimensionality of feature space. When using bag-of-words, we simply throw in a bunch of words into the feature space and hope and pray that they work, because we don't know what words are most informative in a model. Other than handcrafting features and selecting what words to put into the feature space, we can also use the [``feature_selection``](http://scikit-learn.org/stable/modules/feature_selection.html) module to automatically select informative features and eliminate noise. \n",
    "\n",
    "\"``SelectFromModel`` is a meta-transformer that can be used along with any estimator that has a ``coef_`` or ``feature_importances_`` attribute after fitting. The features are considered unimportant and removed, if the corresponding ``coef_`` or ``feature_importances_`` values are below the provided ``threshold`` parameter.\" Basically, the idea is \"to reduce the dimensionality of the data to use with another classifier, they can be used along with ``feature_selection.SelectFromModel`` to select the non-zero coefficients.\" \n",
    "\n",
    "In this following example, we use ``LogisticRegression`` to perform feature elemination. \"l2\" is the penalty, and C controls the sparsity: the smaller C the fewer features selected.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "pipeline = Pipeline([\n",
    "        \n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "                    \n",
    "            ('bag-of-words', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('counts', CountVectorizer()),\n",
    "            ])),\n",
    "\n",
    "            ('votes', Pipeline([\n",
    "                ('selector', ItemSelector(keys=['funny', 'useful', 'cool'])),\n",
    "                ('votes_to_dict', VotesToDictTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "                    \n",
    "            ('sentiments', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('sentiment_transform', SentimentTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "        ],\n",
    "\n",
    "        # weight components in FeatureUnion\n",
    "        transformer_weights={\n",
    "            'bag-of-words': 1.0,\n",
    "            'votes': 0.5,\n",
    "            'sentiments': 1.0,\n",
    "        },\n",
    "    )),\n",
    "\n",
    "    # use SelectFromModel to select informative features\n",
    "    ('feature_selection', SelectFromModel(LogisticRegression(C=0.5, penalty=\"l2\"))),\n",
    "    \n",
    "    # Use a naive bayes classifier on the combined features\n",
    "    ('clf', LogisticRegression()),\n",
    "])\n",
    "\n",
    "\n",
    "pipeline.fit(X_train, y_train)\n",
    "predicted = pipeline.predict(X_test)\n",
    "print(classification_report(predicted, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Grid Search\n",
    "\n",
    "Finally, many classifiers like logistic regression or SVM have certain parameters to tweak for them to get optimal results, and it can be pain in the neck for one to try every combination. Grid Search is an automated method to try every combination and rank the best combinations. In scikit-learn, we use ``model_selection.GridSearchCV``. The bare minimum set of parameters for grid search is an ``estimator`` object and a list or dictionary of parameters. In our case, we are passing a pipline and a dictionary (``Pipeline`` inherits ``BaseEstimator``). \n",
    "Parameters of the estimators in the pipeline can be accessed using the <estimator>__<parameter> syntax, and that will be the key of the dictionary, and the value will be a list of values to experiment. For example, we want to try the combination of different ``max_iter`` and ``C`` values, and the name of the ``LogisticRegression`` in the pipeline is `clf`. Therefore, in the dictionary there are two entries: ``clf__max_iter`` and ``clf__C``. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "pipeline = Pipeline([\n",
    "        \n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "                    \n",
    "            ('bag-of-words', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('counts', CountVectorizer()),\n",
    "            ])),\n",
    "\n",
    "            ('votes', Pipeline([\n",
    "                ('selector', ItemSelector(keys=['funny', 'useful', 'cool'])),\n",
    "                ('votes_to_dict', VotesToDictTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "                    \n",
    "            ('sentiments', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('sentiment_transform', SentimentTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "        ],\n",
    "\n",
    "        # weight components in FeatureUnion\n",
    "        transformer_weights={\n",
    "            'bag-of-words': 1.0,\n",
    "            'votes': 0.5,\n",
    "            'sentiments': 1.0,\n",
    "        },\n",
    "    )),\n",
    "\n",
    "    # Use a naive bayes classifier on the combined features\n",
    "    ('clf', LogisticRegression()),\n",
    "])\n",
    "\n",
    "params = dict(clf__max_iter=[50, 100, 150], clf__C=[1.0, 0.5, 0.1])\n",
    "grid_search = GridSearchCV(pipeline, param_grid=params)\n",
    "grid_search.fit(X_train, y_train)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the output above, when C=0.1 and max_iter=50, we get the best results. To validate the results, let's use these values to train and test a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "        \n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "                    \n",
    "            ('bag-of-words', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('counts', CountVectorizer()),\n",
    "            ])),\n",
    "\n",
    "            ('votes', Pipeline([\n",
    "                ('selector', ItemSelector(keys=['funny', 'useful', 'cool'])),\n",
    "                ('votes_to_dict', VotesToDictTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "                    \n",
    "            ('sentiments', Pipeline([\n",
    "                ('selector', ItemSelector(keys='text')),\n",
    "                ('sentiment_transform', SentimentTransformer()),\n",
    "                ('vectorizer', DictVectorizer()),\n",
    "            ])),\n",
    "        ],\n",
    "\n",
    "        # weight components in FeatureUnion\n",
    "        transformer_weights={\n",
    "            'bag-of-words': 1.0,\n",
    "            'votes': 0.5,\n",
    "            'sentiments': 1.0,\n",
    "        },\n",
    "    )),\n",
    "\n",
    "    # Use a naive bayes classifier on the combined features\n",
    "    ('clf', LogisticRegression(C=0.1, max_iter=50)),\n",
    "])\n",
    "\n",
    "pipeline.fit(X_train, y_train)\n",
    "predicted = pipeline.predict(X_test)\n",
    "print(classification_report(predicted, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, with almost everything the same as the pipeline in Section 2, changing the value of ``C`` and that of ``max_iter`` improves our results (the default value of ``C`` is 1.0 and that of ``max_iter`` is 100). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Conclusion\n",
    "\n",
    "This is just a simple overview of performing feature engineering in scikit-learn, and there are many different models that you can try. For example, with ``GridSearchCV``, you can even try comparing the performance of different classifiers. This two-part tutorial is to help you get familiar and comfortable with scikit-learn and its main modules. Please check its documentation if you need more clarification on how to do certain things, since scikit-learn is one of the best documented libraries that I know of!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}