ml-text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个小例子说明文本分类的基本流程，以及模型选择"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('./data/labeledTrainData.tsv.zip', 'r') as z:\n",
    "    z.extractall('./data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment                                             review\n",
       "0  5814_8          1  With all this stuff going down at the moment w...\n",
       "1  2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "2  7759_3          0  The film starts with a manager (Nicholas Bell)...\n",
       "3  3630_4          0  It must be assumed that those who praised this...\n",
       "4  9495_8          1  Superbly trashy and wondrously unpretentious 8..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t')\n",
    "print(data_train.shape)\n",
    "data_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['review'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "def clean_str(string):\n",
    "    \"\"\"\n",
    "    Tokenization/string cleaning for all datasets \n",
    "    Every dataset is lower cased\n",
    "    \"\"\"\n",
    "    string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", string)     \n",
    "    string = re.sub(r\"\\'s\", \" \\'s\", string) \n",
    "    string = re.sub(r\"\\'ve\", \" \\'ve\", string) \n",
    "    string = re.sub(r\"n\\'t\", \" n\\'t\", string) \n",
    "    string = re.sub(r\"\\'re\", \" \\'re\", string) \n",
    "    string = re.sub(r\"\\'d\", \" \\'d\", string) \n",
    "    string = re.sub(r\"\\'ll\", \" \\'ll\", string) \n",
    "    string = re.sub(r\",\", \" , \", string) \n",
    "    string = re.sub(r\"!\", \" ! \", string) \n",
    "    string = re.sub(r\"\\(\", \" \\( \", string) \n",
    "    string = re.sub(r\"\\)\", \" \\) \", string) \n",
    "    string = re.sub(r\"\\?\", \" \\? \", string) \n",
    "    string = re.sub(r\"\\s{2,}\", \" \", string)    \n",
    "    return string.strip().lower()\n",
    "\n",
    "stops = set(stopwords.words(\"english\"))\n",
    "def review_to_words(raw_review):\n",
    "    review_text = BeautifulSoup(raw_review, \"lxml\").get_text()  \n",
    "    words = clean_str(review_text)\n",
    "    meaningful_words = [w for w in words.split(' ') if not w in stops]\n",
    "    return(\" \".join(meaningful_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "print(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = []\n",
    "texts_with_stopwords = []\n",
    "labels = []\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    text = review_to_words(data_train.review[idx])\n",
    "    texts.append(text)\n",
    "    texts_with_stopwords.append(clean_str(BeautifulSoup(data_train.review[idx], \"lxml\").get_text()))\n",
    "    labels.append(data_train.sentiment[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"stuff going moment mj 've started listening music , watching odd documentary , watched wiz watched moonwalker maybe want get certain insight guy thought really cool eighties maybe make mind whether guilty innocent moonwalker part biography , part feature film remember going see cinema originally released subtle messages mj 's feeling towards press also obvious message drugs bad m'kay visually impressive course michael jackson unless remotely like mj anyway going hate find boring may call mj egotist consenting making movie mj fans would say made fans true really nice actual feature film bit finally starts 20 minutes excluding smooth criminal sequence joe pesci convincing psychopathic powerful drug lord wants mj dead bad beyond mj overheard plans \\\\? nah , joe pesci 's character ranted wanted people know supplying drugs etc dunno , maybe hates mj 's music lots cool things like mj turning car robot whole speed demon sequence also , director must patience saint came filming kiddy bad sequence usually directors hate working one kid let alone whole bunch performing complex dance scene bottom line , movie people like mj one level another \\\\( think people \\\\) , stay away try give wholesome message ironically mj 's bestest buddy movie girl ! michael jackson truly one talented people ever grace planet guilty \\\\? well , attention 've gave subject hmmm well n't know people different behind closed doors , know fact either extremely nice stupid guy one sickest liars hope latter\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"with all this stuff going down at the moment with mj i 've started listening to his music , watching the odd documentary here and there , watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj 's feeling towards the press and also the obvious message of drugs are bad m'kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for 20 minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans \\\\? nah , joe pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates mj 's music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also , the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line , this movie is for people who like mj on one level or another \\\\( which i think is most people \\\\) if not , then stay away it does try and give off a wholesome message and ironically mj 's bestest buddy in this movie is a girl ! michael jackson is truly one of the most talented people ever to grace this planet but is he guilty \\\\? well , with all the attention i 've gave this subject hmmm well i do n't know because people can be different behind closed doors , i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts_with_stopwords[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.88664\n",
      "clf__C: 1\n",
      "vect__min_df: 1\n",
      "vect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=3)),\n",
    "                     ('clf', LogisticRegression(C=3)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'clf__C': (1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.89088\n",
      "clf__C: 1\n",
      "vect__min_df: 1\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))),\n",
    "                     ('clf', LogisticRegression(C=3)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__C': (1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts_with_stopwords, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.89628\n",
      "clf__C: 10\n",
      "vect__min_df: 3\n",
      "vect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', TfidfVectorizer(max_df=0.95, min_df=3)),\n",
    "                     ('clf', LogisticRegression(C=3)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'clf__C': (1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.90036\n",
      "clf__C: 15\n",
      "vect__min_df: 3\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', TfidfVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))),\n",
    "                     ('clf', LogisticRegression(C=3)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__C': (1,3,5,10,15),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts_with_stopwords, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "futher tuning the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))\n",
    "tfidf_features = tfidf_vectorizer.fit_transform(texts)\n",
    "tfidf_features_with_stopwords = tfidf_vectorizer.fit_transform(texts_with_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 180058)\n",
      "(25000, 265165)\n"
     ]
    }
   ],
   "source": [
    "print(tfidf_features.shape)\n",
    "print(tfidf_features_with_stopwords.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.89772\n",
      "LogisticRegression(C=15, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=1e-06,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "parameters = {'C':[1,3,5,10,15], 'tol':[1e-6,1e-5,1e-4,1e-3]}\n",
    "clf = GridSearchCV(lr, parameters, n_jobs=4)\n",
    "clf.fit(tfidf_features, labels)\n",
    "\n",
    "print(clf.best_score_)\n",
    "print(clf.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.90124\n",
      "LogisticRegression(C=15, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=1e-06,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "parameters = {'C':[1,3,5,10,15], 'tol':[1e-6,1e-5,1e-4,1e-3]}\n",
    "clf = GridSearchCV(lr, parameters, n_jobs=4)\n",
    "clf.fit(tfidf_features_with_stopwords, labels)\n",
    "\n",
    "print(clf.best_score_)\n",
    "print(clf.best_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**最终，通过实验可知使用逻辑回归在IMDB数据集上的情感分类最优选择**：\n",
    "\n",
    "- 使用2-grams的TF_IDF权重特征，min_df=3, tol=1e-06\n",
    "- 语料预处理时保留停用词\n",
    "- 最后交叉校验选择LR的最佳参数组合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use character ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from scipy.sparse import hstack\n",
    "\n",
    "tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))\n",
    "tfidf_vectorizer_char = TfidfVectorizer(max_df=0.95, min_df=5, analyzer='char', ngram_range=(1, 4))\n",
    "tfidf_features_with_stopwords = tfidf_vectorizer.fit_transform(texts_with_stopwords)\n",
    "tfidf_features_with_stopwords_char = tfidf_vectorizer_char.fit_transform(texts_with_stopwords)\n",
    "\n",
    "tfidf_matrix_word_char =  hstack((tfidf_features_with_stopwords, tfidf_features_with_stopwords_char))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8754"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(tfidf_features_with_stopwords[:20000],labels[:20000]).score(tfidf_features_with_stopwords[20000:], labels[20000:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.8794 0.8788 0.8826 0.89   0.8768] 0.8815200000000001\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(lr, tfidf_features_with_stopwords, labels, cv=5, n_jobs=4)\n",
    "print(scores,np.mean(scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.88912\n",
      "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "parameters = {'C':[1,3,5,10,15]}\n",
    "clf = GridSearchCV(lr, parameters, n_jobs=4)\n",
    "clf.fit(tfidf_features_with_stopwords, labels)\n",
    "\n",
    "print(clf.best_score_)\n",
    "print(clf.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x155039 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 6837044 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix_word_char[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x61032 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 41993634 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix_word_char[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.896  0.8962 0.8966 0.9058 0.894 ] 0.89772\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "scores = cross_val_score(lr, tfidf_matrix_word_char, labels, cv=5, n_jobs=4)\n",
    "print(scores,np.mean(scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 216071)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix_word_char.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.90068\n",
      "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "parameters = {'C':[1,3,5,10,15]}\n",
    "clf = GridSearchCV(lr, parameters, n_jobs=4)\n",
    "clf.fit(tfidf_matrix_word_char, labels)\n",
    "\n",
    "print(clf.best_score_)\n",
    "print(clf.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.90056\n",
      "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=1e-06,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression(C=10)\n",
    "parameters = {'tol':[1e-6,1e-5,1e-4,1e-3]}\n",
    "clf = GridSearchCV(lr, parameters, n_jobs=4)\n",
    "clf.fit(tfidf_matrix_word_char, labels)\n",
    "\n",
    "print(clf.best_score_)\n",
    "print(clf.best_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MultinomialNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.87736\n",
      "clf__alpha: 1\n",
      "vect__min_df: 1\n",
      "vect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=3)),\n",
    "                     ('clf', MultinomialNB(alpha=1)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'clf__alpha': (0.01,0.1,1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.87808\n",
      "clf__alpha: 1\n",
      "vect__min_df: 1\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=1, ngram_range=(1, 2))),\n",
    "                     ('clf', MultinomialNB(alpha=1)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__alpha': (0.01,0.1,1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts_with_stopwords, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.889\n",
      "clf__alpha: 0.1\n",
      "vect__min_df: 1\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', TfidfVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))),\n",
    "                     ('clf',  MultinomialNB(alpha=1)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__alpha': (0.01,0.1,1,3,5,10),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts_with_stopwords, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.86864\n",
      "clf__n_estimators: 1000\n",
      "vect__min_df: 1\n",
      "vect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=3)),\n",
    "                     ('clf', RandomForestClassifier(n_estimators =250)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'clf__n_estimators': (250, 500, 1000),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.86768\n",
      "clf__n_estimators: 1000\n",
      "vect__min_df: 3\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))),\n",
    "                     ('clf',  RandomForestClassifier(n_estimators =250)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__n_estimators': (500, 1000),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts_with_stopwords, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.86912\n",
      "clf__n_estimators: 1000\n",
      "vect__min_df: 1\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', TfidfVectorizer(max_df=0.95, min_df=3, ngram_range=(1, 2))),\n",
    "                     ('clf',  RandomForestClassifier(n_estimators =250)),\n",
    "])\n",
    "\n",
    "parameters = {'vect__min_df': (1,3,5),\n",
    "              'clf__n_estimators': (250, 500, 1000),\n",
    "}\n",
    "\n",
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=4)\n",
    "gs_clf = gs_clf.fit(texts, labels)\n",
    "print(gs_clf.best_score_)\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}