model_scikit_learn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build text classification models using scikit-learn\n",
    "- Use TfidfVectorizer to transform input texts into tfidf encoded float point matrix\n",
    "- Build a pipeline that include both feature extraction, and classification model\n",
    "- Build and train models\n",
    "- Evaluate model performace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn import metrics\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! ls -ltr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in preprocessed 20 news data into pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2225, 2) 5\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tech</td>\n",
       "      <td>tv future in the hands of viewers with home th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>business</td>\n",
       "      <td>worldcom boss  left books alone  former worldc...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category                                               text\n",
       "0      tech  tv future in the hands of viewers with home th...\n",
       "1  business  worldcom boss  left books alone  former worldc..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('bbc-text.csv')\n",
    "print(df.shape, df['category'].nunique())\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split dataset into train and test subset, we will train model using train subset and test model accuracy using test subset\n",
    "Also check the shape of train and test dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1780,) (445,) (1780,) (445,)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    df['text'], df['category'], test_size=.2, stratify=df['category'], random_state=42)\n",
    "\n",
    "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build and train model pipeline\n",
    "- Will build and train two classification models: [SGD](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html) and [SVM](https://scikit-learn.org/stable/modules/svm.html)\n",
    "- Report on model metrics of each model trained\n",
    "- Save model so we can use later for prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sgd_pipeline():\n",
    "    return Pipeline(\n",
    "        [\n",
    "            (\n",
    "                \"tfidf_vector_com\",\n",
    "                TfidfVectorizer(\n",
    "                    input=\"array\",\n",
    "                    norm=\"l2\",\n",
    "                    max_features=None,\n",
    "                    sublinear_tf=True,\n",
    "                    stop_words=\"english\",\n",
    "                ),\n",
    "            ),\n",
    "            (\n",
    "                \"clf\",\n",
    "                SGDClassifier(\n",
    "                    loss=\"log\",\n",
    "                    penalty=\"l2\",\n",
    "                    class_weight='balanced',\n",
    "                    tol=0.001,\n",
    "                ),\n",
    "            ),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "def svc_pipleline():\n",
    "    return Pipeline(\n",
    "        [\n",
    "            (\n",
    "                \"tfidf_vector_com\",\n",
    "                TfidfVectorizer(\n",
    "                    input=\"array\",\n",
    "                    norm=\"l2\",\n",
    "                    max_features=None,\n",
    "                    sublinear_tf=True,\n",
    "                    stop_words=\"english\",\n",
    "                ),\n",
    "            ),\n",
    "            (\n",
    "                \"clf\",\n",
    "                SVC(\n",
    "                    C=10,\n",
    "                    kernel=\"rbf\",\n",
    "                    gamma=0.1,\n",
    "                    probability=True,\n",
    "                    class_weight=None,\n",
    "                ),\n",
    "            ),\n",
    "        ]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_metrics(pred_test, y_test, pred_train, y_train):\n",
    "    print(\"test accuracy\", str(np.mean(pred_test == y_test)))\n",
    "    print(\"train accuracy\", str(np.mean(pred_train == y_train)))\n",
    "    print(\"\\n Metrics and Confusion for SVM \\n\")\n",
    "    print(metrics.confusion_matrix(y_test, pred_test))\n",
    "    print(metrics.classification_report(y_test, pred_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train model and report SVC model metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test accuracy 0.9842696629213483\n",
      "train accuracy 1.0\n",
      "\n",
      " Metrics and Confusion for SVM \n",
      "\n",
      "[[100   0   2   0   0]\n",
      " [  0  77   0   0   0]\n",
      " [  1   1  81   0   1]\n",
      " [  0   0   0 102   0]\n",
      " [  1   1   0   0  78]]\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     business       0.98      0.98      0.98       102\n",
      "entertainment       0.97      1.00      0.99        77\n",
      "     politics       0.98      0.96      0.97        84\n",
      "        sport       1.00      1.00      1.00       102\n",
      "         tech       0.99      0.97      0.98        80\n",
      "\n",
      "     accuracy                           0.98       445\n",
      "    macro avg       0.98      0.98      0.98       445\n",
      " weighted avg       0.98      0.98      0.98       445\n",
      "\n",
      "CPU times: user 22.3 s, sys: 22.3 ms, total: 22.3 s\n",
      "Wall time: 22.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "svc_pipe = svc_pipleline()\n",
    "svc_pipe.fit(X_train, y_train)\n",
    "pred_test = svc_pipe.predict(X_test)\n",
    "pred_train = svc_pipe.predict(X_train)\n",
    "print_metrics(pred_test, y_test, pred_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train model and report SGD model metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test accuracy 0.9820224719101124\n",
      "train accuracy 1.0\n",
      "\n",
      " Metrics and Confusion for SVM \n",
      "\n",
      "[[ 99   0   2   0   1]\n",
      " [  0  77   0   0   0]\n",
      " [  1   1  81   0   1]\n",
      " [  0   0   0 102   0]\n",
      " [  1   1   0   0  78]]\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     business       0.98      0.97      0.98       102\n",
      "entertainment       0.97      1.00      0.99        77\n",
      "     politics       0.98      0.96      0.97        84\n",
      "        sport       1.00      1.00      1.00       102\n",
      "         tech       0.97      0.97      0.97        80\n",
      "\n",
      "     accuracy                           0.98       445\n",
      "    macro avg       0.98      0.98      0.98       445\n",
      " weighted avg       0.98      0.98      0.98       445\n",
      "\n",
      "CPU times: user 2.38 s, sys: 14.3 ms, total: 2.39 s\n",
      "Wall time: 819 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd_pipe = sgd_pipeline()\n",
    "sgd_pipe.fit(X_train, y_train)\n",
    "pred_test = sgd_pipe.predict(X_test)\n",
    "pred_train = sgd_pipe.predict(X_train)\n",
    "print_metrics(pred_test, y_test, pred_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion:\n",
    "\n",
    "Using tfidf vectorization method we traind tow models: SGD and SVM. \n",
    "Training SGD model is very efficient and it only took 15 CPU seconds to get the model trained, and it achieved 98.2% test accuracy.\n",
    "Training SVM model took much much longer than SGD with 22 seconds, and the test accuracy is marginally better at 98.4%.\n",
    "\n",
    "Now we can save both model to pickle files so we can load it later for prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('scikit_learn_sgd.pickle', 'wb') as f:\n",
    "    pickle.dump(sgd_pipe, f)\n",
    "    \n",
    "with open('scikit_learn_svm.pickle', 'wb') as f:\n",
    "    pickle.dump(svc_pipe, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}