gather_explore_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The first step of building a text classifier is to gather data\n",
    "For demostration purpose I used the bbc text data freely available online at https://www.kaggle.com/yufengdev/bbc-fulltext-and-category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the data\n",
    "Load the data into pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2225, 2)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('bbc-text.csv')\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tech</td>\n",
       "      <td>tv future in the hands of viewers with home th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>business</td>\n",
       "      <td>worldcom boss  left books alone  former worldc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sport</td>\n",
       "      <td>tigers wary of farrell  gamble  leicester say ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sport</td>\n",
       "      <td>yeading face newcastle in fa cup premiership s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>entertainment</td>\n",
       "      <td>ocean s twelve raids box office ocean s twelve...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        category                                               text\n",
       "0           tech  tv future in the hands of viewers with home th...\n",
       "1       business  worldcom boss  left books alone  former worldc...\n",
       "2          sport  tigers wary of farrell  gamble  leicester say ...\n",
       "3          sport  yeading face newcastle in fa cup premiership s...\n",
       "4  entertainment  ocean s twelve raids box office ocean s twelve..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sport            511\n",
       "business         510\n",
       "politics         417\n",
       "tech             401\n",
       "entertainment    386\n",
       "Name: category, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['category'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAE8CAYAAAA7X/FRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3debxVVd3H8c+XQcGhFEVCBrEiDXKokHIoK03JSuwpFVND02hwalS0UitJn9SyTJ8cE+dwSsrKlNLSSkIbFIfENEEQccqhJMHf88daN7bXO5wL93DuXef7fr3u656zzz7nrL3vud+z9tprra2IwMzMytKn0QUwM7Pu53A3MyuQw93MrEAOdzOzAjnczcwK5HA3MyuQw91eQdIFkk5o0HtL0g8lPSVpdiPK0BlJoySFpH49+TWtuTncewFJD0laLGntyrKDJd3UwGLVyw7Ae4HhETG+0YVpJpKOl3Rxo8th3cPh3nv0A45odCG6SlLfLj5lE+ChiHi+HuWxnslHLN3P4d57nAx8UdJ6rR9o65Be0k2SDs63D5B0q6TvSHpa0t8lbZeXz5f0mKTJrV52Q0k3SHpW0s2SNqm89ub5sScl3Sdpr8pjF0j6P0k/k/Q88O42yruxpJn5+fMkfSIvPwg4F9hW0nOSvtbGc1+fy/NPSY9L+lHlse/m7XlG0u2S3lF57HhJV0i6OG/TnZLeIOnovP3zJe3Sav+dKGl2fq9rJQ1q6w8j6dWSzpO0SNIjkk5o+VLrqLzt+Likhfm1vpBf4zWS/iVpg8p7vlXSEkn92yhPX0nHSHogb+vtkkZ0tI8kTQCOAfbO+/4vNWxbX0mn5u16UNKh1c9he3/nyt/jyvz3eAaY2pVttBpEhH96+A/wELAzcDVwQl52MHBTvj0KCKBf5Tk3AQfn2wcAy4ADgb7ACcDDwBnAmsAuwLPAOnn9C/L9d+bHvwvckh9bG5ifX6sf8BbgcWBs5bn/BLYnVR4GtLE9NwNnAgOArYElwE6Vst7Swb64DPhyy2sDO1Qe2w/YIJfrC8CjLe8PHA+8AOyaH78QeDC/Vn/gE8CDrfbfI8Cb8jZfBVzc1v4GfgycldfbCJgNfLKz8rbarpbXvCy/zhZ5v+ycH/8Z8OnK+t8BTm/ntb4E3AlsBgjYCtigxn10cavX6mjbPgXcDQwH1gdubLVfOvo7Hw+8COyR983Armyjf2rIjUYXwD81/JFWhPubSME5mK6H+/2Vx7bI6w+pLHsC2DrfvgC4vPLYOsByYASwN/DbVuU7Cziu8twLO9iWEfm11q0sOxG4oFLWjsL9QuBsUpt8Z/vtKWCrfPt44IbKYx8EngP65vvr5n2yXmX/nVRZfwzwH9KX43/3NzAEWAoMrKy7D/DrrpS38pqbV5Z9Czgv394buDXf7ksK5fHtvNZ9wMQaP1ut99HFlcc627ZfkYM+39+5sl86+zsfD/ymVVlq3kb/dP7jZpleJCLuAn4KTF2Jpy+u3P53fr3Wy9ap3J9fed/ngCeBjUlt4m/LzTtPS3oa2Bd4TVvPbcPGwJMR8Wxl2T+AYTVux5Gk2uhsSXMlfbzlAUlfkHRPbgJ5Gng1sGHlua239/GIWF65D+3sg1zG/q1eD9L+6A8squyPs0i13A7L247W77lxvn0tMEbSa0knnP8ZEe31JhoBPNDWAzXso65s28atylu9XcvfufXnpCvbaJ3wSYze5zjgDuDUyrKWk49rAc/k29WwXRkjWm5IWgcYBCwk/UPeHBHv7eC5HU01uhAYJGndyj/+SFITSKci4lFSEwqSdgBulPQbYChwFLATMDciXpL0FClYV9aIyu2RpGaEx1stn0+q3W4YEctqLW9EzOvgPe+tvOfC/DovSJpB+iLdHLiog3LPB14H3FVdmNvXO9pHrf9uHW4bsIjUJFMte4ta/s4ve78ubqN1wjX3XiaHwo+AwyvLlpD+afbLJ7k+TvrnXhW7SdpB0hrAN4DbImI+6cjhDZL2l9Q//2wj6Y01ln8+8DvgREkDJG0JHARcUsvzJe0pqSVQniIFxHJSs8oyUrtuP0nHAq+qfXPbtJ+kMZLWAr4OXFmp6bdszyLgl8Cpkl4lqY+k10nasZPytuerktaSNJZ0XqN6AvZCUrPV7kBHXRbPBb4habSSLfOJys720WJglKQ+tWwbMAM4QtIwpRP9R1X2y8r+nWvdRuuEw713+jrpBFfVJ0gn0p4AxpL+sVbFpaSjhCeBt5JqU+Ra2C7AJFLt7FHgf0knXmu1D6mNeSFwDam9/oYan7sNcJuk54CZwBER8SBwPfBz4G+kw/8X6Lh5qBYXkc4hPEo6KXh4O+t9DFiDdHLxKeBK0pFER+Vtz83APGAWcEpE/LLlgYi4FXgJuCMiHurgNb5NCt5fko7kziOdsOxsH12Rfz8h6Y4atu2c/B5/Bf5EOiG6jBVfXl3+O3dhG60TyicvzKxCaYDYxRFxbqPLUiXpV8ClPa1cAJLeB/wgIjbpdOWOX6fHbmNv4pq7WS8haRtS19PO+sqvFpIGStpNUj9Jw0hHetes4mv2qG3szRzuZr2ApOmkfuSfbdUDpZEEfI3UXPMn4B7g2JV+sZ65jb2Wm2XMzArkmruZWYEc7mZmBeoRg5g23HDDGDVqVKOLYWbWq9x+++2PR8Tgth7rEeE+atQo5syZ0+himJn1KpL+0d5jbpYxMyuQw93MrEAOdzOzAjnczcwK5HA3MyuQw93MrEAOdzOzAjnczcwK1CMGMXWHUVOva3QRAHjopPc3ughmZrWFu6SHgGdJV1hZFhHjJA0izbk8CngI2CsinsrrH026pNZy4PCIuL7bS27t8hedmXWlWebdEbF1RIzL96cCsyJiNOmSYFMBJI0hXYJtLDABOFNS324ss5mZdWJV2twnAtPz7enAHpXll0fE0nytyHnA+FV4HzMz66Jawz2AX0q6XdKUvGxIvjp6y1XSN8rLh/Hyi+4uyMvMzGw1qfWE6vYRsVDSRsANku7tYF21sewVl3vKXxJTAEaOHFljMczMrBY11dwjYmH+/RjpArjjgcWShgLk34/l1RcAIypPHw4sbOM1z46IcRExbvDgNqcjNjOzldRpuEtaW9K6LbeBXYC7gJnA5LzaZODafHsmMEnSmpI2BUYDs7u74GZm1r5ammWGANdIaln/0oj4haQ/AjMkHQQ8DOwJEBFzJc0A7gaWAYdExPK6lN7MzNrUabhHxN+BrdpY/gSwUzvPmQZMW+XSmZnZSvH0A2ZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYH6NboAZvU0aup1jS4CD530/kYXwZqQa+5mZgVyuJuZFcjhbmZWoJrDXVJfSX+S9NN8f5CkGyTdn3+vX1n3aEnzJN0nadd6FNzMzNrXlZr7EcA9lftTgVkRMRqYle8jaQwwCRgLTADOlNS3e4prZma1qCncJQ0H3g+cW1k8EZieb08H9qgsvzwilkbEg8A8YHz3FNfMzGpRa1fI04AjgXUry4ZExCKAiFgkaaO8fBjwh8p6C/Kyl5E0BZgCMHLkyC4W28y6yt1Cm0unNXdJHwAei4jba3xNtbEsXrEg4uyIGBcR4wYPHlzjS5uZWS1qqblvD+wuaTdgAPAqSRcDiyUNzbX2ocBjef0FwIjK84cDC7uz0GZm1rFOa+4RcXREDI+IUaQTpb+KiP2AmcDkvNpk4Np8eyYwSdKakjYFRgOzu73kZmbWrlWZfuAkYIakg4CHgT0BImKupBnA3cAy4JCIWL7KJTUzs5p1Kdwj4ibgpnz7CWCndtabBkxbxbKZmdlK8sRhZtZ0mqHnkKcfMDMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxADnczswI53M3MCuRwNzMrkMPdzKxAnYa7pAGSZkv6i6S5kr6Wlw+SdIOk+/Pv9SvPOVrSPEn3Sdq1nhtgZmavVEvNfSnwnojYCtgamCDp7cBUYFZEjAZm5ftIGgNMAsYCE4AzJfWtR+HNzKxtnYZ7JM/lu/3zTwATgel5+XRgj3x7InB5RCyNiAeBecD4bi21mZl1qKY2d0l9Jf0ZeAy4ISJuA4ZExCKA/HujvPowYH7l6QvyMjMzW01qCveIWB4RWwPDgfGS3tTB6mrrJV6xkjRF0hxJc5YsWVJbac3MrCZd6i0TEU8DN5Ha0hdLGgqQfz+WV1sAjKg8bTiwsI3XOjsixkXEuMGDB69E0c3MrD219JYZLGm9fHsgsDNwLzATmJxXmwxcm2/PBCZJWlPSpsBoYHZ3F9zMzNrXr4Z1hgLTc4+XPsCMiPippN8DMyQdBDwM7AkQEXMlzQDuBpYBh0TE8voU38zM2tJpuEfEX4E3t7H8CWCndp4zDZi2yqUzM7OV4hGqZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgRzuZmYFcribmRXI4W5mViCHu5lZgToNd0kjJP1a0j2S5ko6Ii8fJOkGSffn3+tXnnO0pHmS7pO0az03wMzMXqmWmvsy4AsR8Ubg7cAhksYAU4FZETEamJXvkx+bBIwFJgBnSupbj8KbmVnbOg33iFgUEXfk288C9wDDgInA9LzadGCPfHsicHlELI2IB4F5wPjuLriZmbWvS23ukkYBbwZuA4ZExCJIXwDARnm1YcD8ytMW5GWtX2uKpDmS5ixZsqTrJTczs3bVHO6S1gGuAj4bEc90tGoby+IVCyLOjohxETFu8ODBtRbDzMxqUFO4S+pPCvZLIuLqvHixpKH58aHAY3n5AmBE5enDgYXdU1wzM6tFLb1lBJwH3BMR3648NBOYnG9PBq6tLJ8kaU1JmwKjgdndV2QzM+tMvxrW2R7YH7hT0p/zsmOAk4AZkg4CHgb2BIiIuZJmAHeTetocEhHLu73kZmbWrk7DPSJuoe12dICd2nnONGDaKpTLzMxWgUeompkVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYEc7mZmBXK4m5kVyOFuZlYgh7uZWYE6DXdJ50t6TNJdlWWDJN0g6f78e/3KY0dLmifpPkm71qvgZmbWvlpq7hcAE1otmwrMiojRwKx8H0ljgEnA2PycMyX17bbSmplZTToN94j4DfBkq8UTgen59nRgj8ryyyNiaUQ8CMwDxndTWc3MrEYr2+Y+JCIWAeTfG+Xlw4D5lfUW5GVmZrYadfcJVbWxLNpcUZoiaY6kOUuWLOnmYpiZNbeVDffFkoYC5N+P5eULgBGV9YYDC9t6gYg4OyLGRcS4wYMHr2QxzMysLSsb7jOByfn2ZODayvJJktaUtCkwGpi9akU0M7Ou6tfZCpIuA94FbChpAXAccBIwQ9JBwMPAngARMVfSDOBuYBlwSEQsr1PZzcysHZ2Ge0Ts085DO7Wz/jRg2qoUyszMVo1HqJqZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgVyuJuZFcjhbmZWIIe7mVmBHO5mZgWqW7hLmiDpPknzJE2t1/uYmdkr1SXcJfUFzgDeB4wB9pE0ph7vZWZmr1Svmvt4YF5E/D0i/gNcDkys03uZmVkriojuf1HpI8CEiDg4398feFtEHFpZZwowJd/dDLiv2wvSdRsCjze6ED2E98UK3hcreF+s0BP2xSYRMbitB/rV6Q3VxrKXfYtExNnA2XV6/5UiaU5EjGt0OXoC74sVvC9W8L5Yoafvi3o1yywARlTuDwcW1um9zMyslXqF+x+B0ZI2lbQGMAmYWaf3MjOzVurSLBMRyyQdClwP9AXOj4i59XivbtajmokazPtiBe+LFbwvVujR+6IuJ1TNzKyxPELVzKxADnczswI53M3MCuRwt5UiqU+r+22NbTCzBnG4t0HS+vn3EElDGl2enigiXgKQtJekAdFkZ+ZbvtwkbSdpy0aXZ3Xxl3r3kbR5/r2OpD27+/Ud7q3kMH+/pIOBnwPrNrhIPU7LP7SkycDRpGHYTUNSn4h4SdK2wGnAgFaP921Myeqv8qU+RNLAiIiSt7deJA0AdpN0DfAzYGm3v0eTVbg6JWkg8FngEOCmiNgvL1ez1U47Iul1wE+B90TEIkn/A6wNXB0Rzze2dPUnaW3gl8ARETFH0luAXSLipAYXrS4kvQaIiFgs6XzSF/pmwP4RMTuv4/+RLsgBfyewJrB1RDwpqV9ELOuO16/X3DK9VkT8O3+Q/woskPRp4LqIeFjSq4EXIqLbv2V7g5Yaa777ImkfbSdpV2B94M3AWsBZDSpiXbXa/jWAfwPbSNoLGAa8Pof8PhGxvFHl7G6S1gJOAO6S9DSwUUR8QNJngBskfSYiLnGwd67VF+By4Buk6VlmSTomIn4uaRAwJCLuWZX3crNMK5KOAIiI3UjTKGwOfETS+4BzgTZnYCudpL65KUKSPhwRD5PmC5oIXBMRewJfB7Zo3S5bgmqw51lPXwROAQ4k1b4OBvYCllFYpSki/gVcAWwB7ET6vyAizgQ+BJwm6fTGlbD3kbQ98F7gkoj4JvAd4NuSPg9cQsqdVVLcP+GqkLQj8BbgFwARcRWpPWw9UtvyHRGxoHElbJxKTXQGab5+IuJzEfGxXNvYBTgKOKdSuy1GJdj/lxTifSLiF8D2EXERaaK8HwE/K+nITlI/gIi4HjgGeArYUtL2uc39V+TPg3WspdYuaQIwndT8+1dJ4yPiQuAAYFvgdxFxzSq/n4+kXrbTDyDt4LuAUyPiwfz4q4D+EfFEdf1GlbdR8mH4thGxf266+jDwUERcJ+lyUi3kJ40tZf1Iehvpc7FDvt+PdGg9EPgS8FJEfKOBRexW+WhteW4m+AFwOOnE31HAq4CfAH+MiEbPad5rSNoAOB04OSL+JOmLwH7AN3Jlsts0dc290nywVj7jfwnwSVL78SRJYwEi4plmDvbKfloT6C/pEFITzI7ADyWNAg4uMdglHVi5+xLwtKT1Wk585c/COOCUlmAvpXtg5WjtauCvEfFoRDxFOoq9HzgI2LGU7a23vJ8OIl16dBxARJwCHA98T9JB3fl+TRvuOaRfkrQm6cN7PvB70omxI4CRwKckbVV9XjMFeyXUB+bfZ5OuPPNW4DsRsRdp5s9NIuK5BhSxrnK32JC0Zq6l/xn4J7ANqWcQko4FPlztIVTSZ0TS1sCSiDgh3x8QyXeAi4G/lLS93a16/invp4tJebOVpA9J6h8RPwYmkLped997N/vfRdIlpIuLTAPeAVxAOvy8HpgKfKsZDzsrh+Sbkk72/InUQ+TYlhqdpG8BW0bEhAYWte4kfY504vTdpHMynwUWk644tiWwU0Q8XeJRXe7y+hPg0Ny+jqSRpBPI34iIFxtZvp6s+nlQmgJ9DeAx4BpSC8Fw4B5S9+Enuvv9m7bmXvFP4NLc9HId8BFg94h4EpgaEY8342FnDvb+pBOoF5E+mDsAAyStpTQqc0vgfxpYzLqQ9JrcBxlJ+5Ka664lnWifTzq0vorUz/1DOdj7lhDsWjHydpjSSO1HSG3E2+Uun5C6uv7Lwd6xSrB/llQzvwu4ENgU+D9SL6vtgaH1eH/X3KXTSH1K98n3hwCXAfsCi0vs+VErSVsAEyPiBEl3AIdFxK2S3kHq4740Il5obCm7n6T3Al8AngSejojP5OWfIA1u+1ZEXFpZv4gau1aMvN0e+AqpZ8w/gN8Bm5C+yJ8mNdN8snEl7R1ypXBj0snojwKfB4ZFxBRJG0TEE5LeEBF/q8v7F/CZrJlePmx8KNAvImZImgkMAr4LfAy4NyK+1Miy9gSSNiH1b34NcHhE/FjSG4BzgMkR8VAjy9fdlAapLY2IFyRdQOrDfUREXFBZZ1fg+8CBEXFLQwpaB5X/jVcBNwL7kHqObRcROymNyH0RWLceTQglk/Q10jmaTfJ4ECSdS+o2e3W93rdpmmUqH95NSc0MbyedML2RVCO5GBgF/L4l2EscjNORluYnSZtIenVE/IMUZHcD60naGDgPuKq0YM8OJfX+6ANcDnwOOETScZV1lpD6tpcU7OtWjlB3JY3tWB/YhbRPAN4FDHSwdyz3ums9v9C/SZ+lr+bHDiUNUlrlvuwdlqXJau4DST1hFuZBA0g6E3gd8MGI+E9l3b5R0BDyzlS+/D4IfJk0Ydo3Sc0vo0jNFPOARyLiuHZfqJeStGZELM3nGW4CvhwRNylNJ/Bt4AHgUWC9iDgkP6fXN8dIWofU1PQAqcLzGOnv/QHgkIj4SW6mOQfYrdAv9W6XWwP+QWram0b6kjyA9P+0MfCxqPOAyKKGSbel1T/g4aQ+pd9teTwiPiPpIlJNZXFleTMFe0vPmOGkf/QDSf/gU4AhpMFJ72pcCetL0j7AHpK+EhH3S7oUOFvSiRHxQ0kfAk4knQg7OD+n1wc7QEQ8J+kXpKaYpyNidG56eg0wRtJ6pEFLxznY29eqZ8wBpHMTVwC7k1oKDiedTO1DqlQvbueluq9MBXw+ayLpFOBXpA/tV4EjgV+TQn0WsGus4kQ9vZGkERExPx9Gfo00y+N2+bEtSEPOnwdOjIgHGljUusk9Y75Hmmbi5Ij4Yz5pfB6pJ9Xxeb2W2n1RR3X5iPZc0nmou4EvAmNJTTQbALdENwyHbwZ5XMDOpH32h9yU+dG87OyIuHp1VQyaok1ZaTKerSLiZxFxPqkWdgppfocDgM9HxD3N1uVRaQTuMfDfI5W/AP+RNE3SRhFxJ6nG8Typ50RxcnPUCxExhXTIfKKkiRHxW+A9wO6Srsyr/wfKO6qLiH9HxL6k7nrrA1eSQn4ADvZOVc5VTSI1Xx0CfFLSoIhYSPrivIY8GHB1HfEVXXPPO31tUtfGM0mzGB5I6hkzktSu/FBEfLhl/RIOtbsi19g/TZrG9djck2hP0pwp0yPiroYWcDWQtHH+JySfc/gSqSnqrLx/do40cVbxJK1BaiP+IPC3iNi9wUXqFSSNJk2LfBjQl5Q3C0hzET2kNBJ1tY4LKDrcW0janxRgz5AG5dwH7EY6FL8AeDVplOG/G1XGRpI0HvgWaf6Qw/MH9UDSCLojSf39i/qgVE4gT2BFn/anSEd0A0knUe8EvhQrRuQ2zZe/0sjUBVHQDJf1kisAR5NGLr8rIu7KXUpPJc0WekRE3Lfay9UMn9VcGxkOPBlpNOGnSGerW9qWd46IGxtayNWoVc+YFyLiBklDSV92a5D6+q8BjImImxtZ1nqSNILUM+YA0qRgbyT3EiEd8e0dBc3yaN1HL79wC7lt/VhSi8BJEfGbvPwY4PsR8cxqL2MzhHuLHPLvI3Xx2zkiFjW4SKtdS+0zd/G7hNTdrWXOkIGkk817ATtGxCMNLGpd6OUX3RgNHB0RH8/3NwSOA26LiIvbeo5Zq54xE0kVgR/lHmeHkS7CcU1E/LCR5WyKE6otcj/2+0hzxyxSE17YNwd7P1L74DHAryXtKuksUj/n44CvN0GwvwXYCJiYj+SINEHcw8BbJfVpGcTmYLcWrYJ9Gul/aE/gXkkjI+J00gC4/SVt1sCiNlfNvZm1bi+WdDJp0rRtgbmkuUOeBz5VHcxVilb/lGeQpu2dTeryN4Q0f8rVpDEQB0TErY0qq/V8kt5M6jK6f27iPJ7UnLd3RPxK0iaRRng3TFPV3JtZJdiOyj1iziFNWXteRBxJGh49llSbLU5l+/ckVWrGk2bmO4v05TYS2JrUTHNrs3WLtc5J2lfSgHyy9GjSyPYdAfJYiMOAGyW9p9HBDk0wQtVWULowyX+Ak3j5lYPGAj8EzoiCrxGbT3qdyooLPM+V9E9Sf/aTI2J2I8tnPZekMaQeVctJVyQ7ihTw75H0POn6ypdLuiPqNMtjV7nmXji9/EowS0k11W8Dh0k6Kj80FrgoIqY3oIirTe7L/ilgvKQT8rIFpJGZ27Za1+2V9l8RcXdE/ByYTJrffm1SN+G1yCNQJa3dU4Id3ObeFCS9kfRBnBIRL+ZeQ+8k9Rr6Q0Qc3tACrmZ5f5wDrEM6+bU56RqwyxpaMOtx2jhX9UbSGJmxwKWkKUyOIZ2z+lxEPNuQgrbB4V6olhFxSrMc9gFuI7UtfzYiluR1Wq6o87kGFrUhJA0iDQt/Hekk2L2NGEVoPVerk/DbAU+QBro9R7qYz9uB30bEBZKG97QmTYd7gXKXrIfz7R+Tpl14ljRY5y/A10lXYH8fsG9E/KtBRW2oPGHYUaTRuB9ohqkWrHaVMSFTSAP7bga2IE2i93tJHwb2Jl2Za04jy9oWh3thlC5efDNptNxgUnvydcCGpIsvbEGau3sj0pzlf2xQUXsMpeukXtOsX3LWPknbkJrwdiCdq9qENKHa9yLiKkmjI+L+RpaxPQ73AinNx30UKdDfH2lK30GkyaA2Jn1IW06wmnHtrEwAAAOUSURBVFmFpH4RsSwf2b2N1BQzgtQ7ZifgZNIlGI+MiMsaV9KOubdMgfIMhp8mzU53fl72JHArae6UDcnT15rZCvnId+s8FcUXgfuBu4DNgDNzhejvwGnAVQ0raA3cz71QEXGfpHcCZ0iaTWpnHwfcWeLUAmarKk9HMgzYgzS3/ZxYMRX0i8DnJA0hXeNgx54+kts194JFupjx/qRBOzOBPhHxqcaWyqznySdPl0fE74HRpIFKd+Wpj8lzxlxGGsQ0uaXDQk/mNvcmoXQd0Ot90tCsfUrXP92OdOWkHYAgnWy/XdLuwI295X/INfcmERHuDWLWAUmvJ/UoOzKPRr02P/RRSX8DduhN/0OuuZtZ05O0NvB9YCvS+alrcx/3TYDXAqMaPT97VznczawpSeoblYudSxpF6iHzAnAFaTKwXjti2eFuZk2n1YVbjiNdy2ARqXvjNFI34l8CN/Wmppgqt7mbWdOpBPsZpNHaC1gxmdwxwBLSbI9rNaqMq8r93M2s6eSLsYwBXh0R+0n6EXBsRDyer6I0TdJr86UXeyXX3M2saeQL1gAMJDXDPC/pIuAfEXFKfuybksZHxN8bUshu4pq7mTWFfFF0JG1Amgl0X2Ap8HFSjxgknQ70L+GqXA53M2sW6wJHAG8mXVIygMMlPQn8WtKtwAYR8YFGFrK7uLeMmRWt1UU3TgZ2JV0c/RZgbkS8lK+R+gjwYm/tHdOaw93MitXGZfIGkWrwXyZN5XsBqYfMtsDpJV071ydUzaxYlRr7VyXNIA1SWj//7gscBlwJPFdSsINr7mZWqJaBSpLeCpwI/IB04nRr4IqIuFbSZsD6EfGHRpa1HhzuZlas3JZ+JfCViLha0gjSrI8TgAeBE1oGNJXGzTJmVpR80Y2W3/cC/YGpABExH7ieNOPjC6UGO7jmbmYFaTVnzHTgTkDAAaQLcOwSEX/Pg5mWVScOK41r7mZWjEqwzyDV2p8D9gbeDlwE3CPpvRGxtORgBw9iMrPCSNoYeDQiTpR0NXBxRDwr6VTSdL69djKwrnDN3cxKswQYIOkB4IGIOE1Sf+DnwKyIuLbjp5fB4W5mRckX2Lic1N7+cK7Jn08ajTqnoYVbjXxC1cyKI6kPaZqBDwJDgUci4tDGlmr1cribWdEkDQCWljYCtTMOdzOzArnN3cysQA53M7MCOdzNzArkcDczK5DD3cysQA53M7MCOdzNzAr0//UYerIRPH6FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df['category'].value_counts().plot.bar(rot=50, title='Number of samples by category');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the data set size is in good balance that each category has similar number of rows. This is really good for training a model. In reality we may find the a great number of sample data belong to just a few top categories, in that case we might want to reblance the training data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now check the statistics of the number of words in each sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    2225.000000\n",
       "mean      390.295281\n",
       "std       241.753128\n",
       "min        90.000000\n",
       "25%       250.000000\n",
       "50%       337.000000\n",
       "75%       479.000000\n",
       "max      4492.000000\n",
       "Name: text, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'].str.split().apply(len).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "samples/words-per-sample ratio is: 6\n"
     ]
    }
   ],
   "source": [
    "print(\"samples/words-per-sample ratio is:\", round(2225 / 390))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So within the total 18768 number of records, there are average **390** words in each text, and the minimum number of words in text is **90** and maximum number of words in text is **4492**, and the medium number of words is **337**\n",
    "\n",
    "#### Now take a look at the text that have the least number of words and most number of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'blunkett hints at election call ex-home secretary david blunkett has given fresh clues that the general election will be announced on monday.  he told bbc radio five live:  i m out in my constituency getting ready for what we presume will be an announcement very shortly at the weekend.  he clarified that he meant he would be in his sheffield seat this weekend  not that he expected an election call then. tony blair is tipped to ask the queen on monday to dissolve parliament ready for a 5 may poll.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the text that have the least number of words\n",
    "df[df['text'].str.split().apply(len) == 90]['text'].values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'terror powers expose  tyranny  the lord chancellor has defended government plans to introduce control orders to keep foreign and british terrorist suspects under house arrest  where there isn t enough evidence to put them on trial.  lord falconer insists that the proposals do not equate to a police state and strike a balance between protecting the public against the threat of terrorism and upholding civil liberties. but thriller writer frederick forsyth tells bbc news of his personal response to the move.    there is a mortal danger aimed at the heart of britain. or so says home secretary charles clarke. my reaction  so what  it is not that i am cynical or just do not care. i care about this country very much.  but in the 66 years that i have been alive  there has not been one hour  of one day  of one month  of one year  when there has not been a threat aimed at us. my point is  the british have always coped without becoming a dictatorship. we have coped with fear without becoming a st'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the text that have the most number of words, show only the first 1000 characters \n",
    "df[df['text'].str.split().apply(len) == 4492]['text'].values[0][:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the data are pretty clean already, if not so we would run clean up process like removing new line and tab characters or removing special characters, or replacing numbers with a token...\n",
    "``` df['text'] = df['text'].str.replace('\\n', ' ').replace('\\t', ' ')```\n",
    "\n",
    "In this case we will leave the data as is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}