NLP/text_classification_Reuter.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "text_classification_Reuter.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ramapriyakp/Portfolio/blob/master/NLP/text_classification_Reuter.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kYlKycacvSNK",
        "colab_type": "text"
      },
      "source": [
        "## Reuters-21578 collection\n",
        "Reuters-21578 is arguably the most commonly used collection for text classification during the last two decades. It has been used in some of the most influential papers on the field.\n",
        "\n",
        "This dataset contains structured information about newswire articles that can be assigned to several classes, therefore making this a multi-label problem. It has a highly skewed distribution of documents over categories, where a large proportion of documents belong to few topics.\n",
        "\n",
        "The collection originally consists of 21,578 documents, including documents without topics and typographical errors. For this reason, a subset and split of the collection is traditionally used. This split also focus only on the categories that have at least one document in the training set and the test set. The dataset has 90 categories with a training set of 7769 documents and a test set of 3019 documents.\n",
        "\n",
        "## Data Collection Stats\n",
        "Understanding your collection is always the first step of any data science problem. Lets have a quick look at the Reuters collection and its documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nzUSZwV5pG7g",
        "colab_type": "code",
        "outputId": "16b585e8-c021-4137-d9d4-43b6a3877631",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ftAmECfOpsPs",
        "colab_type": "code",
        "outputId": "1e145515-617a-4b5a-bd6c-4406f59fed3e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "cd /content/drive/My Drive/NLP"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/NLP\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZ90BCW3qRnb",
        "colab_type": "code",
        "outputId": "00e8c11a-18e5-4637-8f80-e65fd4f55519",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "import nltk\n",
        "nltk.data.path.append('/content/drive/My Drive/NLP')\n",
        "nltk.download('stopwords')\n",
        "nltk.download('reuters')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package reuters to /root/nltk_data...\n",
            "[nltk_data]   Package reuters is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DaOOeH0To6uh",
        "colab_type": "code",
        "outputId": "c624a54d-0580-4f2d-f2b9-bd3ae09b3918",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "from nltk.corpus import reuters\n",
        "\n",
        "# List of document ids\n",
        "documents = reuters.fileids()\n",
        "print(\"Documents: {}\".format(len(documents)))\n",
        "\n",
        "# Train documents\n",
        "train_docs_id = list(filter(lambda doc: doc.startswith(\"train\"), documents))\n",
        "print(\"Total train documents: {}\".format(len(train_docs_id)))\n",
        "\n",
        "# Test documents\n",
        "test_docs_id = list(filter(lambda doc: doc.startswith(\"test\"), documents))\n",
        "print(\"Total test documents: {}\".format(len(test_docs_id)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Documents: 10788\n",
            "Total train documents: 7769\n",
            "Total test documents: 3019\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z2CAPSrFo-2Y",
        "colab_type": "code",
        "outputId": "2afe7d48-fc94-4932-fd0f-f56d806f2480",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        }
      },
      "source": [
        "# Example of a document (with multiple labels)\n",
        "doc = 'training/9865'\n",
        "print(reuters.raw(doc))\n",
        "print()\n",
        "\n",
        "print(reuters.categories(doc))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "FRENCH FREE MARKET CEREAL EXPORT BIDS DETAILED\n",
            "  French operators have requested licences\n",
            "  to export 675,500 tonnes of maize, 245,000 tonnes of barley,\n",
            "  22,000 tonnes of soft bread wheat and 20,000 tonnes of feed\n",
            "  wheat at today's European Community tender, traders said.\n",
            "      Rebates requested ranged from 127.75 to 132.50 European\n",
            "  Currency Units a tonne for maize, 136.00 to 141.00 Ecus a tonne\n",
            "  for barley and 134.25 to 141.81 Ecus for bread wheat, while\n",
            "  rebates requested for feed wheat were 137.65 Ecus, they said.\n",
            "  \n",
            "\n",
            "\n",
            "\n",
            "['barley', 'corn', 'grain', 'wheat']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e0JAY8yVqzpB",
        "colab_type": "code",
        "outputId": "59dbddcf-f63d-42a4-ebd1-a596b29070b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "from operator import itemgetter\n",
        "from pprint import pprint\n",
        "\n",
        "# List of categories \n",
        "categories = reuters.categories();\n",
        "print(\"Number of categories: {}\".format(len(categories)))\n",
        "print()\n",
        "\n",
        "print(categories)\n",
        "print()\n",
        "\n",
        "# Documents per category.\n",
        "category_distribution = [(category, len(reuters.fileids(category))) \n",
        "                         for category in categories]\n",
        "\n",
        "category_distribution = sorted(category_distribution, \n",
        "                               key=itemgetter(1), \n",
        "                               reverse=True)\n",
        "\n",
        "print(\"Most common categories\")\n",
        "pprint(category_distribution[:5])\n",
        "print()\n",
        "\n",
        "print(\"Least common categories\")\n",
        "pprint(category_distribution[-5:])\n",
        "print()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of categories: 90\n",
            "\n",
            "['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', 'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn', 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', 'dmk', 'earn', 'fuel', 'gas', 'gnp', 'gold', 'grain', 'groundnut', 'groundnut-oil', 'heat', 'hog', 'housing', 'income', 'instal-debt', 'interest', 'ipi', 'iron-steel', 'jet', 'jobs', 'l-cattle', 'lead', 'lei', 'lin-oil', 'livestock', 'lumber', 'meal-feed', 'money-fx', 'money-supply', 'naphtha', 'nat-gas', 'nickel', 'nkr', 'nzdlr', 'oat', 'oilseed', 'orange', 'palladium', 'palm-oil', 'palmkernel', 'pet-chem', 'platinum', 'potato', 'propane', 'rand', 'rape-oil', 'rapeseed', 'reserves', 'retail', 'rice', 'rubber', 'rye', 'ship', 'silver', 'sorghum', 'soy-meal', 'soy-oil', 'soybean', 'strategic-metal', 'sugar', 'sun-meal', 'sun-oil', 'sunseed', 'tea', 'tin', 'trade', 'veg-oil', 'wheat', 'wpi', 'yen', 'zinc']\n",
            "\n",
            "Most common categories\n",
            "[('earn', 3964),\n",
            " ('acq', 2369),\n",
            " ('money-fx', 717),\n",
            " ('grain', 582),\n",
            " ('crude', 578)]\n",
            "\n",
            "Least common categories\n",
            "[('castor-oil', 2),\n",
            " ('groundnut-oil', 2),\n",
            " ('lin-oil', 2),\n",
            " ('rye', 2),\n",
            " ('sun-meal', 2)]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uMSZOMNKrAaS",
        "colab_type": "code",
        "outputId": "5f8069d9-0cf9-49d5-e5d5-5fddacfee6db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.preprocessing import MultiLabelBinarizer\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.multiclass import OneVsRestClassifier\n",
        "\n",
        "stop_words = stopwords.words(\"english\")\n",
        "\n",
        "train_docs_id = list(filter(lambda doc: doc.startswith(\"train\"), documents))\n",
        "test_docs_id = list(filter(lambda doc: doc.startswith(\"test\"), documents))\n",
        "\n",
        "train_docs = [reuters.raw(doc_id) for doc_id in train_docs_id]\n",
        "test_docs = [reuters.raw(doc_id) for doc_id in test_docs_id]\n",
        "\n",
        "# Tokenisation \n",
        "vectorizer = TfidfVectorizer(stop_words=stop_words)\n",
        "\n",
        "# Learn and transform train documents\n",
        "vectorised_train_documents = vectorizer.fit_transform(train_docs)\n",
        "vectorised_test_documents = vectorizer.transform(test_docs)\n",
        "\n",
        "# Transform multilabel labels\n",
        "mlb = MultiLabelBinarizer()\n",
        "train_labels = mlb.fit_transform([reuters.categories(doc_id) \n",
        "                                  for doc_id in train_docs_id])\n",
        "test_labels = mlb.transform([reuters.categories(doc_id) \n",
        "                             for doc_id in test_docs_id])\n",
        "\n",
        "# Classifier \n",
        "classifier = OneVsRestClassifier(LinearSVC(random_state=42))\n",
        "classifier.fit(vectorised_train_documents, train_labels)\n",
        "predictions = classifier.predict(vectorised_test_documents)\n",
        "\n",
        "print(\"Number of labels assigned: {}\".format(sum([sum(prediction) \n",
        "                                                  for prediction in predictions])))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of labels assigned: 3126\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djxpwAaIvuwt",
        "colab_type": "text"
      },
      "source": [
        "# How well have we done?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y1azbDjKPuCA",
        "colab_type": "text"
      },
      "source": [
        "**Confusion Matrix**\n",
        "![Confusion Matrix](https://hackernoon.com/hn-images/1*y4HwoAEgx1Js19hCkPM7XA.png)\n",
        "\n",
        "__Precision__ — Also called Positive predictive value\n",
        "The ratio of correct positive predictions to the total predicted positives\n",
        "\n",
        "\n",
        "![Precision](https://hackernoon.com/hn-images/1*7pXTS_beJfTlkntFHiBh7g.png)\n",
        "\n",
        "__Recall__ — Also called Sensitivity, Probability of Detection, True Positive Rate\n",
        "\n",
        "Ratio of correct positive predictions to the total positives examples\n",
        "\n",
        "\n",
        "![Recall](https://hackernoon.com/hn-images/1*VWD0-yDiy5gBZy7ZTIZgXg.png)\n",
        "\n",
        "__F Score__: This is a weighted average of the true positive rate (recall) and precision."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MNq7AgH-reSo",
        "colab_type": "code",
        "outputId": "651185d9-6bdf-4afd-b824-99c0be8c5675",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "source": [
        "from sklearn.metrics import f1_score, precision_score, recall_score\n",
        "\n",
        "# Show our quality\n",
        "precision = precision_score(test_labels, predictions, average='micro')\n",
        "recall = recall_score(test_labels, predictions, average='micro')\n",
        "f1 = f1_score(test_labels, predictions, average='micro')\n",
        "print(\"Micro-average quality numbers\")\n",
        "print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, \n",
        "                                                                     recall, \n",
        "                                                                     f1))\n",
        "\n",
        "precision = precision_score(test_labels, predictions, average='macro')\n",
        "recall = recall_score(test_labels, predictions, average='macro')\n",
        "f1 = f1_score(test_labels, predictions, average='macro')\n",
        "print(\"Macro-average quality numbers\")\n",
        "print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, \n",
        "                                                                     recall, \n",
        "                                                                     f1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Micro-average quality numbers\n",
            "Precision: 0.9517, Recall: 0.7946, F1-measure: 0.8661\n",
            "Macro-average quality numbers\n",
            "Precision: 0.6305, Recall: 0.3715, F1-measure: 0.4451\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
            "  'precision', 'predicted', average, warn_for)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
            "  'precision', 'predicted', average, warn_for)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpBup9hEv3Ox",
        "colab_type": "text"
      },
      "source": [
        "# If you see a weird warning message about \"Precision being ill-defined\":\n",
        "Some metrics can be in a position of indeterminate value when for instance, the classifier decides to not classify any articles in a specific category. This would imply a 0/0 precision. Scikit learn warns us about this, and reports back this quality as 0.0."
      ]
    }
  ]
}