Multi-Label-Learning/Multi_Label_Learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kochlisGit/Advanced-ML/blob/main/Multi_Label_Learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jU-Qf7FLNh8U"
      },
      "source": [
        "**Title**\n",
        "\n",
        "Assignment 2 - Multi-Label Learning\n",
        "\n",
        "**Course**\n",
        "\n",
        "Advanced Machine Learning Topics - Master in Artificial Intelligence\n",
        "\n",
        "**Authors**:\n",
        "\n",
        "1.   Anastasia Papadopoulou\n",
        "2.   Vasileios Kochliaridis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "3XEVImvQOvKq"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHSg3W_ZRD_M"
      },
      "source": [
        "Let's download the DeliciousMIL dataset!\n",
        "\n",
        "DeliciousMIL consists of a subset of tagged web pages from the social bookmarking site delicious.com. The original web pages were obtained from DeliciousT140 dataset. Users of the website delicious.com bookmarked each page with word tags. The class labels of this dataset are: reference, design, programming, internet, computer, web, java, writing, English, grammar, style, language, books, education, philosophy, politics, religion, science, history, and culture.\n",
        "\n",
        "This dataset provides ground-truth class labels to evaluate performance of multi-instance learning models on both instance-level and bag-level label predictions. Each text document is a bag within a multi-instance learning framework consisting of multiple sentences (instances). The goal is to predict document-level and sentence-level class labels on the test set using a model which is trained given only the document-level class labels in the training set. To evaluate performance of such a model, we have manually labeled 1468 randomly selected sentences from the test documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zTpG0YXSK_09",
        "outputId": "c08e0bc4-bce5-4fc6-da17-9200f579dfcc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Download completed!\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "from zipfile import ZipFile\n",
        "\n",
        "dataset_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00418/DeliciousMIL.zip'\n",
        "download_zipfile = 'deliciousmil.zip'\n",
        "train_inputs_fp = 'Data/train-data.dat'\n",
        "train_labels_fp = 'Data/train-label.dat'\n",
        "test_inputs_fp = 'Data/test-data.dat'\n",
        "test_labels_fp = 'Data/test-label.dat'\n",
        "label_names_fp = 'Data/labels.txt'\n",
        "token_names_fp = 'Data/vocabs.txt'\n",
        "\n",
        "req = requests.get(dataset_url)\n",
        "with open(download_zipfile, 'wb') as output_file:\n",
        "  output_file.write(req.content)\n",
        "print('Download completed!\\n')\n",
        "\n",
        "zf = ZipFile(download_zipfile, 'r')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2m4BxjxtRSSD"
      },
      "source": [
        "Reading the labels & tokens (vocals)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VGqLEo6gRRXL",
        "outputId": "3abc9f58-e99a-4fed-e837-3e02608a9462"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Labels\n",
            "{0: 'programming', 1: 'style', 2: 'reference', 3: 'java', 4: 'web', 5: 'internet', 6: 'culture', 7: 'design', 8: 'education', 9: 'language', 10: 'books', 11: 'writing', 12: 'computer', 13: 'english', 14: 'politics', 15: 'history', 16: 'philosophy', 17: 'science', 18: 'religion', 19: 'grammar'}\n",
            "\n",
            "First 5 Tokens\n",
            "0:\tdefer\n",
            "1:\twood\n",
            "2:\tspider\n",
            "3:\thang\n",
            "4:\twoodi\n",
            "5:\tcomic\n",
            "6:\tlocal\n",
            "7:\tseven\n",
            "8:\torthographi\n",
            "9:\torigin\n",
            "Extracted 8520 tokens\n"
          ]
        }
      ],
      "source": [
        "def read_dict_from_textfile(fp):\n",
        "  d = {}\n",
        "\n",
        "  with zf.open(fp, 'r') as txtfile:\n",
        "    lines = txtfile.read().splitlines()\n",
        "    for line in lines:\n",
        "      name, i = line.decode('utf-8').split(', ')\n",
        "      d[int(i)] = name\n",
        "  return d\n",
        "\n",
        "\n",
        "labels_dict = read_dict_from_textfile(label_names_fp)\n",
        "print('Labels\\n' + str(labels_dict))\n",
        "\n",
        "tokens_dict = read_dict_from_textfile(token_names_fp)\n",
        "\n",
        "print('\\nFirst 5 Tokens')\n",
        "for i in range(10):\n",
        "  print('{}:\\t{}'.format(i, tokens_dict[i]))\n",
        "\n",
        "print('Extracted {} tokens'.format(len(tokens_dict)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SS7uFyoepsBA"
      },
      "source": [
        "Reading training & test documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VMpib9Zkpou4",
        "outputId": "15c9fc39-902b-453c-c1b8-d6b9e43721ab"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "First 5 train docs\n",
            "0:\trubi rail helper demo more info auto complet see new helper action \n",
            "1:\tnull length substr locat exec messag messag pleas edit remov follow word content roll stone com news song previou next page good vibrat beach boy smell teen spirit nirvana want hold hand beatl hound dog elvi god know beach boy walk line johnni cash heaven led zeppelin sympathi devil roll stone river deep mountain high turner woman cri bob day buddi holli georgia mind ray charl heartbreak hotel elvi bridg over troubl water simon track tear robinson miracl messag flash five man love woman long tall salli littl richard whole jerri lee lewi california girl beach boy brand new bag jame brown whole love led zeppelin strawberri field forev beatl mysteri train elvi feel good jame brown \n",
            "2:\tsubstr remov addit valu return function result fals progress through version final present adopt wide rang industri project spring focus around provid way manag busi object spring both comprehens modular maximum valu learn curv easi introduc spring increment exist project spring design ground help write code easi test spring ideal framework test driven project spring increasingli import integr technolog role recogn sever larg vendor spring necessarili one more framework depend project spring potenti one stop shop address infrastructur concern typic applic goe place framework don open sourc project februari spring long heritag expert one one laid out basic architectur think behind spring now develop lead contributor devot full time spring develop support flourish open sourc commun help evolv far more achiev individu spring take care plumb left use strut framework gear particular spring elimin prolifer singleton seen mani project experi major problem reduc testabl object orient spring simpli look class properti constructor argument use invers control depend inject discuss below help achiev simplif spring design applic built depend few possibl busi object spring applic depend spring applic built use spring veri easi unit test spring make use implement choic determin applic architectur choos implement busi interfac local without affect call code spring help solv mani problem without use spring provid altern appropri mani applic see consist spring approach mani import spring essenti technolog dedic enabl build applic use desir goal requir sophist framework conceal much complex develop \n",
            "3:\tsqueak squeak import static squeak new style releas instal help power script base environ learn scienc math encourag explor experiment \n",
            "4:\tpro support pro function download trial web servic test tool lead tool web servic test more download use tool test world pro extend version profession support extend function pro two more request respons editor form editor outlin editor thank team ole featur com \n",
            "Extracted 8251 documents\n"
          ]
        }
      ],
      "source": [
        "def read_inputs_from_file(fp, tokens):\n",
        "  inputs = []\n",
        "  \n",
        "  with zf.open(fp, 'r') as txtfile:\n",
        "    lines = txtfile.read().decode('utf-8').splitlines()\n",
        "    for line in lines:\n",
        "      data = line.split(' ')\n",
        "      num_doc_sentences = int(data[0][1: -1])\n",
        "      doc_index = 1\n",
        "      doc = ''\n",
        "\n",
        "      for i in range(num_doc_sentences):\n",
        "        num_sentence_tokens = int(data[doc_index][1: -1])\n",
        "        doc_index += 1\n",
        "\n",
        "        for i in range(doc_index, num_sentence_tokens + doc_index):\n",
        "          doc += tokens[int(data[i])] + ' '\n",
        "        doc_index += num_sentence_tokens\n",
        "      inputs.append(doc)\n",
        "    return inputs\n",
        "\n",
        "train_docs = read_inputs_from_file(train_inputs_fp, tokens_dict)\n",
        "test_docs = read_inputs_from_file(test_inputs_fp, tokens_dict)\n",
        "\n",
        "print('\\nFirst 5 train docs')\n",
        "for i in range(5):\n",
        "  print('{}:\\t{}'.format(i, train_docs[i]))\n",
        "print('Extracted {} documents'.format(len(train_docs)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nHbtjuGG9zQc"
      },
      "source": [
        "Vectorizing documents by Term Frequency - Inverse Document Frequency (TF-IDF). TF-IDF is considered better than Count Vectorizers because it not only focuses on the frequency of words present in the document but also provides the importance of the words. We can then remove the words that are less important for analysis, hence making the model building less complex by reducing the input dimensions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6ZOiG0KO9ukN",
        "outputId": "097facb7-71d6-4949-e627-7a7f0993be5b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((8251, 8520), (3983, 8520))"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "tfidf_vectorizer = TfidfVectorizer(vocabulary=None)\n",
        "train_inputs = tfidf_vectorizer.fit_transform(train_docs)\n",
        "test_inputs = TfidfVectorizer(vocabulary=tfidf_vectorizer.get_feature_names_out()).fit_transform(test_docs)\n",
        "\n",
        "train_inputs.shape, test_inputs.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyQBAboLD2_m"
      },
      "source": [
        "Reading traing & test labels\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dsfo7RjaEFSl",
        "outputId": "47abb19c-18c9-4768-f65e-bb7d8431335e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Labels of the first 5 docs\n",
            "[[1 0 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]\n",
            " [0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0]\n",
            " [1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]\n",
            " [1 0 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 0]\n",
            " [1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((8251, 20), (3983, 20))"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "import numpy as np\n",
        "\n",
        "\n",
        "def read_labels_from_file(fp):\n",
        "  labels = []\n",
        "\n",
        "  with zf.open(fp, 'r') as txtfile:\n",
        "    lines = txtfile.read().decode('utf-8').splitlines()\n",
        "    for line in lines:\n",
        "      doc_label = []\n",
        "\n",
        "      for output in line.split(' '):\n",
        "        doc_label.append(int(output))\n",
        "      labels.append(doc_label)\n",
        "  return np.int8(labels)\n",
        "\n",
        "\n",
        "train_labels = read_labels_from_file(train_labels_fp)\n",
        "test_labels = read_labels_from_file(test_labels_fp)\n",
        "\n",
        "print('Labels of the first 5 docs')\n",
        "print(train_labels[0:5])\n",
        "train_labels.shape, test_labels.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hQUY6bXFGkv"
      },
      "source": [
        "The main thing we notice about each document is that they may have multiple tags. For example, the 1st document which contains the words:\n",
        "\n",
        "`rubi rail helper demo more info auto complet see new helper action`\n",
        "\n",
        "has the following tags:\n",
        "\n",
        "\n",
        "```\n",
        "1. Programming\n",
        "2. Reference\n",
        "3. Java\n",
        "4. Web\n",
        "5. Computer\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the following example, we are going to use 3 training methods for multi-label data:\n",
        "\n",
        "1.   Binary Relevance\n",
        "2.   Regressor Chain\n",
        "3.   Multi-Target Models\n",
        "\n",
        "Then, we are going to evaluate each method using the following multi-label evaluation metrics:\n",
        "\n",
        "1.   Subset Accuracy\n",
        "2.   Coverage Error\n",
        "3.   Ranking Loss\n",
        "4.   Average Precision\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "x1k6t4tj0xHz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.multioutput import MultiOutputClassifier, ClassifierChain\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import zero_one_loss, coverage_error, label_ranking_loss, label_ranking_average_precision_score\n",
        "\n",
        "RANDOM_STATE = 0\n",
        "\n",
        "classifiers = {\n",
        "    'Multi Target Classifier': MultiOutputClassifier(LogisticRegression(class_weight='balanced', random_state=RANDOM_STATE)),\n",
        "    'Chain Classifier': ClassifierChain(LogisticRegression(random_state=RANDOM_STATE)),\n",
        "    'Random Forest Classifier': RandomForestClassifier(n_estimators=100, random_state=RANDOM_STATE),\n",
        "    'K-nearest Neighbors Classifier': KNeighborsClassifier(n_neighbors=5),\n",
        "    'Multi-layer Perceptron Classifier': MLPClassifier(hidden_layer_sizes=(100,), random_state=RANDOM_STATE)\n",
        "}\n",
        "\n",
        "for classifier_name, clf in classifiers.items():\n",
        "  clf.fit(train_inputs, train_labels)\n",
        "  pred_labels = clf.predict(test_inputs)\n",
        "  print('\\nEvaluating {}'.format(classifier_name))\n",
        "  print(classification_report(test_labels,pred_labels,zero_division='warn'))\n",
        "\n",
        "  if classifier_name == 'Chain Classifier' or classifier_name == 'Multi-layer Perceptron Classifier':\n",
        "    proba_labels = clf.predict_proba(test_inputs)\n",
        "  else:\n",
        "    proba_labels = np.array([[k[1] for k in i] for i in clf.predict_proba(test_inputs)]).T\n",
        "  \n",
        "  print('Subset accuracy = {}'.format((1-zero_one_loss(test_labels, pred_labels))))\n",
        "  print('Coverage error = {}'.format((coverage_error(test_labels, pred_labels))))\n",
        "  print('Ranking loss = {}'.format((label_ranking_loss(test_labels, pred_labels))))\n",
        "  print('Average precision = {}'.format((label_ranking_average_precision_score(test_labels, proba_labels))))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eL0sPZf9poTf",
        "outputId": "5669d759-2317-4cef-97c5-0c98e9d238dd"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Evaluating Multi Target Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.70      0.80      0.75       977\n",
            "           1       0.39      0.62      0.48       228\n",
            "           2       0.57      0.61      0.59      1558\n",
            "           3       0.52      0.72      0.61       372\n",
            "           4       0.56      0.68      0.61      1050\n",
            "           5       0.34      0.58      0.43       537\n",
            "           6       0.38      0.68      0.49       702\n",
            "           7       0.54      0.62      0.58      1079\n",
            "           8       0.49      0.65      0.56       803\n",
            "           9       0.50      0.64      0.56       483\n",
            "          10       0.44      0.62      0.52       507\n",
            "          11       0.42      0.58      0.49       478\n",
            "          12       0.36      0.60      0.45       509\n",
            "          13       0.37      0.59      0.45       355\n",
            "          14       0.42      0.66      0.51       392\n",
            "          15       0.37      0.63      0.47       441\n",
            "          16       0.33      0.61      0.43       269\n",
            "          17       0.45      0.56      0.50       501\n",
            "          18       0.47      0.62      0.53       207\n",
            "          19       0.39      0.56      0.46       133\n",
            "\n",
            "   micro avg       0.47      0.64      0.54     11581\n",
            "   macro avg       0.45      0.63      0.52     11581\n",
            "weighted avg       0.49      0.64      0.55     11581\n",
            " samples avg       0.47      0.62      0.50     11581\n",
            "\n",
            "Subset accuracy = 0.04343459703740904\n",
            "Coverage error = 13.48958071805172\n",
            "Ranking loss = 0.3974027055940003\n",
            "Average precision = 0.7110821262970756\n",
            "\n",
            "Evaluating Chain Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.85      0.59      0.69       977\n",
            "           1       0.94      0.14      0.24       228\n",
            "           2       0.67      0.39      0.49      1558\n",
            "           3       0.81      0.48      0.60       372\n",
            "           4       0.71      0.37      0.48      1050\n",
            "           5       0.49      0.14      0.22       537\n",
            "           6       0.60      0.12      0.20       702\n",
            "           7       0.76      0.26      0.39      1079\n",
            "           8       0.71      0.30      0.42       803\n",
            "           9       0.79      0.28      0.41       483\n",
            "          10       0.73      0.21      0.33       507\n",
            "          11       0.86      0.14      0.23       478\n",
            "          12       0.62      0.09      0.16       509\n",
            "          13       0.68      0.18      0.28       355\n",
            "          14       0.75      0.24      0.37       392\n",
            "          15       0.67      0.07      0.12       441\n",
            "          16       0.67      0.09      0.16       269\n",
            "          17       0.87      0.16      0.27       501\n",
            "          18       0.86      0.15      0.26       207\n",
            "          19       0.70      0.17      0.28       133\n",
            "\n",
            "   micro avg       0.73      0.27      0.40     11581\n",
            "   macro avg       0.74      0.23      0.33     11581\n",
            "weighted avg       0.72      0.27      0.38     11581\n",
            " samples avg       0.39      0.26      0.29     11581\n",
            "\n",
            "Subset accuracy = 0.09917147878483557\n",
            "Coverage error = 17.144363545066533\n",
            "Ranking loss = 0.6894205917171637\n",
            "Average precision = 0.6907462391361098\n",
            "\n",
            "Evaluating Random Forest Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.85      0.39      0.53       977\n",
            "           1       0.94      0.07      0.12       228\n",
            "           2       0.72      0.11      0.19      1558\n",
            "           3       1.00      0.13      0.22       372\n",
            "           4       0.84      0.12      0.22      1050\n",
            "           5       0.70      0.01      0.03       537\n",
            "           6       0.70      0.02      0.04       702\n",
            "           7       0.74      0.07      0.13      1079\n",
            "           8       0.76      0.06      0.11       803\n",
            "           9       0.90      0.08      0.14       483\n",
            "          10       0.78      0.04      0.08       507\n",
            "          11       0.67      0.03      0.05       478\n",
            "          12       0.44      0.01      0.02       509\n",
            "          13       0.93      0.07      0.13       355\n",
            "          14       1.00      0.05      0.10       392\n",
            "          15       0.42      0.01      0.02       441\n",
            "          16       0.80      0.01      0.03       269\n",
            "          17       0.79      0.02      0.04       501\n",
            "          18       0.89      0.04      0.07       207\n",
            "          19       0.60      0.02      0.04       133\n",
            "\n",
            "   micro avg       0.81      0.09      0.16     11581\n",
            "   macro avg       0.77      0.07      0.12     11581\n",
            "weighted avg       0.76      0.09      0.15     11581\n",
            " samples avg       0.18      0.08      0.11     11581\n",
            "\n",
            "Subset accuracy = 0.06628169721315591\n",
            "Coverage error = 18.497865930203364\n",
            "Ranking loss = 0.8593266499562767\n",
            "Average precision = 0.6379886183917951\n",
            "\n",
            "Evaluating K-nearest Neighbors Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.65      0.63      0.64       977\n",
            "           1       0.66      0.27      0.39       228\n",
            "           2       0.50      0.47      0.49      1558\n",
            "           3       0.62      0.47      0.54       372\n",
            "           4       0.55      0.45      0.49      1050\n",
            "           5       0.44      0.22      0.29       537\n",
            "           6       0.44      0.11      0.17       702\n",
            "           7       0.54      0.39      0.45      1079\n",
            "           8       0.54      0.28      0.36       803\n",
            "           9       0.65      0.31      0.42       483\n",
            "          10       0.60      0.19      0.29       507\n",
            "          11       0.59      0.23      0.33       478\n",
            "          12       0.47      0.16      0.24       509\n",
            "          13       0.61      0.26      0.37       355\n",
            "          14       0.69      0.15      0.25       392\n",
            "          15       0.41      0.08      0.14       441\n",
            "          16       0.51      0.14      0.22       269\n",
            "          17       0.59      0.14      0.23       501\n",
            "          18       0.82      0.31      0.45       207\n",
            "          19       0.72      0.22      0.34       133\n",
            "\n",
            "   micro avg       0.56      0.32      0.41     11581\n",
            "   macro avg       0.58      0.27      0.35     11581\n",
            "weighted avg       0.55      0.32      0.39     11581\n",
            " samples avg       0.44      0.31      0.33     11581\n",
            "\n",
            "Subset accuracy = 0.06301782575947779\n",
            "Coverage error = 17.055987948782324\n",
            "Ranking loss = 0.6457425175106934\n",
            "Average precision = 0.5295377939645531\n",
            "\n",
            "Evaluating Multi-layer Perceptron Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.68      0.66      0.67       977\n",
            "           1       0.61      0.40      0.48       228\n",
            "           2       0.52      0.51      0.52      1558\n",
            "           3       0.65      0.54      0.59       372\n",
            "           4       0.52      0.50      0.51      1050\n",
            "           5       0.32      0.35      0.34       537\n",
            "           6       0.39      0.35      0.37       702\n",
            "           7       0.52      0.48      0.50      1079\n",
            "           8       0.48      0.42      0.45       803\n",
            "           9       0.54      0.39      0.45       483\n",
            "          10       0.47      0.38      0.42       507\n",
            "          11       0.52      0.36      0.42       478\n",
            "          12       0.40      0.31      0.35       509\n",
            "          13       0.49      0.37      0.42       355\n",
            "          14       0.58      0.41      0.48       392\n",
            "          15       0.41      0.30      0.35       441\n",
            "          16       0.49      0.29      0.36       269\n",
            "          17       0.54      0.34      0.42       501\n",
            "          18       0.85      0.43      0.57       207\n",
            "          19       0.67      0.32      0.44       133\n",
            "\n",
            "   micro avg       0.51      0.44      0.47     11581\n",
            "   macro avg       0.53      0.40      0.45     11581\n",
            "weighted avg       0.52      0.44      0.47     11581\n",
            " samples avg       0.47      0.42      0.41     11581\n",
            "\n",
            "Subset accuracy = 0.054732613607833325\n",
            "Coverage error = 16.111222696459954\n",
            "Ranking loss = 0.5488462330184353\n",
            "Average precision = 0.6368144320479519\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, we are going to try to convert our multi-label problem to a `binary classification` problem. Firstly, We need to count the frequencies of all 20 labels, and pick the label with the highest frequency."
      ],
      "metadata": {
        "id": "SIFLmDD4LhUj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "label_counts = np.sum(train_labels, axis=0)\n",
        "\n",
        "for label, counts in enumerate(label_counts):\n",
        "  print('Label {}:\\t{}'.format(label, counts))\n",
        "\n",
        "most_frequent_label = np.argmax(label_counts)\n",
        "print('\\nMost Frequent Label in the dataset: {}'.format(most_frequent_label))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aHsK3zmN3gdE",
        "outputId": "7ace680c-cc63-4728-f339-991a3efdf901"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Label 0:\t2050\n",
            "Label 1:\t479\n",
            "Label 2:\t3181\n",
            "Label 3:\t799\n",
            "Label 4:\t2203\n",
            "Label 5:\t1211\n",
            "Label 6:\t1471\n",
            "Label 7:\t2221\n",
            "Label 8:\t1559\n",
            "Label 9:\t1004\n",
            "Label 10:\t1034\n",
            "Label 11:\t939\n",
            "Label 12:\t1049\n",
            "Label 13:\t725\n",
            "Label 14:\t830\n",
            "Label 15:\t898\n",
            "Label 16:\t598\n",
            "Label 17:\t1001\n",
            "Label 18:\t411\n",
            "Label 19:\t224\n",
            "\n",
            "Most Frequent Label in the dataset: 2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "most_frequent_label = 2\n",
        "\n",
        "def create_binary_labels(multi_labels):\n",
        "  binary_labels = np.zeros(shape=multi_labels.shape[0], dtype=int)\n",
        "  binary_labels_index = 0\n",
        "  for x in multi_labels:\n",
        "    if (x[most_frequent_label]):\n",
        "      binary_labels[binary_labels_index] = 1\n",
        "    binary_labels_index += 1\n",
        "  return binary_labels\n",
        "\n",
        "train_binary_labels = create_binary_labels(train_labels)\n",
        "test_binary_labels = create_binary_labels(test_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RvWnjukJMjSa",
        "outputId": "4bd1f3ea-0afe-4c6f-a3c0-6bf76a73cdcb"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(8251,) (3983,)\n",
            "[1 1 0 0 0 1 0 0 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def convert_multilabels_to_binary(labels, most_frequent_label):\n",
        "  binary_labels = np.zeros(shape=labels.shape[0], dtype=np.int8)\n",
        "  ones_indices = np.where(labels[:, most_frequent_label] == 1)\n",
        "  binary_labels[ones_indices] = 1\n",
        "  return binary_labels\n",
        "\n",
        "train_binary_labels = convert_multilabels_to_binary(train_labels, most_frequent_label)\n",
        "test_binary_labels = convert_multilabels_to_binary(test_labels, most_frequent_label)\n",
        "\n",
        "for i in range(10):\n",
        "  print('Original Multi Target Label: {}\\tvs Binary Converted Label: {}'.format(train_labels[i], train_binary_labels[i]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EjOiw6fD9BsQ",
        "outputId": "ca3be382-3227-4115-dc9f-cd6a35520a0a"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original Multi Target Label: [1 0 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]\tvs Binary Converted Label: 1\n",
            "Original Multi Target Label: [0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0]\tvs Binary Converted Label: 1\n",
            "Original Multi Target Label: [1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]\tvs Binary Converted Label: 0\n",
            "Original Multi Target Label: [1 0 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 0]\tvs Binary Converted Label: 0\n",
            "Original Multi Target Label: [1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\tvs Binary Converted Label: 0\n",
            "Original Multi Target Label: [0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0]\tvs Binary Converted Label: 1\n",
            "Original Multi Target Label: [1 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0]\tvs Binary Converted Label: 0\n",
            "Original Multi Target Label: [0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]\tvs Binary Converted Label: 0\n",
            "Original Multi Target Label: [0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0]\tvs Binary Converted Label: 1\n",
            "Original Multi Target Label: [0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0]\tvs Binary Converted Label: 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Train and evaluate binary classification"
      ],
      "metadata": {
        "id": "lJUTwa22S-EF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import zero_one_loss\n",
        "\n",
        "classifiers = {\n",
        "    'Gaussian Naive Bayes Classifier': GaussianNB(),\n",
        "    'Logistic Regression Classifier': LogisticRegression(random_state=RANDOM_STATE),\n",
        "    'Support Vector Machine Classifier': LinearSVC(random_state=RANDOM_STATE),\n",
        "    'K-nearest Neighbors Classifier': KNeighborsClassifier(n_neighbors=5),\n",
        "    'Random Forest Classifier': RandomForestClassifier(n_estimators=100, random_state=RANDOM_STATE)\n",
        "}\n",
        "\n",
        "for classifier_name, clf in classifiers.items():\n",
        "  clf.fit(train_inputs.toarray(), train_binary_labels)\n",
        "  pred_labels = clf.predict(test_inputs.toarray())\n",
        "  print('\\nEvaluating {}'.format(classifier_name))\n",
        "  print(classification_report(test_binary_labels,pred_labels,zero_division='warn'))\n",
        "  \n",
        "  print('Subset accuracy = {}'.format((1-zero_one_loss(test_binary_labels, pred_labels))))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vLT2qM_5Sg30",
        "outputId": "14f23842-18e1-4f84-ff6e-84eec664c55d"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Evaluating Gaussian Naive Bayes Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.69      0.40      0.51      2425\n",
            "           1       0.44      0.71      0.54      1558\n",
            "\n",
            "    accuracy                           0.53      3983\n",
            "   macro avg       0.56      0.56      0.52      3983\n",
            "weighted avg       0.59      0.53      0.52      3983\n",
            "\n",
            "Subset accuracy = 0.5254833040421792\n",
            "\n",
            "Evaluating Logistic Regression Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.69      0.87      0.77      2425\n",
            "           1       0.66      0.40      0.49      1558\n",
            "\n",
            "    accuracy                           0.68      3983\n",
            "   macro avg       0.67      0.63      0.63      3983\n",
            "weighted avg       0.68      0.68      0.66      3983\n",
            "\n",
            "Subset accuracy = 0.6834044689932212\n",
            "\n",
            "Evaluating Support Vector Machine Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.70      0.76      0.73      2425\n",
            "           1       0.57      0.49      0.52      1558\n",
            "\n",
            "    accuracy                           0.65      3983\n",
            "   macro avg       0.63      0.62      0.63      3983\n",
            "weighted avg       0.65      0.65      0.65      3983\n",
            "\n",
            "Subset accuracy = 0.653276424805423\n",
            "\n",
            "Evaluating K-nearest Neighbors Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.67      0.69      0.68      2425\n",
            "           1       0.50      0.47      0.49      1558\n",
            "\n",
            "    accuracy                           0.61      3983\n",
            "   macro avg       0.59      0.58      0.58      3983\n",
            "weighted avg       0.60      0.61      0.61      3983\n",
            "\n",
            "Subset accuracy = 0.6080843585237259\n",
            "\n",
            "Evaluating Random Forest Classifier\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.66      0.91      0.76      2425\n",
            "           1       0.65      0.26      0.37      1558\n",
            "\n",
            "    accuracy                           0.66      3983\n",
            "   macro avg       0.65      0.58      0.57      3983\n",
            "weighted avg       0.65      0.66      0.61      3983\n",
            "\n",
            "Subset accuracy = 0.655536028119508\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Multi-Label-Learning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}