Short_Newswire_Multiclassification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Short Newswire Multiclassification.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOZW9pg3TU1U3STwQZo9lxx",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrtzcardo/Short-Newswire-Multiclassification/blob/main/Short_Newswire_Multiclassification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9M50ToJ_kLU3"
      },
      "source": [
        "Attempting to classify short newswires and their topics (published by Reuters in 1986) into the category of topics they are. 46 different topics, each topic has at least 10 examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1OmWnh_ndAGg"
      },
      "source": [
        "from keras.datasets import reuters\n",
        "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(\n",
        "num_words=10000) #restricting the data to only the 10,000 most frequently occurring words found in the data.\n",
        "\n",
        "print(train_labels)\n",
        "#print(len(train_data))\n",
        "#print(len(test_data))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9HR7jblJdOz9"
      },
      "source": [
        "'''Looking at what the data looks like a bit'''\n",
        "\n",
        "print(train_data[45])\n",
        "print(train_labels[45])\n",
        "word_index = reuters.get_word_index()\n",
        "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
        "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[45]])\n",
        "print(decoded_review)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qlQd22yQd108"
      },
      "source": [
        "'''Vectorize data'''\n",
        "import numpy as np\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "  results = np.zeros((len(sequences), dimension)) #matrix len(sequences) x dimension of all zeros\n",
        "  for i, sequences in enumerate(sequences):\n",
        "    results[i, sequences] = 1     \n",
        "  return results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mZshyUzBeyCX"
      },
      "source": [
        "x_train = vectorize_sequences(train_data)\n",
        "x_test = vectorize_sequences(test_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rpYBfKcrqb4y"
      },
      "source": [
        "To vectorize the labels, there are two possibilities: you can cast the label list as an integer\n",
        "tensor, or you can use one-hot encoding. One-hot encoding is a widely used format\n",
        "for categorical data, also called categorical encoding."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_fyHFEHKpUDZ"
      },
      "source": [
        "'''In this case, one-hot encoding of the labels consists of embedding \n",
        "each label as an all-zero vector with a 1 in the place ofthe label index.\n",
        "It looks like this, but Keras already built in.\n",
        "\n",
        "def to_one_hot(labels, dimension=46):\n",
        "  results = np.zeros((len(labels), dimension))\n",
        "  for i, label in enumerate(labels):\n",
        "    results[i, label] = 1\n",
        "  return results\n",
        "\n",
        "one_hot_train_labels = to_one_hot(train_labels)\n",
        "one_hot_test_labels = to_one_hot(test_labels)\n",
        "'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ssCAQ5LVrHn_"
      },
      "source": [
        "from keras.utils.np_utils import to_categorical\n",
        "\n",
        "one_hot_train_labels = to_categorical(train_labels)\n",
        "one_hot_test_labels = to_categorical(test_labels)\n",
        "#print(one_hot_test_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vTLrHoKxrNc_"
      },
      "source": [
        "from keras import models\n",
        "from keras import layers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(46, activation='softmax')) #probability distribution over the 46 different output classes, sums to 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m5-VnLPesq_6"
      },
      "source": [
        "model.compile(optimizer='rmsprop',              #rmsprop kinda just works all the time\n",
        "              loss='categorical_crossentropy',  #best for categorical models\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_tltIrR5tP8-"
      },
      "source": [
        "'''Setting aside a validation set'''\n",
        "\n",
        "x_val = x_train[:1000]\n",
        "partial_x_train = x_train[1000:]\n",
        "\n",
        "y_val = one_hot_train_labels[:1000]\n",
        "partial_y_train = one_hot_train_labels[1000:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CLlRTWlytlki"
      },
      "source": [
        "'''Training model'''\n",
        "\n",
        "history = model.fit(partial_x_train,\n",
        "                    partial_y_train,\n",
        "                    epochs=20,\n",
        "                    batch_size=512,\n",
        "                    validation_data=(x_val, y_val))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jv32oQLvuAdb"
      },
      "source": [
        "'''Plotting the training and validation loss'''\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(1, len(loss) + 1)\n",
        "\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eDngAOx5vuPg"
      },
      "source": [
        "'''Plotting the training and validation accuracy'''\n",
        "\n",
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "plt.plot(epochs, acc, 'bo', label='Training accuracy')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fcDSw0gQwIBR"
      },
      "source": [
        "'''Model begins to over fit at 9 epochs ish so that's where I will be fixing it to'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VuGeUPh9wZrC"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(46, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(partial_x_train,\n",
        "          partial_y_train,\n",
        "          epochs=9,\n",
        "          batch_size=512,\n",
        "          validation_data=(x_val, y_val))\n",
        "\n",
        "results = model.evaluate(x_test, one_hot_test_labels)\n",
        "\n",
        "print(results)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "32NGf9XmwxQq"
      },
      "source": [
        "'''Generating predictions for new data'''\n",
        "\n",
        "predictions = model.predict(x_test)\n",
        "\n",
        "print(predictions[0].shape)     #46 len vector, check\n",
        "print(np.sum(predictions[0]))   #prob sums to 1, check\n",
        "\n",
        "np.argmax(predictions[0])      #looks like label[3] / 4 is the class with 71% prob\n",
        "\n",
        "#for i in predictions[0]:\n",
        "#  print(i)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J3KqY9Q8xl0A"
      },
      "source": [
        "'''A model with an information bottleneck as in one layer has less than 46 nodes'''\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "#model.add(layers.Dense(128, activation='relu'))\n",
        "model.add(layers.Dense(46, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(partial_x_train,\n",
        "          partial_y_train,\n",
        "          epochs=20,\n",
        "          batch_size=128,\n",
        "          validation_data=(x_val, y_val))\n",
        "\n",
        "results = model.evaluate(x_test, one_hot_test_labels)\n",
        "\n",
        "print(results)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-yOEkpllzge_"
      },
      "source": [
        "val_accuracy: 0.6710 Epoch 8/20 with 4 nodes in 2nd layer\n",
        "\n",
        "With 32 nodes, bounced between val_acc of 79 and 80 as of 3rd epoch\n",
        "\n",
        "With 128 nodes, val_acc of stayed around 80 as of epoch 6\n",
        "\n",
        "Similar results taking out the middle layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X-B4qvQhz79W"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}