Spark_NLP_Udemy_MOOC/Healthcare_NLP/GenericSVMClassifierApproach.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ],
      "metadata": {
        "id": "8V_Ow3cAVEYe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **GenericSVMClassifierApproach**"
      ],
      "metadata": {
        "id": "Y0fJRpNJBslT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook will cover the different parameters and usages of `GenericSVMClassifierApproach`.\n",
        "\n",
        "\n",
        "**📖 Learning Objectives:**\n",
        "1. Understand how trains a TensorFlow model for `SVMClassifier` of feature vectors.\n",
        "\n",
        "2. Become comfortable using the different parameters of the annotator.\n",
        "**🔗 Helpful Links:**\n",
        "\n",
        "- Documentation : [GenericSVMClassifierApproach](https://colab.research.google.com/drive/1SCD1jaX-8-GVMdupJ0xEblJjew798NeF#scrollTo=qeclGJmrVLjX)\n",
        "\n",
        "- Python Docs : [GenericSVMClassifierApproach](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/classification/generic_svm_classifier/index.html#module-contents)\n",
        "\n",
        "- For extended examples of usage, see [Spark NLP Workshop repository](https://colab.research.google.com/drive/12JQuR7cSUeYfCu418REcqkSmkYWq8Efb#scrollTo=Ltt3uq8y9MKR).\n"
      ],
      "metadata": {
        "id": "qeclGJmrVLjX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **📜 Background**"
      ],
      "metadata": {
        "id": "OV0hPSCWXslc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`GenericSVMClassifier` is a derivative of [GenericClassifier](https://colab.research.google.com/drive/1O96phialslGizxSt6a0F7KSy-RUvJtTS?usp=sharing) which implements [SVM (Support Vector Machine) classification](https://en.wikipedia.org/wiki/Support_vector_machine). The input to the model is FeatureVector and the output is category annotations with labels and corresponding confidence scores. The scores are standardized using the logistic function so that they vary between 0 and 1."
      ],
      "metadata": {
        "id": "_-hGkZ7_cAZE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🎬 Colab Setup**"
      ],
      "metadata": {
        "id": "E8qy2MI2XySv"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HuLFt0OdBkuo"
      },
      "outputs": [],
      "source": [
        "# Install the johnsnowlabs library to access Spark-OCR and Spark-NLP for Healthcare, Finance, and Legal.\n",
        "! pip install -q johnsnowlabs"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q tensorflow==2.12.0 tensorflow_addons"
      ],
      "metadata": {
        "id": "rf6_VRMVAqPQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ],
      "metadata": {
        "id": "HJGAUiIqBI94"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from johnsnowlabs import nlp, medical, visual\n",
        "\n",
        "# After uploading your license run this to install all licensed Python Wheels and pre-download Jars the Spark Session JVM\n",
        "nlp.install()"
      ],
      "metadata": {
        "id": "SBhRUP21BI7R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from johnsnowlabs import nlp, medical, visual\n",
        "import pandas as pd\n",
        "\n",
        "# Automatically load license data and start a session with all jars user has access to\n",
        "spark = nlp.start()"
      ],
      "metadata": {
        "id": "a_1HS4iCB5ie",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "697c5886-a9dd-45f9-f29d-5be87d7affcd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "👌 Detected license file /content/spark_nlp_for_healthcare_spark_ocr_8448_floating.json\n",
            "👌 Launched \u001b[92mcpu optimized\u001b[39m session with with: 🚀Spark-NLP==5.1.4, 💊Spark-Healthcare==5.1.3, running on ⚡ PySpark==3.1.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "spark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 219
        },
        "id": "aok3umndL-H-",
        "outputId": "dd977c81-41c8-4d96-fd10-d06b43a5df5f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7951cd2fcaf0>"
            ],
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://6861f09b2f72:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.1.2</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>John-Snow-Labs-Spark-Session 🚀 with Jars for: 🚀Spark-NLP==5.1.4, 💊Spark-Healthcare==5.1.3, running on ⚡ PySpark==3.1.2</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🖨️ Input/Output Annotation Types**\n",
        "\n"
      ],
      "metadata": {
        "id": "f_4conOIYj6D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Input: `FEATURE_VECTOR`\n",
        "\n",
        "- Output: `CATEGORY`"
      ],
      "metadata": {
        "id": "0Fc_3iRwYk8N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🔎 Parameters**"
      ],
      "metadata": {
        "id": "IO7xgW6Znobb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- `batchSize`: (int) Batch size\n",
        "\n",
        "- `dropout`: (float) Dropout coefficient\n",
        "\n",
        "- `epochsNumber`: (int) Maximum number of epochs to train\n",
        "\n",
        "- `featureScaling`: (str) Feature scaling method. Possible values are 'zscore', 'minmax' or empty (no scaling)\n",
        "\n",
        "- `fixImbalance`: (boolean) Fix the imbalance in the training set by replicating examples of under represented categories\n",
        "\n",
        "- `labelColumn`: (str) Column with label per each document\n",
        "\n",
        "- `learningRate`: (float) Learning Rate\n",
        "\n",
        "- `modelFile`: (str) Location of file of the model used for classification\n",
        "\n",
        "- `multiClass`: (boolean) If multiClass is set, the model will return all the labels with corresponding scores. By default, multiClass is false.\n",
        "\n",
        "- `outputLogsPath`: (str) Folder path to save training logs. If no path is specified, the logs won't be stored in disk. The path can be a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).\n",
        "\n",
        "- `validationSplit`: (float) The proportion of training dataset to be used as validation set.The model will be validated against this dataset on each Epoch and will not be used for training. The value should be between 0.0 and 1.0."
      ],
      "metadata": {
        "id": "2Vv-dsM2njQT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ✍  Explaining GenericSVMClassifierApproach with Example"
      ],
      "metadata": {
        "id": "g9zdtP-D5spx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 📑 Sample Data"
      ],
      "metadata": {
        "id": "txGhMorxR0s_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#downloading sample datasets\n",
        "!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/ADE_Corpus_V2/ADE-NEG.txt\n",
        "!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/ADE_Corpus_V2/DRUG-AE.rel"
      ],
      "metadata": {
        "id": "FNgWV3Fzjyop"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_neg= pd.read_csv(\"ADE-NEG.txt\", header=None, delimiter=\"\\t\", names=[\"col1\"])\n",
        "df_neg.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "kVetRK19C-sy",
        "outputId": "72cbf30f-1a54-4ab3-9b79-4dbb1e45984e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                col1\n",
              "0  6460590 NEG Clioquinol intoxication occurring ...\n",
              "1  8600337 NEG \"Retinoic acid syndrome\" was preve...\n",
              "2  8402502 NEG BACKGROUND: External beam radiatio...\n",
              "3  8700794 NEG Although the enuresis ceased, she ...\n",
              "4  17662448 NEG A 42-year-old woman had uneventfu..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e82c49b0-6d8c-4823-8f4b-5fe1edfb6084\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6460590 NEG Clioquinol intoxication occurring ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8600337 NEG \"Retinoic acid syndrome\" was preve...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8402502 NEG BACKGROUND: External beam radiatio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8700794 NEG Although the enuresis ceased, she ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>17662448 NEG A 42-year-old woman had uneventfu...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e82c49b0-6d8c-4823-8f4b-5fe1edfb6084')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e82c49b0-6d8c-4823-8f4b-5fe1edfb6084 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e82c49b0-6d8c-4823-8f4b-5fe1edfb6084');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-557df707-b0cd-4e3a-b8bb-e4548d00835a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-557df707-b0cd-4e3a-b8bb-e4548d00835a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-557df707-b0cd-4e3a-b8bb-e4548d00835a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_neg['text'] =  df_neg.col1.str.split('NEG').str[1]\n",
        "df_neg[\"category\"] = \"neg\"\n",
        "df_neg= df_neg[[\"text\", \"category\"]]\n",
        "df_neg.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "h2ArZzdEC-wL",
        "outputId": "493cf300-b8db-48eb-8e78-d382f28d4dec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category\n",
              "0   Clioquinol intoxication occurring in the trea...      neg\n",
              "1   \"Retinoic acid syndrome\" was prevented with s...      neg\n",
              "2   BACKGROUND: External beam radiation therapy o...      neg\n",
              "3   Although the enuresis ceased, she developed t...      neg\n",
              "4   A 42-year-old woman had uneventful bilateral ...      neg"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6396a84c-cd6b-45be-be1d-15d98de2b0f1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Clioquinol intoxication occurring in the trea...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>\"Retinoic acid syndrome\" was prevented with s...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>BACKGROUND: External beam radiation therapy o...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Although the enuresis ceased, she developed t...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A 42-year-old woman had uneventful bilateral ...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6396a84c-cd6b-45be-be1d-15d98de2b0f1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6396a84c-cd6b-45be-be1d-15d98de2b0f1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6396a84c-cd6b-45be-be1d-15d98de2b0f1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b53421fe-a4c4-4ecb-a489-f25b0e07cc0c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b53421fe-a4c4-4ecb-a489-f25b0e07cc0c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b53421fe-a4c4-4ecb-a489-f25b0e07cc0c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_pos= pd.read_csv(\"DRUG-AE.rel\", header=None, delimiter=\"|\")\n",
        "df_pos.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "Yh_2D5czC-y_",
        "outputId": "46b9184b-f707-4e37-ca4a-a4bf4740a4d9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "          0                                                  1  \\\n",
              "0  10030778      Intravenous azithromycin-induced ototoxicity.   \n",
              "1  10048291  Immobilization, while Paget's bone disease was...   \n",
              "2  10048291  Unaccountable severe hypercalcemia in a patien...   \n",
              "3  10082597  METHODS: We report two cases of pseudoporphyri...   \n",
              "4  10082597  METHODS: We report two cases of pseudoporphyri...   \n",
              "\n",
              "                           2    3    4                   5    6    7  \n",
              "0                ototoxicity   43   54        azithromycin   22   34  \n",
              "1  increased calcium-release  960  985  dihydrotachysterol  908  926  \n",
              "2              hypercalcemia   31   44  dihydrotachysterol   94  112  \n",
              "3            pseudoporphyria  620  635            naproxen  646  654  \n",
              "4            pseudoporphyria  620  635           oxaprozin  659  668  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b824842b-119e-4e01-a25d-3ccdd65cc4f6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>10030778</td>\n",
              "      <td>Intravenous azithromycin-induced ototoxicity.</td>\n",
              "      <td>ototoxicity</td>\n",
              "      <td>43</td>\n",
              "      <td>54</td>\n",
              "      <td>azithromycin</td>\n",
              "      <td>22</td>\n",
              "      <td>34</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>10048291</td>\n",
              "      <td>Immobilization, while Paget's bone disease was...</td>\n",
              "      <td>increased calcium-release</td>\n",
              "      <td>960</td>\n",
              "      <td>985</td>\n",
              "      <td>dihydrotachysterol</td>\n",
              "      <td>908</td>\n",
              "      <td>926</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10048291</td>\n",
              "      <td>Unaccountable severe hypercalcemia in a patien...</td>\n",
              "      <td>hypercalcemia</td>\n",
              "      <td>31</td>\n",
              "      <td>44</td>\n",
              "      <td>dihydrotachysterol</td>\n",
              "      <td>94</td>\n",
              "      <td>112</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>10082597</td>\n",
              "      <td>METHODS: We report two cases of pseudoporphyri...</td>\n",
              "      <td>pseudoporphyria</td>\n",
              "      <td>620</td>\n",
              "      <td>635</td>\n",
              "      <td>naproxen</td>\n",
              "      <td>646</td>\n",
              "      <td>654</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>10082597</td>\n",
              "      <td>METHODS: We report two cases of pseudoporphyri...</td>\n",
              "      <td>pseudoporphyria</td>\n",
              "      <td>620</td>\n",
              "      <td>635</td>\n",
              "      <td>oxaprozin</td>\n",
              "      <td>659</td>\n",
              "      <td>668</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b824842b-119e-4e01-a25d-3ccdd65cc4f6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b824842b-119e-4e01-a25d-3ccdd65cc4f6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b824842b-119e-4e01-a25d-3ccdd65cc4f6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-fae4cfcc-5233-4612-b5b1-cc0d86dca4a0\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fae4cfcc-5233-4612-b5b1-cc0d86dca4a0')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-fae4cfcc-5233-4612-b5b1-cc0d86dca4a0 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_pos[\"category\"]= \"pos\"\n",
        "df_pos.rename(columns={1: \"text\"}, inplace=True)\n",
        "df_pos= df_pos[[\"text\", \"category\"]]\n",
        "df_pos.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "99ftTAgoDNuw",
        "outputId": "cfcae3aa-d974-4d02-ddd5-492bb71e5d5d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category\n",
              "0      Intravenous azithromycin-induced ototoxicity.      pos\n",
              "1  Immobilization, while Paget's bone disease was...      pos\n",
              "2  Unaccountable severe hypercalcemia in a patien...      pos\n",
              "3  METHODS: We report two cases of pseudoporphyri...      pos\n",
              "4  METHODS: We report two cases of pseudoporphyri...      pos"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1d0de81c-dd63-4c45-9ce4-35bfddf1a0e9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Intravenous azithromycin-induced ototoxicity.</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Immobilization, while Paget's bone disease was...</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Unaccountable severe hypercalcemia in a patien...</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>METHODS: We report two cases of pseudoporphyri...</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>METHODS: We report two cases of pseudoporphyri...</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1d0de81c-dd63-4c45-9ce4-35bfddf1a0e9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1d0de81c-dd63-4c45-9ce4-35bfddf1a0e9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1d0de81c-dd63-4c45-9ce4-35bfddf1a0e9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8393f376-fe5b-4646-b3f1-1ff65b99f4d1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8393f376-fe5b-4646-b3f1-1ff65b99f4d1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8393f376-fe5b-4646-b3f1-1ff65b99f4d1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ade_df= pd.concat([df_neg, df_pos])\n",
        "ade_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "VPKamnI8DPGL",
        "outputId": "5e161984-99d0-4570-99c9-278e5b77da61"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category\n",
              "0   Clioquinol intoxication occurring in the trea...      neg\n",
              "1   \"Retinoic acid syndrome\" was prevented with s...      neg\n",
              "2   BACKGROUND: External beam radiation therapy o...      neg\n",
              "3   Although the enuresis ceased, she developed t...      neg\n",
              "4   A 42-year-old woman had uneventful bilateral ...      neg"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b22f1fbe-020f-4243-bb56-6d6374bcaff0\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Clioquinol intoxication occurring in the trea...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>\"Retinoic acid syndrome\" was prevented with s...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>BACKGROUND: External beam radiation therapy o...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Although the enuresis ceased, she developed t...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A 42-year-old woman had uneventful bilateral ...</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b22f1fbe-020f-4243-bb56-6d6374bcaff0')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b22f1fbe-020f-4243-bb56-6d6374bcaff0 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b22f1fbe-020f-4243-bb56-6d6374bcaff0');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2b9cb6d6-9ccf-42dd-9787-1e93c6c7cc3e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2b9cb6d6-9ccf-42dd-9787-1e93c6c7cc3e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2b9cb6d6-9ccf-42dd-9787-1e93c6c7cc3e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ade_df[\"category\"].value_counts()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZBpmg-2bDPJX",
        "outputId": "649c11a3-1a2a-4a34-bfa4-e1628615b905"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "neg    16695\n",
              "pos     6821\n",
              "Name: category, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#We will take 50% of data to make a fast training\n",
        "\n",
        "spark_df = spark.createDataFrame(ade_df).sample(0.5, 42) # limit the data\n",
        "\n",
        "trainingData, testData = spark_df.randomSplit([0.8, 0.2], seed = 100)\n",
        "\n",
        "print(\"Training Dataset Count: \" + str(trainingData.count()))\n",
        "print(\"Test Dataset Count: \" + str(testData.count()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QgAIdw18DPMl",
        "outputId": "8a8040e1-9bf2-4801-d0a5-b024adc49bcf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training Dataset Count: 9423\n",
            "Test Dataset Count: 2399\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "spark_df.printSchema()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZuY8Zi8SDWeI",
        "outputId": "573895ef-8cb0-450d-c8f1-00ac6a5f9042"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- text: string (nullable = true)\n",
            " |-- category: string (nullable = true)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "spark_df.head(3)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1rO7ZPdADWg-",
        "outputId": "e101fd54-dc42-4e4b-83cd-25f2f2dec1dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Row(text=' Although the enuresis ceased, she developed throbbing headaches, nausea, vomiting, paresthesia, lethargy, fatigue, and altered mental status over the next 7 days.', category='neg'),\n",
              " Row(text=' Differences in interpretation by patients and healthcare professionals may cause the observed disparities in seriousness and outcome of reported ADRs.', category='neg'),\n",
              " Row(text=' The decedent was also prescribed tramadol, gabapentin, cetirizine, modafinil, carisoprodol, and Xyrem.', category='neg')]"
            ]
          },
          "metadata": {},
          "execution_count": 182
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Preparing input: **FEATURE_VECTOR** for SVM Classifier"
      ],
      "metadata": {
        "id": "CuVEa5bGStHc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***We will use 100d embeddings as input, so text will be vectorized after pipeline below***"
      ],
      "metadata": {
        "id": "XjXmXi--S9wV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "document_assembler = nlp.DocumentAssembler()\\\n",
        "        .setInputCol(\"text\")\\\n",
        "        .setOutputCol(\"document\")\n",
        "\n",
        "tokenizer = nlp.Tokenizer() \\\n",
        "            .setInputCols([\"document\"]) \\\n",
        "            .setOutputCol(\"token\")\n",
        "\n",
        "word_embeddings = nlp.WordEmbeddingsModel.pretrained(\"embeddings_healthcare_100d\",\"en\",\"clinical/models\")\\\n",
        "        .setInputCols([\"document\",\"token\"])\\\n",
        "        .setOutputCol(\"word_embeddings\")\n",
        "\n",
        "sentence_embeddings = nlp.SentenceEmbeddings() \\\n",
        "        .setInputCols([\"document\", \"word_embeddings\"]) \\\n",
        "        .setOutputCol(\"sentence_embeddings\") \\\n",
        "        .setPoolingStrategy(\"AVERAGE\")\n",
        "\n",
        "embeddings_pipeline = nlp.Pipeline(\n",
        "    stages = [\n",
        "        document_assembler,\n",
        "        tokenizer,\n",
        "        word_embeddings,\n",
        "        sentence_embeddings,\n",
        "\n",
        "    ])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EFVK8wWODWjX",
        "outputId": "a4e0cf1e-f380-48d2-f834-286a7910af82"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "embeddings_healthcare_100d download started this may take some time.\n",
            "Approximate size to download 475.8 MB\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trainingData_with_embeddings = embeddings_pipeline.fit(trainingData).transform(trainingData)\n",
        "trainingData_with_embeddings.show(2,truncate=60)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1rEIkIiuVLFS",
        "outputId": "49a3ed33-54e7-40db-83de-50c595c0d041"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------------------------------------------------------+--------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+\n",
            "|                                                        text|category|                                                    document|                                                       token|                                             word_embeddings|                                         sentence_embeddings|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+\n",
            "| A 15-year-old boy had temporary hypertropia, supraductio...|     neg|[{document, 0, 262,  A 15-year-old boy had temporary hype...|[{token, 1, 1, A, {sentence -> 0}, []}, {token, 3, 13, 15...|[{word_embeddings, 1, 1, A, {isOOV -> false, pieceId -> -...|[{sentence_embeddings, 0, 262,  A 15-year-old boy had tem...|\n",
            "| A 17-year-old anuric female patient with end-stage renal...|     neg|[{document, 0, 185,  A 17-year-old anuric female patient ...|[{token, 1, 1, A, {sentence -> 0}, []}, {token, 3, 13, 17...|[{word_embeddings, 1, 1, A, {isOOV -> false, pieceId -> -...|[{sentence_embeddings, 0, 185,  A 17-year-old anuric fema...|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+\n",
            "only showing top 2 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# we will take only sentence embedings per sentence\n",
        "\n",
        "trainingData_with_embeddings = embeddings_pipeline.fit(trainingData).transform(trainingData)\n",
        "trainingData_with_embeddings = trainingData_with_embeddings.select(\"text\",\"category\",\"sentence_embeddings\")\n",
        "trainingData_with_embeddings.show(2,truncate=60)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "or0eCOT-DtsD",
        "outputId": "244940bd-4049-4177-b8f2-ecc5ebe9c9b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "|                                                        text|category|                                         sentence_embeddings|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "| A 15-year-old boy had temporary hypertropia, supraductio...|     neg|[{sentence_embeddings, 0, 262,  A 15-year-old boy had tem...|\n",
            "| A 17-year-old anuric female patient with end-stage renal...|     neg|[{sentence_embeddings, 0, 185,  A 17-year-old anuric fema...|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "only showing top 2 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "testData_with_embeddings = embeddings_pipeline.fit(testData).transform(testData)\n",
        "testData_with_embeddings = testData_with_embeddings.select(\"text\",\"category\",\"sentence_embeddings\")\n",
        "testData_with_embeddings.show(2,truncate=60)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GYtk-buYDtvQ",
        "outputId": "b07a787d-9829-4b6c-9038-f579ea34d32f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "|                                                        text|category|                                         sentence_embeddings|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "| A 29-year-old woman who underwent laser in situ keratomi...|     neg|[{sentence_embeddings, 0, 216,  A 29-year-old woman who u...|\n",
            "| A 52-year-old woman was seen in our outpatient clinic fo...|     neg|[{sentence_embeddings, 0, 198,  A 52-year-old woman was s...|\n",
            "+------------------------------------------------------------+--------+------------------------------------------------------------+\n",
            "only showing top 2 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "testData_with_embeddings.printSchema()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hcyHKzMuDtyB",
        "outputId": "64df4ce9-5ab9-454b-c349-49959194c0c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- text: string (nullable = true)\n",
            " |-- category: string (nullable = true)\n",
            " |-- sentence_embeddings: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "testData_with_embeddings.select(testData_with_embeddings.sentence_embeddings.embeddings).show(3,truncate=120)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DgM2nkSQDt1_",
        "outputId": "8d9048e9-853a-4ecf-b786-fb02e5a20c50"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------------------------------------------------------------------------------------------------------------------+\n",
            "|                                                                                          sentence_embeddings.embeddings|\n",
            "+------------------------------------------------------------------------------------------------------------------------+\n",
            "|[[-0.09864674, 0.021150807, 0.1699883, 0.113804236, -0.05979607, -0.1667684, 0.09705255, 0.08950208, 0.011517063, -0....|\n",
            "|[[0.05888334, 0.1264814, 0.07519952, 0.17283632, -0.20441434, -0.07624834, 0.20318455, -0.026616767, -0.076673746, 0....|\n",
            "|[[0.057804022, 0.047698125, 0.06304918, 0.15662706, -0.04416214, -0.06267688, 0.08489624, -0.0543084, -0.0027059284, ...|\n",
            "+------------------------------------------------------------------------------------------------------------------------+\n",
            "only showing top 3 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **💻Pipeline**"
      ],
      "metadata": {
        "id": "pagXc73HWidS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "log_folder=\"logs\"\n",
        "!mkdir -p $log_folder"
      ],
      "metadata": {
        "id": "_Z3Fme1GD485"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **✅ Graph Creation**"
      ],
      "metadata": {
        "id": "p71mxwc1NXQ5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "graph_folder = \"graph_folder\"\n",
        "\n",
        "gc_svm_graph_builder = medical.TFGraphBuilder()\\\n",
        "    .setModelName(\"svm_classifier\")\\\n",
        "    .setInputCols([\"feature_vector\"]) \\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setGraphFolder(graph_folder)\\\n",
        "    .setGraphFile(\"svm_graph.pb\")"
      ],
      "metadata": {
        "id": "2k2UEqgbD5AU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ⏰**Training with parameters**"
      ],
      "metadata": {
        "id": "TMkd1ewsTfGp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```{Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='lazyAnnotator', doc='Whether this AnnotatorModel acts as lazy in RecursivePipelines'): False,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='labelColumn', doc='Column with one label per document'): 'label',\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='batchSize', doc='Size for each batch in the optimization process'): 32,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='epochsN', doc='Number of epochs for the optimization process'): 10,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='learningRate', doc='Learning rate for the optimization process'): 0.001,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='dropout', doc='Dropout at the output of each layer'): 0.05,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='fixImbalance', doc='A flag indicating whenther to balance the trainig set'): False,\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='featureScaling', doc=\"Feature scaling method. Possible values are 'zscore', 'minmax' or empty (no scaling)\"): '',\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='outputLogsPath', doc='Path to folder where logs will be saved. If no path is specified, no logs are generated'): '',\n",
        " Param(parent='GenericSVMClassifierApproach_ad6f332b1585', name='multiClass', doc='Whether to return only the label with the highest confidence score or all labels'): False}\n",
        "```"
      ],
      "metadata": {
        "id": "7qPfvei2TbQz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will train a new model with GenericSVMClassifierApproach by assesing each parameters.\n",
        "Parameters are in two categories:\n",
        "`labelColumn` , `modelFile`, `outputLogPaths` dont have any effect on model performance or quality. So we will not make any tuning on them.\n"
      ],
      "metadata": {
        "id": "18VoOQkaE4Us"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `epochsNumber`:"
      ],
      "metadata": {
        "id": "4_cTGHOcFxjL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        " How many times will our algorithm see the entire dataset and update its weights."
      ],
      "metadata": {
        "id": "chTG_M15JSgA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "features_asm = medical.FeaturesAssembler()\\\n",
        "    .setInputCols([\"sentence_embeddings\"])\\\n",
        "    .setOutputCol(\"feature_vector\")\n",
        "\n",
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(2)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.015)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    # .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n"
      ],
      "metadata": {
        "id": "mq5UyIE2D5C6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G_TvRydDD5FX",
        "outputId": "44f93f66-9d71-4ad4-c43a-78e100b50834"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! ls -t $log_folder | head -n 1\n",
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R96u2O46P0oj",
        "outputId": "4fb94d6f-91d1-483a-de85-f4a663e13e54"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GenericSVMClassifierApproach_051ce6e2e854.log\n",
            "Training 2 epochs\n",
            "Epoch 1/2\t0.13s\tLoss: 44.346165\tACC: 0.7142985\n",
            "Epoch 2/2\t0.07s\tLoss: 40.961517\tACC: 0.735992\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As seen above, 2 epoch takes .20 s and reached accuracy .735"
      ],
      "metadata": {
        "id": "7MCWedXUIOrQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.015)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    # .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])"
      ],
      "metadata": {
        "id": "A_QvWa6CIh_g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1KBc8glLIuny",
        "outputId": "4e3b202b-35e4-4a40-bd24-6838b5edf8ed"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n",
            "CPU times: user 488 ms, sys: 19.2 ms, total: 507 ms\n",
            "Wall time: 5.69 s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PipelineModel_30e7843d4669"
            ]
          },
          "metadata": {},
          "execution_count": 195
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "48b02rjAIyGq",
        "outputId": "f6ba533d-bb08-4897-8e56-b925255b4fa1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training 30 epochs\n",
            "Epoch 1/30\t0.13s\tLoss: 45.24362\tACC: 0.70614386\n",
            "Epoch 2/30\t0.07s\tLoss: 41.071648\tACC: 0.7361016\n",
            "Epoch 3/30\t0.07s\tLoss: 40.13074\tACC: 0.7475544\n",
            "Epoch 4/30\t0.08s\tLoss: 39.82691\tACC: 0.7580972\n",
            "Epoch 5/30\t0.07s\tLoss: 39.91341\tACC: 0.754668\n",
            "Epoch 6/30\t0.07s\tLoss: 39.645798\tACC: 0.75789666\n",
            "Epoch 7/30\t0.07s\tLoss: 39.610588\tACC: 0.7598117\n",
            "Epoch 8/30\t0.07s\tLoss: 39.639378\tACC: 0.7638383\n",
            "Epoch 9/30\t0.07s\tLoss: 39.57259\tACC: 0.7632556\n",
            "Epoch 10/30\t0.07s\tLoss: 39.657555\tACC: 0.7652909\n",
            "Epoch 11/30\t0.07s\tLoss: 39.3809\tACC: 0.7644717\n",
            "Epoch 12/30\t0.07s\tLoss: 39.323936\tACC: 0.7602233\n",
            "Epoch 13/30\t0.07s\tLoss: 39.669544\tACC: 0.7620181\n",
            "Epoch 14/30\t0.07s\tLoss: 39.400383\tACC: 0.76822156\n",
            "Epoch 15/30\t0.07s\tLoss: 39.99315\tACC: 0.76054007\n",
            "Epoch 16/30\t0.08s\tLoss: 39.142174\tACC: 0.7686038\n",
            "Epoch 17/30\t0.07s\tLoss: 40.122463\tACC: 0.7610025\n",
            "Epoch 18/30\t0.07s\tLoss: 39.373882\tACC: 0.7661208\n",
            "Epoch 19/30\t0.07s\tLoss: 39.80465\tACC: 0.7623495\n",
            "Epoch 20/30\t0.07s\tLoss: 39.46946\tACC: 0.76378345\n",
            "Epoch 21/30\t0.07s\tLoss: 39.46838\tACC: 0.7615811\n",
            "Epoch 22/30\t0.07s\tLoss: 39.451084\tACC: 0.76183236\n",
            "Epoch 23/30\t0.07s\tLoss: 39.52947\tACC: 0.7641042\n",
            "Epoch 24/30\t0.07s\tLoss: 39.743496\tACC: 0.7620328\n",
            "Epoch 25/30\t0.07s\tLoss: 39.464962\tACC: 0.76563704\n",
            "Epoch 26/30\t0.07s\tLoss: 39.500164\tACC: 0.7636565\n",
            "Epoch 27/30\t0.07s\tLoss: 39.608418\tACC: 0.7632596\n",
            "Epoch 28/30\t0.07s\tLoss: 39.406174\tACC: 0.76570255\n",
            "Epoch 29/30\t0.07s\tLoss: 39.36943\tACC: 0.7643554\n",
            "Epoch 30/30\t0.07s\tLoss: 39.71068\tACC: 0.7649595\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "30 epoch takes 5.69 s and reached accuracy  .764"
      ],
      "metadata": {
        "id": "AB7RiTnEI7PY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `batchsize`"
      ],
      "metadata": {
        "id": "jOgHBVeVMSrs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        " how many saples are proccessed in paralel during the learning, it depends on computational power and dataset itself, the smaller batchsize, the longer training time and the smaller memory required"
      ],
      "metadata": {
        "id": "M5MvdCmTPc7Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(4)\\\n",
        "    .setLearningRate(0.015)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    # .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "UJTRck3GMkeV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3xP6ObU8M_6P",
        "outputId": "a92f9ef0-7331-4c37-e2d2-6fb7b0c073f6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n",
            "CPU times: user 666 ms, sys: 41.9 ms, total: 707 ms\n",
            "Wall time: 37.9 s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PipelineModel_9909ef312182"
            ]
          },
          "metadata": {},
          "execution_count": 198
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b93nPwhFM-Wa",
        "outputId": "c8e9e700-65ae-4706-a9c1-88236db7ccdb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training 30 epochs\n",
            "Epoch 1/30\t1.20s\tLoss: 1369.9314\tACC: 0.744942\n",
            "Epoch 2/30\t1.08s\tLoss: 1373.536\tACC: 0.7481607\n",
            "Epoch 3/30\t1.13s\tLoss: 1353.2765\tACC: 0.75541174\n",
            "Epoch 4/30\t1.12s\tLoss: 1351.0005\tACC: 0.7580645\n",
            "Epoch 5/30\t1.09s\tLoss: 1355.1453\tACC: 0.7518746\n",
            "Epoch 6/30\t1.13s\tLoss: 1354.006\tACC: 0.75318336\n",
            "Epoch 7/30\t1.11s\tLoss: 1364.4591\tACC: 0.75180393\n",
            "Epoch 8/30\t1.10s\tLoss: 1359.5323\tACC: 0.75463355\n",
            "Epoch 9/30\t1.12s\tLoss: 1358.5867\tACC: 0.75180393\n",
            "Epoch 10/30\t1.14s\tLoss: 1359.6768\tACC: 0.7512379\n",
            "Epoch 11/30\t1.16s\tLoss: 1345.3046\tACC: 0.75028294\n",
            "Epoch 12/30\t1.15s\tLoss: 1346.5883\tACC: 0.75325406\n",
            "Epoch 13/30\t1.10s\tLoss: 1361.4282\tACC: 0.7518746\n",
            "Epoch 14/30\t1.17s\tLoss: 1345.9159\tACC: 0.7560838\n",
            "Epoch 15/30\t1.10s\tLoss: 1370.4236\tACC: 0.74759483\n",
            "Epoch 16/30\t1.10s\tLoss: 1337.047\tACC: 0.75491655\n",
            "Epoch 17/30\t1.11s\tLoss: 1356.4893\tACC: 0.75382006\n",
            "Epoch 18/30\t1.09s\tLoss: 1360.2317\tACC: 0.75328946\n",
            "Epoch 19/30\t1.12s\tLoss: 1351.7175\tACC: 0.7526528\n",
            "Epoch 20/30\t1.14s\tLoss: 1338.6552\tACC: 0.7578169\n",
            "Epoch 21/30\t1.13s\tLoss: 1344.7694\tACC: 0.7543506\n",
            "Epoch 22/30\t1.15s\tLoss: 1346.7002\tACC: 0.75378466\n",
            "Epoch 23/30\t1.17s\tLoss: 1345.7245\tACC: 0.7567912\n",
            "Epoch 24/30\t1.20s\tLoss: 1331.8018\tACC: 0.75898415\n",
            "Epoch 25/30\t1.20s\tLoss: 1347.3823\tACC: 0.7561899\n",
            "Epoch 26/30\t1.18s\tLoss: 1354.6108\tACC: 0.7516978\n",
            "Epoch 27/30\t1.13s\tLoss: 1355.3135\tACC: 0.74964625\n",
            "Epoch 28/30\t1.18s\tLoss: 1349.8802\tACC: 0.7545628\n",
            "Epoch 29/30\t1.21s\tLoss: 1334.7117\tACC: 0.75700337\n",
            "Epoch 30/30\t1.17s\tLoss: 1358.1816\tACC: 0.7464983\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   in last two run, we set batch size to 128 and 4.\n",
        "*   training time increased from 4.86 to 37.9 s with same epoch number\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "46e_-8jqN6nh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pred_df = model.transform(testData_with_embeddings)\n"
      ],
      "metadata": {
        "id": "gGWuClhlP0rA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pred_df.printSchema()\n",
        "pred_df.select(pred_df.prediction).show(5, truncate=False)\n",
        "pred_df.select(pred_df.category, pred_df.prediction.result).show(5, truncate=False)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TmHD7J5lP0wJ",
        "outputId": "af4f4619-7c6a-4da0-c433-3949259a7d3e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- text: string (nullable = true)\n",
            " |-- category: string (nullable = true)\n",
            " |-- sentence_embeddings: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            " |-- feature_vector: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            " |-- prediction: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            "\n",
            "+-------------------------------------------------------+\n",
            "|prediction                                             |\n",
            "+-------------------------------------------------------+\n",
            "|[{category, 0, 0, neg, {confidence -> 0.8875675}, []}] |\n",
            "|[{category, 0, 0, neg, {confidence -> 0.9180566}, []}] |\n",
            "|[{category, 0, 0, neg, {confidence -> 0.8544874}, []}] |\n",
            "|[{category, 0, 0, neg, {confidence -> 0.89287937}, []}]|\n",
            "|[{category, 0, 0, neg, {confidence -> 0.8806584}, []}] |\n",
            "+-------------------------------------------------------+\n",
            "only showing top 5 rows\n",
            "\n",
            "+--------+-----------------+\n",
            "|category|prediction.result|\n",
            "+--------+-----------------+\n",
            "|neg     |[neg]            |\n",
            "|neg     |[neg]            |\n",
            "|neg     |[neg]            |\n",
            "|neg     |[neg]            |\n",
            "|neg     |[neg]            |\n",
            "+--------+-----------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `learningrate`"
      ],
      "metadata": {
        "id": "uPnDVdSSOWiw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`The learning rate is a hyperparameter in deep learning and many other machine learning algorithms that determines the step size at which the model's weights are updated during training. It is one of the most critical hyperparameters to tune because it has a profound impact on the training process and the final performance of the model.`"
      ],
      "metadata": {
        "id": "Ot6LBSqGW3qS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    # .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "6KsyJzLjOWi3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3fb2da4a-60b0-406c-b4b7-c5d6dc210497",
        "id": "j0l1xZ9ROWi3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n",
            "CPU times: user 475 ms, sys: 40.7 ms, total: 516 ms\n",
            "Wall time: 5.82 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f794d254-ac3d-40fd-a16d-2e9d98cc9a01",
        "id": "7h5ZRpWbOWi3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training 30 epochs\n",
            "Epoch 1/30\t0.13s\tLoss: 63.266453\tACC: 0.6905082\n",
            "Epoch 2/30\t0.08s\tLoss: 45.79635\tACC: 0.71287525\n",
            "Epoch 3/30\t0.08s\tLoss: 44.067856\tACC: 0.7126935\n",
            "Epoch 4/30\t0.07s\tLoss: 43.221058\tACC: 0.7138254\n",
            "Epoch 5/30\t0.08s\tLoss: 42.866653\tACC: 0.71371984\n",
            "Epoch 6/30\t0.08s\tLoss: 42.479755\tACC: 0.7151431\n",
            "Epoch 7/30\t0.08s\tLoss: 42.327938\tACC: 0.7147756\n",
            "Epoch 8/30\t0.10s\tLoss: 42.053\tACC: 0.7160572\n",
            "Epoch 9/30\t0.10s\tLoss: 41.963844\tACC: 0.7157953\n",
            "Epoch 10/30\t0.09s\tLoss: 41.887173\tACC: 0.7163993\n",
            "Epoch 11/30\t0.08s\tLoss: 41.61777\tACC: 0.71952647\n",
            "Epoch 12/30\t0.07s\tLoss: 41.61414\tACC: 0.7205862\n",
            "Epoch 13/30\t0.08s\tLoss: 41.280735\tACC: 0.72134656\n",
            "Epoch 14/30\t0.08s\tLoss: 41.202465\tACC: 0.7227338\n",
            "Epoch 15/30\t0.08s\tLoss: 41.19994\tACC: 0.7234728\n",
            "Epoch 16/30\t0.08s\tLoss: 41.111885\tACC: 0.72556955\n",
            "Epoch 17/30\t0.08s\tLoss: 40.700535\tACC: 0.7287622\n",
            "Epoch 18/30\t0.08s\tLoss: 40.815052\tACC: 0.7278668\n",
            "Epoch 19/30\t0.08s\tLoss: 40.689014\tACC: 0.7289079\n",
            "Epoch 20/30\t0.08s\tLoss: 40.7295\tACC: 0.72763026\n",
            "Epoch 21/30\t0.08s\tLoss: 40.591953\tACC: 0.7324466\n",
            "Epoch 22/30\t0.08s\tLoss: 40.523\tACC: 0.7298727\n",
            "Epoch 23/30\t0.08s\tLoss: 40.559246\tACC: 0.7341505\n",
            "Epoch 24/30\t0.08s\tLoss: 40.493793\tACC: 0.7334368\n",
            "Epoch 25/30\t0.08s\tLoss: 40.400608\tACC: 0.7348708\n",
            "Epoch 26/30\t0.08s\tLoss: 40.361576\tACC: 0.73602545\n",
            "Epoch 27/30\t0.07s\tLoss: 40.012253\tACC: 0.74290913\n",
            "Epoch 28/30\t0.08s\tLoss: 40.05596\tACC: 0.73844963\n",
            "Epoch 29/30\t0.07s\tLoss: 40.039593\tACC: 0.74168634\n",
            "Epoch 30/30\t0.08s\tLoss: 40.20735\tACC: 0.74093264\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "30 epochs, 128 batch size and 0.001 learning  ▶ 4.04 s trainin time and .729 accuracy"
      ],
      "metadata": {
        "id": "1kzgBzWEOWi4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.00001)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    # .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])"
      ],
      "metadata": {
        "id": "6HZ_sglwXXZ9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ktyPNNAcXXdb",
        "outputId": "b37a92f1-75d8-4a4b-84c6-31cc5d631504"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n",
            "CPU times: user 487 ms, sys: 26.9 ms, total: 514 ms\n",
            "Wall time: 5.71 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! ls -t $log_folder | head -n 1\n",
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k4mvk2KTeogF",
        "outputId": "73c19d12-5dfa-4676-ea8d-878d0c66ecb8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GenericSVMClassifierApproach_648c063ac472.log\n",
            "Training 30 epochs\n",
            "Epoch 1/30\t0.13s\tLoss: 82.54853\tACC: 0.33559406\n",
            "Epoch 2/30\t0.08s\tLoss: 81.92425\tACC: 0.34051597\n",
            "Epoch 3/30\t0.07s\tLoss: 81.43048\tACC: 0.35108808\n",
            "Epoch 4/30\t0.07s\tLoss: 80.90814\tACC: 0.35859457\n",
            "Epoch 5/30\t0.07s\tLoss: 80.25813\tACC: 0.3625155\n",
            "Epoch 6/30\t0.07s\tLoss: 79.60011\tACC: 0.379815\n",
            "Epoch 7/30\t0.07s\tLoss: 79.189514\tACC: 0.3871758\n",
            "Epoch 8/30\t0.07s\tLoss: 78.62741\tACC: 0.39786822\n",
            "Epoch 9/30\t0.08s\tLoss: 78.028305\tACC: 0.4155245\n",
            "Epoch 10/30\t0.08s\tLoss: 77.59669\tACC: 0.42686105\n",
            "Epoch 11/30\t0.08s\tLoss: 77.0726\tACC: 0.4451361\n",
            "Epoch 12/30\t0.08s\tLoss: 76.62101\tACC: 0.45760855\n",
            "Epoch 13/30\t0.08s\tLoss: 76.11975\tACC: 0.46958923\n",
            "Epoch 14/30\t0.08s\tLoss: 75.80697\tACC: 0.4820323\n",
            "Epoch 15/30\t0.08s\tLoss: 75.261925\tACC: 0.4988841\n",
            "Epoch 16/30\t0.07s\tLoss: 74.8803\tACC: 0.5162531\n",
            "Epoch 17/30\t0.07s\tLoss: 74.444275\tACC: 0.53161216\n",
            "Epoch 18/30\t0.07s\tLoss: 74.27799\tACC: 0.53538746\n",
            "Epoch 19/30\t0.07s\tLoss: 73.84354\tACC: 0.55228597\n",
            "Epoch 20/30\t0.07s\tLoss: 73.53962\tACC: 0.5622501\n",
            "Epoch 21/30\t0.07s\tLoss: 72.960175\tACC: 0.58106774\n",
            "Epoch 22/30\t0.08s\tLoss: 72.64797\tACC: 0.60165876\n",
            "Epoch 23/30\t0.08s\tLoss: 72.390884\tACC: 0.6061543\n",
            "Epoch 24/30\t0.07s\tLoss: 72.09573\tACC: 0.61597675\n",
            "Epoch 25/30\t0.08s\tLoss: 71.71185\tACC: 0.6273493\n",
            "Epoch 26/30\t0.08s\tLoss: 71.51786\tACC: 0.63224584\n",
            "Epoch 27/30\t0.08s\tLoss: 71.21124\tACC: 0.6447036\n",
            "Epoch 28/30\t0.08s\tLoss: 70.8872\tACC: 0.64998233\n",
            "Epoch 29/30\t0.08s\tLoss: 70.68389\tACC: 0.65363336\n",
            "Epoch 30/30\t0.08s\tLoss: 70.35876\tACC: 0.6639035\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "30 epochs, 128 batch size and 0.001 learning  ▶ 5.82 s trainin time and .740 accuracy\n",
        "\n",
        "30 epochs, 128 batch size and 0.00001 learning  ▶ 5.71 s trainin time and .663 accuracy\n",
        "\n",
        "because we are using small amount of data for training purposes, there is small differences in metrics, but normally, learning rate is very crucial.\n",
        "in this demo, we see the accuracy decreased to sharply, smaller learning rate needs more epoch to reach optimums"
      ],
      "metadata": {
        "id": "xrhLgJQdfuvB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶`featureScaling`"
      ],
      "metadata": {
        "id": "6ISMg7wug-_0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rdf = pd.DataFrame()"
      ],
      "metadata": {
        "id": "AaXYMzBG-TQN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TeQmgaClhqZg",
        "outputId": "6e17ef0d-1575-4856-d83d-86c3dfcb0457"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_ = model.transform(trainingData_with_embeddings)"
      ],
      "metadata": {
        "id": "3DUIFDk0txMe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result_.select(result_.text, result_.prediction.metadata).show(5, truncate=50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lai2H6z4vFxh",
        "outputId": "fe81253b-286d-4120-978f-5c5a7530cc5e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------------------------------------+----------------------------+\n",
            "|                                              text|         prediction.metadata|\n",
            "+--------------------------------------------------+----------------------------+\n",
            "| A 15-year-old boy had temporary hypertropia, s...|[{confidence -> 0.78144157}]|\n",
            "| A 17-year-old anuric female patient with end-s...|[{confidence -> 0.76712304}]|\n",
            "| A 28-year-old female patient complained of int...|[{confidence -> 0.79150796}]|\n",
            "| A 31 year old woman presented with a chest wal...|[{confidence -> 0.84184587}]|\n",
            "| A 43-year-old patient with preexisting mitral ...| [{confidence -> 0.8219223}]|\n",
            "+--------------------------------------------------+----------------------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_.printSchema()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_wedG304t_FI",
        "outputId": "4edd21c9-724e-4787-fe9c-9eb3b9185c67"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- text: string (nullable = true)\n",
            " |-- category: string (nullable = true)\n",
            " |-- sentence_embeddings: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            " |-- feature_vector: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            " |-- prediction: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- annotatorType: string (nullable = true)\n",
            " |    |    |-- begin: integer (nullable = false)\n",
            " |    |    |-- end: integer (nullable = false)\n",
            " |    |    |-- result: string (nullable = true)\n",
            " |    |    |-- metadata: map (nullable = true)\n",
            " |    |    |    |-- key: string\n",
            " |    |    |    |-- value: string (valueContainsNull = true)\n",
            " |    |    |-- embeddings: array (nullable = true)\n",
            " |    |    |    |-- element: float (containsNull = false)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rdf_ = result_.select(result_.text,result_.category, result_.prediction.result[0],result_.prediction.metadata[0]).toPandas()\n",
        "rdf_.columns  = ['text','category', 'predicted category', 'conf']\n",
        "rdf_['confidence'] = rdf_.conf.apply(lambda conf:float(conf['confidence']))\n",
        "rdf_ = rdf_[['text','category', 'predicted category', 'confidence']]\n",
        "rdf = rdf_[['text','category', 'predicted category']]\n",
        "rdf['conf_'] = rdf_['confidence']\n",
        "rdf.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "L6yMOtgz9ZWx",
        "outputId": "58c18b85-88fc-4941-9f4a-1c3ab82b58a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category  \\\n",
              "0   A 15-year-old boy had temporary hypertropia, ...      neg   \n",
              "1   A 17-year-old anuric female patient with end-...      neg   \n",
              "2   A 28-year-old female patient complained of in...      neg   \n",
              "3   A 31 year old woman presented with a chest wa...      neg   \n",
              "4   A 43-year-old patient with preexisting mitral...      neg   \n",
              "\n",
              "  predicted category     conf_  \n",
              "0                neg  0.781442  \n",
              "1                neg  0.767123  \n",
              "2                neg  0.791508  \n",
              "3                neg  0.841846  \n",
              "4                neg  0.821922  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e0edc421-709d-4a03-bf4b-53568e59a634\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "      <th>predicted category</th>\n",
              "      <th>conf_</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A 15-year-old boy had temporary hypertropia, ...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.781442</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A 17-year-old anuric female patient with end-...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.767123</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A 28-year-old female patient complained of in...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.791508</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A 31 year old woman presented with a chest wa...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.841846</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A 43-year-old patient with preexisting mitral...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.821922</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e0edc421-709d-4a03-bf4b-53568e59a634')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e0edc421-709d-4a03-bf4b-53568e59a634 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e0edc421-709d-4a03-bf4b-53568e59a634');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-63fcbfa8-6b89-4db8-a861-cf37e34c46c0\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-63fcbfa8-6b89-4db8-a861-cf37e34c46c0')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-63fcbfa8-6b89-4db8-a861-cf37e34c46c0 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 213
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setFeatureScaling('zscore')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XKWDeN3y-2BJ",
        "outputId": "4a605d41-33e3-41b2-8a97-54731d63362a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_zscore = model.transform(trainingData_with_embeddings)"
      ],
      "metadata": {
        "id": "wH10hjaw_GY0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rdf_zscore = result_zscore.select(result_zscore.text,result_zscore.category, result_zscore.prediction.result[0],result_zscore.prediction.metadata[0]).toPandas()\n",
        "rdf_zscore.columns  = ['text','category', 'predicted category', 'conf']\n",
        "rdf_zscore['confidence'] = rdf_zscore.conf.apply(lambda conf:float(conf['confidence']))\n",
        "rdf['conf_zscore'] = rdf_zscore['confidence']\n",
        "\n",
        "rdf.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "EKXupFq651FV",
        "outputId": "83c20426-d1e1-4246-d055-b72605742f49"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category  \\\n",
              "0   A 15-year-old boy had temporary hypertropia, ...      neg   \n",
              "1   A 17-year-old anuric female patient with end-...      neg   \n",
              "2   A 28-year-old female patient complained of in...      neg   \n",
              "3   A 31 year old woman presented with a chest wa...      neg   \n",
              "4   A 43-year-old patient with preexisting mitral...      neg   \n",
              "\n",
              "  predicted category     conf_  conf_zscore  \n",
              "0                neg  0.781442     0.759435  \n",
              "1                neg  0.767123     0.632024  \n",
              "2                neg  0.791508     0.665319  \n",
              "3                neg  0.841846     0.794640  \n",
              "4                neg  0.821922     0.857194  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2eb720b5-1692-44a1-b659-65bc63eb3c65\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "      <th>predicted category</th>\n",
              "      <th>conf_</th>\n",
              "      <th>conf_zscore</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A 15-year-old boy had temporary hypertropia, ...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.781442</td>\n",
              "      <td>0.759435</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A 17-year-old anuric female patient with end-...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.767123</td>\n",
              "      <td>0.632024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A 28-year-old female patient complained of in...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.791508</td>\n",
              "      <td>0.665319</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A 31 year old woman presented with a chest wa...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.841846</td>\n",
              "      <td>0.794640</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A 43-year-old patient with preexisting mitral...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.821922</td>\n",
              "      <td>0.857194</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2eb720b5-1692-44a1-b659-65bc63eb3c65')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2eb720b5-1692-44a1-b659-65bc63eb3c65 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2eb720b5-1692-44a1-b659-65bc63eb3c65');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9fadb0ed-9723-476b-b138-dbca97cab0b4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9fadb0ed-9723-476b-b138-dbca97cab0b4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9fadb0ed-9723-476b-b138-dbca97cab0b4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 216
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setValidationSplit(0.1)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N0flA9DcAIV0",
        "outputId": "88f47829-7977-4eb8-afdd-224cfa1015c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_minmax = model.transform(trainingData_with_embeddings)\n"
      ],
      "metadata": {
        "id": "2UaY2HxmAIaR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rdf_minmax = result_minmax.select(result_minmax.text,result_minmax.category, result_minmax.prediction.result[0],result_minmax.prediction.metadata[0]).toPandas()\n",
        "rdf_minmax.columns  = ['text','category', 'predicted category', 'conf']\n",
        "rdf_minmax['confidence'] = rdf_minmax.conf.apply(lambda conf:float(conf['confidence']))\n",
        "\n",
        "rdf['conf_minmax'] = rdf_minmax['confidence']\n",
        "rdf.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "JyZkPiLl2SM7",
        "outputId": "7d6259a6-b6c4-4a77-c1e0-ca9d2a38d3fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text category  \\\n",
              "0   A 15-year-old boy had temporary hypertropia, ...      neg   \n",
              "1   A 17-year-old anuric female patient with end-...      neg   \n",
              "2   A 28-year-old female patient complained of in...      neg   \n",
              "3   A 31 year old woman presented with a chest wa...      neg   \n",
              "4   A 43-year-old patient with preexisting mitral...      neg   \n",
              "\n",
              "  predicted category     conf_  conf_zscore  conf_minmax  \n",
              "0                neg  0.781442     0.759435     0.838469  \n",
              "1                neg  0.767123     0.632024     0.836075  \n",
              "2                neg  0.791508     0.665319     0.835654  \n",
              "3                neg  0.841846     0.794640     0.834675  \n",
              "4                neg  0.821922     0.857194     0.809086  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-99fd8cdb-343f-405e-9871-c776e36b1be9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "      <th>predicted category</th>\n",
              "      <th>conf_</th>\n",
              "      <th>conf_zscore</th>\n",
              "      <th>conf_minmax</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A 15-year-old boy had temporary hypertropia, ...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.781442</td>\n",
              "      <td>0.759435</td>\n",
              "      <td>0.838469</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A 17-year-old anuric female patient with end-...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.767123</td>\n",
              "      <td>0.632024</td>\n",
              "      <td>0.836075</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A 28-year-old female patient complained of in...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.791508</td>\n",
              "      <td>0.665319</td>\n",
              "      <td>0.835654</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A 31 year old woman presented with a chest wa...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.841846</td>\n",
              "      <td>0.794640</td>\n",
              "      <td>0.834675</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A 43-year-old patient with preexisting mitral...</td>\n",
              "      <td>neg</td>\n",
              "      <td>neg</td>\n",
              "      <td>0.821922</td>\n",
              "      <td>0.857194</td>\n",
              "      <td>0.809086</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-99fd8cdb-343f-405e-9871-c776e36b1be9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-99fd8cdb-343f-405e-9871-c776e36b1be9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-99fd8cdb-343f-405e-9871-c776e36b1be9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2f958e39-6a3c-4e95-acc1-3c92f11a9ee1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2f958e39-6a3c-4e95-acc1-3c92f11a9ee1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2f958e39-6a3c-4e95-acc1-3c92f11a9ee1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 219
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**As shown above, three different featureScaling option generate different confidence, and the best one may be chosen according to problem/dataset**"
      ],
      "metadata": {
        "id": "qmUDvsA6_9nZ"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "EhC4PWNwXuvC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶`ValidationSplit`"
      ],
      "metadata": {
        "id": "gY9PAwIziW3W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        " when `.setValidationSplit(0.1)` set, logs are also having validation scores, by doing so, we can see our model quality during the training process by comparing validatin accuracy score and training accuracy score."
      ],
      "metadata": {
        "id": "rpnNxFG8ipgE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! ls -t $log_folder | head -n 1\n",
        "! cat $log_folder/$(ls -t $log_folder | head -n 1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1GRRDk38hqgn",
        "outputId": "f113afb2-c4f5-4c66-9208-81a32541d28b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GenericSVMClassifierApproach_4c65c6c5682d.log\n",
            "Training 30 epochs\n",
            "Epoch 1/30\t0.15s\tLoss: 46.637627\tACC: 0.70650655\tValidation ACC: 0.7295864\n",
            "Epoch 2/30\t0.08s\tLoss: 43.55682\tACC: 0.7100047\tValidation ACC: 0.7295864\n",
            "Epoch 3/30\t0.07s\tLoss: 42.528988\tACC: 0.7118703\tValidation ACC: 0.7295864\n",
            "Epoch 4/30\t0.07s\tLoss: 42.17951\tACC: 0.70942163\tValidation ACC: 0.7295864\n",
            "Epoch 5/30\t0.07s\tLoss: 41.408833\tACC: 0.7109375\tValidation ACC: 0.7295864\n",
            "Epoch 6/30\t0.07s\tLoss: 41.22085\tACC: 0.7114039\tValidation ACC: 0.7295864\n",
            "Epoch 7/30\t0.07s\tLoss: 40.84143\tACC: 0.7108209\tValidation ACC: 0.7295864\n",
            "Epoch 8/30\t0.07s\tLoss: 40.79271\tACC: 0.7100047\tValidation ACC: 0.7295864\n",
            "Epoch 9/30\t0.07s\tLoss: 40.288013\tACC: 0.7105877\tValidation ACC: 0.7295864\n",
            "Epoch 10/30\t0.07s\tLoss: 40.098537\tACC: 0.7110541\tValidation ACC: 0.7295864\n",
            "Epoch 11/30\t0.07s\tLoss: 39.95572\tACC: 0.7110541\tValidation ACC: 0.7295864\n",
            "Epoch 12/30\t0.07s\tLoss: 39.82334\tACC: 0.71163714\tValidation ACC: 0.7295864\n",
            "Epoch 13/30\t0.07s\tLoss: 39.824547\tACC: 0.71023786\tValidation ACC: 0.7295864\n",
            "Epoch 14/30\t0.07s\tLoss: 40.10599\tACC: 0.7090718\tValidation ACC: 0.7295864\n",
            "Epoch 15/30\t0.07s\tLoss: 39.83158\tACC: 0.7108209\tValidation ACC: 0.7295864\n",
            "Epoch 16/30\t0.07s\tLoss: 39.79498\tACC: 0.708722\tValidation ACC: 0.7295864\n",
            "Epoch 17/30\t0.07s\tLoss: 39.70448\tACC: 0.7090718\tValidation ACC: 0.7295864\n",
            "Epoch 18/30\t0.07s\tLoss: 39.38595\tACC: 0.71117073\tValidation ACC: 0.7295864\n",
            "Epoch 19/30\t0.07s\tLoss: 39.77536\tACC: 0.7103545\tValidation ACC: 0.7295864\n",
            "Epoch 20/30\t0.07s\tLoss: 39.29447\tACC: 0.71245337\tValidation ACC: 0.7295864\n",
            "Epoch 21/30\t0.07s\tLoss: 39.307087\tACC: 0.71070427\tValidation ACC: 0.7295864\n",
            "Epoch 22/30\t0.07s\tLoss: 39.529385\tACC: 0.7100047\tValidation ACC: 0.7295864\n",
            "Epoch 23/30\t0.07s\tLoss: 39.54308\tACC: 0.71070427\tValidation ACC: 0.7295864\n",
            "Epoch 24/30\t0.07s\tLoss: 39.47582\tACC: 0.70930505\tValidation ACC: 0.7295864\n",
            "Epoch 25/30\t0.07s\tLoss: 39.501244\tACC: 0.7103545\tValidation ACC: 0.7295864\n",
            "Epoch 26/30\t0.07s\tLoss: 39.23721\tACC: 0.7110541\tValidation ACC: 0.7295864\n",
            "Epoch 27/30\t0.07s\tLoss: 39.411274\tACC: 0.70930505\tValidation ACC: 0.7295864\n",
            "Epoch 28/30\t0.07s\tLoss: 39.411396\tACC: 0.71070427\tValidation ACC: 0.7295864\n",
            "Epoch 29/30\t0.07s\tLoss: 39.38477\tACC: 0.7117537\tValidation ACC: 0.7295864\n",
            "Epoch 30/30\t0.07s\tLoss: 39.2411\tACC: 0.7114039\tValidation ACC: 0.7295864\n",
            "Training completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `multiClass`"
      ],
      "metadata": {
        "id": "Rq8TxfhQYkhJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Whether to return only the label with the highest confidence score or all labels"
      ],
      "metadata": {
        "id": "pIivTlv0YxKP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setValidationSplit(0.1)\\\n",
        "    .setMultiClass(True)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "msQUmBA_YwYq",
        "outputId": "e525fa25-297a-422d-8346-8bd4f8d56f9a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = model.transform(testData_with_embeddings)"
      ],
      "metadata": {
        "id": "AKhh_emSiMXE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df.select('prediction').show(5, truncate=False)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uOtNlEJeZcIX",
        "outputId": "f200e5d5-cc5e-4702-a413-84389e36f3cd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------+\n",
            "|prediction                                                                                                    |\n",
            "+--------------------------------------------------------------------------------------------------------------+\n",
            "|[{category, 0, 0, pos, {confidence -> 0.6204573}, []}, {category, 0, 0, neg, {confidence -> 0.82777494}, []}] |\n",
            "|[{category, 0, 0, pos, {confidence -> 0.66048753}, []}, {category, 0, 0, neg, {confidence -> 0.85109913}, []}]|\n",
            "|[{category, 0, 0, pos, {confidence -> 0.6559448}, []}, {category, 0, 0, neg, {confidence -> 0.8332604}, []}]  |\n",
            "|[{category, 0, 0, pos, {confidence -> 0.6407703}, []}, {category, 0, 0, neg, {confidence -> 0.8241285}, []}]  |\n",
            "|[{category, 0, 0, pos, {confidence -> 0.66346145}, []}, {category, 0, 0, neg, {confidence -> 0.8514608}, []}] |\n",
            "+--------------------------------------------------------------------------------------------------------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n",
        "preds_df = pred_df.select('category','prediction.result').toPandas()\n",
        "preds_df['result'] = preds_df.result.apply(lambda x : x[0])\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kec6wLeKpZa1",
        "outputId": "8c183e59-0022-499a-a443-942bd7887f5a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.73      0.97      0.83      1704\n",
            "         pos       0.65      0.12      0.20       695\n",
            "\n",
            "    accuracy                           0.73      2399\n",
            "   macro avg       0.69      0.55      0.52      2399\n",
            "weighted avg       0.71      0.73      0.65      2399\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `setFixImbalance`"
      ],
      "metadata": {
        "id": "zaH_oC5knv5M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "default False\n",
        "when set True:\n",
        "Algorithm duplicates the minor classes randomly"
      ],
      "metadata": {
        "id": "TdbG5LoEnv5a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.01)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(False)\\\n",
        "    .setMultiClass(False)\n",
        "    # .setValidationSplit(0.1)\\\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c455155c-0270-4f08-ccb0-3a6aa9577a6b",
        "id": "q8dg9X6Fnv5b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = model.transform(testData_with_embeddings)"
      ],
      "metadata": {
        "id": "pYKJD7W1nv5b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = pred_df.select('category','prediction.result').toPandas()\n",
        "preds_df['result'] = preds_df.result.apply(lambda x : x[0])\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "965010dd-e84b-4d28-9a8e-cab10e06dc8a",
        "id": "BpGhQpf7nv5c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.73      0.97      0.83      1704\n",
            "         pos       0.65      0.12      0.20       695\n",
            "\n",
            "    accuracy                           0.73      2399\n",
            "   macro avg       0.69      0.55      0.52      2399\n",
            "weighted avg       0.71      0.73      0.65      2399\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.01)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.1)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setMultiClass(False)\n",
        "    # .setValidationSplit(0.1)\\\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)\n",
        "\n",
        "preds_df = model.transform(testData_with_embeddings)\n",
        "preds_df = pred_df.select('category','prediction.result').toPandas()\n",
        "preds_df['result'] = preds_df.result.apply(lambda x : x[0])\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dn2pM5-lA7NS",
        "outputId": "5038a536-2b6e-4c02-c31f-266c61c26e0a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.73      0.97      0.83      1704\n",
            "         pos       0.65      0.12      0.20       695\n",
            "\n",
            "    accuracy                           0.73      2399\n",
            "   macro avg       0.69      0.55      0.52      2399\n",
            "weighted avg       0.71      0.73      0.65      2399\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `setDropout`"
      ],
      "metadata": {
        "id": "uD7gfZX2tbnQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Dropout is a regularization technique commonly used in deep learning algorithms, especially in neural networks, to prevent overfitting and improve the generalization ability of the model. The \"dropout ratio\" is a key hyperparameter associated with dropout, and it refers to the probability that a particular neuron or unit in a neural network layer will be \"dropped out\" or temporarily removed during each training iteration\n",
        "\n",
        "**`default:0.05`**"
      ],
      "metadata": {
        "id": "RrJkGmHpuEoK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# letx compare 0.015 and 0.00"
      ],
      "metadata": {
        "id": "rK5mSHjyuirg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.015)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setMultiClass(False)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5bf20415-6589-4f5d-fad4-769cb58c6ad3",
        "id": "7U9v403Jtbnj"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = model.transform(testData_with_embeddings)"
      ],
      "metadata": {
        "id": "jNFh0jqvtbnj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = pred_df.select('category','prediction.result').toPandas()\n",
        "preds_df['result'] = preds_df.result.apply(lambda x : x[0])\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e46b0c2-5367-45d5-ac57-9e8070c6b61b",
        "id": "kwQjFQSDtbnj"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.73      0.97      0.83      1704\n",
            "         pos       0.65      0.12      0.20       695\n",
            "\n",
            "    accuracy                           0.73      2399\n",
            "   macro avg       0.69      0.55      0.52      2399\n",
            "weighted avg       0.71      0.73      0.65      2399\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gen_clf = medical.GenericSVMClassifierApproach()\\\n",
        "    .setLabelColumn(\"category\")\\\n",
        "    .setInputCols(\"feature_vector\")\\\n",
        "    .setOutputCol(\"prediction\")\\\n",
        "    .setModelFile(f\"{graph_folder}/svm_graph.pb\")\\\n",
        "    .setEpochsNumber(30)\\\n",
        "    .setBatchSize(128)\\\n",
        "    .setLearningRate(0.001)\\\n",
        "    .setFeatureScaling('minmax')\\\n",
        "    .setOutputLogsPath(log_folder)\\\n",
        "    .setDropout(0.00)\\\n",
        "    .setFixImbalance(True)\\\n",
        "    .setMultiClass(False)\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    features_asm,\n",
        "    gc_svm_graph_builder,\n",
        "    gen_clf])\n",
        "model = clf_Pipeline.fit(trainingData_with_embeddings)\n"
      ],
      "metadata": {
        "id": "Nm0Myl_aOWi4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "10f7f289-67eb-4480-d814-afcc63344049"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF Graph Builder configuration:\n",
            "Model name: svm_classifier\n",
            "Graph folder: graph_folder\n",
            "Graph file name: svm_graph.pb\n",
            "Build params: {'input_dim': 100, 'output_dim': 2, 'hidden_layers': [], 'output_act': 'sigmoid', 'loss_func': 'hinge'}\n",
            "svm_classifier graph exported to graph_folder/svm_graph.pb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = model.transform(testData_with_embeddings)\n",
        "preds_df = pred_df.select('category','prediction.result').toPandas()\n",
        "preds_df['result'] = preds_df.result.apply(lambda x : x[0])\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HTVuV-tIOWi4",
        "outputId": "c1fe3d35-f708-4f6b-ed54-6bc864b441b5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.71      1.00      0.83      1704\n",
            "         pos       0.71      0.02      0.03       695\n",
            "\n",
            "    accuracy                           0.71      2399\n",
            "   macro avg       0.71      0.51      0.43      2399\n",
            "weighted avg       0.71      0.71      0.60      2399\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "***f1-score for pos changed dramatically***"
      ],
      "metadata": {
        "id": "l2HbIr5Ywk04"
      }
    }
  ]
}