Spark_NLP_Udemy_MOOC/Healthcare_NLP/GenericSVMClassifierModel.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ],
      "metadata": {
        "id": "8V_Ow3cAVEYe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **GenericSVMClassifierModel**"
      ],
      "metadata": {
        "id": "Y0fJRpNJBslT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook will cover the different parameters and usages of `GenericSVMClassifierModel`.\n",
        "\n",
        "**📖 Learning Objectives:**\n",
        "\n",
        "1. Background: Understand the 'GenericSVMClassifierModel' annotator.\n",
        "\n",
        "2. Colab setup.\n",
        "\n",
        "3. Become comfortable with using the different parameters of the annotator.\n",
        "\n",
        "**🔗 Helpful Links:**\n",
        "\n",
        "- Python Docs : [GenericSVMClassifierModel](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/classification/generic_svm_classifier/index.html#sparknlp_jsl.annotator.classification.generic_svm_classifier.GenericSVMClassifierModel)\n",
        "\n",
        "- Scala Docs: [GenericSVMClassifierModel](https://nlp.johnsnowlabs.com/licensed/api/com/johnsnowlabs/nlp/annotators/classification/GenericSVMClassifierModel.html)\n",
        "\n",
        "- For extended examples of usage, see [Spark NLP Workshop repository](https://colab.research.google.com/drive/12JQuR7cSUeYfCu418REcqkSmkYWq8Efb#scrollTo=Ltt3uq8y9MKR).\n"
      ],
      "metadata": {
        "id": "qeclGJmrVLjX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **📜 Background**"
      ],
      "metadata": {
        "id": "OV0hPSCWXslc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`GenericSVMClassifier` is a derivative of [GenericClassifier](https://nlp.johnsnowlabs.com/docs/en/licensed_annotators#genericclassifier), which implements [SVM (Support Vector Machine) classification](https://en.wikipedia.org/wiki/Support_vector_machine).\n",
        "\n",
        "The input to the model is the `FeatureVector` and the output is category annotations with labels and corresponding confidence scores.\n",
        "\n",
        "The scores are standardized using the logistic function so that they vary between 0 and 1."
      ],
      "metadata": {
        "id": "_-hGkZ7_cAZE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🎬 Colab Setup**"
      ],
      "metadata": {
        "id": "E8qy2MI2XySv"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HuLFt0OdBkuo"
      },
      "outputs": [],
      "source": [
        "# Install the johnsnowlabs library to access Spark-NLP for Healthcare\n",
        "! pip install -q johnsnowlabs==5.1.0"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ],
      "metadata": {
        "id": "HJGAUiIqBI94"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from johnsnowlabs import nlp, medical\n",
        "\n",
        "# After uploading your license run this to install all licensed Python Wheels and pre-download Jars the Spark Session JVM\n",
        "nlp.install()"
      ],
      "metadata": {
        "id": "SBhRUP21BI7R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from johnsnowlabs import nlp, medical\n",
        "import pandas as pd\n",
        "from pyspark.sql import SparkSession\n",
        "from sparknlp.annotator import *\n",
        "from sparknlp_jsl.annotator import *\n",
        "from sparknlp.base import *\n",
        "import sparknlp_jsl\n",
        "import sparknlp\n",
        "import pandas as pd\n",
        "\n",
        "# Automatically load license data and start a session with all jars user has access to\n",
        "spark = nlp.start()"
      ],
      "metadata": {
        "id": "a_1HS4iCB5ie"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "id": "aok3umndL-H-",
        "outputId": "63c11ec1-b82e-4ae3-ad94-a0f3a0b325b8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x790901f2ec20>"
            ],
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://2205ca7c4fea:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.1.2</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>John-Snow-Labs-Spark-Session 🚀 with Jars for: 🚀Spark-NLP==5.1.0, 💊Spark-Healthcare==5.1.0, running on ⚡ PySpark==3.1.2</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🖨️ Input/Output Annotation Types**\n",
        "\n"
      ],
      "metadata": {
        "id": "f_4conOIYj6D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Input: `FEATURE_VECTOR`\n",
        "\n",
        "- Output: `CATEGORY`"
      ],
      "metadata": {
        "id": "0Fc_3iRwYk8N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🔎 Parameters**"
      ],
      "metadata": {
        "id": "IO7xgW6Znobb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`featureScaling`: Feature scaling method. Possible values are '`zscore`', '`minmax`' or empty (no scaling) (default: )\n",
        "\n",
        "`multiClass`: Whether to return only the label with the highest confidence score or all labels (default: False, current: False)\n",
        "\n",
        "\n",
        "`inputCols`: previous annotations columns, if renamed (current: ['features'])\n",
        "\n",
        "`outputCol`: output annotation column. can be left default. (current: class)\n",
        "\n"
      ],
      "metadata": {
        "id": "2Vv-dsM2njQT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ✍  Explaining GenericSVMClassifierModel with an Example"
      ],
      "metadata": {
        "id": "g9zdtP-D5spx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 📑 Sample Data"
      ],
      "metadata": {
        "id": "txGhMorxR0s_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#downloading sample datasets\n",
        "!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/ADE_Corpus_V2/ADE-NEG.txt\n",
        "!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/ADE_Corpus_V2/DRUG-AE.rel"
      ],
      "metadata": {
        "id": "FNgWV3Fzjyop"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_neg= pd.read_csv(\"ADE-NEG.txt\", header=None, delimiter=\"\\t\", names=[\"col1\"])\n",
        "df_neg['text'] =  df_neg.col1.str.split('NEG').str[1]\n",
        "df_neg[\"category\"] = \"neg\"\n",
        "df_neg= df_neg[[\"text\", \"category\"]]\n",
        "\n",
        "df_pos= pd.read_csv(\"DRUG-AE.rel\", header=None, delimiter=\"|\")\n",
        "df_pos[\"category\"]= \"pos\"\n",
        "df_pos.rename(columns={1: \"text\"}, inplace=True)\n",
        "df_pos= df_pos[[\"text\", \"category\"]]\n",
        "\n",
        "ade_df= pd.concat([df_neg, df_pos])\n",
        "ade_df.category = ade_df.category.apply(lambda x: True if x==\"pos\" else False)\n",
        "ade_df.sample(5, random_state=42)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "kVetRK19C-sy",
        "outputId": "48a94b75-56ee-436d-9160-4e08af53eaf3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                    text  category\n",
              "11571   Nineteen months after diagnosis, her mother g...     False\n",
              "4577   An infant girl with choanal atresia, athelia, ...      True\n",
              "5638    A causal relationship between the administrat...     False\n",
              "12799   The absence of eosinophils does not exclude t...     False\n",
              "427    While for ribavirin antidepressant effects are...      True"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-08a68903-d9d7-4131-a874-918a4e1c05b3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>11571</th>\n",
              "      <td>Nineteen months after diagnosis, her mother g...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4577</th>\n",
              "      <td>An infant girl with choanal atresia, athelia, ...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5638</th>\n",
              "      <td>A causal relationship between the administrat...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12799</th>\n",
              "      <td>The absence of eosinophils does not exclude t...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>427</th>\n",
              "      <td>While for ribavirin antidepressant effects are...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-08a68903-d9d7-4131-a874-918a4e1c05b3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-08a68903-d9d7-4131-a874-918a4e1c05b3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-08a68903-d9d7-4131-a874-918a4e1c05b3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-515fb2ca-edc7-444f-b83a-82de16b077d1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-515fb2ca-edc7-444f-b83a-82de16b077d1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-515fb2ca-edc7-444f-b83a-82de16b077d1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql.types import StringType\n",
        "sample_data = ade_df.sample(n= 100, random_state=1)  # taking 100 rows as sample data\n",
        "sample_data = sample_data.reset_index(drop=True)\n",
        "df = spark.createDataFrame(sample_data['text'].tolist(), StringType()).toDF(\"text\")\n",
        "df.show(5,truncate=60)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uwk_enf0-iKq",
        "outputId": "f15b6b3d-667f-4692-e1e2-b9d7dca603df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------------------------------------------------------+\n",
            "|                                                        text|\n",
            "+------------------------------------------------------------+\n",
            "| Multicentric canine lymphoma in a 12-year-old keeshond: ...|\n",
            "| Pyomyositis is a rare disease, encountered mainly in tro...|\n",
            "| Both patients subsequently developed markedly elevated E...|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastroi...|\n",
            "|However, given the clinically significant result to the i...|\n",
            "+------------------------------------------------------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sample_data.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "T6L5D_g--hoX",
        "outputId": "c967b821-9e9a-4b12-c613-fe255dad78eb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text  category\n",
              "0   Multicentric canine lymphoma in a 12-year-old...     False\n",
              "1   Pyomyositis is a rare disease, encountered ma...     False\n",
              "2   Both patients subsequently developed markedly...     False\n",
              "3  Bortezomib-induced paralytic ileus is a potent...      True\n",
              "4  However, given the clinically significant resu...      True"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-58911330-153c-4616-8a6e-f001cdbd0f36\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Multicentric canine lymphoma in a 12-year-old...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Pyomyositis is a rare disease, encountered ma...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Both patients subsequently developed markedly...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Bortezomib-induced paralytic ileus is a potent...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>However, given the clinically significant resu...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-58911330-153c-4616-8a6e-f001cdbd0f36')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-58911330-153c-4616-8a6e-f001cdbd0f36 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-58911330-153c-4616-8a6e-f001cdbd0f36');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-1d8c660f-aece-4464-aec1-933696dba4ba\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1d8c660f-aece-4464-aec1-933696dba4ba')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-1d8c660f-aece-4464-aec1-933696dba4ba button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **💻Pipeline**"
      ],
      "metadata": {
        "id": "pagXc73HWidS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "document_assembler = nlp.DocumentAssembler()\\\n",
        "    .setInputCol(\"text\")\\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "tokenizer = nlp.Tokenizer()\\\n",
        "    .setInputCols(\"document\")\\\n",
        "    .setOutputCol(\"token\")\n",
        "\n",
        "word_embeddings = nlp.WordEmbeddingsModel.pretrained(\"embeddings_clinical\",\"en\",\"clinical/models\")\\\n",
        "    .setInputCols([\"document\",\"token\"])\\\n",
        "    .setOutputCol(\"word_embeddings\")\n",
        "\n",
        "sentence_embeddings = nlp.SentenceEmbeddings() \\\n",
        "    .setInputCols([\"document\", \"word_embeddings\"]) \\\n",
        "    .setOutputCol(\"sentence_embeddings\") \\\n",
        "    .setPoolingStrategy(\"AVERAGE\")\n",
        "\n",
        "features_asm = medical.FeaturesAssembler()\\\n",
        "    .setInputCols([\"sentence_embeddings\"])\\\n",
        "    .setOutputCol(\"features\")\n",
        "\n",
        "generic_classifier = medical.GenericSVMClassifierModel.pretrained(\"generic_svm_classifier_ade\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"features\"])\\\n",
        "    .setOutputCol(\"class\")\n",
        "\n",
        "clf_Pipeline = nlp.Pipeline(stages=[\n",
        "    document_assembler,\n",
        "    tokenizer,\n",
        "    word_embeddings,\n",
        "    sentence_embeddings,\n",
        "    features_asm,\n",
        "    generic_classifier])\n",
        "\n",
        "data = spark.createDataFrame([[\"\"\"None of the patients required treatment for the overdose.\"\"\"],\n",
        " [\"\"\"I feel a bit drowsy & have a little blurred vision after taking an insulin\"\"\"]]).toDF(\"text\")\n",
        "\n"
      ],
      "metadata": {
        "id": "EFVK8wWODWjX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "generic_classifier"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5QSu9PWGYGE",
        "outputId": "1b33e897-ff44-4bc8-8acf-83b3c57f3fdd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GenericSVMClassifierModel_5b8f3a210253"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result = clf_Pipeline.fit(data).transform(df)\n",
        "# result.select(\"text\", \"class.result\").show(truncate=100)"
      ],
      "metadata": {
        "id": "sfIEZ4957d-x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result.select(\"text\", \"class.result\").show(truncate=100)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GxpyXRiFATKz",
        "outputId": "53814f44-e41f-4e5d-a12e-ae3994267e51"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------------------------------------------------------------------------------------------+-------+\n",
            "|                                                                                                text| result|\n",
            "+----------------------------------------------------------------------------------------------------+-------+\n",
            "|                       Multicentric canine lymphoma in a 12-year-old keeshond: chemotherapy options.|[False]|\n",
            "|                             Pyomyositis is a rare disease, encountered mainly in tropical climates.|[False]|\n",
            "| Both patients subsequently developed markedly elevated EBV-DNA titers in association with monocl...|[False]|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastrointestinal side effect of this first-in-c...|[False]|\n",
            "|However, given the clinically significant result to the interaction between tolazoline and cimeti...| [True]|\n",
            "|                                              How much do novel antipsychotics benefit the patients?|[False]|\n",
            "| We hypothesize that during interferon therapy, melanocytes may produce more melanin pigment in t...|[False]|\n",
            "|They seemed to involve multiple aetiological factors, such as autoimmune thyroid disease, the tox...|[False]|\n",
            "|               Two days after completing this regimen, the patient developed a rash with blistering.| [True]|\n",
            "|A diagnosis of masked theophylline poisoning should be considered in similar situations involving...|[False]|\n",
            "| The overall response rate of these 24 refractory lymphomas to gemcitabine-containing regimens wa...|[False]|\n",
            "|Development of sarcoidosis during interferon alpha 2b and ribavirin combination therapy for chron...| [True]|\n",
            "|A patient with coccidioidal meningitis was treated with intrathecally administered amphotericin B...|[False]|\n",
            "|                                                Renal failure associated with the use of dextran-40.|[False]|\n",
            "| However, with increased experience in applying BCG, the side effects now appear to be less promi...|[False]|\n",
            "|                        Hepatotoxicity after high-dose methylprednisolone for demyelinating disease.| [True]|\n",
            "| Histopathological findings included signs of orthokeratotic hyperkeratosis, moderate follicular ...| [True]|\n",
            "| Acute spontaneous TLS is rare, and it has been described in leukemia and lymphoma and in some pa...|[False]|\n",
            "|We present a fatal case of subacute methanol toxicity with associated diffuse brain involvement, ...| [True]|\n",
            "| The reaction was thought to be triggered by the combination of radiation and epidermal growth fa...|[False]|\n",
            "+----------------------------------------------------------------------------------------------------+-------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds_df = result.select(\"text\", \"class.result\").toPandas()\n",
        "preds_df['result'] = preds_df['result'].apply(lambda x : True if x[0]=='True' else False)\n",
        "preds_df\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "oa3hvz0uFR3K",
        "outputId": "0d05ae1f-73bc-47e0-d818-87d162cc51a8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                 text  result\n",
              "0    Multicentric canine lymphoma in a 12-year-old...   False\n",
              "1    Pyomyositis is a rare disease, encountered ma...   False\n",
              "2    Both patients subsequently developed markedly...   False\n",
              "3   Bortezomib-induced paralytic ileus is a potent...   False\n",
              "4   However, given the clinically significant resu...    True\n",
              "..                                                ...     ...\n",
              "95   Subsequently he responded to the West Berlin ...   False\n",
              "96   All three patients likely experienced electro...   False\n",
              "97  We report a case of codeine intoxication in th...   False\n",
              "98   METHODS: The clinical suspicion of chickenpox...   False\n",
              "99   Three cases with touch of an intact anterior ...   False\n",
              "\n",
              "[100 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-65168b05-551d-426d-9444-126d1814251b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Multicentric canine lymphoma in a 12-year-old...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Pyomyositis is a rare disease, encountered ma...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Both patients subsequently developed markedly...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Bortezomib-induced paralytic ileus is a potent...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>However, given the clinically significant resu...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Subsequently he responded to the West Berlin ...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>All three patients likely experienced electro...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>We report a case of codeine intoxication in th...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>METHODS: The clinical suspicion of chickenpox...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Three cases with touch of an intact anterior ...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>100 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-65168b05-551d-426d-9444-126d1814251b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-65168b05-551d-426d-9444-126d1814251b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-65168b05-551d-426d-9444-126d1814251b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-d3e73386-50b1-41f3-995e-c80466227f7e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d3e73386-50b1-41f3-995e-c80466227f7e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-d3e73386-50b1-41f3-995e-c80466227f7e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_c3d4cca5-1696-42dc-88aa-ad9d11ad93c0\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('preds_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_c3d4cca5-1696-42dc-88aa-ad9d11ad93c0 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('preds_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import  classification_report\n",
        "print (classification_report(sample_data['category'], preds_df['result']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mQobVH3tD7om",
        "outputId": "0d78ed57-513c-4a49-e494-059dcc96a072"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.85      0.93      0.89        72\n",
            "        True       0.76      0.57      0.65        28\n",
            "\n",
            "    accuracy                           0.83       100\n",
            "   macro avg       0.81      0.75      0.77       100\n",
            "weighted avg       0.82      0.83      0.82       100\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ⏰Running the **GenericSVMClassifierModel** with parameters"
      ],
      "metadata": {
        "id": "TMkd1ewsTfGp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(generic_classifier.explainParams())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TsawmZHQL2dm",
        "outputId": "3ec12c1a-c975-4f84-dc7a-ee0c9fbacea8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "classes: Categorization classes (current: ['False', 'True'])\n",
            "featureScaling: Feature scaling method. Possible values are 'zscore', 'minmax' or empty (no scaling) (default: )\n",
            "inputCols: previous annotations columns, if renamed (current: ['features'])\n",
            "lazyAnnotator: Whether this AnnotatorModel acts as lazy in RecursivePipelines (default: False)\n",
            "multiClass: Whether to return only the label with the highest confidence score or all labels (default: False, current: False)\n",
            "outputCol: output annotation column. can be left default. (current: class)\n",
            "storageRef: storage unique identifier (default: GenericSVMClassifierModel_5b8f3a210253)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶`featureScaling`\n",
        "\n",
        "\n",
        "buraya feature scaling tanim eklenmesi"
      ],
      "metadata": {
        "id": "6ISMg7wug-_0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "generic_classifier = medical.GenericSVMClassifierModel.pretrained(\"generic_svm_classifier_ade\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"features\"])\\\n",
        "    .setOutputCol(\"class\")\\\n",
        "    .setFeatureScaling('zscore')\n",
        "\n",
        "clf_Pipeline2 = nlp.Pipeline(stages=[\n",
        "    document_assembler,\n",
        "    tokenizer,\n",
        "    word_embeddings,\n",
        "    sentence_embeddings,\n",
        "    features_asm,\n",
        "    generic_classifier])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pg3WZhuYpNCP",
        "outputId": "5c1cb1ee-7df4-4f08-8ca5-2e61eadcd615"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "generic_svm_classifier_ade download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result2 = clf_Pipeline2.fit(data).transform(df)\n",
        "result2.select(\"text\", \"class.result\",\"class.metadata\").show(truncate=100)   # zscore values"
      ],
      "metadata": {
        "id": "wH10hjaw_GY0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f771c075-50d3-4298-f744-d09344524257"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                                                                                                text| result|                    metadata|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                       Multicentric canine lymphoma in a 12-year-old keeshond: chemotherapy options.|[False]|[{confidence -> 0.89175177}]|\n",
            "|                             Pyomyositis is a rare disease, encountered mainly in tropical climates.|[False]|[{confidence -> 0.97560674}]|\n",
            "| Both patients subsequently developed markedly elevated EBV-DNA titers in association with monocl...|[False]|[{confidence -> 0.99982864}]|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastrointestinal side effect of this first-in-c...| [True]| [{confidence -> 0.7110752}]|\n",
            "|However, given the clinically significant result to the interaction between tolazoline and cimeti...| [True]| [{confidence -> 0.9998343}]|\n",
            "|                                              How much do novel antipsychotics benefit the patients?|[False]|[{confidence -> 0.99260116}]|\n",
            "| We hypothesize that during interferon therapy, melanocytes may produce more melanin pigment in t...| [True]|[{confidence -> 0.96625036}]|\n",
            "|They seemed to involve multiple aetiological factors, such as autoimmune thyroid disease, the tox...|[False]|[{confidence -> 0.92676556}]|\n",
            "|               Two days after completing this regimen, the patient developed a rash with blistering.| [True]|[{confidence -> 0.99982035}]|\n",
            "|A diagnosis of masked theophylline poisoning should be considered in similar situations involving...| [True]| [{confidence -> 0.7509141}]|\n",
            "| The overall response rate of these 24 refractory lymphomas to gemcitabine-containing regimens wa...|[False]|[{confidence -> 0.99770486}]|\n",
            "|Development of sarcoidosis during interferon alpha 2b and ribavirin combination therapy for chron...| [True]|[{confidence -> 0.99999905}]|\n",
            "|A patient with coccidioidal meningitis was treated with intrathecally administered amphotericin B...| [True]| [{confidence -> 0.9955476}]|\n",
            "|                                                Renal failure associated with the use of dextran-40.| [True]|[{confidence -> 0.99585235}]|\n",
            "| However, with increased experience in applying BCG, the side effects now appear to be less promi...|[False]|[{confidence -> 0.36976963}]|\n",
            "|                        Hepatotoxicity after high-dose methylprednisolone for demyelinating disease.| [True]|[{confidence -> 0.99999976}]|\n",
            "| Histopathological findings included signs of orthokeratotic hyperkeratosis, moderate follicular ...| [True]|[{confidence -> 0.99984574}]|\n",
            "| Acute spontaneous TLS is rare, and it has been described in leukemia and lymphoma and in some pa...|[False]| [{confidence -> 0.8390809}]|\n",
            "|We present a fatal case of subacute methanol toxicity with associated diffuse brain involvement, ...| [True]|[{confidence -> 0.99999976}]|\n",
            "| The reaction was thought to be triggered by the combination of radiation and epidermal growth fa...|[False]|[{confidence -> 0.98358226}]|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result.select(\"text\", \"class.result\",\"class.metadata\").show(truncate=100)  # default values"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zZkYCC8bppNp",
        "outputId": "1bf0abae-dc8f-4db8-e716-b14c6bccc9d9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                                                                                                text| result|                    metadata|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                       Multicentric canine lymphoma in a 12-year-old keeshond: chemotherapy options.|[False]|[{confidence -> 0.89175177}]|\n",
            "|                             Pyomyositis is a rare disease, encountered mainly in tropical climates.|[False]|[{confidence -> 0.97560674}]|\n",
            "| Both patients subsequently developed markedly elevated EBV-DNA titers in association with monocl...|[False]|[{confidence -> 0.99982864}]|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastrointestinal side effect of this first-in-c...| [True]| [{confidence -> 0.7110752}]|\n",
            "|However, given the clinically significant result to the interaction between tolazoline and cimeti...| [True]| [{confidence -> 0.9998343}]|\n",
            "|                                              How much do novel antipsychotics benefit the patients?|[False]|[{confidence -> 0.99260116}]|\n",
            "| We hypothesize that during interferon therapy, melanocytes may produce more melanin pigment in t...| [True]|[{confidence -> 0.96625036}]|\n",
            "|They seemed to involve multiple aetiological factors, such as autoimmune thyroid disease, the tox...|[False]|[{confidence -> 0.92676556}]|\n",
            "|               Two days after completing this regimen, the patient developed a rash with blistering.| [True]|[{confidence -> 0.99982035}]|\n",
            "|A diagnosis of masked theophylline poisoning should be considered in similar situations involving...| [True]| [{confidence -> 0.7509141}]|\n",
            "| The overall response rate of these 24 refractory lymphomas to gemcitabine-containing regimens wa...|[False]|[{confidence -> 0.99770486}]|\n",
            "|Development of sarcoidosis during interferon alpha 2b and ribavirin combination therapy for chron...| [True]|[{confidence -> 0.99999905}]|\n",
            "|A patient with coccidioidal meningitis was treated with intrathecally administered amphotericin B...| [True]| [{confidence -> 0.9955476}]|\n",
            "|                                                Renal failure associated with the use of dextran-40.| [True]|[{confidence -> 0.99585235}]|\n",
            "| However, with increased experience in applying BCG, the side effects now appear to be less promi...|[False]|[{confidence -> 0.36976963}]|\n",
            "|                        Hepatotoxicity after high-dose methylprednisolone for demyelinating disease.| [True]|[{confidence -> 0.99999976}]|\n",
            "| Histopathological findings included signs of orthokeratotic hyperkeratosis, moderate follicular ...| [True]|[{confidence -> 0.99984574}]|\n",
            "| Acute spontaneous TLS is rare, and it has been described in leukemia and lymphoma and in some pa...|[False]| [{confidence -> 0.8390809}]|\n",
            "|We present a fatal case of subacute methanol toxicity with associated diffuse brain involvement, ...| [True]|[{confidence -> 0.99999976}]|\n",
            "| The reaction was thought to be triggered by the combination of radiation and epidermal growth fa...|[False]|[{confidence -> 0.98358226}]|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "generic_classifier = medical.GenericSVMClassifierModel.pretrained(\"generic_svm_classifier_ade\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"features\"])\\\n",
        "    .setOutputCol(\"class\")\\\n",
        "    .setFeatureScaling('minmax')\n",
        "\n",
        "clf_Pipeline3 = nlp.Pipeline(stages=[\n",
        "    document_assembler,\n",
        "    tokenizer,\n",
        "    word_embeddings,\n",
        "    sentence_embeddings,\n",
        "    features_asm,\n",
        "    generic_classifier])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EKXupFq651FV",
        "outputId": "9591ef7c-0e05-4f85-dc96-84a21d0f0119"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "generic_svm_classifier_ade download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result3 = clf_Pipeline3.fit(data).transform(df)\n",
        "result3.select(\"text\", \"class.result\",\"class.metadata\").show(truncate=100)  #minmax values"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N0flA9DcAIV0",
        "outputId": "7b4bb912-a872-4614-c339-31ac38fc2db3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                                                                                                text| result|                    metadata|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "|                       Multicentric canine lymphoma in a 12-year-old keeshond: chemotherapy options.| [True]| [{confidence -> 0.6639472}]|\n",
            "|                             Pyomyositis is a rare disease, encountered mainly in tropical climates.| [True]|[{confidence -> 0.58102447}]|\n",
            "| Both patients subsequently developed markedly elevated EBV-DNA titers in association with monocl...|[False]| [{confidence -> 0.4665953}]|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastrointestinal side effect of this first-in-c...| [True]|[{confidence -> 0.71549857}]|\n",
            "|However, given the clinically significant result to the interaction between tolazoline and cimeti...| [True]|[{confidence -> 0.86733115}]|\n",
            "|                                              How much do novel antipsychotics benefit the patients?| [True]|[{confidence -> 0.65765846}]|\n",
            "| We hypothesize that during interferon therapy, melanocytes may produce more melanin pigment in t...| [True]|[{confidence -> 0.78535503}]|\n",
            "|They seemed to involve multiple aetiological factors, such as autoimmune thyroid disease, the tox...| [True]|[{confidence -> 0.60473394}]|\n",
            "|               Two days after completing this regimen, the patient developed a rash with blistering.| [True]| [{confidence -> 0.8805499}]|\n",
            "|A diagnosis of masked theophylline poisoning should be considered in similar situations involving...| [True]|[{confidence -> 0.75477517}]|\n",
            "| The overall response rate of these 24 refractory lymphomas to gemcitabine-containing regimens wa...| [True]|[{confidence -> 0.54081035}]|\n",
            "|Development of sarcoidosis during interferon alpha 2b and ribavirin combination therapy for chron...| [True]| [{confidence -> 0.9169581}]|\n",
            "|A patient with coccidioidal meningitis was treated with intrathecally administered amphotericin B...| [True]|[{confidence -> 0.82843184}]|\n",
            "|                                                Renal failure associated with the use of dextran-40.| [True]|[{confidence -> 0.82954353}]|\n",
            "| However, with increased experience in applying BCG, the side effects now appear to be less promi...| [True]|[{confidence -> 0.69373775}]|\n",
            "|                        Hepatotoxicity after high-dose methylprednisolone for demyelinating disease.| [True]| [{confidence -> 0.9289645}]|\n",
            "| Histopathological findings included signs of orthokeratotic hyperkeratosis, moderate follicular ...| [True]|[{confidence -> 0.86991197}]|\n",
            "| Acute spontaneous TLS is rare, and it has been described in leukemia and lymphoma and in some pa...| [True]|[{confidence -> 0.63727444}]|\n",
            "|We present a fatal case of subacute methanol toxicity with associated diffuse brain involvement, ...| [True]|[{confidence -> 0.91892475}]|\n",
            "| The reaction was thought to be triggered by the combination of radiation and epidermal growth fa...| [True]|[{confidence -> 0.49183407}]|\n",
            "+----------------------------------------------------------------------------------------------------+-------+----------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As shown above, three different `featureScaling` options generated different results and confidence values, and the best one may be chosen according to problem/dataset."
      ],
      "metadata": {
        "id": "qmUDvsA6_9nZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ▶ `multiClass`"
      ],
      "metadata": {
        "id": "Rq8TxfhQYkhJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Whether to return only the label with the highest confidence score or all labels"
      ],
      "metadata": {
        "id": "pIivTlv0YxKP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "generic_classifier = medical.GenericSVMClassifierModel.pretrained(\"generic_svm_classifier_ade\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"features\"])\\\n",
        "    .setOutputCol(\"class\")\\\n",
        "    .setMultiClass(True)\n",
        "\n",
        "clf_Pipeline4 = nlp.Pipeline(stages=[\n",
        "    document_assembler,\n",
        "    tokenizer,\n",
        "    word_embeddings,\n",
        "    sentence_embeddings,\n",
        "    features_asm,\n",
        "    generic_classifier])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "msQUmBA_YwYq",
        "outputId": "383bef30-b81e-494f-d5cd-d78404af18d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "generic_svm_classifier_ade download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result4 = clf_Pipeline4.fit(data).transform(df)\n",
        "result4.select(\"text\", \"class.result\",\"class.metadata\").show(truncate=100)\n",
        "\n",
        "# below, result column is not one value, it is list of values showing confidence of each classes"
      ],
      "metadata": {
        "id": "AKhh_emSiMXE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ae8389a-76af-4d79-9eaf-35d26933949b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------------------------------------------------------------------------------------------+-------------+--------------------------------------------------------+\n",
            "|                                                                                                text|       result|                                                metadata|\n",
            "+----------------------------------------------------------------------------------------------------+-------------+--------------------------------------------------------+\n",
            "|                       Multicentric canine lymphoma in a 12-year-old keeshond: chemotherapy options.|[False, True]|  [{confidence -> 0.3259712}, {confidence -> 0.6639472}]|\n",
            "|                             Pyomyositis is a rare disease, encountered mainly in tropical climates.|[False, True]|[{confidence -> 0.33836725}, {confidence -> 0.58102447}]|\n",
            "| Both patients subsequently developed markedly elevated EBV-DNA titers in association with monocl...|[False, True]| [{confidence -> 0.4665953}, {confidence -> 0.35843927}]|\n",
            "|Bortezomib-induced paralytic ileus is a potential gastrointestinal side effect of this first-in-c...|[False, True]|[{confidence -> 0.22667688}, {confidence -> 0.71549857}]|\n",
            "|However, given the clinically significant result to the interaction between tolazoline and cimeti...|[False, True]|[{confidence -> 0.12806472}, {confidence -> 0.86733115}]|\n",
            "|                                              How much do novel antipsychotics benefit the patients?|[False, True]|[{confidence -> 0.35222763}, {confidence -> 0.65765846}]|\n",
            "| We hypothesize that during interferon therapy, melanocytes may produce more melanin pigment in t...|[False, True]|[{confidence -> 0.16957161}, {confidence -> 0.78535503}]|\n",
            "|They seemed to involve multiple aetiological factors, such as autoimmune thyroid disease, the tox...|[False, True]|[{confidence -> 0.30494922}, {confidence -> 0.60473394}]|\n",
            "|               Two days after completing this regimen, the patient developed a rash with blistering.|[False, True]| [{confidence -> 0.17356193}, {confidence -> 0.8805499}]|\n",
            "|A diagnosis of masked theophylline poisoning should be considered in similar situations involving...|[False, True]|[{confidence -> 0.23802176}, {confidence -> 0.75477517}]|\n",
            "| The overall response rate of these 24 refractory lymphomas to gemcitabine-containing regimens wa...|[False, True]| [{confidence -> 0.3957434}, {confidence -> 0.54081035}]|\n",
            "|Development of sarcoidosis during interferon alpha 2b and ribavirin combination therapy for chron...|[False, True]| [{confidence -> 0.08160052}, {confidence -> 0.9169581}]|\n",
            "|A patient with coccidioidal meningitis was treated with intrathecally administered amphotericin B...|[False, True]|[{confidence -> 0.16283986}, {confidence -> 0.82843184}]|\n",
            "|                                                Renal failure associated with the use of dextran-40.|[False, True]|[{confidence -> 0.13947591}, {confidence -> 0.82954353}]|\n",
            "| However, with increased experience in applying BCG, the side effects now appear to be less promi...|[False, True]|[{confidence -> 0.23132768}, {confidence -> 0.69373775}]|\n",
            "|                        Hepatotoxicity after high-dose methylprednisolone for demyelinating disease.|[False, True]|[{confidence -> 0.069665134}, {confidence -> 0.9289645}]|\n",
            "| Histopathological findings included signs of orthokeratotic hyperkeratosis, moderate follicular ...|[False, True]|[{confidence -> 0.16104063}, {confidence -> 0.86991197}]|\n",
            "| Acute spontaneous TLS is rare, and it has been described in leukemia and lymphoma and in some pa...|[False, True]|[{confidence -> 0.29963502}, {confidence -> 0.63727444}]|\n",
            "|We present a fatal case of subacute methanol toxicity with associated diffuse brain involvement, ...|[False, True]|[{confidence -> 0.07455444}, {confidence -> 0.91892475}]|\n",
            "| The reaction was thought to be triggered by the combination of radiation and epidermal growth fa...|[False, True]|[{confidence -> 0.33398497}, {confidence -> 0.49183407}]|\n",
            "+----------------------------------------------------------------------------------------------------+-------------+--------------------------------------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    }
  ]
}