Spark_NLP_Udemy_MOOC/Healthcare_NLP/RENerChunksFilter.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49w2L5Yz2acq"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vaMC-Lec2tr9"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# RENerChunksFilter\n",
        "\n",
        "The `RENerChunksFilter` annotator filters NER chunks that contains the desired NER entity pairs only."
      ],
      "metadata": {
        "id": "8JUTRvSmESv7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**📖 Learning Objectives:**\n",
        "\n",
        "1. Understand how to filters desired relation pairs for the `RelationExtractionDLModel`\n",
        "\n",
        "2. Become comfortable using the different parameters of the annotator.\n",
        "\n",
        "**🔗 Helpful Links:**\n",
        "\n",
        "* For extended examples of usage, see the [Spark NLP Workshop](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/healthcare-nlp/03.0.Clinical_Relation_Extraction.ipynb)\n",
        "\n",
        "* Python Documentation: [RENerChunksFilter](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/re/relation_ner_chunk_filter/index.html#sparknlp_jsl.annotator.re.relation_ner_chunk_filter.RENerChunksFilter.name)\n",
        "\n",
        "* Scala Documentation: [RENerChunksFilter](https://nlp.johnsnowlabs.com/licensed/api/com/johnsnowlabs/nlp/annotators/re/RENerChunksFilter.html)\n",
        "\n",
        "* Relation Extraction Models and Relation Pairs Table: [In this link](https://nlp.johnsnowlabs.com/docs/en/best_practices_pretrained_models#relation-extraction-models-and-relation-pairs-table). Contains the available Relation Extraction models, its labels, associated NER model, and meaningful relation pairs to filter."
      ],
      "metadata": {
        "id": "daTdSpZ9dz6K"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **📜 Background**\n"
      ],
      "metadata": {
        "id": "HXgUBncKfnA6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Filtering the possible relations can be useful to perform additional analysis for a specific use case (e.g., checking adverse drug reactions and drug realations), which can be the input for further analysis using a pretrained `RelationExtractionDLModel`.\n",
        "\n",
        "This annotator filters the NER chunks that contains the desired relation pairs, excluding the annotations between other entities."
      ],
      "metadata": {
        "id": "mtq8-Rrrfu1c"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🎬 Colab Setup**"
      ],
      "metadata": {
        "id": "N5ZJ0EthgVOP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install the johnsnowlabs library to access Spark-NLP for Healthcare\n",
        "! pip install -q johnsnowlabs"
      ],
      "metadata": {
        "id": "WbLu60U6fmxJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ddc00f2-c216-44d3-e481-bc556c745288"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.2/126.2 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m310.8/310.8 MB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m547.3/547.3 kB\u001b[0m \u001b[31m36.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m665.7/665.7 kB\u001b[0m \u001b[31m44.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m58.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.3/139.3 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m66.9/66.9 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.0/12.0 MB\u001b[0m \u001b[31m62.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.1/82.1 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m150.3/150.3 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m63.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Upload license keys\n",
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ],
      "metadata": {
        "id": "hwytACt7gY6N",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "ff28b187-5285-4d45-cc97-30290e2eaecb"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Please Upload your John Snow Labs License using the button below\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-ef7f6fd1-a78e-498c-ae7f-c8255af9de92\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-ef7f6fd1-a78e-498c-ae7f-c8255af9de92\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving spark_nlp_for_healthcare_spark_ocr_8734.json to spark_nlp_for_healthcare_spark_ocr_8734.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Install licensed libraries\n",
        "from johnsnowlabs import nlp\n",
        "\n",
        "\n",
        "nlp.install()"
      ],
      "metadata": {
        "id": "4kSfTrnsfmri",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9ea43ff1-1455-4b71-f8c2-8576320a6cc1"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "👌 Detected license file /content/spark_nlp_for_healthcare_spark_ocr_8734.json\n",
            "🚨 Outdated Medical Secrets in license file. Version=5.2.0 but should be Version=5.2.1\n",
            "🚨 Outdated OCR Secrets in license file. Version=5.1.0 but should be Version=5.1.2\n",
            "👷 Trying to install compatible secrets. Use nlp.settings.enforce_versions=False if you want to install outdated secrets.\n",
            "📋 Stored John Snow Labs License in /root/.johnsnowlabs/licenses/license_number_0_for_Spark-Healthcare_Spark-OCR.json\n",
            "👷 Setting up  John Snow Labs home in /root/.johnsnowlabs, this might take a few minutes.\n",
            "Downloading 🐍+🚀 Python Library spark_nlp-5.2.2-py2.py3-none-any.whl\n",
            "Downloading 🐍+💊 Python Library spark_nlp_jsl-5.2.1-py3-none-any.whl\n",
            "Downloading 🫘+🚀 Java Library spark-nlp-assembly-5.2.2.jar\n",
            "Downloading 🫘+💊 Java Library spark-nlp-jsl-5.2.1.jar\n",
            "🙆 JSL Home setup in /root/.johnsnowlabs\n",
            "👌 Detected license file /content/spark_nlp_for_healthcare_spark_ocr_8734.json\n",
            "👷 Trying to install compatible secrets. Use nlp.settings.enforce_versions=False if you want to install outdated secrets.\n",
            "Installing /root/.johnsnowlabs/py_installs/spark_nlp_jsl-5.2.1-py3-none-any.whl to /usr/bin/python3\n",
            "Installed 1 products:\n",
            "💊 Spark-Healthcare==5.2.1 installed! ✅ Heal the planet with NLP! \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from johnsnowlabs import nlp, medical\n",
        "import pandas as pd\n",
        "from pyspark.sql import DataFrame\n",
        "import pyspark.sql.functions as F\n",
        "import pyspark.sql.types as T\n",
        "\n",
        "\n",
        "# Start Spark Session\n",
        "spark = nlp.start()\n",
        "spark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        },
        "id": "-NycGNEzKVDf",
        "outputId": "b9edb2a4-982a-42f4-ca57-1e88b7dd9dbb"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "👌 Detected license file /content/spark_nlp_for_healthcare_spark_ocr_8734.json\n",
            "👷 Trying to install compatible secrets. Use nlp.settings.enforce_versions=False if you want to install outdated secrets.\n",
            "👌 Launched \u001b[92mcpu optimized\u001b[39m session with with: 🚀Spark-NLP==5.2.2, 💊Spark-Healthcare==5.2.1, running on ⚡ PySpark==3.4.0\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7b300f115960>"
            ],
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://f3b3ef796f66:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.4.0</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>John-Snow-Labs-Spark-Session 🚀 with Jars for: 🚀Spark-NLP==5.2.2, 💊Spark-Healthcare==5.2.1, running on ⚡ PySpark==3.4.0</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🖨️ Input/Output Annotation Types**\n",
        "- Input: `CHUNK, DEPENDENCY`\n",
        "- Output: `CHUNK`"
      ],
      "metadata": {
        "id": "KJuds52Dgvtm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **🔎 Parameters**\n"
      ],
      "metadata": {
        "id": "gs-wO_iMhE42"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- `maxSyntacticDistance` *(Int)*: Maximum syntactic distance between a pair of named entities to consider them as a relation. Increasing this value will increase recall, but also increase the number of false positives.\n",
        "\n",
        "- `relationPairs` *(List[Str])*: List of dash-separated pairs of named entities. For example, [“Biomarker-RelativeDay”] will process all relations between entities of type “Biomarker” and “RelativeDay”.\n",
        "\n",
        "- `relationPairsCaseSensitive` *(Boolean)*: Determines whether relation pairs are case sensitive.\n"
      ],
      "metadata": {
        "id": "LZ5DcZnNhned"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### `setMaxSyntacticDistance`"
      ],
      "metadata": {
        "id": "4iZXmY3iNkEN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This parameter is used for setting the maximal syntactic distance, as threshold.\n",
        "\n",
        "The precision of the RE model is controlled by \"maxSyntacticDistance\", which sets the maximum syntactic distance between named entities, controling the pairs to be classified. A larger value will improve recall at the expense at lower precision."
      ],
      "metadata": {
        "id": "qNhD1W-ABmFz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's compare the result with different values on an sample text.\n",
        "\n",
        "First, we set the `.maxSyntacticDistance` to `10`."
      ],
      "metadata": {
        "id": "bPuN78mjNwjm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "documenter = nlp.DocumentAssembler()\\\n",
        "    .setInputCol(\"text\")\\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "sentencer = nlp.SentenceDetector()\\\n",
        "    .setInputCols([\"document\"])\\\n",
        "    .setOutputCol(\"sentences\")\n",
        "\n",
        "tokenizer = nlp.Tokenizer()\\\n",
        "    .setInputCols([\"sentences\"])\\\n",
        "    .setOutputCol(\"tokens\")\n",
        "\n",
        "words_embedder = nlp.WordEmbeddingsModel()\\\n",
        "    .pretrained(\"embeddings_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"sentences\", \"tokens\"])\\\n",
        "    .setOutputCol(\"embeddings\")\n",
        "\n",
        "pos_tagger = nlp.PerceptronModel()\\\n",
        "    .pretrained(\"pos_clinical\", \"en\", \"clinical/models\") \\\n",
        "    .setInputCols([\"sentences\", \"tokens\"])\\\n",
        "    .setOutputCol(\"pos_tags\")\n",
        "\n",
        "ner_tagger = medical.NerModel.pretrained(\"ner_ade_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols(\"sentences\", \"tokens\", \"embeddings\")\\\n",
        "    .setOutputCol(\"ner_tags\")\n",
        "\n",
        "ner_chunker = medical.NerConverter()\\\n",
        "    .setInputCols([\"sentences\", \"tokens\", \"ner_tags\"])\\\n",
        "    .setOutputCol(\"ner_chunks\")\n",
        "\n",
        "dependency_parser = nlp.DependencyParserModel()\\\n",
        "    .pretrained(\"dependency_conllu\", \"en\")\\\n",
        "    .setInputCols([\"sentences\", \"pos_tags\", \"tokens\"])\\\n",
        "    .setOutputCol(\"dependencies\")\n",
        "\n",
        "ade_re_ner_chunk_filter = medical.RENerChunksFilter() \\\n",
        "    .setInputCols([\"ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(10)\n",
        "\n",
        "ade_re_model = medical.RelationExtractionDLModel()\\\n",
        "    .pretrained('redl_ade_biobert', 'en', \"clinical/models\") \\\n",
        "    .setInputCols([\"re_ner_chunks\", \"sentences\"]) \\\n",
        "    .setPredictionThreshold(0.5)\\\n",
        "    .setOutputCol(\"relations\")\n",
        "\n",
        "pipeline = nlp.Pipeline(stages=[\n",
        "    documenter,\n",
        "    sentencer,\n",
        "    tokenizer,\n",
        "    words_embedder,\n",
        "    pos_tagger,\n",
        "    ner_tagger,\n",
        "    ner_chunker,\n",
        "    dependency_parser,\n",
        "    ade_re_ner_chunk_filter,\n",
        "    ade_re_model\n",
        "])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U4jo1TccMBvS",
        "outputId": "37015b06-ca4c-45ff-f72a-481d16024c68"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "embeddings_clinical download started this may take some time.\n",
            "Approximate size to download 1.6 GB\n",
            "[OK!]\n",
            "pos_clinical download started this may take some time.\n",
            "Approximate size to download 1.5 MB\n",
            "[OK!]\n",
            "ner_ade_clinical download started this may take some time.\n",
            "[OK!]\n",
            "dependency_conllu download started this may take some time.\n",
            "Approximate size to download 16.7 MB\n",
            "[OK!]\n",
            "redl_ade_biobert download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "example =\"\"\"A 44-year-old man taking naproxen for chronic low back pain and a 20-year-old woman on oxaprozin for rheumatoid arthritis presented with tense bullae and cutaneous fragility on the face and the back of the hands.\"\"\"\n",
        "\n",
        "spark_df = spark.createDataFrame([[example]]).toDF(\"text\")\n",
        "\n",
        "results = model.transform(spark_df)"
      ],
      "metadata": {
        "id": "4Ra3_KXbOhLB"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results.select(\n",
        "    F.explode(F.arrays_zip(results.relations.result, results.relations.metadata)).alias(\"cols\")\n",
        ").select(\n",
        "    F.expr(\"cols['0']\").alias(\"relation\"),\n",
        "    F.expr(\"cols['1'].chunk1\").alias(\"chunk1\"),\n",
        "    F.expr(\"cols['1'].entity1\").alias(\"label1\"),\n",
        "    F.expr(\"cols['1'].chunk2\").alias(\"chunk2\"),\n",
        "    F.expr(\"cols['1'].entity2\").alias(\"label2\"),\n",
        "    F.expr(\"cols['1'].syntactic_distance \").alias(\"syntactic_distance \")\n",
        ").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "78AlSeZpOn8W",
        "outputId": "a003b09d-40cf-4351-98d9-bd04ad4296a5"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "|relation|      chunk1|label1|              chunk2|label2|syntactic_distance |\n",
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "|       1|    naproxen|  DRUG|           oxaprozin|  DRUG|                  3|\n",
            "|       1|    naproxen|  DRUG|        tense bullae|   ADE|                  6|\n",
            "|       1|    naproxen|  DRUG|cutaneous fragili...|   ADE|                  7|\n",
            "|       1|   oxaprozin|  DRUG|        tense bullae|   ADE|                  3|\n",
            "|       1|   oxaprozin|  DRUG|cutaneous fragili...|   ADE|                  4|\n",
            "|       1|tense bullae|   ADE|cutaneous fragili...|   ADE|                  3|\n",
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, we will set the `maxSyntacticDistance` to `4` and see the difference from the previous results."
      ],
      "metadata": {
        "id": "TW7RcHVb2Tps"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ade_re_ner_chunk_filter = medical.RENerChunksFilter() \\\n",
        "    .setInputCols([\"ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(4)\n",
        "\n",
        "pipeline = nlp.Pipeline(stages=[\n",
        "    documenter,\n",
        "    sentencer,\n",
        "    tokenizer,\n",
        "    words_embedder,\n",
        "    pos_tagger,\n",
        "    ner_tagger,\n",
        "    ner_chunker,\n",
        "    dependency_parser,\n",
        "    ade_re_ner_chunk_filter,\n",
        "    ade_re_model\n",
        "])\n",
        "\n",
        "model = pipeline.fit(empty_data)"
      ],
      "metadata": {
        "id": "8Zj-PVSq2TSC"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results = model.transform(spark_df)"
      ],
      "metadata": {
        "id": "5P3JNE2S2l6O"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results.select(\n",
        "    F.explode(F.arrays_zip(results.relations.result, results.relations.metadata)).alias(\"cols\")\n",
        ").select(\n",
        "    F.expr(\"cols['0']\").alias(\"relation\"),\n",
        "    F.expr(\"cols['1'].chunk1\").alias(\"chunk1\"),\n",
        "    F.expr(\"cols['1'].entity1\").alias(\"label1\"),\n",
        "    F.expr(\"cols['1'].chunk2\").alias(\"chunk2\"),\n",
        "    F.expr(\"cols['1'].entity2\").alias(\"label2\"),\n",
        "    F.expr(\"cols['1'].syntactic_distance \").alias(\"syntactic_distance \")\n",
        ").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "38oQZVn1GRBK",
        "outputId": "71f2a86a-6039-410a-aeda-ff487aa81f19"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "|relation|      chunk1|label1|              chunk2|label2|syntactic_distance |\n",
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "|       1|    naproxen|  DRUG|           oxaprozin|  DRUG|                  3|\n",
            "|       1|   oxaprozin|  DRUG|        tense bullae|   ADE|                  3|\n",
            "|       1|   oxaprozin|  DRUG|cutaneous fragili...|   ADE|                  4|\n",
            "|       1|tense bullae|   ADE|cutaneous fragili...|   ADE|                  3|\n",
            "+--------+------------+------+--------------------+------+-------------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that the relations with syntactic distance greater than 4 were removed from the results. In this case, as we had two entities close to each other, having a higher maximum syntactic distance was creating false positives, and when controlling this parameter we were able to obtain the expected result."
      ],
      "metadata": {
        "id": "pEULKQIuGTPQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### `setRelationPairs`\n"
      ],
      "metadata": {
        "id": "08YO69T6PJw6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This parameter is used for setting the list of dash-separated pairs of named entities. For example, `[\"drug-ade\"]` will process all relations between entities of type \"drug\" and \"ade\".\n",
        "\n",
        "Now, we will set `setRelationPairs([\"OCCURRENCE-DURATION\"])` parameter and expect only \"OCCURRENCE-DURATION\" and \"DURATION-OCCURENCE\" relations."
      ],
      "metadata": {
        "id": "4zG78d8sGo5u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ner_tagger_clinical = medical.NerModel.pretrained(\"ner_events_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols(\"sentences\", \"tokens\", \"embeddings\")\\\n",
        "    .setOutputCol(\"ner_tags\")\n",
        "\n",
        "clinical_re_ner_chunk_filter = medical.RENerChunksFilter() \\\n",
        "    .setInputCols([\"ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(10)\\\n",
        "    .setRelationPairs([\"OCCURRENCE-DURATION\"])\n",
        "\n",
        "clinical_re_model = medical.RelationExtractionDLModel()\\\n",
        "    .pretrained('redl_temporal_events_biobert', 'en', \"clinical/models\") \\\n",
        "    .setInputCols([\"re_ner_chunks\", \"sentences\"]) \\\n",
        "    .setPredictionThreshold(0.5)\\\n",
        "    .setOutputCol(\"relations\")\n",
        "\n",
        "\n",
        "pipeline = nlp.Pipeline(stages=[\n",
        "    documenter,\n",
        "    sentencer,\n",
        "    tokenizer,\n",
        "    words_embedder,\n",
        "    pos_tagger,\n",
        "    ner_tagger_clinical,\n",
        "    ner_chunker,\n",
        "    dependency_parser,\n",
        "    clinical_re_ner_chunk_filter,\n",
        "    clinical_re_model\n",
        "])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)"
      ],
      "metadata": {
        "id": "nUArSQoxO6aC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0920570b-b685-4cd1-aff3-b24c525fe6ae"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ner_events_clinical download started this may take some time.\n",
            "[OK!]\n",
            "redl_temporal_events_biobert download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "example = \"\"\"The patient is a 68-year-old Caucasian male with past medical history of diabetes mellitus.\n",
        "He was doing fairly well until last week while mowing the lawn, he injured his right foot. \"\"\"\n",
        "\n",
        "spark_df = spark.createDataFrame([[example]]).toDF(\"text\")\n",
        "results = model.transform(spark_df)"
      ],
      "metadata": {
        "id": "pvW1430IQRwN"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets show the OCCURRENCE-DURATION relations by using pandas dataframe."
      ],
      "metadata": {
        "id": "sb72JshKQXKh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "results.select(\n",
        "    F.explode(F.arrays_zip(results.relations.result, results.relations.metadata)).alias(\"cols\")\n",
        ").select(\n",
        "    F.expr(\"cols['0']\").alias(\"relation\"),\n",
        "    F.expr(\"cols['1'].chunk1\").alias(\"chunk1\"),\n",
        "    F.expr(\"cols['1'].entity1\").alias(\"label1\"),\n",
        "    F.expr(\"cols['1'].chunk2\").alias(\"chunk2\"),\n",
        "    F.expr(\"cols['1'].entity2\").alias(\"label2\"),\n",
        "    F.expr(\"cols['1'].syntactic_distance \").alias(\"syntactic_distance \")\n",
        ").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ypOWB5oWQV5A",
        "outputId": "a3bb5fff-fa68-4058-eafd-986a3cd8f3eb"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+-----------------+----------+---------------+----------+-------------------+\n",
            "|relation|           chunk1|    label1|         chunk2|    label2|syntactic_distance |\n",
            "+--------+-----------------+----------+---------------+----------+-------------------+\n",
            "|  BEFORE|doing fairly well|OCCURRENCE|      last week|  DURATION|                  3|\n",
            "| OVERLAP|        last week|  DURATION|mowing the lawn|OCCURRENCE|                  3|\n",
            "+--------+-----------------+----------+---------------+----------+-------------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As seen above, we only have \"OCCURRENCE-DURATION\" relations."
      ],
      "metadata": {
        "id": "ZbnzxOCB4c3u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### `setRelationPairsCaseSensitive`"
      ],
      "metadata": {
        "id": "DEQwOHECQgOw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This parameter is set to determine whether relation pairs are case sensitive (Default: `False`).\n",
        "\n",
        "\n",
        "In this example, the NER model contains `ADE` and `DRUG` entities (uppercased). We first set `relationPairs` to `[\"drug-ade, ade-drug\"]` (lowercased) and `relationPairsCaseSensitive` to `True`."
      ],
      "metadata": {
        "id": "VVGjDWU4QtY9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ade_re_ner_chunk_filter = medical.RENerChunksFilter() \\\n",
        "    .setInputCols([\"ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(10)\\\n",
        "    .setRelationPairs([\"drug-ade, ade-drug\"])\\\n",
        "    .setRelationPairsCaseSensitive(True)\n",
        "\n",
        "pipeline = nlp.Pipeline(stages=[\n",
        "    documenter,\n",
        "    sentencer,\n",
        "    tokenizer,\n",
        "    words_embedder,\n",
        "    pos_tagger,\n",
        "    ner_tagger,\n",
        "    ner_chunker,\n",
        "    dependency_parser,\n",
        "    ade_re_ner_chunk_filter,\n",
        "    ade_re_model\n",
        "])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)"
      ],
      "metadata": {
        "id": "e_Em2-a24qTR"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example = \"\"\"A 44-year-old man taking naproxen for chronic low back pain and a 20-year-old woman on oxaprozin for rheumatoid arthritis presented with tense bullae and cutaneous fragility on the face and the back of the hands.\"\"\"\n",
        "\n",
        "spark_df = spark.createDataFrame([[example]]).toDF(\"text\")\n",
        "\n",
        "results = model.transform(spark_df)"
      ],
      "metadata": {
        "id": "MSGvvm_tRqKk"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results.select(\n",
        "    F.explode(F.arrays_zip(results.relations.result, results.relations.metadata)).alias(\"cols\")\n",
        ").select(\n",
        "    F.expr(\"cols['0']\").alias(\"relation\"),\n",
        "    F.expr(\"cols['1'].chunk1\").alias(\"chunk1\"),\n",
        "    F.expr(\"cols['1'].entity1\").alias(\"label1\"),\n",
        "    F.expr(\"cols['1'].chunk2\").alias(\"chunk2\"),\n",
        "    F.expr(\"cols['1'].entity2\").alias(\"label2\"),\n",
        "    F.expr(\"cols['1'].syntactic_distance \").alias(\"syntactic_distance \")\n",
        ").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "px2W6YnERqEZ",
        "outputId": "a53afd0a-0fe0-4c35-d6df-259cc0943b9c"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+------+------+------+------+-------------------+\n",
            "|relation|chunk1|label1|chunk2|label2|syntactic_distance |\n",
            "+--------+------+------+------+------+-------------------+\n",
            "+--------+------+------+------+------+-------------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As seen above, no relation was identified by the model because of the mismatch between the relation paris and the NER entities."
      ],
      "metadata": {
        "id": "SkIrPh8FS4Jg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will set `relationPairsCaseSensitive` to `False` and expect to see relations."
      ],
      "metadata": {
        "id": "kxCzCQg0pnPQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ade_re_ner_chunk_filter = medical.RENerChunksFilter() \\\n",
        "    .setInputCols([\"ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(10)\\\n",
        "    .setRelationPairs([\"drug-ade, ade-drug\"])\\\n",
        "    .setRelationPairsCaseSensitive(False)\n",
        "\n",
        "pipeline = nlp.Pipeline(stages=[\n",
        "    documenter,\n",
        "    sentencer,\n",
        "    tokenizer,\n",
        "    words_embedder,\n",
        "    pos_tagger,\n",
        "    ner_tagger,\n",
        "    ner_chunker,\n",
        "    dependency_parser,\n",
        "    ade_re_ner_chunk_filter,\n",
        "    ade_re_model\n",
        "])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)"
      ],
      "metadata": {
        "id": "7MFz10Idpkwl"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results = model.transform(spark_df)"
      ],
      "metadata": {
        "id": "9bhQvOtBpj3q"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results.select(\n",
        "    F.explode(F.arrays_zip(results.relations.result, results.relations.metadata)).alias(\"cols\")\n",
        ").select(\n",
        "    F.expr(\"cols['0']\").alias(\"relation\"),\n",
        "    F.expr(\"cols['1'].chunk1\").alias(\"chunk1\"),\n",
        "    F.expr(\"cols['1'].entity1\").alias(\"label1\"),\n",
        "    F.expr(\"cols['1'].chunk2\").alias(\"chunk2\"),\n",
        "    F.expr(\"cols['1'].entity2\").alias(\"label2\"),\n",
        "    F.expr(\"cols['1'].syntactic_distance \").alias(\"syntactic_distance \")\n",
        ").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fc0d8XDRqH-k",
        "outputId": "7647defb-1262-4d92-b950-3c3116c72a71"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+---------+------+--------------------+------+-------------------+\n",
            "|relation|   chunk1|label1|              chunk2|label2|syntactic_distance |\n",
            "+--------+---------+------+--------------------+------+-------------------+\n",
            "|       1| naproxen|  DRUG|        tense bullae|   ADE|                  6|\n",
            "|       1| naproxen|  DRUG|cutaneous fragili...|   ADE|                  7|\n",
            "|       1|oxaprozin|  DRUG|        tense bullae|   ADE|                  3|\n",
            "|       1|oxaprozin|  DRUG|cutaneous fragili...|   ADE|                  4|\n",
            "+--------+---------+------+--------------------+------+-------------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "kJZI9g2AIdRN"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}