Spark_NLP_Udemy_MOOC/Healthcare_NLP/AssertionFilterer.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"sXatvRX899i0"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"]},{"cell_type":"markdown","metadata":{"id":"AOn8d1tcBkK3"},"source":["# **AssertionFilterer**"]},{"cell_type":"markdown","source":["This notebook will cover the different parameters and usages of `AssertionFilterer`. This annotator allows to train an AssertionDLModel.\n","\n","**📖 Learning Objectives:**\n","\n","1. Understand the meaning and use of assertion status.\n","\n","2. Learn how to create a chunk column with metadata for training assertion status detection models.\n","\n","3. Customize your assertion model by using the different parameters of the annotator.\n","\n","\n","**🔗 Helpful Links:**\n","\n","- Documentation : [AssertionFilterer](https://nlp.johnsnowlabs.com/docs/en/licensed_annotators#assertionfilterer)\n","\n","- Python Docs : [AssertionFilterer](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/chunker/assertion_filterer/)\n","\n","- Scala Docs : ?? (link broken)\n","\n","- For extended examples of usage, see the [Spark NLP Workshop repository](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/2.Clinical_Assertion_Model.ipynb)."],"metadata":{"id":"WimihSwD3vtH"}},{"cell_type":"markdown","source":["## **📜 Background**\n"],"metadata":{"id":"qL9lcISyFSLv"}},{"cell_type":"markdown","source":["The goal of assertion models is to classify chunks of text considering their context. The typical example of assertion status detection is negation identification: in the sentence “the patient has no history of diabetes”, the chunk “diabetes” -extracted by a clinical NER model as a Disease- would be classified as Absent by an assertion model due to the word \"no\" in its context. A more complex assertion model can include other labels such as Hypothetical, Past, Planned, Possible, Family, etc.\n","\n","The deep neural network architecture for assertion status detection in Spark NLP is based on a Bi-LSTM framework, and is a modified version of the architecture proposed by Federico Fancellu, Adam Lopez and Bonnie Webber (Neural Networks For Negation Scope Detection.\n","\n","\n","**AssertionFilterer** will allow you to filter out the named entities by the list of acceptable assertion statuses. This annotator would be quite handy if you want to set a white list for the acceptable assertion statuses like present or conditional; and do not want absent conditions get out of your pipeline."],"metadata":{"id":"rlHsEmCBkn5c"}},{"cell_type":"markdown","metadata":{"id":"MfkkKkbVF309"},"source":["## **🎬 Colab Setup**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iMkMQtZNF2n-"},"outputs":[],"source":["!pip install -q johnsnowlabs"]},{"cell_type":"code","source":["from google.colab import files\n","print('Please Upload your John Snow Labs License using the button below')\n","license_keys = files.upload()"],"metadata":{"id":"NulWi4_f4GN5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp\n","\n","nlp.install()"],"metadata":{"id":"1Yf9hK_wwe-r"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp, medical\n","import pyspark.sql.functions as F\n","import pandas as pd\n","\n","spark = nlp.start()"],"metadata":{"id":"2GLXxe1Q0Iln","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687843622453,"user_tz":-120,"elapsed":9891,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"1b384bdc-86db-430b-ddf7-0b0df4f530fe"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["👌 Detected license file /content/4.4.3.spark_nlp_for_healthcare (1).json\n","👌 Launched \u001b[92mcpu optimized\u001b[39m session with with: 🚀Spark-NLP==4.4.1, 💊Spark-Healthcare==4.4.3, running on ⚡ PySpark==3.1.2\n"]}]},{"cell_type":"markdown","source":["## **🖨️ Input/Output Annotation Types**"],"metadata":{"id":"9Fbbk1bqcuA5"}},{"cell_type":"markdown","source":["- Input: `DOCUMENT`, `CHUNK`, `ASSERTION`\n","\n","- Output: `CHUNK`"],"metadata":{"id":"0yFIrr5acsiU"}},{"cell_type":"markdown","source":["## **🔎 AssertionFilterer Parameters**\n"],"metadata":{"id":"b2YJehUKMhb0"}},{"cell_type":"markdown","source":["- `whiteList`: (list) If defined, list of entities to process. The rest will be ignored.\n","\n","- `CaseSensitive`: (bool) Determines whether the definitions of the white listed entities are case sensitive.\n","\n","- `regex`: (list) List of dash-separated pairs of named entities.\n","\n","- `criteria`: (list)  Set tag representing what is the criteria to filter the chunks. possibles values (assertion|isIn|regex). *assertion*: Filter by the assertion *isIn* : Filter by the chunk *regex* : Filter using a regex\n","\n","- `entitiesConfidence`: (Str) Entity pairs to remove based on the confidence level.\n"],"metadata":{"id":"7ttYzdjwm65k"}},{"cell_type":"code","source":["documentAssembler = nlp.DocumentAssembler()\\\n","    .setInputCol(\"text\")\\\n","    .setOutputCol(\"document\")\n","\n","sentenceDetector = nlp.SentenceDetector()\\\n","    .setInputCols([\"document\"])\\\n","    .setOutputCol(\"sentence\")\n","\n","tokenizer = nlp.Tokenizer()\\\n","    .setInputCols([\"sentence\"])\\\n","    .setOutputCol(\"token\")\n","\n","word_embeddings = nlp.WordEmbeddingsModel.pretrained(\"embeddings_clinical\", \"en\", \"clinical/models\")\\\n","    .setInputCols([\"sentence\", \"token\"])\\\n","    .setOutputCol(\"embeddings\")\n","\n","clinical_ner = medical.NerModel.pretrained(\"ner_clinical\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentence\", \"token\", \"embeddings\"]) \\\n","    .setOutputCol(\"ner\")\n","\n","ner_converter = medical.NerConverterInternal() \\\n","    .setInputCols([\"sentence\", \"token\", \"ner\"]) \\\n","    .setOutputCol(\"ner_chunk\")\n","\n","clinical_assertion = medical.AssertionDLModel.pretrained(\"assertion_jsl_augmented\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentence\", \"ner_chunk\", \"embeddings\"]) \\\n","    .setOutputCol(\"assertion\")\n","\n","assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(False)\\\n","    .setWhiteList([\"Present\", \"Planned\", \"Possible\"])\n","\n","nlpPipeline = nlp.Pipeline(stages=[\n","      documentAssembler,\n","      sentenceDetector,\n","      tokenizer,\n","      word_embeddings,\n","      clinical_ner,\n","      ner_converter,\n","      clinical_assertion,\n","      assertion_filterer\n","    ])\n","\n","assertionFilter_model = nlpPipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"yS_r3av3I_Gv","executionInfo":{"status":"ok","timestamp":1687843937155,"user_tz":-120,"elapsed":8525,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"814cde09-4503-46fe-fdae-c0d44a05e81c"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["embeddings_clinical download started this may take some time.\n","Approximate size to download 1.6 GB\n","[OK!]\n","ner_clinical download started this may take some time.\n","[OK!]\n","assertion_jsl_augmented download started this may take some time.\n","[OK!]\n"]}]},{"cell_type":"code","source":["text = 'Patient has a headache for the last 2 weeks, needs to get a head CT, and appears anxious when she walks fast. Alopecia noted. She denies pain.'\n","\n","light_model = nlp.LightPipeline(assertionFilter_model)\n","light_result = light_model.annotate(text)\n","\n","light_result.keys()"],"metadata":{"id":"Bxsotq2prb0o","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687843937615,"user_tz":-120,"elapsed":464,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"ce190aa5-e220-4b18-f427-d9251ad4b9d7"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["dict_keys(['assertion_filtered', 'document', 'ner_chunk', 'assertion', 'token', 'ner', 'embeddings', 'sentence'])"]},"metadata":{},"execution_count":21}]},{"cell_type":"code","source":["list(zip(light_result['ner_chunk'], light_result['assertion']))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"chw7KaKSz62b","executionInfo":{"status":"ok","timestamp":1687843937616,"user_tz":-120,"elapsed":9,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"c12f75b6-0a96-449b-bf4c-9d45aefa5fd4"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[('a headache', 'Present'),\n"," ('a head CT', 'Planned'),\n"," ('anxious', 'Possible'),\n"," ('Alopecia', 'Present'),\n"," ('pain', 'Absent')]"]},"metadata":{},"execution_count":22}]},{"cell_type":"code","source":["chunks=[]\n","entities=[]\n","status=[]\n","confidence=[]\n","\n","light_result = light_model.fullAnnotate(text)[0]\n","\n","for m in light_result['assertion_filtered']:\n","\n","    chunks.append(m.result)\n","    entities.append(m.metadata['entity'])\n","    status.append(m.metadata['assertion'])\n","    confidence.append(m.metadata['confidence'])\n","\n","df = pd.DataFrame({'chunks':chunks, 'entities':entities, 'assertion':status, 'confidence':confidence})\n","\n","df"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":174},"id":"2BD9FFlH0Bu0","executionInfo":{"status":"ok","timestamp":1687843938974,"user_tz":-120,"elapsed":399,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"31c4bafd-42f9-4c2d-8c43-f887ca7e70d7"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["       chunks entities assertion  confidence\n","0  a headache  PROBLEM   Present  0.97150004\n","1   a head CT     TEST   Planned      0.8149\n","2     anxious  PROBLEM  Possible      0.9769\n","3    Alopecia  PROBLEM   Present      0.9949"],"text/html":["\n","  <div id=\"df-8c1e099c-181b-4164-8349-9ea3e41a8881\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>chunks</th>\n","      <th>entities</th>\n","      <th>assertion</th>\n","      <th>confidence</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>a headache</td>\n","      <td>PROBLEM</td>\n","      <td>Present</td>\n","      <td>0.97150004</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>a head CT</td>\n","      <td>TEST</td>\n","      <td>Planned</td>\n","      <td>0.8149</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>anxious</td>\n","      <td>PROBLEM</td>\n","      <td>Possible</td>\n","      <td>0.9769</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Alopecia</td>\n","      <td>PROBLEM</td>\n","      <td>Present</td>\n","      <td>0.9949</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8c1e099c-181b-4164-8349-9ea3e41a8881')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-8c1e099c-181b-4164-8349-9ea3e41a8881 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-8c1e099c-181b-4164-8349-9ea3e41a8881');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":23}]},{"cell_type":"markdown","source":["## setWhiteList()"],"metadata":{"id":"n58ZcgM8V3WW"}},{"cell_type":"code","source":["assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(True)\\\n","    .setWhiteList([\"Present\"])\n","\n","nlpPipeline = nlp.Pipeline(stages=[\n","      documentAssembler,\n","      sentenceDetector,\n","      tokenizer,\n","      word_embeddings,\n","      clinical_ner,\n","      ner_converter,\n","      clinical_assertion,\n","      assertion_filterer\n","    ])\n","\n","assertionFilter_model = nlpPipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))\n","\n","light_model = nlp.LightPipeline(assertionFilter_model)\n","light_result = light_model.annotate(text)"],"metadata":{"id":"TXvzFMIH0Dz1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["chunks=[]\n","entities=[]\n","status=[]\n","confidence=[]\n","\n","light_result = light_model.fullAnnotate(text)[0]\n","\n","for m in light_result['assertion_filtered']:\n","\n","    chunks.append(m.result)\n","    entities.append(m.metadata['entity'])\n","    status.append(m.metadata['assertion'])\n","    confidence.append(m.metadata['confidence'])\n","\n","df = pd.DataFrame({'chunks':chunks, 'entities':entities, 'assertion':status, 'confidence':confidence})\n","\n","df"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"id":"2Yhzy4sM1Zrc","executionInfo":{"status":"ok","timestamp":1687844071906,"user_tz":-120,"elapsed":537,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"9719ea3d-5f62-4388-92f1-9c0761b4cb18"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["       chunks entities assertion  confidence\n","0  a headache  PROBLEM   Present  0.97150004\n","1    Alopecia  PROBLEM   Present      0.9949"],"text/html":["\n","  <div id=\"df-e2fcfa0d-895d-405f-8059-b372044ac416\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>chunks</th>\n","      <th>entities</th>\n","      <th>assertion</th>\n","      <th>confidence</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>a headache</td>\n","      <td>PROBLEM</td>\n","      <td>Present</td>\n","      <td>0.97150004</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Alopecia</td>\n","      <td>PROBLEM</td>\n","      <td>Present</td>\n","      <td>0.9949</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e2fcfa0d-895d-405f-8059-b372044ac416')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e2fcfa0d-895d-405f-8059-b372044ac416 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e2fcfa0d-895d-405f-8059-b372044ac416');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":25}]},{"cell_type":"markdown","source":["As you can see, there are no \"pain, head CT, anxious\" parts in the whitelist as there is only the \"Present\" assertion label.\n","\n","\n","\n"],"metadata":{"id":"JRT-_A6J0srC"}},{"cell_type":"markdown","source":["## setCaseSensitive()"],"metadata":{"id":"ZcpPyvNcWxk9"}},{"cell_type":"code","source":["assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(True)\\\n","    .setWhiteList([\"PRESENT\"])\n","\n","nlpPipeline = nlp.Pipeline(stages=[\n","      documentAssembler,\n","      sentenceDetector,\n","      tokenizer,\n","      word_embeddings,\n","      clinical_ner,\n","      ner_converter,\n","      clinical_assertion,\n","      assertion_filterer\n","    ])\n","\n","assertionFilter_model = nlpPipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))\n","\n","light_model = nlp.LightPipeline(assertionFilter_model)\n","light_result = light_model.annotate(text)"],"metadata":{"id":"n9_MzVtuW3xk"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["light_result['assertion_filtered']\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"40ZE-du4XDCX","executionInfo":{"status":"ok","timestamp":1687851501841,"user_tz":-120,"elapsed":6,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"07044541-f156-4cb5-ffb6-1510548aa7a5"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[]"]},"metadata":{},"execution_count":57}]},{"cell_type":"markdown","source":["As you can observe, enabling the **setCaseSensitive()** parameter to True would hinder our ability to identify the relevant assertion within the **setWhiteList()** entity due to case sensitivity errors. Consequently, to ensure greater flexibility, the **setCaseSensitive()** parameter is set to its default value of False."],"metadata":{"id":"kkmPhWxl2ncC"}},{"cell_type":"markdown","source":["## .setCriteria(\"isin\")"],"metadata":{"id":"nS4U-s_qXSB9"}},{"cell_type":"code","source":["assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(False)\\\n","    .setWhiteList([\"Alopecia\", \"a headache\"])\\\n","    .setCriteria(\"isin\")\n","\n","\n","nlpPipeline = nlp.Pipeline(stages=[\n","      documentAssembler,\n","      sentenceDetector,\n","      tokenizer,\n","      word_embeddings,\n","      clinical_ner,\n","      ner_converter,\n","      clinical_assertion,\n","      assertion_filterer\n","    ])\n","\n","assertionFilter_model = nlpPipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))\n","\n","light_model = nlp.LightPipeline(assertionFilter_model)\n","light_result = light_model.annotate(text)"],"metadata":{"id":"v14PpS3P2oNa"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["light_result['ner_chunk']"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-DUKb3_Kw4Tq","executionInfo":{"status":"ok","timestamp":1687851133815,"user_tz":-120,"elapsed":6,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"8a279c71-97ab-4613-ecbe-4480342cf952"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['a headache', 'a head CT', 'anxious', 'Alopecia', 'pain']"]},"metadata":{},"execution_count":48}]},{"cell_type":"code","source":["light_result['assertion_filtered']"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fe-AJq-2xDSo","executionInfo":{"status":"ok","timestamp":1687851133815,"user_tz":-120,"elapsed":5,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"47fcf088-395f-408f-aa5b-31c001015795"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['a headache', 'Alopecia']"]},"metadata":{},"execution_count":49}]},{"cell_type":"code","source":["assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(False)\\\n","    .setWhiteList([\"gastric problems\"])\\\n","    .setCriteria(\"isin\")\n","\n","\n","nlpPipeline = nlp.Pipeline(stages=[\n","      documentAssembler,\n","      sentenceDetector,\n","      tokenizer,\n","      word_embeddings,\n","      clinical_ner,\n","      ner_converter,\n","      clinical_assertion,\n","      assertion_filterer\n","    ])\n","\n","assertionFilter_model = nlpPipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))"],"metadata":{"id":"9YT7_rvz37za"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["text = 'I feel a bit drowsy & have a little blurred vision, but so far no gastric problems.'\n","\n","filter_df = spark.createDataFrame([[text]]).toDF(\"text\")\n","\n","chunk_filter_result = assertionFilter_model.transform(filter_df)"],"metadata":{"id":"-8YSwhV8x0R6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["chunk_filter_result.select('ner_chunk.result','assertion_filtered.result').show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2Mtcr_KN4G0z","executionInfo":{"status":"ok","timestamp":1687852318940,"user_tz":-120,"elapsed":1107,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"e482295b-184f-4d61-b536-075a7433a52c"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["+---------------------------------------------------------+------------------+\n","|result                                                   |result            |\n","+---------------------------------------------------------+------------------+\n","|[a bit drowsy, a little blurred vision, gastric problems]|[gastric problems]|\n","+---------------------------------------------------------+------------------+\n","\n"]}]},{"cell_type":"markdown","source":["## .setCriteria(\"assertion\")"],"metadata":{"id":"znDzj5JdyXSk"}},{"cell_type":"code","source":["documentAssembler = nlp.DocumentAssembler()\\\n","    .setInputCol(\"text\")\\\n","    .setOutputCol(\"document\")\n","\n","sentenceDetector = nlp.SentenceDetector()\\\n","    .setInputCols([\"document\"])\\\n","    .setOutputCol(\"sentence\")\n","\n","tokenizer = nlp.Tokenizer()\\\n","    .setInputCols([\"sentence\"])\\\n","    .setOutputCol(\"token\")\n","\n","bert_embeddings = nlp.BertEmbeddings.pretrained(\"biobert_pubmed_base_cased\")\\\n","    .setInputCols([\"sentence\", \"token\"])\\\n","    .setOutputCol(\"embeddings\")\n","\n","ade_ner_bert = medical.NerModel.pretrained(\"ner_ade_biobert\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentence\", \"token\", \"embeddings\"]) \\\n","    .setOutputCol(\"ner\")\n","\n","ner_converter = medical.NerConverterInternal() \\\n","    .setInputCols([\"sentence\", \"token\", \"ner\"]) \\\n","    .setOutputCol(\"ner_chunk\")\n","\n","biobert_assertion = medical.AssertionDLModel.pretrained(\"assertion_dl_biobert\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentence\", \"ner_chunk\", \"embeddings\"]) \\\n","    .setOutputCol(\"assertion\")\n","\n","assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(False)\\\n","    .setWhiteList([\"absent\"])\\\n","    .setCriteria(\"assertion\")\n","\n","assertion_pipeline = nlp.Pipeline(stages=[\n","    documentAssembler,\n","    sentenceDetector,\n","    tokenizer,\n","    bert_embeddings,\n","    ade_ner_bert,\n","    ner_converter,\n","    biobert_assertion,\n","    assertion_filterer])\n","\n","assertionFilter_model = assertion_pipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))\n","light_model = nlp.LightPipeline(assertionFilter_model)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XfpPKEV3ztF8","executionInfo":{"status":"ok","timestamp":1687853791993,"user_tz":-120,"elapsed":9070,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"2a856a97-135f-49dc-94a9-0b9219de8584"},"execution_count":109,"outputs":[{"output_type":"stream","name":"stdout","text":["biobert_pubmed_base_cased download started this may take some time.\n","Approximate size to download 386.4 MB\n","[OK!]\n","ner_ade_biobert download started this may take some time.\n","[OK!]\n","assertion_dl_biobert download started this may take some time.\n","[OK!]\n"]}]},{"cell_type":"code","source":["text = 'I feel a bit drowsy & have a little blurred vision, but so far no gastric problems.'\n","light_result = light_model.annotate(text)\n","\n","list(zip(light_result['ner_chunk'], light_result['assertion']))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"AWb16HXE0dsr","executionInfo":{"status":"ok","timestamp":1687853817876,"user_tz":-120,"elapsed":3,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"e11a53b2-80cb-46a1-aacc-1b1f09e79033"},"execution_count":112,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[('drowsy', 'present'),\n"," ('blurred vision', 'present'),\n"," ('gastric problems', 'absent')]"]},"metadata":{},"execution_count":112}]},{"cell_type":"code","source":["light_result[\"assertion_filtered\"]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"1cOekIdJ6vFn","executionInfo":{"status":"ok","timestamp":1687853821361,"user_tz":-120,"elapsed":5,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"14cf263b-f78f-47a0-877d-f95dc325bbf6"},"execution_count":113,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['gastric problems']"]},"metadata":{},"execution_count":113}]},{"cell_type":"markdown","source":["## setCriteria(\"regex\") and setRegex()"],"metadata":{"id":"hMcnTaPu1Rwv"}},{"cell_type":"code","source":["assertion_filterer = medical.AssertionFilterer()\\\n","    .setInputCols(\"sentence\", \"ner_chunk\", \"assertion\")\\\n","    .setOutputCol(\"assertion_filtered\")\\\n","    .setCaseSensitive(False)\\\n","    .setCriteria(\"regex\")\\\n","    .setRegex([\".*/.*\"])\\\n","\n","assertion_pipeline = nlp.Pipeline(stages=[\n","    documentAssembler,\n","    sentenceDetector,\n","    tokenizer,\n","    bert_embeddings,\n","    ade_ner_bert,\n","    ner_converter,\n","    biobert_assertion,\n","    assertion_filterer])\n","\n","assertionFilter_model = assertion_pipeline.fit(spark.createDataFrame([[\"\"]]).toDF(\"text\"))\n","light_model = nlp.LightPipeline(assertionFilter_model)"],"metadata":{"id":"Diq4UlZR4K0b","executionInfo":{"status":"ok","timestamp":1687853830086,"user_tz":-120,"elapsed":530,"user":{"displayName":"Damla","userId":"03285166568766987047"}}},"execution_count":114,"outputs":[]},{"cell_type":"code","source":["text = \"I have an allergic reaction to vancomycin so I have itchy skin, sore throat/burning/itching, numbness of tongue and gums. I would not recommend this drug to anyone, especially since I have never had such an adverse reaction to any other medication.\"\n","\n","light_result = light_model.annotate(text)"],"metadata":{"id":"s_S7Ts8f4ur8","executionInfo":{"status":"ok","timestamp":1687853832265,"user_tz":-120,"elapsed":614,"user":{"displayName":"Damla","userId":"03285166568766987047"}}},"execution_count":115,"outputs":[]},{"cell_type":"code","source":["list(zip(light_result['ner_chunk'], light_result['assertion']))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Vyz2_5mT6EBA","executionInfo":{"status":"ok","timestamp":1687853833515,"user_tz":-120,"elapsed":259,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"0170f5ce-c7ce-4f9e-ad7f-2ca51c541af4"},"execution_count":116,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[('allergic reaction', 'present'),\n"," ('vancomycin', 'present'),\n"," ('itchy skin', 'present'),\n"," ('sore throat/burning/itching', 'present'),\n"," ('numbness of tongue and gums', 'present'),\n"," ('any other medication', 'present')]"]},"metadata":{},"execution_count":116}]},{"cell_type":"code","source":["light_result[\"ner_chunk\"]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"bDHrTXjW5yJB","executionInfo":{"status":"ok","timestamp":1687853834293,"user_tz":-120,"elapsed":4,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"3810e2bc-23d3-43d1-fd42-00a5c9f05a5f"},"execution_count":117,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['allergic reaction',\n"," 'vancomycin',\n"," 'itchy skin',\n"," 'sore throat/burning/itching',\n"," 'numbness of tongue and gums',\n"," 'any other medication']"]},"metadata":{},"execution_count":117}]},{"cell_type":"code","source":["light_result[\"assertion_filtered\"]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"toJ4Alda6CqL","executionInfo":{"status":"ok","timestamp":1687853835209,"user_tz":-120,"elapsed":4,"user":{"displayName":"Damla","userId":"03285166568766987047"}},"outputId":"a8174ddf-1abb-4770-b436-16213330bfc8"},"execution_count":118,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['sore throat/burning/itching']"]},"metadata":{},"execution_count":118}]}],"metadata":{"colab":{"machine_shape":"hm","provenance":[{"file_id":"1seQsaMTaKaIWI_0Y1FJFFr2uViATggmu","timestamp":1684829932210},{"file_id":"1llKG8_dPM96uHeBspUS8Qwd-7M0ZgWon","timestamp":1684826323786}],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":0}