Spark_NLP_Udemy_MOOC/Healthcare_NLP/NerQuestionGenerator.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1o80l29c4srgaknOQ_E1C_lY2cs1NWepM","timestamp":1685644741264},{"file_id":"1OEGqeXa2rqYnelv6snFlJtQomsVuUpjj","timestamp":1683833270367},{"file_id":"1UBiO9kbI1FXDiC5g3iO4D4sG98laDebJ","timestamp":1683644665690},{"file_id":"1M91oRaz13uQc9yB05O6AMISVRzh5LxC8","timestamp":1683051971314},{"file_id":"1gAmDc0oIjjTYctFbdplGlPbexp5AstuO","timestamp":1681680414646},{"file_id":"1k1SVJ-vVIc_vLS4-3ex7f687zjmHp8gR","timestamp":1672339943946}]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\n"],"metadata":{"id":"klIak_Gb_OPJ"}},{"cell_type":"markdown","source":["# **NerQuestionGenerator**"],"metadata":{"id":"bewC1SWN-6jB"}},{"cell_type":"markdown","source":["This notebook will cover the different parameters and usages of `NerQuestionGenerator` annotator. \n","\n","**📖 Learning Objectives:**\n","\n","1. Understand how to use `NerQuestionGenerator`.\n","\n","2. Become comfortable using the different parameters of the annotator.\n","\n","3. Programatically generate question to be used by Question-Answering models.\n","\n","\n","**🔗 Helpful Links:**\n","\n","- Documentation : [NerQuestionGenerator](https://nlp.johnsnowlabs.com/docs/en/licensed_annotator)\n","\n","- Python Docs : [NerQuestionGenerator](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/qa/qa_ner_generator/index.html#sparknlp_jsl.annotator.qa.qa_ner_generator.NerQuestionGenerator)\n","\n","- Scala Docs : [NerQuestionGenerator](https://nlp.johnsnowlabs.com/licensed/api/com/johnsnowlabs/nlp/annotators/qa/NerQuestionGenerator.html)\n","\n","- For extended examples of usage, see the [Spark NLP Workshop repository](https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/healthcare-nlp/)."],"metadata":{"id":"m048uDkB69Rv"}},{"cell_type":"markdown","source":["## **📜 Background**\n"],"metadata":{"id":"B9Wo54MT8jKU"}},{"cell_type":"markdown","source":["`NerQuestionGenerator` takes an NER chunk (obtained by, e.g., `NerConverterInternal`) and generates a questions based on two entity types, a pronoun and a strategy.\n","\n","The question is generated in the form of `[QUESTIONPRONOUN] [ENTITY1] [ENTITY2] [QUESTIONMARK]`. The generated question can be used by `QuestionAnswerer` or `ZeroShotNer` annotators to answer the question or find NER entities."],"metadata":{"id":"yaDBNKJsAovm"}},{"cell_type":"markdown","source":["## **🎬 Colab Setup**"],"metadata":{"id":"A4hMnkhd_ik9"}},{"cell_type":"code","source":["!pip install -q johnsnowlabs"],"metadata":{"id":"xrdvNxjD_yQI","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1685648773551,"user_tz":180,"elapsed":40923,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"f5efa3f3-7f4d-4f54-9e48-5451e14ff825"},"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.2/84.2 kB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.4/212.4 MB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m486.7/486.7 kB\u001b[0m \u001b[31m29.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m639.9/639.9 kB\u001b[0m \u001b[31m33.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m198.6/198.6 kB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m66.9/66.9 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.5/83.5 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m68.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Building wheel for databricks-cli (setup.py) ... \u001b[?25l\u001b[?25hdone\n"]}]},{"cell_type":"code","source":["from johnsnowlabs import nlp\n","\n","\n","nlp.install(force_browser=True)"],"metadata":{"id":"W5D63vBo_0u0","colab":{"base_uri":"https://localhost:8080/","height":251},"executionInfo":{"status":"ok","timestamp":1685648864310,"user_tz":180,"elapsed":90762,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"9eb5fba3-579a-41ab-c21b-d5463127cc18"},"execution_count":4,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.Javascript object>"],"application/javascript":["\n","        var a = document.createElement(\"a\");\n","        a.id=\"auth-btn\"\n","        a.setAttribute(\"target\", \"_blank\");\n","        a.href=\"https://my.johnsnowlabs.com/oauth/authorize/?client_id=2hfGX0iZ5lvyxvLaK3IEzS9Bc9LGfTYCwVvKFfjY&response_type=code&code_challenge_method=S256&code_challenge=NuBXDrW0oIgPdhRoc5115C-lZ8tJuzp21d_Yp7XOSME&redirect_uri=https%3A%2F%2Fqe9k5eh0ke-496ff2e9c6d22116-8000-colab.googleusercontent.com%2Flogin\";\n","        a.style=\"padding:15px 20px;background-color:#0298d9;border-radius:7px;color:white;text-decoration:none;\"\n","        a.innerText=\"Click here to Authorize on My.Johnsnowlabs.com\"\n","        document.body.appendChild(a);\n","        document.body.style = \"text-align:center;padding-top:15px;\"\n","        a.click()\n","      "]},"metadata":{}},{"output_type":"stream","name":"stderr","text":["127.0.0.1 - - [01/Jun/2023 19:47:17] \"GET /login?code=mwOOQWBW7czMUWe41WtmRs7lDsXfWG HTTP/1.1\" 200 -\n"]},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.Javascript object>"],"application/javascript":["document.body.removeChild(a);"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Downloading license...\n","Licenses extracted successfully\n","📋 Stored John Snow Labs License in /root/.johnsnowlabs/licenses/license_number_0_for_Spark-Healthcare_Spark-OCR.json\n","👷 Setting up  John Snow Labs home in /root/.johnsnowlabs, this might take a few minutes.\n","Downloading 🐍+🚀 Python Library spark_nlp-4.4.1-py2.py3-none-any.whl\n","Downloading 🐍+💊 Python Library spark_nlp_jsl-4.4.2-py3-none-any.whl\n","Downloading 🫘+🚀 Java Library spark-nlp-assembly-4.4.1.jar\n","Downloading 🫘+💊 Java Library spark-nlp-jsl-4.4.2.jar\n","🙆 JSL Home setup in /root/.johnsnowlabs\n","Installing /root/.johnsnowlabs/py_installs/spark_nlp_jsl-4.4.2-py3-none-any.whl to /usr/bin/python3\n","Installed 1 products:\n","💊 Spark-Healthcare==4.4.2 installed! ✅ Heal the planet with NLP! \n"]}]},{"cell_type":"code","source":["from johnsnowlabs import nlp, medical\n","import pyspark.sql.functions as F\n","\n","\n","spark = nlp.start()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"yohN0MjG4bqr","executionInfo":{"status":"ok","timestamp":1685648885703,"user_tz":180,"elapsed":21403,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"4d93dcf4-39d3-436c-b92d-40d9d836a91b"},"execution_count":5,"outputs":[{"output_type":"stream","name":"stdout","text":["👌 Launched \u001b[92mcpu optimized\u001b[39m session with with: 🚀Spark-NLP==4.4.1, 💊Spark-Healthcare==4.4.2, running on ⚡ PySpark==3.1.2\n"]}]},{"cell_type":"markdown","source":["## **🖨️ Input/Output Annotation Types**"],"metadata":{"id":"fdTHvykf8wni"}},{"cell_type":"markdown","source":["- Input: `CHUNK`\n","\n","- Output: `DOCUMENT`"],"metadata":{"id":"ejYVNcX98y5j"}},{"cell_type":"markdown","source":["## **🔎 Parameters**\n"],"metadata":{"id":"YVa72oJd9Bk_"}},{"cell_type":"markdown","source":["- `questionPronoun`: Pronoun to be used in the question. E.g., 'When', 'Where', 'Why', 'How', 'Who', 'What'.\n","- `strategyType`: Strategy for the proccess, either `Paired` (default) or `Combined`.\n","- `questionMark`: Whether to add a question mark at the end of the question.\n","- `entities1`: List with the entity types of entities that appear first in the question. \n","- `entities2`: List with the entity types of entities that appear second in the question.\n","\n","\n","All the parameters can be set using the corresponding set method in camel case. For example, `.setQuestionPronoun(True)`."],"metadata":{"id":"AUbv3YL59D8Q"}},{"cell_type":"markdown","source":["### Preparation"],"metadata":{"id":"c6xSWXauZJyc"}},{"cell_type":"markdown","source":["First, let's create a data frame with identified entities that will be used to generate questions. We will use the `EntityRulerApproach` to identify entitites present in a JSON file."],"metadata":{"id":"DrXwpNYxZNOP"}},{"cell_type":"code","source":["import json\n","\n","entities = [\n","          {\n","            \"label\": \"Person\",\n","            \"patterns\": [\"Jon\", \"John\", \"John's\"]\n","          },\n","          {\n","            \"label\": \"Organization\",\n","            \"patterns\": [\"St. Mary's Hospital\", \"St. Mary's\"]\n","          },\n","          {\n","              \"label\": \"Condition\",\n","              \"patterns\": [\"vital signs\", \"heartbeat\", \"oxygen saturation levels\"]\n","          }\n","         ]\n","\n","with open('./entities.json', 'w') as jsonfile:\n","    json.dump(entities, jsonfile)"],"metadata":{"id":"Imw1w4fAZM2N","executionInfo":{"status":"ok","timestamp":1685649761450,"user_tz":180,"elapsed":337,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}}},"execution_count":10,"outputs":[]},{"cell_type":"code","source":["document_assembler = nlp.DocumentAssembler().setInputCol(\"text\").setOutputCol(\"document\")\n","\n","entity_ruler = nlp.EntityRulerApproach() \\\n","                  .setInputCols([\"document\"]) \\\n","                  .setOutputCol(\"entity\") \\\n","                  .setPatternsResource(\"./entities.json\")\\\n","                  .setCaseSensitive(False)\n","\n","prep_pipeline = nlp.Pipeline(stages=[\n","    document_assembler,\n","    entity_ruler\n","])"],"metadata":{"id":"3VxHgbLegKnw","executionInfo":{"status":"ok","timestamp":1685649763792,"user_tz":180,"elapsed":452,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}}},"execution_count":11,"outputs":[]},{"cell_type":"code","source":["example = \"\"\"At St. Mary's Hospital, the healthcare team closely monitored John's vital signs with unwavering attention. They recorded his heartbeat and oxygen saturation levels, promptly addressing any deviations from normal. Their dedication and expertise at St. Mary's played a vital role in ensuring John's stability and fostering a swift recovery.\"\"\"\n","df = spark.createDataFrame([[example]]).toDF(\"text\")\n","\n","# Apply the initial steps\n","df = prep_pipeline.fit(df).transform(df)\n","\n","df.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ycpUSO37e93Z","executionInfo":{"status":"ok","timestamp":1685649766666,"user_tz":180,"elapsed":1575,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"422e5e03-50ea-4e82-d5e5-64651c819074"},"execution_count":12,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+--------------------+--------------------+\n","|                text|            document|              entity|\n","+--------------------+--------------------+--------------------+\n","|At St. Mary's Hos...|[{document, 0, 33...|[{chunk, 3, 21, S...|\n","+--------------------+--------------------+--------------------+\n","\n"]}]},{"cell_type":"code","source":["df.select(F.explode(F.arrays_zip(df.entity.result, df.entity.metadata)).alias(\"cols\")).select(\n","    F.expr(\"cols['0']\").alias(\"chunk\"),\n","    F.expr(\"cols['1'].entity\").alias(\"entity\")\n",").show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Col4BkS6jk1a","executionInfo":{"status":"ok","timestamp":1685650066046,"user_tz":180,"elapsed":2090,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"80da2a20-16df-49c6-9108-2b925548854a"},"execution_count":18,"outputs":[{"output_type":"stream","name":"stdout","text":["+------------------------+------------+\n","|chunk                   |entity      |\n","+------------------------+------------+\n","|St. Mary's Hospital     |Organization|\n","|John's                  |Person      |\n","|vital signs             |Condition   |\n","|heartbeat               |Condition   |\n","|oxygen saturation levels|Condition   |\n","|John's                  |Person      |\n","+------------------------+------------+\n","\n"]}]},{"cell_type":"markdown","source":["### `questionPronoun`, `entities1`, `entities2`"],"metadata":{"id":"7T_3otzSBllg"}},{"cell_type":"markdown","source":["Using `What`: "],"metadata":{"id":"d_X6COPXjDrZ"}},{"cell_type":"code","source":["qagenerator = (\n","    medical.NerQuestionGenerator()\n","    .setInputCols([\"entity\"])\n","    .setOutputCol(\"question\")\n","    .setQuestionPronoun(\"What\")\n","    .setEntities1([\"Person\"])\n","    .setEntities2([\"Condition\"])\n",")\n"],"metadata":{"id":"YdyH_OvPNr_q","executionInfo":{"status":"ok","timestamp":1685649855955,"user_tz":180,"elapsed":334,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}}},"execution_count":14,"outputs":[]},{"cell_type":"code","source":["qagenerator.transform(df).select(\"question\").show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"C6uPZ_LSjGwf","executionInfo":{"status":"ok","timestamp":1685649859339,"user_tz":180,"elapsed":1233,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"5297875a-9826-42d0-a32b-c5969d50f9e7"},"execution_count":15,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------------------------------------------------------------------------------------------------------------------------+\n","|question                                                                                                                              |\n","+--------------------------------------------------------------------------------------------------------------------------------------+\n","|[{document, 62, 79, What John's vital signs , {sentence -> 0}, []}, {document, 291, 134, What John's heartbeat , {sentence -> 0}, []}]|\n","+--------------------------------------------------------------------------------------------------------------------------------------+\n","\n"]}]},{"cell_type":"markdown","source":["Using `Where`:"],"metadata":{"id":"CXCgb2zYj-F2"}},{"cell_type":"code","source":["qagenerator = (\n","    medical.NerQuestionGenerator()\n","    .setInputCols([\"entity\"])\n","    .setOutputCol(\"question\")\n","    .setQuestionPronoun(\"Where\")\n","    .setEntities1([\"Person\"])\n","    .setEntities2([\"Condition\"])\n",")\n","\n","qagenerator.transform(df).select(\"question\").show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"_IHrDqDYj9ot","executionInfo":{"status":"ok","timestamp":1685650200650,"user_tz":180,"elapsed":491,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"524069c7-5f03-4520-8df5-4288736174d5"},"execution_count":20,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------------------------------------------------------------------------------------------------------------------------------+\n","|question                                                                                                                                |\n","+----------------------------------------------------------------------------------------------------------------------------------------+\n","|[{document, 62, 79, Where John's vital signs , {sentence -> 0}, []}, {document, 291, 134, Where John's heartbeat , {sentence -> 0}, []}]|\n","+----------------------------------------------------------------------------------------------------------------------------------------+\n","\n"]}]},{"cell_type":"markdown","source":["### `strategyType`"],"metadata":{"id":"xx5yOlkwElEA"}},{"cell_type":"markdown","source":["- If set to `Paired` (default), applies a one-vs-one strategy. In this case, the number of chunks in Entity 1 must be aligned with the number of chunks in Entity 2. E.g., if Entity 1 has 3 chunks and Entity 2 has 3 chunks, the first chunk of Entity 1 will be grouped with first chunk of Entity 2,\n","the second with second, third with third, etc.\n","\n","- If set to `Combined`, applies a one-vs-all strategy. In this case, the number of chunks in Entity 1 don't need to be the same as the number of chunks in Entity 2, and each chunk in Entity 1 will be grouped with all chunks in Entity 2."],"metadata":{"id":"rn-5a-CtMgC9"}},{"cell_type":"code","source":["qagenerator = (\n","    medical.NerQuestionGenerator()\n","    .setInputCols([\"entity\"])\n","    .setOutputCol(\"question\")\n","    .setQuestionPronoun(\"Where\")\n","    .setEntities1([\"Person\"])\n","    .setEntities2([\"Organization\"])\n","    .setStrategyType(\"Paired\")\n",")\n","\n","qagenerator.transform(df).select(\"question.result\").show(truncate=False)"],"metadata":{"id":"TsLx-rxREnlo","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1685650249999,"user_tz":180,"elapsed":866,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"6ee519f8-0cd1-47e8-be6d-ea96014a6e5e"},"execution_count":21,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------------------------------------------+\n","|result                                              |\n","+----------------------------------------------------+\n","|[Where John's vital signs , Where John's heartbeat ]|\n","+----------------------------------------------------+\n","\n"]}]},{"cell_type":"code","source":["qagenerator = (\n","    medical.NerQuestionGenerator()\n","    .setInputCols([\"entity\"])\n","    .setOutputCol(\"question\")\n","    .setQuestionPronoun(\"How\")\n","    .setEntities1([\"Person\"])\n","    .setEntities2([\"Condition\"])\n","    .setStrategyType(\"Combined\")\n",")\n","\n","qagenerator.transform(df).select(\"question.result\").show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"n8Ojp9VElnpZ","executionInfo":{"status":"ok","timestamp":1685650344233,"user_tz":180,"elapsed":3,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"0ee85740-39f0-4349-9768-c0a7d65dae23"},"execution_count":23,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------------------------------------------------------------------------------------------------------------------------------------+\n","|result                                                                                                                                        |\n","+----------------------------------------------------------------------------------------------------------------------------------------------+\n","|[John's vital signs, John's heartbeat, John's oxygen saturation levels, John's vital signs, John's heartbeat, John's oxygen saturation levels]|\n","+----------------------------------------------------------------------------------------------------------------------------------------------+\n","\n"]}]},{"cell_type":"markdown","source":["### `questionMark`"],"metadata":{"id":"Y_J8-cpXEoEy"}},{"cell_type":"code","source":["qagenerator = (\n","    medical.NerQuestionGenerator()\n","    .setInputCols([\"entity\"])\n","    .setOutputCol(\"question\")\n","    .setQuestionPronoun(\"How is\")\n","    .setEntities1([\"Person\"])\n","    .setEntities2([\"Condition\"])\n","    .setStrategyType(\"Paired\")\n","    .setQuestionMark(True)\n",")\n","\n","qagenerator.transform(df).select(\"question.result\").show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"m7e8FzH9Uqrq","executionInfo":{"status":"ok","timestamp":1685650442270,"user_tz":180,"elapsed":1438,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"fc7afb8f-2b76-430f-b09d-a3d0c729b03e"},"execution_count":26,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------------------------------------------+\n","|result                                                  |\n","+--------------------------------------------------------+\n","|[How is John's vital signs ?, How is John's heartbeat ?]|\n","+--------------------------------------------------------+\n","\n"]}]},{"cell_type":"markdown","source":["## Fast inference with [LightPipelines](https://nlp.johnsnowlabs.com/docs/en/concepts#using-spark-nlps-lightpipeline)"],"metadata":{"id":"gbYfY2mnWdpb"}},{"cell_type":"markdown","source":["We can use Spark NLP's `LightPipeline` to run fast inference directly on text (or list of text) instead of using spark data frames. \n","\n","Let's check how to do that."],"metadata":{"id":"4N3hD_k1WtO0"}},{"cell_type":"code","source":["pipeline = nlp.Pipeline(stages=[prep_pipeline, qagenerator])\n","\n","lp = nlp.LightPipeline(pipeline.fit(df.select(\"text\")))"],"metadata":{"id":"52gfCJAXmMun","executionInfo":{"status":"ok","timestamp":1685650515544,"user_tz":180,"elapsed":388,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}}},"execution_count":28,"outputs":[]},{"cell_type":"code","source":["result = lp.annotate(example)\n","result"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KOejJgPVWTEb","executionInfo":{"status":"ok","timestamp":1685650533569,"user_tz":180,"elapsed":2,"user":{"displayName":"David Cecchini","userId":"15563969817876100862"}},"outputId":"b5191ae3-d1cf-4334-9f7e-efe238353d0d"},"execution_count":29,"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'document': [\"At St. Mary's Hospital, the healthcare team closely monitored John's vital signs with unwavering attention. They recorded his heartbeat and oxygen saturation levels, promptly addressing any deviations from normal. Their dedication and expertise at St. Mary's played a vital role in ensuring John's stability and fostering a swift recovery.\"],\n"," 'entity': [\"St. Mary's Hospital\",\n","  \"John's\",\n","  'vital signs',\n","  'heartbeat',\n","  'oxygen saturation levels',\n","  \"John's\"],\n"," 'question': [\"How is John's vital signs ?\", \"How is John's heartbeat ?\"]}"]},"metadata":{},"execution_count":29}]},{"cell_type":"code","source":[],"metadata":{"id":"K_hZp6ZKXVzq"},"execution_count":null,"outputs":[]}]}