2023-05-21-ner_ade_emb_clinical_medium_en (#244)

* Add model 2023-05-21-ner_ade_emb_clinical_medium_en * Add model 2023-05-21-ner_ade_emb_clinical_large_en * Add model 2023-05-24-ner_cellular_emb_clinical_medium_en * Add model 2023-05-24-ner_cellular_emb_clinical_large_en --------- Co-authored-by: Damla-Gurbaz <dml.grbz.01@gmail.com>
JohnSnowLabs · May 24, 2023 · 083df29 · 083df29
1 parent 5999c14
commit 083df29
Show file tree

Hide file tree

Showing 4 changed files with 654 additions and 0 deletions.
diff --git a/docs/_posts/Damla-Gurbaz/2023-05-21-ner_ade_emb_clinical_large_en.md b/docs/_posts/Damla-Gurbaz/2023-05-21-ner_ade_emb_clinical_large_en.md
@@ -0,0 +1,152 @@
+---
+layout: model
+title: Detect Adverse Drug Events (clinical_large)
+author: John Snow Labs
+name: ner_ade_emb_clinical_large
+date: 2023-05-21
+tags: [ner, ade, drug, licensed, clinical, en]
+task: Named Entity Recognition
+language: en
+edition: Healthcare NLP 4.4.2
+spark_version: 3.0
+supported: true
+annotator: MedicalNerModel
+article_header:
+  type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Detect adverse reactions to drugs in reviews, tweets, and medical text using a pre-trained NER model.
+
+## Predicted Entities
+
+`DRUG`, `ADE`
+
+{:.btn-box}
+[Live Demo](https://demo.johnsnowlabs.com/healthcare/ADE/){:.button.button-orange}
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/16.Adverse_Drug_Event_ADE_NER_and_Classifier.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_ade_emb_clinical_large_en_4.4.2_3.0_1684710290191.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden}
+[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_ade_emb_clinical_large_en_4.4.2_3.0_1684710290191.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
+
+## How to use
+
+
+
+<div class="tabs-box" markdown="1">
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+```python
+document_assembler = DocumentAssembler()\
+    .setInputCol("text")\
+    .setOutputCol("document")
+
+sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")\
+    .setInputCols(["document"])\
+    .setOutputCol("sentence")
+
+tokenizer = Tokenizer()\
+    .setInputCols(["sentence"])\
+    .setOutputCol("token")
+
+clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_large", "en", "clinical/models")\
+    .setInputCols(["sentence", "token"])\
+    .setOutputCol("embeddings")
+
+ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_large", "en", "clinical/models")\
+    .setInputCols(["sentence", "token","embeddings"])\
+    .setOutputCol("ner")
+
+ner_converter = NerConverterInternal()\
+    .setInputCols(['sentence', 'token', 'ner'])\
+    .setOutputCol('ner_chunk')
+
+pipeline = Pipeline(stages=[
+    document_assembler, 
+    sentence_detector,
+    tokenizer,
+    clinical_embeddings,
+    ner_model,
+    ner_converter   
+    ])
+
+sample_df = spark.createDataFrame([["Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps."]]).toDF("text")
+
+result = pipeline.fit(sample_df).transform(sample_df)
+```
+```scala
+val document_assembler = new DocumentAssembler()
+    .setInputCol("text")
+    .setOutputCol("document")
+
+val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")
+    .setInputCols("document")
+    .setOutputCol("sentence")
+
+val tokenizer = new Tokenizer()
+    .setInputCols("sentence")
+    .setOutputCol("token")
+
+val clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_large", "en", "clinical/models")
+    .setInputCols(Array("sentence", "token"))
+    .setOutputCol("embeddings")
+
+val ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_large", "en", "clinical/models")
+    .setInputCols(Array("sentence", "token", "embeddings"))
+    .setOutputCol("ner")
+
+val ner_converter = new NerConverterInternal()
+    .setInputCols(Array("sentence", "token", "ner"))
+    .setOutputCol("ner_chunk")
+
+val pipeline = new Pipeline().setStages(Array(
+    document_assembler, 
+    sentence_detector,
+    tokenizer,
+    clinical_embeddings,
+    ner_model,
+    ner_converter))
+
+val sample_data = Seq("Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps.").toDS.toDF("text")
+
+val result = pipeline.fit(sample_data).transform(sample_data)
+```
+</div>
+
+## Results
+
+```bash
++--------------+-----+---+---------+
+|chunk         |begin|end|ner_label|
++--------------+-----+---+---------+
+|Lipitor       |12   |18 |DRUG     |
+|severe fatigue|52   |65 |ADE      |
+|voltaren      |97   |104|DRUG     |
+|cramps        |152  |157|ADE      |
++--------------+-----+---+---------+
+```
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|ner_ade_emb_clinical_large|
+|Compatibility:|Healthcare NLP 4.4.2+|
+|License:|Licensed|
+|Edition:|Official|
+|Input Labels:|[sentence, token, embeddings]|
+|Output Labels:|[ner]|
+|Language:|en|
+|Size:|2.7 MB|
+
+## Benchmarking
+
+```bash
+       label  precision    recall  f1-score   support
+        DRUG       0.92      0.91      0.92     16032
+         ADE       0.82      0.80      0.81      6142
+   micro-avg       0.89      0.88      0.89     22174
+   macro-avg       0.87      0.86      0.86     22174
+weighted-avg       0.89      0.88      0.89     22174
+```
diff --git a/docs/_posts/Damla-Gurbaz/2023-05-21-ner_ade_emb_clinical_medium_en.md b/docs/_posts/Damla-Gurbaz/2023-05-21-ner_ade_emb_clinical_medium_en.md
@@ -0,0 +1,152 @@
+---
+layout: model
+title: Detect Adverse Drug Events (clinical_medium)
+author: John Snow Labs
+name: ner_ade_emb_clinical_medium
+date: 2023-05-21
+tags: [en, clinical, ade, drug, licensed, ner]
+task: Named Entity Recognition
+language: en
+edition: Healthcare NLP 4.4.2
+spark_version: 3.0
+supported: true
+annotator: MedicalNerModel
+article_header:
+  type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Detect adverse reactions to drugs in reviews, tweets, and medical text using a pre-trained NER model.
+
+## Predicted Entities
+
+`DRUG`, `ADE`
+
+{:.btn-box}
+[Live Demo](https://demo.johnsnowlabs.com/healthcare/ADE/){:.button.button-orange}
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/16.Adverse_Drug_Event_ADE_NER_and_Classifier.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_ade_emb_clinical_medium_en_4.4.2_3.0_1684646733993.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden}
+[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_ade_emb_clinical_medium_en_4.4.2_3.0_1684646733993.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
+
+## How to use
+
+
+
+<div class="tabs-box" markdown="1">
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+```python
+document_assembler = DocumentAssembler()\
+    .setInputCol("text")\
+    .setOutputCol("document")
+
+sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")\
+    .setInputCols(["document"])\
+    .setOutputCol("sentence")
+
+tokenizer = Tokenizer()\
+    .setInputCols(["sentence"])\
+    .setOutputCol("token")
+
+clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_medium", "en", "clinical/models")\
+    .setInputCols(["sentence", "token"])\
+    .setOutputCol("embeddings")
+
+ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_medium", "en", "clinical/models")\
+    .setInputCols(["sentence", "token", "embeddings"])\
+    .setOutputCol("ner")
+
+ner_converter = NerConverterInternal()\
+    .setInputCols(['sentence', 'token', 'ner'])\
+    .setOutputCol('ner_chunk')
+
+pipeline = Pipeline(stages=[
+    document_assembler, 
+    sentence_detector,
+    tokenizer,
+    clinical_embeddings,
+    ner_model,
+    ner_converter   
+    ])
+
+sample_df = spark.createDataFrame([["Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps."]]).toDF("text")
+
+result = pipeline.fit(sample_df).transform(sample_df)
+```
+```scala
+val document_assembler = new DocumentAssembler()
+    .setInputCol("text")
+    .setOutputCol("document")
+
+val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")
+    .setInputCols("document")
+    .setOutputCol("sentence")
+
+val tokenizer = new Tokenizer()
+    .setInputCols("sentence")
+    .setOutputCol("token")
+
+val clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_medium", "en", "clinical/models")
+    .setInputCols(Array("sentence", "token"))
+    .setOutputCol("embeddings")
+
+val ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_medium", "en", "clinical/models")
+    .setInputCols(Array("sentence", "token", "embeddings"))
+    .setOutputCol("ner")
+
+val ner_converter = new NerConverterInternal()
+    .setInputCols(Array("sentence", "token", "ner"))
+    .setOutputCol("ner_chunk")
+
+val pipeline = new Pipeline().setStages(Array(
+    document_assembler, 
+    sentence_detector,
+    tokenizer,
+    clinical_embeddings,
+    ner_model,
+    ner_converter))
+
+val sample_data = Seq("Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps.").toDS.toDF("text")
+
+val result = pipeline.fit(sample_data).transform(sample_data)
+```
+</div>
+
+## Results
+
+```bash
++--------------+-----+---+---------+
+|chunk         |begin|end|ner_label|
++--------------+-----+---+---------+
+|Lipitor       |12   |18 |DRUG     |
+|severe fatigue|52   |65 |ADE      |
+|voltaren      |97   |104|DRUG     |
+|cramps        |152  |157|ADE      |
++--------------+-----+---+---------+
+```
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|ner_ade_emb_clinical_medium|
+|Compatibility:|Healthcare NLP 4.4.2+|
+|License:|Licensed|
+|Edition:|Official|
+|Input Labels:|[sentence, token, embeddings]|
+|Output Labels:|[ner]|
+|Language:|en|
+|Size:|2.7 MB|
+
+## Benchmarking
+
+```bash
+       label  precision    recall  f1-score   support
+        DRUG       0.92      0.91      0.91     15895
+         ADE       0.83      0.77      0.80      6077
+   micro-avg       0.89      0.87      0.88     21972
+   macro-avg       0.87      0.84      0.86     21972
+weighted-avg       0.89      0.87      0.88     21972
+```