Spark_NLP_Udemy_MOOC/Healthcare_NLP/RelationExtractionApproach.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"sXatvRX899i0"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"]},{"cell_type":"markdown","metadata":{"id":"AOn8d1tcBkK3"},"source":["# **RelationExtractionApproach**"]},{"cell_type":"markdown","source":["In this notebook, we will examine the `RelationExtractionApproach` to to train an `RelationExtractionModel`.\n","\n","\n","**📖 Learning Objectives:**\n","\n","1. Learn how to preprocess your data before training a Relation Extraction model.\n","\n","2. Understand how to train a Relation Extraction model using `RelationExtractionApproach`.\n","\n","3. Understand how to resume Relation Extraction model training.\n","\n","\n","\n","\n","**🔗 Helpful Links:**\n","\n","For extended examples of usage, see the [Spark NLP Workshop](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/healthcare-nlp/03.0.Clinical_Relation_Extraction.ipynb)\n","\n","Python Documentation: [RelationExtractionApproach](https://nlp.johnsnowlabs.com/licensed/api/python/reference/autosummary/sparknlp_jsl/annotator/re/relation_extraction/index.html#sparknlp_jsl.annotator.re.relation_extraction.RelationExtractionApproach.name)\n","\n","Scala Documentation: [RelationExtractionApproach](https://nlp.johnsnowlabs.com/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionApproach.html) <br/>\n","\n"],"metadata":{"id":"WimihSwD3vtH"}},{"cell_type":"markdown","source":["## **📜 Background**\n"],"metadata":{"id":"qL9lcISyFSLv"}},{"cell_type":"markdown","source":["`RelationExtractionApproach` is used for training a TensorFlow model for relation extraction. <br/>\n","\n","To train a custom relation extraction model, you need to first creat a Tensorflow graph using either the `TfGraphBuilder` annotator or the `tf_graph` module. Then, set the path to the Tensorflow graph using the method `.setModelFile(\"path/to/tensorflow_graph.pb\")`.\n","\n","If the parameter `relationDirectionCol` is set, the model will be trained using the direction information (see the parameter decription for details). Otherwise, the model won’t have direction between the relation of the entities.\n","\n","After training a model (using the .fit() method), the resulting object is of class `RelationExtractionModel`.\n","\n","\n","\n","\n","\n","\n"],"metadata":{"id":"TjDKOoZ4Fc8G"}},{"cell_type":"markdown","metadata":{"id":"MfkkKkbVF309"},"source":["## **🎬 Colab Setup**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iMkMQtZNF2n-"},"outputs":[],"source":["!pip install -q johnsnowlabs"]},{"cell_type":"code","source":["from google.colab import files\n","print('Please Upload your John Snow Labs License using the button below')\n","license_keys = files.upload()"],"metadata":{"id":"R7CfxWMkEzIX"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp, medical, visual\n","\n","# After uploading your license run this to install all licensed Python Wheels and pre-download Jars the Spark Session JVM\n","nlp.install()"],"metadata":{"id":"NulWi4_f4GN5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from johnsnowlabs import nlp, medical\n","\n","spark = nlp.start()"],"metadata":{"id":"2GLXxe1Q0Iln"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from pyspark.sql import DataFrame\n","import pyspark.sql.functions as F\n","import pyspark.sql.types as T\n","import pyspark.sql as SQL\n","from pyspark import keyword_only"],"metadata":{"id":"DRAgk0aVNREB","executionInfo":{"status":"ok","timestamp":1687514131604,"user_tz":-60,"elapsed":6,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":["## **🖨️ Input/Output Annotation Types**"],"metadata":{"id":"9Fbbk1bqcuA5"}},{"cell_type":"markdown","source":["- Input: `WORD_EMBEDDINGS`, `POS`, `CHUNK`, `DEPENDENCY`\n","\n","- Output: `CATEGORY`"],"metadata":{"id":"0yFIrr5acsiU"}},{"cell_type":"markdown","source":["## **📂 Training Data**"],"metadata":{"id":"dt9KfxnM5Pz6"}},{"cell_type":"markdown","source":["Downloading sample data:"],"metadata":{"id":"17FrqktW3wkp"}},{"cell_type":"code","source":["!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/i2b2_clinical_rel_dataset.csv"],"metadata":{"id":"FmFnWt9jMhoF","executionInfo":{"status":"ok","timestamp":1687514374296,"user_tz":-60,"elapsed":321,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":8,"outputs":[]},{"cell_type":"code","source":["data = spark.read.option(\"header\",\"true\").format(\"csv\").load(\"i2b2_clinical_rel_dataset.csv\")\n","\n","data = data.select( 'sentence','firstCharEnt1','firstCharEnt2','lastCharEnt1','lastCharEnt2', \"chunk1\", \"chunk2\", \"label1\", \"label2\",'rel','dataset')\n","\n","data.show(10)\n","\n","# you only need these columns>> 'sentence','firstCharEnt1','firstCharEnt2','lastCharEnt1','lastCharEnt2', \"chunk1\", \"chunk2\", \"label1\", \"label2\",'rel'\n","# ('dataset' column is optional)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"MngkV4G333VQ","executionInfo":{"status":"ok","timestamp":1687514380677,"user_tz":-60,"elapsed":4673,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"835988a0-ef33-43ad-aea4-30ed8436653e"},"execution_count":9,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+-------------+-------------+------------+------------+--------------------+--------------------+---------+---------+-----+-------+\n","|            sentence|firstCharEnt1|firstCharEnt2|lastCharEnt1|lastCharEnt2|              chunk1|              chunk2|   label1|   label2|  rel|dataset|\n","+--------------------+-------------+-------------+------------+------------+--------------------+--------------------+---------+---------+-----+-------+\n","|VITAL SIGNS - Tem...|           49|           75|          64|          84|    respiratory rate|          saturation|     test|     test|    O|   test|\n","|No lotions , crea...|            3|           34|           9|          42|             lotions|           incisions|treatment|  problem|TrNAP|   test|\n","|Because of expect...|           11|           58|          54|          68|expected long ter...|         a picc line|treatment|treatment|    O|  train|\n","|She states this l...|           16|           82|          31|          92|    light-headedness|         diaphoresis|  problem|  problem|  PIP|  train|\n","|Initial electroca...|          196|          145|         239|         176|an inferior and r...|1-mm st depressio...|  problem|  problem|  PIP|   test|\n","|Abdominal ultraso...|            1|           54|          20|          71|abdominal ultrasound|  gallbladder sludge|     test|  problem| TeRP|   test|\n","|At that time he w...|           99|          139|         133|         173|ir placement of a...|his abdominopelvi...|treatment|  problem| TrAP|   test|\n","|He was transferre...|          143|          195|         154|         213|        reintubation| worsening pneumonia|treatment|  problem| TrAP|   test|\n","|5) Anemia - previ...|           12|           35|          23|          47|        previous w/u|       elev ferritin|     test|  problem| TeRP|   test|\n","|The patient &apos...|          161|          149|         179|         156| po pain medications|            his pain|treatment|  problem| TrAP|  train|\n","+--------------------+-------------+-------------+------------+------------+--------------------+--------------------+---------+---------+-----+-------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"code","source":["data.groupby('dataset').count().show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"q6dqICWN6c7F","executionInfo":{"status":"ok","timestamp":1687514383613,"user_tz":-60,"elapsed":2938,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"e44e40e2-e1aa-40f1-a0de-68a44205f2d7"},"execution_count":10,"outputs":[{"output_type":"stream","name":"stdout","text":["+-------+-----+\n","|dataset|count|\n","+-------+-----+\n","|  train|  350|\n","|   test|  650|\n","+-------+-----+\n","\n"]}]},{"cell_type":"code","source":["data.groupby('rel').count().show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Kk68iX-K6c0C","executionInfo":{"status":"ok","timestamp":1687514384614,"user_tz":-60,"elapsed":1004,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"af593870-7543-4dbf-d069-9f85a4e27eab"},"execution_count":11,"outputs":[{"output_type":"stream","name":"stdout","text":["+-----+-----+\n","|  rel|count|\n","+-----+-----+\n","| TrIP|   14|\n","| TrAP|  164|\n","| TeCP|   26|\n","|    O|  414|\n","|TrNAP|   14|\n","| TrCP|   28|\n","|  PIP|  153|\n","| TrWP|   11|\n","| TeRP|  176|\n","+-----+-----+\n","\n"]}]},{"cell_type":"code","source":["#from sparknlp_jsl.training import REDatasetHelper\n","\n","# map entity columns to dataset columns\n","column_map = {\n","    \"begin1\": \"firstCharEnt1\",\n","    \"end1\": \"lastCharEnt1\",\n","    \"begin2\": \"firstCharEnt2\",\n","    \"end2\": \"lastCharEnt2\",\n","    \"chunk1\": \"chunk1\",\n","    \"chunk2\": \"chunk2\",\n","    \"label1\": \"label1\",\n","    \"label2\": \"label2\"\n","}\n","\n","# apply preprocess function to dataframe\n","data = medical.REDatasetHelper(data).create_annotation_column(\n","    column_map,\n","    ner_column_name=\"train_ner_chunks\" # optional, default train_ner_chunks\n",")\n","\n","train_data = data.where(\"dataset='train'\")\n","test_data = data.where(\"dataset='test'\")"],"metadata":{"id":"8W9uUW5X6cuo","executionInfo":{"status":"ok","timestamp":1687514385891,"user_tz":-60,"elapsed":1280,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":12,"outputs":[]},{"cell_type":"code","source":["data.show(10)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"S-UM-9J66hw6","executionInfo":{"status":"ok","timestamp":1687514387043,"user_tz":-60,"elapsed":1154,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"d03ee2b3-11a9-46da-d821-1a09550c4c7a"},"execution_count":13,"outputs":[{"output_type":"stream","name":"stdout","text":["+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+---------+\n","|lastCharEnt1|   label1|    train_ner_chunks|lastCharEnt2|firstCharEnt2|              chunk1|  rel|            sentence|              chunk2|dataset|firstCharEnt1|   label2|\n","+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+---------+\n","|          64|     test|[{chunk, 49, 64, ...|          84|           75|    respiratory rate|    O|VITAL SIGNS - Tem...|          saturation|   test|           49|     test|\n","|           9|treatment|[{chunk, 3, 9, lo...|          42|           34|             lotions|TrNAP|No lotions , crea...|           incisions|   test|            3|  problem|\n","|          54|treatment|[{chunk, 11, 54, ...|          68|           58|expected long ter...|    O|Because of expect...|         a picc line|  train|           11|treatment|\n","|          31|  problem|[{chunk, 16, 31, ...|          92|           82|    light-headedness|  PIP|She states this l...|         diaphoresis|  train|           16|  problem|\n","|         239|  problem|[{chunk, 196, 239...|         176|          145|an inferior and r...|  PIP|Initial electroca...|1-mm st depressio...|   test|          196|  problem|\n","|          20|     test|[{chunk, 1, 20, a...|          71|           54|abdominal ultrasound| TeRP|Abdominal ultraso...|  gallbladder sludge|   test|            1|  problem|\n","|         133|treatment|[{chunk, 99, 133,...|         173|          139|ir placement of a...| TrAP|At that time he w...|his abdominopelvi...|   test|           99|  problem|\n","|         154|treatment|[{chunk, 143, 154...|         213|          195|        reintubation| TrAP|He was transferre...| worsening pneumonia|   test|          143|  problem|\n","|          23|     test|[{chunk, 12, 23, ...|          47|           35|        previous w/u| TeRP|5) Anemia - previ...|       elev ferritin|   test|           12|  problem|\n","|         179|treatment|[{chunk, 161, 179...|         156|          149| po pain medications| TrAP|The patient &apos...|            his pain|  train|          161|  problem|\n","+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+---------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"markdown","source":["## **✅ Graph Creation**\n","\n","We will use `TFGraphBuilder` annotator which can be used to create graphs automatically in the model training pipeline.\n","\n","`TFGraphBuilder` inspects the data and creates the proper graph if a suitable version of TensorFlow is available. The graph is stored in the defined folder and loaded by the approach.\n","\n","You can also create a custom graph by using `tf_graph` module in Spark NLP for Healthcare."],"metadata":{"id":"HFINJvvX6mRa"}},{"cell_type":"code","source":["!pip install -q tensorflow==2.11.0 tensorflow-addons"],"metadata":{"id":"2Ge5vd5H6hq7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["'''\n","# custom graph\n","\n","medical.tf_graph.print_model_params(\"relation_extraction\")\n","\n","medical.tf_graph.build(\"relation_extraction\",\n","             build_params={\"input_dim\": 10000,\n","                          \"output_dim\": 10,\n","                          'batch_norm':1,\n","                          \"hidden_layers\": [300, 200],\n","                          \"hidden_act\": \"relu\",\n","                          'hidden_act_l2':1},\n","                          'model_location'=\".\",\n","                          'model_filename'=\"re_in6000D_out10.pb\")\n","'''\n","\n","'''\n","# ready to use graph\n","!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/generic_classifier_graph/RE_in1200D_out20.pb\n","'''"],"metadata":{"id":"U6tn30Zo6hk5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["graph_folder= \"./tf_graphs\""],"metadata":{"id":"EZlcRtXy6hes","executionInfo":{"status":"ok","timestamp":1687514454814,"user_tz":-60,"elapsed":7,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":15,"outputs":[]},{"cell_type":"code","source":["re_graph_builder = medical.TFGraphBuilder()\\\n","    .setModelName(\"relation_extraction\")\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"]) \\\n","    .setLabelColumn(\"rel\")\\\n","    .setGraphFolder(graph_folder)\\\n","    .setGraphFile(\"re_graph.pb\")\\\n","    .setHiddenLayers([300, 200])\\\n","    .setHiddenAct(\"relu\")\\\n","    .setHiddenActL2(True)\\\n","    .setHiddenWeightsL2(False)\\\n","    .setBatchNorm(False)"],"metadata":{"id":"spzfA4FE33Cx","executionInfo":{"status":"ok","timestamp":1687514454815,"user_tz":-60,"elapsed":6,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":["## **🔎 Parameters**\n"],"metadata":{"id":"b2YJehUKMhb0"}},{"cell_type":"markdown","source":["- `FromEntity`: (begin_col: str, end_col: str, label_col: str) Sets from entity\n","\n",">`begin_col` Column that has a reference of where the chunk begins </br>\n","\n","> `end_col`: Column that has a reference of where the chunk ends\n","\n","> `label_col`: Column that has a reference what are the type of chunk\n","\n","- `ToEntity`: (begin_col: str, end_col: str, label_col: str) Sets to entity\n","\n","> `begin_col` Column that has a reference of where the chunk begins </br>\n","\n","> `end_col`: Column that has a reference of where the chunk ends\n","\n","> `label_col`: Column that has a reference what are the type of chunk\n","\n","- `CustomLabels`: (labels: dict[str, str]) Sets custom relation labels\n","\n","> `labels`: Dictionary which maps old to new labels\n","\n","- `RelationDirectionCol`: (col: str) Relation direction column (possible values are: \"none\", \"left\" or \"right\"). If this parameter is not set, the model will not have direction between the relation of the entities\n","\n","> `col` Column contains the relation direction values\n","\n","- `PretrainedModelPath` (value: str) Path to an already trained model saved to disk, which is used as a starting point for training the new model\n","\n","- `ОverrideExistingLabels` (bool) Whether to override already learned labels when using a pretrained model to initialize the new model. Default is ‘true’\n","\n","- `batchSize`: (Int) Size for each batch in the optimization process\n","\n","- `EpochsNumber` (Int) Maximum number of epochs to train\n","\n","- `Dropout`: (Float) Dropout at the output of each layer\n","\n","- `LearningRate`: (Float) Learning rate for the optimization process\n","\n","- `OutputLogsPath`: (Str) Folder path to save training logs. If no path is specified, the logs won't be stored in disk. The path can be a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).\n","\n","- `ModelFile`: (Str) The path to the Tensorflow graph\n","\n","- `FixImbalance` (Float) Fix the imbalance in the training set by replicating examples of under represented categories\n","\n","- `ValidationSplit` (Float) The proportion of training dataset to be used as validation set\n","\n","- `OverrideExistingLabels` (Boolean) Controls whether to override already learned lebels when using a pretrained model to initialize the new model. A value of true will override existing labels\n","\n","- `MultiClass` (Boolean) If multiClass is set, the model will return all the labels with corresponding scores. By default, multiClass is false.\n","\n","- `ModelFile` (Str) Location of file of the model used for classification\n","\n","- `MaxSyntacticDistance` (Int) Maximal syntactic distance, as threshold (Default: 0)\n","\n","\n","\n","\n","\n","\n","\n","\n","\n","\n","\n","\n"],"metadata":{"id":"oidLDoS94asU"}},{"cell_type":"markdown","source":["## **🦾 Model Training**"],"metadata":{"id":"qZOEBPodEgjP"}},{"cell_type":"markdown","source":["Now, we will define `RelationExtractionApproach` and generate a pipeline along with required annotators and TFGraph."],"metadata":{"id":"csJ9LhL8re-R"}},{"cell_type":"code","source":["documenter = nlp.DocumentAssembler()\\\n","    .setInputCol(\"sentence\")\\\n","    .setOutputCol(\"sentences\")\n","\n","tokenizer = nlp.Tokenizer()\\\n","    .setInputCols([\"sentences\"])\\\n","    .setOutputCol(\"tokens\")\\\n","\n","words_embedder = nlp.WordEmbeddingsModel()\\\n","    .pretrained(\"embeddings_clinical\", \"en\", \"clinical/models\")\\\n","    .setInputCols([\"sentences\", \"tokens\"])\\\n","    .setOutputCol(\"embeddings\")\n","\n","pos_tagger = nlp.PerceptronModel()\\\n","    .pretrained(\"pos_clinical\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentences\", \"tokens\"])\\\n","    .setOutputCol(\"pos_tags\")\n","\n","dependency_parser = nlp.DependencyParserModel()\\\n","    .pretrained(\"dependency_conllu\", \"en\")\\\n","    .setInputCols([\"sentences\", \"pos_tags\", \"tokens\"])\\\n","    .setOutputCol(\"dependencies\")\n","\n","# set training params and upload model graph (see ../Healthcare/8.Generic_Classifier.ipynb)\n","reApproach = medical.RelationExtractionApproach()\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\\\n","    .setLabelColumn(\"rel\")\\\n","    .setEpochsNumber(80)\\\n","    .setBatchSize(200)\\\n","    .setDropout(0.5)\\\n","    .setLearningRate(0.001)\\\n","    .setModelFile(f\"{graph_folder}/re_graph.pb\")\\\n","    .setFixImbalance(True)\\\n","    .setFromEntity(\"firstCharEnt1\", \"lastCharEnt1\", \"label1\")\\\n","    .setToEntity(\"firstCharEnt2\", \"lastCharEnt2\", \"label2\")\\\n","    .setOutputLogsPath('/content')\n","\n","finisher = nlp.Finisher()\\\n","    .setInputCols([\"relations\"])\\\n","    .setOutputCols([\"relations_out\"])\\\n","    .setCleanAnnotations(False)\\\n","    .setValueSplitSymbol(\",\")\\\n","    .setAnnotationSplitSymbol(\",\")\\\n","    .setOutputAsArray(False)\n","\n","train_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    re_graph_builder,\n","    reApproach,\n","    finisher\n","])"],"metadata":{"id":"njQKuLQCxNJC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687514563935,"user_tz":-60,"elapsed":109125,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"ecb6263e-4789-4cb9-c1e8-7a09e1876477"},"execution_count":17,"outputs":[{"output_type":"stream","name":"stdout","text":["embeddings_clinical download started this may take some time.\n","Approximate size to download 1.6 GB\n","[OK!]\n","pos_clinical download started this may take some time.\n","Approximate size to download 1.5 MB\n","[OK!]\n","dependency_conllu download started this may take some time.\n","Approximate size to download 16.7 MB\n","[OK!]\n"]}]},{"cell_type":"code","source":["%%time\n","rel_model = train_pipeline.fit(train_data)"],"metadata":{"id":"Axs1zT8lxNEk","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687514581508,"user_tz":-60,"elapsed":17578,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"6a56e987-29fd-4175-a1bf-b6c51e64e691"},"execution_count":18,"outputs":[{"output_type":"stream","name":"stdout","text":["TF Graph Builder configuration:\n","Model name: relation_extraction\n","Graph folder: ./tf_graphs\n","Graph file name: re_graph.pb\n","Build params: {'input_dim': 1149, 'output_dim': 27, 'hidden_layers': [300, 200], 'hidden_act': 'relu', 'hidden_act_l2': True, 'hidden_weights_l2': False, 'batch_norm': False}\n","relation_extraction graph exported to ./tf_graphs/re_graph.pb\n","CPU times: user 3.79 s, sys: 298 ms, total: 4.08 s\n","Wall time: 17.7 s\n"]}]},{"cell_type":"code","source":["rel_model.stages"],"metadata":{"id":"S15PuZGQxTbn","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687514581509,"user_tz":-60,"elapsed":16,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"5fd71c83-1dfb-46b1-82d2-087926a21244"},"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[DocumentAssembler_6af920082bd8,\n"," REGEX_TOKENIZER_09748e8634ee,\n"," WORD_EMBEDDINGS_MODEL_9004b1d00302,\n"," POS_6f55785005bf,\n"," dependency_e7755462ba78,\n"," TFGraphBuilderModel_a2779a0336f2,\n"," RelationExtractionModel_d6427987b089,\n"," Finisher_4cc624e0fa46]"]},"metadata":{},"execution_count":19}]},{"cell_type":"code","source":["rel_model.stages[-2]"],"metadata":{"id":"HaiI0t8BxTXr","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687514581509,"user_tz":-60,"elapsed":14,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"bbb89bff-7535-4bf5-eca5-286d824f1790"},"execution_count":20,"outputs":[{"output_type":"execute_result","data":{"text/plain":["RelationExtractionModel_d6427987b089"]},"metadata":{},"execution_count":20}]},{"cell_type":"markdown","source":["Saving the model"],"metadata":{"id":"zMOGw-ghxKcm"}},{"cell_type":"code","source":["rel_model.stages[-2].write().overwrite().save('custom_RE_model')"],"metadata":{"id":"R5p2WvAov7HE","executionInfo":{"status":"ok","timestamp":1687514581908,"user_tz":-60,"elapsed":403,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":21,"outputs":[]},{"cell_type":"markdown","source":["### Evaluating The Performance of the Model\n","\n","We will create a pipeline and use our trained RE model by calling with `.load` method. Then we will get predictions by transforming our test set on this pipeline."],"metadata":{"id":"Kts8Mzc2xzTS"}},{"cell_type":"code","source":["customReModel = medical.RelationExtractionModel.load(\"custom_RE_model\")\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations_pred\")\\\n","    .setMaxSyntacticDistance(0)\n","\n","test_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    customReModel])\n","\n","test_results = test_pipeline.fit(test_data).transform(test_data)"],"metadata":{"id":"oOlpwph_xXb3","executionInfo":{"status":"ok","timestamp":1687515811734,"user_tz":-60,"elapsed":3835,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":23,"outputs":[]},{"cell_type":"code","source":["test_results.show(5)"],"metadata":{"id":"9IC2nPRIxXWp","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687515892351,"user_tz":-60,"elapsed":2198,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"a7cd3fce-4f41-4c6b-a55f-e25f1a330578"},"execution_count":24,"outputs":[{"output_type":"stream","name":"stdout","text":["+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+-------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","|lastCharEnt1|   label1|    train_ner_chunks|lastCharEnt2|firstCharEnt2|              chunk1|  rel|            sentence|              chunk2|dataset|firstCharEnt1| label2|           sentences|              tokens|          embeddings|            pos_tags|        dependencies|      relations_pred|\n","+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+-------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","|          64|     test|[{chunk, 49, 64, ...|          84|           75|    respiratory rate|    O|VITAL SIGNS - Tem...|          saturation|   test|           49|   test|[{document, 0, 10...|[{token, 0, 4, VI...|[{word_embeddings...|[{pos, 0, 4, NN, ...|[{dependency, 0, ...|[{category, 49, 8...|\n","|           9|treatment|[{chunk, 3, 9, lo...|          42|           34|             lotions|TrNAP|No lotions , crea...|           incisions|   test|            3|problem|[{document, 0, 44...|[{token, 0, 1, No...|[{word_embeddings...|[{pos, 0, 1, DD, ...|[{dependency, 0, ...|[{category, 3, 42...|\n","|         239|  problem|[{chunk, 196, 239...|         176|          145|an inferior and r...|  PIP|Initial electroca...|1-mm st depressio...|   test|          196|problem|[{document, 0, 24...|[{token, 0, 6, In...|[{word_embeddings...|[{pos, 0, 6, JJ, ...|[{dependency, 0, ...|[{category, 145, ...|\n","|          20|     test|[{chunk, 1, 20, a...|          71|           54|abdominal ultrasound| TeRP|Abdominal ultraso...|  gallbladder sludge|   test|            1|problem|[{document, 0, 95...|[{token, 0, 8, Ab...|[{word_embeddings...|[{pos, 0, 8, JJ, ...|[{dependency, 0, ...|[{category, 1, 71...|\n","|         133|treatment|[{chunk, 99, 133,...|         173|          139|ir placement of a...| TrAP|At that time he w...|his abdominopelvi...|   test|           99|problem|[{document, 0, 17...|[{token, 0, 1, At...|[{word_embeddings...|[{pos, 0, 1, II, ...|[{dependency, 0, ...|[{category, 99, 1...|\n","+------------+---------+--------------------+------------+-------------+--------------------+-----+--------------------+--------------------+-------+-------------+-------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n","only showing top 5 rows\n","\n"]}]},{"cell_type":"markdown","source":["When you check the `relations_pred` column, you can see that some of the relations between the chunk pairs were not detected this time.\n","\n","Now we will get metrics using the ground truth (`rel`) and prediction (`relations_pred`) columns in the result dataframe."],"metadata":{"id":"VjKncyMHyctT"}},{"cell_type":"code","source":["pd_test_results = test_results.select('rel', 'relations_pred.result').toPandas()\n","pd_test_results.head()"],"metadata":{"id":"FTzrGzU2xXO9","colab":{"base_uri":"https://localhost:8080/","height":204},"executionInfo":{"status":"ok","timestamp":1687515978067,"user_tz":-60,"elapsed":9381,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"48913bea-77b0-47a4-fa53-cbeefbfaa4f7"},"execution_count":25,"outputs":[{"output_type":"execute_result","data":{"text/plain":["     rel  result\n","0      O     [O]\n","1  TrNAP     [O]\n","2    PIP   [PIP]\n","3   TeRP  [TeRP]\n","4   TrAP  [TrAP]"],"text/html":["\n","  <div id=\"df-24496329-f7ba-4ffb-8fa9-50159c907513\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rel</th>\n","      <th>result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>TrNAP</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>PIP</td>\n","      <td>[PIP]</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>TeRP</td>\n","      <td>[TeRP]</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>TrAP</td>\n","      <td>[TrAP]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-24496329-f7ba-4ffb-8fa9-50159c907513')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-24496329-f7ba-4ffb-8fa9-50159c907513 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-24496329-f7ba-4ffb-8fa9-50159c907513');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":25}]},{"cell_type":"markdown","source":["We will explode the `result` column and fill null values as `O` label."],"metadata":{"id":"OqH4Z8F0yoko"}},{"cell_type":"code","source":["pd_test_results = pd_test_results.explode(\"result\").fillna(\"O\")\n","pd_test_results.result.value_counts()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cZgQRfb1yjpT","executionInfo":{"status":"ok","timestamp":1687515998190,"user_tz":-60,"elapsed":413,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"fd742798-f39b-460f-e764-4bc5cc59fefe"},"execution_count":26,"outputs":[{"output_type":"execute_result","data":{"text/plain":["O        343\n","PIP      106\n","TeRP      99\n","TrAP      96\n","TeCP       2\n","TrNAP      2\n","TrCP       1\n","TrIP       1\n","Name: result, dtype: int64"]},"metadata":{},"execution_count":26}]},{"cell_type":"code","source":["pd_test_results.rel.value_counts()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Wc7Tf6vXyjkT","executionInfo":{"status":"ok","timestamp":1687516008730,"user_tz":-60,"elapsed":410,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"e1dbfd41-7a09-413e-ac4e-cd4169912b54"},"execution_count":27,"outputs":[{"output_type":"execute_result","data":{"text/plain":["O        274\n","TeRP     116\n","TrAP     108\n","PIP       88\n","TrCP      20\n","TeCP      15\n","TrIP      11\n","TrNAP      9\n","TrWP       9\n","Name: rel, dtype: int64"]},"metadata":{},"execution_count":27}]},{"cell_type":"code","source":["from sklearn.metrics import classification_report\n","print(classification_report(pd_test_results[\"rel\"], pd_test_results[\"result\"]))"],"metadata":{"id":"-s9OQDDBv7Cs","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1687516028330,"user_tz":-60,"elapsed":1852,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"c440fcb2-d34f-4d53-ba44-14d2a5f7b3c0"},"execution_count":28,"outputs":[{"output_type":"stream","name":"stdout","text":["              precision    recall  f1-score   support\n","\n","           O       0.66      0.83      0.74       274\n","         PIP       0.59      0.72      0.65        88\n","        TeCP       0.50      0.07      0.12        15\n","        TeRP       0.76      0.65      0.70       116\n","        TrAP       0.64      0.56      0.60       108\n","        TrCP       0.00      0.00      0.00        20\n","        TrIP       1.00      0.09      0.17        11\n","       TrNAP       0.00      0.00      0.00         9\n","        TrWP       0.00      0.00      0.00         9\n","\n","    accuracy                           0.66       650\n","   macro avg       0.46      0.32      0.33       650\n","weighted avg       0.63      0.66      0.63       650\n","\n"]}]},{"cell_type":"markdown","source":["### Training by Setting the Relation Direction"],"metadata":{"id":"NaQboMiKy6-z"}},{"cell_type":"markdown","source":["We have `setRelationDirectionCol` parameter that is used during training with a new separate column that specifies the relationship directions. The column should contain one of the following values:\n","\n"," - `rightwards`: The first entity in the text is also the first argument of the relation (as well as the second entity in the text is the second argument). In other words, the relation arguments are ordered *left to right* in the text.\n"," - `leftwards`: The first entity in the text is the second argument of the relation (and the second entity in the text is the first argument).\n"," - `both`: Order doesn't matter (relation is symmetric)."],"metadata":{"id":"cviClFUBy_6Y"}},{"cell_type":"markdown","source":["Let's modify our example training dataset accordingly"],"metadata":{"id":"mindw9_0zbJE"}},{"cell_type":"code","source":["data.columns"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"4JsMQ4oUy3z3","executionInfo":{"status":"ok","timestamp":1687516241507,"user_tz":-60,"elapsed":415,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"40a3748f-e04b-47e0-be2c-41071804c302"},"execution_count":29,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['lastCharEnt1',\n"," 'label1',\n"," 'train_ner_chunks',\n"," 'lastCharEnt2',\n"," 'firstCharEnt2',\n"," 'chunk1',\n"," 'rel',\n"," 'sentence',\n"," 'chunk2',\n"," 'dataset',\n"," 'firstCharEnt1',\n"," 'label2']"]},"metadata":{},"execution_count":29}]},{"cell_type":"code","source":["import pyspark.sql.types as T\n","import pyspark.sql.functions as F\n","\n","@F.udf(T.StringType())\n","def encodeRelationDirection(rel, begin1, begin2):\n","    if rel != \"O\":\n","        if begin1 > begin2:\n","            return \"leftwards\"\n","        else:\n","            return \"rightwards\"\n","    else:\n","        return \"both\"\n","\n","\n","\n","data = data.withColumn(\"rel_dir\", encodeRelationDirection(\"rel\", \"firstCharEnt1\", \"lastCharEnt2\"))\n","\n","train_data = data.where(\"dataset='train'\")\n","test_data = data.where(\"dataset='test'\")"],"metadata":{"id":"rUw8HmdpznRw","executionInfo":{"status":"ok","timestamp":1687516250475,"user_tz":-60,"elapsed":429,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":30,"outputs":[]},{"cell_type":"markdown","source":["Checking the new training data with **rel_dir** column"],"metadata":{"id":"RskOJWcezqXY"}},{"cell_type":"code","source":["train_data.select(\"chunk1\",\"label1\",\"label2\",\"chunk2\",\"rel\",\"rel_dir\").show(10)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"J6zCKyHAznMT","executionInfo":{"status":"ok","timestamp":1687516269847,"user_tz":-60,"elapsed":788,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"6ba9730d-e7ba-4cc5-f765-9b1ffff18440"},"execution_count":31,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+---------+---------+--------------------+----+----------+\n","|              chunk1|   label1|   label2|              chunk2| rel|   rel_dir|\n","+--------------------+---------+---------+--------------------+----+----------+\n","|expected long ter...|treatment|treatment|         a picc line|   O|      both|\n","|    light-headedness|  problem|  problem|         diaphoresis| PIP|rightwards|\n","| po pain medications|treatment|  problem|            his pain|TrAP| leftwards|\n","|bilateral pleural...|  problem|  problem|increased work of...| PIP|rightwards|\n","|    her urine output|     test|  problem|           decreased|TeRP|rightwards|\n","|his psychiatric i...|  problem|  problem|his neurologic in...| PIP|rightwards|\n","|   white blood cells|     test|     test|     red blood cells|   O|      both|\n","|            chloride|     test|     test|                 bun|   O|      both|\n","|     further work-up|     test|  problem|his neurologic co...|TeCP|rightwards|\n","|         four liters|treatment|     test|      blood pressure|   O|      both|\n","+--------------------+---------+---------+--------------------+----+----------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"code","source":["train_data\\\n","    .selectExpr(\"concat(label1, \\\", \\\", label2) AS args\", \"rel\", \"rel_dir\")\\\n","    .groupBy(\"rel\", \"args\", \"rel_dir\").count().where(\"count > 10\").orderBy(\"rel\").show(1000, truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fTSNiZjJzuUl","executionInfo":{"status":"ok","timestamp":1687516279951,"user_tz":-60,"elapsed":1871,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"2d0e0f91-01f6-41f9-b621-b2e0305fdf85"},"execution_count":32,"outputs":[{"output_type":"stream","name":"stdout","text":["+----+--------------------+----------+-----+\n","|rel |args                |rel_dir   |count|\n","+----+--------------------+----------+-----+\n","|O   |treatment, treatment|both      |26   |\n","|O   |test, test          |both      |59   |\n","|O   |problem, problem    |both      |38   |\n","|PIP |problem, problem    |rightwards|47   |\n","|PIP |problem, problem    |leftwards |18   |\n","|TeRP|test, problem       |rightwards|57   |\n","|TrAP|treatment, problem  |rightwards|29   |\n","|TrAP|treatment, problem  |leftwards |27   |\n","+----+--------------------+----------+-----+\n","\n"]}]},{"cell_type":"markdown","source":["Generating a new `RelationExtractionApproach()` with `.setRelationDirectionCol(\"rel_dir\")` and training a new model."],"metadata":{"id":"f7HmtM9Xzy7U"}},{"cell_type":"code","source":["documenter = nlp.DocumentAssembler()\\\n","    .setInputCol(\"sentence\")\\\n","    .setOutputCol(\"sentences\")\n","\n","tokenizer = nlp.Tokenizer()\\\n","    .setInputCols([\"sentences\"])\\\n","    .setOutputCol(\"tokens\")\\\n","\n","words_embedder = nlp.WordEmbeddingsModel()\\\n","    .pretrained(\"embeddings_clinical\", \"en\", \"clinical/models\")\\\n","    .setInputCols([\"sentences\", \"tokens\"])\\\n","    .setOutputCol(\"embeddings\")\n","\n","pos_tagger = nlp.PerceptronModel()\\\n","    .pretrained(\"pos_clinical\", \"en\", \"clinical/models\") \\\n","    .setInputCols([\"sentences\", \"tokens\"])\\\n","    .setOutputCol(\"pos_tags\")\n","\n","dependency_parser = nlp.DependencyParserModel()\\\n","    .pretrained(\"dependency_conllu\", \"en\")\\\n","    .setInputCols([\"sentences\", \"pos_tags\", \"tokens\"])\\\n","    .setOutputCol(\"dependencies\")\n","\n","reApproach = medical.RelationExtractionApproach()\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\\\n","    .setLabelColumn(\"rel\")\\\n","    .setEpochsNumber(80)\\\n","    .setBatchSize(200)\\\n","    .setDropout(0.5)\\\n","    .setLearningRate(0.001)\\\n","    .setModelFile(f\"{graph_folder}/re_graph.pb\")\\\n","    .setFixImbalance(True)\\\n","    .setFromEntity(\"firstCharEnt1\", \"lastCharEnt1\", \"label1\")\\\n","    .setToEntity(\"firstCharEnt2\", \"lastCharEnt2\", \"label2\")\\\n","    .setOutputLogsPath('/rel_log')\\\n","    .setRelationDirectionCol(\"rel_dir\")\n","\n","train_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    re_graph_builder,\n","    reApproach,\n","    finisher\n","])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"m1vObwUhy3vO","executionInfo":{"status":"ok","timestamp":1687516319048,"user_tz":-60,"elapsed":10470,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"cd666fa0-6bb0-4eb6-fcb3-48f54ce280a4"},"execution_count":33,"outputs":[{"output_type":"stream","name":"stdout","text":["embeddings_clinical download started this may take some time.\n","Approximate size to download 1.6 GB\n","[OK!]\n","pos_clinical download started this may take some time.\n","Approximate size to download 1.5 MB\n","[OK!]\n","dependency_conllu download started this may take some time.\n","Approximate size to download 16.7 MB\n","[OK!]\n"]}]},{"cell_type":"code","source":["%%time\n","rel_model = train_pipeline.fit(train_data)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ocMIP5Ulz5IU","executionInfo":{"status":"ok","timestamp":1687516329469,"user_tz":-60,"elapsed":7280,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"bf21b76b-178d-4bf1-a347-4bee44d7c29a"},"execution_count":34,"outputs":[{"output_type":"stream","name":"stdout","text":["TF Graph Builder configuration:\n","Model name: relation_extraction\n","Graph folder: ./tf_graphs\n","Graph file name: re_graph.pb\n","Build params: {'input_dim': 1149, 'output_dim': 27, 'hidden_layers': [300, 200], 'hidden_act': 'relu', 'hidden_act_l2': True, 'hidden_weights_l2': False, 'batch_norm': False}\n","relation_extraction graph exported to ./tf_graphs/re_graph.pb\n","CPU times: user 766 ms, sys: 60.2 ms, total: 827 ms\n","Wall time: 6.89 s\n"]}]},{"cell_type":"code","source":["rel_model.stages"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"TbL3UxmKz5Bw","executionInfo":{"status":"ok","timestamp":1687516329470,"user_tz":-60,"elapsed":8,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"958357a9-a398-4267-cfba-c607767eebd5"},"execution_count":35,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[DocumentAssembler_b9ce0bf136eb,\n"," REGEX_TOKENIZER_d6c09339511d,\n"," WORD_EMBEDDINGS_MODEL_9004b1d00302,\n"," POS_6f55785005bf,\n"," dependency_e7755462ba78,\n"," TFGraphBuilderModel_f3f1c4b0c3bc,\n"," RelationExtractionModel_8664e0f7a637,\n"," Finisher_4cc624e0fa46]"]},"metadata":{},"execution_count":35}]},{"cell_type":"markdown","source":["Saving the model"],"metadata":{"id":"ancLD-aN0Et5"}},{"cell_type":"code","source":["rel_model.stages[-2].write().overwrite().save('rel_dir_RE_model')"],"metadata":{"id":"vqww67aTz48w","executionInfo":{"status":"ok","timestamp":1687516338679,"user_tz":-60,"elapsed":873,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":36,"outputs":[]},{"cell_type":"markdown","source":["**Evaluating The Performance of the Model Trained with Directions**"],"metadata":{"id":"tdfzyTNc0Aa5"}},{"cell_type":"code","source":["customReModelDir = medical.RelationExtractionModel.load(\"rel_dir_RE_model\")\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations_pred\")\\\n","    .setMaxSyntacticDistance(0)\n","\n","test_pipeline_dir = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    customReModelDir])\n","\n","test_results_dir = test_pipeline_dir.fit(test_data).transform(test_data)"],"metadata":{"id":"kKSKh0gX0AFt","executionInfo":{"status":"ok","timestamp":1687516378515,"user_tz":-60,"elapsed":3998,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":37,"outputs":[]},{"cell_type":"markdown","source":["Now we will get metrics using the ground truth (`rel`) and prediction (`relations_pred`) columns in the result dataframe."],"metadata":{"id":"gLdSfCo80IZ3"}},{"cell_type":"code","source":["pd_test_results_dir = test_results_dir.select('rel', 'relations_pred.result').toPandas()\n","pd_test_results_dir.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"BrXJKmy8z_-O","executionInfo":{"status":"ok","timestamp":1687516401550,"user_tz":-60,"elapsed":8503,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"1c6298dd-e525-46e6-9354-632724340db4"},"execution_count":38,"outputs":[{"output_type":"execute_result","data":{"text/plain":["     rel  result\n","0      O     [O]\n","1  TrNAP     [O]\n","2    PIP   [PIP]\n","3   TeRP  [TeRP]\n","4   TrAP  [TrAP]"],"text/html":["\n","  <div id=\"df-5ec4e417-cc5e-48a4-a2d5-d38560f21539\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rel</th>\n","      <th>result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>TrNAP</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>PIP</td>\n","      <td>[PIP]</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>TeRP</td>\n","      <td>[TeRP]</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>TrAP</td>\n","      <td>[TrAP]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5ec4e417-cc5e-48a4-a2d5-d38560f21539')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5ec4e417-cc5e-48a4-a2d5-d38560f21539 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5ec4e417-cc5e-48a4-a2d5-d38560f21539');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":38}]},{"cell_type":"markdown","source":["We will explode the `result` column and fill null values as `O` label and get metrics."],"metadata":{"id":"eUQx0aBv0Nq2"}},{"cell_type":"code","source":["from sklearn.metrics import classification_report\n","\n","pd_test_results_dir = pd_test_results_dir.explode(\"result\").fillna(\"O\")\n","\n","print(classification_report(pd_test_results_dir[\"rel\"], pd_test_results_dir[\"result\"]))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"e9bcFLNk0Nao","executionInfo":{"status":"ok","timestamp":1687516414974,"user_tz":-60,"elapsed":380,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"6202ad63-7d72-4b31-b7d0-12b2f2d85fe5"},"execution_count":39,"outputs":[{"output_type":"stream","name":"stdout","text":["              precision    recall  f1-score   support\n","\n","           O       0.67      0.87      0.76       274\n","         PIP       0.59      0.64      0.61        88\n","        TeCP       0.00      0.00      0.00        15\n","        TeRP       0.85      0.61      0.71       116\n","        TrAP       0.70      0.68      0.69       108\n","        TrCP       0.00      0.00      0.00        20\n","        TrIP       0.00      0.00      0.00        11\n","       TrNAP       0.14      0.11      0.12         9\n","        TrWP       0.00      0.00      0.00         9\n","\n","    accuracy                           0.68       650\n","   macro avg       0.33      0.32      0.32       650\n","weighted avg       0.63      0.68      0.64       650\n","\n"]}]},{"cell_type":"markdown","source":["As you can see, this model is slightly better than the previous one."],"metadata":{"id":"cIGLW9Ru0XH2"}},{"cell_type":"markdown","source":["### Resume RelationExtractionApproach Model Training\n","\n","Steps:\n","- Train a new model for a few epochs.\n","- Load the same model and train for more epochs on the same taxnonomy, and evaluate.\n","- Train a model already trained on a different dataset."],"metadata":{"id":"nwQfAk_b1eJa"}},{"cell_type":"markdown","source":["**Split the test data into two parts:**\n","- We keep the first part separate and use it for training the model further, as it will be totally unseen data from the same taxonomy.\n","\n","- The second part will be used to testing and evaluating"],"metadata":{"id":"jDeZFSoc2mjW"}},{"cell_type":"code","source":["(test_data_1, test_data_2) = test_data.randomSplit([0.5, 0.5], seed = 100)"],"metadata":{"id":"qgBdM1_Q058T","executionInfo":{"status":"ok","timestamp":1687517049844,"user_tz":-60,"elapsed":928,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":40,"outputs":[]},{"cell_type":"markdown","source":["#### Train a new model, pause, and resume training on the same dataset."],"metadata":{"id":"0UC9CX_f2wli"}},{"cell_type":"markdown","source":["**Create a Graph**"],"metadata":{"id":"dTSkRuzd28Wp"}},{"cell_type":"code","source":["#from sparknlp_jsl.annotator import TFGraphBuilder\n","\n","graph_folder= \"./tf_graphs\"\n","\n","re_graph_builder = medical.TFGraphBuilder()\\\n","    .setModelName(\"relation_extraction\")\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"]) \\\n","    .setLabelColumn(\"rel\")\\\n","    .setGraphFolder(graph_folder)\\\n","    .setGraphFile(\"re_graph.pb\")\\\n","    .setHiddenLayers([300, 200])\\\n","    .setHiddenAct(\"relu\")\\\n","    .setHiddenActL2(True)\\\n","    .setHiddenWeightsL2(False)\\\n","    .setBatchNorm(False)"],"metadata":{"id":"CfgerE3E052N","executionInfo":{"status":"ok","timestamp":1687517138968,"user_tz":-60,"elapsed":551,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":41,"outputs":[]},{"cell_type":"markdown","source":["**Train for 100 epochs**"],"metadata":{"id":"3vmXlpBX3EKT"}},{"cell_type":"code","source":["reApproach = medical.RelationExtractionApproach()\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\\\n","    .setLabelColumn(\"rel\")\\\n","    .setEpochsNumber(100)\\\n","    .setBatchSize(200)\\\n","    .setDropout(0.5)\\\n","    .setLearningRate(0.001)\\\n","    .setModelFile(f\"{graph_folder}/re_graph.pb\")\\\n","    .setFixImbalance(True)\\\n","    .setFromEntity(\"firstCharEnt1\", \"lastCharEnt1\", \"label1\")\\\n","    .setToEntity(\"firstCharEnt2\", \"lastCharEnt2\", \"label2\")\\\n","    .setOutputLogsPath('/content')\\\n","    .setRelationDirectionCol(\"rel_dir\")\\\n","    .setMaxSyntacticDistance(10)\n","\n","finisher = nlp.Finisher()\\\n","    .setInputCols([\"relations\"])\\\n","    .setOutputCols([\"relations_out\"])\\\n","    .setCleanAnnotations(False)\\\n","    .setValueSplitSymbol(\",\")\\\n","    .setAnnotationSplitSymbol(\",\")\\\n","    .setOutputAsArray(False)\n","\n","train_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    re_graph_builder,\n","    reApproach,\n","    finisher\n","])"],"metadata":{"id":"vYdpBlbL3BVJ","executionInfo":{"status":"ok","timestamp":1687517189307,"user_tz":-60,"elapsed":310,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":42,"outputs":[]},{"cell_type":"code","source":["%%time\n","rel_model = train_pipeline.fit(train_data)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"BKtIDvQl3BQN","executionInfo":{"status":"ok","timestamp":1687517207035,"user_tz":-60,"elapsed":7901,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"a314caf6-55c5-47e5-8e61-d9b91a4755e9"},"execution_count":43,"outputs":[{"output_type":"stream","name":"stdout","text":["TF Graph Builder configuration:\n","Model name: relation_extraction\n","Graph folder: ./tf_graphs\n","Graph file name: re_graph.pb\n","Build params: {'input_dim': 1149, 'output_dim': 27, 'hidden_layers': [300, 200], 'hidden_act': 'relu', 'hidden_act_l2': True, 'hidden_weights_l2': False, 'batch_norm': False}\n","relation_extraction graph exported to ./tf_graphs/re_graph.pb\n","CPU times: user 866 ms, sys: 35.8 ms, total: 902 ms\n","Wall time: 7.5 s\n"]}]},{"cell_type":"code","source":["result_test = rel_model.transform(test_data_2)"],"metadata":{"id":"qdUtux-c3QmA","executionInfo":{"status":"ok","timestamp":1687517214391,"user_tz":-60,"elapsed":2760,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":44,"outputs":[]},{"cell_type":"markdown","source":["**Evaluate**"],"metadata":{"id":"0qCphZSm3Tij"}},{"cell_type":"code","source":["result_test_df = result_test.select('rel', 'relations.result').toPandas()\n","result_test_df.sample(5)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"euJcBvNH3Qgz","executionInfo":{"status":"ok","timestamp":1687517233961,"user_tz":-60,"elapsed":5017,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"0ea64959-c0b4-4c3d-a821-4f86f0dc1d0d"},"execution_count":45,"outputs":[{"output_type":"execute_result","data":{"text/plain":["      rel  result\n","69   TeRP     [O]\n","191   PIP   [PIP]\n","242     O     [O]\n","291  TrAP  [TrAP]\n","70   TeRP     [O]"],"text/html":["\n","  <div id=\"df-ddc7e866-34ce-43a2-8c58-662ad5ec068a\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rel</th>\n","      <th>result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>69</th>\n","      <td>TeRP</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>191</th>\n","      <td>PIP</td>\n","      <td>[PIP]</td>\n","    </tr>\n","    <tr>\n","      <th>242</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>291</th>\n","      <td>TrAP</td>\n","      <td>[TrAP]</td>\n","    </tr>\n","    <tr>\n","      <th>70</th>\n","      <td>TeRP</td>\n","      <td>[O]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ddc7e866-34ce-43a2-8c58-662ad5ec068a')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-ddc7e866-34ce-43a2-8c58-662ad5ec068a button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-ddc7e866-34ce-43a2-8c58-662ad5ec068a');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":45}]},{"cell_type":"code","source":["from sklearn.metrics import classification_report\n","\n","pd_test_results = result_test_df.explode(\"result\").fillna(\"O\")\n","\n","print(classification_report(pd_test_results[\"rel\"], pd_test_results[\"result\"]))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"SoxCLzlL3X8r","executionInfo":{"status":"ok","timestamp":1687517235183,"user_tz":-60,"elapsed":5,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"3b2ebd2c-aa28-4ecd-a59e-cf5d4fe5bf26"},"execution_count":46,"outputs":[{"output_type":"stream","name":"stdout","text":["              precision    recall  f1-score   support\n","\n","           O       0.58      0.93      0.71       136\n","         PIP       0.60      0.55      0.58        38\n","        TeCP       1.00      0.25      0.40         4\n","        TeRP       0.77      0.16      0.27        61\n","        TrAP       0.67      0.65      0.66        48\n","        TrCP       0.00      0.00      0.00        12\n","        TrIP       0.00      0.00      0.00         7\n","       TrNAP       0.00      0.00      0.00         5\n","        TrWP       0.00      0.00      0.00         3\n","\n","    accuracy                           0.60       314\n","   macro avg       0.40      0.28      0.29       314\n","weighted avg       0.59      0.60      0.54       314\n","\n"]}]},{"cell_type":"markdown","source":["Save to disk"],"metadata":{"id":"Niq2vaMU3adg"}},{"cell_type":"code","source":["rel_model.stages[-2].write().overwrite().save('RE_model_100e')"],"metadata":{"id":"yLH8nDyC3X3H","executionInfo":{"status":"ok","timestamp":1687517293144,"user_tz":-60,"elapsed":369,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":48,"outputs":[]},{"cell_type":"markdown","source":["#### Train using the saved model on unseen dataset\n","\n","We use unseen data from the same taxonomy"],"metadata":{"id":"ohjIwLpv3xli"}},{"cell_type":"code","source":["reApproach_finetune = medical.RelationExtractionApproach()\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\\\n","    .setLabelColumn(\"rel\")\\\n","    .setEpochsNumber(30)\\\n","    .setBatchSize(200)\\\n","    .setDropout(0.5)\\\n","    .setLearningRate(0.001)\\\n","    .setFixImbalance(True)\\\n","    .setFromEntity(\"firstCharEnt1\", \"lastCharEnt1\", \"label1\")\\\n","    .setToEntity(\"firstCharEnt2\", \"lastCharEnt2\", \"label2\")\\\n","    .setRelationDirectionCol(\"rel_dir\")\\\n","    .setPretrainedModelPath(\"RE_model_100e\")\\\n","    .setОverrideExistingLabels(False)\n","\n","finetune_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    re_graph_builder,\n","    reApproach_finetune,\n","    finisher\n","])"],"metadata":{"id":"9Oj_0VXuy3q-","executionInfo":{"status":"ok","timestamp":1687517382563,"user_tz":-60,"elapsed":318,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":50,"outputs":[]},{"cell_type":"code","source":["%%time\n","rel_model = finetune_pipeline.fit(test_data_1)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nFRDSCQz3vIT","executionInfo":{"status":"ok","timestamp":1687517398113,"user_tz":-60,"elapsed":8105,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"1aa32ffe-6bad-4ff4-f5a6-c20e6cefccc2"},"execution_count":51,"outputs":[{"output_type":"stream","name":"stdout","text":["TF Graph Builder configuration:\n","Model name: relation_extraction\n","Graph folder: ./tf_graphs\n","Graph file name: re_graph.pb\n","Build params: {'input_dim': 1149, 'output_dim': 27, 'hidden_layers': [300, 200], 'hidden_act': 'relu', 'hidden_act_l2': True, 'hidden_weights_l2': False, 'batch_norm': False}\n","relation_extraction graph exported to ./tf_graphs/re_graph.pb\n","CPU times: user 863 ms, sys: 34 ms, total: 897 ms\n","Wall time: 7.8 s\n"]}]},{"cell_type":"code","source":["result = rel_model.transform(test_data_2)"],"metadata":{"id":"a4dlOBBu3vDe","executionInfo":{"status":"ok","timestamp":1687517403333,"user_tz":-60,"elapsed":2204,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":52,"outputs":[]},{"cell_type":"markdown","source":["**Evaluate**"],"metadata":{"id":"zaE7Y1wD4CLx"}},{"cell_type":"code","source":["result_test_df = result.select('rel', 'relations.result').toPandas()\n","result_test_df.sample(5)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"ro99gHmM3u-Y","executionInfo":{"status":"ok","timestamp":1687517426720,"user_tz":-60,"elapsed":5150,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"cfa29803-d889-40e2-d265-5f179a979a06"},"execution_count":53,"outputs":[{"output_type":"execute_result","data":{"text/plain":["      rel  result\n","87      O     [O]\n","14   TeCP  [TeRP]\n","77   TeCP  [TeRP]\n","153     O     [O]\n","132  TrAP      []"],"text/html":["\n","  <div id=\"df-01c2043c-8ca4-4a9c-b114-d5ba6b7fb7ef\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rel</th>\n","      <th>result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>87</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>TeCP</td>\n","      <td>[TeRP]</td>\n","    </tr>\n","    <tr>\n","      <th>77</th>\n","      <td>TeCP</td>\n","      <td>[TeRP]</td>\n","    </tr>\n","    <tr>\n","      <th>153</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>132</th>\n","      <td>TrAP</td>\n","      <td>[]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-01c2043c-8ca4-4a9c-b114-d5ba6b7fb7ef')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-01c2043c-8ca4-4a9c-b114-d5ba6b7fb7ef button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-01c2043c-8ca4-4a9c-b114-d5ba6b7fb7ef');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":53}]},{"cell_type":"code","source":["from sklearn.metrics import classification_report\n","\n","pd_test_results = result_test_df.explode(\"result\").fillna(\"O\")\n","\n","print(classification_report(pd_test_results[\"rel\"], pd_test_results[\"result\"]))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YkpxA7rb4B7m","executionInfo":{"status":"ok","timestamp":1687517430144,"user_tz":-60,"elapsed":416,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"51a55745-7443-4fb0-f2a5-4cb90593e954"},"execution_count":54,"outputs":[{"output_type":"stream","name":"stdout","text":["              precision    recall  f1-score   support\n","\n","           O       0.77      0.87      0.82       136\n","         PIP       0.77      0.71      0.74        38\n","        TeCP       0.00      0.00      0.00         4\n","        TeRP       0.83      0.79      0.81        61\n","        TrAP       0.63      0.75      0.69        48\n","        TrCP       0.25      0.08      0.12        12\n","        TrIP       0.67      0.29      0.40         7\n","       TrNAP       0.00      0.00      0.00         5\n","        TrWP       0.33      0.33      0.33         3\n","\n","    accuracy                           0.74       314\n","   macro avg       0.47      0.42      0.43       314\n","weighted avg       0.71      0.74      0.72       314\n","\n"]}]},{"cell_type":"markdown","source":["Save to disk"],"metadata":{"id":"3FhOKg0e4cZ4"}},{"cell_type":"code","source":["rel_model.stages[-2].write().overwrite().save('RE_model_finetuned')"],"metadata":{"id":"Bb9SIjcG4br6","executionInfo":{"status":"ok","timestamp":1687517525993,"user_tz":-60,"elapsed":1340,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":55,"outputs":[]},{"cell_type":"markdown","source":["#### Take a model trained on a different dataset (pretrained model) and train on this dataset"],"metadata":{"id":"E0xzOEp-4h4E"}},{"cell_type":"code","source":["clinical_re_Model = medical.RelationExtractionModel()\\\n","    .pretrained(\"re_clinical\", \"en\", 'clinical/models')\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\n","\n","clinical_re_Model.getClasses()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"syzN0GZw4bjY","executionInfo":{"status":"ok","timestamp":1687517585909,"user_tz":-60,"elapsed":5553,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"268263ca-fe79-4705-91c9-3fab2207489a"},"execution_count":56,"outputs":[{"output_type":"stream","name":"stdout","text":["re_clinical download started this may take some time.\n","[OK!]\n"]},{"output_type":"execute_result","data":{"text/plain":["['TrWP', 'TrNAP', 'TrCP', 'PIP', 'TeCP', 'TeRP', 'TrIP', 'TrAP', 'O']"]},"metadata":{},"execution_count":56}]},{"cell_type":"markdown","source":["**Now train a model using this model as base**"],"metadata":{"id":"ELHpio_t4wLB"}},{"cell_type":"code","source":["reApproach_finetune = medical.RelationExtractionApproach()\\\n","    .setInputCols([\"embeddings\", \"pos_tags\", \"train_ner_chunks\", \"dependencies\"])\\\n","    .setOutputCol(\"relations\")\\\n","    .setLabelColumn(\"rel\")\\\n","    .setEpochsNumber(30)\\\n","    .setBatchSize(200)\\\n","    .setDropout(0.5)\\\n","    .setLearningRate(0.001)\\\n","    .setFixImbalance(True)\\\n","    .setFromEntity(\"firstCharEnt1\", \"lastCharEnt1\", \"label1\")\\\n","    .setToEntity(\"firstCharEnt2\", \"lastCharEnt2\", \"label2\")\\\n","    .setPretrainedModelPath(\"/root/cache_pretrained/re_clinical_en_2.5.5_2.4_1596928426753\")\\\n","    .setОverrideExistingLabels(False)\n","\n","finetune_pipeline = nlp.Pipeline(stages=[\n","    documenter,\n","    tokenizer,\n","    words_embedder,\n","    pos_tagger,\n","    dependency_parser,\n","    re_graph_builder,\n","    reApproach_finetune,\n","    finisher\n","])"],"metadata":{"id":"Ne-1OecX4v6t","executionInfo":{"status":"ok","timestamp":1687517609834,"user_tz":-60,"elapsed":453,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":57,"outputs":[]},{"cell_type":"code","source":["%%time\n","rel_model = finetune_pipeline.fit(train_data)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ZWNrbuCJ4v1L","executionInfo":{"status":"ok","timestamp":1687517626629,"user_tz":-60,"elapsed":6214,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"021fa9c7-48b3-4682-a7be-f35bef7337b3"},"execution_count":58,"outputs":[{"output_type":"stream","name":"stdout","text":["TF Graph Builder configuration:\n","Model name: relation_extraction\n","Graph folder: ./tf_graphs\n","Graph file name: re_graph.pb\n","Build params: {'input_dim': 1149, 'output_dim': 27, 'hidden_layers': [300, 200], 'hidden_act': 'relu', 'hidden_act_l2': True, 'hidden_weights_l2': False, 'batch_norm': False}\n","relation_extraction graph exported to ./tf_graphs/re_graph.pb\n","CPU times: user 779 ms, sys: 20 ms, total: 799 ms\n","Wall time: 5.91 s\n"]}]},{"cell_type":"code","source":["result = rel_model.transform(test_data_2)"],"metadata":{"id":"_z8kDKR943qf","executionInfo":{"status":"ok","timestamp":1687517634906,"user_tz":-60,"elapsed":1980,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":59,"outputs":[]},{"cell_type":"markdown","source":["**Evaluate**"],"metadata":{"id":"Iqz7iC0P46KI"}},{"cell_type":"code","source":["result_test_df = result.select('rel', 'relations.result').toPandas()\n","result_test_df.sample(5)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"TKOurlA943kf","executionInfo":{"status":"ok","timestamp":1687517652502,"user_tz":-60,"elapsed":4670,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"e5e7cec8-7b5f-43a3-88d8-eabaa852b325"},"execution_count":60,"outputs":[{"output_type":"execute_result","data":{"text/plain":["      rel  result\n","157     O     [O]\n","31      O     [O]\n","261  TrAP  [TrAP]\n","145     O   [PIP]\n","177   PIP  [TrAP]"],"text/html":["\n","  <div id=\"df-5eb80a0d-e4d1-43c8-a887-de3127439841\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rel</th>\n","      <th>result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>157</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>31</th>\n","      <td>O</td>\n","      <td>[O]</td>\n","    </tr>\n","    <tr>\n","      <th>261</th>\n","      <td>TrAP</td>\n","      <td>[TrAP]</td>\n","    </tr>\n","    <tr>\n","      <th>145</th>\n","      <td>O</td>\n","      <td>[PIP]</td>\n","    </tr>\n","    <tr>\n","      <th>177</th>\n","      <td>PIP</td>\n","      <td>[TrAP]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5eb80a0d-e4d1-43c8-a887-de3127439841')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5eb80a0d-e4d1-43c8-a887-de3127439841 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5eb80a0d-e4d1-43c8-a887-de3127439841');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":60}]},{"cell_type":"code","source":["from sklearn.metrics import classification_report\n","\n","pd_test_results = result_test_df.explode(\"result\").fillna(\"O\")\n","\n","print(classification_report(pd_test_results[\"rel\"], pd_test_results[\"result\"]))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XxoE7N2I4B2G","executionInfo":{"status":"ok","timestamp":1687517654707,"user_tz":-60,"elapsed":420,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}},"outputId":"cf3a1f6c-d2a6-48bf-b2a1-55724649ac67"},"execution_count":61,"outputs":[{"output_type":"stream","name":"stdout","text":["              precision    recall  f1-score   support\n","\n","           O       0.59      0.67      0.63       136\n","         PIP       0.38      0.13      0.20        38\n","        TeCP       0.00      0.00      0.00         4\n","        TeRP       0.80      0.57      0.67        61\n","        TrAP       0.36      0.73      0.48        48\n","        TrCP       1.00      0.08      0.15        12\n","        TrIP       1.00      0.14      0.25         7\n","       TrNAP       0.00      0.00      0.00         5\n","        TrWP       0.00      0.00      0.00         3\n","\n","    accuracy                           0.54       314\n","   macro avg       0.46      0.26      0.26       314\n","weighted avg       0.57      0.54      0.51       314\n","\n"]}]},{"cell_type":"markdown","source":["Save to disk"],"metadata":{"id":"gD4np84K5BR1"}},{"cell_type":"code","source":["rel_model.stages[-2].write().overwrite().save('RE_pretrained_model_finetuned')"],"metadata":{"id":"K5zsROTF5BFX","executionInfo":{"status":"ok","timestamp":1687517671551,"user_tz":-60,"elapsed":471,"user":{"displayName":"Ahmet Emin Tek","userId":"14855809472179427810"}}},"execution_count":62,"outputs":[]}],"metadata":{"colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":0}