introduction_to_applying_machine_learning/identify_key_insights_from_textual_document/document_relationship_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "67ade5de",
   "metadata": {},
   "source": [
    "# Document Understanding Solution - Relationship Extraction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108d89c0",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98d5231",
   "metadata": {},
   "source": [
    "\n",
    "Relation Extraction (RE) is the task of extracting semantic relationships from text, which usually occur between two or more entities. In this notebook,  we demonstrate two use cases of Relation Extraction:\n",
    "\n",
    "1. How to fine-tune a pre-trained Transformer model on a custom dataset, and then run inference on the fine-tuned model.\n",
    "2. How to run [SageMaker Automatic Model Tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html) (a hyperparameter optimization procedure) to find the best model compared with the model fine-tuned in point 1. The performance of the optimal model and model fine-tuned in point 1 is evaluated on a hold-out test data. \n",
    "\n",
    "**Note**: When running this notebook on SageMaker Studio, you should make\n",
    "sure the `PyTorch 1.10 Python 3.8 CPU Optimized` image/kernel is used. When\n",
    "running this notebook on SageMaker Notebook Instance, you should make\n",
    "sure the 'sagemaker-soln' kernel is used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbc29af0",
   "metadata": {},
   "source": [
    "This solution relies on a config file to run the provisioned AWS resources. Run the cell below to generate that file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2236f4fc",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -U sagemaker ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccea7163",
   "metadata": {},
   "source": [
    "## 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7935b1eb",
   "metadata": {},
   "source": [
    "We start by importing a variety of packages that are used throughout\n",
    "the notebook. One of the most important packages is the Amazon SageMaker\n",
    "Python SDK (i.e. `import sagemaker`). We also import modules from our own\n",
    "custom (and editable) package that can be found at `../package`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77b2acd2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from pathlib import Path\n",
    "import sagemaker\n",
    "from sagemaker.pytorch import PyTorch\n",
    "import sys\n",
    "import config\n",
    "\n",
    "IAM_ROLE = sagemaker.get_execution_role()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()\n",
    "DEFAULT_BUCKET = sess.default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd7a42f5",
   "metadata": {},
   "source": [
    "## 2. Finetune the pre-trained model on a custom dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc85884",
   "metadata": {},
   "source": [
    "This is a Relationship Extraction model built on a [Bert-base-uncased](https://huggingface.co/bert-base-uncased) using transformers from the [transformers](https://huggingface.co/transformers/) library. \n",
    "\n",
    "The model for fine-tuning attaches a linear classification layer that takes a pair of token embeddings outputted by the Text Embedding model\n",
    "and initializes the layer parameters to random values. The fine-tuning step fine-tunes \n",
    "all the model parameters to minimize prediction error on the input data and returns the fine-tuned model. The Text Embedding model we use in this demonstartion is [Bert-base-uncased](https://huggingface.co/bert-base-uncased) from the [transformers](https://huggingface.co/transformers/) library. The dataset we fine-tune the model is [SemEval-2010 Task 8](https://aclanthology.org/S10-1006/). The SemEval-2 Task 8 is a dataset for multi-way classification of mutually exclusive semantic relations between pairs of nominals.\n",
    "\n",
    "\n",
    "The model returned by fine-tuning can be further deployed for inference. Below are the instructions \n",
    "for how the training data should be formatted for input to the model. \n",
    "\n",
    "- **Input:**  A directory containing a `txt` format file.\n",
    "    - Each observation contains three components, text, semantic relation label, and comment (optional), each of which takes a line in the `txt` format file. Observations are separated by an empty line. For each observation, there are markers highlighting the two terms in the text and their semantic relation label in the line below.\n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    " \n",
    "Below is an example of `txt` format file. Note. Desipte of the same semantic relation label, pairs of entities with different order relations are counted as different labels. For an example, `Component-Whole(e2,e1)` and `Component-Whole(e1,e2)` are different semantic relation labels. The data for training and validation are downloaded into directory `../data/semeval2010t8` in the following section.\n",
    "\n",
    "|   |\n",
    "|--- |\n",
    "|1  \"The system as described above has its greatest application in an arrayed <e1>configuration</e1> of antenna <e2>elements</e2>.\"|\n",
    "|Component-Whole(e2,e1)|\n",
    "|Comment: Not a collection: there is structure here, organisation.|\n",
    "||\n",
    "|2  \"The <e1>child</e1> was carefully wrapped and bound into the <e2>cradle</e2> by means of a cord.\"|\n",
    "|Other|\n",
    "|Comment: NA|\n",
    "| |\n",
    "|3  \"The <e1>author</e1> of a keygen uses a <e2>disassembler</e2> to look at the raw assembly code.\"|\n",
    "|Instrument-Agency(e2,e1)|\n",
    "|Comment: NA|\n",
    "||\n",
    "|...   |\n",
    " \n",
    "\n",
    "\n",
    "Citation:\n",
    "@inproceedings{hendrickx-etal-2010-semeval,\n",
    "    title = \"{S}em{E}val-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals\",\n",
    "    author = \"Hendrickx, Iris  and\n",
    "      Kim, Su Nam  and\n",
    "      Kozareva, Zornitsa  and\n",
    "      Nakov, Preslav  and\n",
    "      {\\'O} S{\\'e}aghdha, Diarmuid  and\n",
    "      Pad{\\'o}, Sebastian  and\n",
    "      Pennacchiotti, Marco  and\n",
    "      Romano, Lorenza  and\n",
    "      Szpakowicz, Stan\",\n",
    "    booktitle = \"Proceedings of the 5th International Workshop on Semantic Evaluation\",\n",
    "    month = jul,\n",
    "    year = \"2010\",\n",
    "    address = \"Uppsala, Sweden\",\n",
    "    publisher = \"Association for Computational Linguistics\",\n",
    "    url = \"https://www.aclweb.org/anthology/S10-1006\",\n",
    "    pages = \"33--38\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccdda27c",
   "metadata": {},
   "source": [
    "### 2.1. Download, preprocess, and upload the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdcf4099",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!aws s3 cp --recursive $config.SOURCE_S3_PATH/artifacts/data/semeval2010t8/ data/semeval2010t8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db03c166",
   "metadata": {},
   "source": [
    "The dataset has been partitioned into `train.txt`, `validation.txt`, and `test.txt` data. Thus we don't need split the train data as what we do in previous notebooks. The`train.txt` and `validation.txt` are used as training and validation data. The `test.txt` is used as hold-out test data to evaluate model performance with / without hyperparameter optimization. Next, we upload them into S3 path which are used as input for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3308c785",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "\n",
    "bucket = DEFAULT_BUCKET\n",
    "prefix = \"RE\"\n",
    "\n",
    "boto3.Session().resource(\"s3\").Bucket(bucket).Object(\n",
    "    os.path.join(prefix, \"train/train.txt\")\n",
    ").upload_file(\"data/semeval2010t8/train/train.txt\")\n",
    "\n",
    "boto3.Session().resource(\"s3\").Bucket(bucket).Object(\n",
    "    os.path.join(prefix, \"validation/validation.txt\")\n",
    ").upload_file(\"data/semeval2010t8/validation/validation.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82996a9",
   "metadata": {},
   "source": [
    "### 2.2. Set Training parameters\n",
    "\n",
    "Now that we are done with all the setup that is needed, we are ready to fine-tune our relation extraction model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8f3d23b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "    \"pretrained-model\": \"bert-base-uncased\",\n",
    "    \"learning-rate\": 0.0002,\n",
    "    \"max-epoch\": 2,\n",
    "    \"weight-decay\": 0,\n",
    "    \"batch-size\": 16,\n",
    "    \"accumulate-grad-batches\": 2,\n",
    "    \"gradient-clip-val\": 1.0,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed2f2887",
   "metadata": {},
   "source": [
    "### 3.2. Fine-tuning without hyperparameter optimization\n",
    "\n",
    "We use the PyTorch from the Amazon SageMaker Python SDK. The entry script is located under `../containers/relationship_extraction/entry_point.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8a216f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "training_job_name = f\"{config.SOLUTION_PREFIX}-re-finetune\"\n",
    "\n",
    "train_instance_type = config.TRAINING_INSTANCE_TYPE\n",
    "\n",
    "re_estimator = PyTorch(\n",
    "    framework_version=\"1.10.0\",\n",
    "    py_version=\"py38\",\n",
    "    entry_point=\"entry_point.py\",\n",
    "    source_dir=\"containers/relationship_extraction\",\n",
    "    hyperparameters=hyperparameters,\n",
    "    role=IAM_ROLE,\n",
    "    instance_count=1,\n",
    "    instance_type=train_instance_type,\n",
    "    output_path=f\"s3://{bucket}/{prefix}/output\",\n",
    "    code_location=f\"s3://{bucket}/{prefix}/output\",\n",
    "    base_job_name=training_job_name,\n",
    "    tags=[{\"Key\": config.TAG_KEY, \"Value\": config.SOLUTION_PREFIX}],\n",
    "    sagemaker_session=sess,\n",
    "    volume_size=30,\n",
    "    env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"500\"},\n",
    "    debugger_hook_config=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fecf61c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "re_estimator.fit(\n",
    "    {\n",
    "        \"train\": f\"s3://{bucket}/{prefix}/train/\",\n",
    "        \"validation\": f\"s3://{bucket}/{prefix}/validation/\",\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f49c0223",
   "metadata": {},
   "source": [
    "## 3.3. Deploy & run Inference on the fine-tuned model\n",
    "\n",
    "A trained model does nothing on its own. We now want to use the model to perform inference. For this example, it means predicting the semantic relation label of two text string within an input text. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fdf9bd4",
   "metadata": {},
   "source": [
    "We use the unique solution prefix to name the model and endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7436ca6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "inference_instance_type = config.HOSTING_INSTANCE_TYPE\n",
    "\n",
    "unique_hash = str(uuid.uuid4())[:6]\n",
    "endpoint_name_finetune = f\"{config.SOLUTION_PREFIX}-{unique_hash}-re-finetune-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd10ca09",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "\n",
    "finetuned_predictor = re_estimator.deploy(\n",
    "    endpoint_name=endpoint_name_finetune,\n",
    "    instance_type=inference_instance_type,\n",
    "    initial_instance_count=1,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer(),\n",
    ")\n",
    "\n",
    "time.sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76caa220",
   "metadata": {},
   "source": [
    "When calling our new endpoint from the notebook, we use a Amazon\n",
    "SageMaker SDK\n",
    "[`Predictor`](https://sagemaker.readthedocs.io/en/stable/predictors.html).\n",
    "A `Predictor` is used to send data to an endpoint (as part of a request),\n",
    "and interpret the response. Our `estimator.deploy` command returned a\n",
    "`Predictor` but, by default, it sends and receive numpy arrays. Our\n",
    "endpoint expects to receive (and also sends) JSON formatted objects, so\n",
    "we modify the `Predictor` to use JSON instead of the PyTorch endpoint\n",
    "default of numpy arrays. JSON is used here because it is a standard\n",
    "endpoint format and the endpoint response can contain nested data\n",
    "structures."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca33e45",
   "metadata": {},
   "source": [
    "With our model successfully deployed and our predictor configured, we can\n",
    "try out the relationship extraction model out on example inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57466e04",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor.predict(\n",
    "    data={\n",
    "        \"sequence\": \"Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning (ML) models quickly.\",\n",
    "        \"entity_one_start\": 0,\n",
    "        \"entity_one_end\": 6,\n",
    "        \"entity_two_start\": 7,\n",
    "        \"entity_two_end\": 16,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bbabb77",
   "metadata": {},
   "source": [
    "Next, let's query the deployed endpoint to get for the prediction for each test example located in `../data/semeval2010t8/test/test.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96197594",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils_relation_extraction import parse_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4e0403d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "examples, ground_truth = parse_file(\"data/semeval2010t8/test/test.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e1c589d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prediction_labels = []\n",
    "for each_example in examples:\n",
    "    prediction_labels.append(finetuned_predictor.predict(each_example)[\"Label\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328f2c02",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "\n",
    "accuracy = accuracy_score(prediction_labels, ground_truth)\n",
    "f1_macro = f1_score(prediction_labels, ground_truth, average=\"macro\")\n",
    "f1_micro = f1_score(prediction_labels, ground_truth, average=\"micro\")\n",
    "\n",
    "result = {\"Accuracy\": [accuracy], \"F1 Macro\": [f1_macro], \"F1 Micro\": [f1_micro]}\n",
    "\n",
    "result = pd.DataFrame.from_dict(result, orient=\"index\", columns=[\"No HPO\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e50b2130",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7527ce9c",
   "metadata": {},
   "source": [
    "Since the task is essentially multiclass classification task, we use accuracy, f1 macro, and f1 micro as the evaluation scores. For each of them, higher value indicates better results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f91ba8a6",
   "metadata": {},
   "source": [
    "## 3. Finetune the pre-trained model on a custom dataset with automatic model tuning (AMT)\n",
    "\n",
    "Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose. We use a [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) object to interact with Amazon SageMaker hyperparameter tuning APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d757ded2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tuner import (\n",
    "    ContinuousParameter,\n",
    "    IntegerParameter,\n",
    "    CategoricalParameter,\n",
    "    HyperparameterTuner,\n",
    ")\n",
    "\n",
    "\n",
    "# Define objective metric per framework, based on which the best model is selected.\n",
    "metric_definitions = {\n",
    "    \"metrics\": [{\"Name\": \"validation_accuracy\", \"Regex\": \"valid_accuracy=([0-9\\\\.]+)\"}],\n",
    "    \"type\": \"Maximize\",\n",
    "}\n",
    "\n",
    "# You can select from the hyperparameters supported by the model, and configure ranges of values to be searched for training the optimal model.(https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-ranges.html)\n",
    "hyperparameter_ranges = {\n",
    "    \"learning-rate\": ContinuousParameter(0.0001, 0.001, scaling_type=\"Logarithmic\"),\n",
    "    # \"max-epoch\": IntegerParameter(3, 8),\n",
    "}\n",
    "\n",
    "# Increase the total number of training jobs run by AMT, for increased accuracy (and training time).\n",
    "max_jobs = 6\n",
    "# Change parallel training jobs run by AMT to reduce total training time, constrained by your account limits.\n",
    "# if max_jobs=max_parallel_jobs then Bayesian search turns to Random.\n",
    "max_parallel_jobs = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7565f6ef",
   "metadata": {},
   "source": [
    "### 3.1. Fine-tuning with hyperparameter optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96157a30",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuning_job_name = f\"{config.SOLUTION_PREFIX}-re-hpo\"\n",
    "\n",
    "hyperparameters = {\n",
    "    \"max-epoch\": 2,\n",
    "    \"weight-decay\": 0,\n",
    "    \"batch-size\": 16,\n",
    "    \"accumulate-grad-batches\": 2,\n",
    "    \"gradient-clip-val\": 1.0,\n",
    "}\n",
    "\n",
    "\n",
    "estimator = PyTorch(\n",
    "    framework_version=\"1.10.0\",\n",
    "    py_version=\"py38\",\n",
    "    entry_point=\"entry_point.py\",\n",
    "    source_dir=\"containers/relationship_extraction\",\n",
    "    hyperparameters=hyperparameters,\n",
    "    role=IAM_ROLE,\n",
    "    instance_count=1,\n",
    "    instance_type=train_instance_type,\n",
    "    output_path=f\"s3://{bucket}/{prefix}/output\",\n",
    "    code_location=f\"s3://{bucket}/{prefix}/output\",\n",
    "    base_job_name=tuning_job_name,\n",
    "    tags=[{\"Key\": config.TAG_KEY, \"Value\": config.SOLUTION_PREFIX}],\n",
    "    sagemaker_session=sess,\n",
    "    volume_size=30,\n",
    "    env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"500\"},\n",
    "    debugger_hook_config=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b19aee8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "re_tuner = HyperparameterTuner(\n",
    "    estimator,\n",
    "    metric_definitions[\"metrics\"][0][\"Name\"],\n",
    "    hyperparameter_ranges,\n",
    "    metric_definitions[\"metrics\"],\n",
    "    max_jobs=max_jobs,\n",
    "    max_parallel_jobs=max_parallel_jobs,\n",
    "    objective_type=metric_definitions[\"type\"],\n",
    "    base_tuning_job_name=tuning_job_name,\n",
    ")\n",
    "\n",
    "re_tuner.fit(\n",
    "    {\n",
    "        \"train\": f\"s3://{bucket}/{prefix}/train/\",\n",
    "        \"validation\": f\"s3://{bucket}/{prefix}/validation/\",\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f707d9",
   "metadata": {},
   "source": [
    "### 3.2. Deploy & run Inference on the fine-tuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b859f2ce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "\n",
    "endpoint_name_hpo = f\"{config.SOLUTION_PREFIX}-re-hpo-endpoint\"\n",
    "\n",
    "finetuned_predictor_hpo = re_tuner.deploy(\n",
    "    endpoint_name=endpoint_name_hpo,\n",
    "    instance_type=inference_instance_type,\n",
    "    initial_instance_count=1,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer(),\n",
    ")\n",
    "\n",
    "time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a23c5a0d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prediction_labels_hpo = []\n",
    "for each_example in examples:\n",
    "    prediction_labels_hpo.append(finetuned_predictor_hpo.predict(each_example)[\"Label\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b82213f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "\n",
    "accuracy_hpo = accuracy_score(prediction_labels_hpo, ground_truth)\n",
    "f1_macro_hpo = f1_score(prediction_labels_hpo, ground_truth, average=\"macro\")\n",
    "f1_micro_hpo = f1_score(prediction_labels_hpo, ground_truth, average=\"micro\")\n",
    "\n",
    "\n",
    "result_hpo = {\n",
    "    \"Accuracy\": [accuracy_hpo],\n",
    "    \"F1 Macro\": [f1_macro_hpo],\n",
    "    \"F1 Micro\": [f1_micro_hpo],\n",
    "}\n",
    "\n",
    "result_hpo = pd.DataFrame.from_dict(result_hpo, orient=\"index\", columns=[\"With HPO\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c15c3e73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.concat([result, result_hpo], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfe0c7cb",
   "metadata": {},
   "source": [
    "We can see results with hyperparameter optimization shows better performance on the hold-out test data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a44e9ad1",
   "metadata": {},
   "source": [
    "## 3.3. Clean Up the endpoint\n",
    "\n",
    "When you've finished with the summarization endpoint (and associated\n",
    "endpoint-config), make sure that you delete it to avoid accidental\n",
    "charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93c3d741",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#### # Delete the SageMaker endpoint and the attached resources\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()\n",
    "\n",
    "finetuned_predictor_hpo.delete_model()\n",
    "finetuned_predictor_hpo.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "759bd122",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/introduction_to_applying_machine_learning|identify_key_insights_from_textual_document|document_relationship_extraction.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/pytorch-1.13-cpu-py39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}