notebooks/api.demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Characterization of Stigmatizing Language in Medical Records\n",
    "\n",
    "This notebook serves as a quickstart guide for interacting with our stigmatizing language characterization toolkit. As you move through the notebook, you will learn not only *how* to characterize stigmatizing language in medical records, but also *why* we should care about stigmatizing language in medical records.\n",
    "\n",
    "## Motivation\n",
    "\n",
    "Widespread disparities in clinical outcomes exist between different demographic groups in the United States. A new line of work in [medical sociology](https://pubmed.ncbi.nlm.nih.gov/34130567/) has demonstrated physicians often use stigmatizing language in electronic medical records within certain groups, such as black patients, which may exacerbate disparities. This documentation practice may not only negatively frame patients to future providers and thus influence their quality of care, but also discourage patients from seeking treatment [altogether](https://rdcu.be/dfbrc).\n",
    "\n",
    "A major challenge with stigmatizing language in medical records is that it is frequently invoked unconsciously, reflecting an internalized bias by the healthcare provider. The first step in confronting this bias is highlighting it for the provider.\n",
    "\n",
    "## Prior Work\n",
    "\n",
    "The idea of using natural language processing to characterize stigmatizing language in medical records is relatively new. Most prior work has relied on using simple [word counts](https://rdcu.be/dfbvx). The main shortcoming with such analyses is that they lack an ability to discriminate between genuine and innocuous cases of stigmatizing language (e.g., \"aggressive behavior\" vs. \"aggressive treatment regimen\"). \n",
    "\n",
    "[Sun et al.](https://doi.org/10.1377/hlthaff.2021.01423) was the first to use machine learning to better characterize instances of stigmatizing language in medical records. Although it was a step forward, their system was limited to predicting 3 sentiment-like classes (i.e., positive, negative, out-of-context) and relied on non-neural, non-contextual models.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "To ensure a smooth experience, we must first make sure you have access to the necessary resources and computation environment.\n",
    "\n",
    "### Compute Environment\n",
    "\n",
    "Our toolkit was developed and tested using Python 3.10. We cannot guarantee that other versions of Python will support the entirety of our codebase. That said, we expect the majority of functionality to be preserved as long as you are using Python >= 3.7.\n",
    "\n",
    "We **strongly** recommend using a virtual environment manager (e.g., `conda`) when working with this codebase. This will help limit unintended consequences that arise due to e.g., dependency upgrades. The `conda` [documentation](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html) provides all the information you need to set up your first environment.\n",
    "\n",
    "Once your environment has been created and activated, you can install the `stigma` toolkit with a single command (executed from the root of the repository):\n",
    "\n",
    "```\n",
    "pip install -e .\n",
    "```\n",
    "\n",
    "This command will install all external dependencies, as well as the `stigma` package itself. It is *extremely important* to keep the `-e` environment flag, as it will ensure default data and model paths are preserved.\n",
    "\n",
    "### Data\n",
    "\n",
    "To complete this tutorial, no external data (e.g., MIMIC) is required. However, if you are interested in training your own models, you will need to download the [MIMIC-IV (v2.2)](https://physionet.org/content/mimiciv/2.2/) and [MIMIC-IV-Notes (v2.2)](https://physionet.org/content/mimic-iv-note/2.2/) datasets. Access to both of these resources requires IRB training, authenticated credentials, and the signing of a data usage agreement. If all of these criteria are satisfied, you can download the minimally necessary files to train new models or reproduce our past experiments using `bash scripts/acquire/get_mimic.sh`.\n",
    "\n",
    "### Models\n",
    "\n",
    "Although you *do not* need access to MIMIC to complete this tutorial, you *do* need access to our pretrained stigmatizing language classifiers. These can be acquired from PhysioNet after signing our data usage agreement. If you already have access to MIMIC, downloading our models should only require you sign our [data usage agreement](https://doi.org/10.13026/n938-2385).\n",
    "\n",
    "We have opted to keep our models behind a gate for a few reasons. First, although we do not expect our training procedure to encode sensitive information regarding the MIMIC dataset, the risk is nonzero and worth respecting. Furthermore, if we release models in the future which do allow end-users to extract sensitive information, existing end-users will be able to acquire them with minimal additional overhead.\n",
    "\n",
    "With that said, note that our models can be acquired through two methods. The first (recommended) method requires running a single command `bash scripts/acquire/get_models_and_labels.sh` and entering your PhysioNet credentials when prompted. This script will download the most up-to-date models and place them in the appropriate directories. If for some reason you encounter issues downloading our models using the `bash` script, you can do so through your [web browser](https://doi.org/10.13026/n938-2385). Upon downloading the zip file hosted on PhysioNet, please unzip the contents and place them within the `data/resources/` directory. If all goes as expected, you should now have `data/resources/models/` and `data/resources/annotations/` folders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "We begin by importing the three primary modules for interfacing easily with our stigmatizing language models.\n",
    "\n",
    "The `StigmaSearch` module uses regular expressions to identify anchors (i.e., keywords) which are commonly used in a stigmatizing manner. The module can also be used to prepare data for input into downstream models.\n",
    "\n",
    "The `StigmaBaselineModel` and `StigmaBertModel` modules act wrappers to our classifiers, each trained to characterize a different type of stigmatizing language. The `StigmaBertModel` module supports both CPU and GPU inference. Neural models leverage the `torch` and `transformers` libraries, while non-neural models leverage `scikit-learn` and `numpy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Notebook Imports\n",
    "from IPython.display import display\n",
    "\n",
    "## Environment\n",
    "import os\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
    "\n",
    "## API Imports\n",
    "from stigma import settings\n",
    "from stigma import StigmaSearch\n",
    "from stigma import StigmaBaselineModel, StigmaBertModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples of Stigmatizing Language\n",
    "\n",
    "Below, we have created some realistic examples of stigmatizing language usage in an electronic medical record. Although the examples below are short, we note that our toolkit is designed to be applied to full clinical notes without substantial preprocessing. Our models operate on normalized versions of text and do not require any additional metadata regarding the note. There are, however, some important caveats.\n",
    "\n",
    "#### Limitation 1: Reliance on Predefined Keywords\n",
    "\n",
    "Our models are trained to characterize stigmatizing language within statements containing one of our predefined stigmatizing keywords. The full list of keywords is included in `data/resources/keywords/keywords.json`. Experiments suggest that our current models *do not* generalize well to statements containing keywords not seen at training time, or those not containing a stigmatizing keyword altogether. If you are interested in making inferences for statements lacking a keyword or containing a new keyword, we highly recommend you annotate additional data and retrain the existing classifiers with the augmented dataset.\n",
    "\n",
    "#### Limitation 2: Robustness to Domain Shift\n",
    "\n",
    "In our ACL paper, [\"Characterization of Stigmatizing Language in Medical Records\"](https://aclanthology.org/2023.acl-short.28/), we conduct a suite of domain transfer experiments. We evaluate transfer between the public MIMIC-IV dataset and a private dataset consisting of records from our research institution. We observe statistically significant drops in performance between transferring between the datasets. A qualitative error analysis suggests that this arises due to differences in the joint keyword-label distributions for each dataset, as well as speciality-specific nuances in the usage of language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = [\n",
    "    \"\"\"\n",
    "    Despite my best advice, the patient remains adamant about leaving the hospital today. \n",
    "    Social services is aware of the situation.\n",
    "    \"\"\",\n",
    "\n",
    "    \"\"\"\n",
    "    Patient Doe remains lethargic and slow-moving. They insist that they have adhered to a \n",
    "    'drug-free lifestyle', though blood tests suggest otherwise.\n",
    "    \"\"\",\n",
    "    \n",
    "    \"\"\"\n",
    "    Miss Doe is a charming, 73 year old women who visits us today with a chief complaint \n",
    "    of heart pain. Unfortunately, not a good historian.\n",
    "    \"\"\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identifying Stigmatizing Language\n",
    "\n",
    "The first step to characterizing stigmatizing language in the EHR is finding it. As mentioned above, we use a set of predefined keywords which were curated by domain-experts to identify candidate instances of stigmatizing language.\n",
    "\n",
    "In the cell below, we initialize our search module and apply it to the examples enumerated above. \n",
    "\n",
    "You have the option of changing the `context_size` parameter in `StigmaSearch`. This value dictates the number of words to the left and right, respectively, that are maintained as context for the stigmatizing keyword. Our models are trained using a context size of 10, a value chosen based on guidance from our clinical collaborators. Future work may examine the optimality of this parameter choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_id</th>\n",
       "      <th>keyword_category</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>keyword</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>adamant</td>\n",
       "      <td>49</td>\n",
       "      <td>56</td>\n",
       "      <td>adamant</td>\n",
       "      <td>despite my best advice, the patient remains ad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>adamant</td>\n",
       "      <td>57</td>\n",
       "      <td>63</td>\n",
       "      <td>insist</td>\n",
       "      <td>patient doe remains lethargic and slow-moving....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>compliance</td>\n",
       "      <td>79</td>\n",
       "      <td>86</td>\n",
       "      <td>adhered</td>\n",
       "      <td>doe remains lethargic and slow-moving. they in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>other</td>\n",
       "      <td>19</td>\n",
       "      <td>27</td>\n",
       "      <td>charming</td>\n",
       "      <td>miss doe is a charming, 73 year old women who ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>other</td>\n",
       "      <td>136</td>\n",
       "      <td>145</td>\n",
       "      <td>historian</td>\n",
       "      <td>of heart pain. unfortunately, not a good histo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   document_id keyword_category  start  end    keyword  \\\n",
       "0            0          adamant     49   56    adamant   \n",
       "1            1          adamant     57   63     insist   \n",
       "2            1       compliance     79   86    adhered   \n",
       "3            2            other     19   27   charming   \n",
       "4            2            other    136  145  historian   \n",
       "\n",
       "                                                text  \n",
       "0  despite my best advice, the patient remains ad...  \n",
       "1  patient doe remains lethargic and slow-moving....  \n",
       "2  doe remains lethargic and slow-moving. they in...  \n",
       "3  miss doe is a charming, 73 year old women who ...  \n",
       "4  of heart pain. unfortunately, not a good histo...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Initialize Search Wrapper\n",
    "search_tool = StigmaSearch(context_size=10)\n",
    "\n",
    "## Run Search\n",
    "search_results = search_tool.search(examples)\n",
    "\n",
    "## Show Results\n",
    "_ = display(search_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Ready to Make Inferences\n",
    "\n",
    "If all ran as expected, you should see a pandas DataFrame above containing candidate instances of stigmatizing language drawn from the previously defined examples. The DataFrame contains 6 columns:\n",
    "\n",
    "* `document_id`: The integer index of the example passed to the search tool.\n",
    "* `keyword_category`: The potential class of stigmatizing language. Our current models operationalize these classes as separate modeling tasks. We include a taxonomy of the tasks below.\n",
    "* `start`: An integer indicating the starting character of the matched keyword within the original document.\n",
    "* `end`: An integer indicating the ending character of the matched keyword within the original document.\n",
    "* `text`: A snippet of text which makes up the \"context\" for the keyword. The length of this text will depend on the `context_size` parameter initialized within the `StigmaSearch` class.\n",
    "\n",
    "We can re-use the `StigmaSearch` module to extract the inputs which will be passed to our machine learning classifiers. The `format_for_model()` method ingests the results DataFrame, along with a desired keyword category, and outputs document IDs, keywords, and text windows. Each of these items is formatted as a list.\n",
    "\n",
    "## Types of Stigmatizing Language (Task Taxonomy)\n",
    "\n",
    "We formulate three independent classification tasks that discriminate between instances of bias based on their impact. Note that we use the \"keyword category\" field as shorthand to reference each modeling task. Expanded descriptions of the tasks and their respective classes can be found in Table 4 of [our paper](https://aclanthology.org/2023.acl-short.28/).\n",
    "\n",
    "| Task | Keyword Category | Classes | Description |\n",
    "|------|---------------------------|---------|-------------|\n",
    "| Credibility & Obstinacy | \"adamant\" | Disbelief, Difficult, Exclude | Insinuation of doubt regarding a patient's testimony or describes the patient as obstinate.|\n",
    "| Compliance | \"compliance\" | Negative, Neutral, Positive | Patient does not appear to follow medical advice.|\n",
    "| Descriptors | \"other\" | Negative, Neutral, Positive, Exclude | Evaluates descriptions of patient behavior and demeanor.|\n",
    "\n",
    "## Model Setup\n",
    "\n",
    "Once we have decided which type of stigmatizing language we are interested in characterizing, we can initialize the `StigmaBaselineModel` or `StigmaBertModel` class with one of our pretrained models. There are a few important arguments to be aware of:\n",
    "\n",
    "* `model`: A string identifier indicating which model to load. You can see the list of default models using `StigmaBaselineModel.show_default_models()` or `StigmaBertModel.show_default_models()`. Alternatively, you can pass a full file-path to a pretrained model.\n",
    "* `tokenizer` (`StigmaBertModel` only): If you are passing a non-default model, you should include a path to the tokenizer associated with the model. Under the hood, this loads a Hugging Face tokenizer.\n",
    "* `preprocessing_params` (`StigmaBaselineModel` only): Path to a \"preprocessing_params.joblib\" file. Generated at root of output model directory during baseline model training. Only necessary if not using one of the default models.\n",
    "* `keyword_category`: A string indicating which of the tasks from above you are interested in.\n",
    "* `batch_size`: An integer indicating how many instances (context windows) will be processed in each batch. This will depend on your compute environment (e.g., RAM or vRAM).\n",
    "* `device` (`StigmaBertModel` only): Either \"cpu\" or \"cuda\" depending on whether you want to use a GPU to accelerate inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      " \"adamant\": [\n",
      "  \"adamant\",\n",
      "  \"adamantly\",\n",
      "  \"adament\",\n",
      "  \"adamently\",\n",
      "  \"claim\",\n",
      "  \"claimed\",\n",
      "  \"claiming\",\n",
      "  \"claims\",\n",
      "  \"insist\",\n",
      "  \"insisted\",\n",
      "  \"insistence\",\n",
      "  \"insisting\",\n",
      "  \"insists\"\n",
      " ],\n",
      " \"compliance\": [\n",
      "  \"adherance\",\n",
      "  \"adhere\",\n",
      "  \"adhered\",\n",
      "  \"adherence\",\n",
      "  \"adherent\",\n",
      "  \"adheres\",\n",
      "  \"adhering\",\n",
      "  \"compliance\",\n",
      "  \"compliant\",\n",
      "  \"complied\",\n",
      "  \"complies\",\n",
      "  \"comply\",\n",
      "  \"complying\",\n",
      "  \"declined\",\n",
      "  \"declines\",\n",
      "  \"declining\",\n",
      "  \"nonadherance\",\n",
      "  \"nonadherence\",\n",
      "  \"nonadherent\",\n",
      "  \"noncompliance\",\n",
      "  \"noncompliant\",\n",
      "  \"refusal\",\n",
      "  \"refuse\",\n",
      "  \"refused\",\n",
      "  \"refuses\",\n",
      "  \"refusing\"\n",
      " ],\n",
      " \"other\": [\n",
      "  \"aggression\",\n",
      "  \"aggressive\",\n",
      "  \"aggressively\",\n",
      "  \"agitated\",\n",
      "  \"agitation\",\n",
      "  \"anger\",\n",
      "  \"angered\",\n",
      "  \"angers\",\n",
      "  \"angrier\",\n",
      "  \"angrily\",\n",
      "  \"angry\",\n",
      "  \"argumentative\",\n",
      "  \"argumentatively\",\n",
      "  \"belligerence\",\n",
      "  \"belligerent\",\n",
      "  \"belligerently\",\n",
      "  \"charming\",\n",
      "  \"combative\",\n",
      "  \"combatively\",\n",
      "  \"confrontational\",\n",
      "  \"cooperative\",\n",
      "  \"defensive\",\n",
      "  \"delightful\",\n",
      "  \"disheveled\",\n",
      "  \"drug seeking\",\n",
      "  \"drug-seeking\",\n",
      "  \"exaggerate\",\n",
      "  \"exaggerates\",\n",
      "  \"exaggerating\",\n",
      "  \"historian\",\n",
      "  \"lovely\",\n",
      "  \"malinger\",\n",
      "  \"malingered\",\n",
      "  \"malingerer\",\n",
      "  \"malingering\",\n",
      "  \"malingers\",\n",
      "  \"narcotic seeking\",\n",
      "  \"narcotic-seeking\",\n",
      "  \"pleasant\",\n",
      "  \"pleasantly\",\n",
      "  \"poorly groomed\",\n",
      "  \"poorly-groomed\",\n",
      "  \"secondary gain\",\n",
      "  \"uncooperative\",\n",
      "  \"unkempt\",\n",
      "  \"unmotivated\",\n",
      "  \"unwilling\",\n",
      "  \"unwillingly\",\n",
      "  \"well groomed\",\n",
      "  \"well-groomed\"\n",
      " ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "## See Keywords and Their Categories\n",
    "_ = StigmaSearch.show_default_keyword_categories()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Baseline (Non-BERT) Models\n",
      "{\n",
      " \"mimic-iv-discharge_majority_overall\": {\n",
      "  \"model_type\": \"baseline\",\n",
      "  \"preprocessing\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-majority/preprocessing.params.joblib\",\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-majority/keyword/majority/adamant_fold-0\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-majority/keyword/majority/compliance_fold-0\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-majority/keyword/majority/other_fold-0\"\n",
      "  }\n",
      " },\n",
      " \"mimic-iv-discharge_majority_keyword\": {\n",
      "  \"model_type\": \"baseline\",\n",
      "  \"preprocessing\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/preprocessing.params.joblib\",\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword/linear/adamant_fold-0\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword/linear/compliance_fold-0\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword/linear/other_fold-0\"\n",
      "  }\n",
      " },\n",
      " \"mimic-iv-discharge_logistic-regression_context\": {\n",
      "  \"model_type\": \"baseline\",\n",
      "  \"preprocessing\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/preprocessing.params.joblib\",\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/tokens_tfidf/linear/adamant_fold-0\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/tokens_tfidf/linear/compliance_fold-0\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/tokens_tfidf/linear/other_fold-0\"\n",
      "  }\n",
      " },\n",
      " \"mimic-iv-discharge_logistic-regression_keyword-context\": {\n",
      "  \"model_type\": \"baseline\",\n",
      "  \"preprocessing\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/preprocessing.params.joblib\",\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword_tokens_tfidf/linear/adamant_fold-0\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword_tokens_tfidf/linear/compliance_fold-0\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_baseline-statistical/keyword_tokens_tfidf/linear/other_fold-0\"\n",
      "  }\n",
      " }\n",
      "}\n",
      ">> BERT Models\n",
      "{\n",
      " \"mimic-iv-discharge_base-bert\": {\n",
      "  \"model_type\": \"bert\",\n",
      "  \"preprocessing\": null,\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_base-bert/adamant_fold-0/checkpoint-250\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_base-bert/compliance_fold-0/checkpoint-100\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_base-bert/other_fold-0/checkpoint-250\"\n",
      "  }\n",
      " },\n",
      " \"mimic-iv-discharge_clinical-bert\": {\n",
      "  \"model_type\": \"bert\",\n",
      "  \"preprocessing\": null,\n",
      "  \"tasks\": {\n",
      "   \"adamant\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_clinical-bert/adamant_fold-0/checkpoint-50\",\n",
      "   \"compliance\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_clinical-bert/compliance_fold-0/checkpoint-400\",\n",
      "   \"other\": \"/Users/kharrigian/Dev/research/johns-hopkins/ehr-stigma/data/resources/models/mimic-iv-discharge_clinical-bert/other_fold-0/checkpoint-350\"\n",
      "  }\n",
      " }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "## See Available Models\n",
    "print(\">> Baseline (Non-BERT) Models\")\n",
    "_ = StigmaBaselineModel.show_default_models()\n",
    "print(\">> BERT Models\")\n",
    "_ = StigmaBertModel.show_default_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example IDs: [2, 2]\n",
      "Example Keywords: ['charming', 'historian']\n",
      "Example Text: ['miss doe is a charming, 73 year old women who visits us today with a', 'of heart pain. unfortunately, not a good historian.']\n",
      "[Loading Model Parameters]\n",
      "[Initializing Model Architecture]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at emilyalsentzer/Bio_ClinicalBERT were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Initializing Model Weights]\n"
     ]
    }
   ],
   "source": [
    "## Choose Keyword Category\n",
    "keyword_category = \"other\"\n",
    "\n",
    "## Prepare Inputs for the Model\n",
    "example_ids, example_keywords, example_text = search_tool.format_for_model(search_results=search_results,\n",
    "                                                                           keyword_category=keyword_category)\n",
    "\n",
    "print(f\"Example IDs: {example_ids}\")\n",
    "print(f\"Example Keywords: {example_keywords}\")\n",
    "print(f\"Example Text: {example_text}\")\n",
    "\n",
    "## Initialize Baseline Models\n",
    "majority_model = StigmaBaselineModel(model=\"mimic-iv-discharge_majority_overall\",\n",
    "                                     keyword_category=keyword_category,\n",
    "                                     batch_size=32)\n",
    "keyword_context_model = StigmaBaselineModel(model=\"mimic-iv-discharge_logistic-regression_keyword-context\",\n",
    "                                            keyword_category=keyword_category,\n",
    "                                            batch_size=32)\n",
    "\n",
    "## Initialize BERT Model (Note the alternative option for specifying model parameters)\n",
    "bert_model = StigmaBertModel(model=settings.MODELS[\"mimic-iv-discharge_clinical-bert\"][\"tasks\"][keyword_category],\n",
    "                             tokenizer=\"emilyalsentzer/Bio_ClinicalBERT\",\n",
    "                             keyword_category=keyword_category,\n",
    "                             batch_size=8,\n",
    "                             device=\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making Inferences\n",
    "\n",
    "Once the model has been initialized and your data has been prepared, you are ready to characterize stigmatizing language!\n",
    "\n",
    "Our API makes it as simple as calling a `predict()` method on the list of text and keywords extracted above. The output is a DataFrame with one row per input instance. Probabilities for each class of the respective task are shown in each column. To make a single class prediction, you can use the pandas `idxmax` argument to extract the argmax label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Making Predictions Using Majority Overall Classifier\n",
      "[Extracting Negated Keywords]\n",
      "[Tokenizing Text Data]\n",
      "[Learning N-Grams within Vocabulary (Tokens w/o Keyword)]\n",
      "[Applying Phrase Transformation 1/2\n",
      "[Rephrasing]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 9521.69it/s]\n",
      "[Applying Phrase Transformation 2/2\n",
      "[Rephrasing]: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 21024.08it/s]\n",
      "[Learning N-Grams within Vocabulary (Tokens w/ Keyword)]\n",
      "[Applying Phrase Transformation 1/2\n",
      "[Rephrasing]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 6673.51it/s]\n",
      "[Applying Phrase Transformation 2/2\n",
      "[Rephrasing]: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 11366.68it/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Making Predictions]: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 137.27it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exclude</th>\n",
       "      <th>negative</th>\n",
       "      <th>neutral</th>\n",
       "      <th>positive</th>\n",
       "      <th>argmax</th>\n",
       "      <th>context</th>\n",
       "      <th>keyword</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.222004</td>\n",
       "      <td>0.561231</td>\n",
       "      <td>0.042567</td>\n",
       "      <td>0.174198</td>\n",
       "      <td>negative</td>\n",
       "      <td>miss doe is a charming, 73 year old women who ...</td>\n",
       "      <td>charming</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.222004</td>\n",
       "      <td>0.561231</td>\n",
       "      <td>0.042567</td>\n",
       "      <td>0.174198</td>\n",
       "      <td>negative</td>\n",
       "      <td>of heart pain. unfortunately, not a good histo...</td>\n",
       "      <td>historian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    exclude  negative   neutral  positive    argmax  \\\n",
       "0  0.222004  0.561231  0.042567  0.174198  negative   \n",
       "1  0.222004  0.561231  0.042567  0.174198  negative   \n",
       "\n",
       "                                             context    keyword  \n",
       "0  miss doe is a charming, 73 year old women who ...   charming  \n",
       "1  of heart pain. unfortunately, not a good histo...  historian  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Making Predictions Using Keyword + Context Classifier\n",
      "[Extracting Negated Keywords]\n",
      "[Tokenizing Text Data]\n",
      "[Learning N-Grams within Vocabulary (Tokens w/o Keyword)]\n",
      "[Applying Phrase Transformation 1/2\n",
      "[Rephrasing]: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 20661.60it/s]\n",
      "[Applying Phrase Transformation 2/2\n",
      "[Rephrasing]: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 12354.36it/s]\n",
      "[Learning N-Grams within Vocabulary (Tokens w/ Keyword)]\n",
      "[Applying Phrase Transformation 1/2\n",
      "[Rephrasing]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 7695.97it/s]\n",
      "[Applying Phrase Transformation 2/2\n",
      "[Rephrasing]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 9127.97it/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Making Predictions]: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 105.49it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exclude</th>\n",
       "      <th>negative</th>\n",
       "      <th>neutral</th>\n",
       "      <th>positive</th>\n",
       "      <th>argmax</th>\n",
       "      <th>context</th>\n",
       "      <th>keyword</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.037252</td>\n",
       "      <td>0.058752</td>\n",
       "      <td>0.018344</td>\n",
       "      <td>0.885651</td>\n",
       "      <td>positive</td>\n",
       "      <td>miss doe is a charming, 73 year old women who ...</td>\n",
       "      <td>charming</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.063075</td>\n",
       "      <td>0.627966</td>\n",
       "      <td>0.237639</td>\n",
       "      <td>0.071320</td>\n",
       "      <td>negative</td>\n",
       "      <td>of heart pain. unfortunately, not a good histo...</td>\n",
       "      <td>historian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    exclude  negative   neutral  positive    argmax  \\\n",
       "0  0.037252  0.058752  0.018344  0.885651  positive   \n",
       "1  0.063075  0.627966  0.237639  0.071320  negative   \n",
       "\n",
       "                                             context    keyword  \n",
       "0  miss doe is a charming, 73 year old women who ...   charming  \n",
       "1  of heart pain. unfortunately, not a good histo...  historian  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Making Predictions Using Clinical BERT Classifier\n",
      "[Running Evaluation]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.98it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exclude</th>\n",
       "      <th>negative</th>\n",
       "      <th>neutral</th>\n",
       "      <th>positive</th>\n",
       "      <th>argmax</th>\n",
       "      <th>context</th>\n",
       "      <th>keyword</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000714</td>\n",
       "      <td>0.001035</td>\n",
       "      <td>0.000712</td>\n",
       "      <td>0.997539</td>\n",
       "      <td>positive</td>\n",
       "      <td>miss doe is a charming, 73 year old women who ...</td>\n",
       "      <td>charming</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.005566</td>\n",
       "      <td>0.988215</td>\n",
       "      <td>0.005839</td>\n",
       "      <td>0.000379</td>\n",
       "      <td>negative</td>\n",
       "      <td>of heart pain. unfortunately, not a good histo...</td>\n",
       "      <td>historian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    exclude  negative   neutral  positive    argmax  \\\n",
       "0  0.000714  0.001035  0.000712  0.997539  positive   \n",
       "1  0.005566  0.988215  0.005839  0.000379  negative   \n",
       "\n",
       "                                             context    keyword  \n",
       "0  miss doe is a charming, 73 year old women who ...   charming  \n",
       "1  of heart pain. unfortunately, not a good histo...  historian  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Iterate Through Models\n",
    "for model_name, model in zip([\"Majority Overall\",\"Keyword + Context\",\"Clinical BERT\"],\n",
    "                             [majority_model, keyword_context_model, bert_model]):\n",
    "\n",
    "    print(f\">> Making Predictions Using {model_name} Classifier\")\n",
    "\n",
    "    ## Run Prediction Procedure\n",
    "    predictions = model.predict(text=example_text,\n",
    "                                keywords=example_keywords)\n",
    "\n",
    "    ## Augment Predictions with Inputs\n",
    "    predictions[\"argmax\"] = predictions.idxmax(axis=1)\n",
    "    predictions[\"context\"] = example_text\n",
    "    predictions[\"keyword\"] = example_keywords\n",
    "\n",
    "    ## Show Predictions\n",
    "    _ = display(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concluding Thoughts\n",
    "\n",
    "If you reached this point, you should now be ready to start using our models! Congratulations!\n",
    "\n",
    "This notebook and our high-level API is intended to provide an easy entry point to characterizing stigmatizing language in medical records. If you are interested in developing custom solutions, we encourage you to read through and try out some of the code in the `scripts/` and `stigma/` directories. The API you interacted with today only supports a subset of functionality that is available throughout the broader package.\n",
    "\n",
    "If you have ideas for improving the toolkit or encounter any issues with our code, please feel free to submit an issue on our GitHub page or contact [Keith Harrigian](mailto:kharrigian@jhu.edu)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}