BioDEX_DSPy.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3g7UMaMuGhZt"
      },
      "source": [
        "# BioDEX-DSPy\n",
        "\n",
        "Biomedical publications can describe adverse drug reactions. Extracting and coding these reactions is a real-world task which is vital for public drug safety, but automating this procedure has been challenging [(D'Oosterlinck et al., 2023)](https://arxiv.org/abs/2305.13395).\n",
        "\n",
        "In this notebook, we will build a state-of-the-art reaction extractor using the [DSPy framework](https://github.com/stanfordnlp/dspy) [(Khattab et al., 2023)](https://arxiv.org/pdf/2310.03714.pdf). We will combine in-context learning and retrieval to build a pipeline that reads a full biomedical paper, predicts adverse reactions, and encodeds these reactions into one of ~26k admissable reaction terms."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wc3E6-aHVi5"
      },
      "source": [
        "## Setup\n",
        "- downloads data\n",
        "- installs packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "And-63zyGQjz",
        "outputId": "7b6df92f-710d-4e93-91f1-1bda93042b98"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The autoreload extension is already loaded. To reload it, use:\n",
            "  %reload_ext autoreload\n",
            "File ‘embeddings[FremyCompany--BioLORD-STAMB2-v1].pt’ already there; not retrieving.\n",
            "File ‘reaction_terms.txt’ already there; not retrieving.\n",
            "File ‘cache.zip’ already there; not retrieving.\n",
            "caution: filename not matched:  -y\n",
            "Requirement already satisfied: dspy-ai in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (2.0.7)\n",
            "Requirement already satisfied: datasets in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (2.14.4)\n",
            "Requirement already satisfied: sentence_transformers in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (2.2.2)\n",
            "Requirement already satisfied: torch in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (2.1.1)\n",
            "Requirement already satisfied: openai in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (0.28.1)\n",
            "Requirement already satisfied: backoff<=2.2.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (2.2.1)\n",
            "Requirement already satisfied: joblib<=1.3.2 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (1.3.2)\n",
            "Requirement already satisfied: pandas<=2.1.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (2.0.3)\n",
            "Requirement already satisfied: regex<=2023.10.3 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (2023.8.8)\n",
            "Requirement already satisfied: ujson<=5.8.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (5.8.0)\n",
            "Requirement already satisfied: tqdm<=4.66.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (4.66.1)\n",
            "Requirement already satisfied: requests<=2.31.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (2.31.0)\n",
            "Requirement already satisfied: optuna<=3.4.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from dspy-ai) (3.4.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (1.25.2)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (12.0.1)\n",
            "Requirement already satisfied: dill<0.3.8,>=0.3.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (0.3.7)\n",
            "Requirement already satisfied: xxhash in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (3.3.0)\n",
            "Requirement already satisfied: multiprocess in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (0.70.15)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (3.8.5)\n",
            "Requirement already satisfied: huggingface-hub<1.0.0,>=0.14.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (0.16.4)\n",
            "Requirement already satisfied: packaging in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (23.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from datasets) (6.0.1)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (4.31.0)\n",
            "Requirement already satisfied: torchvision in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (0.16.1)\n",
            "Requirement already satisfied: scikit-learn in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (1.3.2)\n",
            "Requirement already satisfied: scipy in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (1.11.3)\n",
            "Requirement already satisfied: nltk in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (3.8.1)\n",
            "Requirement already satisfied: sentencepiece in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sentence_transformers) (0.1.99)\n",
            "Requirement already satisfied: filelock in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torch) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torch) (4.7.1)\n",
            "Requirement already satisfied: sympy in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torch) (1.12)\n",
            "Requirement already satisfied: networkx in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torch) (3.1)\n",
            "Requirement already satisfied: jinja2 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torch) (3.1.2)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (3.2.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: alembic>=1.5.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from optuna<=3.4.0->dspy-ai) (1.12.1)\n",
            "Requirement already satisfied: colorlog in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from optuna<=3.4.0->dspy-ai) (6.7.0)\n",
            "Requirement already satisfied: sqlalchemy>=1.3.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from optuna<=3.4.0->dspy-ai) (2.0.23)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from pandas<=2.1.1->dspy-ai) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from pandas<=2.1.1->dspy-ai) (2023.3)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from pandas<=2.1.1->dspy-ai) (2023.3)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from requests<=2.31.0->dspy-ai) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from requests<=2.31.0->dspy-ai) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from requests<=2.31.0->dspy-ai) (2023.7.22)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.13.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.3.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from jinja2->torch) (2.1.3)\n",
            "Requirement already satisfied: click in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from nltk->sentence_transformers) (8.1.7)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from scikit-learn->sentence_transformers) (3.2.0)\n",
            "Requirement already satisfied: mpmath>=0.19 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from sympy->torch) (1.3.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from torchvision->sentence_transformers) (10.1.0)\n",
            "Requirement already satisfied: Mako in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from alembic>=1.5.0->optuna<=3.4.0->dspy-ai) (1.3.0)\n",
            "Requirement already satisfied: six>=1.5 in /Users/kldooste/miniconda3/envs/trl/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas<=2.1.1->dspy-ai) (1.16.0)\n"
          ]
        }
      ],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "import sys\n",
        "import os\n",
        "\n",
        "# Set up the cache for this notebook\n",
        "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join('.', 'cache')\n",
        "\n",
        "# download biomedical terms and embeddings\n",
        "!wget -nc 'https://www.dropbox.com/scl/fi/5ywdea0xjkb10os1o6ryj/embeddings-FremyCompany-BioLORD-STAMB2-v1.pt?rlkey=nek172noiyrpn588jt7dunl66&dl=0' -O 'embeddings[FremyCompany--BioLORD-STAMB2-v1].pt'\n",
        "!wget -nc 'https://www.dropbox.com/scl/fi/f92z0vg42icsn5g89f3wu/reaction_terms.txt?rlkey=ot8qasqr3r9getbn9epyji0aa&dl=0' -O \"reaction_terms.txt\"\n",
        "!wget -nc 'https://www.dropbox.com/scl/fi/cgu0eal9m7q0xrswp49g5/cache.zip?rlkey=x3lnpc5vz1t0di7igzthrky8h&dl=0' -O \"cache.zip\"\n",
        "!unzip -n -q cache.zip -y\n",
        "\n",
        "# install packages\n",
        "!pip install dspy-ai datasets sentence_transformers torch openai\n",
        "\n",
        "import datasets\n",
        "import dspy\n",
        "from dspy.evaluate import Evaluate\n",
        "import tqdm\n",
        "import sentence_transformers\n",
        "from sentence_transformers import SentenceTransformer\n",
        "import torch\n",
        "import os\n",
        "from functools import lru_cache\n",
        "from collections import defaultdict, Counter\n",
        "import math\n",
        "import re\n",
        "from functools import partial\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Q7JRIQCVqj8"
      },
      "source": [
        "## Setting up the data\n",
        "We will be using the BioDEX dataset [(D'Oosterlinck et al., 2023)](https://arxiv.org/abs/2305.13395). BioDEX features full biomedical publications and associated expert-created drug safety reports. For our purposes, we will use the biomedical reactions discussed in these reports.\n",
        "\n",
        "- [BioDEX Github](https://github.com/KarelDO/BioDEX)\n",
        "- [BioDEX HuggingFace](https://huggingface.co/BioDEX)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VCUFnd_cbCy_"
      },
      "source": [
        "Let's first write some helper functions to parse and normalize biomedical reactions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "RpwN113lH6HD",
        "outputId": "d07858f3-5201-453f-abed-455b87d69abe"
      },
      "outputs": [],
      "source": [
        "# normalize one reaction string\n",
        "def normalize(reaction: str) -> str:\n",
        "    # Remove leading and trailing newlines\n",
        "    reaction = reaction.strip('\\n')\n",
        "\n",
        "    # Remove leading and trailing punctuation and newlines\n",
        "    reaction = re.sub(r'^[^\\w\\s]+|[^\\w\\s]+$', '', reaction, flags=re.UNICODE)\n",
        "\n",
        "    # Remove leading and trailing newlines\n",
        "    reaction = reaction.strip('\\n')\n",
        "\n",
        "    return reaction.strip().lower()\n",
        "\n",
        "# given a csv string of reactions, parse into a list\n",
        "def extract_reactions_from_string(reactions: str) -> list[str]:\n",
        "  return [normalize(r) for r in reactions.split(',')]\n",
        "\n",
        "# given a list of csv trings of reactions, parse into a list\n",
        "def extract_reactions_from_strings(reactions: list[str]) -> list[str]:\n",
        "  reactions = [normalize(r) for r in reactions]\n",
        "  reactions = \", \".join(reactions)\n",
        "  return extract_reactions_from_string(reactions)\n",
        "\n",
        "# process a biodex datapoint\n",
        "def preprocess_example(example: dict) -> dspy.Example:\n",
        "    title = example['title']\n",
        "    abstract = example['abstract']\n",
        "    context = example['fulltext_processed'].split('\\n\\nTEXT:\\n', 1)[-1]\n",
        "    reactions = extract_reactions_from_string(example['reactions'])\n",
        "\n",
        "    example = dict(title=title, abstract=abstract, context=context, reactions=reactions)\n",
        "    example['labels'] = dspy.Example(reactions=reactions)\n",
        "\n",
        "    return example"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ikbtTKSbL_e"
      },
      "source": [
        "Great, now let's load the dataset and create a tiny training and validation set to easily apply in-context learning without being too expensive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "4xa9DkssVrJm",
        "outputId": "55ef8169-5b4b-4219-9ae3-d8a8240e001f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  9%|▊         | 1000/11543 [00:00<00:01, 6514.35it/s]\n",
            " 17%|█▋        | 500/2886 [00:00<00:00, 6430.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1000 500\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "dataset = datasets.load_dataset(\"BioDEX/BioDEX-Reactions\")\n",
        "official_trainset, official_devset = dataset['train'], dataset['validation']\n",
        "trainset, devset = [], []\n",
        "\n",
        "for example in tqdm.tqdm(official_trainset):\n",
        "    if len(trainset) >= 1000: break\n",
        "    trainset.append(preprocess_example(example))\n",
        "\n",
        "for example in tqdm.tqdm(official_devset):\n",
        "    if len(devset) >= 500: break\n",
        "    devset.append(preprocess_example(example))\n",
        "\n",
        "trainsetX = [dspy.Example(**x).with_inputs('title', 'abstract', 'context', 'labels') for x in trainset]\n",
        "trainset = [dspy.Example(**x).with_inputs('title', 'abstract', 'context') for x in trainset]\n",
        "devsetX = [dspy.Example(**x).with_inputs('title', 'abstract', 'context', 'labels') for x in devset]\n",
        "devset = [dspy.Example(**x).with_inputs('title', 'abstract', 'context') for x in devset]\n",
        "\n",
        "print(len(trainset), len(devset))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13JR0wybbVgn"
      },
      "source": [
        "Let's look at one example to familiarize ourself.\n",
        "\n",
        "Each datapoints has a:\n",
        "- titel\n",
        "- abstract\n",
        "- context (body of the paper)\n",
        "- a list of associated reactions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "id": "gTL1bTAKXOH6",
        "outputId": "7448b0cb-607c-4679-cb30-20aa1e77f009"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "HIV-1 Drug Resistance by Ultra-Deep Sequencing Following Short Course Zidovudine, Single-Dose Nevirapine, and Single-Dose Tenofovir with Emtricitabine for Prevention of Mother-to-Child Transmission.\n",
            "\n",
            "Antiretroviral drug resistance following pMTCT strategies remains a significant problem. With rapid advancements in next generation sequencing technologies, there is more focus on HIV drug-resistant variants of low frequency, or the so-called minority variants. In South Africa, AZT monotherapy for pMTCT, similar to World Health Organization option A, has been used since 2008. In 2010, a single dose of co-formulated TDF/FTC was included in the strategy for prevention of resistance conferred by single-dose nevirapine (sd NVP). The study was conducted in KwaZulu-Natal, South Africa, among pMTCT participants who received AZT monotherapy from 14 weeks of gestation, intrapartum AZT and sd NVP, and postpartum sd TDF/FTC. Twenty-six specimens collected at 6 weeks post-delivery were successfully sequenced using 454 ultra-deep sequencing. Non-nucleoside reverse transcriptase inhibitor (NNRTI) resistance was detected in 17 of 26 (65%) patients, 2 (7%) had Thymidine analogue mutations, and 3 (11%) had K65R. Of the 17 patients with NNRTI resistance, 11 (65%) had high-level NNRTI resistance, whereas 6 (35%) had intermediate NNRTI resistance. The levels of NNRTI resistance are much higher than would be expected, given the inclusion of antepartum AZT and postpartum TDF/FTC. This high level of NNRTI resistance could impact future NNRTI-containing treatment for a large proportion of pMTCT-exposed women. The detection of Thymidine analogue mutations highlights the need to understand the clinical impact of these on AZT-containing antiretroviral treatment in women exposed to AZT monotherapy.\n",
            "\n",
            "['drug resistance', 'exposure during pregnancy', 'viral mutation identified']\n"
          ]
        }
      ],
      "source": [
        "# show one title and abstract, as well as the reactions in the final expert-created drug safety report\n",
        "print(trainset[0].title)\n",
        "print()\n",
        "print(trainset[0].abstract)\n",
        "print()\n",
        "print(trainset[0].labels().reactions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XaHeiYtCXGvo"
      },
      "source": [
        "## Setting up evaluation code\n",
        "\n",
        "We will optimize Recall@10: given 10 unique reaction predictions from our pipeline, how many of the true predictions did we find.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "PM8kejUNWaBw",
        "outputId": "86f421d3-c3ad-449f-a99d-99b4f372fc7e"
      },
      "outputs": [],
      "source": [
        "def metric_recall(gold: list[str], pred: list[str]) -> float:\n",
        "  \"\"\" Given a gold and predicted list of reactions, normalize and compute recall.\"\"\"\n",
        "  gold = [normalize(r) for r in gold]\n",
        "  pred = [normalize(r) for r in pred]\n",
        "\n",
        "  gold, pred = set(gold), set(pred)\n",
        "\n",
        "  intersection = gold.intersection(pred)\n",
        "\n",
        "  recall = len(intersection) / len(gold)\n",
        "  return recall\n",
        "\n",
        "def metric_recallK(gold: list[str], pred: list[str], K:int=10) -> float:\n",
        "  return metric_recall(gold, pred[:K])\n",
        "\n",
        "# wrap the recall@K metric so it can take dspy Examples\n",
        "def dspy_metric_recall10(gold: dspy.Example, pred: dspy.Example, trace=None) -> float:\n",
        "  return metric_recallK(gold.reactions, pred.reactions, K=10)\n",
        "\n",
        "def dspy_metric_recall20(gold: dspy.Example, pred: dspy.Example, trace=None) -> float:\n",
        "  return metric_recallK(gold.reactions, pred.reactions, K=20)\n",
        "  \n",
        "def dspy_metric_recall30(gold: dspy.Example, pred: dspy.Example, trace=None) -> float:\n",
        "  return metric_recallK(gold.reactions, pred.reactions, K=30)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8DISiPz-bMVR"
      },
      "source": [
        "## Set up a neural grounding\n",
        "The output space of BioDEX contains ~26k distinct reaction classes. Even naively enumerating these classes, without demonstrations, might overflow the in-context learning window. Thus, we need a more scalable way of combining reactions with in-context learning. To solve this, we set up a retrieval-based grounder. This grounded embeds the reaction names and retrieves the K nearest neighbors given an ungrounded reaction prediction. Additionally, the grounder returns the predicted semantic similarty, so that we can use this later on as a measure of prediction confidence. Optinally, the grounder incorporates the prior distribution of reactions in the BioDEX trainset, so that prior statistics can be combined with in-context learning.\n",
        "\n",
        "We use the BioLORD model [(Remy et al., 2022)](https://arxiv.org/abs/2210.11892) to produce our embeddings, but feel free to experiment with any type of embeddings.\n",
        "\n",
        "Switch the runtime to GPU when you are using a new encoder to speed up embedding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "CXwDHrO_bO2w",
        "outputId": "032e33db-f6cb-4c59-e3f9-c52f3d90953e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "No sentence-transformers model found with name /Users/kldooste/.cache/torch/sentence_transformers/FremyCompany_BioLORD-STAMB2-v1. Creating a new one with MEAN pooling.\n"
          ]
        }
      ],
      "source": [
        "class ReactionGrounder():\n",
        "  \"\"\" Matches a given reaction string to one of the BioDEX reaction terms.\n",
        "  Implements several ways of grounding reactions. Each grounding function returns a list of length K.\n",
        "  Each list item consists of a similarity score and the associated grounded term. \"\"\"\n",
        "\n",
        "  def __init__(self, model_name:str='FremyCompany/BioLORD-STAMB2-v1', trainset: list[dict]=[]):\n",
        "\n",
        "      self.model_name = model_name\n",
        "      self.friendly_model_name = self.model_name.replace('/','--')\n",
        "      self.trainset = trainset\n",
        "\n",
        "      self.model = SentenceTransformer(self.model_name)\n",
        "      self.model.to('cpu')\n",
        "\n",
        "      self.reaction_terms = self._load_reaction_terms()\n",
        "      self.reaction_terms_to_count = self._calculate_counts()\n",
        "      self.reaction_embeddings = self._load_embeddings()\n",
        "\n",
        "  def _load_reaction_terms(self) -> list[str]:\n",
        "      \"\"\"Get all reaction terms and normalize them.\"\"\"\n",
        "      reaction_filename = 'reaction_terms.txt'\n",
        "      return [normalize(r) for r in open(reaction_filename).read().splitlines()]\n",
        "\n",
        "  def _load_embeddings(self) -> torch.Tensor:\n",
        "      \"\"\"Load or create embeddings for all reaction terms.\"\"\"\n",
        "      reaction_embeddings_filename = f'embeddings[{self.friendly_model_name}].pt'\n",
        "\n",
        "      # If the file exists, load. Else, create embeddings.\n",
        "      if os.path.isfile(reaction_embeddings_filename):\n",
        "          with open(reaction_embeddings_filename, \"rb\") as f:\n",
        "              reaction_embeddings = torch.load(f, map_location=torch.device('cpu'))\n",
        "      else:\n",
        "          self.model.to(torch.device('cuda' if torch.cuda.is_available() else 'cpu'))\n",
        "          reaction_embeddings = self.model.encode(self.reaction_terms, convert_to_tensor = True, show_progress_bar = True)\n",
        "          with open(reaction_embeddings_filename, \"wb\") as f:\n",
        "              torch.save(reaction_embeddings, f)\n",
        "          self.model.to(torch.device('cpu'))\n",
        "      return reaction_embeddings\n",
        "\n",
        "  def _calculate_counts(self) -> dict[str, int]:\n",
        "      \"\"\" Given a training set, count how many times each reaction occured as prior.\"\"\"\n",
        "      reactions = []\n",
        "      for example in self.trainset:\n",
        "        reactions.extend(extract_reactions_from_string(example['reactions']))\n",
        "      counts = defaultdict(lambda: 0, Counter(reactions))\n",
        "      return counts\n",
        "\n",
        "  @lru_cache(maxsize=100000)\n",
        "  def ground(self, reaction: str, K:int=3) -> list[tuple[float, str]]:\n",
        "      \"\"\" Finds K closest matches based on semantic embedding similarity. \"\"\"\n",
        "      query_embeddings = self.model.encode(reaction, convert_to_tensor = True)\n",
        "      query_result = sentence_transformers.util.semantic_search(query_embeddings, self.reaction_embeddings, query_chunk_size=64, top_k=K)[0]\n",
        "\n",
        "      # get (score, term) tuples\n",
        "      matches = []\n",
        "      for result in query_result:\n",
        "        score = result['score']\n",
        "        term = self.reaction_terms[result['corpus_id']]\n",
        "        matches.append((score, term))\n",
        "\n",
        "      return sorted(matches, reverse=True)\n",
        "\n",
        "  @lru_cache(maxsize=100000)\n",
        "  def ground_with_prior(self, reaction, K=3):\n",
        "      \"\"\" Finds 3*K closest matches based on semantic embedding similarity.\n",
        "      Incorporates the prior counts, returns K most likely terms.\"\"\"\n",
        "      matches = self.ground(reaction, K=3*K)\n",
        "\n",
        "      # heuristically incorporate prior into the similarity scores\n",
        "      new_matches = []\n",
        "      for score, term in matches:\n",
        "        prior = self.reaction_terms_to_count[term]\n",
        "\n",
        "        score = score * max(2,math.log(prior + math.e))\n",
        "        # score = score * math.log(prior + math.e)\n",
        "        # score = score * math.log(prior + .1)\n",
        "        new_matches.append((score, term))\n",
        "\n",
        "      return sorted(new_matches, reverse=True)[:K]\n",
        "\n",
        "# create the grounder\n",
        "grounder = ReactionGrounder(trainset=official_trainset)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqtL2TnCcPkd"
      },
      "source": [
        "Let's play with some of the groundings! Look how different queries get grounded, and how incorporating a prior term changes some of the retrieved reactions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 436
        },
        "id": "xDCINl8ubO1K",
        "outputId": "e0f58154-1b16-4e26-fb82-ef696377dacb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Query:  pain\n",
            "\t Ground without prior:\n",
            "\t\tpain - 1.000000238418579 \n",
            "\t\ttenderness - 0.8098210096359253 \n",
            "\t\tinflammatory pain - 0.7327480316162109 \n",
            "\t\theadache - 0.7172471284866333 \n",
            "\t\tdiscomfort - 0.7165692448616028 \n",
            "\t Ground with prior:\n",
            "\t\tpain - 4.679520433152032 \n",
            "\t\theadache - 3.8610861348756327 \n",
            "\t\ttenderness - 2.5640926425656185 \n",
            "\t\tdiscomfort - 2.0598477859496542 \n",
            "\t\tmusculoskeletal pain - 1.8701876261560002 \n",
            "Query:  fever\n",
            "\t Ground without prior:\n",
            "\t\tpyrexia - 0.923504114151001 \n",
            "\t\tbody temperature increased - 0.8943093419075012 \n",
            "\t\thyperpyrexia - 0.8813413381576538 \n",
            "\t\tfebrile infection - 0.7856964468955994 \n",
            "\t\tbody temperature abnormal - 0.7851898670196533 \n",
            "\t Ground with prior:\n",
            "\t\tpyrexia - 5.612130707861001 \n",
            "\t\tchills - 2.5693159432673554 \n",
            "\t\thyperthermia - 2.3369348762331836 \n",
            "\t\tbody temperature increased - 2.2742648513245327 \n",
            "\t\thyperpyrexia - 2.09049834993291 \n",
            "Query:  i have a runny nose\n",
            "\t Ground without prior:\n",
            "\t\tnasal congestion - 0.6704621315002441 \n",
            "\t\tnasal flaring - 0.6388666033744812 \n",
            "\t\trebound nasal congestion - 0.6226147413253784 \n",
            "\t\tnasal pruritus - 0.6181106567382812 \n",
            "\t\trhinorrhoea - 0.611099362373352 \n",
            "\t Ground with prior:\n",
            "\t\tnasal congestion - 1.4518335707262786 \n",
            "\t\trhinorrhoea - 1.323288113763861 \n",
            "\t\tnasal flaring - 1.2777332067489624 \n",
            "\t\trebound nasal congestion - 1.2452294826507568 \n",
            "\t\tnasal pruritus - 1.2362213134765625 \n"
          ]
        }
      ],
      "source": [
        "K = 5\n",
        "queries = [\n",
        "    'pain',\n",
        "    'fever',\n",
        "    'i have a runny nose'\n",
        "]\n",
        "\n",
        "for q in queries:\n",
        "  result = grounder.ground(q, K=K)\n",
        "  result_with_prior = grounder.ground_with_prior(q, K=K)\n",
        "\n",
        "  print(\"Query: \", q)\n",
        "\n",
        "  print(\"\\t Ground without prior:\")\n",
        "  for score, term in result:\n",
        "    print(f\"\\t\\t{term} - {score} \")\n",
        "  print(\"\\t Ground with prior:\")\n",
        "  for score, term in result_with_prior:\n",
        "    print(f\"\\t\\t{term} - {score} \")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kqXyOOZZashX"
      },
      "source": [
        "## DSPy program\n",
        "We are almost ready to build our DSPy program!\n",
        "\n",
        "Our program will have the following input-output behavior:\n",
        "- input: title, abstract, body of a paper\n",
        "- output: a list of predicted reactions, sorted by how confident we are.\n",
        "\n",
        "There are many potential pipelines we can construct. In this work, we will:\n",
        "1. chunk a paper in different windows\n",
        "2. predict ungrounded reaction terms per chunk using in-context learning\n",
        "3. find the top K nearest grounded neighbors per ungrounded reaction using the grounder defined above\n",
        "4. resolve all grounded reactions into a final prediction using the similarity scores from the grounding step\n",
        "\n",
        "Because of the abstractions provided by DSPy, we can quickly iterate on the design of this pipeline. For example, if a reaction term is predicted in two chunks, should we add the confidence of this term or take the maximum for the final prediction? This is all trivally implemented in DSPy.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vmqY1OQ3PuJ7"
      },
      "source": [
        "Let's first implement the function that is used to resolve a list of predicted reactions and scores:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "ivB79U6oD3we",
        "outputId": "d6accba2-9f86-4e56-fdda-ef2ea9af021d"
      },
      "outputs": [],
      "source": [
        "from operator import add\n",
        "\n",
        "def resolve_reactions(reactions: list[(float, str)], resolve_f = add) -> list[str]:\n",
        "  \"\"\" Applies a resolve function across all duplicate predicted reactions to aggregate their similarity score.\n",
        "  Sorts the resulting reactions according to aggregated score.\"\"\"\n",
        "\n",
        "  reactions_to_score = defaultdict(lambda: .0)\n",
        "\n",
        "  for score, term in reactions:\n",
        "    reactions_to_score[term] = resolve_f(reactions_to_score[term], score)\n",
        "\n",
        "  reactions = sorted(reactions_to_score.items(), key=lambda x: x[1], reverse=True)\n",
        "  reactions = [(r[1], r[0]) for r in reactions]\n",
        "  return reactions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPoeOlgaQDpO"
      },
      "source": [
        "Let's also implement a chunker class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "5MUDQzCUp8cf",
        "outputId": "77311771-9433-488e-b0a8-2167a31dae41"
      },
      "outputs": [],
      "source": [
        "class Chunker:\n",
        "    def __init__(self, context_window=3000, max_windows=5):\n",
        "        self.context_window = context_window\n",
        "        self.max_windows = max_windows\n",
        "        self.window_overlap = 0.02\n",
        "\n",
        "    def __call__(self, paper):\n",
        "        snippet_idx = 0\n",
        "\n",
        "        while snippet_idx < self.max_windows and paper:\n",
        "            endpos = int(self.context_window * (1.0 + self.window_overlap))\n",
        "            snippet, paper = paper[:endpos], paper[endpos:]\n",
        "\n",
        "            next_newline_pos = snippet.rfind('\\n')\n",
        "            if paper and next_newline_pos != -1 and next_newline_pos >= self.context_window // 2:\n",
        "                paper = snippet[next_newline_pos+1:] + paper\n",
        "                snippet = snippet[:next_newline_pos]\n",
        "\n",
        "            yield snippet_idx, snippet.strip()\n",
        "            snippet_idx += 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lfmfZ5zoQJ8t"
      },
      "source": [
        "Great! Now let's build the DSPy program, starting with the signature for the ChainOfThought in-context module. Notice how this is everything we need to use ChainOfThought. No sloppy prompt-engineering, but well-defined interfaces:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "TWMKdU8eQJu6",
        "outputId": "fd42ab74-b27b-4daa-c783-9c00fc4fecdc"
      },
      "outputs": [],
      "source": [
        "class PredictReactions(dspy.Signature):\n",
        "    __doc__ = f\"\"\"Given a snippet from a medical article, identify the adverse drug reactions affecting the patient. If none are mentioned in the snippet, say '\\n'.\"\"\"\n",
        "\n",
        "    title = dspy.InputField()\n",
        "    context = dspy.InputField()\n",
        "    reactions = dspy.OutputField(desc=\"list of comma-separated adverse drug reactions\", format=lambda x: ', '.join(x) if isinstance(x, list) else x)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0FgHCguRc_Ln"
      },
      "source": [
        "Awesome, let's create our pipeline. `PredictThenGround` will use ChainOfThought and chunking to predict reactions, and will optionally use a grounding function to resolve these to the final space of reactions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "hVrLgbZvbJ97",
        "outputId": "3262dc61-458c-4008-f26d-aaa954593643"
      },
      "outputs": [],
      "source": [
        "# Grounding with prior\n",
        "HINT = \"Is any of the following reactions discussed in the article snippet? The valid candidates are:\"\n",
        "\n",
        "class PredictThenGround(dspy.Module):\n",
        "    def __init__(self, context_window=3000, max_windows=5, num_preds=1, grounding_function = lambda r: [(1.0, r)], resolve_function = add):\n",
        "        super().__init__()\n",
        "\n",
        "        # devides a biomedical paper into chunks\n",
        "        self.chunk = Chunker(context_window=context_window, max_windows=max_windows)\n",
        "        # given a paper title and body, predict a list of ungrounded reactions using a CoT\n",
        "        self.predict = dspy.ChainOfThoughtWithHint(PredictReactions, n=num_preds)\n",
        "        # ungrounded reaction -> grounded neighbors and similarity scores\n",
        "        self.grounding_function = grounding_function\n",
        "        # operator to combine similarity scores over multiple duplicate predictions\n",
        "        self.resolve_function = resolve_function\n",
        "\n",
        "    def forward(self, title, abstract, context, labels=None):\n",
        "        hint = f\"{HINT} {', '.join(labels.reactions)}.\" if labels else None\n",
        "        reactions = []\n",
        "\n",
        "        # for each chunk in the paper\n",
        "        for _, snippet in self.chunk(abstract + '\\n\\n' + context):\n",
        "            # use the LM to predict ungrounded reactions\n",
        "            chunk_reactions = self.predict(title=title, context=[snippet], hint=hint)\n",
        "            reactions.extend(extract_reactions_from_strings(chunk_reactions.completions.reactions))\n",
        "\n",
        "        # for each ungrounded reaction, get grounded reactions and grounding confidence\n",
        "        grounded_reactions = sorted([r for sublist in [self.grounding_function(r) for r in reactions] for r in sublist], reverse=True)\n",
        "        # aggregate / pool duplicate predictions and sort based on confidence\n",
        "        resolved_reactions = resolve_reactions(grounded_reactions, resolve_f=self.resolve_function)\n",
        "        # get a final list of grounded reactions\n",
        "        reactions = [r[1] for r in resolved_reactions]\n",
        "\n",
        "        # track all of these predictions\n",
        "        return dspy.Prediction(reactions=reactions, resolved_reactions=resolved_reactions, grounded_reactions=grounded_reactions, ungrounded_reactions=reactions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UscPSLizdQe3"
      },
      "source": [
        "Let's get ready to run some models! We'll use `gpt-3.5-turbo-1106`. All executions are cached, so you don't need an API to run this notebook. However, if you change anything, you might need to add your OpenAI API key to execute new GPT calls."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "eBYBiXWZyPjy",
        "outputId": "6eedc1f6-ad46-41b9-9cd9-fc495d485763"
      },
      "outputs": [],
      "source": [
        "# set DSPy to use gpt-3.5\n",
        "turbo11 = dspy.OpenAI(model='gpt-3.5-turbo-1106', max_tokens=150)\n",
        "dspy.settings.configure(lm=turbo11)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0gTKH8STdiWX"
      },
      "source": [
        "The following code creates an evaluation helper which we will use to evaluate all our DSPy programs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "kowkXwp-T_cz",
        "outputId": "119ca58c-6177-451d-e412-a167b450d355"
      },
      "outputs": [],
      "source": [
        "# create an evaluation helper function\n",
        "evaluateR10 = Evaluate(devset=trainset[100:150], metric=dspy_metric_recall10, num_threads=8, display_progress=True, display_table=0, max_errors=100)\n",
        "evaluateR20 = Evaluate(devset=trainset[100:150], metric=dspy_metric_recall20, num_threads=8, display_progress=True, display_table=0, max_errors=100)\n",
        "evaluateR30 = Evaluate(devset=trainset[100:150], metric=dspy_metric_recall30, num_threads=8, display_progress=True, display_table=0, max_errors=100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tNnIlnVhdsEQ"
      },
      "source": [
        "### Simplest CoT pipeline\n",
        "Let's evaluate the simplest CoT pipeline possible, which uses no grounding and only processes the first chunk of the paper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "0TjOZmXEUDie",
        "outputId": "9f8f0227-ede9-4347-e7ca-6cf7a84013f3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Predicted reactions:  ['thymidine analogue mutations', 'nnrti resistance']\n",
            "Gold reactions:  ['drug resistance', 'exposure during pregnancy', 'viral mutation identified']\n",
            "Recall@10:  0.0\n"
          ]
        }
      ],
      "source": [
        "# create the pipeline\n",
        "pipeline_no_grounding_no_chunking = PredictThenGround(max_windows=1)\n",
        "\n",
        "# get an example\n",
        "example = trainset[0]\n",
        "example_x = example.inputs()\n",
        "example_y = example.labels()\n",
        "\n",
        "# get a prediction\n",
        "prediction = pipeline_no_grounding_no_chunking(**example_x)\n",
        "\n",
        "# print the reactions predicted\n",
        "print('Predicted reactions: ', prediction.reactions)\n",
        "print('Gold reactions: ', example_y.reactions)\n",
        "print('Recall@10: ', dspy_metric_recall10(example_y, prediction))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upZLu0zuVb0s"
      },
      "source": [
        "This does not look that great. Let's run a more thorough evaluation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "id": "49Jqzuw3UnRj",
        "outputId": "c99bc2c6-a502-43db-8d3c-8ec3c49f70f4"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 2.5325460487225193 / 50  (5.1): 100%|██████████| 50/50 [00:00<00:00, 493.04it/s] "
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 2.5325460487225193 / 50  (5.1%)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "5.07"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluateR10(pipeline_no_grounding_no_chunking)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RMOgcH0AeEEq"
      },
      "source": [
        "5.07% Recall@10, not great."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nFtM2PLReI7Z"
      },
      "source": [
        "### CoT pipeline with chunking\n",
        "Let's evaluate the CoT pipeline now over 5 chunks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "id": "r4sN5dOFeRH6",
        "outputId": "d795e151-7c18-44ac-a82c-0e3c0c3d1db6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Predicted reactions:  ['nnrti resistance', 'y188c', 'y181c', 'v90i', 'v108i', 'v106m', 'v106a', 'thymidine analogue mutations (tams', 'thymidine analogue mutations', 'tams', 't69s', 'n/a', 'l100v', 'k70r', 'k65r', 'k103n', 'k101e', 'intermediate nnrti resistance', 'high-level resistance to nvp and efv', 'g190a', 'f227l', 'a98g', '']\n",
            "Gold reactions:  ['drug resistance', 'exposure during pregnancy', 'viral mutation identified']\n",
            "Recall@10:  0.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 4.870875160271446 / 50  (9.7): 100%|██████████| 50/50 [00:00<00:00, 405.58it/s]  "
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 4.870875160271446 / 50  (9.7%)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "9.74"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# create the pipeline\n",
        "pipeline_no_grounding = PredictThenGround(max_windows=5)\n",
        "\n",
        "# get an example\n",
        "example = trainset[0]\n",
        "example_x = example.inputs()\n",
        "example_y = example.labels()\n",
        "\n",
        "# get a prediction\n",
        "prediction = pipeline_no_grounding(**example_x)\n",
        "\n",
        "# print the reactions predicted\n",
        "print('Predicted reactions: ', prediction.reactions)\n",
        "print('Gold reactions: ', example_y.reactions)\n",
        "print('Recall@10: ', dspy_metric_recall10(example_y, prediction))\n",
        "\n",
        "evaluateR10(pipeline_no_grounding)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7OpdzTeeYRH"
      },
      "source": [
        "Still no traction on this example, but at least we are already outputting more reactions, increasing the final evaluation metric to 9.74!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOh-hNCzelNx"
      },
      "source": [
        "### CoT pipeline with chunking and per-chunk grounding\n",
        "Let's add the simplest grounding function. For every ungrounded prediction the LM makes, our pipeline will now find the 3 nearest grounded reaction terms.\n",
        "\n",
        "When the same reaction term is predicted multiple times, we resolve to adding the similarity scores."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "10yY5Rt1wjBp",
        "outputId": "4a82803b-2345-4be5-e52d-c9398d28d031"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Predicted reactions:  ['carbohydrate antigen 125', 'multiple-drug resistance', 'nat2 polymorphism', 'hiv tropism identified', 'carbohydrate antigen 27.29', 'nat1 polymorphism', 'hiv infection cdc category a', 'carbohydrate antigen 19-9', 'mutagenic effect', 'ret gene mutation', 'hiv antigen', 'k-ras status assay', 'carbohydrate antigen 549', 'carbohydrate antigen 50', 'c-kit gene mutation', 'transgenerational epigenetic inheritance', 'acquired gene mutation', 'sustained viral response', 'pathogen resistance', 'vascular resistance systemic', 'human t-cell lymphotropic virus infection', 'human t-cell lymphotropic virus type i infection', 'n-ras gene mutation', 'k-ras gene mutation', 'nyha classification', 'slow virus infection', 'melas syndrome', 'no adverse event', 'varicella post vaccine', 'covid-19', 'blood group o', 'viral mutation identified', 'hiv infection cdc group i', 'intermediate syndrome', 'reverse tri-iodothyronine', 'blood group b', 'aase syndrome', 'troponin t', 'tri-iodothyronine', 'c-kit gene status assay', 'divorced', 'homosexuality', 'married']\n",
            "Gold reactions:  ['drug resistance', 'exposure during pregnancy', 'viral mutation identified']\n",
            "Recall@10:  0.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.755123995371672 / 50  (25.5): 100%|██████████| 50/50 [00:10<00:00,  4.81it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.755123995371672 / 50  (25.5%)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "25.51"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "K = 3\n",
        "\n",
        "# greate the grounding function\n",
        "grounding_function = partial(grounder.ground, K=K)\n",
        "\n",
        "# greate the pipeline\n",
        "pipeline_with_ground = PredictThenGround(max_windows=5, grounding_function=grounding_function)\n",
        "\n",
        "# get an example\n",
        "example = trainset[0]\n",
        "example_x = example.inputs()\n",
        "example_y = example.labels()\n",
        "\n",
        "# get a predcition\n",
        "prediction = pipeline_with_ground(**example_x)\n",
        "\n",
        "# print the reactions predicted\n",
        "print('Predicted reactions: ', prediction.reactions)\n",
        "print('Gold reactions: ', example_y.reactions)\n",
        "print('Recall@10: ', dspy_metric_recall10(example_y, prediction))\n",
        "evaluateR10(pipeline_with_ground)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "INiTIUvEWLjc"
      },
      "source": [
        "Our final metric jumped to 25.51! For our initial example, we actually correctly predict 'viral mutation identified', but not as one of the top-10 reactions. Let's try to solve this by incorporating the prior distribution of the reactions in the grounding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmIGKErkgwVN"
      },
      "source": [
        "### CoT pipeline with chunking and per-chunk grounding and priors\n",
        "Let's use the grounding function that takes the priors into account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "hGOInOe8Wds1",
        "outputId": "be155825-d832-4cc3-88a9-906ebf8ceef5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Predicted reactions:  ['multiple-drug resistance', 'pathogen resistance', 'drug resistance', 'carbohydrate antigen 125', 'viral mutation identified', 'hiv tropism identified', 'carbohydrate antigen 27.29', 'acquired gene mutation', 'carbohydrate antigen 19-9', 'mutagenic effect', 'no adverse event', 'hiv antigen', 'hiv infection cdc category a', 'k-ras status assay', 'carbohydrate antigen 549', 'covid-19', 'virologic failure', 'carbohydrate antigen 50', 'c-kit gene mutation', 'transgenerational epigenetic inheritance', 'bk virus infection', 'ret gene mutation', 'human t-cell lymphotropic virus infection', 'human t-cell lymphotropic virus type i infection', 'n-ras gene mutation', 'nat2 polymorphism', 'k-ras gene mutation', 'nyha classification', 'slow virus infection', 'melas syndrome', 'covid-19 pneumonia', 'blood group o', 'intermediate syndrome', 'reverse tri-iodothyronine', 'aase syndrome', 'troponin t', 'tri-iodothyronine', 'divorced', 'homosexuality', 'married']\n",
            "Gold reactions:  ['drug resistance', 'exposure during pregnancy', 'viral mutation identified']\n",
            "Recall@10:  0.6666666666666666\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.939415830127901 / 50  (29.9): 100%|██████████| 50/50 [00:10<00:00,  4.83it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.939415830127901 / 50  (29.9%)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "29.88"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# let's make a better pipeline with grounding, now taking the prior into account.\n",
        "\n",
        "K = 3\n",
        "\n",
        "grounding_function = partial(grounder.ground_with_prior, K=K)\n",
        "pipeline_with_ground_and_prior = PredictThenGround(max_windows=5, grounding_function=grounding_function)\n",
        "\n",
        "# get an example\n",
        "example = trainset[0]\n",
        "example_x = example.inputs()\n",
        "example_y = example.labels()\n",
        "\n",
        "# get a predcition\n",
        "prediction = pipeline_with_ground_and_prior(**example_x)\n",
        "\n",
        "# print the reactions predicted\n",
        "print('Predicted reactions: ', prediction.reactions)\n",
        "print('Gold reactions: ', example_y.reactions)\n",
        "print('Recall@10: ', dspy_metric_recall10(example_y, prediction))\n",
        "\n",
        "evaluateR10(pipeline_with_ground_and_prior)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wxQsLhJBzxNZ"
      },
      "source": [
        "29.88%, Great! We got strong zero-shot performance by combining in-context learning with grounding and priors. Using the DSPy framework, this was all rather easy to code up and highly modular."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Analyze best program\n",
        "We're going to look at some prediction to understand what is going on. Open the output in a text editor if you want to see it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Gold reactions: \n",
            "\t ['blindness', 'consciousness fluctuating', 'encephalitis', 'encephalopathy', 'eye pain', 'eyelid ptosis', 'infection reactivation', 'loss of consciousness', 'mydriasis', 'off label use', 'ophthalmoplegia', 'orbital apex syndrome', 'paraesthesia', 'periorbital oedema', 'pupil fixed', 'pyrexia', 'varicella zoster virus infection']\n",
            "Predicted reactions: \n",
            "\t ['ophthalmoplegia', 'gaze palsy', 'extraocular muscle paresis', 'encephalopathy', 'no adverse event', 'encephalitis', 'hepatic encephalopathy', 'toxic encephalopathy', 'meningitis', 'meningitis aseptic', 'encephalitis viral', 'adverse drug reaction', 'panencephalitis', 'orbital apex syndrome', 'meningitis noninfective', 'no reaction on previous exposure to drug', 'orbital compartment syndrome', 'cavernous sinus syndrome']\n",
            "Ungrounded reactions: \n",
            "\t ['ophthalmoplegia', 'gaze palsy', 'extraocular muscle paresis', 'encephalopathy', 'no adverse event', 'encephalitis', 'hepatic encephalopathy', 'toxic encephalopathy', 'meningitis', 'meningitis aseptic', 'encephalitis viral', 'adverse drug reaction', 'panencephalitis', 'orbital apex syndrome', 'meningitis noninfective', 'no reaction on previous exposure to drug', 'orbital compartment syndrome', 'cavernous sinus syndrome']\n",
            "Recall@10:  0.17647058823529413\n",
            "Recall@20:  0.23529411764705882\n",
            "Recall@30:  0.23529411764705882\n",
            "Resolved reactions: \n",
            "\t\t ophthalmoplegia \t 6.130776065402945\n",
            "\t\t gaze palsy \t 5.7369917343615295\n",
            "\t\t extraocular muscle paresis \t 5.377774715423584\n",
            "\t\t encephalopathy \t 4.462664458635802\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t encephalitis \t 3.357534317725296\n",
            "\t\t hepatic encephalopathy \t 2.529780236592046\n",
            "\t\t toxic encephalopathy \t 2.487066355711432\n",
            "\t\t meningitis \t 2.3586926819038965\n",
            "\t\t meningitis aseptic \t 2.0873249991348004\n",
            "\t\t encephalitis viral \t 1.8044583591029408\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t panencephalitis \t 1.7522270679473877\n",
            "\t\t orbital apex syndrome \t 1.6739182472229004\n",
            "\t\t meningitis noninfective \t 1.5326719284057617\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t orbital compartment syndrome \t 1.4479095935821533\n",
            "\t\t cavernous sinus syndrome \t 1.2661194801330566\n",
            "Grounded reactions: \n",
            "\t\t encephalopathy \t 4.462664458635802\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t encephalitis \t 3.357534317725296\n",
            "\t\t hepatic encephalopathy \t 2.529780236592046\n",
            "\t\t toxic encephalopathy \t 2.487066355711432\n",
            "\t\t meningitis \t 2.3586926819038965\n",
            "\t\t meningitis aseptic \t 2.0873249991348004\n",
            "\t\t ophthalmoplegia \t 2.0435920218009818\n",
            "\t\t ophthalmoplegia \t 2.0435920218009818\n",
            "\t\t ophthalmoplegia \t 2.0435920218009818\n",
            "\t\t gaze palsy \t 1.91233057812051\n",
            "\t\t gaze palsy \t 1.91233057812051\n",
            "\t\t gaze palsy \t 1.91233057812051\n",
            "\t\t encephalitis viral \t 1.8044583591029408\n",
            "\t\t extraocular muscle paresis \t 1.7925915718078613\n",
            "\t\t extraocular muscle paresis \t 1.7925915718078613\n",
            "\t\t extraocular muscle paresis \t 1.7925915718078613\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t panencephalitis \t 1.7522270679473877\n",
            "\t\t orbital apex syndrome \t 1.6739182472229004\n",
            "\t\t meningitis noninfective \t 1.5326719284057617\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t orbital compartment syndrome \t 1.4479095935821533\n",
            "\t\t cavernous sinus syndrome \t 1.2661194801330566\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['colitis', 'crohn^s disease', 'human herpesvirus 6 infection', 'lymphocele', 'nephropathy toxic', 'proteinuria']\n",
            "Predicted reactions: \n",
            "\t ['colitis', 'diarrhoea', 'abdominal pain', 'constipation', 'colitis ulcerative', 'abdominal pain upper', 'abdominal discomfort', 'anaemia', 'decreased appetite', 'necrotising colitis', 'neutropenic colitis', 'gastrointestinal haemorrhage', 'diarrhoea infectious', 'haematochezia', 'colitis ischaemic', 'large intestinal stenosis', 'ulcer', \"crohn's disease\", 'intestinal obstruction', 'skin ulcer', 'diarrhoea haemorrhagic', 'enterocolitis', 'abdominal abscess', 'anorexia nervosa', 'colitis microscopic', 'microcytic anaemia', 'large intestinal ulcer', 'malabsorption', 'lymphocytic infiltration', 'deficiency anaemia', 'intestinal stenosis', 'ileus', 'colonic abscess', 'abscess intestinal', 'obstructive airways disorder', 'ulcer haemorrhage', 'lymphocyte morphology', 'lymphocyte morphology abnormal', 'obstruction']\n",
            "Ungrounded reactions: \n",
            "\t ['colitis', 'diarrhoea', 'abdominal pain', 'constipation', 'colitis ulcerative', 'abdominal pain upper', 'abdominal discomfort', 'anaemia', 'decreased appetite', 'necrotising colitis', 'neutropenic colitis', 'gastrointestinal haemorrhage', 'diarrhoea infectious', 'haematochezia', 'colitis ischaemic', 'large intestinal stenosis', 'ulcer', \"crohn's disease\", 'intestinal obstruction', 'skin ulcer', 'diarrhoea haemorrhagic', 'enterocolitis', 'abdominal abscess', 'anorexia nervosa', 'colitis microscopic', 'microcytic anaemia', 'large intestinal ulcer', 'malabsorption', 'lymphocytic infiltration', 'deficiency anaemia', 'intestinal stenosis', 'ileus', 'colonic abscess', 'abscess intestinal', 'obstructive airways disorder', 'ulcer haemorrhage', 'lymphocyte morphology', 'lymphocyte morphology abnormal', 'obstruction']\n",
            "Recall@10:  0.16666666666666666\n",
            "Recall@20:  0.16666666666666666\n",
            "Recall@30:  0.16666666666666666\n",
            "Resolved reactions: \n",
            "\t\t colitis \t 13.748082829608238\n",
            "\t\t diarrhoea \t 12.09264971021761\n",
            "\t\t abdominal pain \t 10.613744159131917\n",
            "\t\t constipation \t 7.271667607991146\n",
            "\t\t colitis ulcerative \t 6.866757047191675\n",
            "\t\t abdominal pain upper \t 6.69277894817889\n",
            "\t\t abdominal discomfort \t 6.009032015633339\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t decreased appetite \t 4.333810472449917\n",
            "\t\t necrotising colitis \t 4.307463866772462\n",
            "\t\t neutropenic colitis \t 3.727973092870651\n",
            "\t\t gastrointestinal haemorrhage \t 3.5999979122550867\n",
            "\t\t diarrhoea infectious \t 3.337033987045288\n",
            "\t\t haematochezia \t 3.2648843820676046\n",
            "\t\t colitis ischaemic \t 3.2450929249047333\n",
            "\t\t large intestinal stenosis \t 3.14389967918396\n",
            "\t\t ulcer \t 2.8195961875147804\n",
            "\t\t crohn's disease \t 2.771578788757324\n",
            "\t\t intestinal obstruction \t 2.4914456763733397\n",
            "\t\t skin ulcer \t 2.325113400965082\n",
            "\t\t diarrhoea haemorrhagic \t 2.2280814950262258\n",
            "\t\t enterocolitis \t 1.904506481310922\n",
            "\t\t abdominal abscess \t 1.887594922176759\n",
            "\t\t anorexia nervosa \t 1.8405749797821045\n",
            "\t\t colitis microscopic \t 1.8313005355997138\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t large intestinal ulcer \t 1.826811661742159\n",
            "\t\t malabsorption \t 1.8021268267206885\n",
            "\t\t lymphocytic infiltration \t 1.7875225110209714\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t intestinal stenosis \t 1.7640082836151123\n",
            "\t\t ileus \t 1.543657888341521\n",
            "\t\t colonic abscess \t 1.5026090145111084\n",
            "\t\t abscess intestinal \t 1.4440722465515137\n",
            "\t\t obstructive airways disorder \t 1.3550532111051332\n",
            "\t\t ulcer haemorrhage \t 1.3120054006576538\n",
            "\t\t lymphocyte morphology \t 1.283441424369812\n",
            "\t\t lymphocyte morphology abnormal \t 1.1942050457000732\n",
            "\t\t obstruction \t 1.1097859144210815\n",
            "Grounded reactions: \n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t decreased appetite \t 4.333810472449917\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t gastrointestinal haemorrhage \t 3.5999979122550867\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t haematochezia \t 3.2648843820676046\n",
            "\t\t colitis \t 3.1900441181608845\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t colitis ulcerative \t 2.9992854427853133\n",
            "\t\t colitis \t 2.969056518951196\n",
            "\t\t ulcer \t 2.8195961875147804\n",
            "\t\t colitis \t 2.6884798441359994\n",
            "\t\t colitis \t 2.6884798441359994\n",
            "\t\t intestinal obstruction \t 2.4914456763733397\n",
            "\t\t skin ulcer \t 2.325113400965082\n",
            "\t\t diarrhoea haemorrhagic \t 2.2280814950262258\n",
            "\t\t colitis \t 2.2120225042241586\n",
            "\t\t colitis ulcerative \t 1.933735802203181\n",
            "\t\t colitis ulcerative \t 1.933735802203181\n",
            "\t\t enterocolitis \t 1.904506481310922\n",
            "\t\t abdominal abscess \t 1.887594922176759\n",
            "\t\t neutropenic colitis \t 1.8639865464353256\n",
            "\t\t neutropenic colitis \t 1.8639865464353256\n",
            "\t\t large intestinal stenosis \t 1.850825309753418\n",
            "\t\t anorexia nervosa \t 1.8405749797821045\n",
            "\t\t colitis microscopic \t 1.8313005355997138\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t large intestinal ulcer \t 1.826811661742159\n",
            "\t\t malabsorption \t 1.8021268267206885\n",
            "\t\t lymphocytic infiltration \t 1.7875225110209714\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t intestinal stenosis \t 1.7640082836151123\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t colitis ischaemic \t 1.6225464624523667\n",
            "\t\t colitis ischaemic \t 1.6225464624523667\n",
            "\t\t ileus \t 1.543657888341521\n",
            "\t\t necrotising colitis \t 1.5410546061977477\n",
            "\t\t necrotising colitis \t 1.5410546061977477\n",
            "\t\t colonic abscess \t 1.5026090145111084\n",
            "\t\t abscess intestinal \t 1.4440722465515137\n",
            "\t\t crohn's disease \t 1.385789394378662\n",
            "\t\t crohn's disease \t 1.385789394378662\n",
            "\t\t obstructive airways disorder \t 1.3550532111051332\n",
            "\t\t ulcer haemorrhage \t 1.3120054006576538\n",
            "\t\t large intestinal stenosis \t 1.293074369430542\n",
            "\t\t lymphocyte morphology \t 1.283441424369812\n",
            "\t\t necrotising colitis \t 1.2253546543769671\n",
            "\t\t lymphocyte morphology abnormal \t 1.1942050457000732\n",
            "\t\t obstruction \t 1.1097859144210815\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['disease progression', 'drug resistance']\n",
            "Predicted reactions: \n",
            "\t ['no adverse event', 'metastases to liver', 'adverse drug reaction', 'rash', 'dyspnoea', 'respiratory failure', 'hepatocellular carcinoma', 'hepatotoxicity', 'drug-induced liver injury', 'oedema peripheral', 'interstitial lung disease', 'abdominal distension', 'hepatic cancer metastatic', 'rash erythematous', 'liver injury', 'acute respiratory failure', 'no reaction on previous exposure to drug', 'dyspnoea exertional', 'sepsis', 'dysphagia', 'rash maculo-papular', 'oedema', 'acute respiratory distress syndrome', 'peripheral swelling', 'lung disorder', 'abdominal discomfort', 'acute hepatic failure', 'respiratory arrest', 'drug intolerance', 'pulmonary fibrosis', 'metastases to lymph nodes', 'lymphangiosis carcinomatosa', 'orthopnoea', 'lung infiltration', 'hypophagia', 'drug hypersensitivity', 'flatulence', 'bacterial sepsis', 'eating disorder', 'leukaemic infiltration pulmonary', 'yersinia sepsis']\n",
            "Ungrounded reactions: \n",
            "\t ['no adverse event', 'metastases to liver', 'adverse drug reaction', 'rash', 'dyspnoea', 'respiratory failure', 'hepatocellular carcinoma', 'hepatotoxicity', 'drug-induced liver injury', 'oedema peripheral', 'interstitial lung disease', 'abdominal distension', 'hepatic cancer metastatic', 'rash erythematous', 'liver injury', 'acute respiratory failure', 'no reaction on previous exposure to drug', 'dyspnoea exertional', 'sepsis', 'dysphagia', 'rash maculo-papular', 'oedema', 'acute respiratory distress syndrome', 'peripheral swelling', 'lung disorder', 'abdominal discomfort', 'acute hepatic failure', 'respiratory arrest', 'drug intolerance', 'pulmonary fibrosis', 'metastases to lymph nodes', 'lymphangiosis carcinomatosa', 'orthopnoea', 'lung infiltration', 'hypophagia', 'drug hypersensitivity', 'flatulence', 'bacterial sepsis', 'eating disorder', 'leukaemic infiltration pulmonary', 'yersinia sepsis']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t no adverse event \t 7.029132318072656\n",
            "\t\t metastases to liver \t 6.564517658532073\n",
            "\t\t adverse drug reaction \t 5.8107564675048975\n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t dyspnoea \t 4.881158693892149\n",
            "\t\t respiratory failure \t 4.807865089249589\n",
            "\t\t hepatocellular carcinoma \t 4.2053394122270795\n",
            "\t\t hepatotoxicity \t 4.195707930132005\n",
            "\t\t drug-induced liver injury \t 4.1298486388681415\n",
            "\t\t oedema peripheral \t 4.089979427243733\n",
            "\t\t interstitial lung disease \t 4.074278365794382\n",
            "\t\t abdominal distension \t 4.0210938151668305\n",
            "\t\t hepatic cancer metastatic \t 3.3137824535369873\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t liver injury \t 3.0880320986726657\n",
            "\t\t acute respiratory failure \t 2.9996367400764714\n",
            "\t\t no reaction on previous exposure to drug \t 2.972557544708252\n",
            "\t\t dyspnoea exertional \t 2.9540996054856214\n",
            "\t\t sepsis \t 2.9288881899836947\n",
            "\t\t dysphagia \t 2.869299223349528\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t oedema \t 2.7821311885496858\n",
            "\t\t acute respiratory distress syndrome \t 2.6114878194182087\n",
            "\t\t peripheral swelling \t 2.603086127082601\n",
            "\t\t lung disorder \t 2.5561430172427753\n",
            "\t\t abdominal discomfort \t 2.3473828383535347\n",
            "\t\t acute hepatic failure \t 2.3029274501717696\n",
            "\t\t respiratory arrest \t 2.204772307572368\n",
            "\t\t drug intolerance \t 2.094932655699067\n",
            "\t\t pulmonary fibrosis \t 1.9964777773528997\n",
            "\t\t metastases to lymph nodes \t 1.9112728287558631\n",
            "\t\t lymphangiosis carcinomatosa \t 1.87962284293694\n",
            "\t\t orthopnoea \t 1.8224309663889289\n",
            "\t\t lung infiltration \t 1.803896852457912\n",
            "\t\t hypophagia \t 1.7415160574114559\n",
            "\t\t drug hypersensitivity \t 1.7058099849149198\n",
            "\t\t flatulence \t 1.452634316166269\n",
            "\t\t bacterial sepsis \t 1.4263611142237302\n",
            "\t\t eating disorder \t 1.40944145538695\n",
            "\t\t leukaemic infiltration pulmonary \t 1.195487141609192\n",
            "\t\t yersinia sepsis \t 1.053065299987793\n",
            "Grounded reactions: \n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t dyspnoea \t 4.881158693892149\n",
            "\t\t respiratory failure \t 4.807865089249589\n",
            "\t\t hepatotoxicity \t 4.195707930132005\n",
            "\t\t drug-induced liver injury \t 4.1298486388681415\n",
            "\t\t oedema peripheral \t 4.089979427243733\n",
            "\t\t interstitial lung disease \t 4.074278365794382\n",
            "\t\t abdominal distension \t 4.0210938151668305\n",
            "\t\t metastases to liver \t 3.8072496322634874\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t liver injury \t 3.0880320986726657\n",
            "\t\t acute respiratory failure \t 2.9996367400764714\n",
            "\t\t dyspnoea exertional \t 2.9540996054856214\n",
            "\t\t sepsis \t 2.9288881899836947\n",
            "\t\t dysphagia \t 2.869299223349528\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t oedema \t 2.7821311885496858\n",
            "\t\t metastases to liver \t 2.7572680262685854\n",
            "\t\t acute respiratory distress syndrome \t 2.6114878194182087\n",
            "\t\t peripheral swelling \t 2.603086127082601\n",
            "\t\t lung disorder \t 2.5561430172427753\n",
            "\t\t abdominal discomfort \t 2.3473828383535347\n",
            "\t\t hepatocellular carcinoma \t 2.3150834567229484\n",
            "\t\t acute hepatic failure \t 2.3029274501717696\n",
            "\t\t adverse drug reaction \t 2.2695005011312164\n",
            "\t\t respiratory arrest \t 2.204772307572368\n",
            "\t\t drug intolerance \t 2.094932655699067\n",
            "\t\t pulmonary fibrosis \t 1.9964777773528997\n",
            "\t\t metastases to lymph nodes \t 1.9112728287558631\n",
            "\t\t hepatic cancer metastatic \t 1.9036779403686523\n",
            "\t\t hepatocellular carcinoma \t 1.8902559555041316\n",
            "\t\t lymphangiosis carcinomatosa \t 1.87962284293694\n",
            "\t\t orthopnoea \t 1.8224309663889289\n",
            "\t\t lung infiltration \t 1.803896852457912\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t hypophagia \t 1.7415160574114559\n",
            "\t\t drug hypersensitivity \t 1.7058099849149198\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t flatulence \t 1.452634316166269\n",
            "\t\t bacterial sepsis \t 1.4263611142237302\n",
            "\t\t hepatic cancer metastatic \t 1.410104513168335\n",
            "\t\t eating disorder \t 1.40944145538695\n",
            "\t\t leukaemic infiltration pulmonary \t 1.195487141609192\n",
            "\t\t yersinia sepsis \t 1.053065299987793\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['blood creatine phosphokinase increased', 'condition aggravated', 'muscular weakness', 'pain', 'rhabdomyolysis', 'vasculitis']\n",
            "Predicted reactions: \n",
            "\t ['no adverse event', 'drug ineffective', 'adverse drug reaction', 'blood creatinine increased', 'drug interaction', 'blood creatine phosphokinase increased', 'chromaturia', 'creatinine urine increased', 'creatinine renal clearance increased', 'blood creatine phosphokinase bb increased', 'blood creatine phosphokinase mm increased', 'no reaction on previous exposure to drug', 'urine abnormality', 'isosthenuria', 'nothing by mouth order', 'ideas of reference']\n",
            "Ungrounded reactions: \n",
            "\t ['no adverse event', 'drug ineffective', 'adverse drug reaction', 'blood creatinine increased', 'drug interaction', 'blood creatine phosphokinase increased', 'chromaturia', 'creatinine urine increased', 'creatinine renal clearance increased', 'blood creatine phosphokinase bb increased', 'blood creatine phosphokinase mm increased', 'no reaction on previous exposure to drug', 'urine abnormality', 'isosthenuria', 'nothing by mouth order', 'ideas of reference']\n",
            "Recall@10:  0.16666666666666666\n",
            "Recall@20:  0.16666666666666666\n",
            "Recall@30:  0.16666666666666666\n",
            "Resolved reactions: \n",
            "\t\t no adverse event \t 8.398469768691935\n",
            "\t\t drug ineffective \t 7.668228517817285\n",
            "\t\t adverse drug reaction \t 6.538073863608667\n",
            "\t\t blood creatinine increased \t 4.5601085049382055\n",
            "\t\t drug interaction \t 3.5600697628287725\n",
            "\t\t blood creatine phosphokinase increased \t 3.4277960149395943\n",
            "\t\t chromaturia \t 2.2648713099984663\n",
            "\t\t creatinine urine increased \t 1.9017962217330933\n",
            "\t\t creatinine renal clearance increased \t 1.776202917098999\n",
            "\t\t blood creatine phosphokinase bb increased \t 1.6544320583343506\n",
            "\t\t blood creatine phosphokinase mm increased \t 1.647878885269165\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t urine abnormality \t 1.176031470298767\n",
            "\t\t isosthenuria \t 1.1442487239837646\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "Grounded reactions: \n",
            "\t\t blood creatinine increased \t 4.5601085049382055\n",
            "\t\t drug ineffective \t 3.9798800178096756\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t drug interaction \t 3.5600697628287725\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t blood creatine phosphokinase increased \t 3.4277960149395943\n",
            "\t\t no adverse event \t 3.3319030318452447\n",
            "\t\t adverse drug reaction \t 2.723724239272734\n",
            "\t\t chromaturia \t 2.2648713099984663\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t creatinine urine increased \t 1.9017962217330933\n",
            "\t\t creatinine renal clearance increased \t 1.776202917098999\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t blood creatine phosphokinase bb increased \t 1.6544320583343506\n",
            "\t\t blood creatine phosphokinase mm increased \t 1.647878885269165\n",
            "\t\t no adverse event \t 1.5520005778103618\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t urine abnormality \t 1.176031470298767\n",
            "\t\t isosthenuria \t 1.1442487239837646\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['vascular pseudoaneurysm']\n",
            "Predicted reactions: \n",
            "\t ['vascular pseudoaneurysm', 'anaphylactic reaction', 'haemorrhage', 'haematoma', 'vascular pseudoaneurysm ruptured', 'urticaria', 'necrosis', 'heparin neutralisation therapy', 'anticoagulant therapy', 'embolism', 'anaphylactic shock', 'thrombosis', 'epistaxis', 'skin necrosis', 'embolism venous', 'disseminated intravascular coagulation', 'heparin-induced thrombocytopenia', 'anaphylactoid reaction', 'no adverse event', 'gangrene', 'urticarial dermatitis', 'spinal cord compression', 'idiopathic urticaria', 'aneurysm ruptured', 'procoagulant therapy', 'extravasation blood', 'traumatic haematoma', 'cardiac murmur', 'subcutaneous haematoma', 'cardiac murmur functional', 'aortic pseudoaneurysm', 'heart sounds', 'traumatic spinal cord compression', 'nerve compression', 'hypercoagulation', 'clot retraction', 'intervertebral disc space narrowing', 'shunt stenosis', 'joint space narrowing', 'nyha classification', 'blood group o']\n",
            "Ungrounded reactions: \n",
            "\t ['vascular pseudoaneurysm', 'anaphylactic reaction', 'haemorrhage', 'haematoma', 'vascular pseudoaneurysm ruptured', 'urticaria', 'necrosis', 'heparin neutralisation therapy', 'anticoagulant therapy', 'embolism', 'anaphylactic shock', 'thrombosis', 'epistaxis', 'skin necrosis', 'embolism venous', 'disseminated intravascular coagulation', 'heparin-induced thrombocytopenia', 'anaphylactoid reaction', 'no adverse event', 'gangrene', 'urticarial dermatitis', 'spinal cord compression', 'idiopathic urticaria', 'aneurysm ruptured', 'procoagulant therapy', 'extravasation blood', 'traumatic haematoma', 'cardiac murmur', 'subcutaneous haematoma', 'cardiac murmur functional', 'aortic pseudoaneurysm', 'heart sounds', 'traumatic spinal cord compression', 'nerve compression', 'hypercoagulation', 'clot retraction', 'intervertebral disc space narrowing', 'shunt stenosis', 'joint space narrowing', 'nyha classification', 'blood group o']\n",
            "Recall@10:  1.0\n",
            "Recall@20:  1.0\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t vascular pseudoaneurysm \t 5.465170664076959\n",
            "\t\t anaphylactic reaction \t 3.9140394426688054\n",
            "\t\t haemorrhage \t 3.777245383321886\n",
            "\t\t haematoma \t 3.6517250675119537\n",
            "\t\t vascular pseudoaneurysm ruptured \t 3.511780619621277\n",
            "\t\t urticaria \t 3.511342095149185\n",
            "\t\t necrosis \t 3.3917624062153626\n",
            "\t\t heparin neutralisation therapy \t 3.2009897232055664\n",
            "\t\t anticoagulant therapy \t 3.1306207180023193\n",
            "\t\t embolism \t 3.058974680153086\n",
            "\t\t anaphylactic shock \t 2.984264099110596\n",
            "\t\t thrombosis \t 2.888833047007379\n",
            "\t\t epistaxis \t 2.695905684383067\n",
            "\t\t skin necrosis \t 2.4892014136157647\n",
            "\t\t embolism venous \t 2.1999899284703903\n",
            "\t\t disseminated intravascular coagulation \t 2.192407746810269\n",
            "\t\t heparin-induced thrombocytopenia \t 2.052450362312442\n",
            "\t\t anaphylactoid reaction \t 2.000655512077942\n",
            "\t\t no adverse event \t 1.9108820812637108\n",
            "\t\t gangrene \t 1.892486889807253\n",
            "\t\t urticarial dermatitis \t 1.749977946281433\n",
            "\t\t spinal cord compression \t 1.694601684495076\n",
            "\t\t idiopathic urticaria \t 1.6895856857299805\n",
            "\t\t aneurysm ruptured \t 1.6496007442474365\n",
            "\t\t procoagulant therapy \t 1.6424626111984253\n",
            "\t\t extravasation blood \t 1.639888882637024\n",
            "\t\t traumatic haematoma \t 1.631518840789795\n",
            "\t\t cardiac murmur \t 1.5918537976108271\n",
            "\t\t subcutaneous haematoma \t 1.5433260941907045\n",
            "\t\t cardiac murmur functional \t 1.491403341293335\n",
            "\t\t aortic pseudoaneurysm \t 1.471808671951294\n",
            "\t\t heart sounds \t 1.3848036527633667\n",
            "\t\t traumatic spinal cord compression \t 1.2598544359207153\n",
            "\t\t nerve compression \t 1.2408795356750488\n",
            "\t\t hypercoagulation \t 1.2140457937799103\n",
            "\t\t clot retraction \t 1.1910510063171387\n",
            "\t\t intervertebral disc space narrowing \t 1.0984150171279907\n",
            "\t\t shunt stenosis \t 1.047274112701416\n",
            "\t\t joint space narrowing \t 0.9984000325202942\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "Grounded reactions: \n",
            "\t\t anaphylactic reaction \t 3.9140394426688054\n",
            "\t\t haemorrhage \t 3.777245383321886\n",
            "\t\t haematoma \t 3.6517250675119537\n",
            "\t\t urticaria \t 3.511342095149185\n",
            "\t\t necrosis \t 3.3917624062153626\n",
            "\t\t embolism \t 3.058974680153086\n",
            "\t\t vascular pseudoaneurysm \t 3.0339493896541745\n",
            "\t\t anaphylactic shock \t 2.984264099110596\n",
            "\t\t thrombosis \t 2.888833047007379\n",
            "\t\t epistaxis \t 2.695905684383067\n",
            "\t\t skin necrosis \t 2.4892014136157647\n",
            "\t\t vascular pseudoaneurysm \t 2.431221274422785\n",
            "\t\t embolism venous \t 2.1999899284703903\n",
            "\t\t disseminated intravascular coagulation \t 2.192407746810269\n",
            "\t\t heparin-induced thrombocytopenia \t 2.052450362312442\n",
            "\t\t anaphylactoid reaction \t 2.000655512077942\n",
            "\t\t vascular pseudoaneurysm ruptured \t 1.9195047616958618\n",
            "\t\t no adverse event \t 1.9108820812637108\n",
            "\t\t gangrene \t 1.892486889807253\n",
            "\t\t urticarial dermatitis \t 1.749977946281433\n",
            "\t\t anticoagulant therapy \t 1.7440216541290283\n",
            "\t\t spinal cord compression \t 1.694601684495076\n",
            "\t\t idiopathic urticaria \t 1.6895856857299805\n",
            "\t\t heparin neutralisation therapy \t 1.6537048816680908\n",
            "\t\t aneurysm ruptured \t 1.6496007442474365\n",
            "\t\t procoagulant therapy \t 1.6424626111984253\n",
            "\t\t extravasation blood \t 1.639888882637024\n",
            "\t\t traumatic haematoma \t 1.631518840789795\n",
            "\t\t vascular pseudoaneurysm ruptured \t 1.592275857925415\n",
            "\t\t cardiac murmur \t 1.5918537976108271\n",
            "\t\t heparin neutralisation therapy \t 1.5472848415374756\n",
            "\t\t subcutaneous haematoma \t 1.5433260941907045\n",
            "\t\t cardiac murmur functional \t 1.491403341293335\n",
            "\t\t aortic pseudoaneurysm \t 1.471808671951294\n",
            "\t\t anticoagulant therapy \t 1.386599063873291\n",
            "\t\t heart sounds \t 1.3848036527633667\n",
            "\t\t traumatic spinal cord compression \t 1.2598544359207153\n",
            "\t\t nerve compression \t 1.2408795356750488\n",
            "\t\t hypercoagulation \t 1.2140457937799103\n",
            "\t\t clot retraction \t 1.1910510063171387\n",
            "\t\t intervertebral disc space narrowing \t 1.0984150171279907\n",
            "\t\t shunt stenosis \t 1.047274112701416\n",
            "\t\t joint space narrowing \t 0.9984000325202942\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['bacteraemia', 'drug ineffective', 'enterococcal infection']\n",
            "Predicted reactions: \n",
            "\t ['respiratory failure', 'acute respiratory failure', 'death', 'acute kidney injury', 'drug resistance', 'bronchopulmonary aspergillosis', 'aspergillus infection', 'drug interaction', 'no adverse event', 'respiratory arrest', 'tachypnoea', 'drug intolerance', 'immune reconstitution inflammatory syndrome', 'multiple-drug resistance', 'infection reactivation', 'adverse drug reaction', 'sudden death', 'enterococcal infection', 'respiratory symptom', 'breath sounds abnormal', 'clinical death', 'acinetobacter infection', 'cerebral aspergillosis', 'pneumonia acinetobacter', 'mechanical ventilation', 'lung assist device therapy', 'atypical mycobacterial pneumonia', 'adaptive servo-ventilation', 'steroid therapy', 'steroid activity', 'systemic inflammatory response syndrome', 'escherichia bacteraemia', 'steroid dependence', 'enterococcal bacteraemia', 'fungal infection', 'amphetamines', 'antifungal treatment', 'fungal sepsis', 'thiopurine methyltransferase', 'coproporphyrinogen', 'matrix metalloproteinase-3', 'matrix-assisted laser desorption/ionisation time-of-flight mass spectrometry', 'antibiotic level therapeutic', 'antibiotic level', 'barbiturates']\n",
            "Ungrounded reactions: \n",
            "\t ['respiratory failure', 'acute respiratory failure', 'death', 'acute kidney injury', 'drug resistance', 'bronchopulmonary aspergillosis', 'aspergillus infection', 'drug interaction', 'no adverse event', 'respiratory arrest', 'tachypnoea', 'drug intolerance', 'immune reconstitution inflammatory syndrome', 'multiple-drug resistance', 'infection reactivation', 'adverse drug reaction', 'sudden death', 'enterococcal infection', 'respiratory symptom', 'breath sounds abnormal', 'clinical death', 'acinetobacter infection', 'cerebral aspergillosis', 'pneumonia acinetobacter', 'mechanical ventilation', 'lung assist device therapy', 'atypical mycobacterial pneumonia', 'adaptive servo-ventilation', 'steroid therapy', 'steroid activity', 'systemic inflammatory response syndrome', 'escherichia bacteraemia', 'steroid dependence', 'enterococcal bacteraemia', 'fungal infection', 'amphetamines', 'antifungal treatment', 'fungal sepsis', 'thiopurine methyltransferase', 'coproporphyrinogen', 'matrix metalloproteinase-3', 'matrix-assisted laser desorption/ionisation time-of-flight mass spectrometry', 'antibiotic level therapeutic', 'antibiotic level', 'barbiturates']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.3333333333333333\n",
            "Recall@30:  0.3333333333333333\n",
            "Resolved reactions: \n",
            "\t\t respiratory failure \t 9.991565538188041\n",
            "\t\t acute respiratory failure \t 6.932394180861193\n",
            "\t\t death \t 5.228785869945281\n",
            "\t\t acute kidney injury \t 4.449860041177979\n",
            "\t\t drug resistance \t 3.485903570098309\n",
            "\t\t bronchopulmonary aspergillosis \t 3.382731602414802\n",
            "\t\t aspergillus infection \t 3.3101534577058365\n",
            "\t\t drug interaction \t 2.981877989187109\n",
            "\t\t no adverse event \t 2.823716409967973\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t tachypnoea \t 2.6872349863421627\n",
            "\t\t drug intolerance \t 2.637850107159869\n",
            "\t\t immune reconstitution inflammatory syndrome \t 2.6010123259019866\n",
            "\t\t multiple-drug resistance \t 2.170966501795213\n",
            "\t\t infection reactivation \t 2.1385124396618087\n",
            "\t\t adverse drug reaction \t 2.097649271380158\n",
            "\t\t sudden death \t 2.062174121736474\n",
            "\t\t enterococcal infection \t 2.05858493687948\n",
            "\t\t respiratory symptom \t 1.920194149017334\n",
            "\t\t breath sounds abnormal \t 1.8726977338056963\n",
            "\t\t clinical death \t 1.842085599899292\n",
            "\t\t acinetobacter infection \t 1.7727336152549282\n",
            "\t\t cerebral aspergillosis \t 1.7430955032004327\n",
            "\t\t pneumonia acinetobacter \t 1.71113920211792\n",
            "\t\t mechanical ventilation \t 1.678187370300293\n",
            "\t\t lung assist device therapy \t 1.5705232620239258\n",
            "\t\t atypical mycobacterial pneumonia \t 1.562268614768982\n",
            "\t\t adaptive servo-ventilation \t 1.5518798828125\n",
            "\t\t steroid therapy \t 1.5434775352478027\n",
            "\t\t steroid activity \t 1.439610481262207\n",
            "\t\t systemic inflammatory response syndrome \t 1.4377558579253644\n",
            "\t\t escherichia bacteraemia \t 1.4314505626147749\n",
            "\t\t steroid dependence \t 1.3697452545166016\n",
            "\t\t enterococcal bacteraemia \t 1.3357478380203247\n",
            "\t\t fungal infection \t 1.3138322961431819\n",
            "\t\t amphetamines \t 1.3134913444519043\n",
            "\t\t antifungal treatment \t 0.8479658961296082\n",
            "\t\t fungal sepsis \t 0.7851443916439836\n",
            "\t\t thiopurine methyltransferase \t 0.7033562660217285\n",
            "\t\t coproporphyrinogen \t 0.6455943584442139\n",
            "\t\t matrix metalloproteinase-3 \t 0.6042932271957397\n",
            "\t\t matrix-assisted laser desorption/ionisation time-of-flight mass spectrometry \t 0.592769980430603\n",
            "\t\t antibiotic level therapeutic \t 0.5723983645439148\n",
            "\t\t antibiotic level \t 0.5657370686531067\n",
            "\t\t barbiturates \t 0.5440603494644165\n",
            "Grounded reactions: \n",
            "\t\t respiratory failure \t 5.630202200309923\n",
            "\t\t death \t 5.228785869945281\n",
            "\t\t acute kidney injury \t 4.449860041177979\n",
            "\t\t respiratory failure \t 4.361363337878118\n",
            "\t\t acute respiratory failure \t 3.9063727090956677\n",
            "\t\t drug resistance \t 3.485903570098309\n",
            "\t\t bronchopulmonary aspergillosis \t 3.382731602414802\n",
            "\t\t aspergillus infection \t 3.3101534577058365\n",
            "\t\t acute respiratory failure \t 3.026021471765525\n",
            "\t\t drug interaction \t 2.981877989187109\n",
            "\t\t no adverse event \t 2.823716409967973\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t tachypnoea \t 2.6872349863421627\n",
            "\t\t drug intolerance \t 2.637850107159869\n",
            "\t\t immune reconstitution inflammatory syndrome \t 2.6010123259019866\n",
            "\t\t multiple-drug resistance \t 2.170966501795213\n",
            "\t\t infection reactivation \t 2.1385124396618087\n",
            "\t\t adverse drug reaction \t 2.097649271380158\n",
            "\t\t sudden death \t 2.062174121736474\n",
            "\t\t enterococcal infection \t 2.05858493687948\n",
            "\t\t respiratory symptom \t 1.920194149017334\n",
            "\t\t breath sounds abnormal \t 1.8726977338056963\n",
            "\t\t clinical death \t 1.842085599899292\n",
            "\t\t acinetobacter infection \t 1.7727336152549282\n",
            "\t\t cerebral aspergillosis \t 1.7430955032004327\n",
            "\t\t pneumonia acinetobacter \t 1.71113920211792\n",
            "\t\t mechanical ventilation \t 1.678187370300293\n",
            "\t\t lung assist device therapy \t 1.5705232620239258\n",
            "\t\t atypical mycobacterial pneumonia \t 1.562268614768982\n",
            "\t\t adaptive servo-ventilation \t 1.5518798828125\n",
            "\t\t steroid therapy \t 1.5434775352478027\n",
            "\t\t steroid activity \t 1.439610481262207\n",
            "\t\t systemic inflammatory response syndrome \t 1.4377558579253644\n",
            "\t\t escherichia bacteraemia \t 1.4314505626147749\n",
            "\t\t steroid dependence \t 1.3697452545166016\n",
            "\t\t enterococcal bacteraemia \t 1.3357478380203247\n",
            "\t\t fungal infection \t 1.3138322961431819\n",
            "\t\t antifungal treatment \t 0.8479658961296082\n",
            "\t\t fungal sepsis \t 0.7851443916439836\n",
            "\t\t amphetamines \t 0.7385751008987427\n",
            "\t\t thiopurine methyltransferase \t 0.7033562660217285\n",
            "\t\t coproporphyrinogen \t 0.6455943584442139\n",
            "\t\t matrix metalloproteinase-3 \t 0.6042932271957397\n",
            "\t\t matrix-assisted laser desorption/ionisation time-of-flight mass spectrometry \t 0.592769980430603\n",
            "\t\t amphetamines \t 0.5749162435531616\n",
            "\t\t antibiotic level therapeutic \t 0.5723983645439148\n",
            "\t\t antibiotic level \t 0.5657370686531067\n",
            "\t\t barbiturates \t 0.5440603494644165\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['interstitial lung disease', 'lymphangiosis carcinomatosa', 'superior vena cava syndrome']\n",
            "Predicted reactions: \n",
            "\t ['dyspnoea', 'erythema', 'rash erythematous', 'rash', 'dyspnoea exertional', 'face oedema', 'swelling face', 'orthopnoea', 'periorbital oedema', 'generalised oedema', 'cough', 'induration', 'oedema', 'skin induration', 'productive cough', 'interstitial lung disease', 'swelling', 'lymphoedema', 'lymphatic obstruction', 'pulmonary toxicity', 'habit cough', 'erythema infectiosum', 'lymphatic insufficiency', 'superior vena cava dilatation', 'application site induration', 'non-pitting oedema', 'lung induration', 'erythema migrans', 'coronary sinus dilatation', 'erythema annulare', 'inferior vena cava dilatation', 'idiopathic interstitial pneumonia']\n",
            "Ungrounded reactions: \n",
            "\t ['dyspnoea', 'erythema', 'rash erythematous', 'rash', 'dyspnoea exertional', 'face oedema', 'swelling face', 'orthopnoea', 'periorbital oedema', 'generalised oedema', 'cough', 'induration', 'oedema', 'skin induration', 'productive cough', 'interstitial lung disease', 'swelling', 'lymphoedema', 'lymphatic obstruction', 'pulmonary toxicity', 'habit cough', 'erythema infectiosum', 'lymphatic insufficiency', 'superior vena cava dilatation', 'application site induration', 'non-pitting oedema', 'lung induration', 'erythema migrans', 'coronary sinus dilatation', 'erythema annulare', 'inferior vena cava dilatation', 'idiopathic interstitial pneumonia']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.3333333333333333\n",
            "Recall@30:  0.3333333333333333\n",
            "Resolved reactions: \n",
            "\t\t dyspnoea \t 17.058059769618687\n",
            "\t\t erythema \t 13.490022873216322\n",
            "\t\t rash erythematous \t 12.780773607269552\n",
            "\t\t rash \t 11.058680104261496\n",
            "\t\t dyspnoea exertional \t 9.4637907539061\n",
            "\t\t face oedema \t 7.522437638372639\n",
            "\t\t swelling face \t 7.4375245584176\n",
            "\t\t orthopnoea \t 6.620345169255847\n",
            "\t\t periorbital oedema \t 6.246988513243066\n",
            "\t\t generalised oedema \t 4.753449746407294\n",
            "\t\t cough \t 4.706989534697944\n",
            "\t\t induration \t 3.5938727855682373\n",
            "\t\t oedema \t 3.443481670474922\n",
            "\t\t skin induration \t 3.3518940210342407\n",
            "\t\t productive cough \t 3.1677841580402895\n",
            "\t\t interstitial lung disease \t 2.5771014217316806\n",
            "\t\t swelling \t 2.419026311645221\n",
            "\t\t lymphoedema \t 1.9999998807907104\n",
            "\t\t lymphatic obstruction \t 1.8137116432189941\n",
            "\t\t pulmonary toxicity \t 1.7170810166863055\n",
            "\t\t habit cough \t 1.68760347366333\n",
            "\t\t erythema infectiosum \t 1.6582627296447754\n",
            "\t\t lymphatic insufficiency \t 1.5512126684188843\n",
            "\t\t superior vena cava dilatation \t 1.48533296585083\n",
            "\t\t application site induration \t 1.4613252878189087\n",
            "\t\t non-pitting oedema \t 1.457883596420288\n",
            "\t\t lung induration \t 1.4379675388336182\n",
            "\t\t erythema migrans \t 1.3184146881103516\n",
            "\t\t coronary sinus dilatation \t 1.316967248916626\n",
            "\t\t erythema annulare \t 1.307340145111084\n",
            "\t\t inferior vena cava dilatation \t 1.2540686130523682\n",
            "\t\t idiopathic interstitial pneumonia \t 1.1886913776397705\n",
            "Grounded reactions: \n",
            "\t\t dyspnoea \t 5.686019923206229\n",
            "\t\t dyspnoea \t 5.686019923206229\n",
            "\t\t dyspnoea \t 5.686019923206229\n",
            "\t\t cough \t 4.706989534697944\n",
            "\t\t erythema \t 4.496674291072107\n",
            "\t\t erythema \t 4.496674291072107\n",
            "\t\t erythema \t 4.496674291072107\n",
            "\t\t rash \t 3.6862267014204986\n",
            "\t\t rash \t 3.6862267014204986\n",
            "\t\t rash \t 3.6862267014204986\n",
            "\t\t rash erythematous \t 3.4987119947861336\n",
            "\t\t rash erythematous \t 3.4987119947861336\n",
            "\t\t rash erythematous \t 3.4987119947861336\n",
            "\t\t oedema \t 3.443481670474922\n",
            "\t\t productive cough \t 3.1677841580402895\n",
            "\t\t dyspnoea exertional \t 3.1545969179687\n",
            "\t\t dyspnoea exertional \t 3.1545969179687\n",
            "\t\t dyspnoea exertional \t 3.1545969179687\n",
            "\t\t swelling face \t 3.0533447384246375\n",
            "\t\t face oedema \t 2.8339406476022075\n",
            "\t\t generalised oedema \t 2.6903782834171186\n",
            "\t\t interstitial lung disease \t 2.5771014217316806\n",
            "\t\t swelling \t 2.419026311645221\n",
            "\t\t face oedema \t 2.344248495385216\n",
            "\t\t face oedema \t 2.344248495385216\n",
            "\t\t rash erythematous \t 2.2846376229111525\n",
            "\t\t periorbital oedema \t 2.230554403032441\n",
            "\t\t orthopnoea \t 2.206781723085282\n",
            "\t\t orthopnoea \t 2.206781723085282\n",
            "\t\t orthopnoea \t 2.206781723085282\n",
            "\t\t swelling face \t 2.1920899099964815\n",
            "\t\t swelling face \t 2.1920899099964815\n",
            "\t\t generalised oedema \t 2.0630714629901754\n",
            "\t\t periorbital oedema \t 2.008217055105312\n",
            "\t\t periorbital oedema \t 2.008217055105312\n",
            "\t\t induration \t 2.000000238418579\n",
            "\t\t lymphoedema \t 1.9999998807907104\n",
            "\t\t skin induration \t 1.8277112245559692\n",
            "\t\t lymphatic obstruction \t 1.8137116432189941\n",
            "\t\t pulmonary toxicity \t 1.7170810166863055\n",
            "\t\t habit cough \t 1.68760347366333\n",
            "\t\t erythema infectiosum \t 1.6582627296447754\n",
            "\t\t induration \t 1.5938725471496582\n",
            "\t\t lymphatic insufficiency \t 1.5512126684188843\n",
            "\t\t skin induration \t 1.5241827964782715\n",
            "\t\t superior vena cava dilatation \t 1.48533296585083\n",
            "\t\t application site induration \t 1.4613252878189087\n",
            "\t\t non-pitting oedema \t 1.457883596420288\n",
            "\t\t lung induration \t 1.4379675388336182\n",
            "\t\t erythema migrans \t 1.3184146881103516\n",
            "\t\t coronary sinus dilatation \t 1.316967248916626\n",
            "\t\t erythema annulare \t 1.307340145111084\n",
            "\t\t inferior vena cava dilatation \t 1.2540686130523682\n",
            "\t\t idiopathic interstitial pneumonia \t 1.1886913776397705\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['dyspnoea', 'megacolon', 'tachycardia']\n",
            "Predicted reactions: \n",
            "\t ['lactic acidosis', 'nausea', 'vomiting', 'leukocytosis', 'metabolic acidosis', 'white blood cell count increased', 'diarrhoea', 'anaemia', 'pyrexia', 'abdominal pain', 'acidosis', 'colitis ulcerative', 'lymphocytosis', 'constipation', 'abdominal pain upper', 'vomiting projectile', 'large intestinal ulcer', 'abdominal discomfort', 'colitis', 'blood lactic acid increased', 'hyperthermia', 'body temperature increased', 'adverse drug reaction', 'hyperlactacidaemia', 'megacolon', 'paradoxical drug reaction', 'microcytic anaemia', 'deficiency anaemia', 'toxic dilatation of intestine', 'diarrhoea infectious', 'signet-ring cell carcinoma', 'congenital megacolon', 'mixed adenoneuroendocrine carcinoma', 'adverse reaction', 'small cell carcinoma', 'colitis erosive', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['lactic acidosis', 'nausea', 'vomiting', 'leukocytosis', 'metabolic acidosis', 'white blood cell count increased', 'diarrhoea', 'anaemia', 'pyrexia', 'abdominal pain', 'acidosis', 'colitis ulcerative', 'lymphocytosis', 'constipation', 'abdominal pain upper', 'vomiting projectile', 'large intestinal ulcer', 'abdominal discomfort', 'colitis', 'blood lactic acid increased', 'hyperthermia', 'body temperature increased', 'adverse drug reaction', 'hyperlactacidaemia', 'megacolon', 'paradoxical drug reaction', 'microcytic anaemia', 'deficiency anaemia', 'toxic dilatation of intestine', 'diarrhoea infectious', 'signet-ring cell carcinoma', 'congenital megacolon', 'mixed adenoneuroendocrine carcinoma', 'adverse reaction', 'small cell carcinoma', 'colitis erosive', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.3333333333333333\n",
            "Resolved reactions: \n",
            "\t\t lactic acidosis \t 11.973229168334411\n",
            "\t\t nausea \t 11.724094031085794\n",
            "\t\t vomiting \t 11.278298153600627\n",
            "\t\t leukocytosis \t 8.878663899415118\n",
            "\t\t metabolic acidosis \t 8.075387946754724\n",
            "\t\t white blood cell count increased \t 6.202309958195587\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t acidosis \t 5.248282548052457\n",
            "\t\t colitis ulcerative \t 5.210904626759177\n",
            "\t\t lymphocytosis \t 3.780032006027975\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t vomiting projectile \t 3.332826614379883\n",
            "\t\t large intestinal ulcer \t 3.276744329698592\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t colitis \t 2.969056518951196\n",
            "\t\t blood lactic acid increased \t 2.547429472642968\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t adverse drug reaction \t 2.2732936113096534\n",
            "\t\t hyperlactacidaemia \t 2.017960381466964\n",
            "\t\t megacolon \t 1.8767872655046876\n",
            "\t\t paradoxical drug reaction \t 1.85227180176138\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t toxic dilatation of intestine \t 1.6862962245941162\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t signet-ring cell carcinoma \t 1.5599799156188965\n",
            "\t\t congenital megacolon \t 1.47506582736969\n",
            "\t\t mixed adenoneuroendocrine carcinoma \t 1.3688544034957886\n",
            "\t\t adverse reaction \t 1.3655628802685056\n",
            "\t\t small cell carcinoma \t 1.2961148023605347\n",
            "\t\t colitis erosive \t 1.2318897247314453\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t leukocytosis \t 4.439331949707559\n",
            "\t\t leukocytosis \t 4.439331949707559\n",
            "\t\t lactic acidosis \t 4.258704122622189\n",
            "\t\t lactic acidosis \t 4.258704122622189\n",
            "\t\t metabolic acidosis \t 4.037693973377362\n",
            "\t\t metabolic acidosis \t 4.037693973377362\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t lactic acidosis \t 3.455820923090034\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t white blood cell count increased \t 3.1011549790977937\n",
            "\t\t white blood cell count increased \t 3.1011549790977937\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t colitis ulcerative \t 2.9992854427853133\n",
            "\t\t colitis \t 2.969056518951196\n",
            "\t\t acidosis \t 2.6241412740262287\n",
            "\t\t acidosis \t 2.6241412740262287\n",
            "\t\t blood lactic acid increased \t 2.547429472642968\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t adverse drug reaction \t 2.2732936113096534\n",
            "\t\t colitis ulcerative \t 2.2116191839738635\n",
            "\t\t hyperlactacidaemia \t 2.017960381466964\n",
            "\t\t lymphocytosis \t 1.8900160030139874\n",
            "\t\t lymphocytosis \t 1.8900160030139874\n",
            "\t\t megacolon \t 1.8767872655046876\n",
            "\t\t paradoxical drug reaction \t 1.85227180176138\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t large intestinal ulcer \t 1.826811661742159\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t toxic dilatation of intestine \t 1.6862962245941162\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t signet-ring cell carcinoma \t 1.5599799156188965\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t congenital megacolon \t 1.47506582736969\n",
            "\t\t large intestinal ulcer \t 1.4499326679564328\n",
            "\t\t mixed adenoneuroendocrine carcinoma \t 1.3688544034957886\n",
            "\t\t adverse reaction \t 1.3655628802685056\n",
            "\t\t small cell carcinoma \t 1.2961148023605347\n",
            "\t\t colitis erosive \t 1.2318897247314453\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['drug ineffective']\n",
            "Predicted reactions: \n",
            "\t ['hepatitis b reactivation', 'acute hepatic failure', 'hepatic failure', 'hepatitis b', 'hepatitis fulminant', 'epstein-barr virus infection reactivation', 'acute hepatitis b']\n",
            "Ungrounded reactions: \n",
            "\t ['hepatitis b reactivation', 'acute hepatic failure', 'hepatic failure', 'hepatitis b', 'hepatitis fulminant', 'epstein-barr virus infection reactivation', 'acute hepatitis b']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t hepatitis b reactivation \t 8.694513778894448\n",
            "\t\t acute hepatic failure \t 7.68435192309903\n",
            "\t\t hepatic failure \t 7.186838440714409\n",
            "\t\t hepatitis b \t 6.896550685303634\n",
            "\t\t hepatitis fulminant \t 4.67725178842387\n",
            "\t\t epstein-barr virus infection reactivation \t 2.880279952249622\n",
            "\t\t acute hepatitis b \t 1.5251129865646362\n",
            "Grounded reactions: \n",
            "\t\t acute hepatic failure \t 3.842175961549515\n",
            "\t\t acute hepatic failure \t 3.842175961549515\n",
            "\t\t hepatic failure \t 3.5934192203572044\n",
            "\t\t hepatic failure \t 3.5934192203572044\n",
            "\t\t hepatitis b reactivation \t 3.285348043773798\n",
            "\t\t hepatitis b reactivation \t 3.285348043773798\n",
            "\t\t hepatitis fulminant \t 2.338625894211935\n",
            "\t\t hepatitis fulminant \t 2.338625894211935\n",
            "\t\t hepatitis b \t 2.3030511247621823\n",
            "\t\t hepatitis b \t 2.3030511247621823\n",
            "\t\t hepatitis b \t 2.2904484357792687\n",
            "\t\t hepatitis b reactivation \t 2.123817691346852\n",
            "\t\t acute hepatitis b \t 1.5251129865646362\n",
            "\t\t epstein-barr virus infection reactivation \t 1.440139976124811\n",
            "\t\t epstein-barr virus infection reactivation \t 1.440139976124811\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['acute kidney injury', 'bone marrow failure', 'drug ineffective', 'febrile neutropenia', 'infection', 'liver injury', 'neutropenia', 'sepsis']\n",
            "Predicted reactions: \n",
            "\t ['sepsis', 'neutropenia', 'acute kidney injury', 'thrombocytopenia', 'febrile neutropenia', 'no adverse event', 'agranulocytosis', 'disease recurrence', 'bone marrow failure', 'renal failure', 'renal impairment', 'platelet count decreased', 'liver injury', 'drug-induced liver injury', 'infection', 'hepatotoxicity', 'cytopenia', 'neutropenic sepsis', 'immune thrombocytopenia', 'neutrophil count decreased', 'bacterial sepsis', 'infection reactivation', 'adverse drug reaction', 'drug effective for unapproved indication', 'neoplasm recurrence', 'cyclic neutropenia', 'systemic infection', 'sensory level normal', 'no reaction on previous exposure to drug']\n",
            "Ungrounded reactions: \n",
            "\t ['sepsis', 'neutropenia', 'acute kidney injury', 'thrombocytopenia', 'febrile neutropenia', 'no adverse event', 'agranulocytosis', 'disease recurrence', 'bone marrow failure', 'renal failure', 'renal impairment', 'platelet count decreased', 'liver injury', 'drug-induced liver injury', 'infection', 'hepatotoxicity', 'cytopenia', 'neutropenic sepsis', 'immune thrombocytopenia', 'neutrophil count decreased', 'bacterial sepsis', 'infection reactivation', 'adverse drug reaction', 'drug effective for unapproved indication', 'neoplasm recurrence', 'cyclic neutropenia', 'systemic infection', 'sensory level normal', 'no reaction on previous exposure to drug']\n",
            "Recall@10:  0.625\n",
            "Recall@20:  0.875\n",
            "Recall@30:  0.875\n",
            "Resolved reactions: \n",
            "\t\t sepsis \t 9.199019547654464\n",
            "\t\t neutropenia \t 9.05616181640497\n",
            "\t\t acute kidney injury \t 6.150001909562591\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t febrile neutropenia \t 5.6047658009866765\n",
            "\t\t no adverse event \t 5.481275748858888\n",
            "\t\t agranulocytosis \t 5.190351446394612\n",
            "\t\t disease recurrence \t 4.598552619995\n",
            "\t\t bone marrow failure \t 3.7630314102615303\n",
            "\t\t renal failure \t 3.697433668391888\n",
            "\t\t renal impairment \t 3.6073114540752194\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t liver injury \t 3.3109571761167342\n",
            "\t\t drug-induced liver injury \t 3.1055287935969482\n",
            "\t\t infection \t 3.053486914605728\n",
            "\t\t hepatotoxicity \t 2.8723139223082055\n",
            "\t\t cytopenia \t 2.748830171956887\n",
            "\t\t neutropenic sepsis \t 2.615243795857332\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t neutrophil count decreased \t 2.2487662784546107\n",
            "\t\t bacterial sepsis \t 2.2384407444825314\n",
            "\t\t infection reactivation \t 1.858154262202406\n",
            "\t\t adverse drug reaction \t 1.8188675208367635\n",
            "\t\t drug effective for unapproved indication \t 1.787482230901558\n",
            "\t\t neoplasm recurrence \t 1.7670640282489554\n",
            "\t\t cyclic neutropenia \t 1.6033440828323364\n",
            "\t\t systemic infection \t 1.5126262089733533\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "Grounded reactions: \n",
            "\t\t acute kidney injury \t 6.150001909562591\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t sepsis \t 5.7860335313383615\n",
            "\t\t febrile neutropenia \t 5.6047658009866765\n",
            "\t\t neutropenia \t 4.788381703759212\n",
            "\t\t disease recurrence \t 4.598552619995\n",
            "\t\t neutropenia \t 4.267780112645757\n",
            "\t\t bone marrow failure \t 3.7630314102615303\n",
            "\t\t renal failure \t 3.697433668391888\n",
            "\t\t renal impairment \t 3.6073114540752194\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t sepsis \t 3.412986016316102\n",
            "\t\t liver injury \t 3.3109571761167342\n",
            "\t\t no adverse event \t 3.179032740849157\n",
            "\t\t drug-induced liver injury \t 3.1055287935969482\n",
            "\t\t infection \t 3.053486914605728\n",
            "\t\t hepatotoxicity \t 2.8723139223082055\n",
            "\t\t cytopenia \t 2.748830171956887\n",
            "\t\t agranulocytosis \t 2.682857824787392\n",
            "\t\t neutropenic sepsis \t 2.615243795857332\n",
            "\t\t agranulocytosis \t 2.5074936216072192\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t neutrophil count decreased \t 2.2487662784546107\n",
            "\t\t bacterial sepsis \t 2.2384407444825314\n",
            "\t\t infection reactivation \t 1.858154262202406\n",
            "\t\t adverse drug reaction \t 1.8188675208367635\n",
            "\t\t drug effective for unapproved indication \t 1.787482230901558\n",
            "\t\t neoplasm recurrence \t 1.7670640282489554\n",
            "\t\t cyclic neutropenia \t 1.6033440828323364\n",
            "\t\t systemic infection \t 1.5126262089733533\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['infection', 'off label use', 'performance status decreased', 'product use issue', 'sepsis', 'weight decreased']\n",
            "Predicted reactions: \n",
            "\t ['sepsis', 'weight decreased', 'weight increased', 'neutropenic sepsis', 'drug resistance', 'agranulocytosis', 'neurotoxicity', 'performance status decreased', 'pathogen resistance', 'therapeutic response decreased', 'therapy partial responder', 'abnormal loss of weight', 'adverse drug reaction', 'no adverse event', 'multiple-drug resistance', 'poisoning', 'rebound effect', 'eastern cooperative oncology group performance status', 'paradoxical drug reaction', 'systemic toxicity', 'delayed effects of radiation', 'radiofrequency-induced thermochemotherapeutic effect', 'nyha classification', 'blood group o', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['sepsis', 'weight decreased', 'weight increased', 'neutropenic sepsis', 'drug resistance', 'agranulocytosis', 'neurotoxicity', 'performance status decreased', 'pathogen resistance', 'therapeutic response decreased', 'therapy partial responder', 'abnormal loss of weight', 'adverse drug reaction', 'no adverse event', 'multiple-drug resistance', 'poisoning', 'rebound effect', 'eastern cooperative oncology group performance status', 'paradoxical drug reaction', 'systemic toxicity', 'delayed effects of radiation', 'radiofrequency-induced thermochemotherapeutic effect', 'nyha classification', 'blood group o', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t sepsis \t 5.306338027132753\n",
            "\t\t weight decreased \t 4.6804568828751725\n",
            "\t\t weight increased \t 3.3927835595690716\n",
            "\t\t neutropenic sepsis \t 2.817679894395045\n",
            "\t\t drug resistance \t 2.6801084187944877\n",
            "\t\t agranulocytosis \t 2.6437867644509647\n",
            "\t\t neurotoxicity \t 2.545051157970916\n",
            "\t\t performance status decreased \t 2.4424858308096877\n",
            "\t\t pathogen resistance \t 2.238131041314796\n",
            "\t\t therapeutic response decreased \t 2.2122035731297673\n",
            "\t\t therapy partial responder \t 2.166942811526932\n",
            "\t\t abnormal loss of weight \t 2.0250111320033213\n",
            "\t\t adverse drug reaction \t 1.9463123741995398\n",
            "\t\t no adverse event \t 1.9108820812637108\n",
            "\t\t multiple-drug resistance \t 1.9086766297347255\n",
            "\t\t poisoning \t 1.7953364779208636\n",
            "\t\t rebound effect \t 1.5098337349544864\n",
            "\t\t eastern cooperative oncology group performance status \t 1.3494648933410645\n",
            "\t\t paradoxical drug reaction \t 1.3445859485967204\n",
            "\t\t systemic toxicity \t 1.294503927230835\n",
            "\t\t delayed effects of radiation \t 1.1969966888427734\n",
            "\t\t radiofrequency-induced thermochemotherapeutic effect \t 1.1540576219558716\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t sepsis \t 5.306338027132753\n",
            "\t\t weight decreased \t 4.6804568828751725\n",
            "\t\t weight increased \t 3.3927835595690716\n",
            "\t\t neutropenic sepsis \t 2.817679894395045\n",
            "\t\t drug resistance \t 2.6801084187944877\n",
            "\t\t agranulocytosis \t 2.6437867644509647\n",
            "\t\t neurotoxicity \t 2.545051157970916\n",
            "\t\t performance status decreased \t 2.4424858308096877\n",
            "\t\t pathogen resistance \t 2.238131041314796\n",
            "\t\t therapeutic response decreased \t 2.2122035731297673\n",
            "\t\t therapy partial responder \t 2.166942811526932\n",
            "\t\t abnormal loss of weight \t 2.0250111320033213\n",
            "\t\t adverse drug reaction \t 1.9463123741995398\n",
            "\t\t no adverse event \t 1.9108820812637108\n",
            "\t\t multiple-drug resistance \t 1.9086766297347255\n",
            "\t\t poisoning \t 1.7953364779208636\n",
            "\t\t rebound effect \t 1.5098337349544864\n",
            "\t\t eastern cooperative oncology group performance status \t 1.3494648933410645\n",
            "\t\t paradoxical drug reaction \t 1.3445859485967204\n",
            "\t\t systemic toxicity \t 1.294503927230835\n",
            "\t\t delayed effects of radiation \t 1.1969966888427734\n",
            "\t\t radiofrequency-induced thermochemotherapeutic effect \t 1.1540576219558716\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['drug ineffective']\n",
            "Predicted reactions: \n",
            "\t ['no adverse event', 'diarrhoea', 'adverse drug reaction', 'drug ineffective', 'infusion related reaction', 'gastrointestinal haemorrhage', 'haematochezia', 'infusion related hypersensitivity reaction', 'infusion site reaction', 'colitis ulcerative', 'immune-mediated adverse reaction', 'remission not achieved', 'cancer in remission', 'chronic leukaemia in remission', 'adverse reaction', 'disease risk factor', 'colonoscopy abnormal', 'colitis erosive', 'carcinogenicity', 'disease propensity', 'nothing by mouth order', 'ideas of reference']\n",
            "Ungrounded reactions: \n",
            "\t ['no adverse event', 'diarrhoea', 'adverse drug reaction', 'drug ineffective', 'infusion related reaction', 'gastrointestinal haemorrhage', 'haematochezia', 'infusion related hypersensitivity reaction', 'infusion site reaction', 'colitis ulcerative', 'immune-mediated adverse reaction', 'remission not achieved', 'cancer in remission', 'chronic leukaemia in remission', 'adverse reaction', 'disease risk factor', 'colonoscopy abnormal', 'colitis erosive', 'carcinogenicity', 'disease propensity', 'nothing by mouth order', 'ideas of reference']\n",
            "Recall@10:  1.0\n",
            "Recall@20:  1.0\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t no adverse event \t 4.883903609655606\n",
            "\t\t diarrhoea \t 4.330632219613925\n",
            "\t\t adverse drug reaction \t 4.1089068884480255\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t infusion related reaction \t 3.3939057235511294\n",
            "\t\t gastrointestinal haemorrhage \t 3.191432703009352\n",
            "\t\t haematochezia \t 3.0092463564109044\n",
            "\t\t infusion related hypersensitivity reaction \t 1.783442497253418\n",
            "\t\t infusion site reaction \t 1.7203346490859985\n",
            "\t\t colitis ulcerative \t 1.6868579270157131\n",
            "\t\t immune-mediated adverse reaction \t 1.5687072732240033\n",
            "\t\t remission not achieved \t 1.5212418520823996\n",
            "\t\t cancer in remission \t 1.4271535873413086\n",
            "\t\t chronic leukaemia in remission \t 1.4085389375686646\n",
            "\t\t adverse reaction \t 1.3993354819746062\n",
            "\t\t disease risk factor \t 1.1853526830673218\n",
            "\t\t colonoscopy abnormal \t 1.1421291828155518\n",
            "\t\t colitis erosive \t 1.1392080783843994\n",
            "\t\t carcinogenicity \t 1.133631706237793\n",
            "\t\t disease propensity \t 1.1307069063186646\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "Grounded reactions: \n",
            "\t\t diarrhoea \t 4.330632219613925\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t infusion related reaction \t 3.3939057235511294\n",
            "\t\t no adverse event \t 3.3319030318452447\n",
            "\t\t gastrointestinal haemorrhage \t 3.191432703009352\n",
            "\t\t haematochezia \t 3.0092463564109044\n",
            "\t\t adverse drug reaction \t 2.0651852472989334\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t infusion related hypersensitivity reaction \t 1.783442497253418\n",
            "\t\t infusion site reaction \t 1.7203346490859985\n",
            "\t\t colitis ulcerative \t 1.6868579270157131\n",
            "\t\t immune-mediated adverse reaction \t 1.5687072732240033\n",
            "\t\t no adverse event \t 1.5520005778103618\n",
            "\t\t remission not achieved \t 1.5212418520823996\n",
            "\t\t cancer in remission \t 1.4271535873413086\n",
            "\t\t chronic leukaemia in remission \t 1.4085389375686646\n",
            "\t\t adverse reaction \t 1.3993354819746062\n",
            "\t\t disease risk factor \t 1.1853526830673218\n",
            "\t\t colonoscopy abnormal \t 1.1421291828155518\n",
            "\t\t colitis erosive \t 1.1392080783843994\n",
            "\t\t carcinogenicity \t 1.133631706237793\n",
            "\t\t disease propensity \t 1.1307069063186646\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['metastases to meninges']\n",
            "Predicted reactions: \n",
            "\t ['no adverse event', 'drug interaction', 'sensory level normal', 'no reaction on previous exposure to drug', 'drug intolerance', 'adverse drug reaction', 'egfr gene mutation', 'blood brain barrier defect', 'egfr status assay', 'bispectral index', 'egfr gene overexpression', 'human epidermal growth factor receptor decreased']\n",
            "Ungrounded reactions: \n",
            "\t ['no adverse event', 'drug interaction', 'sensory level normal', 'no reaction on previous exposure to drug', 'drug intolerance', 'adverse drug reaction', 'egfr gene mutation', 'blood brain barrier defect', 'egfr status assay', 'bispectral index', 'egfr gene overexpression', 'human epidermal growth factor receptor decreased']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t no adverse event \t 9.208972032038924\n",
            "\t\t drug interaction \t 3.0057810698854865\n",
            "\t\t sensory level normal \t 2.689606189727783\n",
            "\t\t no reaction on previous exposure to drug \t 2.6601758003234863\n",
            "\t\t drug intolerance \t 2.5640184172817464\n",
            "\t\t adverse drug reaction \t 2.5197662998162627\n",
            "\t\t egfr gene mutation \t 1.0632562308390736\n",
            "\t\t blood brain barrier defect \t 0.9710973501205444\n",
            "\t\t egfr status assay \t 0.8930081725120544\n",
            "\t\t bispectral index \t 0.8710999488830566\n",
            "\t\t egfr gene overexpression \t 0.8602426052093506\n",
            "\t\t human epidermal growth factor receptor decreased \t 0.8433263301849365\n",
            "Grounded reactions: \n",
            "\t\t drug interaction \t 3.0057810698854865\n",
            "\t\t drug intolerance \t 2.5640184172817464\n",
            "\t\t adverse drug reaction \t 2.5197662998162627\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t egfr gene mutation \t 1.0632562308390736\n",
            "\t\t blood brain barrier defect \t 0.9710973501205444\n",
            "\t\t egfr status assay \t 0.8930081725120544\n",
            "\t\t bispectral index \t 0.8710999488830566\n",
            "\t\t egfr gene overexpression \t 0.8602426052093506\n",
            "\t\t human epidermal growth factor receptor decreased \t 0.8433263301849365\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['cardiac failure', 'cardiomegaly', 'hepatic function abnormal', 'pleural effusion', 'renal impairment', 'thrombocytopenia']\n",
            "Predicted reactions: \n",
            "\t ['cardiac failure', 'cardiac failure congestive', 'myocardial ischaemia', 'cardiomyopathy', 'cardiac failure acute', 'pericardial effusion', 'deep vein thrombosis', 'acute myocardial infarction', 'coronary artery disease', 'atrial thrombosis', 'pericarditis', 'fatigue', 'blood cholesterol increased', 'venous thrombosis', 'ischaemia', 'pleural effusion', 'dyspnoea', 'hypercholesterolaemia', 'arrhythmia', 'thrombosis', 'brain natriuretic peptide increased', 'asthenia', 'pancytopenia', 'renal impairment', 'intracardiac thrombus', 'renal failure', 'bradycardia', 'hepatic failure', 'palpitations', 'dyspnoea exertional', 'myocarditis', 'acute coronary syndrome', 'cardiac disorder', 'cardiac dysfunction', 'cardiotoxicity', 'tricuspid valve incompetence', 'peripheral ischaemia', 'arteriospasm coronary', 'c-reactive protein increased', 'cardiomegaly', 'mitral valve incompetence', 'congestive cardiomyopathy', 'coronary artery stenosis', 'vasospasm', 'arteriosclerosis', 'cardiac failure chronic', 'hyperlipidaemia', 'brain natriuretic peptide', 'high density lipoprotein increased', 'orthopnoea', 'inflammatory marker increased', 'restrictive cardiomyopathy', 'vasoconstriction', 'prohormone brain natriuretic peptide', 'coronary artery thrombosis', 'mental fatigue', 'infectious pleural effusion', 'pericardial drainage', 'pleuropericarditis', 'tricuspid valve disease', 'n-terminal prohormone brain natriuretic peptide increased', 'toxic cardiomyopathy', 'prohormone brain natriuretic peptide increased', 'reticulocytopenia', 'inflammatory marker test', 'bicytopenia', 'dilatation atrial', 'cardiac ventricular thrombosis', 'right atrial dilatation', 'hypokinesia', 'atherosclerotic plaque rupture', 'fractional flow reserve', 'forced vital capacity', 'stroke volume', 'parkinsonian rest tremor', 'synkinesis', 'myocardial strain', 'gleason grading score', 'post vaccination challenge strain shedding']\n",
            "Ungrounded reactions: \n",
            "\t ['cardiac failure', 'cardiac failure congestive', 'myocardial ischaemia', 'cardiomyopathy', 'cardiac failure acute', 'pericardial effusion', 'deep vein thrombosis', 'acute myocardial infarction', 'coronary artery disease', 'atrial thrombosis', 'pericarditis', 'fatigue', 'blood cholesterol increased', 'venous thrombosis', 'ischaemia', 'pleural effusion', 'dyspnoea', 'hypercholesterolaemia', 'arrhythmia', 'thrombosis', 'brain natriuretic peptide increased', 'asthenia', 'pancytopenia', 'renal impairment', 'intracardiac thrombus', 'renal failure', 'bradycardia', 'hepatic failure', 'palpitations', 'dyspnoea exertional', 'myocarditis', 'acute coronary syndrome', 'cardiac disorder', 'cardiac dysfunction', 'cardiotoxicity', 'tricuspid valve incompetence', 'peripheral ischaemia', 'arteriospasm coronary', 'c-reactive protein increased', 'cardiomegaly', 'mitral valve incompetence', 'congestive cardiomyopathy', 'coronary artery stenosis', 'vasospasm', 'arteriosclerosis', 'cardiac failure chronic', 'hyperlipidaemia', 'brain natriuretic peptide', 'high density lipoprotein increased', 'orthopnoea', 'inflammatory marker increased', 'restrictive cardiomyopathy', 'vasoconstriction', 'prohormone brain natriuretic peptide', 'coronary artery thrombosis', 'mental fatigue', 'infectious pleural effusion', 'pericardial drainage', 'pleuropericarditis', 'tricuspid valve disease', 'n-terminal prohormone brain natriuretic peptide increased', 'toxic cardiomyopathy', 'prohormone brain natriuretic peptide increased', 'reticulocytopenia', 'inflammatory marker test', 'bicytopenia', 'dilatation atrial', 'cardiac ventricular thrombosis', 'right atrial dilatation', 'hypokinesia', 'atherosclerotic plaque rupture', 'fractional flow reserve', 'forced vital capacity', 'stroke volume', 'parkinsonian rest tremor', 'synkinesis', 'myocardial strain', 'gleason grading score', 'post vaccination challenge strain shedding']\n",
            "Recall@10:  0.16666666666666666\n",
            "Recall@20:  0.3333333333333333\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t cardiac failure \t 15.193568475072704\n",
            "\t\t cardiac failure congestive \t 11.39592192711338\n",
            "\t\t myocardial ischaemia \t 10.678990860277075\n",
            "\t\t cardiomyopathy \t 7.861909684589173\n",
            "\t\t cardiac failure acute \t 7.093771651687671\n",
            "\t\t pericardial effusion \t 6.6904968340772495\n",
            "\t\t deep vein thrombosis \t 5.780012882754894\n",
            "\t\t acute myocardial infarction \t 5.671245888671413\n",
            "\t\t coronary artery disease \t 5.477392275047135\n",
            "\t\t atrial thrombosis \t 5.399145645544899\n",
            "\t\t pericarditis \t 5.156885709183468\n",
            "\t\t fatigue \t 5.0459531444195\n",
            "\t\t blood cholesterol increased \t 4.527768248634385\n",
            "\t\t venous thrombosis \t 4.5128696139560205\n",
            "\t\t ischaemia \t 4.501604365606802\n",
            "\t\t pleural effusion \t 4.478829863177229\n",
            "\t\t dyspnoea \t 4.460679595804341\n",
            "\t\t hypercholesterolaemia \t 4.143233045927635\n",
            "\t\t arrhythmia \t 4.0568851380667255\n",
            "\t\t thrombosis \t 3.7546265268854278\n",
            "\t\t brain natriuretic peptide increased \t 3.705595069281859\n",
            "\t\t asthenia \t 3.563371940198621\n",
            "\t\t pancytopenia \t 3.3639625944350016\n",
            "\t\t renal impairment \t 3.2372358652165443\n",
            "\t\t intracardiac thrombus \t 3.2195568350266166\n",
            "\t\t renal failure \t 3.1918203022257265\n",
            "\t\t bradycardia \t 3.1848756486667864\n",
            "\t\t hepatic failure \t 3.0338417856476196\n",
            "\t\t palpitations \t 2.7963924129188387\n",
            "\t\t dyspnoea exertional \t 2.7855659377121276\n",
            "\t\t myocarditis \t 2.7744383083748874\n",
            "\t\t acute coronary syndrome \t 2.5528617520531225\n",
            "\t\t cardiac disorder \t 2.462105613886922\n",
            "\t\t cardiac dysfunction \t 2.461150465126438\n",
            "\t\t cardiotoxicity \t 2.3751084441806998\n",
            "\t\t tricuspid valve incompetence \t 2.3725365592028993\n",
            "\t\t peripheral ischaemia \t 2.3415241943197764\n",
            "\t\t arteriospasm coronary \t 2.1543118383431694\n",
            "\t\t c-reactive protein increased \t 2.1093095923981977\n",
            "\t\t cardiomegaly \t 2.098798409409469\n",
            "\t\t mitral valve incompetence \t 2.093047399265651\n",
            "\t\t congestive cardiomyopathy \t 2.080209275801203\n",
            "\t\t coronary artery stenosis \t 2.0383409721662202\n",
            "\t\t vasospasm \t 1.9999998807907104\n",
            "\t\t arteriosclerosis \t 1.9183125882723153\n",
            "\t\t cardiac failure chronic \t 1.8566801542677305\n",
            "\t\t hyperlipidaemia \t 1.8129706676464519\n",
            "\t\t brain natriuretic peptide \t 1.780796766281128\n",
            "\t\t high density lipoprotein increased \t 1.7624353170394897\n",
            "\t\t orthopnoea \t 1.755833806594105\n",
            "\t\t inflammatory marker increased \t 1.735298991203308\n",
            "\t\t restrictive cardiomyopathy \t 1.7307666540145874\n",
            "\t\t vasoconstriction \t 1.709447979927063\n",
            "\t\t prohormone brain natriuretic peptide \t 1.7078843116760254\n",
            "\t\t coronary artery thrombosis \t 1.7070953845977783\n",
            "\t\t mental fatigue \t 1.660275936126709\n",
            "\t\t infectious pleural effusion \t 1.6089239639672863\n",
            "\t\t pericardial drainage \t 1.6022896766662598\n",
            "\t\t pleuropericarditis \t 1.584289789199829\n",
            "\t\t tricuspid valve disease \t 1.5794159173965454\n",
            "\t\t n-terminal prohormone brain natriuretic peptide increased \t 1.5747636556625366\n",
            "\t\t toxic cardiomyopathy \t 1.4557414054870605\n",
            "\t\t prohormone brain natriuretic peptide increased \t 1.454856276512146\n",
            "\t\t reticulocytopenia \t 1.4208530187606812\n",
            "\t\t inflammatory marker test \t 1.412926197052002\n",
            "\t\t bicytopenia \t 1.3997321128845215\n",
            "\t\t dilatation atrial \t 1.375124454498291\n",
            "\t\t cardiac ventricular thrombosis \t 1.3623605750832932\n",
            "\t\t right atrial dilatation \t 1.3438618183135986\n",
            "\t\t hypokinesia \t 1.2988794827591885\n",
            "\t\t atherosclerotic plaque rupture \t 1.2171039581298828\n",
            "\t\t fractional flow reserve \t 0.9587522149085999\n",
            "\t\t forced vital capacity \t 0.9420304298400879\n",
            "\t\t stroke volume \t 0.9163352847099304\n",
            "\t\t parkinsonian rest tremor \t 0.9149860143661499\n",
            "\t\t synkinesis \t 0.901874840259552\n",
            "\t\t myocardial strain \t 0.8588438630104065\n",
            "\t\t gleason grading score \t 0.8536218404769897\n",
            "\t\t post vaccination challenge strain shedding \t 0.8182023763656616\n",
            "Grounded reactions: \n",
            "\t\t fatigue \t 5.0459531444195\n",
            "\t\t cardiac failure \t 4.694152825747449\n",
            "\t\t pleural effusion \t 4.478829863177229\n",
            "\t\t dyspnoea \t 4.460679595804341\n",
            "\t\t pericardial effusion \t 4.106244591302996\n",
            "\t\t arrhythmia \t 4.0568851380667255\n",
            "\t\t cardiac failure \t 3.9217558355742135\n",
            "\t\t thrombosis \t 3.7546265268854278\n",
            "\t\t cardiac failure \t 3.637473294436047\n",
            "\t\t cardiomyopathy \t 3.5756624406183315\n",
            "\t\t asthenia \t 3.563371940198621\n",
            "\t\t cardiac failure congestive \t 3.5206522679973076\n",
            "\t\t pancytopenia \t 3.3639625944350016\n",
            "\t\t acute myocardial infarction \t 3.3103714837772724\n",
            "\t\t renal impairment \t 3.2372358652165443\n",
            "\t\t renal failure \t 3.1918203022257265\n",
            "\t\t bradycardia \t 3.1848756486667864\n",
            "\t\t myocardial ischaemia \t 3.177641491601147\n",
            "\t\t hepatic failure \t 3.0338417856476196\n",
            "\t\t deep vein thrombosis \t 3.0018923946256155\n",
            "\t\t cardiac failure \t 2.9401865193149948\n",
            "\t\t cardiac failure congestive \t 2.8748879043988356\n",
            "\t\t coronary artery disease \t 2.8571127380468426\n",
            "\t\t palpitations \t 2.7963924129188387\n",
            "\t\t dyspnoea exertional \t 2.7855659377121276\n",
            "\t\t deep vein thrombosis \t 2.7781204881292787\n",
            "\t\t myocarditis \t 2.7744383083748874\n",
            "\t\t cardiac failure acute \t 2.7580446188634125\n",
            "\t\t cardiac failure congestive \t 2.7217017602339233\n",
            "\t\t pericarditis \t 2.694835957297503\n",
            "\t\t myocardial ischaemia \t 2.6712175412635473\n",
            "\t\t coronary artery disease \t 2.6202795370002927\n",
            "\t\t ischaemia \t 2.5901175193459545\n",
            "\t\t pericardial effusion \t 2.5842522427742534\n",
            "\t\t acute coronary syndrome \t 2.5528617520531225\n",
            "\t\t venous thrombosis \t 2.535826818533349\n",
            "\t\t myocardial ischaemia \t 2.4876773924958053\n",
            "\t\t cardiac disorder \t 2.462105613886922\n",
            "\t\t pericarditis \t 2.4620497518859654\n",
            "\t\t cardiac dysfunction \t 2.461150465126438\n",
            "\t\t cardiotoxicity \t 2.3751084441806998\n",
            "\t\t tricuspid valve incompetence \t 2.3725365592028993\n",
            "\t\t acute myocardial infarction \t 2.360874404894141\n",
            "\t\t myocardial ischaemia \t 2.3424544349165757\n",
            "\t\t peripheral ischaemia \t 2.3415241943197764\n",
            "\t\t blood cholesterol increased \t 2.320358703273786\n",
            "\t\t cardiac failure acute \t 2.3102605862661956\n",
            "\t\t cardiac failure congestive \t 2.278679994483314\n",
            "\t\t cardiomyopathy \t 2.2781760917425515\n",
            "\t\t blood cholesterol increased \t 2.207409545360599\n",
            "\t\t arteriospasm coronary \t 2.1543118383431694\n",
            "\t\t hypercholesterolaemia \t 2.1308716022904615\n",
            "\t\t c-reactive protein increased \t 2.1093095923981977\n",
            "\t\t cardiomegaly \t 2.098798409409469\n",
            "\t\t mitral valve incompetence \t 2.093047399265651\n",
            "\t\t congestive cardiomyopathy \t 2.080209275801203\n",
            "\t\t coronary artery stenosis \t 2.0383409721662202\n",
            "\t\t cardiac failure acute \t 2.0254664465580636\n",
            "\t\t hypercholesterolaemia \t 2.012361443637174\n",
            "\t\t cardiomyopathy \t 2.0080711522282897\n",
            "\t\t vasospasm \t 1.9999998807907104\n",
            "\t\t atrial thrombosis \t 1.9923200425351966\n",
            "\t\t venous thrombosis \t 1.9770427954226715\n",
            "\t\t brain natriuretic peptide increased \t 1.9446591451811297\n",
            "\t\t arteriosclerosis \t 1.9183125882723153\n",
            "\t\t ischaemia \t 1.9114868462608474\n",
            "\t\t intracardiac thrombus \t 1.861782575334741\n",
            "\t\t cardiac failure chronic \t 1.8566801542677305\n",
            "\t\t hyperlipidaemia \t 1.8129706676464519\n",
            "\t\t atrial thrombosis \t 1.7967944507062654\n",
            "\t\t brain natriuretic peptide \t 1.780796766281128\n",
            "\t\t high density lipoprotein increased \t 1.7624353170394897\n",
            "\t\t brain natriuretic peptide increased \t 1.7609359241007294\n",
            "\t\t orthopnoea \t 1.755833806594105\n",
            "\t\t inflammatory marker increased \t 1.735298991203308\n",
            "\t\t restrictive cardiomyopathy \t 1.7307666540145874\n",
            "\t\t vasoconstriction \t 1.709447979927063\n",
            "\t\t prohormone brain natriuretic peptide \t 1.7078843116760254\n",
            "\t\t coronary artery thrombosis \t 1.7070953845977783\n",
            "\t\t mental fatigue \t 1.660275936126709\n",
            "\t\t atrial thrombosis \t 1.610031152303437\n",
            "\t\t infectious pleural effusion \t 1.6089239639672863\n",
            "\t\t pericardial drainage \t 1.6022896766662598\n",
            "\t\t pleuropericarditis \t 1.584289789199829\n",
            "\t\t tricuspid valve disease \t 1.5794159173965454\n",
            "\t\t n-terminal prohormone brain natriuretic peptide increased \t 1.5747636556625366\n",
            "\t\t toxic cardiomyopathy \t 1.4557414054870605\n",
            "\t\t prohormone brain natriuretic peptide increased \t 1.454856276512146\n",
            "\t\t reticulocytopenia \t 1.4208530187606812\n",
            "\t\t inflammatory marker test \t 1.412926197052002\n",
            "\t\t bicytopenia \t 1.3997321128845215\n",
            "\t\t dilatation atrial \t 1.375124454498291\n",
            "\t\t cardiac ventricular thrombosis \t 1.3623605750832932\n",
            "\t\t intracardiac thrombus \t 1.3577742596918758\n",
            "\t\t right atrial dilatation \t 1.3438618183135986\n",
            "\t\t hypokinesia \t 1.2988794827591885\n",
            "\t\t atherosclerotic plaque rupture \t 1.2171039581298828\n",
            "\t\t fractional flow reserve \t 0.9587522149085999\n",
            "\t\t forced vital capacity \t 0.9420304298400879\n",
            "\t\t stroke volume \t 0.9163352847099304\n",
            "\t\t parkinsonian rest tremor \t 0.9149860143661499\n",
            "\t\t synkinesis \t 0.901874840259552\n",
            "\t\t myocardial strain \t 0.8588438630104065\n",
            "\t\t gleason grading score \t 0.8536218404769897\n",
            "\t\t post vaccination challenge strain shedding \t 0.8182023763656616\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['drug ineffective']\n",
            "Predicted reactions: \n",
            "\t ['sepsis', 'septic shock', 'multiple organ dysfunction syndrome', 'no adverse event', 'shock', 'diarrhoea', 'renal failure', 'death', 'depressed level of consciousness', 'loss of consciousness', 'constipation', 'drug ineffective', 'altered state of consciousness', 'necrosis', 'necrotising fasciitis', 'neutropenic sepsis', 'skin necrosis', 'sudden death', 'bacterial sepsis', 'circulatory collapse', 'adverse drug reaction', 'amputation', 'gangrene', 'debridement', 'organ failure', 'limb amputation', 'clinical death', 'necrotising fasciitis streptococcal', 'burn debridement', 'necrotising fasciitis staphylococcal', 'leg amputation', 'diarrhoea infectious', 'no reaction on previous exposure to drug', 'joint debridement', 'drug screen negative', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['sepsis', 'septic shock', 'multiple organ dysfunction syndrome', 'no adverse event', 'shock', 'diarrhoea', 'renal failure', 'death', 'depressed level of consciousness', 'loss of consciousness', 'constipation', 'drug ineffective', 'altered state of consciousness', 'necrosis', 'necrotising fasciitis', 'neutropenic sepsis', 'skin necrosis', 'sudden death', 'bacterial sepsis', 'circulatory collapse', 'adverse drug reaction', 'amputation', 'gangrene', 'debridement', 'organ failure', 'limb amputation', 'clinical death', 'necrotising fasciitis streptococcal', 'burn debridement', 'necrotising fasciitis staphylococcal', 'leg amputation', 'diarrhoea infectious', 'no reaction on previous exposure to drug', 'joint debridement', 'drug screen negative', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  1.0\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t sepsis \t 13.946157822856332\n",
            "\t\t septic shock \t 10.747946611229324\n",
            "\t\t multiple organ dysfunction syndrome \t 9.99318387275541\n",
            "\t\t no adverse event \t 6.824761875697967\n",
            "\t\t shock \t 6.227117226143714\n",
            "\t\t diarrhoea \t 6.134954194154853\n",
            "\t\t renal failure \t 5.959955565224317\n",
            "\t\t death \t 5.647981780445181\n",
            "\t\t depressed level of consciousness \t 4.327112271338578\n",
            "\t\t loss of consciousness \t 3.9058931226391485\n",
            "\t\t constipation \t 3.7138263321455196\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t altered state of consciousness \t 3.585676854245646\n",
            "\t\t necrosis \t 3.1249477378833723\n",
            "\t\t necrotising fasciitis \t 2.689090222595603\n",
            "\t\t neutropenic sepsis \t 2.615243795857332\n",
            "\t\t skin necrosis \t 2.596260494515278\n",
            "\t\t sudden death \t 2.329618247112669\n",
            "\t\t bacterial sepsis \t 2.2384407444825314\n",
            "\t\t circulatory collapse \t 2.1937769036671755\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t amputation \t 2.0\n",
            "\t\t gangrene \t 1.9976923678253349\n",
            "\t\t debridement \t 1.9766271114349365\n",
            "\t\t organ failure \t 1.91442462443019\n",
            "\t\t limb amputation \t 1.9069154262542725\n",
            "\t\t clinical death \t 1.8975193500518799\n",
            "\t\t necrotising fasciitis streptococcal \t 1.7609443664550781\n",
            "\t\t burn debridement \t 1.7426750659942627\n",
            "\t\t necrotising fasciitis staphylococcal \t 1.7188953161239624\n",
            "\t\t leg amputation \t 1.679053544998169\n",
            "\t\t diarrhoea infectious \t 1.6109651327133179\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t joint debridement \t 1.5147638320922852\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t diarrhoea \t 6.134954194154853\n",
            "\t\t sepsis \t 5.7860335313383615\n",
            "\t\t death \t 5.647981780445181\n",
            "\t\t septic shock \t 5.373973305614662\n",
            "\t\t septic shock \t 5.373973305614662\n",
            "\t\t multiple organ dysfunction syndrome \t 5.043167240294808\n",
            "\t\t multiple organ dysfunction syndrome \t 4.950016632460601\n",
            "\t\t depressed level of consciousness \t 4.327112271338578\n",
            "\t\t sepsis \t 4.080062145758985\n",
            "\t\t sepsis \t 4.080062145758985\n",
            "\t\t loss of consciousness \t 3.9058931226391485\n",
            "\t\t constipation \t 3.7138263321455196\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t altered state of consciousness \t 3.585676854245646\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t no adverse event \t 3.3319030318452447\n",
            "\t\t necrosis \t 3.1249477378833723\n",
            "\t\t shock \t 3.113558613071857\n",
            "\t\t shock \t 3.113558613071857\n",
            "\t\t renal failure \t 3.012501388215662\n",
            "\t\t renal failure \t 2.947454177008655\n",
            "\t\t necrotising fasciitis \t 2.689090222595603\n",
            "\t\t neutropenic sepsis \t 2.615243795857332\n",
            "\t\t skin necrosis \t 2.596260494515278\n",
            "\t\t sudden death \t 2.329618247112669\n",
            "\t\t bacterial sepsis \t 2.2384407444825314\n",
            "\t\t circulatory collapse \t 2.1937769036671755\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t amputation \t 2.0\n",
            "\t\t gangrene \t 1.9976923678253349\n",
            "\t\t debridement \t 1.9766271114349365\n",
            "\t\t organ failure \t 1.91442462443019\n",
            "\t\t limb amputation \t 1.9069154262542725\n",
            "\t\t clinical death \t 1.8975193500518799\n",
            "\t\t necrotising fasciitis streptococcal \t 1.7609443664550781\n",
            "\t\t burn debridement \t 1.7426750659942627\n",
            "\t\t necrotising fasciitis staphylococcal \t 1.7188953161239624\n",
            "\t\t leg amputation \t 1.679053544998169\n",
            "\t\t diarrhoea infectious \t 1.6109651327133179\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t joint debridement \t 1.5147638320922852\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['abdominal pain', 'condition aggravated', 'disease progression', 'hepatic enzyme increased', 'hyperbilirubinaemia', 'metastasis', 'off label use', 'thrombocytopenia']\n",
            "Predicted reactions: \n",
            "\t ['thrombocytopenia', 'platelet count decreased', 'pulmonary embolism', 'abdominal pain', 'immune thrombocytopenia', 'anaemia', 'hypotension', 'venous thrombosis', 'dizziness', 'hypoxia', 'embolism', 'deep vein thrombosis', 'back pain', 'hepatic enzyme increased', 'haemoptysis', 'alanine aminotransferase increased', 'hyperbilirubinaemia', 'thrombosis', 'blood pressure decreased', 'pulmonary microemboli', 'transaminases increased', 'abdominal pain upper', 'blood bilirubin increased', 'pulmonary thrombosis', 'oxygen saturation decreased', 'lymphadenopathy', 'jaundice', 'abdominal discomfort', 'pulmonary haemorrhage', 'vertigo', 'fluid overload', 'haematemesis', 'abdominal pain lower', 'thrombocytosis', 'retroperitoneal lymphadenopathy', 'tumour embolism', 'dizziness postural', 'blood pressure systolic decreased', 'microcytic anaemia', 'abdominal lymphadenopathy', 'deficiency anaemia', 'anaemic hypoxia', 'renal mass', 'venous thrombosis limb', 'hypervolaemia', 'metastatic pulmonary embolism', 'fluid imbalance', 'decreased ventricular afterload', 'kidney enlargement', 'decreased ventricular preload', 'renal hypertrophy', 'pulmonary function test decreased', 'focal cortical resection', 'hippocampal sclerosis', 'fibrous cortical defect']\n",
            "Ungrounded reactions: \n",
            "\t ['thrombocytopenia', 'platelet count decreased', 'pulmonary embolism', 'abdominal pain', 'immune thrombocytopenia', 'anaemia', 'hypotension', 'venous thrombosis', 'dizziness', 'hypoxia', 'embolism', 'deep vein thrombosis', 'back pain', 'hepatic enzyme increased', 'haemoptysis', 'alanine aminotransferase increased', 'hyperbilirubinaemia', 'thrombosis', 'blood pressure decreased', 'pulmonary microemboli', 'transaminases increased', 'abdominal pain upper', 'blood bilirubin increased', 'pulmonary thrombosis', 'oxygen saturation decreased', 'lymphadenopathy', 'jaundice', 'abdominal discomfort', 'pulmonary haemorrhage', 'vertigo', 'fluid overload', 'haematemesis', 'abdominal pain lower', 'thrombocytosis', 'retroperitoneal lymphadenopathy', 'tumour embolism', 'dizziness postural', 'blood pressure systolic decreased', 'microcytic anaemia', 'abdominal lymphadenopathy', 'deficiency anaemia', 'anaemic hypoxia', 'renal mass', 'venous thrombosis limb', 'hypervolaemia', 'metastatic pulmonary embolism', 'fluid imbalance', 'decreased ventricular afterload', 'kidney enlargement', 'decreased ventricular preload', 'renal hypertrophy', 'pulmonary function test decreased', 'focal cortical resection', 'hippocampal sclerosis', 'fibrous cortical defect']\n",
            "Recall@10:  0.25\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t thrombocytopenia \t 23.22565696081648\n",
            "\t\t platelet count decreased \t 13.085106448589798\n",
            "\t\t pulmonary embolism \t 9.591149825095293\n",
            "\t\t abdominal pain \t 8.921612354781521\n",
            "\t\t immune thrombocytopenia \t 7.178773239438796\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t venous thrombosis \t 5.250054938504053\n",
            "\t\t dizziness \t 5.002053790283374\n",
            "\t\t hypoxia \t 4.670192482229418\n",
            "\t\t embolism \t 4.535427724923405\n",
            "\t\t deep vein thrombosis \t 4.474129803001089\n",
            "\t\t back pain \t 4.13865273466732\n",
            "\t\t hepatic enzyme increased \t 4.064588720886751\n",
            "\t\t haemoptysis \t 4.040627044853302\n",
            "\t\t alanine aminotransferase increased \t 3.896521712897921\n",
            "\t\t hyperbilirubinaemia \t 3.6651814274006633\n",
            "\t\t thrombosis \t 3.4670128681864028\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t pulmonary microemboli \t 3.3926249742507935\n",
            "\t\t transaminases increased \t 3.3594641535268277\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t blood bilirubin increased \t 3.334217327028947\n",
            "\t\t pulmonary thrombosis \t 3.280269742012024\n",
            "\t\t oxygen saturation decreased \t 3.158836194448269\n",
            "\t\t lymphadenopathy \t 3.157761431258889\n",
            "\t\t jaundice \t 3.057157854339577\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t pulmonary haemorrhage \t 2.695031027873436\n",
            "\t\t vertigo \t 2.6231935713626284\n",
            "\t\t fluid overload \t 2.519879235391242\n",
            "\t\t haematemesis \t 2.24625185302505\n",
            "\t\t abdominal pain lower \t 2.0637268128847355\n",
            "\t\t thrombocytosis \t 2.03881583099053\n",
            "\t\t retroperitoneal lymphadenopathy \t 1.9999998807907104\n",
            "\t\t tumour embolism \t 1.9179171323776245\n",
            "\t\t dizziness postural \t 1.9157160247242142\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t abdominal lymphadenopathy \t 1.8193693161010742\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t anaemic hypoxia \t 1.765486717224121\n",
            "\t\t renal mass \t 1.7591931819915771\n",
            "\t\t venous thrombosis limb \t 1.61574125289917\n",
            "\t\t hypervolaemia \t 1.6100788877314578\n",
            "\t\t metastatic pulmonary embolism \t 1.5466313362121582\n",
            "\t\t fluid imbalance \t 1.462489366531372\n",
            "\t\t decreased ventricular afterload \t 1.4331474304199219\n",
            "\t\t kidney enlargement \t 1.399002969500664\n",
            "\t\t decreased ventricular preload \t 1.3907315731048584\n",
            "\t\t renal hypertrophy \t 1.3879657983779907\n",
            "\t\t pulmonary function test decreased \t 1.3833017349243164\n",
            "\t\t focal cortical resection \t 1.125450611114502\n",
            "\t\t hippocampal sclerosis \t 1.0270581245422363\n",
            "\t\t fibrous cortical defect \t 1.0055245161056519\n",
            "Grounded reactions: \n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t dizziness \t 5.002053790283374\n",
            "\t\t thrombocytopenia \t 4.899794014108922\n",
            "\t\t pulmonary embolism \t 4.818007634490475\n",
            "\t\t pulmonary embolism \t 4.773142190604817\n",
            "\t\t hypoxia \t 4.670192482229418\n",
            "\t\t deep vein thrombosis \t 4.474129803001089\n",
            "\t\t back pain \t 4.13865273466732\n",
            "\t\t hepatic enzyme increased \t 4.064588720886751\n",
            "\t\t haemoptysis \t 4.040627044853302\n",
            "\t\t alanine aminotransferase increased \t 3.896521712897921\n",
            "\t\t hyperbilirubinaemia \t 3.6651814274006633\n",
            "\t\t abdominal pain \t 3.614740275215563\n",
            "\t\t thrombosis \t 3.4670128681864028\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t transaminases increased \t 3.3594641535268277\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t blood bilirubin increased \t 3.334217327028947\n",
            "\t\t oxygen saturation decreased \t 3.158836194448269\n",
            "\t\t lymphadenopathy \t 3.157761431258889\n",
            "\t\t jaundice \t 3.057157854339577\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t venous thrombosis \t 2.9140365375370334\n",
            "\t\t platelet count decreased \t 2.8133011249199846\n",
            "\t\t pulmonary haemorrhage \t 2.695031027873436\n",
            "\t\t vertigo \t 2.6231935713626284\n",
            "\t\t fluid overload \t 2.519879235391242\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t embolism \t 2.373857983928098\n",
            "\t\t venous thrombosis \t 2.3360184009670193\n",
            "\t\t haematemesis \t 2.24625185302505\n",
            "\t\t embolism \t 2.1615697409953074\n",
            "\t\t abdominal pain lower \t 2.0637268128847355\n",
            "\t\t thrombocytosis \t 2.03881583099053\n",
            "\t\t retroperitoneal lymphadenopathy \t 1.9999998807907104\n",
            "\t\t tumour embolism \t 1.9179171323776245\n",
            "\t\t dizziness postural \t 1.9157160247242142\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t abdominal lymphadenopathy \t 1.8193693161010742\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t anaemic hypoxia \t 1.765486717224121\n",
            "\t\t renal mass \t 1.7591931819915771\n",
            "\t\t pulmonary microemboli \t 1.70327889919281\n",
            "\t\t pulmonary microemboli \t 1.6893460750579834\n",
            "\t\t pulmonary thrombosis \t 1.6442219018936157\n",
            "\t\t pulmonary thrombosis \t 1.6360478401184082\n",
            "\t\t venous thrombosis limb \t 1.61574125289917\n",
            "\t\t hypervolaemia \t 1.6100788877314578\n",
            "\t\t metastatic pulmonary embolism \t 1.5466313362121582\n",
            "\t\t fluid imbalance \t 1.462489366531372\n",
            "\t\t decreased ventricular afterload \t 1.4331474304199219\n",
            "\t\t kidney enlargement \t 1.399002969500664\n",
            "\t\t decreased ventricular preload \t 1.3907315731048584\n",
            "\t\t renal hypertrophy \t 1.3879657983779907\n",
            "\t\t pulmonary function test decreased \t 1.3833017349243164\n",
            "\t\t focal cortical resection \t 1.125450611114502\n",
            "\t\t hippocampal sclerosis \t 1.0270581245422363\n",
            "\t\t fibrous cortical defect \t 1.0055245161056519\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['encephalitis', 'off label use', 'pemphigoid', 'rash maculo-papular', 'thrombocytopenia']\n",
            "Predicted reactions: \n",
            "\t ['dermatitis bullous', 'pemphigoid', 'blister', 'dermatitis', 'rash pruritic', 'pemphigus', 'urticaria', 'pruritus', 'eczema', 'rash', 'colitis', 'hepatitis', 'drug interaction', 'rash maculo-papular', 'tubulointerstitial nephritis', 'adverse drug reaction', 'stevens-johnson syndrome', 'urticaria papular', 'drug intolerance', 'idiopathic urticaria', 'nephropathy', 'erythema multiforme', 'toxic epidermal necrolysis', 'nephritis', 'erythema', 'lupus-like syndrome', 'encephalitis', 'enterocolitis', 'skin hyperpigmentation', 'hypophysitis', 'rash macular', 'systemic lupus erythematosus', 'vitiligo', 'enteritis', 'skin plaque', 'hepatitis acute', 'hepatitis e', 'rash erythematous', 'dermatitis allergic', 'granuloma', 'lichenoid keratosis', 'pruritus allergic', 'lichenification', 'lymphocytic hypophysitis', 'granuloma skin', 'pigmentation disorder', 'cutaneous lupus erythematosus', 'granulomatous dermatitis', 'rash vesicular', 'lichen striatus', 'encephalitis viral', 'oral mucosal blistering', 'mucosa vesicle', 'panencephalitis', 'skin abrasion', 'oropharyngeal blistering', 'scratch', 'scab', 'genital blister', 'hypopituitarism']\n",
            "Ungrounded reactions: \n",
            "\t ['dermatitis bullous', 'pemphigoid', 'blister', 'dermatitis', 'rash pruritic', 'pemphigus', 'urticaria', 'pruritus', 'eczema', 'rash', 'colitis', 'hepatitis', 'drug interaction', 'rash maculo-papular', 'tubulointerstitial nephritis', 'adverse drug reaction', 'stevens-johnson syndrome', 'urticaria papular', 'drug intolerance', 'idiopathic urticaria', 'nephropathy', 'erythema multiforme', 'toxic epidermal necrolysis', 'nephritis', 'erythema', 'lupus-like syndrome', 'encephalitis', 'enterocolitis', 'skin hyperpigmentation', 'hypophysitis', 'rash macular', 'systemic lupus erythematosus', 'vitiligo', 'enteritis', 'skin plaque', 'hepatitis acute', 'hepatitis e', 'rash erythematous', 'dermatitis allergic', 'granuloma', 'lichenoid keratosis', 'pruritus allergic', 'lichenification', 'lymphocytic hypophysitis', 'granuloma skin', 'pigmentation disorder', 'cutaneous lupus erythematosus', 'granulomatous dermatitis', 'rash vesicular', 'lichen striatus', 'encephalitis viral', 'oral mucosal blistering', 'mucosa vesicle', 'panencephalitis', 'skin abrasion', 'oropharyngeal blistering', 'scratch', 'scab', 'genital blister', 'hypopituitarism']\n",
            "Recall@10:  0.2\n",
            "Recall@20:  0.4\n",
            "Recall@30:  0.6\n",
            "Resolved reactions: \n",
            "\t\t dermatitis bullous \t 12.744536286147003\n",
            "\t\t pemphigoid \t 10.258156193722185\n",
            "\t\t blister \t 8.561830517826218\n",
            "\t\t dermatitis \t 5.820042073986915\n",
            "\t\t rash pruritic \t 5.7066477282282335\n",
            "\t\t pemphigus \t 5.643931094702858\n",
            "\t\t urticaria \t 5.497690802568533\n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t eczema \t 4.511105606400868\n",
            "\t\t rash \t 4.204907471001762\n",
            "\t\t colitis \t 4.1700437246299\n",
            "\t\t hepatitis \t 4.154471272652927\n",
            "\t\t drug interaction \t 3.693773994980663\n",
            "\t\t rash maculo-papular \t 3.5853229924046217\n",
            "\t\t tubulointerstitial nephritis \t 3.388881160010349\n",
            "\t\t adverse drug reaction \t 3.118395371754109\n",
            "\t\t stevens-johnson syndrome \t 3.0773251446367045\n",
            "\t\t urticaria papular \t 3.023838520050049\n",
            "\t\t drug intolerance \t 3.006131832584599\n",
            "\t\t idiopathic urticaria \t 2.8751953840255737\n",
            "\t\t nephropathy \t 2.7502505492597344\n",
            "\t\t erythema multiforme \t 2.6418833873664216\n",
            "\t\t toxic epidermal necrolysis \t 2.637583938251987\n",
            "\t\t nephritis \t 2.5430404724093343\n",
            "\t\t erythema \t 2.505305105450717\n",
            "\t\t lupus-like syndrome \t 2.432824617228884\n",
            "\t\t encephalitis \t 2.4295234505806587\n",
            "\t\t enterocolitis \t 2.4026285574737845\n",
            "\t\t skin hyperpigmentation \t 2.397138160508922\n",
            "\t\t hypophysitis \t 2.37195072422162\n",
            "\t\t rash macular \t 2.37167932072596\n",
            "\t\t systemic lupus erythematosus \t 2.285332854446194\n",
            "\t\t vitiligo \t 2.2740093783938033\n",
            "\t\t enteritis \t 2.2588994068989323\n",
            "\t\t skin plaque \t 2.1800615703155137\n",
            "\t\t hepatitis acute \t 2.178056352378092\n",
            "\t\t hepatitis e \t 2.1488345812914766\n",
            "\t\t rash erythematous \t 2.1423383864445698\n",
            "\t\t dermatitis allergic \t 2.082993292567891\n",
            "\t\t granuloma \t 2.0721752053635187\n",
            "\t\t lichenoid keratosis \t 2.0183572101158154\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t lichenification \t 1.7748680114746094\n",
            "\t\t lymphocytic hypophysitis \t 1.7747119387033037\n",
            "\t\t granuloma skin \t 1.7687172889709473\n",
            "\t\t pigmentation disorder \t 1.7624768527714194\n",
            "\t\t cutaneous lupus erythematosus \t 1.7258412837982178\n",
            "\t\t granulomatous dermatitis \t 1.675887107849121\n",
            "\t\t rash vesicular \t 1.6666852228021538\n",
            "\t\t lichen striatus \t 1.6217113733291626\n",
            "\t\t encephalitis viral \t 1.6146922915115665\n",
            "\t\t oral mucosal blistering \t 1.553694248199463\n",
            "\t\t mucosa vesicle \t 1.541529893875122\n",
            "\t\t panencephalitis \t 1.5279465913772583\n",
            "\t\t skin abrasion \t 1.4776091575622559\n",
            "\t\t oropharyngeal blistering \t 1.4588536024093628\n",
            "\t\t scratch \t 1.4527802467346191\n",
            "\t\t scab \t 1.4460267713647386\n",
            "\t\t genital blister \t 1.4425745010375977\n",
            "\t\t hypopituitarism \t 1.372517514063937\n",
            "Grounded reactions: \n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t rash \t 4.204907471001762\n",
            "\t\t colitis \t 4.1700437246299\n",
            "\t\t hepatitis \t 4.154471272652927\n",
            "\t\t drug interaction \t 3.693773994980663\n",
            "\t\t rash maculo-papular \t 3.5853229924046217\n",
            "\t\t tubulointerstitial nephritis \t 3.388881160010349\n",
            "\t\t dermatitis \t 3.3220925853245977\n",
            "\t\t blister \t 3.2286903579809643\n",
            "\t\t rash pruritic \t 3.2020435906031124\n",
            "\t\t adverse drug reaction \t 3.118395371754109\n",
            "\t\t stevens-johnson syndrome \t 3.0773251446367045\n",
            "\t\t drug intolerance \t 3.006131832584599\n",
            "\t\t urticaria \t 2.854597489630878\n",
            "\t\t pemphigoid \t 2.8435665110199153\n",
            "\t\t blister \t 2.8422126708542894\n",
            "\t\t nephropathy \t 2.7502505492597344\n",
            "\t\t pemphigoid \t 2.6574421550315304\n",
            "\t\t pemphigoid \t 2.6574421550315304\n",
            "\t\t urticaria \t 2.6430933129376544\n",
            "\t\t erythema multiforme \t 2.6418833873664216\n",
            "\t\t toxic epidermal necrolysis \t 2.637583938251987\n",
            "\t\t nephritis \t 2.5430404724093343\n",
            "\t\t erythema \t 2.505305105450717\n",
            "\t\t rash pruritic \t 2.5046041376251207\n",
            "\t\t dermatitis \t 2.4979494886623175\n",
            "\t\t blister \t 2.4909274889909647\n",
            "\t\t lupus-like syndrome \t 2.432824617228884\n",
            "\t\t encephalitis \t 2.4295234505806587\n",
            "\t\t enterocolitis \t 2.4026285574737845\n",
            "\t\t skin hyperpigmentation \t 2.397138160508922\n",
            "\t\t hypophysitis \t 2.37195072422162\n",
            "\t\t rash macular \t 2.37167932072596\n",
            "\t\t eczema \t 2.365717522964346\n",
            "\t\t dermatitis bullous \t 2.3042321077733923\n",
            "\t\t systemic lupus erythematosus \t 2.285332854446194\n",
            "\t\t vitiligo \t 2.2740093783938033\n",
            "\t\t enteritis \t 2.2588994068989323\n",
            "\t\t skin plaque \t 2.1800615703155137\n",
            "\t\t hepatitis acute \t 2.178056352378092\n",
            "\t\t dermatitis bullous \t 2.171697520578562\n",
            "\t\t hepatitis e \t 2.1488345812914766\n",
            "\t\t eczema \t 2.145388083436522\n",
            "\t\t rash erythematous \t 2.1423383864445698\n",
            "\t\t dermatitis bullous \t 2.1166532847533093\n",
            "\t\t pemphigoid \t 2.099705372639209\n",
            "\t\t dermatitis allergic \t 2.082993292567891\n",
            "\t\t dermatitis bullous \t 2.0747401927193723\n",
            "\t\t granuloma \t 2.0721752053635187\n",
            "\t\t dermatitis bullous \t 2.038606590161184\n",
            "\t\t dermatitis bullous \t 2.038606590161184\n",
            "\t\t lichenoid keratosis \t 2.0183572101158154\n",
            "\t\t pemphigus \t 1.956088514445872\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t pemphigus \t 1.8439212901284932\n",
            "\t\t pemphigus \t 1.8439212901284932\n",
            "\t\t lichenification \t 1.7748680114746094\n",
            "\t\t lymphocytic hypophysitis \t 1.7747119387033037\n",
            "\t\t granuloma skin \t 1.7687172889709473\n",
            "\t\t pigmentation disorder \t 1.7624768527714194\n",
            "\t\t cutaneous lupus erythematosus \t 1.7258412837982178\n",
            "\t\t granulomatous dermatitis \t 1.675887107849121\n",
            "\t\t rash vesicular \t 1.6666852228021538\n",
            "\t\t lichen striatus \t 1.6217113733291626\n",
            "\t\t encephalitis viral \t 1.6146922915115665\n",
            "\t\t oral mucosal blistering \t 1.553694248199463\n",
            "\t\t urticaria papular \t 1.5508558750152588\n",
            "\t\t mucosa vesicle \t 1.541529893875122\n",
            "\t\t panencephalitis \t 1.5279465913772583\n",
            "\t\t idiopathic urticaria \t 1.4902148246765137\n",
            "\t\t skin abrasion \t 1.4776091575622559\n",
            "\t\t urticaria papular \t 1.47298264503479\n",
            "\t\t oropharyngeal blistering \t 1.4588536024093628\n",
            "\t\t scratch \t 1.4527802467346191\n",
            "\t\t scab \t 1.4460267713647386\n",
            "\t\t genital blister \t 1.4425745010375977\n",
            "\t\t idiopathic urticaria \t 1.38498055934906\n",
            "\t\t hypopituitarism \t 1.372517514063937\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['cystoid macular oedema', 'off label use', 'retinal degeneration', 'retinal disorder', 'retinal toxicity', 'retinopathy', 'vision blurred', 'visual acuity reduced', 'visual field defect']\n",
            "Predicted reactions: \n",
            "\t ['cystoid macular oedema', 'no adverse event', 'macular oedema', 'adverse drug reaction', 'vision blurred', 'visual field defect', 'maculopathy', 'drug ineffective', 'drug interaction', 'visual impairment', 'visual acuity reduced', 'headache', 'foveal reflex abnormal', 'scintillating scotoma', 'no reaction on previous exposure to drug', 'macular reflex abnormal', 'hemianopia heteronymous', 'uncircumcised']\n",
            "Ungrounded reactions: \n",
            "\t ['cystoid macular oedema', 'no adverse event', 'macular oedema', 'adverse drug reaction', 'vision blurred', 'visual field defect', 'maculopathy', 'drug ineffective', 'drug interaction', 'visual impairment', 'visual acuity reduced', 'headache', 'foveal reflex abnormal', 'scintillating scotoma', 'no reaction on previous exposure to drug', 'macular reflex abnormal', 'hemianopia heteronymous', 'uncircumcised']\n",
            "Recall@10:  0.3333333333333333\n",
            "Recall@20:  0.4444444444444444\n",
            "Recall@30:  0.4444444444444444\n",
            "Resolved reactions: \n",
            "\t\t cystoid macular oedema \t 6.012217366762455\n",
            "\t\t no adverse event \t 5.260651337453635\n",
            "\t\t macular oedema \t 4.872438666929911\n",
            "\t\t adverse drug reaction \t 4.494352222459574\n",
            "\t\t vision blurred \t 4.059771221140152\n",
            "\t\t visual field defect \t 4.020122451090205\n",
            "\t\t maculopathy \t 4.012976047123599\n",
            "\t\t drug ineffective \t 3.9798800178096756\n",
            "\t\t drug interaction \t 3.5600697628287725\n",
            "\t\t visual impairment \t 3.0149204563433614\n",
            "\t\t visual acuity reduced \t 2.526790467291728\n",
            "\t\t headache \t 1.649351795645265\n",
            "\t\t foveal reflex abnormal \t 1.5658787488937378\n",
            "\t\t scintillating scotoma \t 1.505533218383789\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t macular reflex abnormal \t 1.2940754890441895\n",
            "\t\t hemianopia heteronymous \t 1.2239603996276855\n",
            "\t\t uncircumcised \t 0.8278310298919678\n",
            "Grounded reactions: \n",
            "\t\t vision blurred \t 4.059771221140152\n",
            "\t\t drug ineffective \t 3.9798800178096756\n",
            "\t\t drug interaction \t 3.5600697628287725\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t cystoid macular oedema \t 3.175218610831104\n",
            "\t\t visual impairment \t 3.0149204563433614\n",
            "\t\t cystoid macular oedema \t 2.8369987559313503\n",
            "\t\t adverse drug reaction \t 2.723724239272734\n",
            "\t\t macular oedema \t 2.632656177869577\n",
            "\t\t visual acuity reduced \t 2.526790467291728\n",
            "\t\t macular oedema \t 2.239782489060334\n",
            "\t\t visual field defect \t 2.1743059943460525\n",
            "\t\t maculopathy \t 2.1410644528951344\n",
            "\t\t maculopathy \t 1.871911594228465\n",
            "\t\t visual field defect \t 1.8458164567441526\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t no adverse event \t 1.7460851784173068\n",
            "\t\t headache \t 1.649351795645265\n",
            "\t\t foveal reflex abnormal \t 1.5658787488937378\n",
            "\t\t scintillating scotoma \t 1.505533218383789\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t macular reflex abnormal \t 1.2940754890441895\n",
            "\t\t hemianopia heteronymous \t 1.2239603996276855\n",
            "\t\t uncircumcised \t 0.8278310298919678\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['immune-mediated uveitis', 'optic neuritis']\n",
            "Predicted reactions: \n",
            "\t ['uveitis', 'papilloedema', 'papillitis', 'iridocyclitis', 'optic neuritis', 'vision blurred', 'eye pain', 'endophthalmitis', 'visual impairment', 'pain', 'visual acuity reduced', 'facial pain', 'adhesion', 'symblepharon', 'conjunctival adhesion', 'anterior chamber inflammation']\n",
            "Ungrounded reactions: \n",
            "\t ['uveitis', 'papilloedema', 'papillitis', 'iridocyclitis', 'optic neuritis', 'vision blurred', 'eye pain', 'endophthalmitis', 'visual impairment', 'pain', 'visual acuity reduced', 'facial pain', 'adhesion', 'symblepharon', 'conjunctival adhesion', 'anterior chamber inflammation']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t uveitis \t 6.770867274716891\n",
            "\t\t papilloedema \t 6.169516585414692\n",
            "\t\t papillitis \t 5.724093675613403\n",
            "\t\t iridocyclitis \t 5.045782556539239\n",
            "\t\t optic neuritis \t 4.741397621239144\n",
            "\t\t vision blurred \t 4.059771221140152\n",
            "\t\t eye pain \t 3.496388934571243\n",
            "\t\t endophthalmitis \t 3.282711160549524\n",
            "\t\t visual impairment \t 3.0149204563433614\n",
            "\t\t pain \t 2.9825023426487447\n",
            "\t\t visual acuity reduced \t 2.526790467291728\n",
            "\t\t facial pain \t 1.7753881631393884\n",
            "\t\t adhesion \t 1.3597692410092057\n",
            "\t\t symblepharon \t 1.3321807384490967\n",
            "\t\t conjunctival adhesion \t 1.31605863571167\n",
            "\t\t anterior chamber inflammation \t 1.0668048858642578\n",
            "Grounded reactions: \n",
            "\t\t vision blurred \t 4.059771221140152\n",
            "\t\t eye pain \t 3.496388934571243\n",
            "\t\t visual impairment \t 3.0149204563433614\n",
            "\t\t pain \t 2.9825023426487447\n",
            "\t\t visual acuity reduced \t 2.526790467291728\n",
            "\t\t uveitis \t 2.4295279387213173\n",
            "\t\t uveitis \t 2.4295279387213173\n",
            "\t\t papilloedema \t 2.1786173557236075\n",
            "\t\t papilloedema \t 2.1786173557236075\n",
            "\t\t papillitis \t 2.0\n",
            "\t\t papillitis \t 2.0\n",
            "\t\t uveitis \t 1.911811397274256\n",
            "\t\t papilloedema \t 1.8122818739674769\n",
            "\t\t iridocyclitis \t 1.7833917752052955\n",
            "\t\t iridocyclitis \t 1.7833917752052955\n",
            "\t\t facial pain \t 1.7753881631393884\n",
            "\t\t papillitis \t 1.7240936756134033\n",
            "\t\t optic neuritis \t 1.6586027192931199\n",
            "\t\t optic neuritis \t 1.6586027192931199\n",
            "\t\t endophthalmitis \t 1.641355580274762\n",
            "\t\t endophthalmitis \t 1.641355580274762\n",
            "\t\t iridocyclitis \t 1.4789990061286473\n",
            "\t\t optic neuritis \t 1.4241921826529043\n",
            "\t\t adhesion \t 1.3597692410092057\n",
            "\t\t symblepharon \t 1.3321807384490967\n",
            "\t\t conjunctival adhesion \t 1.31605863571167\n",
            "\t\t anterior chamber inflammation \t 1.0668048858642578\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['acute respiratory distress syndrome', 'intentional overdose', 'intentional self-injury', 'myalgia', 'non-cardiogenic pulmonary oedema', 'pulmonary oedema', 'renal failure', 'suicide attempt', 'vomiting']\n",
            "Predicted reactions: \n",
            "\t ['pulmonary oedema', 'acute pulmonary oedema', 'non-cardiogenic pulmonary oedema', 'hypoxia', 'oxygen saturation decreased', 'respiratory alkalosis', 'respiratory acidosis', 'metabolic alkalosis', 'anaemic hypoxia']\n",
            "Ungrounded reactions: \n",
            "\t ['pulmonary oedema', 'acute pulmonary oedema', 'non-cardiogenic pulmonary oedema', 'hypoxia', 'oxygen saturation decreased', 'respiratory alkalosis', 'respiratory acidosis', 'metabolic alkalosis', 'anaemic hypoxia']\n",
            "Recall@10:  0.2222222222222222\n",
            "Recall@20:  0.2222222222222222\n",
            "Recall@30:  0.2222222222222222\n",
            "Resolved reactions: \n",
            "\t\t pulmonary oedema \t 19.251443038435667\n",
            "\t\t acute pulmonary oedema \t 11.146192391779188\n",
            "\t\t non-cardiogenic pulmonary oedema \t 11.100133637219297\n",
            "\t\t hypoxia \t 4.670192482229418\n",
            "\t\t oxygen saturation decreased \t 3.158836194448269\n",
            "\t\t respiratory alkalosis \t 2.981546217004304\n",
            "\t\t respiratory acidosis \t 2.6636989009666303\n",
            "\t\t metabolic alkalosis \t 2.09414557787063\n",
            "\t\t anaemic hypoxia \t 1.765486717224121\n",
            "Grounded reactions: \n",
            "\t\t hypoxia \t 4.670192482229418\n",
            "\t\t pulmonary oedema \t 3.8502886076871334\n",
            "\t\t pulmonary oedema \t 3.8502886076871334\n",
            "\t\t pulmonary oedema \t 3.8502886076871334\n",
            "\t\t pulmonary oedema \t 3.8502886076871334\n",
            "\t\t pulmonary oedema \t 3.8502886076871334\n",
            "\t\t oxygen saturation decreased \t 3.158836194448269\n",
            "\t\t respiratory alkalosis \t 2.981546217004304\n",
            "\t\t respiratory acidosis \t 2.6636989009666303\n",
            "\t\t acute pulmonary oedema \t 2.2292384783558377\n",
            "\t\t acute pulmonary oedema \t 2.2292384783558377\n",
            "\t\t acute pulmonary oedema \t 2.2292384783558377\n",
            "\t\t acute pulmonary oedema \t 2.2292384783558377\n",
            "\t\t acute pulmonary oedema \t 2.2292384783558377\n",
            "\t\t non-cardiogenic pulmonary oedema \t 2.2200267274438596\n",
            "\t\t non-cardiogenic pulmonary oedema \t 2.2200267274438596\n",
            "\t\t non-cardiogenic pulmonary oedema \t 2.2200267274438596\n",
            "\t\t non-cardiogenic pulmonary oedema \t 2.2200267274438596\n",
            "\t\t non-cardiogenic pulmonary oedema \t 2.2200267274438596\n",
            "\t\t metabolic alkalosis \t 2.09414557787063\n",
            "\t\t anaemic hypoxia \t 1.765486717224121\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['drug ineffective', 'intraocular pressure increased', 'squamous cell carcinoma of lung']\n",
            "Predicted reactions: \n",
            "\t ['visual impairment', 'drug ineffective', 'no adverse event', 'visual acuity reduced', 'blindness', 'intraocular pressure increased', 'adverse drug reaction', 'retinopathy', 'ocular hypertension', 'ocular hyperaemia', 'subretinal fluid', 'retinal ischaemia', 'vision correction operation', 'delayed visual maturation', 'myoclonic epilepsy', 'mitochondrial encephalomyopathy', 'mitochondrial myopathy']\n",
            "Ungrounded reactions: \n",
            "\t ['visual impairment', 'drug ineffective', 'no adverse event', 'visual acuity reduced', 'blindness', 'intraocular pressure increased', 'adverse drug reaction', 'retinopathy', 'ocular hypertension', 'ocular hyperaemia', 'subretinal fluid', 'retinal ischaemia', 'vision correction operation', 'delayed visual maturation', 'myoclonic epilepsy', 'mitochondrial encephalomyopathy', 'mitochondrial myopathy']\n",
            "Recall@10:  0.6666666666666666\n",
            "Recall@20:  0.6666666666666666\n",
            "Recall@30:  0.6666666666666666\n",
            "Resolved reactions: \n",
            "\t\t visual impairment \t 5.610972338987952\n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t no adverse event \t 3.3319030318452447\n",
            "\t\t visual acuity reduced \t 3.2916196417967654\n",
            "\t\t blindness \t 3.1118307265245253\n",
            "\t\t intraocular pressure increased \t 2.421795224630516\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t retinopathy \t 1.666353689143476\n",
            "\t\t ocular hypertension \t 1.6419931268527055\n",
            "\t\t ocular hyperaemia \t 1.4366058045916066\n",
            "\t\t subretinal fluid \t 1.3844163354702297\n",
            "\t\t retinal ischaemia \t 1.313911739103495\n",
            "\t\t vision correction operation \t 1.2588937282562256\n",
            "\t\t delayed visual maturation \t 1.0914995670318604\n",
            "\t\t myoclonic epilepsy \t 0.8560168743133545\n",
            "\t\t mitochondrial encephalomyopathy \t 0.8171674013137817\n",
            "\t\t mitochondrial myopathy \t 0.8070541620254517\n",
            "Grounded reactions: \n",
            "\t\t drug ineffective \t 3.688348500007609\n",
            "\t\t visual impairment \t 3.657488256713437\n",
            "\t\t no adverse event \t 3.3319030318452447\n",
            "\t\t visual acuity reduced \t 3.2916196417967654\n",
            "\t\t blindness \t 3.1118307265245253\n",
            "\t\t intraocular pressure increased \t 2.421795224630516\n",
            "\t\t adverse drug reaction \t 2.0437216411490926\n",
            "\t\t visual impairment \t 1.953484082274515\n",
            "\t\t retinopathy \t 1.666353689143476\n",
            "\t\t ocular hypertension \t 1.6419931268527055\n",
            "\t\t ocular hyperaemia \t 1.4366058045916066\n",
            "\t\t subretinal fluid \t 1.3844163354702297\n",
            "\t\t retinal ischaemia \t 1.313911739103495\n",
            "\t\t vision correction operation \t 1.2588937282562256\n",
            "\t\t delayed visual maturation \t 1.0914995670318604\n",
            "\t\t myoclonic epilepsy \t 0.8560168743133545\n",
            "\t\t mitochondrial encephalomyopathy \t 0.8171674013137817\n",
            "\t\t mitochondrial myopathy \t 0.8070541620254517\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['deep vein thrombosis', 'drug dependence', 'embolism venous', 'paradoxical embolism', 'product use in unapproved indication', 'pulmonary embolism', 'sedation', 'weight increased']\n",
            "Predicted reactions: \n",
            "\t ['deep vein thrombosis', 'venous thrombosis', 'thrombosis', 'pulmonary embolism', 'embolism venous', 'embolism', 'ischaemic stroke', 'cerebrovascular accident', 'cerebral infarction', 'post embolisation syndrome', 'paradoxical embolism', 'pseudostroke']\n",
            "Ungrounded reactions: \n",
            "\t ['deep vein thrombosis', 'venous thrombosis', 'thrombosis', 'pulmonary embolism', 'embolism venous', 'embolism', 'ischaemic stroke', 'cerebrovascular accident', 'cerebral infarction', 'post embolisation syndrome', 'paradoxical embolism', 'pseudostroke']\n",
            "Recall@10:  0.375\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t deep vein thrombosis \t 26.342892768783422\n",
            "\t\t venous thrombosis \t 19.19339575596106\n",
            "\t\t thrombosis \t 16.00721755882066\n",
            "\t\t pulmonary embolism \t 8.78525698157041\n",
            "\t\t embolism venous \t 7.027213673740417\n",
            "\t\t embolism \t 4.7861297353498315\n",
            "\t\t ischaemic stroke \t 3.3804260396201307\n",
            "\t\t cerebrovascular accident \t 3.325813330977923\n",
            "\t\t cerebral infarction \t 2.9480826971849337\n",
            "\t\t post embolisation syndrome \t 1.456144094467163\n",
            "\t\t paradoxical embolism \t 1.362370491027832\n",
            "\t\t pseudostroke \t 1.2106891870498657\n",
            "Grounded reactions: \n",
            "\t\t pulmonary embolism \t 4.392628490785205\n",
            "\t\t pulmonary embolism \t 4.392628490785205\n",
            "\t\t deep vein thrombosis \t 4.251239003753496\n",
            "\t\t deep vein thrombosis \t 4.251239003753496\n",
            "\t\t deep vein thrombosis \t 4.251239003753496\n",
            "\t\t deep vein thrombosis \t 3.4618197113115814\n",
            "\t\t ischaemic stroke \t 3.3804260396201307\n",
            "\t\t deep vein thrombosis \t 3.375785348737116\n",
            "\t\t deep vein thrombosis \t 3.375785348737116\n",
            "\t\t deep vein thrombosis \t 3.375785348737116\n",
            "\t\t cerebrovascular accident \t 3.325813330977923\n",
            "\t\t cerebral infarction \t 2.9480826971849337\n",
            "\t\t embolism venous \t 2.8459342787028072\n",
            "\t\t venous thrombosis \t 2.7935935724810883\n",
            "\t\t venous thrombosis \t 2.7935935724810883\n",
            "\t\t venous thrombosis \t 2.7935935724810883\n",
            "\t\t venous thrombosis \t 2.764274478305286\n",
            "\t\t thrombosis \t 2.6984993861132485\n",
            "\t\t thrombosis \t 2.6984993861132485\n",
            "\t\t thrombosis \t 2.6984993861132485\n",
            "\t\t venous thrombosis \t 2.682780186737502\n",
            "\t\t venous thrombosis \t 2.682780186737502\n",
            "\t\t venous thrombosis \t 2.682780186737502\n",
            "\t\t thrombosis \t 2.637239800160305\n",
            "\t\t thrombosis \t 2.637239800160305\n",
            "\t\t thrombosis \t 2.637239800160305\n",
            "\t\t embolism \t 2.3930648676749158\n",
            "\t\t embolism \t 2.3930648676749158\n",
            "\t\t embolism venous \t 2.090639697518805\n",
            "\t\t embolism venous \t 2.090639697518805\n",
            "\t\t post embolisation syndrome \t 1.456144094467163\n",
            "\t\t paradoxical embolism \t 1.362370491027832\n",
            "\t\t pseudostroke \t 1.2106891870498657\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['diarrhoea', 'peripheral swelling']\n",
            "Predicted reactions: \n",
            "\t ['pyrexia', 'leukocytosis', 'diarrhoea', 'hyperthermia', 'body temperature increased', 'hypotension', 'abdominal pain', 'tachycardia', 'dyspnoea', 'neutrophilia', 'constipation', 'neutrophil count increased', 'tachypnoea', 'drug interaction', 'pathogen resistance', 'ileus', 'blood pressure decreased', 'oedema peripheral', 'abdominal pain upper', 'adverse drug reaction', 'heart rate increased', 'white blood cell count increased', 'drug intolerance', 'abdominal discomfort', 'dyspnoea exertional', 'intestinal obstruction', 'peripheral swelling', 'ileus paralytic', 'drug resistance', 'respiratory rate increased', 'swelling', 'diarrhoea haemorrhagic', 'anastomotic leak', 'gastrointestinal anastomotic leak', 'heart rate decreased', 'lymphocytosis', 'multiple-drug resistance', 'blood pressure systolic decreased', 'orthopnoea', 'defaecation urgency', 'hyperventilation', 'post procedural diarrhoea', 'reproductive tract anastomotic leak', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['pyrexia', 'leukocytosis', 'diarrhoea', 'hyperthermia', 'body temperature increased', 'hypotension', 'abdominal pain', 'tachycardia', 'dyspnoea', 'neutrophilia', 'constipation', 'neutrophil count increased', 'tachypnoea', 'drug interaction', 'pathogen resistance', 'ileus', 'blood pressure decreased', 'oedema peripheral', 'abdominal pain upper', 'adverse drug reaction', 'heart rate increased', 'white blood cell count increased', 'drug intolerance', 'abdominal discomfort', 'dyspnoea exertional', 'intestinal obstruction', 'peripheral swelling', 'ileus paralytic', 'drug resistance', 'respiratory rate increased', 'swelling', 'diarrhoea haemorrhagic', 'anastomotic leak', 'gastrointestinal anastomotic leak', 'heart rate decreased', 'lymphocytosis', 'multiple-drug resistance', 'blood pressure systolic decreased', 'orthopnoea', 'defaecation urgency', 'hyperventilation', 'post procedural diarrhoea', 'reproductive tract anastomotic leak', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  0.5\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t pyrexia \t 16.928646666392435\n",
            "\t\t leukocytosis \t 11.389371838643845\n",
            "\t\t diarrhoea \t 10.441174895140392\n",
            "\t\t hyperthermia \t 7.12888509866385\n",
            "\t\t body temperature increased \t 7.008995559689889\n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t tachycardia \t 5.134140508754524\n",
            "\t\t dyspnoea \t 4.881158693892149\n",
            "\t\t neutrophilia \t 4.7439017312018175\n",
            "\t\t constipation \t 4.742479399651104\n",
            "\t\t neutrophil count increased \t 4.220789009964791\n",
            "\t\t tachypnoea \t 3.975027796114034\n",
            "\t\t drug interaction \t 3.693773994980663\n",
            "\t\t pathogen resistance \t 3.680751416819575\n",
            "\t\t ileus \t 3.4568940526668106\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t oedema peripheral \t 3.3827222679789863\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t adverse drug reaction \t 3.118395371754109\n",
            "\t\t heart rate increased \t 3.092190087658065\n",
            "\t\t white blood cell count increased \t 3.0126779085642634\n",
            "\t\t drug intolerance \t 3.006131832584599\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t dyspnoea exertional \t 2.9540996054856214\n",
            "\t\t intestinal obstruction \t 2.7762387466280543\n",
            "\t\t peripheral swelling \t 2.69208414564793\n",
            "\t\t ileus paralytic \t 2.6711384991293654\n",
            "\t\t drug resistance \t 2.6352631971103184\n",
            "\t\t respiratory rate increased \t 2.290193035087287\n",
            "\t\t swelling \t 2.2072042909667218\n",
            "\t\t diarrhoea haemorrhagic \t 2.1791557636761265\n",
            "\t\t anastomotic leak \t 2.000000238418579\n",
            "\t\t gastrointestinal anastomotic leak \t 1.935065507888794\n",
            "\t\t heart rate decreased \t 1.9178425693854937\n",
            "\t\t lymphocytosis \t 1.9146116326726614\n",
            "\t\t multiple-drug resistance \t 1.9101625464813412\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t orthopnoea \t 1.8224309663889289\n",
            "\t\t defaecation urgency \t 1.6869020462036133\n",
            "\t\t hyperventilation \t 1.595197319984436\n",
            "\t\t post procedural diarrhoea \t 1.573487639427185\n",
            "\t\t reproductive tract anastomotic leak \t 1.5503830909729004\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t pyrexia \t 5.671394872002146\n",
            "\t\t pyrexia \t 5.645121086529286\n",
            "\t\t diarrhoea \t 5.636904802170626\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t tachycardia \t 5.134140508754524\n",
            "\t\t dyspnoea \t 4.881158693892149\n",
            "\t\t diarrhoea \t 4.804270092969767\n",
            "\t\t constipation \t 4.742479399651104\n",
            "\t\t leukocytosis \t 4.3659599569849\n",
            "\t\t tachypnoea \t 3.975027796114034\n",
            "\t\t drug interaction \t 3.693773994980663\n",
            "\t\t pathogen resistance \t 3.680751416819575\n",
            "\t\t leukocytosis \t 3.5117059408294726\n",
            "\t\t leukocytosis \t 3.5117059408294726\n",
            "\t\t ileus \t 3.4568940526668106\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t oedema peripheral \t 3.3827222679789863\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t adverse drug reaction \t 3.118395371754109\n",
            "\t\t heart rate increased \t 3.092190087658065\n",
            "\t\t white blood cell count increased \t 3.0126779085642634\n",
            "\t\t drug intolerance \t 3.006131832584599\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t dyspnoea exertional \t 2.9540996054856214\n",
            "\t\t intestinal obstruction \t 2.7762387466280543\n",
            "\t\t peripheral swelling \t 2.69208414564793\n",
            "\t\t ileus paralytic \t 2.6711384991293654\n",
            "\t\t drug resistance \t 2.6352631971103184\n",
            "\t\t hyperthermia \t 2.580580241404274\n",
            "\t\t body temperature increased \t 2.5111109238791993\n",
            "\t\t neutrophilia \t 2.3719508656009087\n",
            "\t\t neutrophilia \t 2.3719508656009087\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t respiratory rate increased \t 2.290193035087287\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t body temperature increased \t 2.2236197844861563\n",
            "\t\t hyperthermia \t 2.211369981026393\n",
            "\t\t swelling \t 2.2072042909667218\n",
            "\t\t diarrhoea haemorrhagic \t 2.1791557636761265\n",
            "\t\t neutrophil count increased \t 2.1103945049823953\n",
            "\t\t neutrophil count increased \t 2.1103945049823953\n",
            "\t\t anastomotic leak \t 2.000000238418579\n",
            "\t\t gastrointestinal anastomotic leak \t 1.935065507888794\n",
            "\t\t heart rate decreased \t 1.9178425693854937\n",
            "\t\t lymphocytosis \t 1.9146116326726614\n",
            "\t\t multiple-drug resistance \t 1.9101625464813412\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t orthopnoea \t 1.8224309663889289\n",
            "\t\t defaecation urgency \t 1.6869020462036133\n",
            "\t\t hyperventilation \t 1.595197319984436\n",
            "\t\t post procedural diarrhoea \t 1.573487639427185\n",
            "\t\t reproductive tract anastomotic leak \t 1.5503830909729004\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['extravasation', 'pleural effusion', 'pleuritic pain']\n",
            "Predicted reactions: \n",
            "\t ['pleural effusion', 'pleurisy', 'chest pain', 'pyrexia', 'pneumonia', 'pain', 'leukocytosis', 'pneumonitis', 'pericardial effusion', 'necrosis', 'pleuritic pain', 'headache', 'infectious pleural effusion', 'white blood cell count increased', 'disease progression', 'disease recurrence', 'traumatic lung injury', 'tenderness', 'pericarditis', 'organising pneumonia', 'hyperthermia', 'pulmonary toxicity', 'body temperature increased', 'acute lung injury', 'cell death', 'lymphocytosis', 'injury', 'rebound effect', 'foetal death', 'pericardial drainage', 'tissue injury', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['pleural effusion', 'pleurisy', 'chest pain', 'pyrexia', 'pneumonia', 'pain', 'leukocytosis', 'pneumonitis', 'pericardial effusion', 'necrosis', 'pleuritic pain', 'headache', 'infectious pleural effusion', 'white blood cell count increased', 'disease progression', 'disease recurrence', 'traumatic lung injury', 'tenderness', 'pericarditis', 'organising pneumonia', 'hyperthermia', 'pulmonary toxicity', 'body temperature increased', 'acute lung injury', 'cell death', 'lymphocytosis', 'injury', 'rebound effect', 'foetal death', 'pericardial drainage', 'tissue injury', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.3333333333333333\n",
            "Recall@20:  0.6666666666666666\n",
            "Recall@30:  0.6666666666666666\n",
            "Resolved reactions: \n",
            "\t\t pleural effusion \t 10.26827979343471\n",
            "\t\t pleurisy \t 6.667043378322933\n",
            "\t\t chest pain \t 5.872614928153152\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t pneumonia \t 5.076254120286797\n",
            "\t\t pain \t 4.679520433152032\n",
            "\t\t leukocytosis \t 4.3659599569849\n",
            "\t\t pneumonitis \t 4.244462576464894\n",
            "\t\t pericardial effusion \t 4.106244591302996\n",
            "\t\t necrosis \t 3.9152794599142258\n",
            "\t\t pleuritic pain \t 3.8913245487370394\n",
            "\t\t headache \t 3.8610861348756327\n",
            "\t\t infectious pleural effusion \t 3.6982584703600416\n",
            "\t\t white blood cell count increased \t 3.0126779085642634\n",
            "\t\t disease progression \t 2.735777131181288\n",
            "\t\t disease recurrence \t 2.627789443334814\n",
            "\t\t traumatic lung injury \t 2.618433609166826\n",
            "\t\t tenderness \t 2.5640926425656185\n",
            "\t\t pericarditis \t 2.4620497518859654\n",
            "\t\t organising pneumonia \t 2.3673436655944413\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t pulmonary toxicity \t 2.326712230612549\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t acute lung injury \t 2.0945176138752726\n",
            "\t\t cell death \t 2.0\n",
            "\t\t lymphocytosis \t 1.9146116326726614\n",
            "\t\t injury \t 1.8680367291377304\n",
            "\t\t rebound effect \t 1.8152606908137703\n",
            "\t\t foetal death \t 1.7023365411954088\n",
            "\t\t pericardial drainage \t 1.6022896766662598\n",
            "\t\t tissue injury \t 1.52091383934021\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t pleural effusion \t 5.134139896717355\n",
            "\t\t pleural effusion \t 5.134139896717355\n",
            "\t\t pneumonia \t 5.076254120286797\n",
            "\t\t pain \t 4.679520433152032\n",
            "\t\t leukocytosis \t 4.3659599569849\n",
            "\t\t pneumonitis \t 4.244462576464894\n",
            "\t\t pericardial effusion \t 4.106244591302996\n",
            "\t\t headache \t 3.8610861348756327\n",
            "\t\t white blood cell count increased \t 3.0126779085642634\n",
            "\t\t chest pain \t 2.936307464076576\n",
            "\t\t chest pain \t 2.936307464076576\n",
            "\t\t disease progression \t 2.735777131181288\n",
            "\t\t disease recurrence \t 2.627789443334814\n",
            "\t\t traumatic lung injury \t 2.618433609166826\n",
            "\t\t tenderness \t 2.5640926425656185\n",
            "\t\t pericarditis \t 2.4620497518859654\n",
            "\t\t organising pneumonia \t 2.3673436655944413\n",
            "\t\t necrosis \t 2.351741799829748\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t pulmonary toxicity \t 2.326712230612549\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t acute lung injury \t 2.0945176138752726\n",
            "\t\t cell death \t 2.0\n",
            "\t\t pleuritic pain \t 1.9456622743685197\n",
            "\t\t pleuritic pain \t 1.9456622743685197\n",
            "\t\t lymphocytosis \t 1.9146116326726614\n",
            "\t\t injury \t 1.8680367291377304\n",
            "\t\t infectious pleural effusion \t 1.8491292351800208\n",
            "\t\t infectious pleural effusion \t 1.8491292351800208\n",
            "\t\t rebound effect \t 1.8152606908137703\n",
            "\t\t pleurisy \t 1.745601793131002\n",
            "\t\t pleurisy \t 1.745601793131002\n",
            "\t\t foetal death \t 1.7023365411954088\n",
            "\t\t pericardial drainage \t 1.6022896766662598\n",
            "\t\t pleurisy \t 1.5879198960304641\n",
            "\t\t pleurisy \t 1.5879198960304641\n",
            "\t\t necrosis \t 1.563537660084478\n",
            "\t\t tissue injury \t 1.52091383934021\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['condition aggravated', 'drug ineffective']\n",
            "Predicted reactions: \n",
            "\t ['gastrointestinal haemorrhage', 'haematochezia', 'nausea', 'vomiting', 'shock', 'circulatory collapse', 'syncope', 'diarrhoea', 'death', 'haematemesis', 'rash', 'upper gastrointestinal haemorrhage', 'gingival bleeding', 'cerebral haemorrhage', 'epistaxis', 'haematuria', 'constipation', 'haemorrhage intracranial', 'vomiting projectile', 'rash erythematous', 'coagulopathy', 'rash maculo-papular', 'respiratory depression', 'melaena', 'vaginal haemorrhage', 'conjunctival haemorrhage', 'sudden death', 'cardiovascular insufficiency', 'clinical death', 'mouth haemorrhage', 'central nervous system haemorrhage', 'hypoventilation', 'hypopnoea', 'eye haemorrhage', 'diarrhoea infectious', 'menorrhagia', 'blood urine present', 'disseminated intravascular coagulation', 'metrorrhagia', 'haemorrhage urinary tract', 'hypercoagulation', 'tooth socket haemorrhage', 'hypocoagulable state', 'conjunctival disorder', 'respiratory tract haemorrhage', 'ischaemia', 'coronary artery dissection', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['gastrointestinal haemorrhage', 'haematochezia', 'nausea', 'vomiting', 'shock', 'circulatory collapse', 'syncope', 'diarrhoea', 'death', 'haematemesis', 'rash', 'upper gastrointestinal haemorrhage', 'gingival bleeding', 'cerebral haemorrhage', 'epistaxis', 'haematuria', 'constipation', 'haemorrhage intracranial', 'vomiting projectile', 'rash erythematous', 'coagulopathy', 'rash maculo-papular', 'respiratory depression', 'melaena', 'vaginal haemorrhage', 'conjunctival haemorrhage', 'sudden death', 'cardiovascular insufficiency', 'clinical death', 'mouth haemorrhage', 'central nervous system haemorrhage', 'hypoventilation', 'hypopnoea', 'eye haemorrhage', 'diarrhoea infectious', 'menorrhagia', 'blood urine present', 'disseminated intravascular coagulation', 'metrorrhagia', 'haemorrhage urinary tract', 'hypercoagulation', 'tooth socket haemorrhage', 'hypocoagulable state', 'conjunctival disorder', 'respiratory tract haemorrhage', 'ischaemia', 'coronary artery dissection', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t gastrointestinal haemorrhage \t 17.96600882341855\n",
            "\t\t haematochezia \t 14.465320033339413\n",
            "\t\t nausea \t 11.724094031085794\n",
            "\t\t vomiting \t 11.278298153600627\n",
            "\t\t shock \t 11.146574155365427\n",
            "\t\t circulatory collapse \t 10.284375439431386\n",
            "\t\t syncope \t 6.396524969140847\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t death \t 5.647981780445181\n",
            "\t\t haematemesis \t 5.396397989287135\n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t upper gastrointestinal haemorrhage \t 5.01226321751287\n",
            "\t\t gingival bleeding \t 4.531700932815455\n",
            "\t\t cerebral haemorrhage \t 4.057717313499216\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t haematuria \t 3.861773156608939\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t haemorrhage intracranial \t 3.382156994880847\n",
            "\t\t vomiting projectile \t 3.332826614379883\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t coagulopathy \t 2.9748052625553356\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t respiratory depression \t 2.7784895937545877\n",
            "\t\t melaena \t 2.7292993808310504\n",
            "\t\t vaginal haemorrhage \t 2.577792132806105\n",
            "\t\t conjunctival haemorrhage \t 2.5406455554300176\n",
            "\t\t sudden death \t 2.329618247112669\n",
            "\t\t cardiovascular insufficiency \t 2.0804061174359703\n",
            "\t\t clinical death \t 1.8975193500518799\n",
            "\t\t mouth haemorrhage \t 1.8688815754692356\n",
            "\t\t central nervous system haemorrhage \t 1.8605918884277344\n",
            "\t\t hypoventilation \t 1.7408309078419075\n",
            "\t\t hypopnoea \t 1.715468681305503\n",
            "\t\t eye haemorrhage \t 1.6857184186032474\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t menorrhagia \t 1.6673335354861751\n",
            "\t\t blood urine present \t 1.6461281776428223\n",
            "\t\t disseminated intravascular coagulation \t 1.636710582077133\n",
            "\t\t metrorrhagia \t 1.6150825023651123\n",
            "\t\t haemorrhage urinary tract \t 1.5763455629348755\n",
            "\t\t hypercoagulation \t 1.5386027353209348\n",
            "\t\t tooth socket haemorrhage \t 1.4860259294509888\n",
            "\t\t hypocoagulable state \t 1.4660186767578125\n",
            "\t\t conjunctival disorder \t 1.4618439674377441\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t ischaemia \t 0.9710260176816675\n",
            "\t\t coronary artery dissection \t 0.8747293328881569\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t death \t 5.647981780445181\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t gastrointestinal haemorrhage \t 4.333961769638751\n",
            "\t\t gastrointestinal haemorrhage \t 4.333961769638751\n",
            "\t\t cerebral haemorrhage \t 4.057717313499216\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t haematuria \t 3.861773156608939\n",
            "\t\t shock \t 3.8060674523458577\n",
            "\t\t shock \t 3.8060674523458577\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t shock \t 3.5344392506737115\n",
            "\t\t circulatory collapse \t 3.5180397583734684\n",
            "\t\t circulatory collapse \t 3.5180397583734684\n",
            "\t\t haemorrhage intracranial \t 3.382156994880847\n",
            "\t\t circulatory collapse \t 3.24829592268445\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t gastrointestinal haemorrhage \t 3.2014155323117746\n",
            "\t\t gastrointestinal haemorrhage \t 3.2014155323117746\n",
            "\t\t syncope \t 3.1982624845704235\n",
            "\t\t syncope \t 3.1982624845704235\n",
            "\t\t haematochezia \t 3.1368045069882324\n",
            "\t\t haematochezia \t 3.1368045069882324\n",
            "\t\t coagulopathy \t 2.9748052625553356\n",
            "\t\t gastrointestinal haemorrhage \t 2.895254219517501\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t gingival bleeding \t 2.815209896185384\n",
            "\t\t respiratory depression \t 2.7784895937545877\n",
            "\t\t haematochezia \t 2.744231475579997\n",
            "\t\t haematochezia \t 2.744231475579997\n",
            "\t\t melaena \t 2.7292993808310504\n",
            "\t\t haematochezia \t 2.703248068202953\n",
            "\t\t haematemesis \t 2.6981989946435676\n",
            "\t\t haematemesis \t 2.6981989946435676\n",
            "\t\t vaginal haemorrhage \t 2.577792132806105\n",
            "\t\t conjunctival haemorrhage \t 2.5406455554300176\n",
            "\t\t upper gastrointestinal haemorrhage \t 2.506131608756435\n",
            "\t\t upper gastrointestinal haemorrhage \t 2.506131608756435\n",
            "\t\t sudden death \t 2.329618247112669\n",
            "\t\t cardiovascular insufficiency \t 2.0804061174359703\n",
            "\t\t clinical death \t 1.8975193500518799\n",
            "\t\t mouth haemorrhage \t 1.8688815754692356\n",
            "\t\t central nervous system haemorrhage \t 1.8605918884277344\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t hypoventilation \t 1.7408309078419075\n",
            "\t\t gingival bleeding \t 1.7164910366300707\n",
            "\t\t hypopnoea \t 1.715468681305503\n",
            "\t\t eye haemorrhage \t 1.6857184186032474\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t menorrhagia \t 1.6673335354861751\n",
            "\t\t blood urine present \t 1.6461281776428223\n",
            "\t\t disseminated intravascular coagulation \t 1.636710582077133\n",
            "\t\t metrorrhagia \t 1.6150825023651123\n",
            "\t\t haemorrhage urinary tract \t 1.5763455629348755\n",
            "\t\t hypercoagulation \t 1.5386027353209348\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t tooth socket haemorrhage \t 1.4860259294509888\n",
            "\t\t hypocoagulable state \t 1.4660186767578125\n",
            "\t\t conjunctival disorder \t 1.4618439674377441\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t ischaemia \t 0.9710260176816675\n",
            "\t\t coronary artery dissection \t 0.8747293328881569\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['drug interaction', 'idiosyncratic drug reaction', 'labelled drug-drug interaction medication error', 'stevens-johnson syndrome']\n",
            "Predicted reactions: \n",
            "\t ['stevens-johnson syndrome', 'toxic epidermal necrolysis', 'erythema multiforme']\n",
            "Ungrounded reactions: \n",
            "\t ['stevens-johnson syndrome', 'toxic epidermal necrolysis', 'erythema multiforme']\n",
            "Recall@10:  0.25\n",
            "Recall@20:  0.25\n",
            "Recall@30:  0.25\n",
            "Resolved reactions: \n",
            "\t\t stevens-johnson syndrome \t 19.274861690392687\n",
            "\t\t toxic epidermal necrolysis \t 14.290383759965605\n",
            "\t\t erythema multiforme \t 11.061811038991813\n",
            "Grounded reactions: \n",
            "\t\t stevens-johnson syndrome \t 3.945810862333697\n",
            "\t\t stevens-johnson syndrome \t 3.945810862333697\n",
            "\t\t stevens-johnson syndrome \t 3.945810862333697\n",
            "\t\t stevens-johnson syndrome \t 3.7187145516957973\n",
            "\t\t stevens-johnson syndrome \t 3.7187145516957973\n",
            "\t\t toxic epidermal necrolysis \t 2.9600264371780303\n",
            "\t\t toxic epidermal necrolysis \t 2.9600264371780303\n",
            "\t\t toxic epidermal necrolysis \t 2.9600264371780303\n",
            "\t\t toxic epidermal necrolysis \t 2.7051522242157575\n",
            "\t\t toxic epidermal necrolysis \t 2.7051522242157575\n",
            "\t\t erythema multiforme \t 2.2394656404855007\n",
            "\t\t erythema multiforme \t 2.2394656404855007\n",
            "\t\t erythema multiforme \t 2.2394656404855007\n",
            "\t\t erythema multiforme \t 2.171707058767656\n",
            "\t\t erythema multiforme \t 2.171707058767656\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['constipation', 'malaise', 'nausea', 'neuropathy peripheral', 'stomatitis']\n",
            "Predicted reactions: \n",
            "\t ['neuropathy peripheral', 'fatigue', 'peripheral sensory neuropathy', 'nausea', 'vomiting', 'malaise', 'asthenia', 'polyneuropathy', 'anaemia', 'infection', 'pruritus', 'oedema peripheral', 'decreased appetite', 'stomatitis', 'alopecia', 'mucosal inflammation', 'vomiting projectile', 'rash pruritic', 'dry skin', 'general physical health deterioration', 'paraesthesia', 'oedema', 'peripheral swelling', 'dry mouth', 'depressed mood', 'dry eye', 'pruritus allergic', 'anorexia nervosa', 'microcytic anaemia', 'skin infection', 'malabsorption', 'deficiency anaemia', 'alopecia areata', 'emotional distress', 'hypotrichosis', 'quality of life decreased', 'oral mucosal eruption', 'systemic infection', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['neuropathy peripheral', 'fatigue', 'peripheral sensory neuropathy', 'nausea', 'vomiting', 'malaise', 'asthenia', 'polyneuropathy', 'anaemia', 'infection', 'pruritus', 'oedema peripheral', 'decreased appetite', 'stomatitis', 'alopecia', 'mucosal inflammation', 'vomiting projectile', 'rash pruritic', 'dry skin', 'general physical health deterioration', 'paraesthesia', 'oedema', 'peripheral swelling', 'dry mouth', 'depressed mood', 'dry eye', 'pruritus allergic', 'anorexia nervosa', 'microcytic anaemia', 'skin infection', 'malabsorption', 'deficiency anaemia', 'alopecia areata', 'emotional distress', 'hypotrichosis', 'quality of life decreased', 'oral mucosal eruption', 'systemic infection', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.6\n",
            "Recall@20:  0.8\n",
            "Recall@30:  0.8\n",
            "Resolved reactions: \n",
            "\t\t neuropathy peripheral \t 18.885842198269202\n",
            "\t\t fatigue \t 15.693427835768098\n",
            "\t\t peripheral sensory neuropathy \t 14.129809991928717\n",
            "\t\t nausea \t 11.724094031085794\n",
            "\t\t vomiting \t 11.278298153600627\n",
            "\t\t malaise \t 10.300275319734396\n",
            "\t\t asthenia \t 8.295383124881033\n",
            "\t\t polyneuropathy \t 8.26262559285308\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t infection \t 5.048049090731403\n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t oedema peripheral \t 4.340547898850344\n",
            "\t\t decreased appetite \t 4.333810472449917\n",
            "\t\t stomatitis \t 4.056728178298654\n",
            "\t\t alopecia \t 3.9191473714809497\n",
            "\t\t mucosal inflammation \t 3.427398275931924\n",
            "\t\t vomiting projectile \t 3.332826614379883\n",
            "\t\t rash pruritic \t 3.2020435906031124\n",
            "\t\t dry skin \t 3.123169781090493\n",
            "\t\t general physical health deterioration \t 2.968292136082459\n",
            "\t\t paraesthesia \t 2.813274611131038\n",
            "\t\t oedema \t 2.7892550733747465\n",
            "\t\t peripheral swelling \t 2.7694235044314213\n",
            "\t\t dry mouth \t 2.339698321151943\n",
            "\t\t depressed mood \t 2.006308666618418\n",
            "\t\t dry eye \t 1.8913195524309292\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t anorexia nervosa \t 1.8405749797821045\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t skin infection \t 1.8268614394106404\n",
            "\t\t malabsorption \t 1.8021268267206885\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t alopecia areata \t 1.6729514598846436\n",
            "\t\t emotional distress \t 1.546303453672579\n",
            "\t\t hypotrichosis \t 1.5419704914093018\n",
            "\t\t quality of life decreased \t 1.531426393292228\n",
            "\t\t oral mucosal eruption \t 1.4788228273391724\n",
            "\t\t systemic infection \t 1.4236079807237438\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t anaemia \t 5.873887542192253\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t infection \t 5.048049090731403\n",
            "\t\t neuropathy peripheral \t 5.028002643159886\n",
            "\t\t neuropathy peripheral \t 4.756222046655719\n",
            "\t\t neuropathy peripheral \t 4.650277870798836\n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t neuropathy peripheral \t 4.451339637654763\n",
            "\t\t malaise \t 4.390965090244717\n",
            "\t\t oedema peripheral \t 4.340547898850344\n",
            "\t\t decreased appetite \t 4.333810472449917\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t stomatitis \t 4.056728178298654\n",
            "\t\t fatigue \t 3.950780031536054\n",
            "\t\t alopecia \t 3.9191473714809497\n",
            "\t\t peripheral sensory neuropathy \t 3.8003819544162334\n",
            "\t\t peripheral sensory neuropathy \t 3.729043644653584\n",
            "\t\t peripheral sensory neuropathy \t 3.4459467757397646\n",
            "\t\t mucosal inflammation \t 3.427398275931924\n",
            "\t\t rash pruritic \t 3.2020435906031124\n",
            "\t\t peripheral sensory neuropathy \t 3.154437617119135\n",
            "\t\t dry skin \t 3.123169781090493\n",
            "\t\t general physical health deterioration \t 2.968292136082459\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t polyneuropathy \t 2.8499750333840117\n",
            "\t\t polyneuropathy \t 2.833305823434517\n",
            "\t\t paraesthesia \t 2.813274611131038\n",
            "\t\t oedema \t 2.7892550733747465\n",
            "\t\t peripheral swelling \t 2.7694235044314213\n",
            "\t\t polyneuropathy \t 2.5793447360345514\n",
            "\t\t dry mouth \t 2.339698321151943\n",
            "\t\t depressed mood \t 2.006308666618418\n",
            "\t\t dry eye \t 1.8913195524309292\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t anorexia nervosa \t 1.8405749797821045\n",
            "\t\t microcytic anaemia \t 1.8295872699349633\n",
            "\t\t skin infection \t 1.8268614394106404\n",
            "\t\t malabsorption \t 1.8021268267206885\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t deficiency anaemia \t 1.772294282913208\n",
            "\t\t alopecia areata \t 1.6729514598846436\n",
            "\t\t emotional distress \t 1.546303453672579\n",
            "\t\t hypotrichosis \t 1.5419704914093018\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t quality of life decreased \t 1.531426393292228\n",
            "\t\t oral mucosal eruption \t 1.4788228273391724\n",
            "\t\t systemic infection \t 1.4236079807237438\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['respiratory failure']\n",
            "Predicted reactions: \n",
            "\t ['respiratory failure', 'acute respiratory failure', 'no adverse event', 'adverse drug reaction', 'acute kidney injury', 'blood creatine phosphokinase increased', 'respiratory arrest', 'drug effective for unapproved indication', 'gastrointestinal toxicity', 'no reaction on previous exposure to drug', 'blood creatine phosphokinase bb increased', 'blood creatine phosphokinase mm increased', 'adverse reaction']\n",
            "Ungrounded reactions: \n",
            "\t ['respiratory failure', 'acute respiratory failure', 'no adverse event', 'adverse drug reaction', 'acute kidney injury', 'blood creatine phosphokinase increased', 'respiratory arrest', 'drug effective for unapproved indication', 'gastrointestinal toxicity', 'no reaction on previous exposure to drug', 'blood creatine phosphokinase bb increased', 'blood creatine phosphokinase mm increased', 'adverse reaction']\n",
            "Recall@10:  1.0\n",
            "Recall@20:  1.0\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t respiratory failure \t 9.991565538188041\n",
            "\t\t acute respiratory failure \t 6.932394180861193\n",
            "\t\t no adverse event \t 6.693598899885485\n",
            "\t\t adverse drug reaction \t 5.741384846022128\n",
            "\t\t acute kidney injury \t 4.449860041177979\n",
            "\t\t blood creatine phosphokinase increased \t 2.8460075964513925\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t drug effective for unapproved indication \t 1.787482230901558\n",
            "\t\t gastrointestinal toxicity \t 1.5162345914479136\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t blood creatine phosphokinase bb increased \t 1.3851399421691895\n",
            "\t\t blood creatine phosphokinase mm increased \t 1.3728007078170776\n",
            "\t\t adverse reaction \t 1.1163059451148316\n",
            "Grounded reactions: \n",
            "\t\t respiratory failure \t 5.630202200309923\n",
            "\t\t acute kidney injury \t 4.449860041177979\n",
            "\t\t respiratory failure \t 4.361363337878118\n",
            "\t\t acute respiratory failure \t 3.9063727090956677\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t no adverse event \t 3.179032740849157\n",
            "\t\t acute respiratory failure \t 3.026021471765525\n",
            "\t\t blood creatine phosphokinase increased \t 2.8460075964513925\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t adverse drug reaction \t 2.151889341998524\n",
            "\t\t adverse drug reaction \t 1.8188675208367635\n",
            "\t\t drug effective for unapproved indication \t 1.787482230901558\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t gastrointestinal toxicity \t 1.5162345914479136\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t blood creatine phosphokinase bb increased \t 1.3851399421691895\n",
            "\t\t blood creatine phosphokinase mm increased \t 1.3728007078170776\n",
            "\t\t adverse reaction \t 1.1163059451148316\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['neoplasm progression']\n",
            "Predicted reactions: \n",
            "\t ['nausea', 'vomiting', 'hypertension', 'proteinuria', 'diarrhoea', 'blood pressure increased', 'constipation', 'fatigue', 'protein urine', 'white coat hypertension', 'rash', 'protein urine present', 'vomiting projectile', 'asthenia', 'drug-induced liver injury', 'hepatotoxicity', 'adverse drug reaction', 'palmar-plantar erythrodysaesthesia syndrome', 'no adverse event', 'diarrhoea infectious', 'rash erythematous', 'muscle spasms', 'malaise', 'rash maculo-papular', 'liver injury', 'drug intolerance', 'muscle twitching', 'hand and foot syndrome secondary to sickle cell anaemia', 'myofascial spasm', 'adverse event', 'no reaction on previous exposure to drug', 'hand-arm vibration syndrome']\n",
            "Ungrounded reactions: \n",
            "\t ['nausea', 'vomiting', 'hypertension', 'proteinuria', 'diarrhoea', 'blood pressure increased', 'constipation', 'fatigue', 'protein urine', 'white coat hypertension', 'rash', 'protein urine present', 'vomiting projectile', 'asthenia', 'drug-induced liver injury', 'hepatotoxicity', 'adverse drug reaction', 'palmar-plantar erythrodysaesthesia syndrome', 'no adverse event', 'diarrhoea infectious', 'rash erythematous', 'muscle spasms', 'malaise', 'rash maculo-papular', 'liver injury', 'drug intolerance', 'muscle twitching', 'hand and foot syndrome secondary to sickle cell anaemia', 'myofascial spasm', 'adverse event', 'no reaction on previous exposure to drug', 'hand-arm vibration syndrome']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t nausea \t 17.252317863139503\n",
            "\t\t vomiting \t 17.238577151740074\n",
            "\t\t hypertension \t 15.523672360067\n",
            "\t\t proteinuria \t 13.059271551196275\n",
            "\t\t diarrhoea \t 12.09264971021761\n",
            "\t\t blood pressure increased \t 10.444455309014119\n",
            "\t\t constipation \t 7.271667607991146\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t protein urine \t 5.84251856803894\n",
            "\t\t white coat hypertension \t 5.4718111753463745\n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t protein urine present \t 5.135669946670532\n",
            "\t\t vomiting projectile \t 5.1335859298706055\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t drug-induced liver injury \t 4.002464690556923\n",
            "\t\t hepatotoxicity \t 3.8643640219752218\n",
            "\t\t adverse drug reaction \t 3.783585779221305\n",
            "\t\t palmar-plantar erythrodysaesthesia syndrome \t 3.7533656106075752\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t diarrhoea infectious \t 3.337033987045288\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t muscle spasms \t 3.019920538457368\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t liver injury \t 2.7385522851503676\n",
            "\t\t drug intolerance \t 1.7952964948109484\n",
            "\t\t muscle twitching \t 1.6729952390189946\n",
            "\t\t hand and foot syndrome secondary to sickle cell anaemia \t 1.6250860691070557\n",
            "\t\t myofascial spasm \t 1.5287535190582275\n",
            "\t\t adverse event \t 1.522006539762788\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t hand-arm vibration syndrome \t 1.4710776805877686\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t rash \t 5.297680058631821\n",
            "\t\t hypertension \t 5.174557453355667\n",
            "\t\t hypertension \t 5.174557453355667\n",
            "\t\t hypertension \t 5.174557453355667\n",
            "\t\t proteinuria \t 4.353090517065425\n",
            "\t\t proteinuria \t 4.353090517065425\n",
            "\t\t proteinuria \t 4.353090517065425\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t drug-induced liver injury \t 4.002464690556923\n",
            "\t\t hepatotoxicity \t 3.8643640219752218\n",
            "\t\t palmar-plantar erythrodysaesthesia syndrome \t 3.7533656106075752\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t no adverse event \t 3.514566159036328\n",
            "\t\t blood pressure increased \t 3.4814851030047063\n",
            "\t\t blood pressure increased \t 3.4814851030047063\n",
            "\t\t blood pressure increased \t 3.4814851030047063\n",
            "\t\t rash erythematous \t 3.2309815931917236\n",
            "\t\t muscle spasms \t 3.019920538457368\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t rash maculo-papular \t 2.8233759503956306\n",
            "\t\t liver injury \t 2.7385522851503676\n",
            "\t\t adverse drug reaction \t 2.0129577960344647\n",
            "\t\t protein urine \t 1.9475061893463135\n",
            "\t\t protein urine \t 1.9475061893463135\n",
            "\t\t protein urine \t 1.9475061893463135\n",
            "\t\t white coat hypertension \t 1.8239370584487915\n",
            "\t\t white coat hypertension \t 1.8239370584487915\n",
            "\t\t white coat hypertension \t 1.8239370584487915\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t drug intolerance \t 1.7952964948109484\n",
            "\t\t adverse drug reaction \t 1.7706279831868403\n",
            "\t\t protein urine present \t 1.7118899822235107\n",
            "\t\t protein urine present \t 1.7118899822235107\n",
            "\t\t protein urine present \t 1.7118899822235107\n",
            "\t\t muscle twitching \t 1.6729952390189946\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t hand and foot syndrome secondary to sickle cell anaemia \t 1.6250860691070557\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t myofascial spasm \t 1.5287535190582275\n",
            "\t\t adverse event \t 1.522006539762788\n",
            "\t\t no reaction on previous exposure to drug \t 1.486278772354126\n",
            "\t\t hand-arm vibration syndrome \t 1.4710776805877686\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['dizziness', 'dry skin', 'gastrointestinal disorder', 'heart rate increased', 'insomnia', 'memory impairment', 'product use in unapproved indication', 'self-medication']\n",
            "Predicted reactions: \n",
            "\t ['dizziness', 'dehydration', 'insomnia', 'chills', 'diarrhoea', 'urticaria', 'muscle spasms', 'memory impairment', 'dry skin', 'sinusitis', 'angioedema', 'influenza like illness', 'oropharyngeal pain', 'vertigo', 'influenza', 'abdominal discomfort', 'amnesia', 'dry mouth', 'hypovolaemia', 'hiccups', 'disturbance in attention', 'yawning', 'nasal congestion', 'thirst', 'dizziness postural', 'dry eye', 'paranasal sinus inflammation', 'middle insomnia', 'sinusitis noninfective', 'terminal insomnia', 'throat irritation', 'muscle twitching', 'trismus', 'abdominal symptom', 'hypokinesia', 'body temperature increased', 'nasal oedema', 'snoring', 'adverse drug reaction', 'nasal obstruction', 'oropharyngeal discomfort', 'h1n1 influenza', 'allergic oedema', 'feeling of body temperature change', 'motor dysfunction', 'decreased activity', 'rebound effect', 'divorced', 'homosexuality', 'married', 'adverse reaction']\n",
            "Ungrounded reactions: \n",
            "\t ['dizziness', 'dehydration', 'insomnia', 'chills', 'diarrhoea', 'urticaria', 'muscle spasms', 'memory impairment', 'dry skin', 'sinusitis', 'angioedema', 'influenza like illness', 'oropharyngeal pain', 'vertigo', 'influenza', 'abdominal discomfort', 'amnesia', 'dry mouth', 'hypovolaemia', 'hiccups', 'disturbance in attention', 'yawning', 'nasal congestion', 'thirst', 'dizziness postural', 'dry eye', 'paranasal sinus inflammation', 'middle insomnia', 'sinusitis noninfective', 'terminal insomnia', 'throat irritation', 'muscle twitching', 'trismus', 'abdominal symptom', 'hypokinesia', 'body temperature increased', 'nasal oedema', 'snoring', 'adverse drug reaction', 'nasal obstruction', 'oropharyngeal discomfort', 'h1n1 influenza', 'allergic oedema', 'feeling of body temperature change', 'motor dysfunction', 'decreased activity', 'rebound effect', 'divorced', 'homosexuality', 'married', 'adverse reaction']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t dizziness \t 5.002053790283374\n",
            "\t\t dehydration \t 4.550906760236026\n",
            "\t\t insomnia \t 4.462664192640257\n",
            "\t\t chills \t 4.08963795804121\n",
            "\t\t diarrhoea \t 3.9154458879188785\n",
            "\t\t urticaria \t 3.3045640409872075\n",
            "\t\t muscle spasms \t 3.207420991651697\n",
            "\t\t memory impairment \t 3.158432195933962\n",
            "\t\t dry skin \t 3.123169781090493\n",
            "\t\t sinusitis \t 3.0781540194884047\n",
            "\t\t angioedema \t 2.8432643689229766\n",
            "\t\t influenza like illness \t 2.7811899065299444\n",
            "\t\t oropharyngeal pain \t 2.633267588359446\n",
            "\t\t vertigo \t 2.6231935713626284\n",
            "\t\t influenza \t 2.609691510630165\n",
            "\t\t abdominal discomfort \t 2.563302105233214\n",
            "\t\t amnesia \t 2.536772709989819\n",
            "\t\t dry mouth \t 2.339698321151943\n",
            "\t\t hypovolaemia \t 2.1895029997202227\n",
            "\t\t hiccups \t 2.1052060899614222\n",
            "\t\t disturbance in attention \t 2.031408393270929\n",
            "\t\t yawning \t 2.0\n",
            "\t\t nasal congestion \t 1.9720924937236772\n",
            "\t\t thirst \t 1.9237989293717024\n",
            "\t\t dizziness postural \t 1.9157160247242142\n",
            "\t\t dry eye \t 1.8913195524309292\n",
            "\t\t paranasal sinus inflammation \t 1.889520287513733\n",
            "\t\t middle insomnia \t 1.8854784965515137\n",
            "\t\t sinusitis noninfective \t 1.8407843112945557\n",
            "\t\t terminal insomnia \t 1.840500831604004\n",
            "\t\t throat irritation \t 1.7543901205062866\n",
            "\t\t muscle twitching \t 1.7422123466967707\n",
            "\t\t trismus \t 1.6567017876049368\n",
            "\t\t abdominal symptom \t 1.6428523063659668\n",
            "\t\t hypokinesia \t 1.6314031751836313\n",
            "\t\t body temperature increased \t 1.6129273230371426\n",
            "\t\t nasal oedema \t 1.6099530458450317\n",
            "\t\t snoring \t 1.5973937511444092\n",
            "\t\t adverse drug reaction \t 1.569063882295079\n",
            "\t\t nasal obstruction \t 1.565278172492981\n",
            "\t\t oropharyngeal discomfort \t 1.5394014120101929\n",
            "\t\t h1n1 influenza \t 1.514762282371521\n",
            "\t\t allergic oedema \t 1.5139275789260864\n",
            "\t\t feeling of body temperature change \t 1.3963960409164429\n",
            "\t\t motor dysfunction \t 1.2664529622056964\n",
            "\t\t decreased activity \t 1.189793348312378\n",
            "\t\t rebound effect \t 1.0422110728564835\n",
            "\t\t divorced \t 0.9876983165740967\n",
            "\t\t homosexuality \t 0.9583614468574524\n",
            "\t\t married \t 0.9419969320297241\n",
            "\t\t adverse reaction \t 0.9012870611401393\n",
            "Grounded reactions: \n",
            "\t\t dizziness \t 5.002053790283374\n",
            "\t\t dehydration \t 4.550906760236026\n",
            "\t\t insomnia \t 4.462664192640257\n",
            "\t\t chills \t 4.08963795804121\n",
            "\t\t diarrhoea \t 3.9154458879188785\n",
            "\t\t urticaria \t 3.3045640409872075\n",
            "\t\t muscle spasms \t 3.207420991651697\n",
            "\t\t memory impairment \t 3.158432195933962\n",
            "\t\t dry skin \t 3.123169781090493\n",
            "\t\t sinusitis \t 3.0781540194884047\n",
            "\t\t angioedema \t 2.8432643689229766\n",
            "\t\t influenza like illness \t 2.7811899065299444\n",
            "\t\t oropharyngeal pain \t 2.633267588359446\n",
            "\t\t vertigo \t 2.6231935713626284\n",
            "\t\t influenza \t 2.609691510630165\n",
            "\t\t abdominal discomfort \t 2.563302105233214\n",
            "\t\t amnesia \t 2.536772709989819\n",
            "\t\t dry mouth \t 2.339698321151943\n",
            "\t\t hypovolaemia \t 2.1895029997202227\n",
            "\t\t hiccups \t 2.1052060899614222\n",
            "\t\t disturbance in attention \t 2.031408393270929\n",
            "\t\t yawning \t 2.0\n",
            "\t\t nasal congestion \t 1.9720924937236772\n",
            "\t\t thirst \t 1.9237989293717024\n",
            "\t\t dizziness postural \t 1.9157160247242142\n",
            "\t\t dry eye \t 1.8913195524309292\n",
            "\t\t paranasal sinus inflammation \t 1.889520287513733\n",
            "\t\t middle insomnia \t 1.8854784965515137\n",
            "\t\t sinusitis noninfective \t 1.8407843112945557\n",
            "\t\t terminal insomnia \t 1.840500831604004\n",
            "\t\t throat irritation \t 1.7543901205062866\n",
            "\t\t muscle twitching \t 1.7422123466967707\n",
            "\t\t trismus \t 1.6567017876049368\n",
            "\t\t abdominal symptom \t 1.6428523063659668\n",
            "\t\t hypokinesia \t 1.6314031751836313\n",
            "\t\t body temperature increased \t 1.6129273230371426\n",
            "\t\t nasal oedema \t 1.6099530458450317\n",
            "\t\t snoring \t 1.5973937511444092\n",
            "\t\t adverse drug reaction \t 1.569063882295079\n",
            "\t\t nasal obstruction \t 1.565278172492981\n",
            "\t\t oropharyngeal discomfort \t 1.5394014120101929\n",
            "\t\t h1n1 influenza \t 1.514762282371521\n",
            "\t\t allergic oedema \t 1.5139275789260864\n",
            "\t\t feeling of body temperature change \t 1.3963960409164429\n",
            "\t\t motor dysfunction \t 1.2664529622056964\n",
            "\t\t decreased activity \t 1.189793348312378\n",
            "\t\t rebound effect \t 1.0422110728564835\n",
            "\t\t adverse reaction \t 0.9012870611401393\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['cerebral haemorrhage', 'haematochezia', 'intestinal haemorrhage']\n",
            "Predicted reactions: \n",
            "\t ['cerebral haemorrhage', 'haemorrhage intracranial', 'gastrointestinal haemorrhage', 'thrombocytopenia', 'haematochezia', 'thrombotic microangiopathy', 'rectal haemorrhage', 'haemorrhagic cerebral infarction', 'proteinuria', 'platelet count decreased', 'drug interaction', 'no adverse event', 'nephrotic syndrome', 'haemolysis', 'immune thrombocytopenia', 'adverse drug reaction', 'haemolytic anaemia', 'post procedural haemorrhage', 'thrombotic thrombocytopenic purpura', 'protein urine', 'procedural haemorrhage', 'intravascular haemolysis', 'renal vascular thrombosis', 'haemorrhagic disorder', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['cerebral haemorrhage', 'haemorrhage intracranial', 'gastrointestinal haemorrhage', 'thrombocytopenia', 'haematochezia', 'thrombotic microangiopathy', 'rectal haemorrhage', 'haemorrhagic cerebral infarction', 'proteinuria', 'platelet count decreased', 'drug interaction', 'no adverse event', 'nephrotic syndrome', 'haemolysis', 'immune thrombocytopenia', 'adverse drug reaction', 'haemolytic anaemia', 'post procedural haemorrhage', 'thrombotic thrombocytopenic purpura', 'protein urine', 'procedural haemorrhage', 'intravascular haemolysis', 'renal vascular thrombosis', 'haemorrhagic disorder', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.6666666666666666\n",
            "Recall@20:  0.6666666666666666\n",
            "Recall@30:  0.6666666666666666\n",
            "Resolved reactions: \n",
            "\t\t cerebral haemorrhage \t 7.56273686155143\n",
            "\t\t haemorrhage intracranial \t 6.546443992015869\n",
            "\t\t gastrointestinal haemorrhage \t 6.156442734380514\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t haematochezia \t 5.638407969116589\n",
            "\t\t thrombotic microangiopathy \t 4.038096108222449\n",
            "\t\t rectal haemorrhage \t 3.923293837449644\n",
            "\t\t haemorrhagic cerebral infarction \t 3.754475116130026\n",
            "\t\t proteinuria \t 3.7242686502224576\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t nephrotic syndrome \t 2.5756188898594936\n",
            "\t\t haemolysis \t 2.4191706928928727\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t haemolytic anaemia \t 2.112988443622762\n",
            "\t\t post procedural haemorrhage \t 2.05483221613213\n",
            "\t\t thrombotic thrombocytopenic purpura \t 2.0487456381253617\n",
            "\t\t protein urine \t 1.6748175621032715\n",
            "\t\t procedural haemorrhage \t 1.6302410999742358\n",
            "\t\t intravascular haemolysis \t 1.4817737340927124\n",
            "\t\t renal vascular thrombosis \t 1.4342122077941895\n",
            "\t\t haemorrhagic disorder \t 1.400338053703308\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t thrombotic microangiopathy \t 4.038096108222449\n",
            "\t\t cerebral haemorrhage \t 3.835710446704849\n",
            "\t\t cerebral haemorrhage \t 3.7270264148465815\n",
            "\t\t proteinuria \t 3.7242686502224576\n",
            "\t\t gastrointestinal haemorrhage \t 3.6318778969917456\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t haemorrhage intracranial \t 3.310697010160459\n",
            "\t\t haematochezia \t 3.2893457629718337\n",
            "\t\t haemorrhage intracranial \t 3.23574698185541\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t nephrotic syndrome \t 2.5756188898594936\n",
            "\t\t gastrointestinal haemorrhage \t 2.5245648373887684\n",
            "\t\t haemolysis \t 2.4191706928928727\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t haematochezia \t 2.3490622061447555\n",
            "\t\t rectal haemorrhage \t 2.175264797979567\n",
            "\t\t haemolytic anaemia \t 2.112988443622762\n",
            "\t\t post procedural haemorrhage \t 2.05483221613213\n",
            "\t\t thrombotic thrombocytopenic purpura \t 2.0487456381253617\n",
            "\t\t haemorrhagic cerebral infarction \t 1.895494047829348\n",
            "\t\t haemorrhagic cerebral infarction \t 1.858981068300678\n",
            "\t\t rectal haemorrhage \t 1.748029039470077\n",
            "\t\t protein urine \t 1.6748175621032715\n",
            "\t\t procedural haemorrhage \t 1.6302410999742358\n",
            "\t\t intravascular haemolysis \t 1.4817737340927124\n",
            "\t\t renal vascular thrombosis \t 1.4342122077941895\n",
            "\t\t haemorrhagic disorder \t 1.400338053703308\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['anxiety', 'palpitations']\n",
            "Predicted reactions: \n",
            "\t ['urticaria', 'palpitations', 'tachycardia', 'rash', 'anaphylactic reaction', 'shock', 'hypersensitivity', 'arrhythmia', 'urticarial dermatitis', 'pruritus', 'anxiety', 'syncope', 'type i hypersensitivity', 'dermatitis contact', 'angioedema', 'hyperhidrosis', 'urticaria physical', 'anaphylactic shock', 'idiopathic urticaria', 'loss of consciousness', 'rash erythematous', 'rash pruritic', 'bradycardia', 'drug eruption', 'wheezing', 'rash maculo-papular', 'cough', 'asthma', 'immune-mediated adverse reaction', 'sneezing', 'fixed eruption', 'bronchospasm', 'pruritus allergic', 'night sweats', 'nervousness', 'dermatitis allergic', 'cold sweat', 'anxiety disorder', 'eczema', 'fear', 'phobia', 'nosophobia', 'circumoral oedema', 'sunscreen sensitivity', 'urticaria chronic', 'rubber sensitivity', 'rhinorrhoea', 'product origin unknown', 'uncircumcised', 'unevaluable specimen']\n",
            "Ungrounded reactions: \n",
            "\t ['urticaria', 'palpitations', 'tachycardia', 'rash', 'anaphylactic reaction', 'shock', 'hypersensitivity', 'arrhythmia', 'urticarial dermatitis', 'pruritus', 'anxiety', 'syncope', 'type i hypersensitivity', 'dermatitis contact', 'angioedema', 'hyperhidrosis', 'urticaria physical', 'anaphylactic shock', 'idiopathic urticaria', 'loss of consciousness', 'rash erythematous', 'rash pruritic', 'bradycardia', 'drug eruption', 'wheezing', 'rash maculo-papular', 'cough', 'asthma', 'immune-mediated adverse reaction', 'sneezing', 'fixed eruption', 'bronchospasm', 'pruritus allergic', 'night sweats', 'nervousness', 'dermatitis allergic', 'cold sweat', 'anxiety disorder', 'eczema', 'fear', 'phobia', 'nosophobia', 'circumoral oedema', 'sunscreen sensitivity', 'urticaria chronic', 'rubber sensitivity', 'rhinorrhoea', 'product origin unknown', 'uncircumcised', 'unevaluable specimen']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  1.0\n",
            "Recall@30:  1.0\n",
            "Resolved reactions: \n",
            "\t\t urticaria \t 16.812618228127167\n",
            "\t\t palpitations \t 11.624104012592715\n",
            "\t\t tachycardia \t 9.862323495360707\n",
            "\t\t rash \t 8.278838158838205\n",
            "\t\t anaphylactic reaction \t 6.617473331796354\n",
            "\t\t shock \t 6.4809807531383035\n",
            "\t\t hypersensitivity \t 6.287847694766967\n",
            "\t\t arrhythmia \t 6.0141780542787675\n",
            "\t\t urticarial dermatitis \t 5.379542112350464\n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t anxiety \t 4.451066126406115\n",
            "\t\t syncope \t 4.0555739661109\n",
            "\t\t type i hypersensitivity \t 3.806528506027923\n",
            "\t\t dermatitis contact \t 3.6212079524993896\n",
            "\t\t angioedema \t 3.612186778828024\n",
            "\t\t hyperhidrosis \t 3.509711373717248\n",
            "\t\t urticaria physical \t 3.4870078563690186\n",
            "\t\t anaphylactic shock \t 3.322092189300348\n",
            "\t\t idiopathic urticaria \t 3.3069069385528564\n",
            "\t\t loss of consciousness \t 3.2819181126629884\n",
            "\t\t rash erythematous \t 3.2680464537851015\n",
            "\t\t rash pruritic \t 3.2020435906031124\n",
            "\t\t bradycardia \t 3.190329147923831\n",
            "\t\t drug eruption \t 3.0435911969452287\n",
            "\t\t wheezing \t 3.0310164906160337\n",
            "\t\t rash maculo-papular \t 2.894086345489155\n",
            "\t\t cough \t 2.8832718021006483\n",
            "\t\t asthma \t 2.272675620892331\n",
            "\t\t immune-mediated adverse reaction \t 2.0548758934211615\n",
            "\t\t sneezing \t 2.0\n",
            "\t\t fixed eruption \t 1.9904037224354616\n",
            "\t\t bronchospasm \t 1.9555477702665942\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t night sweats \t 1.854261768691071\n",
            "\t\t nervousness \t 1.8404547154319442\n",
            "\t\t dermatitis allergic \t 1.8070721765958344\n",
            "\t\t cold sweat \t 1.7777384519577026\n",
            "\t\t anxiety disorder \t 1.7616612911224365\n",
            "\t\t eczema \t 1.7454347268121446\n",
            "\t\t fear \t 1.7124559879302979\n",
            "\t\t phobia \t 1.6714109182357788\n",
            "\t\t nosophobia \t 1.622959852218628\n",
            "\t\t circumoral oedema \t 1.6116259098052979\n",
            "\t\t sunscreen sensitivity \t 1.5202758312225342\n",
            "\t\t urticaria chronic \t 1.5199646949768066\n",
            "\t\t rubber sensitivity \t 1.480731725692749\n",
            "\t\t rhinorrhoea \t 1.4377885163599236\n",
            "\t\t product origin unknown \t 0.7179691195487976\n",
            "\t\t uncircumcised \t 0.64505934715271\n",
            "\t\t unevaluable specimen \t 0.6158665418624878\n",
            "Grounded reactions: \n",
            "\t\t rash \t 5.242995932180842\n",
            "\t\t pruritus \t 4.592270702953962\n",
            "\t\t anxiety \t 4.451066126406115\n",
            "\t\t anaphylactic reaction \t 4.056355817359044\n",
            "\t\t syncope \t 4.0555739661109\n",
            "\t\t palpitations \t 3.926286199908406\n",
            "\t\t palpitations \t 3.848908906342155\n",
            "\t\t palpitations \t 3.848908906342155\n",
            "\t\t urticaria \t 3.8003824074571204\n",
            "\t\t urticaria \t 3.8003824074571204\n",
            "\t\t angioedema \t 3.612186778828024\n",
            "\t\t hyperhidrosis \t 3.509711373717248\n",
            "\t\t tachycardia \t 3.47866259016131\n",
            "\t\t urticaria \t 3.3612779644030493\n",
            "\t\t anaphylactic shock \t 3.322092189300348\n",
            "\t\t loss of consciousness \t 3.2819181126629884\n",
            "\t\t shock \t 3.2716559643615977\n",
            "\t\t rash erythematous \t 3.2680464537851015\n",
            "\t\t hypersensitivity \t 3.267484497608131\n",
            "\t\t shock \t 3.209324788776706\n",
            "\t\t rash pruritic \t 3.2020435906031124\n",
            "\t\t tachycardia \t 3.1918304525996986\n",
            "\t\t tachycardia \t 3.1918304525996986\n",
            "\t\t bradycardia \t 3.190329147923831\n",
            "\t\t drug eruption \t 3.0435911969452287\n",
            "\t\t rash \t 3.035842226657363\n",
            "\t\t wheezing \t 3.0310164906160337\n",
            "\t\t hypersensitivity \t 3.020363197158835\n",
            "\t\t urticaria \t 3.0081554716961247\n",
            "\t\t arrhythmia \t 3.0070890271393838\n",
            "\t\t arrhythmia \t 3.0070890271393838\n",
            "\t\t rash maculo-papular \t 2.894086345489155\n",
            "\t\t cough \t 2.8832718021006483\n",
            "\t\t urticaria \t 2.8424199771137544\n",
            "\t\t anaphylactic reaction \t 2.5611175144373095\n",
            "\t\t asthma \t 2.272675620892331\n",
            "\t\t immune-mediated adverse reaction \t 2.0548758934211615\n",
            "\t\t sneezing \t 2.0\n",
            "\t\t fixed eruption \t 1.9904037224354616\n",
            "\t\t bronchospasm \t 1.9555477702665942\n",
            "\t\t dermatitis contact \t 1.9554272890090942\n",
            "\t\t type i hypersensitivity \t 1.9329972547219274\n",
            "\t\t type i hypersensitivity \t 1.8735312513059952\n",
            "\t\t urticarial dermatitis \t 1.8621516227722168\n",
            "\t\t urticarial dermatitis \t 1.8621516227722168\n",
            "\t\t pruritus allergic \t 1.8611410856246948\n",
            "\t\t night sweats \t 1.854261768691071\n",
            "\t\t nervousness \t 1.8404547154319442\n",
            "\t\t dermatitis allergic \t 1.8070721765958344\n",
            "\t\t cold sweat \t 1.7777384519577026\n",
            "\t\t anxiety disorder \t 1.7616612911224365\n",
            "\t\t eczema \t 1.7454347268121446\n",
            "\t\t urticaria physical \t 1.7435039281845093\n",
            "\t\t urticaria physical \t 1.7435039281845093\n",
            "\t\t fear \t 1.7124559879302979\n",
            "\t\t idiopathic urticaria \t 1.6836657524108887\n",
            "\t\t phobia \t 1.6714109182357788\n",
            "\t\t dermatitis contact \t 1.6657806634902954\n",
            "\t\t urticarial dermatitis \t 1.6552388668060303\n",
            "\t\t idiopathic urticaria \t 1.6232411861419678\n",
            "\t\t nosophobia \t 1.622959852218628\n",
            "\t\t circumoral oedema \t 1.6116259098052979\n",
            "\t\t sunscreen sensitivity \t 1.5202758312225342\n",
            "\t\t urticaria chronic \t 1.5199646949768066\n",
            "\t\t rubber sensitivity \t 1.480731725692749\n",
            "\t\t rhinorrhoea \t 1.4377885163599236\n",
            "\t\t product origin unknown \t 0.7179691195487976\n",
            "\t\t uncircumcised \t 0.64505934715271\n",
            "\t\t unevaluable specimen \t 0.6158665418624878\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['catheter site extravasation', 'device deployment issue', 'non-cardiac chest pain']\n",
            "Predicted reactions: \n",
            "\t ['pain', 'necrosis', 'headache', 'chest pain', 'tenderness', 'cell death', 'pleuritic pain', 'injury', 'foetal death', 'pleurisy', 'tissue injury']\n",
            "Ungrounded reactions: \n",
            "\t ['pain', 'necrosis', 'headache', 'chest pain', 'tenderness', 'cell death', 'pleuritic pain', 'injury', 'foetal death', 'pleurisy', 'tissue injury']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t pain \t 4.679520433152032\n",
            "\t\t necrosis \t 3.9152794599142258\n",
            "\t\t headache \t 3.8610861348756327\n",
            "\t\t chest pain \t 2.936307464076576\n",
            "\t\t tenderness \t 2.5640926425656185\n",
            "\t\t cell death \t 2.0\n",
            "\t\t pleuritic pain \t 1.9456622743685197\n",
            "\t\t injury \t 1.8680367291377304\n",
            "\t\t foetal death \t 1.7023365411954088\n",
            "\t\t pleurisy \t 1.5879198960304641\n",
            "\t\t tissue injury \t 1.52091383934021\n",
            "Grounded reactions: \n",
            "\t\t pain \t 4.679520433152032\n",
            "\t\t headache \t 3.8610861348756327\n",
            "\t\t chest pain \t 2.936307464076576\n",
            "\t\t tenderness \t 2.5640926425656185\n",
            "\t\t necrosis \t 2.351741799829748\n",
            "\t\t cell death \t 2.0\n",
            "\t\t pleuritic pain \t 1.9456622743685197\n",
            "\t\t injury \t 1.8680367291377304\n",
            "\t\t foetal death \t 1.7023365411954088\n",
            "\t\t pleurisy \t 1.5879198960304641\n",
            "\t\t necrosis \t 1.563537660084478\n",
            "\t\t tissue injury \t 1.52091383934021\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['abdominal distension', 'abdominal pain', 'disease progression', 'disease recurrence', 'dyspnoea', 'metastasis', 'multi-organ failure', 'pleural effusion']\n",
            "Predicted reactions: \n",
            "\t ['abdominal distension', 'pleural effusion', 'abdominal discomfort', 'dyspnoea', 'ascites', 'abdominal pain', 'dyspnoea exertional', 'abdominal pain upper', 'flatulence', 'infectious pleural effusion', 'pleurisy', 'urinary ascites', 'multiple organ dysfunction syndrome', 'orthopnoea', 'decreased appetite', 'biliary ascites', 'dysphagia', 'respiratory distress', 'renal failure', 'hepatic failure', 'polyuria', 'hypophagia', 'dysuria', 'pulmonary arterial pressure increased', 'pulmonary arterial wedge pressure increased', 'swelling', 'urine output increased', 'pulmonary arterial pressure abnormal', 'haemorrhagic ascites', 'lack of satiety']\n",
            "Ungrounded reactions: \n",
            "\t ['abdominal distension', 'pleural effusion', 'abdominal discomfort', 'dyspnoea', 'ascites', 'abdominal pain', 'dyspnoea exertional', 'abdominal pain upper', 'flatulence', 'infectious pleural effusion', 'pleurisy', 'urinary ascites', 'multiple organ dysfunction syndrome', 'orthopnoea', 'decreased appetite', 'biliary ascites', 'dysphagia', 'respiratory distress', 'renal failure', 'hepatic failure', 'polyuria', 'hypophagia', 'dysuria', 'pulmonary arterial pressure increased', 'pulmonary arterial wedge pressure increased', 'swelling', 'urine output increased', 'pulmonary arterial pressure abnormal', 'haemorrhagic ascites', 'lack of satiety']\n",
            "Recall@10:  0.5\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t abdominal distension \t 18.2938249280702\n",
            "\t\t pleural effusion \t 15.402419690152065\n",
            "\t\t abdominal discomfort \t 15.335701647525065\n",
            "\t\t dyspnoea \t 14.772312239219344\n",
            "\t\t ascites \t 13.023140851693864\n",
            "\t\t abdominal pain \t 10.613744159131917\n",
            "\t\t dyspnoea exertional \t 8.571385535448735\n",
            "\t\t abdominal pain upper \t 6.69277894817889\n",
            "\t\t flatulence \t 5.64776187618137\n",
            "\t\t infectious pleural effusion \t 5.547387705540062\n",
            "\t\t pleurisy \t 5.236805379393006\n",
            "\t\t urinary ascites \t 5.144556045532227\n",
            "\t\t multiple organ dysfunction syndrome \t 4.850651828504094\n",
            "\t\t orthopnoea \t 3.932315798832767\n",
            "\t\t decreased appetite \t 3.5789135812513\n",
            "\t\t biliary ascites \t 3.241389274597168\n",
            "\t\t dysphagia \t 3.229596603067633\n",
            "\t\t respiratory distress \t 3.1837566186690363\n",
            "\t\t renal failure \t 3.049016659190021\n",
            "\t\t hepatic failure \t 2.5002703418358503\n",
            "\t\t polyuria \t 2.48670244273389\n",
            "\t\t hypophagia \t 2.0564494780684686\n",
            "\t\t dysuria \t 1.9432772860015253\n",
            "\t\t pulmonary arterial pressure increased \t 1.9125250577926636\n",
            "\t\t pulmonary arterial wedge pressure increased \t 1.899442434310913\n",
            "\t\t swelling \t 1.885655729530309\n",
            "\t\t urine output increased \t 1.8012830625807774\n",
            "\t\t pulmonary arterial pressure abnormal \t 1.7398574352264404\n",
            "\t\t haemorrhagic ascites \t 1.6081058979034424\n",
            "\t\t lack of satiety \t 1.1368011236190796\n",
            "Grounded reactions: \n",
            "\t\t dyspnoea \t 5.381599277281312\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t abdominal pain \t 5.3068720795659585\n",
            "\t\t pleural effusion \t 5.134139896717355\n",
            "\t\t pleural effusion \t 5.134139896717355\n",
            "\t\t pleural effusion \t 5.134139896717355\n",
            "\t\t dyspnoea \t 4.931075863129843\n",
            "\t\t multiple organ dysfunction syndrome \t 4.850651828504094\n",
            "\t\t ascites \t 4.462664724631348\n",
            "\t\t ascites \t 4.462664724631348\n",
            "\t\t dyspnoea \t 4.459637098808189\n",
            "\t\t ascites \t 4.097811402431167\n",
            "\t\t abdominal distension \t 4.038096348911762\n",
            "\t\t abdominal distension \t 4.038096348911762\n",
            "\t\t abdominal distension \t 4.038096348911762\n",
            "\t\t abdominal distension \t 4.038096348911762\n",
            "\t\t decreased appetite \t 3.5789135812513\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t abdominal pain upper \t 3.346389474089445\n",
            "\t\t dysphagia \t 3.229596603067633\n",
            "\t\t respiratory distress \t 3.1837566186690363\n",
            "\t\t dyspnoea exertional \t 3.167626700380651\n",
            "\t\t renal failure \t 3.049016659190021\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t abdominal discomfort \t 3.0045160078166697\n",
            "\t\t dyspnoea exertional \t 2.8847997966652037\n",
            "\t\t dyspnoea exertional \t 2.518959038402881\n",
            "\t\t hepatic failure \t 2.5002703418358503\n",
            "\t\t polyuria \t 2.48670244273389\n",
            "\t\t abdominal discomfort \t 2.331667407972932\n",
            "\t\t abdominal discomfort \t 2.331667407972932\n",
            "\t\t abdominal discomfort \t 2.331667407972932\n",
            "\t\t abdominal discomfort \t 2.331667407972932\n",
            "\t\t abdominal distension \t 2.141439532423151\n",
            "\t\t hypophagia \t 2.0564494780684686\n",
            "\t\t orthopnoea \t 2.0090918183849933\n",
            "\t\t dysuria \t 1.9432772860015253\n",
            "\t\t orthopnoea \t 1.923223980447774\n",
            "\t\t pulmonary arterial pressure increased \t 1.9125250577926636\n",
            "\t\t pulmonary arterial wedge pressure increased \t 1.899442434310913\n",
            "\t\t swelling \t 1.885655729530309\n",
            "\t\t infectious pleural effusion \t 1.8491292351800208\n",
            "\t\t infectious pleural effusion \t 1.8491292351800208\n",
            "\t\t infectious pleural effusion \t 1.8491292351800208\n",
            "\t\t urine output increased \t 1.8012830625807774\n",
            "\t\t urinary ascites \t 1.7638704776763916\n",
            "\t\t urinary ascites \t 1.7638704776763916\n",
            "\t\t pleurisy \t 1.745601793131002\n",
            "\t\t pleurisy \t 1.745601793131002\n",
            "\t\t pleurisy \t 1.745601793131002\n",
            "\t\t pulmonary arterial pressure abnormal \t 1.7398574352264404\n",
            "\t\t biliary ascites \t 1.620694637298584\n",
            "\t\t biliary ascites \t 1.620694637298584\n",
            "\t\t urinary ascites \t 1.6168150901794434\n",
            "\t\t haemorrhagic ascites \t 1.6081058979034424\n",
            "\t\t flatulence \t 1.4119404690453425\n",
            "\t\t flatulence \t 1.4119404690453425\n",
            "\t\t flatulence \t 1.4119404690453425\n",
            "\t\t flatulence \t 1.4119404690453425\n",
            "\t\t lack of satiety \t 1.1368011236190796\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['cardiac arrest', 'cardiogenic shock', 'electrocardiogram st segment elevation', 'gastrointestinal haemorrhage', 'left ventricular dysfunction', 'mitral valve incompetence', 'multiple organ dysfunction syndrome', 'tricuspid valve incompetence', 'ventricular hypokinesia', 'ventricular tachycardia']\n",
            "Predicted reactions: \n",
            "\t ['cardiogenic shock', 'hypotension', 'multiple organ dysfunction syndrome', 'no adverse event', 'cardiac arrest', 'hyperglycaemia', 'respiratory failure', 'blood pressure decreased', 'ventricular tachycardia', 'shock', 'cardio-respiratory arrest', 'drug interaction', 'renal failure', 'blood glucose increased', 'syncope', 'torsade de pointes', 'ventricular fibrillation', 'adverse drug reaction', 'hypertriglyceridaemia', 'gastrointestinal haemorrhage', 'vascular stent thrombosis', 'embolism', 'circulatory collapse', 'blood pressure systolic decreased', 'therapeutic embolisation', 'vascular graft thrombosis', 'implant site thrombosis', 'device embolisation', 'sudden death', 'upper gastrointestinal haemorrhage', 'sudden cardiac death', 'lower gastrointestinal haemorrhage', 'nyha classification', 'blood group o']\n",
            "Ungrounded reactions: \n",
            "\t ['cardiogenic shock', 'hypotension', 'multiple organ dysfunction syndrome', 'no adverse event', 'cardiac arrest', 'hyperglycaemia', 'respiratory failure', 'blood pressure decreased', 'ventricular tachycardia', 'shock', 'cardio-respiratory arrest', 'drug interaction', 'renal failure', 'blood glucose increased', 'syncope', 'torsade de pointes', 'ventricular fibrillation', 'adverse drug reaction', 'hypertriglyceridaemia', 'gastrointestinal haemorrhage', 'vascular stent thrombosis', 'embolism', 'circulatory collapse', 'blood pressure systolic decreased', 'therapeutic embolisation', 'vascular graft thrombosis', 'implant site thrombosis', 'device embolisation', 'sudden death', 'upper gastrointestinal haemorrhage', 'sudden cardiac death', 'lower gastrointestinal haemorrhage', 'nyha classification', 'blood group o']\n",
            "Recall@10:  0.4\n",
            "Recall@20:  0.5\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t cardiogenic shock \t 5.953761840264138\n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t multiple organ dysfunction syndrome \t 4.950016632460601\n",
            "\t\t no adverse event \t 4.731864655024923\n",
            "\t\t cardiac arrest \t 4.311064229745204\n",
            "\t\t hyperglycaemia \t 4.24485735499562\n",
            "\t\t respiratory failure \t 3.486224428891502\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t ventricular tachycardia \t 3.3426988974362057\n",
            "\t\t shock \t 3.2754312268612478\n",
            "\t\t cardio-respiratory arrest \t 3.1914974815810915\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t renal failure \t 3.012501388215662\n",
            "\t\t blood glucose increased \t 2.90086120512768\n",
            "\t\t syncope \t 2.7566367854134843\n",
            "\t\t torsade de pointes \t 2.5715498152059255\n",
            "\t\t ventricular fibrillation \t 2.506268448337308\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t hypertriglyceridaemia \t 2.345180534253753\n",
            "\t\t gastrointestinal haemorrhage \t 2.2929790259161265\n",
            "\t\t vascular stent thrombosis \t 2.2587936263914385\n",
            "\t\t embolism \t 2.2159611420232896\n",
            "\t\t circulatory collapse \t 2.1937769036671755\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t therapeutic embolisation \t 1.7237930297851562\n",
            "\t\t vascular graft thrombosis \t 1.7050495147705078\n",
            "\t\t implant site thrombosis \t 1.6757512092590332\n",
            "\t\t device embolisation \t 1.6232554912567139\n",
            "\t\t sudden death \t 1.4360339989974016\n",
            "\t\t upper gastrointestinal haemorrhage \t 1.4072497779401214\n",
            "\t\t sudden cardiac death \t 1.3998578478985797\n",
            "\t\t lower gastrointestinal haemorrhage \t 1.1225504885872184\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "Grounded reactions: \n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t multiple organ dysfunction syndrome \t 4.950016632460601\n",
            "\t\t cardiac arrest \t 4.311064229745204\n",
            "\t\t hyperglycaemia \t 4.24485735499562\n",
            "\t\t cardiogenic shock \t 3.844135561911765\n",
            "\t\t respiratory failure \t 3.486224428891502\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t ventricular tachycardia \t 3.3426988974362057\n",
            "\t\t shock \t 3.2754312268612478\n",
            "\t\t cardio-respiratory arrest \t 3.1914974815810915\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t renal failure \t 3.012501388215662\n",
            "\t\t blood glucose increased \t 2.90086120512768\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t syncope \t 2.7566367854134843\n",
            "\t\t torsade de pointes \t 2.5715498152059255\n",
            "\t\t ventricular fibrillation \t 2.506268448337308\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t hypertriglyceridaemia \t 2.345180534253753\n",
            "\t\t gastrointestinal haemorrhage \t 2.2929790259161265\n",
            "\t\t vascular stent thrombosis \t 2.2587936263914385\n",
            "\t\t embolism \t 2.2159611420232896\n",
            "\t\t circulatory collapse \t 2.1937769036671755\n",
            "\t\t cardiogenic shock \t 2.109626278352373\n",
            "\t\t no adverse event \t 1.9108820812637108\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t therapeutic embolisation \t 1.7237930297851562\n",
            "\t\t vascular graft thrombosis \t 1.7050495147705078\n",
            "\t\t implant site thrombosis \t 1.6757512092590332\n",
            "\t\t device embolisation \t 1.6232554912567139\n",
            "\t\t sudden death \t 1.4360339989974016\n",
            "\t\t upper gastrointestinal haemorrhage \t 1.4072497779401214\n",
            "\t\t sudden cardiac death \t 1.3998578478985797\n",
            "\t\t lower gastrointestinal haemorrhage \t 1.1225504885872184\n",
            "\t\t nyha classification \t 0.8501802682876587\n",
            "\t\t blood group o \t 0.7983784675598145\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['therapeutic response unexpected']\n",
            "Predicted reactions: \n",
            "\t ['leukopenia', 'pneumocystis jirovecii pneumonia', 'lymphopenia', 'bone marrow failure', 'herpes zoster', 'varicella zoster virus infection', 'white blood cell count decreased', 'cytopenia', 'pneumocystis jirovecii infection', 'pneumonia fungal', 'herpes zoster infection neurological', 'myeloid maturation arrest', 'lymphocyte stimulation test negative', 'lymphocyte transformation test negative', 'le cells absent']\n",
            "Ungrounded reactions: \n",
            "\t ['leukopenia', 'pneumocystis jirovecii pneumonia', 'lymphopenia', 'bone marrow failure', 'herpes zoster', 'varicella zoster virus infection', 'white blood cell count decreased', 'cytopenia', 'pneumocystis jirovecii infection', 'pneumonia fungal', 'herpes zoster infection neurological', 'myeloid maturation arrest', 'lymphocyte stimulation test negative', 'lymphocyte transformation test negative', 'le cells absent']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t leukopenia \t 4.58706500895965\n",
            "\t\t pneumocystis jirovecii pneumonia \t 4.161628681255536\n",
            "\t\t lymphopenia \t 4.122580188942518\n",
            "\t\t bone marrow failure \t 3.908695553829053\n",
            "\t\t herpes zoster \t 3.7546265268854278\n",
            "\t\t varicella zoster virus infection \t 2.990010016231453\n",
            "\t\t white blood cell count decreased \t 2.8824758497631158\n",
            "\t\t cytopenia \t 2.8720323620283668\n",
            "\t\t pneumocystis jirovecii infection \t 2.4999629420729548\n",
            "\t\t pneumonia fungal \t 2.1548030256379884\n",
            "\t\t herpes zoster infection neurological \t 1.7743511199951172\n",
            "\t\t myeloid maturation arrest \t 1.6331427097320557\n",
            "\t\t lymphocyte stimulation test negative \t 1.0378832817077637\n",
            "\t\t lymphocyte transformation test negative \t 1.0199410915374756\n",
            "\t\t le cells absent \t 0.9671817421913147\n",
            "Grounded reactions: \n",
            "\t\t leukopenia \t 4.58706500895965\n",
            "\t\t pneumocystis jirovecii pneumonia \t 4.161628681255536\n",
            "\t\t lymphopenia \t 4.122580188942518\n",
            "\t\t bone marrow failure \t 3.908695553829053\n",
            "\t\t herpes zoster \t 3.7546265268854278\n",
            "\t\t varicella zoster virus infection \t 2.990010016231453\n",
            "\t\t white blood cell count decreased \t 2.8824758497631158\n",
            "\t\t cytopenia \t 2.8720323620283668\n",
            "\t\t pneumocystis jirovecii infection \t 2.4999629420729548\n",
            "\t\t pneumonia fungal \t 2.1548030256379884\n",
            "\t\t herpes zoster infection neurological \t 1.7743511199951172\n",
            "\t\t myeloid maturation arrest \t 1.6331427097320557\n",
            "\t\t lymphocyte stimulation test negative \t 1.0378832817077637\n",
            "\t\t lymphocyte transformation test negative \t 1.0199410915374756\n",
            "\t\t le cells absent \t 0.9671817421913147\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['cardiovascular disorder', 'cytomegalovirus infection', 'haemorrhage', 'infection', 'neoplasm malignant', 'nephropathy toxic', 'polyomavirus-associated nephropathy', 'viraemia']\n",
            "Predicted reactions: \n",
            "\t ['nephropathy toxic', 'nephropathy', 'diarrhoea', 'leukopenia', 'infection', 'renal tubular necrosis', 'constipation', 'lymphopenia', 'no adverse event', 'white blood cell count decreased', 'skin infection', 'polyomavirus-associated nephropathy', 'diarrhoea infectious', 'no reaction on previous exposure to drug', 'potassium wasting nephropathy', 'opportunistic infection', 'drug screen negative', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['nephropathy toxic', 'nephropathy', 'diarrhoea', 'leukopenia', 'infection', 'renal tubular necrosis', 'constipation', 'lymphopenia', 'no adverse event', 'white blood cell count decreased', 'skin infection', 'polyomavirus-associated nephropathy', 'diarrhoea infectious', 'no reaction on previous exposure to drug', 'potassium wasting nephropathy', 'opportunistic infection', 'drug screen negative', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.25\n",
            "Recall@20:  0.375\n",
            "Recall@30:  0.375\n",
            "Resolved reactions: \n",
            "\t\t nephropathy toxic \t 7.117108728984112\n",
            "\t\t nephropathy \t 6.569188071398086\n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t leukopenia \t 5.271554433812283\n",
            "\t\t infection \t 4.997446794693291\n",
            "\t\t renal tubular necrosis \t 4.763761854122631\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t lymphopenia \t 3.587280720150924\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t white blood cell count decreased \t 3.3213578333799996\n",
            "\t\t skin infection \t 1.8569168529109508\n",
            "\t\t polyomavirus-associated nephropathy \t 1.7552796548015281\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t potassium wasting nephropathy \t 1.5163118839263916\n",
            "\t\t opportunistic infection \t 1.4856870262782793\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t divorced \t 0.9876983165740967\n",
            "\t\t homosexuality \t 0.9583614468574524\n",
            "\t\t married \t 0.9419969320297241\n",
            "Grounded reactions: \n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t leukopenia \t 5.271554433812283\n",
            "\t\t infection \t 4.997446794693291\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t lymphopenia \t 3.587280720150924\n",
            "\t\t nephropathy toxic \t 3.558554364492056\n",
            "\t\t nephropathy toxic \t 3.558554364492056\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t white blood cell count decreased \t 3.3213578333799996\n",
            "\t\t renal tubular necrosis \t 2.3818809270613155\n",
            "\t\t renal tubular necrosis \t 2.3818809270613155\n",
            "\t\t nephropathy \t 2.2691208118865522\n",
            "\t\t nephropathy \t 2.1500336297557667\n",
            "\t\t nephropathy \t 2.1500336297557667\n",
            "\t\t skin infection \t 1.8569168529109508\n",
            "\t\t polyomavirus-associated nephropathy \t 1.7552796548015281\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t potassium wasting nephropathy \t 1.5163118839263916\n",
            "\t\t opportunistic infection \t 1.4856870262782793\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['diarrhoea', 'drug interaction', 'drug level increased', 'inhibitory drug interaction', 'myopathy', 'myositis', 'nausea', 'rhabdomyolysis', 'vomiting']\n",
            "Predicted reactions: \n",
            "\t ['rhabdomyolysis', 'myopathy', 'muscle necrosis', 'myopathy toxic', 'neuromuscular toxicity', 'musculoskeletal toxicity', 'immune-mediated myositis']\n",
            "Ungrounded reactions: \n",
            "\t ['rhabdomyolysis', 'myopathy', 'muscle necrosis', 'myopathy toxic', 'neuromuscular toxicity', 'musculoskeletal toxicity', 'immune-mediated myositis']\n",
            "Recall@10:  0.2222222222222222\n",
            "Recall@20:  0.2222222222222222\n",
            "Recall@30:  0.2222222222222222\n",
            "Resolved reactions: \n",
            "\t\t rhabdomyolysis \t 18.718075038502047\n",
            "\t\t myopathy \t 10.180999824644607\n",
            "\t\t muscle necrosis \t 6.116580390681759\n",
            "\t\t myopathy toxic \t 3.398179769515991\n",
            "\t\t neuromuscular toxicity \t 1.7328468561172485\n",
            "\t\t musculoskeletal toxicity \t 1.6643071174621582\n",
            "\t\t immune-mediated myositis \t 1.6241151470656772\n",
            "Grounded reactions: \n",
            "\t\t rhabdomyolysis \t 4.679518759625512\n",
            "\t\t rhabdomyolysis \t 4.679518759625512\n",
            "\t\t rhabdomyolysis \t 4.679518759625512\n",
            "\t\t rhabdomyolysis \t 4.679518759625512\n",
            "\t\t myopathy \t 2.2910267179403756\n",
            "\t\t myopathy \t 1.9724932766760577\n",
            "\t\t myopathy \t 1.9724932766760577\n",
            "\t\t myopathy \t 1.9724932766760577\n",
            "\t\t myopathy \t 1.9724932766760577\n",
            "\t\t myopathy toxic \t 1.7467899322509766\n",
            "\t\t neuromuscular toxicity \t 1.7328468561172485\n",
            "\t\t musculoskeletal toxicity \t 1.6643071174621582\n",
            "\t\t myopathy toxic \t 1.6513898372650146\n",
            "\t\t immune-mediated myositis \t 1.6241151470656772\n",
            "\t\t muscle necrosis \t 1.5291450976704397\n",
            "\t\t muscle necrosis \t 1.5291450976704397\n",
            "\t\t muscle necrosis \t 1.5291450976704397\n",
            "\t\t muscle necrosis \t 1.5291450976704397\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['asthenia', 'confusional state', 'discomfort', 'disturbance in attention', 'dizziness', 'emotional distress', 'fatigue', 'fear', 'feeling abnormal', 'feeling jittery', 'headache', 'hyperhidrosis', 'irritability', 'nausea', 'nervousness', 'palpitations', 'restlessness', 'salivary hypersecretion', 'tension']\n",
            "Predicted reactions: \n",
            "\t ['dizziness', 'vertigo', 'anxiety', 'agitation', 'irritability', 'nausea', 'vomiting', 'dizziness postural', 'tremor', 'disorientation', 'restlessness', 'no adverse event', 'nervousness', 'fear', 'phobia', 'tension', 'vomiting projectile', 'divorced', 'homosexuality', 'married', 'sensory level normal', 'no reaction on previous exposure to drug']\n",
            "Ungrounded reactions: \n",
            "\t ['dizziness', 'vertigo', 'anxiety', 'agitation', 'irritability', 'nausea', 'vomiting', 'dizziness postural', 'tremor', 'disorientation', 'restlessness', 'no adverse event', 'nervousness', 'fear', 'phobia', 'tension', 'vomiting projectile', 'divorced', 'homosexuality', 'married', 'sensory level normal', 'no reaction on previous exposure to drug']\n",
            "Recall@10:  0.15789473684210525\n",
            "Recall@20:  0.3684210526315789\n",
            "Recall@30:  0.3684210526315789\n",
            "Resolved reactions: \n",
            "\t\t dizziness \t 14.39980350284974\n",
            "\t\t vertigo \t 7.899818396295163\n",
            "\t\t anxiety \t 7.340246010887228\n",
            "\t\t agitation \t 6.822297626846895\n",
            "\t\t irritability \t 6.329333425673038\n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t dizziness postural \t 3.8146716666385654\n",
            "\t\t tremor \t 3.028023357293735\n",
            "\t\t disorientation \t 2.803871060985377\n",
            "\t\t restlessness \t 2.5022906479824174\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t nervousness \t 2.0585813624348677\n",
            "\t\t fear \t 1.9279080629348755\n",
            "\t\t phobia \t 1.7623804807662964\n",
            "\t\t tension \t 1.5349245071411133\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t divorced \t 0.9876983165740967\n",
            "\t\t homosexuality \t 0.9583614468574524\n",
            "\t\t married \t 0.9419969320297241\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t dizziness \t 5.002053790283374\n",
            "\t\t dizziness \t 4.791142285428328\n",
            "\t\t dizziness \t 4.606607427138039\n",
            "\t\t agitation \t 3.845323871172489\n",
            "\t\t irritability \t 3.799134069604212\n",
            "\t\t anxiety \t 3.748260402944435\n",
            "\t\t anxiety \t 3.591985607942793\n",
            "\t\t tremor \t 3.028023357293735\n",
            "\t\t agitation \t 2.9769737556744067\n",
            "\t\t disorientation \t 2.803871060985377\n",
            "\t\t vertigo \t 2.742583438102107\n",
            "\t\t vertigo \t 2.6231935713626284\n",
            "\t\t vertigo \t 2.5340413868304275\n",
            "\t\t irritability \t 2.5301993560688256\n",
            "\t\t restlessness \t 2.5022906479824174\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t nervousness \t 2.0585813624348677\n",
            "\t\t fear \t 1.9279080629348755\n",
            "\t\t dizziness postural \t 1.9157160247242142\n",
            "\t\t dizziness postural \t 1.8989556419143514\n",
            "\t\t phobia \t 1.7623804807662964\n",
            "\t\t tension \t 1.5349245071411133\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['blindness transient', 'posterior capsule rupture', 'retinal toxicity']\n",
            "Predicted reactions: \n",
            "\t ['visual impairment', 'blindness', 'visual acuity reduced', 'retinal toxicity', 'retinopathy', 'commotio retinae', 'transient ischaemic attack', 'retinal exudates', 'night blindness', 'delayed dark adaptation', 'neuronophagia', 'retinal infiltrates', 'neuritic plaques', 'subretinal hyperreflective exudation', 'demyelination', 'upper motor neurone lesion', 'photophobia', 'red blood cell poikilocytes', 'lower motor neurone lesion', 'white blood cell morphology', 'anisocytosis', 'diffuse axonal injury', 'vitreal cells', 'photopsia']\n",
            "Ungrounded reactions: \n",
            "\t ['visual impairment', 'blindness', 'visual acuity reduced', 'retinal toxicity', 'retinopathy', 'commotio retinae', 'transient ischaemic attack', 'retinal exudates', 'night blindness', 'delayed dark adaptation', 'neuronophagia', 'retinal infiltrates', 'neuritic plaques', 'subretinal hyperreflective exudation', 'demyelination', 'upper motor neurone lesion', 'photophobia', 'red blood cell poikilocytes', 'lower motor neurone lesion', 'white blood cell morphology', 'anisocytosis', 'diffuse axonal injury', 'vitreal cells', 'photopsia']\n",
            "Recall@10:  0.3333333333333333\n",
            "Recall@20:  0.3333333333333333\n",
            "Recall@30:  0.3333333333333333\n",
            "Resolved reactions: \n",
            "\t\t visual impairment \t 11.027810613844519\n",
            "\t\t blindness \t 10.557305504302853\n",
            "\t\t visual acuity reduced \t 10.174964944701594\n",
            "\t\t retinal toxicity \t 4.922299756685022\n",
            "\t\t retinopathy \t 4.106218656069575\n",
            "\t\t commotio retinae \t 3.1119542121887207\n",
            "\t\t transient ischaemic attack \t 1.947302140611915\n",
            "\t\t retinal exudates \t 1.4407125033739223\n",
            "\t\t night blindness \t 1.4089443683624268\n",
            "\t\t delayed dark adaptation \t 1.3461735248565674\n",
            "\t\t neuronophagia \t 1.3015031814575195\n",
            "\t\t retinal infiltrates \t 1.282507300376892\n",
            "\t\t neuritic plaques \t 1.275619387626648\n",
            "\t\t subretinal hyperreflective exudation \t 1.2410024404525757\n",
            "\t\t demyelination \t 1.236578190595169\n",
            "\t\t upper motor neurone lesion \t 1.1560032367706299\n",
            "\t\t photophobia \t 1.0450401659274628\n",
            "\t\t red blood cell poikilocytes \t 1.033910870552063\n",
            "\t\t lower motor neurone lesion \t 1.0278916358947754\n",
            "\t\t white blood cell morphology \t 1.0237988233566284\n",
            "\t\t anisocytosis \t 1.0211577415466309\n",
            "\t\t diffuse axonal injury \t 0.9675099849700928\n",
            "\t\t vitreal cells \t 0.8405426740646362\n",
            "\t\t photopsia \t 0.8272359591070325\n",
            "Grounded reactions: \n",
            "\t\t visual impairment \t 3.6953587150919476\n",
            "\t\t visual acuity reduced \t 3.294697022187288\n",
            "\t\t blindness \t 3.178197675565822\n",
            "\t\t blindness \t 2.6017316080025434\n",
            "\t\t visual impairment \t 2.514314635847487\n",
            "\t\t visual impairment \t 2.4992308489918207\n",
            "\t\t retinal toxicity \t 2.461149878342511\n",
            "\t\t retinal toxicity \t 2.461149878342511\n",
            "\t\t blindness \t 2.4280824522598308\n",
            "\t\t visual acuity reduced \t 2.3627831678985727\n",
            "\t\t blindness \t 2.3492937684746575\n",
            "\t\t visual acuity reduced \t 2.336409586030904\n",
            "\t\t visual impairment \t 2.3189064139132642\n",
            "\t\t visual acuity reduced \t 2.1810751685848295\n",
            "\t\t retinopathy \t 2.0531093280347874\n",
            "\t\t retinopathy \t 2.0531093280347874\n",
            "\t\t transient ischaemic attack \t 1.947302140611915\n",
            "\t\t commotio retinae \t 1.5559771060943604\n",
            "\t\t commotio retinae \t 1.5559771060943604\n",
            "\t\t retinal exudates \t 1.4407125033739223\n",
            "\t\t night blindness \t 1.4089443683624268\n",
            "\t\t delayed dark adaptation \t 1.3461735248565674\n",
            "\t\t neuronophagia \t 1.3015031814575195\n",
            "\t\t retinal infiltrates \t 1.282507300376892\n",
            "\t\t neuritic plaques \t 1.275619387626648\n",
            "\t\t subretinal hyperreflective exudation \t 1.2410024404525757\n",
            "\t\t demyelination \t 1.236578190595169\n",
            "\t\t upper motor neurone lesion \t 1.1560032367706299\n",
            "\t\t photophobia \t 1.0450401659274628\n",
            "\t\t red blood cell poikilocytes \t 1.033910870552063\n",
            "\t\t lower motor neurone lesion \t 1.0278916358947754\n",
            "\t\t white blood cell morphology \t 1.0237988233566284\n",
            "\t\t anisocytosis \t 1.0211577415466309\n",
            "\t\t diffuse axonal injury \t 0.9675099849700928\n",
            "\t\t vitreal cells \t 0.8405426740646362\n",
            "\t\t photopsia \t 0.8272359591070325\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['acute hepatic failure', 'overdose', 'suicide attempt']\n",
            "Predicted reactions: \n",
            "\t ['no adverse event', 'drug-induced liver injury', 'hepatotoxicity', 'liver injury', 'sensory level normal', 'no reaction on previous exposure to drug', 'nothing by mouth order', 'ideas of reference', 'divorced', 'homosexuality', 'married']\n",
            "Ungrounded reactions: \n",
            "\t ['no adverse event', 'drug-induced liver injury', 'hepatotoxicity', 'liver injury', 'sensory level normal', 'no reaction on previous exposure to drug', 'nothing by mouth order', 'ideas of reference', 'divorced', 'homosexuality', 'married']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t no adverse event \t 6.156486593829824\n",
            "\t\t drug-induced liver injury \t 3.64954749992604\n",
            "\t\t hepatotoxicity \t 2.940607492847523\n",
            "\t\t liver injury \t 2.7169937075830584\n",
            "\t\t sensory level normal \t 1.3448030948638916\n",
            "\t\t no reaction on previous exposure to drug \t 1.3300879001617432\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "Grounded reactions: \n",
            "\t\t drug-induced liver injury \t 3.64954749992604\n",
            "\t\t hepatotoxicity \t 2.940607492847523\n",
            "\t\t liver injury \t 2.7169937075830584\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t no adverse event \t 1.5520005778103618\n",
            "\t\t nothing by mouth order \t 0.7308026552200317\n",
            "\t\t ideas of reference \t 0.7196863889694214\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "\t\t divorced \t 0.49384915828704834\n",
            "\t\t homosexuality \t 0.4791807234287262\n",
            "\t\t married \t 0.47099846601486206\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['diabetic ketoacidosis', 'muscle rigidity', 'stiff person syndrome', 'treatment noncompliance']\n",
            "Predicted reactions: \n",
            "\t ['muscle rigidity', 'musculoskeletal stiffness', 'muscle spasms', 'diabetic ketoacidosis', 'parkinsonism', 'gait disturbance', 'pain', 'stiff person syndrome', 'musculoskeletal pain', 'gait inability', 'ketoacidosis', 'stiff leg syndrome', 'euglycaemic diabetic ketoacidosis', 'rapid-onset dystonia-parkinsonism', 'joint stiffness', 'myalgia', 'abdominal pain', 'abdominal rigidity', 'mobility decreased', 'walking disability']\n",
            "Ungrounded reactions: \n",
            "\t ['muscle rigidity', 'musculoskeletal stiffness', 'muscle spasms', 'diabetic ketoacidosis', 'parkinsonism', 'gait disturbance', 'pain', 'stiff person syndrome', 'musculoskeletal pain', 'gait inability', 'ketoacidosis', 'stiff leg syndrome', 'euglycaemic diabetic ketoacidosis', 'rapid-onset dystonia-parkinsonism', 'joint stiffness', 'myalgia', 'abdominal pain', 'abdominal rigidity', 'mobility decreased', 'walking disability']\n",
            "Recall@10:  0.75\n",
            "Recall@20:  0.75\n",
            "Recall@30:  0.75\n",
            "Resolved reactions: \n",
            "\t\t muscle rigidity \t 17.92944299964168\n",
            "\t\t musculoskeletal stiffness \t 13.402991732795716\n",
            "\t\t muscle spasms \t 8.546059353898848\n",
            "\t\t diabetic ketoacidosis \t 8.004395746975218\n",
            "\t\t parkinsonism \t 7.1250994963574295\n",
            "\t\t gait disturbance \t 6.691575400874883\n",
            "\t\t pain \t 5.953093081371559\n",
            "\t\t stiff person syndrome \t 5.8508312702178955\n",
            "\t\t musculoskeletal pain \t 5.339045474211926\n",
            "\t\t gait inability \t 4.226985687933372\n",
            "\t\t ketoacidosis \t 3.9901077596732484\n",
            "\t\t stiff leg syndrome \t 3.9057029485702515\n",
            "\t\t euglycaemic diabetic ketoacidosis \t 3.8846182823181152\n",
            "\t\t rapid-onset dystonia-parkinsonism \t 3.6238853931427\n",
            "\t\t joint stiffness \t 3.380776882171631\n",
            "\t\t myalgia \t 3.131673059341655\n",
            "\t\t abdominal pain \t 2.9163216291803185\n",
            "\t\t abdominal rigidity \t 1.9519474506378174\n",
            "\t\t mobility decreased \t 1.8117242648906369\n",
            "\t\t walking disability \t 1.7378997802734375\n",
            "Grounded reactions: \n",
            "\t\t diabetic ketoacidosis \t 4.002197873487609\n",
            "\t\t diabetic ketoacidosis \t 4.002197873487609\n",
            "\t\t gait disturbance \t 3.5688123725176224\n",
            "\t\t muscle rigidity \t 3.5180401777565384\n",
            "\t\t muscle rigidity \t 3.5180401777565384\n",
            "\t\t muscle rigidity \t 3.429461021350077\n",
            "\t\t musculoskeletal stiffness \t 3.161219217522348\n",
            "\t\t musculoskeletal stiffness \t 3.161219217522348\n",
            "\t\t myalgia \t 3.131673059341655\n",
            "\t\t gait disturbance \t 3.1227630283572605\n",
            "\t\t pain \t 2.9765465406857796\n",
            "\t\t pain \t 2.9765465406857796\n",
            "\t\t abdominal pain \t 2.9163216291803185\n",
            "\t\t muscle spasms \t 2.900492891557591\n",
            "\t\t muscle spasms \t 2.822783231170628\n",
            "\t\t muscle spasms \t 2.822783231170628\n",
            "\t\t muscle rigidity \t 2.6202769045699554\n",
            "\t\t muscle rigidity \t 2.6202769045699554\n",
            "\t\t parkinsonism \t 2.4103361411669066\n",
            "\t\t parkinsonism \t 2.4103361411669066\n",
            "\t\t musculoskeletal stiffness \t 2.3825021031124503\n",
            "\t\t musculoskeletal stiffness \t 2.349025597319285\n",
            "\t\t musculoskeletal stiffness \t 2.349025597319285\n",
            "\t\t parkinsonism \t 2.304427214023616\n",
            "\t\t gait inability \t 2.2606622676400923\n",
            "\t\t muscle rigidity \t 2.2233478136386156\n",
            "\t\t ketoacidosis \t 1.9950538798366242\n",
            "\t\t ketoacidosis \t 1.9950538798366242\n",
            "\t\t gait inability \t 1.966323420293279\n",
            "\t\t abdominal rigidity \t 1.9519474506378174\n",
            "\t\t stiff person syndrome \t 1.9502770900726318\n",
            "\t\t stiff person syndrome \t 1.9502770900726318\n",
            "\t\t stiff person syndrome \t 1.9502770900726318\n",
            "\t\t euglycaemic diabetic ketoacidosis \t 1.9423091411590576\n",
            "\t\t euglycaemic diabetic ketoacidosis \t 1.9423091411590576\n",
            "\t\t musculoskeletal pain \t 1.8828014958678798\n",
            "\t\t mobility decreased \t 1.8117242648906369\n",
            "\t\t walking disability \t 1.7378997802734375\n",
            "\t\t musculoskeletal pain \t 1.7281219891720232\n",
            "\t\t musculoskeletal pain \t 1.7281219891720232\n",
            "\t\t joint stiffness \t 1.6903884410858154\n",
            "\t\t joint stiffness \t 1.6903884410858154\n",
            "\t\t stiff leg syndrome \t 1.3019009828567505\n",
            "\t\t stiff leg syndrome \t 1.3019009828567505\n",
            "\t\t stiff leg syndrome \t 1.3019009828567505\n",
            "\t\t rapid-onset dystonia-parkinsonism \t 1.2079617977142334\n",
            "\t\t rapid-onset dystonia-parkinsonism \t 1.2079617977142334\n",
            "\t\t rapid-onset dystonia-parkinsonism \t 1.2079617977142334\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['treatment failure']\n",
            "Predicted reactions: \n",
            "\t ['hypotension', 'pyrexia', 'hyponatraemia', 'anaemia', 'hypokalaemia', 'adverse drug reaction', 'no adverse event', 'proteinuria', 'blood pressure decreased', 'hyperkalaemia', 'drug interaction', 'blood potassium decreased', 'normochromic normocytic anaemia', 'blood sodium decreased', 'hyperthermia', 'body temperature increased', 'normocytic anaemia', 'protein urine', 'blood pressure systolic decreased', 'hyponatraemic syndrome', 'protein urine present', 'adverse event', 'immunosuppressant drug therapy', 'blood corticosterone', 'adverse reaction', 'adrenocortical steroid therapy', 'immunosuppression', 'hydroxycorticosteroids urine', 'cytarabine syndrome', 'cyclotron therapy', 'chemotherapy multiple agents systemic', 'product used for unknown indication', 'uncircumcised']\n",
            "Ungrounded reactions: \n",
            "\t ['hypotension', 'pyrexia', 'hyponatraemia', 'anaemia', 'hypokalaemia', 'adverse drug reaction', 'no adverse event', 'proteinuria', 'blood pressure decreased', 'hyperkalaemia', 'drug interaction', 'blood potassium decreased', 'normochromic normocytic anaemia', 'blood sodium decreased', 'hyperthermia', 'body temperature increased', 'normocytic anaemia', 'protein urine', 'blood pressure systolic decreased', 'hyponatraemic syndrome', 'protein urine present', 'adverse event', 'immunosuppressant drug therapy', 'blood corticosterone', 'adverse reaction', 'adrenocortical steroid therapy', 'immunosuppression', 'hydroxycorticosteroids urine', 'cytarabine syndrome', 'cyclotron therapy', 'chemotherapy multiple agents systemic', 'product used for unknown indication', 'uncircumcised']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t hyponatraemia \t 4.747725420279235\n",
            "\t\t anaemia \t 4.7132358577946905\n",
            "\t\t hypokalaemia \t 4.587792470321669\n",
            "\t\t adverse drug reaction \t 4.4900814349641776\n",
            "\t\t no adverse event \t 4.418558154468798\n",
            "\t\t proteinuria \t 4.353090517065425\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t hyperkalaemia \t 3.080406448671787\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t blood potassium decreased \t 2.5457074423043164\n",
            "\t\t normochromic normocytic anaemia \t 2.5215344961054336\n",
            "\t\t blood sodium decreased \t 2.3375169783518346\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t normocytic anaemia \t 2.258235059915684\n",
            "\t\t protein urine \t 1.9475061893463135\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t hyponatraemic syndrome \t 1.850398302078247\n",
            "\t\t protein urine present \t 1.7118899822235107\n",
            "\t\t adverse event \t 1.620637377016047\n",
            "\t\t immunosuppressant drug therapy \t 1.5767894983291626\n",
            "\t\t blood corticosterone \t 1.4073514938354492\n",
            "\t\t adverse reaction \t 1.2803392715421253\n",
            "\t\t adrenocortical steroid therapy \t 1.2646713256835938\n",
            "\t\t immunosuppression \t 1.1252529707941652\n",
            "\t\t hydroxycorticosteroids urine \t 1.1218518018722534\n",
            "\t\t cytarabine syndrome \t 0.7993156909942627\n",
            "\t\t cyclotron therapy \t 0.7989180088043213\n",
            "\t\t chemotherapy multiple agents systemic \t 0.7469836473464966\n",
            "\t\t product used for unknown indication \t 0.7064741207569913\n",
            "\t\t uncircumcised \t 0.6763926148414612\n",
            "Grounded reactions: \n",
            "\t\t hypotension \t 5.706173885713593\n",
            "\t\t pyrexia \t 5.612130707861001\n",
            "\t\t hyponatraemia \t 4.747725420279235\n",
            "\t\t anaemia \t 4.7132358577946905\n",
            "\t\t hypokalaemia \t 4.587792470321669\n",
            "\t\t proteinuria \t 4.353090517065425\n",
            "\t\t blood pressure decreased \t 3.4016533848946198\n",
            "\t\t hyperkalaemia \t 3.080406448671787\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t blood potassium decreased \t 2.5457074423043164\n",
            "\t\t normochromic normocytic anaemia \t 2.5215344961054336\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t blood sodium decreased \t 2.3375169783518346\n",
            "\t\t hyperthermia \t 2.3369348762331836\n",
            "\t\t body temperature increased \t 2.2742648513245327\n",
            "\t\t normocytic anaemia \t 2.258235059915684\n",
            "\t\t adverse drug reaction \t 2.1003312261140965\n",
            "\t\t protein urine \t 1.9475061893463135\n",
            "\t\t blood pressure systolic decreased \t 1.871436517881605\n",
            "\t\t hyponatraemic syndrome \t 1.850398302078247\n",
            "\t\t protein urine present \t 1.7118899822235107\n",
            "\t\t adverse event \t 1.620637377016047\n",
            "\t\t no adverse event \t 1.5975755807075858\n",
            "\t\t blood corticosterone \t 1.4073514938354492\n",
            "\t\t adverse reaction \t 1.2803392715421253\n",
            "\t\t adrenocortical steroid therapy \t 1.2646713256835938\n",
            "\t\t immunosuppression \t 1.1252529707941652\n",
            "\t\t hydroxycorticosteroids urine \t 1.1218518018722534\n",
            "\t\t immunosuppressant drug therapy \t 0.8058457374572754\n",
            "\t\t cytarabine syndrome \t 0.7993156909942627\n",
            "\t\t cyclotron therapy \t 0.7989180088043213\n",
            "\t\t immunosuppressant drug therapy \t 0.7709437608718872\n",
            "\t\t chemotherapy multiple agents systemic \t 0.7469836473464966\n",
            "\t\t product used for unknown indication \t 0.7064741207569913\n",
            "\t\t uncircumcised \t 0.6763926148414612\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['agranulocytosis', 'neutropenia', 'neutrophil count decreased', 'off label use']\n",
            "Predicted reactions: \n",
            "\t ['neutropenia', 'agranulocytosis', 'pancytopenia', 'neutrophil count decreased']\n",
            "Ungrounded reactions: \n",
            "\t ['neutropenia', 'agranulocytosis', 'pancytopenia', 'neutrophil count decreased']\n",
            "Recall@10:  0.75\n",
            "Recall@20:  0.75\n",
            "Recall@30:  0.75\n",
            "Resolved reactions: \n",
            "\t\t neutropenia \t 50.803088143211184\n",
            "\t\t agranulocytosis \t 30.54284896645661\n",
            "\t\t pancytopenia \t 15.789380278231576\n",
            "\t\t neutrophil count decreased \t 14.38160676732248\n",
            "Grounded reactions: \n",
            "\t\t neutropenia \t 6.053777817245455\n",
            "\t\t neutropenia \t 6.053777817245455\n",
            "\t\t neutropenia \t 6.053777817245455\n",
            "\t\t neutropenia \t 6.053777817245455\n",
            "\t\t neutropenia \t 6.053777817245455\n",
            "\t\t neutropenia \t 5.13354976424598\n",
            "\t\t neutropenia \t 5.13354976424598\n",
            "\t\t neutropenia \t 5.13354976424598\n",
            "\t\t neutropenia \t 5.13354976424598\n",
            "\t\t pancytopenia \t 3.947345069557894\n",
            "\t\t pancytopenia \t 3.947345069557894\n",
            "\t\t pancytopenia \t 3.947345069557894\n",
            "\t\t pancytopenia \t 3.947345069557894\n",
            "\t\t agranulocytosis \t 3.7066767346992497\n",
            "\t\t agranulocytosis \t 3.7066767346992497\n",
            "\t\t agranulocytosis \t 3.7066767346992497\n",
            "\t\t agranulocytosis \t 3.7066767346992497\n",
            "\t\t agranulocytosis \t 3.1432284055319233\n",
            "\t\t agranulocytosis \t 3.1432284055319233\n",
            "\t\t agranulocytosis \t 3.1432284055319233\n",
            "\t\t agranulocytosis \t 3.1432284055319233\n",
            "\t\t agranulocytosis \t 3.1432284055319233\n",
            "\t\t neutrophil count decreased \t 2.8763213534644962\n",
            "\t\t neutrophil count decreased \t 2.8763213534644962\n",
            "\t\t neutrophil count decreased \t 2.8763213534644962\n",
            "\t\t neutrophil count decreased \t 2.8763213534644962\n",
            "\t\t neutrophil count decreased \t 2.8763213534644962\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['affective disorder', 'aphasia', 'bk virus infection', 'blindness cortical', 'coma', 'confusional state', 'cytomegalovirus infection', 'disorientation', 'epstein-barr viraemia', 'epstein-barr virus infection', 'hemiparesis', 'hemiplegia', 'hypogammaglobulinaemia', 'jc virus infection', 'kidney transplant rejection', 'learning disorder', 'lymphopenia', 'muscular weakness', 'polyomavirus-associated nephropathy', 'progressive multifocal leukoencephalopathy', 'respiratory failure', 'visual impairment']\n",
            "Predicted reactions: \n",
            "\t ['leukopenia', 'confusional state', 'lymphopenia', 'progressive multifocal leukoencephalopathy', 'disorientation', 'delirium', 'cell-mediated immune deficiency', 'white blood cell count decreased', 'respiratory failure', 'immunodeficiency', 'leukoencephalopathy', 'coma', 'humoral immune defect', 'condition aggravated', 'hemiparesis', 'aphasia', 'loss of consciousness', 'depressed level of consciousness', 'acute respiratory failure', 'subacute sclerosing panencephalitis', 'hemiplegia', 'respiratory arrest', 'mental status changes', 'speech disorder', 'general physical health deterioration', 'blindness', 'visual impairment', 'decreased immune responsiveness', 'affective disorder', 'mood altered', 'blindness cortical', 'bipolar disorder', 'paraparesis', 'graft versus host disease', 'agraphia', 'movement disorder', 'ill-defined disorder', 'general physical condition abnormal', 'cd30 expression', 'ctla4 deficiency', 'fip1l1/pdgfr alpha fusion kinase positive', 'cd49d expression']\n",
            "Ungrounded reactions: \n",
            "\t ['leukopenia', 'confusional state', 'lymphopenia', 'progressive multifocal leukoencephalopathy', 'disorientation', 'delirium', 'cell-mediated immune deficiency', 'white blood cell count decreased', 'respiratory failure', 'immunodeficiency', 'leukoencephalopathy', 'coma', 'humoral immune defect', 'condition aggravated', 'hemiparesis', 'aphasia', 'loss of consciousness', 'depressed level of consciousness', 'acute respiratory failure', 'subacute sclerosing panencephalitis', 'hemiplegia', 'respiratory arrest', 'mental status changes', 'speech disorder', 'general physical health deterioration', 'blindness', 'visual impairment', 'decreased immune responsiveness', 'affective disorder', 'mood altered', 'blindness cortical', 'bipolar disorder', 'paraparesis', 'graft versus host disease', 'agraphia', 'movement disorder', 'ill-defined disorder', 'general physical condition abnormal', 'cd30 expression', 'ctla4 deficiency', 'fip1l1/pdgfr alpha fusion kinase positive', 'cd49d expression']\n",
            "Recall@10:  0.22727272727272727\n",
            "Recall@20:  0.36363636363636365\n",
            "Recall@30:  0.5\n",
            "Resolved reactions: \n",
            "\t\t leukopenia \t 9.1741300179193\n",
            "\t\t confusional state \t 8.546133613737073\n",
            "\t\t lymphopenia \t 8.245160377885036\n",
            "\t\t progressive multifocal leukoencephalopathy \t 7.509253948943684\n",
            "\t\t disorientation \t 6.99135399667551\n",
            "\t\t delirium \t 6.927432072560242\n",
            "\t\t cell-mediated immune deficiency \t 6.7397540807724\n",
            "\t\t white blood cell count decreased \t 5.7649516995262315\n",
            "\t\t respiratory failure \t 5.630202200309923\n",
            "\t\t immunodeficiency \t 5.397982786806696\n",
            "\t\t leukoencephalopathy \t 4.562878278084333\n",
            "\t\t coma \t 4.462664724631348\n",
            "\t\t humoral immune defect \t 4.144334077835083\n",
            "\t\t condition aggravated \t 3.726176772597141\n",
            "\t\t hemiparesis \t 3.6032752030374056\n",
            "\t\t aphasia \t 3.6032747734935797\n",
            "\t\t loss of consciousness \t 3.5536181662468214\n",
            "\t\t depressed level of consciousness \t 3.1718845654396515\n",
            "\t\t acute respiratory failure \t 3.026021471765525\n",
            "\t\t subacute sclerosing panencephalitis \t 2.850191116333008\n",
            "\t\t hemiplegia \t 2.8414755242236134\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t mental status changes \t 2.585089881333372\n",
            "\t\t speech disorder \t 2.488515418290937\n",
            "\t\t general physical health deterioration \t 2.4267685996990203\n",
            "\t\t blindness \t 2.422828204561584\n",
            "\t\t visual impairment \t 2.391915703505945\n",
            "\t\t decreased immune responsiveness \t 2.312892198562622\n",
            "\t\t affective disorder \t 2.0852936079841955\n",
            "\t\t mood altered \t 2.0492865140278735\n",
            "\t\t blindness cortical \t 1.9579867124557495\n",
            "\t\t bipolar disorder \t 1.7843972035438\n",
            "\t\t paraparesis \t 1.7502155853312291\n",
            "\t\t graft versus host disease \t 1.6933558642733624\n",
            "\t\t agraphia \t 1.6200354099273682\n",
            "\t\t movement disorder \t 1.387778158265902\n",
            "\t\t ill-defined disorder \t 1.319844546978087\n",
            "\t\t general physical condition abnormal \t 1.056087851524353\n",
            "\t\t cd30 expression \t 1.0237032175064087\n",
            "\t\t ctla4 deficiency \t 0.9598333835601807\n",
            "\t\t fip1l1/pdgfr alpha fusion kinase positive \t 0.9538519382476807\n",
            "\t\t cd49d expression \t 0.9519875645637512\n",
            "Grounded reactions: \n",
            "\t\t respiratory failure \t 5.630202200309923\n",
            "\t\t leukopenia \t 4.58706500895965\n",
            "\t\t leukopenia \t 4.58706500895965\n",
            "\t\t confusional state \t 4.577451547592006\n",
            "\t\t coma \t 4.462664724631348\n",
            "\t\t lymphopenia \t 4.122580188942518\n",
            "\t\t lymphopenia \t 4.122580188942518\n",
            "\t\t confusional state \t 3.968682066145066\n",
            "\t\t progressive multifocal leukoencephalopathy \t 3.754626974471842\n",
            "\t\t progressive multifocal leukoencephalopathy \t 3.754626974471842\n",
            "\t\t condition aggravated \t 3.726176772597141\n",
            "\t\t delirium \t 3.6333082239733354\n",
            "\t\t hemiparesis \t 3.6032752030374056\n",
            "\t\t aphasia \t 3.6032747734935797\n",
            "\t\t loss of consciousness \t 3.5536181662468214\n",
            "\t\t disorientation \t 3.496261045539627\n",
            "\t\t disorientation \t 3.4950929511358835\n",
            "\t\t delirium \t 3.2941238485869064\n",
            "\t\t depressed level of consciousness \t 3.1718845654396515\n",
            "\t\t acute respiratory failure \t 3.026021471765525\n",
            "\t\t white blood cell count decreased \t 2.8824758497631158\n",
            "\t\t white blood cell count decreased \t 2.8824758497631158\n",
            "\t\t hemiplegia \t 2.8414755242236134\n",
            "\t\t respiratory arrest \t 2.730395957950146\n",
            "\t\t mental status changes \t 2.585089881333372\n",
            "\t\t speech disorder \t 2.488515418290937\n",
            "\t\t general physical health deterioration \t 2.4267685996990203\n",
            "\t\t blindness \t 2.422828204561584\n",
            "\t\t visual impairment \t 2.391915703505945\n",
            "\t\t leukoencephalopathy \t 2.2814391390421664\n",
            "\t\t leukoencephalopathy \t 2.2814391390421664\n",
            "\t\t affective disorder \t 2.0852936079841955\n",
            "\t\t mood altered \t 2.0492865140278735\n",
            "\t\t blindness cortical \t 1.9579867124557495\n",
            "\t\t immunodeficiency \t 1.799327595602232\n",
            "\t\t immunodeficiency \t 1.799327595602232\n",
            "\t\t immunodeficiency \t 1.799327595602232\n",
            "\t\t bipolar disorder \t 1.7843972035438\n",
            "\t\t paraparesis \t 1.7502155853312291\n",
            "\t\t graft versus host disease \t 1.6933558642733624\n",
            "\t\t agraphia \t 1.6200354099273682\n",
            "\t\t cell-mediated immune deficiency \t 1.4793038368225098\n",
            "\t\t cell-mediated immune deficiency \t 1.4793038368225098\n",
            "\t\t cell-mediated immune deficiency \t 1.4793038368225098\n",
            "\t\t subacute sclerosing panencephalitis \t 1.425095558166504\n",
            "\t\t subacute sclerosing panencephalitis \t 1.425095558166504\n",
            "\t\t movement disorder \t 1.387778158265902\n",
            "\t\t humoral immune defect \t 1.3814446926116943\n",
            "\t\t humoral immune defect \t 1.3814446926116943\n",
            "\t\t humoral immune defect \t 1.3814446926116943\n",
            "\t\t ill-defined disorder \t 1.319844546978087\n",
            "\t\t cell-mediated immune deficiency \t 1.3061997890472412\n",
            "\t\t decreased immune responsiveness \t 1.301480770111084\n",
            "\t\t general physical condition abnormal \t 1.056087851524353\n",
            "\t\t cd30 expression \t 1.0237032175064087\n",
            "\t\t decreased immune responsiveness \t 1.011411428451538\n",
            "\t\t cell-mediated immune deficiency \t 0.9956427812576294\n",
            "\t\t ctla4 deficiency \t 0.9598333835601807\n",
            "\t\t fip1l1/pdgfr alpha fusion kinase positive \t 0.9538519382476807\n",
            "\t\t cd49d expression \t 0.9519875645637512\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['colitis ischaemic', 'cough', 'drug administration error', 'epistaxis', 'haematochezia', 'haemoglobin decreased', 'haemoptysis', 'labelled drug-drug interaction medication error', 'mucosal haemorrhage', 'nasal injury']\n",
            "Predicted reactions: \n",
            "\t ['epistaxis', 'haemoptysis', 'gastrointestinal haemorrhage', 'haematochezia', 'pulmonary haemorrhage', 'gingival bleeding', 'haematemesis', 'respiratory tract haemorrhage', 'rectal haemorrhage', 'cough', 'haemoglobin decreased', 'colitis', 'coagulopathy', 'productive cough', 'colitis ischaemic', 'intestinal ischaemia', 'petechiae', 'ecchymosis', 'prothrombin time prolonged', 'mucosal haemorrhage', 'activated partial thromboplastin time prolonged', 'skin ulcer', 'generalised oedema', 'skin haemorrhage', 'red blood cell count decreased', 'sputum discoloured', 'haematocrit decreased', 'habit cough', 'post procedural haemorrhage', 'sputum increased', 'mucosal dryness', 'mucosal roughness', 'oedema mucosal', 'post abortion haemorrhage', 'apheresis related complication', 'gastrointestinal mucosal disorder']\n",
            "Ungrounded reactions: \n",
            "\t ['epistaxis', 'haemoptysis', 'gastrointestinal haemorrhage', 'haematochezia', 'pulmonary haemorrhage', 'gingival bleeding', 'haematemesis', 'respiratory tract haemorrhage', 'rectal haemorrhage', 'cough', 'haemoglobin decreased', 'colitis', 'coagulopathy', 'productive cough', 'colitis ischaemic', 'intestinal ischaemia', 'petechiae', 'ecchymosis', 'prothrombin time prolonged', 'mucosal haemorrhage', 'activated partial thromboplastin time prolonged', 'skin ulcer', 'generalised oedema', 'skin haemorrhage', 'red blood cell count decreased', 'sputum discoloured', 'haematocrit decreased', 'habit cough', 'post procedural haemorrhage', 'sputum increased', 'mucosal dryness', 'mucosal roughness', 'oedema mucosal', 'post abortion haemorrhage', 'apheresis related complication', 'gastrointestinal mucosal disorder']\n",
            "Recall@10:  0.4\n",
            "Recall@20:  0.7\n",
            "Recall@30:  0.7\n",
            "Resolved reactions: \n",
            "\t\t epistaxis \t 15.859849196623776\n",
            "\t\t haemoptysis \t 15.28819893163864\n",
            "\t\t gastrointestinal haemorrhage \t 10.982451469637251\n",
            "\t\t haematochezia \t 9.666448975946434\n",
            "\t\t pulmonary haemorrhage \t 8.085093083620308\n",
            "\t\t gingival bleeding \t 6.865964146520283\n",
            "\t\t haematemesis \t 6.73875555907515\n",
            "\t\t respiratory tract haemorrhage \t 5.839825630187988\n",
            "\t\t rectal haemorrhage \t 4.682424248256914\n",
            "\t\t cough \t 4.5383007071091255\n",
            "\t\t haemoglobin decreased \t 4.401738138323119\n",
            "\t\t colitis \t 3.1572513980358456\n",
            "\t\t coagulopathy \t 3.1084798371023625\n",
            "\t\t productive cough \t 3.007559967466014\n",
            "\t\t colitis ischaemic \t 2.8232664558338305\n",
            "\t\t intestinal ischaemia \t 2.538051513971083\n",
            "\t\t petechiae \t 2.483219338343491\n",
            "\t\t ecchymosis \t 2.4759699586107997\n",
            "\t\t prothrombin time prolonged \t 2.3146905756226226\n",
            "\t\t mucosal haemorrhage \t 2.3030083864671025\n",
            "\t\t activated partial thromboplastin time prolonged \t 2.19622875384905\n",
            "\t\t skin ulcer \t 1.9914699117962489\n",
            "\t\t generalised oedema \t 1.8674266360574259\n",
            "\t\t skin haemorrhage \t 1.8536370757443423\n",
            "\t\t red blood cell count decreased \t 1.7229278087615967\n",
            "\t\t sputum discoloured \t 1.703094482421875\n",
            "\t\t haematocrit decreased \t 1.6975230979332185\n",
            "\t\t habit cough \t 1.6540231704711914\n",
            "\t\t post procedural haemorrhage \t 1.6467792779645394\n",
            "\t\t sputum increased \t 1.581804633140564\n",
            "\t\t mucosal dryness \t 1.389777660369873\n",
            "\t\t mucosal roughness \t 1.2578771114349365\n",
            "\t\t oedema mucosal \t 1.255593422175998\n",
            "\t\t post abortion haemorrhage \t 1.2408088445663452\n",
            "\t\t apheresis related complication \t 1.1667125225067139\n",
            "\t\t gastrointestinal mucosal disorder \t 1.1046419143676758\n",
            "Grounded reactions: \n",
            "\t\t cough \t 4.5383007071091255\n",
            "\t\t haemoglobin decreased \t 4.401738138323119\n",
            "\t\t haemoptysis \t 4.040627044853302\n",
            "\t\t haemoptysis \t 4.040627044853302\n",
            "\t\t haemoptysis \t 4.040627044853302\n",
            "\t\t gastrointestinal haemorrhage \t 3.976545701753294\n",
            "\t\t gastrointestinal haemorrhage \t 3.976545701753294\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t epistaxis \t 3.964962299155944\n",
            "\t\t haematochezia \t 3.622850438748298\n",
            "\t\t haematochezia \t 3.622850438748298\n",
            "\t\t haemoptysis \t 3.1663177970787326\n",
            "\t\t colitis \t 3.1572513980358456\n",
            "\t\t coagulopathy \t 3.1084798371023625\n",
            "\t\t gastrointestinal haemorrhage \t 3.029360066130664\n",
            "\t\t productive cough \t 3.007559967466014\n",
            "\t\t colitis ischaemic \t 2.8232664558338305\n",
            "\t\t pulmonary haemorrhage \t 2.695031027873436\n",
            "\t\t pulmonary haemorrhage \t 2.695031027873436\n",
            "\t\t pulmonary haemorrhage \t 2.695031027873436\n",
            "\t\t intestinal ischaemia \t 2.538051513971083\n",
            "\t\t petechiae \t 2.483219338343491\n",
            "\t\t ecchymosis \t 2.4759699586107997\n",
            "\t\t haematochezia \t 2.4207480984498373\n",
            "\t\t rectal haemorrhage \t 2.341212124128457\n",
            "\t\t rectal haemorrhage \t 2.341212124128457\n",
            "\t\t prothrombin time prolonged \t 2.3146905756226226\n",
            "\t\t mucosal haemorrhage \t 2.3030083864671025\n",
            "\t\t haematemesis \t 2.24625185302505\n",
            "\t\t haematemesis \t 2.24625185302505\n",
            "\t\t haematemesis \t 2.24625185302505\n",
            "\t\t activated partial thromboplastin time prolonged \t 2.19622875384905\n",
            "\t\t skin ulcer \t 1.9914699117962489\n",
            "\t\t generalised oedema \t 1.8674266360574259\n",
            "\t\t skin haemorrhage \t 1.8536370757443423\n",
            "\t\t red blood cell count decreased \t 1.7229278087615967\n",
            "\t\t gingival bleeding \t 1.7164910366300707\n",
            "\t\t gingival bleeding \t 1.7164910366300707\n",
            "\t\t gingival bleeding \t 1.7164910366300707\n",
            "\t\t gingival bleeding \t 1.7164910366300707\n",
            "\t\t sputum discoloured \t 1.703094482421875\n",
            "\t\t haematocrit decreased \t 1.6975230979332185\n",
            "\t\t habit cough \t 1.6540231704711914\n",
            "\t\t post procedural haemorrhage \t 1.6467792779645394\n",
            "\t\t sputum increased \t 1.581804633140564\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t respiratory tract haemorrhage \t 1.459956407546997\n",
            "\t\t mucosal dryness \t 1.389777660369873\n",
            "\t\t mucosal roughness \t 1.2578771114349365\n",
            "\t\t oedema mucosal \t 1.255593422175998\n",
            "\t\t post abortion haemorrhage \t 1.2408088445663452\n",
            "\t\t apheresis related complication \t 1.1667125225067139\n",
            "\t\t gastrointestinal mucosal disorder \t 1.1046419143676758\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['left ventricular dysfunction', 'off label use', 'product use issue', 'torsade de pointes']\n",
            "Predicted reactions: \n",
            "\t ['ventricular tachycardia', 'torsade de pointes', 'cardiac failure', 'ventricular fibrillation', 'cardiac failure congestive', 'cardiac arrest', 'death', 'cardiac failure chronic', 'no adverse event', 'cardio-respiratory arrest', 'long qt syndrome', 'sudden death', 'cardiac failure acute', 'sudden cardiac death', 'no reaction on previous exposure to drug', 'drug screen negative']\n",
            "Ungrounded reactions: \n",
            "\t ['ventricular tachycardia', 'torsade de pointes', 'cardiac failure', 'ventricular fibrillation', 'cardiac failure congestive', 'cardiac arrest', 'death', 'cardiac failure chronic', 'no adverse event', 'cardio-respiratory arrest', 'long qt syndrome', 'sudden death', 'cardiac failure acute', 'sudden cardiac death', 'no reaction on previous exposure to drug', 'drug screen negative']\n",
            "Recall@10:  0.25\n",
            "Recall@20:  0.25\n",
            "Recall@30:  0.25\n",
            "Resolved reactions: \n",
            "\t\t ventricular tachycardia \t 19.64818103394544\n",
            "\t\t torsade de pointes \t 18.78924108475203\n",
            "\t\t cardiac failure \t 12.070567360212117\n",
            "\t\t ventricular fibrillation \t 11.641575350581089\n",
            "\t\t cardiac failure congestive \t 9.322882188833018\n",
            "\t\t cardiac arrest \t 8.749663789229036\n",
            "\t\t death \t 4.6716247072123895\n",
            "\t\t cardiac failure chronic \t 4.172833922484804\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t cardio-respiratory arrest \t 3.464489725047134\n",
            "\t\t long qt syndrome \t 3.093998518950557\n",
            "\t\t sudden death \t 2.8165026730437854\n",
            "\t\t cardiac failure acute \t 2.437953099675713\n",
            "\t\t sudden cardiac death \t 1.9976202086794246\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "Grounded reactions: \n",
            "\t\t cardiac arrest \t 5.145854624214851\n",
            "\t\t death \t 4.6716247072123895\n",
            "\t\t cardiac failure \t 4.040988077093343\n",
            "\t\t cardiac failure \t 4.040988077093343\n",
            "\t\t cardiac failure \t 3.9885912060254314\n",
            "\t\t ventricular tachycardia \t 3.7610142078135356\n",
            "\t\t cardiac arrest \t 3.603809165014185\n",
            "\t\t torsade de pointes \t 3.5472668249272434\n",
            "\t\t torsade de pointes \t 3.5472668249272434\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t cardio-respiratory arrest \t 3.464489725047134\n",
            "\t\t ventricular fibrillation \t 3.418570404717827\n",
            "\t\t ventricular tachycardia \t 3.407368803830975\n",
            "\t\t ventricular tachycardia \t 3.407368803830975\n",
            "\t\t torsade de pointes \t 3.1727706399986735\n",
            "\t\t torsade de pointes \t 3.1727706399986735\n",
            "\t\t cardiac failure congestive \t 3.1444443065603327\n",
            "\t\t cardiac failure congestive \t 3.1444443065603327\n",
            "\t\t ventricular tachycardia \t 3.0887797592487263\n",
            "\t\t ventricular tachycardia \t 3.0887797592487263\n",
            "\t\t cardiac failure congestive \t 3.033993575712353\n",
            "\t\t ventricular tachycardia \t 2.8948696999725034\n",
            "\t\t torsade de pointes \t 2.8236452129269973\n",
            "\t\t sudden death \t 2.8165026730437854\n",
            "\t\t ventricular fibrillation \t 2.791513097776266\n",
            "\t\t ventricular fibrillation \t 2.791513097776266\n",
            "\t\t ventricular fibrillation \t 2.639978750310731\n",
            "\t\t torsade de pointes \t 2.525520941973198\n",
            "\t\t cardiac failure acute \t 2.437953099675713\n",
            "\t\t cardiac failure chronic \t 2.086416961242402\n",
            "\t\t cardiac failure chronic \t 2.086416961242402\n",
            "\t\t sudden cardiac death \t 1.9976202086794246\n",
            "\t\t long qt syndrome \t 1.5469992594752786\n",
            "\t\t long qt syndrome \t 1.5469992594752786\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['bone marrow failure', 'death', 'pineal germinoma', 'seizure']\n",
            "Predicted reactions: \n",
            "\t ['nausea', 'vomiting', 'thrombocytopenia', 'cytopenia', 'fatigue', 'infection', 'asthenia', 'pancytopenia', 'alopecia', 'platelet count decreased', 'vomiting projectile', 'malaise', 'white blood cell count decreased', 'aplastic anaemia', 'traumatic lung injury', 'pulmonary toxicity', 'immune thrombocytopenia', 'electrolyte imbalance', 'polyuria', 'urine output decreased', 'acute lung injury', 'infertility', 'urine output increased', 'full blood count decreased', 'skin infection', 'infertility female', 'infertility male', 'alopecia areata', 'hypotrichosis', 'systemic infection', 'electrolyte depletion', 'salt diathesis', 'hypopituitarism', 'postpartum hypopituitarism', 'postoperative hypopituitarism']\n",
            "Ungrounded reactions: \n",
            "\t ['nausea', 'vomiting', 'thrombocytopenia', 'cytopenia', 'fatigue', 'infection', 'asthenia', 'pancytopenia', 'alopecia', 'platelet count decreased', 'vomiting projectile', 'malaise', 'white blood cell count decreased', 'aplastic anaemia', 'traumatic lung injury', 'pulmonary toxicity', 'immune thrombocytopenia', 'electrolyte imbalance', 'polyuria', 'urine output decreased', 'acute lung injury', 'infertility', 'urine output increased', 'full blood count decreased', 'skin infection', 'infertility female', 'infertility male', 'alopecia areata', 'hypotrichosis', 'systemic infection', 'electrolyte depletion', 'salt diathesis', 'hypopituitarism', 'postpartum hypopituitarism', 'postoperative hypopituitarism']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t nausea \t 11.724094031085794\n",
            "\t\t vomiting \t 11.278298153600627\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t cytopenia \t 6.003473487094201\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t infection \t 5.048049090731403\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t pancytopenia \t 3.9962727719448883\n",
            "\t\t alopecia \t 3.9191473714809497\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t vomiting projectile \t 3.332826614379883\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t white blood cell count decreased \t 2.9039410277620634\n",
            "\t\t aplastic anaemia \t 2.6072991833124997\n",
            "\t\t traumatic lung injury \t 2.5201690682661964\n",
            "\t\t pulmonary toxicity \t 2.4372072352301197\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t electrolyte imbalance \t 2.387897453617204\n",
            "\t\t polyuria \t 2.3526869368224137\n",
            "\t\t urine output decreased \t 2.16081034713137\n",
            "\t\t acute lung injury \t 2.026546095413319\n",
            "\t\t infertility \t 1.999999761581421\n",
            "\t\t urine output increased \t 1.9943014343844017\n",
            "\t\t full blood count decreased \t 1.869966983795166\n",
            "\t\t skin infection \t 1.8268614394106404\n",
            "\t\t infertility female \t 1.810597538948059\n",
            "\t\t infertility male \t 1.8085129261016846\n",
            "\t\t alopecia areata \t 1.6729514598846436\n",
            "\t\t hypotrichosis \t 1.5419704914093018\n",
            "\t\t systemic infection \t 1.4236079807237438\n",
            "\t\t electrolyte depletion \t 1.3591617345809937\n",
            "\t\t salt diathesis \t 1.3170294761657715\n",
            "\t\t hypopituitarism \t 1.223095221286283\n",
            "\t\t postpartum hypopituitarism \t 1.1719930171966553\n",
            "\t\t postoperative hypopituitarism \t 1.161359429359436\n",
            "Grounded reactions: \n",
            "\t\t nausea \t 6.195870199032087\n",
            "\t\t thrombocytopenia \t 6.108620982235853\n",
            "\t\t vomiting \t 5.960278998139449\n",
            "\t\t fatigue \t 5.871323902116022\n",
            "\t\t nausea \t 5.5282238320537065\n",
            "\t\t vomiting \t 5.318019155461177\n",
            "\t\t infection \t 5.048049090731403\n",
            "\t\t asthenia \t 4.147691562440516\n",
            "\t\t pancytopenia \t 3.9962727719448883\n",
            "\t\t alopecia \t 3.9191473714809497\n",
            "\t\t platelet count decreased \t 3.4239351078899376\n",
            "\t\t cytopenia \t 3.0187339964167927\n",
            "\t\t cytopenia \t 2.984739490677408\n",
            "\t\t malaise \t 2.9546551147448397\n",
            "\t\t white blood cell count decreased \t 2.9039410277620634\n",
            "\t\t aplastic anaemia \t 2.6072991833124997\n",
            "\t\t traumatic lung injury \t 2.5201690682661964\n",
            "\t\t pulmonary toxicity \t 2.4372072352301197\n",
            "\t\t immune thrombocytopenia \t 2.3929244131462655\n",
            "\t\t electrolyte imbalance \t 2.387897453617204\n",
            "\t\t polyuria \t 2.3526869368224137\n",
            "\t\t urine output decreased \t 2.16081034713137\n",
            "\t\t acute lung injury \t 2.026546095413319\n",
            "\t\t infertility \t 1.999999761581421\n",
            "\t\t urine output increased \t 1.9943014343844017\n",
            "\t\t full blood count decreased \t 1.869966983795166\n",
            "\t\t skin infection \t 1.8268614394106404\n",
            "\t\t infertility female \t 1.810597538948059\n",
            "\t\t infertility male \t 1.8085129261016846\n",
            "\t\t vomiting projectile \t 1.8007593154907227\n",
            "\t\t alopecia areata \t 1.6729514598846436\n",
            "\t\t hypotrichosis \t 1.5419704914093018\n",
            "\t\t vomiting projectile \t 1.5320672988891602\n",
            "\t\t systemic infection \t 1.4236079807237438\n",
            "\t\t electrolyte depletion \t 1.3591617345809937\n",
            "\t\t salt diathesis \t 1.3170294761657715\n",
            "\t\t hypopituitarism \t 1.223095221286283\n",
            "\t\t postpartum hypopituitarism \t 1.1719930171966553\n",
            "\t\t postoperative hypopituitarism \t 1.161359429359436\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['general physical health deterioration']\n",
            "Predicted reactions: \n",
            "\t ['diarrhoea', 'arrhythmia', 'hypoaesthesia', 'constipation', 'no adverse event', 'paraesthesia', 'bradycardia', 'peroneal nerve palsy', 'oedema', 'palpitations', 'fluid retention', 'erectile dysfunction', 'hyporeflexia', 'generalised oedema', 'conduction disorder', 'no reaction on previous exposure to drug', 'areflexia', 'cardiac disorder', 'peripheral sensory neuropathy', 'diarrhoea haemorrhagic', 'hyperreflexia', 'organic erectile dysfunction', 'male orgasmic disorder', 'diarrhoea infectious', 'ventricle rupture', 'decreased vibratory sense', 'cardiac perforation', 'interventricular septum rupture', 'sensory loss', 'drug screen negative', 'muscle atrophy', \"hoover's sign of leg paresis\", 'sensory level normal']\n",
            "Ungrounded reactions: \n",
            "\t ['diarrhoea', 'arrhythmia', 'hypoaesthesia', 'constipation', 'no adverse event', 'paraesthesia', 'bradycardia', 'peroneal nerve palsy', 'oedema', 'palpitations', 'fluid retention', 'erectile dysfunction', 'hyporeflexia', 'generalised oedema', 'conduction disorder', 'no reaction on previous exposure to drug', 'areflexia', 'cardiac disorder', 'peripheral sensory neuropathy', 'diarrhoea haemorrhagic', 'hyperreflexia', 'organic erectile dysfunction', 'male orgasmic disorder', 'diarrhoea infectious', 'ventricle rupture', 'decreased vibratory sense', 'cardiac perforation', 'interventricular septum rupture', 'sensory loss', 'drug screen negative', 'muscle atrophy', \"hoover's sign of leg paresis\", 'sensory level normal']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t diarrhoea \t 11.100383532479785\n",
            "\t\t arrhythmia \t 7.0979289271683115\n",
            "\t\t hypoaesthesia \t 6.955068044550989\n",
            "\t\t constipation \t 6.860273770422825\n",
            "\t\t no adverse event \t 5.795101851862453\n",
            "\t\t paraesthesia \t 4.781130908165631\n",
            "\t\t bradycardia \t 3.3308983180567067\n",
            "\t\t peroneal nerve palsy \t 3.140773372844622\n",
            "\t\t oedema \t 3.1040384872355387\n",
            "\t\t palpitations \t 2.919083949456183\n",
            "\t\t fluid retention \t 2.816503176673739\n",
            "\t\t erectile dysfunction \t 2.6187293832937852\n",
            "\t\t hyporeflexia \t 2.461149878342511\n",
            "\t\t generalised oedema \t 2.3031985888001145\n",
            "\t\t conduction disorder \t 2.25151477735456\n",
            "\t\t no reaction on previous exposure to drug \t 2.207563281059265\n",
            "\t\t areflexia \t 2.0960735033919047\n",
            "\t\t cardiac disorder \t 2.0592639654622733\n",
            "\t\t peripheral sensory neuropathy \t 2.0527599710174482\n",
            "\t\t diarrhoea haemorrhagic \t 1.9479871655239045\n",
            "\t\t hyperreflexia \t 1.8551945096372586\n",
            "\t\t organic erectile dysfunction \t 1.8046739101409912\n",
            "\t\t male orgasmic disorder \t 1.7004733085632324\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t ventricle rupture \t 1.6176679134368896\n",
            "\t\t decreased vibratory sense \t 1.509394884109497\n",
            "\t\t cardiac perforation \t 1.4951987266540527\n",
            "\t\t interventricular septum rupture \t 1.4305514097213745\n",
            "\t\t sensory loss \t 1.326208815591014\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t muscle atrophy \t 1.2895458507865032\n",
            "\t\t hoover's sign of leg paresis \t 1.1194219589233398\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "Grounded reactions: \n",
            "\t\t diarrhoea \t 6.046324855108805\n",
            "\t\t diarrhoea \t 5.054058677370979\n",
            "\t\t arrhythmia \t 4.122580680392373\n",
            "\t\t constipation \t 3.635833803995573\n",
            "\t\t no adverse event \t 3.4928588438527224\n",
            "\t\t bradycardia \t 3.3308983180567067\n",
            "\t\t constipation \t 3.2244399664272527\n",
            "\t\t oedema \t 3.1040384872355387\n",
            "\t\t arrhythmia \t 2.9753482467759382\n",
            "\t\t palpitations \t 2.919083949456183\n",
            "\t\t fluid retention \t 2.816503176673739\n",
            "\t\t erectile dysfunction \t 2.6187293832937852\n",
            "\t\t paraesthesia \t 2.534609237330823\n",
            "\t\t hyporeflexia \t 2.461149878342511\n",
            "\t\t hypoaesthesia \t 2.412780326101924\n",
            "\t\t generalised oedema \t 2.3031985888001145\n",
            "\t\t no adverse event \t 2.302243008009731\n",
            "\t\t hypoaesthesia \t 2.2833896171093517\n",
            "\t\t hypoaesthesia \t 2.2588981013397134\n",
            "\t\t conduction disorder \t 2.25151477735456\n",
            "\t\t paraesthesia \t 2.2465216708348072\n",
            "\t\t areflexia \t 2.0960735033919047\n",
            "\t\t cardiac disorder \t 2.0592639654622733\n",
            "\t\t peripheral sensory neuropathy \t 2.0527599710174482\n",
            "\t\t diarrhoea haemorrhagic \t 1.9479871655239045\n",
            "\t\t hyperreflexia \t 1.8551945096372586\n",
            "\t\t organic erectile dysfunction \t 1.8046739101409912\n",
            "\t\t peroneal nerve palsy \t 1.734848083413078\n",
            "\t\t male orgasmic disorder \t 1.7004733085632324\n",
            "\t\t diarrhoea infectious \t 1.668516993522644\n",
            "\t\t ventricle rupture \t 1.6176679134368896\n",
            "\t\t no reaction on previous exposure to drug \t 1.5425193309783936\n",
            "\t\t decreased vibratory sense \t 1.509394884109497\n",
            "\t\t cardiac perforation \t 1.4951987266540527\n",
            "\t\t interventricular septum rupture \t 1.4305514097213745\n",
            "\t\t peroneal nerve palsy \t 1.4059252894315442\n",
            "\t\t sensory loss \t 1.326208815591014\n",
            "\t\t drug screen negative \t 1.3233342170715332\n",
            "\t\t muscle atrophy \t 1.2895458507865032\n",
            "\t\t hoover's sign of leg paresis \t 1.1194219589233398\n",
            "\t\t sensory level normal \t 0.6724015474319458\n",
            "\t\t no reaction on previous exposure to drug \t 0.6650439500808716\n",
            "--------\n",
            "Gold reactions: \n",
            "\t ['condition aggravated', 'drug ineffective']\n",
            "Predicted reactions: \n",
            "\t ['hyperthyroidism', 'drug interaction', 'no adverse event', 'secondary hyperthyroidism', 'thyrotoxic crisis', 'adverse drug reaction', 'secondary adrenocortical insufficiency', 'adrenocortical insufficiency acute', 'hyperadrenocorticism']\n",
            "Ungrounded reactions: \n",
            "\t ['hyperthyroidism', 'drug interaction', 'no adverse event', 'secondary hyperthyroidism', 'thyrotoxic crisis', 'adverse drug reaction', 'secondary adrenocortical insufficiency', 'adrenocortical insufficiency acute', 'hyperadrenocorticism']\n",
            "Recall@10:  0.0\n",
            "Recall@20:  0.0\n",
            "Recall@30:  0.0\n",
            "Resolved reactions: \n",
            "\t\t hyperthyroidism \t 9.672674287684375\n",
            "\t\t drug interaction \t 6.037885113551009\n",
            "\t\t no adverse event \t 5.6419651475224235\n",
            "\t\t secondary hyperthyroidism \t 5.4533915519714355\n",
            "\t\t thyrotoxic crisis \t 5.376206603670788\n",
            "\t\t adverse drug reaction \t 4.779500417700162\n",
            "\t\t secondary adrenocortical insufficiency \t 4.575803259732765\n",
            "\t\t adrenocortical insufficiency acute \t 4.566376326826499\n",
            "\t\t hyperadrenocorticism \t 4.111531019210815\n",
            "Grounded reactions: \n",
            "\t\t hyperthyroidism \t 3.2242247625614584\n",
            "\t\t hyperthyroidism \t 3.2242247625614584\n",
            "\t\t hyperthyroidism \t 3.2242247625614584\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t drug interaction \t 3.0189425567755044\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t no adverse event \t 2.8209825737612118\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t adverse drug reaction \t 2.389750208850081\n",
            "\t\t secondary hyperthyroidism \t 1.8177971839904785\n",
            "\t\t secondary hyperthyroidism \t 1.8177971839904785\n",
            "\t\t secondary hyperthyroidism \t 1.8177971839904785\n",
            "\t\t thyrotoxic crisis \t 1.7920688678902625\n",
            "\t\t thyrotoxic crisis \t 1.7920688678902625\n",
            "\t\t thyrotoxic crisis \t 1.7920688678902625\n",
            "\t\t secondary adrenocortical insufficiency \t 1.525267753244255\n",
            "\t\t secondary adrenocortical insufficiency \t 1.525267753244255\n",
            "\t\t secondary adrenocortical insufficiency \t 1.525267753244255\n",
            "\t\t adrenocortical insufficiency acute \t 1.5221254422754997\n",
            "\t\t adrenocortical insufficiency acute \t 1.5221254422754997\n",
            "\t\t adrenocortical insufficiency acute \t 1.5221254422754997\n",
            "\t\t hyperadrenocorticism \t 1.3705103397369385\n",
            "\t\t hyperadrenocorticism \t 1.3705103397369385\n",
            "\t\t hyperadrenocorticism \t 1.3705103397369385\n",
            "--------\n"
          ]
        }
      ],
      "source": [
        "def print_score_and_term(ls):\n",
        "    for score, term in ls:\n",
        "        print('\\t\\t', term, '\\t', score)\n",
        "\n",
        "for example in trainset[100:150]:\n",
        "    # get an example\n",
        "    example_x = example.inputs()\n",
        "    example_y = example.labels()\n",
        "\n",
        "    # get a predcition\n",
        "    prediction = pipeline_with_ground_and_prior(**example_x)\n",
        "\n",
        "    # print the reactions predicted\n",
        "    print('Gold reactions: ',)\n",
        "    print('\\t', example_y.reactions)\n",
        "    print('Predicted reactions: ')\n",
        "    print('\\t', prediction.reactions)\n",
        "    print('Ungrounded reactions: ')\n",
        "    print('\\t', prediction.ungrounded_reactions)\n",
        "    print('Recall@10: ', dspy_metric_recall10(example_y, prediction))\n",
        "    print('Recall@20: ', dspy_metric_recall20(example_y, prediction))\n",
        "    print('Recall@30: ', dspy_metric_recall30(example_y, prediction))\n",
        "    print('Resolved reactions: ')\n",
        "    print_score_and_term(prediction.resolved_reactions)\n",
        "    print('Grounded reactions: ')\n",
        "    print_score_and_term(prediction.grounded_reactions)\n",
        "    print('--------')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dCOcn43XDr0"
      },
      "source": [
        "# Compiling programs\n",
        "Typically with in-context learning, we'd spend some time writing a better prompt or gathering good few-shot demonstrations. However, this is brittle and time-consuming. Luckily DSPy can help us by compiling our prompt for us!\n",
        "\n",
        "This pushes the performance to 35%!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 657
        },
        "id": "rmTyOpeJzE2i",
        "outputId": "2cd9a67a-8fb4-4cd4-e206-ceade17ec57b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "./cache/compiler\n",
            "Going to sample between 1 and 2 traces per predictor.\n",
            "Will attempt to train 20 candidate sets.\n",
            "-3 range(0, 20)\n",
            "-2 range(0, 20)\n",
            "-1 range(0, 20)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:29,  3.30it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.112314319667261 / 50  (30.2): 100%|██████████| 50/50 [00:08<00:00,  5.77it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.112314319667261 / 50  (30.2%)\n",
            "Score: 30.22 for set: [2]\n",
            "New best score: 30.22 for seed 0\n",
            "Scores so far: [30.22]\n",
            "Best score: 30.22\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:34,  2.88it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.80803624480095 / 50  (29.6): 100%|██████████| 50/50 [00:07<00:00,  6.54it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.80803624480095 / 50  (29.6%)\n",
            "Score: 29.62 for set: [1]\n",
            "Scores so far: [30.22, 29.62]\n",
            "Best score: 30.22\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:13,  7.40it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.35727866904337 / 50  (34.7): 100%|██████████| 50/50 [00:14<00:00,  3.57it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.35727866904337 / 50  (34.7%)\n",
            "Score: 34.71 for set: [1]\n",
            "New best score: 34.71 for seed 2\n",
            "Scores so far: [30.22, 29.62, 34.71]\n",
            "Best score: 34.71\n",
            "Average of max per entry across top 1 scores: 0.3471455733808675\n",
            "Average of max per entry across top 2 scores: 0.4029887106357694\n",
            "Average of max per entry across top 3 scores: 0.4114887106357694\n",
            "Average of max per entry across top 5 scores: 0.4114887106357694\n",
            "Average of max per entry across top 8 scores: 0.4114887106357694\n",
            "Average of max per entry across top 9999 scores: 0.4114887106357694\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:35,  2.83it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 16.347177658942364 / 50  (32.7): 100%|██████████| 50/50 [00:10<00:00,  4.62it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 16.347177658942364 / 50  (32.7%)\n",
            "Score: 32.69 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69]\n",
            "Best score: 34.71\n",
            "Average of max per entry across top 1 scores: 0.3471455733808675\n",
            "Average of max per entry across top 2 scores: 0.3970546642899584\n",
            "Average of max per entry across top 3 scores: 0.42389780154486034\n",
            "Average of max per entry across top 5 scores: 0.4323978015448603\n",
            "Average of max per entry across top 8 scores: 0.4323978015448603\n",
            "Average of max per entry across top 9999 scores: 0.4323978015448603\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:25,  3.96it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.726203208556147 / 50  (35.5): 100%|██████████| 50/50 [00:09<00:00,  5.24it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.726203208556147 / 50  (35.5%)\n",
            "Score: 35.45 for set: [1]\n",
            "New best score: 35.45 for seed 4\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.39821093285799164\n",
            "Average of max per entry across top 3 scores: 0.41245335710041586\n",
            "Average of max per entry across top 5 scores: 0.4346200237670825\n",
            "Average of max per entry across top 8 scores: 0.4346200237670825\n",
            "Average of max per entry across top 9999 scores: 0.4346200237670825\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:21,  4.66it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.51916221033868 / 50  (25.0): 100%|██████████| 50/50 [00:06<00:00,  7.50it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.51916221033868 / 50  (25.0%)\n",
            "Score: 25.04 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.39821093285799164\n",
            "Average of max per entry across top 3 scores: 0.41245335710041586\n",
            "Average of max per entry across top 5 scores: 0.4346200237670825\n",
            "Average of max per entry across top 8 scores: 0.4587361853832441\n",
            "Average of max per entry across top 9999 scores: 0.4587361853832441\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:29,  3.38it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.85307721174594 / 50  (29.7): 100%|██████████| 50/50 [00:12<00:00,  3.98it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.85307721174594 / 50  (29.7%)\n",
            "Score: 29.71 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.39821093285799164\n",
            "Average of max per entry across top 3 scores: 0.41245335710041586\n",
            "Average of max per entry across top 5 scores: 0.4358393220126966\n",
            "Average of max per entry across top 8 scores: 0.46645548362885825\n",
            "Average of max per entry across top 9999 scores: 0.46645548362885825\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:30,  3.26it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.73581402257873 / 50  (23.5): 100%|██████████| 50/50 [00:04<00:00, 11.94it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.73581402257873 / 50  (23.5%)\n",
            "Score: 23.47 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.39821093285799164\n",
            "Average of max per entry across top 3 scores: 0.41245335710041586\n",
            "Average of max per entry across top 5 scores: 0.4358393220126966\n",
            "Average of max per entry across top 8 scores: 0.48645548362885827\n",
            "Average of max per entry across top 9999 scores: 0.48645548362885827\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:05, 17.72it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.458036244800951 / 50  (26.9): 100%|██████████| 50/50 [00:05<00:00,  9.04it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.458036244800951 / 50  (26.9%)\n",
            "Score: 26.92 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.39821093285799164\n",
            "Average of max per entry across top 3 scores: 0.41245335710041586\n",
            "Average of max per entry across top 5 scores: 0.4358393220126966\n",
            "Average of max per entry across top 8 scores: 0.4931221502955249\n",
            "Average of max per entry across top 9999 scores: 0.5131221502955249\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:15,  6.25it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.537046939988116 / 50  (35.1): 100%|██████████| 50/50 [00:09<00:00,  5.20it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 17.537046939988116 / 50  (35.1%)\n",
            "Score: 35.07 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.44912982768865123\n",
            "Average of max per entry across top 8 scores: 0.4893491259342653\n",
            "Average of max per entry across top 9999 scores: 0.5209652875504269\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:13,  7.10it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.391889483065953 / 50  (28.8): 100%|██████████| 50/50 [00:04<00:00, 10.86it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 14.391889483065953 / 50  (28.8%)\n",
            "Score: 28.78 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.44912982768865123\n",
            "Average of max per entry across top 8 scores: 0.4551824592675986\n",
            "Average of max per entry across top 9999 scores: 0.5234652875504269\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:09, 10.26it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.664111705288178 / 50  (25.3): 100%|██████████| 50/50 [00:02<00:00, 17.71it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.664111705288178 / 50  (25.3%)\n",
            "Score: 25.33 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.44912982768865123\n",
            "Average of max per entry across top 8 scores: 0.4551824592675986\n",
            "Average of max per entry across top 9999 scores: 0.5234652875504269\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:24,  3.92it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.636824123588829 / 50  (25.3): 100%|██████████| 50/50 [00:05<00:00,  8.90it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 12.636824123588829 / 50  (25.3%)\n",
            "Score: 25.27 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.44912982768865123\n",
            "Average of max per entry across top 8 scores: 0.4551824592675986\n",
            "Average of max per entry across top 9999 scores: 0.5434652875504269\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:07, 12.68it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.349702911467615 / 50  (30.7): 100%|██████████| 50/50 [00:05<00:00,  9.38it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.349702911467615 / 50  (30.7%)\n",
            "Score: 30.7 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.46546316102198454\n",
            "Average of max per entry across top 8 scores: 0.4846824592675986\n",
            "Average of max per entry across top 9999 scores: 0.5454652875504269\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:32,  3.03it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.963324420677363 / 50  (27.9): 100%|██████████| 50/50 [00:09<00:00,  5.46it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.963324420677363 / 50  (27.9%)\n",
            "Score: 27.93 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.46546316102198454\n",
            "Average of max per entry across top 8 scores: 0.4846824592675986\n",
            "Average of max per entry across top 9999 scores: 0.549465287550427\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:21,  4.51it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.328490790255495 / 50  (30.7): 100%|██████████| 50/50 [00:06<00:00,  7.39it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 15.328490790255495 / 50  (30.7%)\n",
            "Score: 30.66 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93, 30.66]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.46546316102198454\n",
            "Average of max per entry across top 8 scores: 0.4871824592675986\n",
            "Average of max per entry across top 9999 scores: 0.554465287550427\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:28,  3.48it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.475445632798573 / 50  (27.0): 100%|██████████| 50/50 [00:05<00:00,  9.03it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 13.475445632798573 / 50  (27.0%)\n",
            "Score: 26.95 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93, 30.66, 26.95]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.46546316102198454\n",
            "Average of max per entry across top 8 scores: 0.4871824592675986\n",
            "Average of max per entry across top 9999 scores: 0.554465287550427\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  2%|▏         | 2/100 [00:00<00:15,  6.27it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 2 full traces after 3 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.933065953654191 / 50  (23.9): 100%|██████████| 50/50 [00:03<00:00, 16.60it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.933065953654191 / 50  (23.9%)\n",
            "Score: 23.87 for set: [2]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93, 30.66, 26.95, 23.87]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.46546316102198454\n",
            "Average of max per entry across top 8 scores: 0.4871824592675986\n",
            "Average of max per entry across top 9999 scores: 0.556965287550427\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<01:03,  1.55it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 16.45903149138443 / 50  (32.9): 100%|██████████| 50/50 [00:10<00:00,  4.98it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 16.45903149138443 / 50  (32.9%)\n",
            "Score: 32.92 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93, 30.66, 26.95, 23.87, 32.92]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.4424126559714795\n",
            "Average of max per entry across top 8 scores: 0.4910793226381462\n",
            "Average of max per entry across top 9999 scores: 0.556965287550427\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  1%|          | 1/100 [00:00<00:07, 13.62it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 1 full traces after 2 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.264111705288176 / 50  (22.5): 100%|██████████| 50/50 [00:03<00:00, 15.22it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 11.264111705288176 / 50  (22.5%)\n",
            "Score: 22.53 for set: [1]\n",
            "Scores so far: [30.22, 29.62, 34.71, 32.69, 35.45, 25.04, 29.71, 23.47, 26.92, 35.07, 28.78, 25.33, 25.27, 30.7, 27.93, 30.66, 26.95, 23.87, 32.92, 22.53]\n",
            "Best score: 35.45\n",
            "Average of max per entry across top 1 scores: 0.3545240641711229\n",
            "Average of max per entry across top 2 scores: 0.38235204991087346\n",
            "Average of max per entry across top 3 scores: 0.4226298276886512\n",
            "Average of max per entry across top 5 scores: 0.4424126559714795\n",
            "Average of max per entry across top 8 scores: 0.4910793226381462\n",
            "Average of max per entry across top 9999 scores: 0.556965287550427\n",
            "20 candidate programs found.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",
        "tp = BootstrapFewShotWithRandomSearch(metric=dspy_metric_recall10, max_bootstrapped_demos=2, max_labeled_demos=0, max_rounds=1,\n",
        "                                       num_candidate_programs=20, num_threads=8, teacher_settings=dict(lm=turbo11))\n",
        "\n",
        "\n",
        "compiledR = tp.compile(pipeline_with_ground_and_prior, teacher=pipeline_with_ground_and_prior, trainset=trainsetX[0:100], valset=trainset[100:150], restrict=range(20))\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}