notebooks/materializing_meta_datasets/materializing-meta-datasets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5f6904e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "from bigbio.dataloader import BigBioConfigHelpers\n",
    "from bigbio.utils.constants import Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be90c109",
   "metadata": {},
   "source": [
    "# BigBioConfigHelpers\n",
    "\n",
    "Start by creating an instance of BigBioConfigHelpers. This will help locate and filter datasets available in the BigBIO package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d250c451",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "found 417 dataset configs from 122 datasets\n"
     ]
    }
   ],
   "source": [
    "conhelps = BigBioConfigHelpers()\n",
    "print(\"found {} dataset configs from {} datasets\".format(\n",
    "    len(conhelps),\n",
    "    len(conhelps.available_dataset_names)\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51b1c5d",
   "metadata": {},
   "source": [
    "Each dataset has at least one source config and at least one bigbio config. Source configs attempt to preserve the original structure of the dataset while bigbio configs are normalized into one of several bigbio [task schemas](https://github.com/bigscience-workshop/biomedical/blob/master/task_schemas.md). Some datasets have several source configs and/or bigbio configs (e.g. multi-lingual datasets or datasets supporting multiple cross-validation folds). This is why the number of configs is greater than twice the number of datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "918fc969",
   "metadata": {},
   "source": [
    "### Examine One Helper\n",
    "\n",
    "conhelps is list-like and elements can be accesed with integer indices. Lets examine one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "950deb8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BigBioConfigHelper(script='/home/galtay/repos/biomedical/bigbio/hub/hub_repos/an_em/an_em.py',\n",
      "                   dataset_name='an_em',\n",
      "                   tasks={'COREFERENCE_RESOLUTION',\n",
      "                          'NAMED_ENTITY_RECOGNITION',\n",
      "                          'RELATION_EXTRACTION'},\n",
      "                   languages=['English'],\n",
      "                   config=BigBioConfig(name='an_em_bigbio_kb',\n",
      "                                       version=1.0.0,\n",
      "                                       data_dir=None,\n",
      "                                       data_files=None,\n",
      "                                       description='AnEM BigBio schema',\n",
      "                                       schema='bigbio_kb',\n",
      "                                       subset_id='an_em'),\n",
      "                   is_local=False,\n",
      "                   is_pubmed=True,\n",
      "                   is_bigbio_schema=True,\n",
      "                   bigbio_schema_caps='KB',\n",
      "                   is_large=False,\n",
      "                   is_resource=False,\n",
      "                   is_default=False,\n",
      "                   is_broken=False,\n",
      "                   bigbio_version='1.0.0',\n",
      "                   source_version='1.0.4',\n",
      "                   citation='@inproceedings{ohta-etal-2012-open,\\n'\n",
      "                            '  author    = {Ohta, Tomoko and Pyysalo, Sampo '\n",
      "                            \"and Tsujii, Jun{'}ichi and Ananiadou, Sophia},\\n\"\n",
      "                            '  title     = {Open-domain Anatomical Entity '\n",
      "                            'Mention Detection},\\n'\n",
      "                            '  journal   = {},\\n'\n",
      "                            '  volume    = {W12-43},\\n'\n",
      "                            '  year      = {2012},\\n'\n",
      "                            '  url       = '\n",
      "                            '{https://aclanthology.org/W12-4304},\\n'\n",
      "                            '  doi       = {},\\n'\n",
      "                            '  biburl    = {},\\n'\n",
      "                            '  bibsource = {},\\n'\n",
      "                            '  publisher = {Association for Computational '\n",
      "                            'Linguistics}\\n'\n",
      "                            '}\\n',\n",
      "                   description='AnEM corpus is a domain- and '\n",
      "                               'species-independent resource manually '\n",
      "                               'annotated for anatomical\\n'\n",
      "                               'entity mentions using a fine-grained '\n",
      "                               'classification system. The corpus consists of '\n",
      "                               '500 documents\\n'\n",
      "                               '(over 90,000 words) selected randomly from '\n",
      "                               'citation abstracts and full-text papers with\\n'\n",
      "                               'the aim of making the corpus representative of '\n",
      "                               'the entire available biomedical scientific\\n'\n",
      "                               'literature. The corpus annotation covers '\n",
      "                               'mentions of both healthy and pathological '\n",
      "                               'anatomical\\n'\n",
      "                               'entities and contains over 3,000 annotated '\n",
      "                               'mentions.\\n',\n",
      "                   homepage='http://www.nactem.ac.uk/anatomy/',\n",
      "                   display_name='AnEM',\n",
      "                   license='Creative Commons Attribution Share Alike 3.0 '\n",
      "                           'Unported')\n"
     ]
    }
   ],
   "source": [
    "pprint(conhelps[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323b6e00",
   "metadata": {},
   "source": [
    "### Show All Available Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f9eb7c61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['an_em', 'anat_em', 'ask_a_patient', 'bc5cdr', 'bc7_litcovid', 'bio_sim_verb', 'bio_simlex', 'bioasq_2021_mesinesp', 'bioasq_task_b', 'bioasq_task_c_2017', 'bioinfer', 'biology_how_why_corpus', 'biomrc', 'bionlp_shared_task_2009', 'bionlp_st_2011_epi', 'bionlp_st_2011_ge', 'bionlp_st_2011_id', 'bionlp_st_2011_rel', 'bionlp_st_2013_cg', 'bionlp_st_2013_ge', 'bionlp_st_2013_gro', 'bionlp_st_2013_pc', 'bionlp_st_2019_bb', 'biored', 'biorelex', 'bioscope', 'biosses', 'blurb', 'cantemist', 'cas', 'cellfinder', 'chebi_nactem', 'chemdner', 'chemprot', 'chia', 'citation_gia_test_collection', 'codiesp', 'cpi', 'ctebmsp', 'ddi_corpus', 'distemist', 'drugprot', 'ebm_pico', 'ehr_rel', 'essai', 'euadr', 'evidence_inference', 'gad', 'genetag', 'genia_ptm_event_corpus', 'genia_relation_corpus', 'genia_term_corpus', 'geokhoj_v1', 'gnormplus', 'hallmarks_of_cancer', 'hprd50', 'iepa', 'jnlpba', 'linnaeus', 'lll', 'mayosrs', 'med_qa', 'medal', 'meddialog', 'meddocan', 'medhop', 'medical_data', 'mediqa_nli', 'mediqa_qa', 'mediqa_rqe', 'medmentions', 'mednli', 'meqsum', 'minimayosrs', 'mirna', 'mlee', 'mqp', 'msh_wsd', 'muchmore', 'multi_xscience', 'n2c2_2006_deid', 'n2c2_2006_smokers', 'n2c2_2008', 'n2c2_2009', 'n2c2_2010', 'n2c2_2011', 'n2c2_2014_deid', 'n2c2_2018_track1', 'n2c2_2018_track2', 'ncbi_disease', 'nlm_gene', 'nlm_wsd', 'nlmchem', 'ntcir_13_medweb', 'osiris', 'paramed', 'pdr', 'pharmaconer', 'pico_extraction', 'pmc_patients', 'progene', 'psytar', 'pubhealth', 'pubmed_qa', 'pubtator_central', 'quaero', 'scai_chemical', 'scai_disease', 'scicite', 'scielo', 'scifact', 'sciq', 'scitail', 'seth_corpus', 'spl_adr_200db', 'swedish_medical_ner', 'tmvar_v1', 'tmvar_v2', 'tmvar_v3', 'twadrl', 'umnsrs', 'verspoor_2013']\n"
     ]
    }
   ],
   "source": [
    "print(conhelps.available_dataset_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb92025",
   "metadata": {},
   "source": [
    "### Show Helpers for Specific Dataset\n",
    "\n",
    "We can also get the helpers for a specific dataset using the dataset name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed60355b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "BigBioConfig(name='bc5cdr_source',\n",
      "             version=1.5.16,\n",
      "             data_dir=None,\n",
      "             data_files=None,\n",
      "             description='BC5CDR source schema',\n",
      "             schema='source',\n",
      "             subset_id='bc5cdr')\n",
      "BigBioConfig(name='bc5cdr_bigbio_kb',\n",
      "             version=1.0.0,\n",
      "             data_dir=None,\n",
      "             data_files=None,\n",
      "             description='BC5CDR simplified BigBio schema',\n",
      "             schema='bigbio_kb',\n",
      "             subset_id='bc5cdr')\n"
     ]
    }
   ],
   "source": [
    "bc5cdr_helpers = conhelps.for_dataset(\"bc5cdr\")\n",
    "print(len(bc5cdr_helpers))\n",
    "pprint(bc5cdr_helpers[0].config)\n",
    "pprint(bc5cdr_helpers[1].config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61c9a92",
   "metadata": {},
   "source": [
    "# Loading Datasets\n",
    "\n",
    "Each config helper provides a wrapper to the [load_dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/loading_methods#datasets.load_dataset) function from Huggingface's [datasets](https://huggingface.co/docs/datasets/) package. This wrapper will automatically populate the first two arguments of load_dataset,\n",
    "\n",
    "* path: path to the dataloader script\n",
    "* name: name of the dataset configuration\n",
    "\n",
    "If you have a specific dataset and config in mind, you can,\n",
    "\n",
    "* fetch the helper from conhelps with the for_config_name method\n",
    "* load the dataset using the load_dataset wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f27e0146",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'path': 'bigbio/bc5cdr', 'name': 'bc5cdr_source'}\n",
      "{'path': 'bigbio/bc5cdr', 'name': 'bc5cdr_bigbio_kb'}\n"
     ]
    }
   ],
   "source": [
    "# these are the kwargs that will be passed to the datasets.load_dataset function by the wrapper function below\n",
    "print(conhelps.for_config_name(\"bc5cdr_source\").get_load_dataset_kwargs())\n",
    "print(conhelps.for_config_name(\"bc5cdr_bigbio_kb\").get_load_dataset_kwargs())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "520925a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bc5cdr (/home/galtay/.cache/huggingface/datasets/bigbio___bc5cdr/bc5cdr_source/1.5.16/68f03988d9e501c974d9f9987183bf06474858d1318ed0d4e51cfc4584f0f51f)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "327d8cf833ef4e6da0e9496265365551",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bc5cdr (/home/galtay/.cache/huggingface/datasets/bigbio___bc5cdr/bc5cdr_bigbio_kb/1.0.0/68f03988d9e501c974d9f9987183bf06474858d1318ed0d4e51cfc4584f0f51f)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d2d49a9b8e84ff683305b968ba02549",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# below is equivalent to load_dataset(path=\"bigbio/bc5cdr\", name=\"bc5cdr_source\")\n",
    "bc5cdr_source = conhelps.for_config_name(\"bc5cdr_source\").load_dataset()\n",
    "\n",
    "# below is equivalent to load_dataset(path=\"bigbio/bc5cdr\", \"name=\"bc5cdr_bigbio_kb\")\n",
    "bc5cdr_bigbio = conhelps.for_config_name(\"bc5cdr_bigbio_kb\").load_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca4e1b97",
   "metadata": {},
   "source": [
    "This wrapper function will pass through any other kwargs you may need to use. For example data_dir for datasets that are not public,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2824eb8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note this will not work unless you have the n2c2 dataset locally\n",
    "\n",
    "#n2c2_2011_source = (\n",
    "#    conhelps.\n",
    "#    for_config_name(\"n2c2_2011_source\").\n",
    "#    load_dataset(data_dir=\"/path/to/n2c2_2011/data\")\n",
    "#)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3445d0cb",
   "metadata": {},
   "source": [
    "# Filter and Load Multiple Datasets\n",
    "\n",
    "You can use any attribute of a BigBioConfigHelper to filter the collection. Here are some examples,"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "547b2cc3",
   "metadata": {},
   "source": [
    "### Source schema for n2c2 datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82433fe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "n2c2_source_helpers = conhelps.filtered(\n",
    "    lambda x:\n",
    "        x.dataset_name.startswith(\"n2c2\")\n",
    "        and not x.is_bigbio_schema\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d5bda24f",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('n2c2_2006_deid', 'n2c2_2006_deid_source'),\n",
       " ('n2c2_2006_smokers', 'n2c2_2006_smokers_source'),\n",
       " ('n2c2_2008', 'n2c2_2008_source'),\n",
       " ('n2c2_2009', 'n2c2_2009_source'),\n",
       " ('n2c2_2010', 'n2c2_2010_source'),\n",
       " ('n2c2_2011', 'n2c2_2011_source'),\n",
       " ('n2c2_2014_deid', 'n2c2_2014_source'),\n",
       " ('n2c2_2018_track1', 'n2c2_2018_track1_source'),\n",
       " ('n2c2_2018_track2', 'n2c2_2018_track2_source')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(ch.dataset_name, ch.config.name) for ch in n2c2_source_helpers]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eb036cb",
   "metadata": {},
   "source": [
    "### BigBIO schema datasets that are public and support textual entailment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "837043e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "entailment_helpers = conhelps.filtered(\n",
    "    lambda x:\n",
    "        x.is_bigbio_schema\n",
    "        and not x.is_local\n",
    "        and Tasks.TEXTUAL_ENTAILMENT.name in x.tasks\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "edc6cc5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('scitail', 'scitail_bigbio_te')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(ch.dataset_name, ch.config.name) for ch in entailment_helpers]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4060da45",
   "metadata": {},
   "source": [
    "### BigBIO schema datasets that are public and not \"large\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "60b7bfc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "bb_public_helpers = conhelps.filtered(\n",
    "    lambda x:\n",
    "        x.is_bigbio_schema\n",
    "        and not x.is_local\n",
    "        and not x.is_large\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4934837",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('an_em', 'an_em_bigbio_kb'),\n",
       " ('anat_em', 'anat_em_bigbio_kb'),\n",
       " ('ask_a_patient', 'ask_a_patient_bigbio_kb'),\n",
       " ('bc5cdr', 'bc5cdr_bigbio_kb'),\n",
       " ('bc7_litcovid', 'bc7_litcovid_bigbio_text'),\n",
       " ('bio_sim_verb', 'bio_sim_verb_bigbio_pairs'),\n",
       " ('bio_simlex', 'bio_simlex_bigbio_pairs'),\n",
       " ('bioasq_2021_mesinesp', 'bioasq_2021_mesinesp_subtrack1_all_bigbio_text'),\n",
       " ('bioasq_2021_mesinesp',\n",
       "  'bioasq_2021_mesinesp_subtrack1_only_articles_bigbio_text'),\n",
       " ('bioasq_2021_mesinesp', 'bioasq_2021_mesinesp_subtrack2_bigbio_text'),\n",
       " ('bioasq_2021_mesinesp', 'bioasq_2021_mesinesp_subtrack3_bigbio_text'),\n",
       " ('bioinfer', 'bioinfer_bigbio_kb'),\n",
       " ('biology_how_why_corpus', 'biology_how_why_corpus_bigbio_qa'),\n",
       " ('biomrc', 'biomrc_small_A_bigbio_qa'),\n",
       " ('biomrc', 'biomrc_tiny_A_bigbio_qa'),\n",
       " ('biomrc', 'biomrc_small_B_bigbio_qa'),\n",
       " ('biomrc', 'biomrc_tiny_B_bigbio_qa'),\n",
       " ('bionlp_shared_task_2009', 'bionlp_shared_task_2009_bigbio_kb'),\n",
       " ('bionlp_st_2011_epi', 'bionlp_st_2011_epi_bigbio_kb'),\n",
       " ('bionlp_st_2011_ge', 'bionlp_st_2011_ge_bigbio_kb'),\n",
       " ('bionlp_st_2011_id', 'bionlp_st_2011_id_bigbio_kb'),\n",
       " ('bionlp_st_2011_rel', 'bionlp_st_2011_rel_bigbio_kb'),\n",
       " ('bionlp_st_2013_cg', 'bionlp_st_2013_cg_bigbio_kb'),\n",
       " ('bionlp_st_2013_ge', 'bionlp_st_2013_ge_bigbio_kb'),\n",
       " ('bionlp_st_2013_gro', 'bionlp_st_2013_gro_bigbio_kb'),\n",
       " ('bionlp_st_2013_pc', 'bionlp_st_2013_pc_bigbio_kb'),\n",
       " ('bionlp_st_2019_bb', 'bionlp_st_2019_bb_bigbio_kb'),\n",
       " ('biored', 'biored_bigbio_kb'),\n",
       " ('biorelex', 'biorelex_bigbio_kb'),\n",
       " ('bioscope', 'bioscope_bigbio_kb'),\n",
       " ('bioscope', 'bioscope_abstracts_bigbio_kb'),\n",
       " ('bioscope', 'bioscope_papers_bigbio_kb'),\n",
       " ('bioscope', 'bioscope_medical_texts_bigbio_kb'),\n",
       " ('biosses', 'biosses_bigbio_pairs'),\n",
       " ('cantemist', 'cantemist_bigbio_kb'),\n",
       " ('cantemist', 'cantemist_bigbio_text'),\n",
       " ('cellfinder', 'cellfinder_bigbio_kb'),\n",
       " ('cellfinder', 'cellfinder_splits_bigbio_kb'),\n",
       " ('chebi_nactem', 'chebi_nactem_abstr_ann1_bigbio_kb'),\n",
       " ('chebi_nactem', 'chebi_nactem_abstr_ann2_bigbio_kb'),\n",
       " ('chebi_nactem', 'chebi_nactem_fullpaper_bigbio_kb'),\n",
       " ('chemdner', 'chemdner_bigbio_kb'),\n",
       " ('chemdner', 'chemdner_bigbio_text'),\n",
       " ('chemprot', 'chemprot_bigbio_kb'),\n",
       " ('chia', 'chia_bigbio_kb'),\n",
       " ('citation_gia_test_collection', 'citation_gia_test_collection_bigbio_kb'),\n",
       " ('codiesp', 'codiesp_D_bigbio_text'),\n",
       " ('codiesp', 'codiesp_P_bigbio_text'),\n",
       " ('codiesp', 'codiesp_X_bigbio_kb'),\n",
       " ('codiesp', 'codiesp_extra_mesh_bigbio_text'),\n",
       " ('codiesp', 'codiesp_extra_cie_bigbio_text'),\n",
       " ('cpi', 'cpi_bigbio_kb'),\n",
       " ('ctebmsp', 'ctebmsp_abstracts_bigbio_kb'),\n",
       " ('ctebmsp', 'ctebmsp_eudract_bigbio_kb'),\n",
       " ('ddi_corpus', 'ddi_corpus_bigbio_kb'),\n",
       " ('distemist', 'distemist_entities_bigbio_kb'),\n",
       " ('distemist', 'distemist_linking_bigbio_kb'),\n",
       " ('drugprot', 'drugprot_bigbio_kb'),\n",
       " ('ebm_pico', 'ebm_pico_bigbio_kb'),\n",
       " ('ehr_rel', 'ehr_rel_bigbio_pairs'),\n",
       " ('euadr', 'euadr_bigbio_kb'),\n",
       " ('evidence_inference', 'evidence-inference_bigbio_qa'),\n",
       " ('gad', 'gad_fold0_bigbio_text'),\n",
       " ('gad', 'gad_fold1_bigbio_text'),\n",
       " ('gad', 'gad_fold2_bigbio_text'),\n",
       " ('gad', 'gad_fold3_bigbio_text'),\n",
       " ('gad', 'gad_fold4_bigbio_text'),\n",
       " ('gad', 'gad_fold5_bigbio_text'),\n",
       " ('gad', 'gad_fold6_bigbio_text'),\n",
       " ('gad', 'gad_fold7_bigbio_text'),\n",
       " ('gad', 'gad_fold8_bigbio_text'),\n",
       " ('gad', 'gad_fold9_bigbio_text'),\n",
       " ('gad', 'gad_blurb_bigbio_text'),\n",
       " ('genetag', 'genetaggold_bigbio_kb'),\n",
       " ('genetag', 'genetagcorrect_bigbio_kb'),\n",
       " ('genia_ptm_event_corpus', 'genia_ptm_event_corpus_bigbio_kb'),\n",
       " ('genia_relation_corpus', 'genia_relation_corpus_bigbio_kb'),\n",
       " ('genia_term_corpus', 'genia_term_corpus_bigbio_kb'),\n",
       " ('geokhoj_v1', 'geokhoj_v1_bigbio_text'),\n",
       " ('gnormplus', 'gnormplus_bigbio_kb'),\n",
       " ('hallmarks_of_cancer', 'hallmarks_of_cancer_bigbio_text'),\n",
       " ('hprd50', 'hprd50_bigbio_kb'),\n",
       " ('iepa', 'iepa_bigbio_kb'),\n",
       " ('jnlpba', 'jnlpba_bigbio_kb'),\n",
       " ('linnaeus', 'linnaeus_bigbio_kb'),\n",
       " ('linnaeus', 'linnaeus_filtered_bigbio_kb'),\n",
       " ('lll', 'lll_bigbio_kb'),\n",
       " ('mayosrs', 'mayosrs_bigbio_pairs'),\n",
       " ('med_qa', 'med_qa_en_bigbio_qa'),\n",
       " ('med_qa', 'med_qa_zh_bigbio_qa'),\n",
       " ('med_qa', 'med_qa_tw_bigbio_qa'),\n",
       " ('med_qa', 'med_qa_tw_en_bigbio_qa'),\n",
       " ('med_qa', 'med_qa_tw_zh_bigbio_qa'),\n",
       " ('meddialog', 'meddialog_en_bigbio_text'),\n",
       " ('meddocan', 'meddocan_bigbio_kb'),\n",
       " ('medhop', 'medhop_bigbio_qa'),\n",
       " ('mediqa_qa', 'mediqa_qa_bigbio_qa'),\n",
       " ('mediqa_rqe', 'mediqa_rqe_bigbio_pairs'),\n",
       " ('medmentions', 'medmentions_full_bigbio_kb'),\n",
       " ('medmentions', 'medmentions_st21pv_bigbio_kb'),\n",
       " ('meqsum', 'meqsum_bigbio_t2t'),\n",
       " ('minimayosrs', 'minimayosrs_bigbio_pairs'),\n",
       " ('mirna', 'mirna_bigbio_kb'),\n",
       " ('mlee', 'mlee_bigbio_kb'),\n",
       " ('mqp', 'mqp_bigbio_pairs'),\n",
       " ('muchmore', 'muchmore_bigbio_kb'),\n",
       " ('muchmore', 'muchmore_en_bigbio_kb'),\n",
       " ('muchmore', 'muchmore_de_bigbio_kb'),\n",
       " ('muchmore', 'muchmore_bigbio_t2t'),\n",
       " ('multi_xscience', 'multi_xscience_bigbio_t2t'),\n",
       " ('ncbi_disease', 'ncbi_disease_bigbio_kb'),\n",
       " ('nlm_gene', 'nlm_gene_bigbio_kb'),\n",
       " ('nlmchem', 'nlmchem_bigbio_kb'),\n",
       " ('nlmchem', 'nlmchem_bigbio_text'),\n",
       " ('osiris', 'osiris_bigbio_kb'),\n",
       " ('paramed', 'paramed_bigbio_t2t'),\n",
       " ('pdr', 'pdr_bigbio_kb'),\n",
       " ('pharmaconer', 'pharmaconer_bigbio_kb'),\n",
       " ('pharmaconer', 'pharmaconer_bigbio_text'),\n",
       " ('pico_extraction', 'pico_extraction_bigbio_kb'),\n",
       " ('pmc_patients', 'pmc_patients_bigbio_pairs'),\n",
       " ('progene', 'progene_bigbio_kb'),\n",
       " ('pubhealth', 'pubhealth_bigbio_pairs'),\n",
       " ('pubmed_qa', 'pubmed_qa_artificial_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_unlabeled_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold0_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold1_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold2_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold3_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold4_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold5_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold6_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold7_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold8_bigbio_qa'),\n",
       " ('pubmed_qa', 'pubmed_qa_labeled_fold9_bigbio_qa'),\n",
       " ('pubtator_central', 'pubtator_central_sample_bigbio_kb'),\n",
       " ('quaero', 'quaero_emea_bigbio_kb'),\n",
       " ('quaero', 'quaero_medline_bigbio_kb'),\n",
       " ('scai_chemical', 'scai_chemical_bigbio_kb'),\n",
       " ('scai_disease', 'scai_disease_bigbio_kb'),\n",
       " ('scicite', 'scicite_bigbio_text'),\n",
       " ('scielo', 'scielo_en_es_bigbio_t2t'),\n",
       " ('scielo', 'scielo_en_pt_bigbio_t2t'),\n",
       " ('scifact', 'scifact_rationale_bigbio_pairs'),\n",
       " ('scifact', 'scifact_labelprediction_bigbio_pairs'),\n",
       " ('sciq', 'sciq_bigbio_qa'),\n",
       " ('scitail', 'scitail_bigbio_te'),\n",
       " ('seth_corpus', 'seth_corpus_bigbio_kb'),\n",
       " ('spl_adr_200db', 'spl_adr_200db_train_bigbio_kb'),\n",
       " ('spl_adr_200db', 'spl_adr_200db_unannotated_bigbio_kb'),\n",
       " ('swedish_medical_ner', 'swedish_medical_ner_wiki_bigbio_kb'),\n",
       " ('swedish_medical_ner', 'swedish_medical_ner_lt_bigbio_kb'),\n",
       " ('swedish_medical_ner', 'swedish_medical_ner_1177_bigbio_kb'),\n",
       " ('tmvar_v1', 'tmvar_v1_bigbio_kb'),\n",
       " ('tmvar_v2', 'tmvar_v2_bigbio_kb'),\n",
       " ('tmvar_v3', 'tmvar_v3_bigbio_kb'),\n",
       " ('twadrl', 'twadrl_bigbio_kb'),\n",
       " ('umnsrs', 'umnsrs_similarity_mod_bigbio_pairs'),\n",
       " ('umnsrs', 'umnsrs_similarity_bigbio_pairs'),\n",
       " ('umnsrs', 'umnsrs_relatedness_mod_bigbio_pairs'),\n",
       " ('umnsrs', 'umnsrs_relatedness_bigbio_pairs'),\n",
       " ('verspoor_2013', 'verspoor_2013_bigbio_kb')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(ch.dataset_name, ch.config.name) for ch in bb_public_helpers]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1962f52",
   "metadata": {},
   "source": [
    "### Loading filtered datasets\n",
    "\n",
    "Note that the `filtered` method returns another instance of `BigBioConfigHelpers`. This means you can iterate over any of the helpers defined above and load all of the datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "25e5d0f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162\n"
     ]
    }
   ],
   "source": [
    "print(len(bb_public_helpers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "84e9cf45",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset an_em (/home/galtay/.cache/huggingface/datasets/bigbio___an_em/an_em_bigbio_kb/1.0.0/6531a2dc4f5c90c7ee6ebe9ac031d53c179f04090e41ab5259420ea5b6abb8c6)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae810f499bca4204afa60b1cacb7f653",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset anat_em (/home/galtay/.cache/huggingface/datasets/bigbio___anat_em/anat_em_bigbio_kb/1.0.0/26e5c978c82396e267dec0e126b2c1b1e8a63a5dc5ef51225225870a2a3e3d88)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f4cda65127a4b4c9e044ff50bc7fbd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset ask_a_patient (/home/galtay/.cache/huggingface/datasets/bigbio___ask_a_patient/ask_a_patient_bigbio_kb/1.0.0/ec2512406bc8163402097ad96a2851e0525fe0dd080650077188bd53bdb15745)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba18779d54474bd4aa8cc54621c6039d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/30 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bc5cdr (/home/galtay/.cache/huggingface/datasets/bigbio___bc5cdr/bc5cdr_bigbio_kb/1.0.0/68f03988d9e501c974d9f9987183bf06474858d1318ed0d4e51cfc4584f0f51f)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f48cd6d0402448f0be1e42d598f4b4cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bc7_litcovid (/home/galtay/.cache/huggingface/datasets/bigbio___bc7_litcovid/bc7_litcovid_bigbio_text/1.0.0/772ea31933b422bf949a43dd6d13f6155d5cc2058e180f2616cede06040a8c11)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29739c46657c40c094cf31acde45cbcb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bio_sim_verb (/home/galtay/.cache/huggingface/datasets/bigbio___bio_sim_verb/bio_sim_verb_bigbio_pairs/1.0.0/8ab449c032843c956c6ffdbdecfc16971d825d7c6d167765343ac197f1e8ccff)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "151a789fd6bb432bacacf7892450040a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bio_simlex (/home/galtay/.cache/huggingface/datasets/bigbio___bio_simlex/bio_simlex_bigbio_pairs/1.0.0/43ae58c304e1b1283aeb72fb60f2ab506fe04bc7c98b6ea86e0a0e3ea799d6f5)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d74865c3c8c647eea304c093c1f423e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bioasq_2021_mesinesp (/home/galtay/.cache/huggingface/datasets/bigbio___bioasq_2021_mesinesp/bioasq_2021_mesinesp_subtrack1_all_bigbio_text/1.0.0/2efb6ae440bb40e4b5821e1ca05e07f40cd0c5a3bcb48579569d1b40c9c6f724)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d62221ef895493bb1af9e2a0b59f4e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bioasq_2021_mesinesp (/home/galtay/.cache/huggingface/datasets/bigbio___bioasq_2021_mesinesp/bioasq_2021_mesinesp_subtrack1_only_articles_bigbio_text/1.0.0/2efb6ae440bb40e4b5821e1ca05e07f40cd0c5a3bcb48579569d1b40c9c6f724)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4ef4306d73d4fd3a5dd6407e401f64e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bioasq_2021_mesinesp (/home/galtay/.cache/huggingface/datasets/bigbio___bioasq_2021_mesinesp/bioasq_2021_mesinesp_subtrack2_bigbio_text/1.0.0/2efb6ae440bb40e4b5821e1ca05e07f40cd0c5a3bcb48579569d1b40c9c6f724)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3791c1e8e9b547df8f66dd5e5018c26d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NOTE the first time you run this cell, the public datasets will be downloaded and cached.\n",
    "# As an example we just load the first 10 datasets \n",
    "\n",
    "bb_public_datasets = {}\n",
    "for helper in bb_public_helpers[:10]:\n",
    "    bb_public_datasets[helper.config.name] = helper.load_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a52519",
   "metadata": {},
   "source": [
    "# Dataset Metadata\n",
    "\n",
    "Each BigBioConfigHelper provides a get_metadata method that will calculate schema specific metadata for configs implementing a BigBIO schema. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "37ac86b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset bc5cdr (/home/galtay/.cache/huggingface/datasets/bigbio___bc5cdr/bc5cdr_bigbio_kb/1.0.0/68f03988d9e501c974d9f9987183bf06474858d1318ed0d4e51cfc4584f0f51f)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce6c0931205d499496d09048f4e2fbec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'train': BigBioKbMetadata(samples_count=500, passages_count=1000, passages_char_count=652177, passages_type_counter={'title': 500, 'abstract': 500}, entities_count=9570, entities_normalized_count=9599, entities_type_counter={'Chemical': 5207, 'Disease': 4363}, entities_db_name_counter={'MESH': 9599}, entities_unique_db_ids_count=1328, events_count=0, events_type_counter={}, events_arguments_count=0, events_arguments_role_counter={}, coreferences_count=0, relations_count=15072, relations_type_counter={'CID': 15072}, relations_db_name_counter={}, relations_unique_db_ids_count=0),\n",
       " 'test': BigBioKbMetadata(samples_count=500, passages_count=1000, passages_char_count=676751, passages_type_counter={'title': 500, 'abstract': 500}, entities_count=9928, entities_normalized_count=9919, entities_type_counter={'Chemical': 5394, 'Disease': 4534}, entities_db_name_counter={'MESH': 9919}, entities_unique_db_ids_count=1315, events_count=0, events_type_counter={}, events_arguments_count=0, events_arguments_role_counter={}, coreferences_count=0, relations_count=16250, relations_type_counter={'CID': 16250}, relations_db_name_counter={}, relations_unique_db_ids_count=0),\n",
       " 'validation': BigBioKbMetadata(samples_count=500, passages_count=1000, passages_char_count=646528, passages_type_counter={'title': 500, 'abstract': 500}, entities_count=9773, entities_normalized_count=9817, entities_type_counter={'Chemical': 5352, 'Disease': 4421}, entities_db_name_counter={'MESH': 9817}, entities_unique_db_ids_count=1264, events_count=0, events_type_counter={}, events_arguments_count=0, events_arguments_role_counter={}, coreferences_count=0, relations_count=16491, relations_type_counter={'CID': 16491}, relations_db_name_counter={}, relations_unique_db_ids_count=0)}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conhelps.for_config_name('bc5cdr_bigbio_kb').get_metadata()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7d84e4d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}