general-examples/getting-started-with-epigraphdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "spoken-decline",
   "metadata": {
    "id": "uGuMTl7Wy2PM",
    "papermill": {
     "duration": 0.026869,
     "end_time": "2021-03-22T10:46:08.924783",
     "exception": false,
     "start_time": "2021-03-22T10:46:08.897914",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Getting started with EpiGraphDB in Python\n",
    "\n",
    "This notebook is provided as a brief introductory guide to working with the EpiGraphDB platform through Python. Here we will demonstrate a few basic operations that can be carried out using the platform, but for more advanced methods please refer to the [API endpoint documentation](http://docs.epigraphdb.org/api/api-endpoints/).\n",
    "\n",
    "A Python wrapper for EpiGraphDB's API is currently in the works, but for now we will be querying it directly using the `requests` library - knowledge of this package is advantageous but not essential."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "alternative-breakfast",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:08.981591Z",
     "iopub.status.busy": "2021-03-22T10:46:08.981131Z",
     "iopub.status.idle": "2021-03-22T10:46:09.230873Z",
     "shell.execute_reply": "2021-03-22T10:46:09.230156Z"
    },
    "id": "Ncm8DbWly2PP",
    "papermill": {
     "duration": 0.279664,
     "end_time": "2021-03-22T10:46:09.231048",
     "exception": false,
     "start_time": "2021-03-22T10:46:08.951384",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from pprint import pprint\n",
    "\n",
    "import requests\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "basic-essex",
   "metadata": {
    "id": "fRZvk6Oqy2Pd",
    "papermill": {
     "duration": 0.025877,
     "end_time": "2021-03-22T10:46:09.285934",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.260057",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "First, we will ping the API to check our connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "connected-orbit",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:09.328528Z",
     "iopub.status.busy": "2021-03-22T10:46:09.328168Z",
     "iopub.status.idle": "2021-03-22T10:46:09.330760Z",
     "shell.execute_reply": "2021-03-22T10:46:09.331202Z"
    },
    "papermill": {
     "duration": 0.02045,
     "end_time": "2021-03-22T10:46:09.331349",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.310899",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Default parameters\n",
    "API_URL = \"https://api.epigraphdb.org\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "pending-cylinder",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:09.362703Z",
     "iopub.status.busy": "2021-03-22T10:46:09.362161Z",
     "iopub.status.idle": "2021-03-22T10:46:09.363952Z",
     "shell.execute_reply": "2021-03-22T10:46:09.364515Z"
    },
    "papermill": {
     "duration": 0.019114,
     "end_time": "2021-03-22T10:46:09.364689",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.345575",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "API_URL = \"https://api.epigraphdb.org\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "unavailable-mexican",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:09.398588Z",
     "iopub.status.busy": "2021-03-22T10:46:09.398218Z",
     "iopub.status.idle": "2021-03-22T10:46:09.550357Z",
     "shell.execute_reply": "2021-03-22T10:46:09.549947Z"
    },
    "id": "w9N2Z-mIy2Pd",
    "outputId": "e680f9de-f8e5-42c8-fabc-0e3c56554133",
    "papermill": {
     "duration": 0.169735,
     "end_time": "2021-03-22T10:46:09.550466",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.380731",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.epigraphdb.org\n",
      "If this line gets printed, ping was sucessful.\n"
     ]
    }
   ],
   "source": [
    "# Here we use the .get() method to send a GET request to the /ping endpoint of the API\n",
    "print(API_URL)\n",
    "endpoint = \"/ping\"\n",
    "response_object = requests.get(API_URL + endpoint)\n",
    "\n",
    "# Check that the ping was sucessful\n",
    "try:\n",
    "    response_object.raise_for_status()\n",
    "    print(\"If this line gets printed, ping was sucessful.\")\n",
    "except requests.exceptions.HTTPError as err:\n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appreciated-sympathy",
   "metadata": {
    "id": "UJyGttrGy2Pl",
    "papermill": {
     "duration": 0.018565,
     "end_time": "2021-03-22T10:46:09.587542",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.568977",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "***\n",
    "## 1. Using EpiGraphDB to obtain biological mappings\n",
    "\n",
    "In this first section, we will take an arbitrary list of genes and query the EpiGraph API to find the proteins that they map to. We will be using the `POST` HTTP method which requires its parameters to be passed in JSON format, a conversion that is easy to do using the `json` library. To find the correct names of the parameters that we are about to set, we can use the [EpiGraphDB API documentation](http://docs.epigraphdb.org/api/api-endpoints/) and find the endpoint of interest. From there we simply read off the parameters that we want to pass, and can take a look at the example request as a reference if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "virtual-adolescent",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:09.632215Z",
     "iopub.status.busy": "2021-03-22T10:46:09.631847Z",
     "iopub.status.idle": "2021-03-22T10:46:09.985441Z",
     "shell.execute_reply": "2021-03-22T10:46:09.985981Z"
    },
    "id": "2HQaupzoy2Pm",
    "outputId": "7067b90d-8fab-4b2f-d54b-d5686e8dd622",
    "papermill": {
     "duration": 0.379908,
     "end_time": "2021-03-22T10:46:09.986154",
     "exception": false,
     "start_time": "2021-03-22T10:46:09.606246",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene.name</th>\n",
       "      <th>gene.ensembl_id</th>\n",
       "      <th>protein.uniprot_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TP53</td>\n",
       "      <td>ENSG00000141510</td>\n",
       "      <td>P04637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BRCA1</td>\n",
       "      <td>ENSG00000012048</td>\n",
       "      <td>P38398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TNF</td>\n",
       "      <td>ENSG00000232810</td>\n",
       "      <td>P01375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gene.name  gene.ensembl_id protein.uniprot_id\n",
       "0      TP53  ENSG00000141510             P04637\n",
       "1     BRCA1  ENSG00000012048             P38398\n",
       "2       TNF  ENSG00000232810             P01375"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1.1 Mapping genes to proteins\n",
    "\n",
    "# Set parameters and convert to JSON format\n",
    "params = {\"gene_name_list\": [\"TP53\", \"BRCA1\", \"TNF\"]}\n",
    "json_params = json.dumps(params)\n",
    "\n",
    "# Define which endpoint of the API we would like to connect with\n",
    "endpoint = \"/mappings/gene-to-protein\"\n",
    "\n",
    "# Send the POST request\n",
    "response_object = requests.post(API_URL + endpoint, data=json_params)\n",
    "\n",
    "# Check for successful request\n",
    "response_object.raise_for_status()\n",
    "\n",
    "results = response_object.json()[\"results\"]\n",
    "gene_protein_df = pd.json_normalize(results)\n",
    "\n",
    "gene_protein_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "colored-occupation",
   "metadata": {
    "id": "XkfSu116y2Pq",
    "papermill": {
     "duration": 0.027376,
     "end_time": "2021-03-22T10:46:10.044305",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.016929",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In the above cell, we queried EpiGraphDB for the proteins that have been mapped to the genes *TP53*, *BRCA1*, and *TNF*. Our query went through successfully and we received an associated protein for each. The columns in our output dataframe take the general form `entity.property` and this will remain consistent throughout this notebook. \n",
    "\n",
    "Specific descriptions for the properties of each entity can be found in EpiGraphDB's [data dictionary](https://docs.epigraphdb.org/graph-database/meta-nodes/). Simply click on the relevant entity in the table of contents  on the right hand side (or scroll down to the relevant section), then locate the property of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "heavy-metabolism",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:10.091284Z",
     "iopub.status.busy": "2021-03-22T10:46:10.088874Z",
     "iopub.status.idle": "2021-03-22T10:46:10.265365Z",
     "shell.execute_reply": "2021-03-22T10:46:10.264751Z"
    },
    "id": "0yOtBAk-y2Pq",
    "outputId": "8d1701ee-f69a-48e6-a859-f9e13f69b76a",
    "papermill": {
     "duration": 0.196421,
     "end_time": "2021-03-22T10:46:10.265511",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.069090",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uniprot_id</th>\n",
       "      <th>pathway_count</th>\n",
       "      <th>pathway_reactome_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>P04637</td>\n",
       "      <td>84</td>\n",
       "      <td>[R-HSA-983231, R-HSA-9006925, R-HSA-8953897, R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>P38398</td>\n",
       "      <td>38</td>\n",
       "      <td>[R-HSA-912446, R-HSA-8953750, R-HSA-74160, R-H...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>P01375</td>\n",
       "      <td>15</td>\n",
       "      <td>[R-HSA-75893, R-HSA-73887, R-HSA-6785807, R-HS...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  uniprot_id  pathway_count                                pathway_reactome_id\n",
       "0     P04637             84  [R-HSA-983231, R-HSA-9006925, R-HSA-8953897, R...\n",
       "1     P38398             38  [R-HSA-912446, R-HSA-8953750, R-HSA-74160, R-H...\n",
       "2     P01375             15  [R-HSA-75893, R-HSA-73887, R-HSA-6785807, R-HS..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1.2 Proteins to pathways\n",
    "\n",
    "# As above, this is another POST request, so we need our data in JSON format\n",
    "json_params = json.dumps(\n",
    "    {\"uniprot_id_list\": gene_protein_df[\"protein.uniprot_id\"].tolist()}\n",
    ")\n",
    "\n",
    "# Send the request\n",
    "endpoint = \"/protein/in-pathway\"\n",
    "response_object = requests.post(API_URL + endpoint, data=json_params)\n",
    "\n",
    "# Check for successful request\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Store results\n",
    "results = response_object.json()[\"results\"]\n",
    "protein_pathway_df = pd.json_normalize(results)\n",
    "\n",
    "protein_pathway_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dental-english",
   "metadata": {
    "id": "KySWgtOby2Px",
    "papermill": {
     "duration": 0.028369,
     "end_time": "2021-03-22T10:46:10.321678",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.293309",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Above, we took the proteins that had been mapped to our genes of interest and queried the platform for their associated pathway data. The API found multiple such pathways for each gene and has returned the respective reactome IDs to us as lists.\n",
    "\n",
    "\n",
    "It is worth noting here that so far we have only been accessing the `'results'` key in the nested dictionairy returned by the `.json()` method of our response object. The other available key is `'metadata'` (see the output below) which provides us with information about the request itself, including the specific [Cypher](https://neo4j.com/developer/cypher/) query that the platform ran to get these results. If you would like to know more about the use of Cypher in these requests, there is a section dedicated to this at the end of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "domestic-terminology",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:10.366418Z",
     "iopub.status.busy": "2021-03-22T10:46:10.366066Z",
     "iopub.status.idle": "2021-03-22T10:46:10.368683Z",
     "shell.execute_reply": "2021-03-22T10:46:10.368343Z"
    },
    "id": "YFJBxuway2Py",
    "outputId": "f5adec6e-6c94-4560-be4a-cd24963eb21b",
    "papermill": {
     "duration": 0.021786,
     "end_time": "2021-03-22T10:46:10.368760",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.346974",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'empty_results': False,\n",
      " 'query': 'MATCH p=(protein:Protein)-[r:PROTEIN_IN_PATHWAY]-(pathway:Pathway) '\n",
      "          \"WHERE protein.uniprot_id IN ['P04637', 'P38398', 'P01375'] RETURN \"\n",
      "          'protein.uniprot_id AS uniprot_id, count(p) AS pathway_count, '\n",
      "          'collect(pathway.id) AS pathway_reactome_id',\n",
      " 'total_seconds': 0.013254}\n"
     ]
    }
   ],
   "source": [
    "metadata = response_object.json()[\"metadata\"]\n",
    "\n",
    "pprint(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rocky-deficit",
   "metadata": {
    "id": "5mY0MSXby2P2",
    "papermill": {
     "duration": 0.015369,
     "end_time": "2021-03-22T10:46:10.399683",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.384314",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "***\n",
    "## 2. Epidemiological relationship analysis\n",
    "\n",
    "In the cell below, we will query EpiGraphDB to get metadata relating to GWAS studies of a target trait- body mass index. Following that, queries will be performed to get pre-computed Mendelian randomization (MR) results involving the same trait (generated by [MR-Eve](https://github.com/explodecomputer/mr-eve)). \n",
    "\n",
    "Here we will be using a different HTTP method than before- the `GET` method, which is in fact easier to use in Python because the parameters can be passed directly as a dictionary. To learn more about the differences between `GET` and `POST`, please see [this guide](https://www.w3schools.com/tags/ref_httpmethods.asp). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "personal-rolling",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:10.438612Z",
     "iopub.status.busy": "2021-03-22T10:46:10.436303Z",
     "iopub.status.idle": "2021-03-22T10:46:10.632355Z",
     "shell.execute_reply": "2021-03-22T10:46:10.631825Z"
    },
    "id": "DfJ6swgRy2P6",
    "outputId": "93b0bfe6-87d0-4638-ad4e-b03c8477d289",
    "papermill": {
     "duration": 0.216894,
     "end_time": "2021-03-22T10:46:10.632458",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.415564",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node.note</th>\n",
       "      <th>node._name</th>\n",
       "      <th>node.year</th>\n",
       "      <th>node.mr</th>\n",
       "      <th>node.author</th>\n",
       "      <th>node.sex</th>\n",
       "      <th>node.pmid</th>\n",
       "      <th>node.population</th>\n",
       "      <th>node.sample_size</th>\n",
       "      <th>node.nsnp</th>\n",
       "      <th>node.build</th>\n",
       "      <th>node.trait</th>\n",
       "      <th>node._source</th>\n",
       "      <th>node.id</th>\n",
       "      <th>node._id</th>\n",
       "      <th>node.subcategory</th>\n",
       "      <th>node.category</th>\n",
       "      <th>node.sd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dominance model? If so then not necessarily of...</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Wood</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>26961502.0</td>\n",
       "      <td>European</td>\n",
       "      <td>120286.0</td>\n",
       "      <td>8654252</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ieu-a-1089</td>\n",
       "      <td>ieu-a-1089</td>\n",
       "      <td>Anthropometric</td>\n",
       "      <td>Risk factor</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Locke AE</td>\n",
       "      <td>Females</td>\n",
       "      <td>25673413.0</td>\n",
       "      <td>European</td>\n",
       "      <td>171977.0</td>\n",
       "      <td>2494613</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ieu-a-974</td>\n",
       "      <td>ieu-a-974</td>\n",
       "      <td>Anthropometric</td>\n",
       "      <td>Risk factor</td>\n",
       "      <td>4.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Randall JC</td>\n",
       "      <td>Females</td>\n",
       "      <td>23754948.0</td>\n",
       "      <td>European</td>\n",
       "      <td>73137.0</td>\n",
       "      <td>2736876</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ieu-a-95</td>\n",
       "      <td>ieu-a-95</td>\n",
       "      <td>Anthropometric</td>\n",
       "      <td>Risk factor</td>\n",
       "      <td>4.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Akiyama M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28892062.0</td>\n",
       "      <td>East Asian</td>\n",
       "      <td>158284.0</td>\n",
       "      <td>5952516</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ebi-a-GCST004904</td>\n",
       "      <td>ebi-a-GCST004904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Hoffmann TJ</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30108127.0</td>\n",
       "      <td>European</td>\n",
       "      <td>315347.0</td>\n",
       "      <td>27854527</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ebi-a-GCST006368</td>\n",
       "      <td>ebi-a-GCST006368</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           node.note       node._name  \\\n",
       "0  Dominance model? If so then not necessarily of...  Body mass index   \n",
       "1                                                NaN  Body mass index   \n",
       "2                                                NaN  Body mass index   \n",
       "3                                                NaN  Body mass index   \n",
       "4                                                NaN  Body mass index   \n",
       "\n",
       "  node.year node.mr  node.author           node.sex   node.pmid  \\\n",
       "0    2016.0       0         Wood  Males and Females  26961502.0   \n",
       "1    2015.0       1     Locke AE            Females  25673413.0   \n",
       "2    2013.0       1   Randall JC            Females  23754948.0   \n",
       "3    2017.0       1    Akiyama M                NaN  28892062.0   \n",
       "4    2018.0       1  Hoffmann TJ                NaN  30108127.0   \n",
       "\n",
       "  node.population node.sample_size node.nsnp   node.build       node.trait  \\\n",
       "0        European         120286.0   8654252  HG19/GRCh37  Body mass index   \n",
       "1        European         171977.0   2494613  HG19/GRCh37  Body mass index   \n",
       "2        European          73137.0   2736876  HG19/GRCh37  Body mass index   \n",
       "3      East Asian         158284.0   5952516  HG19/GRCh37  Body mass index   \n",
       "4        European         315347.0  27854527  HG19/GRCh37  Body mass index   \n",
       "\n",
       "            node._source           node.id          node._id node.subcategory  \\\n",
       "0  [OpenGWAS-2020-10-13]        ieu-a-1089        ieu-a-1089   Anthropometric   \n",
       "1  [OpenGWAS-2020-10-13]         ieu-a-974         ieu-a-974   Anthropometric   \n",
       "2  [OpenGWAS-2020-10-13]          ieu-a-95          ieu-a-95   Anthropometric   \n",
       "3  [OpenGWAS-2020-10-13]  ebi-a-GCST004904  ebi-a-GCST004904              NaN   \n",
       "4  [OpenGWAS-2020-10-13]  ebi-a-GCST006368  ebi-a-GCST006368              NaN   \n",
       "\n",
       "  node.category node.sd  \n",
       "0   Risk factor     NaN  \n",
       "1   Risk factor    4.77  \n",
       "2   Risk factor    4.77  \n",
       "3           NaN     NaN  \n",
       "4           NaN     NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2.1 Getting GWAS studies from EpiGraphDB\n",
    "\n",
    "# Create a dictionary for the parameters to be passed\n",
    "params = {\"name\": \"Body mass index\"}\n",
    "\n",
    "# Send the request\n",
    "endpoint = \"/meta/nodes/Gwas/search\"\n",
    "response_object = requests.get(API_URL + endpoint, params=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Store the results of the query and display\n",
    "result = response_object.json()[\"results\"]\n",
    "gwas_df = pd.json_normalize(result)\n",
    "\n",
    "gwas_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "random-latex",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:10.681587Z",
     "iopub.status.busy": "2021-03-22T10:46:10.681109Z",
     "iopub.status.idle": "2021-03-22T10:46:11.897099Z",
     "shell.execute_reply": "2021-03-22T10:46:11.897673Z"
    },
    "id": "c8243Kgry2P-",
    "outputId": "540d1918-0119-4a80-be59-7ac11c1fdf34",
    "papermill": {
     "duration": 1.245503,
     "end_time": "2021-03-22T10:46:11.897874",
     "exception": false,
     "start_time": "2021-03-22T10:46:10.652371",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exposure.id</th>\n",
       "      <th>exposure.trait</th>\n",
       "      <th>outcome.id</th>\n",
       "      <th>outcome.trait</th>\n",
       "      <th>mr.b</th>\n",
       "      <th>mr.se</th>\n",
       "      <th>mr.pval</th>\n",
       "      <th>mr.method</th>\n",
       "      <th>mr.selection</th>\n",
       "      <th>mr.moescore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ieu-a-974</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>ebi-a-GCST005062</td>\n",
       "      <td>Fibrinogen levels</td>\n",
       "      <td>0.193038</td>\n",
       "      <td>0.002236</td>\n",
       "      <td>0.0</td>\n",
       "      <td>FE IVW</td>\n",
       "      <td>DF</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ebi-a-GCST004904</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>ukb-b-2303</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>0.594919</td>\n",
       "      <td>0.015287</td>\n",
       "      <td>0.0</td>\n",
       "      <td>FE IVW</td>\n",
       "      <td>Tophits</td>\n",
       "      <td>0.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ieu-a-835</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>ukb-b-18096</td>\n",
       "      <td>Leg fat mass (right)</td>\n",
       "      <td>0.607879</td>\n",
       "      <td>0.013882</td>\n",
       "      <td>0.0</td>\n",
       "      <td>FE IVW</td>\n",
       "      <td>DF + HF</td>\n",
       "      <td>0.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ieu-a-835</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>ukb-b-12854</td>\n",
       "      <td>Arm fat percentage (right)</td>\n",
       "      <td>0.525881</td>\n",
       "      <td>0.013035</td>\n",
       "      <td>0.0</td>\n",
       "      <td>FE IVW</td>\n",
       "      <td>DF + HF</td>\n",
       "      <td>0.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ieu-a-835</td>\n",
       "      <td>Body mass index</td>\n",
       "      <td>ieu-a-93</td>\n",
       "      <td>Overweight</td>\n",
       "      <td>1.671103</td>\n",
       "      <td>0.034076</td>\n",
       "      <td>0.0</td>\n",
       "      <td>FE IVW</td>\n",
       "      <td>Tophits</td>\n",
       "      <td>0.84</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        exposure.id   exposure.trait        outcome.id  \\\n",
       "0         ieu-a-974  Body mass index  ebi-a-GCST005062   \n",
       "1  ebi-a-GCST004904  Body mass index        ukb-b-2303   \n",
       "2         ieu-a-835  Body mass index       ukb-b-18096   \n",
       "3         ieu-a-835  Body mass index       ukb-b-12854   \n",
       "4         ieu-a-835  Body mass index          ieu-a-93   \n",
       "\n",
       "                outcome.trait      mr.b     mr.se  mr.pval mr.method  \\\n",
       "0           Fibrinogen levels  0.193038  0.002236      0.0    FE IVW   \n",
       "1       Body mass index (BMI)  0.594919  0.015287      0.0    FE IVW   \n",
       "2        Leg fat mass (right)  0.607879  0.013882      0.0    FE IVW   \n",
       "3  Arm fat percentage (right)  0.525881  0.013035      0.0    FE IVW   \n",
       "4                  Overweight  1.671103  0.034076      0.0    FE IVW   \n",
       "\n",
       "  mr.selection  mr.moescore  \n",
       "0           DF         1.00  \n",
       "1      Tophits         0.81  \n",
       "2      DF + HF         0.94  \n",
       "3      DF + HF         0.96  \n",
       "4      Tophits         0.84  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2.2 Getting MR results for a trait\n",
    "\n",
    "# Set parameters\n",
    "params = {\"exposure_trait\": \"Body mass index\", \"pval_threshold\": 1e-10}\n",
    "\n",
    "# Send request\n",
    "endpoint = \"/mr\"\n",
    "response_object = requests.get(API_URL + endpoint, params=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Store and display results\n",
    "result = response_object.json()[\"results\"]\n",
    "BMI_MR_df = pd.json_normalize(result)\n",
    "\n",
    "BMI_MR_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "robust-treat",
   "metadata": {
    "id": "HNs1RSRAy2QD",
    "papermill": {
     "duration": 0.030266,
     "end_time": "2021-03-22T10:46:11.958785",
     "exception": false,
     "start_time": "2021-03-22T10:46:11.928519",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The dataframe above displays the results of our query. We requested all traits for which an MR analysis using body mass index as the exposure variable returned a causal estimate with a p-value lower than 1e-10. Information regarding the specific MR parameters, as well as the exposure and outcome variables, has been displayed in the table for all traits that matched our search conditions.\n",
    "\n",
    "In the parameters we set in 2.2, another viable parameter name is `'outcome_trait'` which takes the same type of values as `'exposure_trait'`. Either one or both of these parameters can be passed during an MR query, which allows users to refine which results are returned to them depending on their own analytical preferences."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sharp-electric",
   "metadata": {
    "id": "0U5bVcaiy2QE",
    "papermill": {
     "duration": 0.015838,
     "end_time": "2021-03-22T10:46:11.999150",
     "exception": false,
     "start_time": "2021-03-22T10:46:11.983312",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "***\n",
    "## 3. Looking for literature evidence\n",
    "\n",
    "Accessing information in the literature is a ubiquitous task in research, be it for novel hypothesis generation or as part of evidence triangulation. EpiGraphDB facilitates fast processing of this information by allowing access to a host of literature-mined relationships that have been structured into semantic triples. These take the general form *(subject, predicate, object)* and have been generated using contemporary natural language processing techniques applied to a massive amount of published biomedical research papers by [SemMedDB](https://skr3.nlm.nih.gov/SemMedDB/). In the following section we will query the API for the literature relationship between a given gene and an outcome trait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "asian-seeking",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:12.036013Z",
     "iopub.status.busy": "2021-03-22T10:46:12.035652Z",
     "iopub.status.idle": "2021-03-22T10:46:12.233746Z",
     "shell.execute_reply": "2021-03-22T10:46:12.234307Z"
    },
    "id": "5mHcF50Ky2QF",
    "outputId": "b61dd8f3-9889-48c5-9220-cbb68badfc9e",
    "papermill": {
     "duration": 0.219519,
     "end_time": "2021-03-22T10:46:12.234478",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.014959",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pubmed_id</th>\n",
       "      <th>gene.name</th>\n",
       "      <th>st.predicate</th>\n",
       "      <th>lt.name</th>\n",
       "      <th>lt.id</th>\n",
       "      <th>lt.type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[23131344]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>PREDISPOSES</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[21155887, 17484863]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>NEG_ASSOCIATED_WITH</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[31728561]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>CAUSES</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[21155887, 18383521, 18383363, 25159710, 18341...</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>ASSOCIATED_WITH</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[27852544]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>AFFECTS</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           pubmed_id gene.name  \\\n",
       "0                                         [23131344]     IL23R   \n",
       "1                               [21155887, 17484863]     IL23R   \n",
       "2                                         [31728561]     IL23R   \n",
       "3  [21155887, 18383521, 18383363, 25159710, 18341...     IL23R   \n",
       "4                                         [27852544]     IL23R   \n",
       "\n",
       "          st.predicate                      lt.name     lt.id lt.type  \n",
       "0          PREDISPOSES  Inflammatory Bowel Diseases  C0021390  [dsyn]  \n",
       "1  NEG_ASSOCIATED_WITH  Inflammatory Bowel Diseases  C0021390  [dsyn]  \n",
       "2               CAUSES  Inflammatory Bowel Diseases  C0021390  [dsyn]  \n",
       "3      ASSOCIATED_WITH  Inflammatory Bowel Diseases  C0021390  [dsyn]  \n",
       "4              AFFECTS  Inflammatory Bowel Diseases  C0021390  [dsyn]  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Establish parameters\n",
    "params = {\"gene_name\": \"IL23R\", \"object_name\": \"Inflammatory bowel disease\"}\n",
    "\n",
    "# Send the request\n",
    "endpoint = \"/literature/gene\"\n",
    "response_object = requests.get(API_URL + endpoint, params=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Store the results of the query and display\n",
    "result = response_object.json()[\"results\"]\n",
    "lit_df = pd.json_normalize(result)\n",
    "\n",
    "lit_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "editorial-leather",
   "metadata": {
    "id": "R2QLx5gly2QP",
    "papermill": {
     "duration": 0.029498,
     "end_time": "2021-03-22T10:46:12.294332",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.264834",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The dataframe outputted above shows the results of our query- four unique predicates were found between the gene *IL23R* and the trait *Inflammatory bowel disease* and are displayed in the `st.predicate` column. Our leftmost column contains the pubmed IDs of the papers from which this triple was derived. These IDs allow us to access the respective papers by navigating to `https://pubmed.ncbi.nlm.nih.gov/*insert_pubmed_id_here*`. In this particular case it seems that *ASSOCIATED_WITH* is the most common predicate linking our gene to the trait, but we can't see exactly how many papers there are due to how pandas displays lists. Let's add a paper count to the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "another-february",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:12.342067Z",
     "iopub.status.busy": "2021-03-22T10:46:12.341703Z",
     "iopub.status.idle": "2021-03-22T10:46:12.344856Z",
     "shell.execute_reply": "2021-03-22T10:46:12.344191Z"
    },
    "id": "y-qL52MPy2QQ",
    "outputId": "36e07b79-7424-4e3d-db39-a533798e5ef8",
    "papermill": {
     "duration": 0.030604,
     "end_time": "2021-03-22T10:46:12.344996",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.314392",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pubmed_id</th>\n",
       "      <th>gene.name</th>\n",
       "      <th>st.predicate</th>\n",
       "      <th>lt.name</th>\n",
       "      <th>lt.id</th>\n",
       "      <th>lt.type</th>\n",
       "      <th>publication_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[23131344]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>PREDISPOSES</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[21155887, 17484863]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>NEG_ASSOCIATED_WITH</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[31728561]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>CAUSES</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[21155887, 18383521, 18383363, 25159710, 18341...</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>ASSOCIATED_WITH</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[27852544]</td>\n",
       "      <td>IL23R</td>\n",
       "      <td>AFFECTS</td>\n",
       "      <td>Inflammatory Bowel Diseases</td>\n",
       "      <td>C0021390</td>\n",
       "      <td>[dsyn]</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           pubmed_id gene.name  \\\n",
       "0                                         [23131344]     IL23R   \n",
       "1                               [21155887, 17484863]     IL23R   \n",
       "2                                         [31728561]     IL23R   \n",
       "3  [21155887, 18383521, 18383363, 25159710, 18341...     IL23R   \n",
       "4                                         [27852544]     IL23R   \n",
       "\n",
       "          st.predicate                      lt.name     lt.id lt.type  \\\n",
       "0          PREDISPOSES  Inflammatory Bowel Diseases  C0021390  [dsyn]   \n",
       "1  NEG_ASSOCIATED_WITH  Inflammatory Bowel Diseases  C0021390  [dsyn]   \n",
       "2               CAUSES  Inflammatory Bowel Diseases  C0021390  [dsyn]   \n",
       "3      ASSOCIATED_WITH  Inflammatory Bowel Diseases  C0021390  [dsyn]   \n",
       "4              AFFECTS  Inflammatory Bowel Diseases  C0021390  [dsyn]   \n",
       "\n",
       "   publication_count  \n",
       "0                  1  \n",
       "1                  2  \n",
       "2                  1  \n",
       "3                 21  \n",
       "4                  1  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = [len(papers_list) for papers_list in lit_df[\"pubmed_id\"]]\n",
    "lit_df[\"publication_count\"] = counts\n",
    "\n",
    "lit_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civilian-marathon",
   "metadata": {
    "id": "baG-x7qsy2QZ",
    "papermill": {
     "duration": 0.015927,
     "end_time": "2021-03-22T10:46:12.382549",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.366622",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "-----\n",
    "## 4. EpiGraphDB node search\n",
    "\n",
    "EpiGraphDB stores data as nodes (entities) and edges (relationships) of a wide range of types. The `/meta` endpoints of the API offer us information about the structure of the graph itself- for example, the available classes of nodes can be listed through the `/meta/nodes/list` endpoint. Let’s do that now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "creative-jumping",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:12.419296Z",
     "iopub.status.busy": "2021-03-22T10:46:12.418945Z",
     "iopub.status.idle": "2021-03-22T10:46:12.559780Z",
     "shell.execute_reply": "2021-03-22T10:46:12.560224Z"
    },
    "id": "UzPP4rl4y2Qc",
    "outputId": "1e9a681f-a8b0-4c87-f0a6-e08a7a3cea65",
    "papermill": {
     "duration": 0.161567,
     "end_time": "2021-03-22T10:46:12.560348",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.398781",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Disease',\n",
       " 'Drug',\n",
       " 'Efo',\n",
       " 'Gene',\n",
       " 'Gwas',\n",
       " 'Literature',\n",
       " 'LiteratureTerm',\n",
       " 'Pathway',\n",
       " 'Protein',\n",
       " 'Tissue',\n",
       " 'Variant']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4.1 Getting a list of available meta-nodes\n",
    "\n",
    "# Send the request\n",
    "endpoint = \"/meta/nodes/list\"\n",
    "response_object = requests.get(API_URL + endpoint)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Store the results of the query and display\n",
    "result = response_object.json()\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "authentic-power",
   "metadata": {
    "id": "_n7D0Y4Ey2RK",
    "papermill": {
     "duration": 0.021844,
     "end_time": "2021-03-22T10:46:12.603960",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.582116",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "This list above corresponds to EpiGraphDB's meta nodes, whose documentation can be found [here](https://docs.epigraphdb.org/graph-database/meta-nodes/) along with their available properties.\n",
    "\n",
    "In the following, we will demonstrate how we can search by name for a node of interest, using the endpoint `/meta/nodes/{meta_node}/search`, where viable values for `{meta_node}` are those terms listed above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "frozen-tracy",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:12.650624Z",
     "iopub.status.busy": "2021-03-22T10:46:12.650170Z",
     "iopub.status.idle": "2021-03-22T10:46:12.837405Z",
     "shell.execute_reply": "2021-03-22T10:46:12.837936Z"
    },
    "id": "HkRDtvfuy2RL",
    "outputId": "e5f1e1bd-c7d1-4d71-8d0d-4c1a15b61a57",
    "papermill": {
     "duration": 0.212702,
     "end_time": "2021-03-22T10:46:12.838107",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.625405",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node.trait</th>\n",
       "      <th>node.id</th>\n",
       "      <th>node.sample_size</th>\n",
       "      <th>node.year</th>\n",
       "      <th>node.author</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Breast cancer</td>\n",
       "      <td>ebi-a-GCST007236</td>\n",
       "      <td>89677.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Breast cancer</td>\n",
       "      <td>ebi-a-GCST004988</td>\n",
       "      <td>139274.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Breast cancer (Combined Oncoarray; iCOGS; GWAS...</td>\n",
       "      <td>ieu-a-1126</td>\n",
       "      <td>228951.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Breast cancer (GWAS)</td>\n",
       "      <td>ieu-a-1168</td>\n",
       "      <td>33832.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Breast cancer (GWAS)</td>\n",
       "      <td>ieu-a-1131</td>\n",
       "      <td>32498.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Breast cancer (Oncoarray)</td>\n",
       "      <td>ieu-a-1129</td>\n",
       "      <td>106776.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Breast cancer (Survival)</td>\n",
       "      <td>ieu-a-1165</td>\n",
       "      <td>37954.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>Guo Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Breast cancer (iCOGS)</td>\n",
       "      <td>ieu-a-1162</td>\n",
       "      <td>89677.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Breast cancer (iCOGS)</td>\n",
       "      <td>ieu-a-1130</td>\n",
       "      <td>89677.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>Michailidou K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Breast cancer anti-estrogen resistance protein 3</td>\n",
       "      <td>prot-a-234</td>\n",
       "      <td>3301.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>Sun BB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          node.trait           node.id  \\\n",
       "0                                      Breast cancer  ebi-a-GCST007236   \n",
       "1                                      Breast cancer  ebi-a-GCST004988   \n",
       "2  Breast cancer (Combined Oncoarray; iCOGS; GWAS...        ieu-a-1126   \n",
       "3                               Breast cancer (GWAS)        ieu-a-1168   \n",
       "4                               Breast cancer (GWAS)        ieu-a-1131   \n",
       "5                          Breast cancer (Oncoarray)        ieu-a-1129   \n",
       "6                           Breast cancer (Survival)        ieu-a-1165   \n",
       "7                              Breast cancer (iCOGS)        ieu-a-1162   \n",
       "8                              Breast cancer (iCOGS)        ieu-a-1130   \n",
       "9   Breast cancer anti-estrogen resistance protein 3        prot-a-234   \n",
       "\n",
       "  node.sample_size node.year    node.author  \n",
       "0          89677.0    2015.0  Michailidou K  \n",
       "1         139274.0    2017.0  Michailidou K  \n",
       "2         228951.0    2017.0  Michailidou K  \n",
       "3          33832.0    2015.0  Michailidou K  \n",
       "4          32498.0    2017.0  Michailidou K  \n",
       "5         106776.0    2017.0  Michailidou K  \n",
       "6          37954.0    2015.0          Guo Q  \n",
       "7          89677.0    2015.0  Michailidou K  \n",
       "8          89677.0    2017.0  Michailidou K  \n",
       "9           3301.0    2018.0         Sun BB  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4.2 Searching for specific entities by name\n",
    "\n",
    "# Set params\n",
    "params = {\"name\": \"breast cancer\"}\n",
    "\n",
    "# Make request\n",
    "meta_node = \"Gwas\"\n",
    "endpoint = f\"/meta/nodes/{meta_node}/search\"\n",
    "response_object = requests.get(API_URL + endpoint, params=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Convert to pandas\n",
    "results = pd.json_normalize(response_object.json()[\"results\"])\n",
    "target_node_id = results[\"node.id\"][3]  # Store one ID for use in the next cell\n",
    "\n",
    "results[[\"node.trait\", \"node.id\", \"node.sample_size\", \"node.year\", \"node.author\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "experimental-convergence",
   "metadata": {
    "id": "Te4D_BWYy2RN",
    "papermill": {
     "duration": 0.032437,
     "end_time": "2021-03-22T10:46:12.902741",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.870304",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Above we used the `name` parameter of the endpoint to search for any GWAS nodes that fuzzily matched our specified string. Several did, and some of their basic node properties are displayed above. Fuzzy matching is useful because you don't need to know the exact name of the entity or its ID in order to look it up. \n",
    "\n",
    "On the other hand, once you have identified your entity of interest, it is often sensible to move forward using the node's ID for the sake of unambiguity. Fortunately we can also search for traits using their ID, as demonstrated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "suburban-extraction",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:12.949505Z",
     "iopub.status.busy": "2021-03-22T10:46:12.947155Z",
     "iopub.status.idle": "2021-03-22T10:46:13.094394Z",
     "shell.execute_reply": "2021-03-22T10:46:13.094923Z"
    },
    "id": "cN6ssg7Ay2RO",
    "outputId": "74cddd42-8a0f-488b-a580-0b229aaeec1b",
    "papermill": {
     "duration": 0.168871,
     "end_time": "2021-03-22T10:46:13.095092",
     "exception": false,
     "start_time": "2021-03-22T10:46:12.926221",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node.ncase</th>\n",
       "      <th>node._name</th>\n",
       "      <th>node.year</th>\n",
       "      <th>node.mr</th>\n",
       "      <th>node.author</th>\n",
       "      <th>node.sex</th>\n",
       "      <th>node.pmid</th>\n",
       "      <th>node.population</th>\n",
       "      <th>node.sample_size</th>\n",
       "      <th>node.nsnp</th>\n",
       "      <th>node.build</th>\n",
       "      <th>node.ncontrol</th>\n",
       "      <th>node.trait</th>\n",
       "      <th>node._source</th>\n",
       "      <th>node.id</th>\n",
       "      <th>node._id</th>\n",
       "      <th>node.subcategory</th>\n",
       "      <th>node.category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15748.0</td>\n",
       "      <td>Breast cancer (GWAS)</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Michailidou K</td>\n",
       "      <td>Females</td>\n",
       "      <td>25751625.0</td>\n",
       "      <td>European</td>\n",
       "      <td>33832.0</td>\n",
       "      <td>13011123</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>18084.0</td>\n",
       "      <td>Breast cancer (GWAS)</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ieu-a-1168</td>\n",
       "      <td>ieu-a-1168</td>\n",
       "      <td>Cancer</td>\n",
       "      <td>Disease</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  node.ncase            node._name node.year node.mr    node.author node.sex  \\\n",
       "0    15748.0  Breast cancer (GWAS)    2015.0       1  Michailidou K  Females   \n",
       "\n",
       "    node.pmid node.population node.sample_size node.nsnp   node.build  \\\n",
       "0  25751625.0        European          33832.0  13011123  HG19/GRCh37   \n",
       "\n",
       "  node.ncontrol            node.trait           node._source     node.id  \\\n",
       "0       18084.0  Breast cancer (GWAS)  [OpenGWAS-2020-10-13]  ieu-a-1168   \n",
       "\n",
       "     node._id node.subcategory node.category  \n",
       "0  ieu-a-1168           Cancer       Disease  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4.3 Searching for a node by ID\n",
    "\n",
    "# Set params\n",
    "params = {\"id\": target_node_id}  # From previous cell\n",
    "\n",
    "# Make request\n",
    "meta_node = \"Gwas\"\n",
    "endpoint = f\"/meta/nodes/{meta_node}/search\"\n",
    "response_object = requests.get(API_URL + endpoint, params=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Convert to pandas\n",
    "results = pd.json_normalize(response_object.json()[\"results\"])\n",
    "\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "textile-partnership",
   "metadata": {
    "id": "3o81lvE7y2RR",
    "papermill": {
     "duration": 0.034615,
     "end_time": "2021-03-22T10:46:13.162276",
     "exception": false,
     "start_time": "2021-03-22T10:46:13.127661",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "***\n",
    "## Advanced examples- Cypher\n",
    "\n",
    "Until now, to get information from the platform we have been simply creating a dictionary or JSON object containing our parameters and then sending it to the correct endpoint of the API using the `requests` library. This is a good way to use EpiGraphDB and the API has been designed specifically to allow this method of use, as we have (inexhaustively) demonstrated above. It works because the API automatically converts the HTTP requests that it receives into a Cypher query, which it then passes to the Neo4j database on which EpiGraphDB is built. The database passes back the result of the query, which is then returned to us in Python as a response object. Each response object contains metadata that includes the exact Cypher query that was called on the database, as shown in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "certified-nowhere",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:13.208633Z",
     "iopub.status.busy": "2021-03-22T10:46:13.208286Z",
     "iopub.status.idle": "2021-03-22T10:46:13.346474Z",
     "shell.execute_reply": "2021-03-22T10:46:13.346998Z"
    },
    "id": "JSavRQgAy2RR",
    "outputId": "bfb344cb-7737-4d0b-fb3d-0dadac874914",
    "papermill": {
     "duration": 0.16235,
     "end_time": "2021-03-22T10:46:13.347179",
     "exception": false,
     "start_time": "2021-03-22T10:46:13.184829",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MATCH (gene:Gene)-[gp:GENE_TO_PROTEIN]-(protein:Protein) WHERE gene.name IN ['TP53'] RETURN gene {.ensembl_id, .name}, protein {.uniprot_id}\n"
     ]
    }
   ],
   "source": [
    "# 4.1 Cypher\n",
    "\n",
    "params = {\"gene_name_list\": [\"TP53\"]}\n",
    "json_params = json.dumps(params)\n",
    "endpoint = \"/mappings/gene-to-protein\"\n",
    "response_object = requests.post(API_URL + endpoint, data=json_params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Extract and print the Cypher query\n",
    "cypher_query = response_object.json()[\"metadata\"][\"query\"]\n",
    "print(cypher_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "soviet-justice",
   "metadata": {
    "id": "Q9KkV4Hmy2RU",
    "papermill": {
     "duration": 0.032842,
     "end_time": "2021-03-22T10:46:13.413001",
     "exception": false,
     "start_time": "2021-03-22T10:46:13.380159",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The text printed above is the exact Cypher query that was run in section 1.1, behind the scenes. The basic structure of these queries is as follows:\n",
    "\n",
    "                                            MATCH subgraph\n",
    "\n",
    "                                            WHERE condition\n",
    "\n",
    "                                            RETURN data\n",
    "\n",
    "Note that the subgraph should take this general form: *(node)-[relationship]-(node)*, but for both nodes and relationships we write them as `my_variable_name:Meta_node` so that we can access their properties through the variable name we assigned them (my_variable_name), and use those properties to define our conditions and what data we want returned. Information on the available properties for each class of entity can be found in EpiGraphDB's documentation, specifically [here for nodes](https://docs.epigraphdb.org/graph-database/meta-nodes/) and [here for relationships](https://docs.epigraphdb.org/graph-database/meta-relationships/).\n",
    "\n",
    "Further documentation on Neo4j Cypher is available in the [Neo4j documentation](https://neo4j.com/docs/cypher-manual/current/).\n",
    "\n",
    "Now let's write and send our own basic query to get traits with high genetic correlation to body mass index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "northern-belief",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-22T10:46:13.463480Z",
     "iopub.status.busy": "2021-03-22T10:46:13.462863Z",
     "iopub.status.idle": "2021-03-22T10:46:13.945160Z",
     "shell.execute_reply": "2021-03-22T10:46:13.945689Z"
    },
    "id": "rWGXPd0wy2RU",
    "outputId": "610b7bda-e6ef-4e12-bf72-72c4d4e06341",
    "papermill": {
     "duration": 0.510954,
     "end_time": "2021-03-22T10:46:13.945858",
     "exception": false,
     "start_time": "2021-03-22T10:46:13.434904",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trait1.note</th>\n",
       "      <th>trait1._name</th>\n",
       "      <th>trait1.year</th>\n",
       "      <th>trait1.mr</th>\n",
       "      <th>trait1.author</th>\n",
       "      <th>trait1.consortium</th>\n",
       "      <th>trait1.sex</th>\n",
       "      <th>trait1.population</th>\n",
       "      <th>trait1.unit</th>\n",
       "      <th>trait1.sample_size</th>\n",
       "      <th>...</th>\n",
       "      <th>trait2.sample_size</th>\n",
       "      <th>trait2.nsnp</th>\n",
       "      <th>trait2.build</th>\n",
       "      <th>trait2.trait</th>\n",
       "      <th>trait2._source</th>\n",
       "      <th>trait2.id</th>\n",
       "      <th>trait2._id</th>\n",
       "      <th>trait2.category</th>\n",
       "      <th>corr.p</th>\n",
       "      <th>corr.rg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23104: Output from GWAS pipeline using Phesant...</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Ben Elsworth</td>\n",
       "      <td>MRC-IEU</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>European</td>\n",
       "      <td>SD</td>\n",
       "      <td>454884.0</td>\n",
       "      <td>...</td>\n",
       "      <td>462166.0</td>\n",
       "      <td>9851867</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Waist circumference</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ukb-b-9405</td>\n",
       "      <td>ukb-b-9405</td>\n",
       "      <td>Continuous</td>\n",
       "      <td>1.000000e-308</td>\n",
       "      <td>0.9027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23104: Output from GWAS pipeline using Phesant...</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Ben Elsworth</td>\n",
       "      <td>MRC-IEU</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>European</td>\n",
       "      <td>SD</td>\n",
       "      <td>454884.0</td>\n",
       "      <td>...</td>\n",
       "      <td>336639.0</td>\n",
       "      <td>10894596</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Waist circumference</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ukb-a-382</td>\n",
       "      <td>ukb-a-382</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000e-308</td>\n",
       "      <td>0.9027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23104: Output from GWAS pipeline using Phesant...</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Ben Elsworth</td>\n",
       "      <td>MRC-IEU</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>European</td>\n",
       "      <td>SD</td>\n",
       "      <td>454884.0</td>\n",
       "      <td>...</td>\n",
       "      <td>462166.0</td>\n",
       "      <td>9851867</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Waist circumference</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ukb-b-9405</td>\n",
       "      <td>ukb-b-9405</td>\n",
       "      <td>Continuous</td>\n",
       "      <td>1.000000e-308</td>\n",
       "      <td>0.9023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23104: Output from GWAS pipeline using Phesant...</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Ben Elsworth</td>\n",
       "      <td>MRC-IEU</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>European</td>\n",
       "      <td>SD</td>\n",
       "      <td>454884.0</td>\n",
       "      <td>...</td>\n",
       "      <td>336639.0</td>\n",
       "      <td>10894596</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Waist circumference</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ukb-a-382</td>\n",
       "      <td>ukb-a-382</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000e-308</td>\n",
       "      <td>0.9023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>23104: Output from GWAS pipeline using Phesant...</td>\n",
       "      <td>Body mass index (BMI)</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>1</td>\n",
       "      <td>Ben Elsworth</td>\n",
       "      <td>MRC-IEU</td>\n",
       "      <td>Males and Females</td>\n",
       "      <td>European</td>\n",
       "      <td>SD</td>\n",
       "      <td>454884.0</td>\n",
       "      <td>...</td>\n",
       "      <td>331164.0</td>\n",
       "      <td>10894596</td>\n",
       "      <td>HG19/GRCh37</td>\n",
       "      <td>Arm fat mass (left)</td>\n",
       "      <td>[OpenGWAS-2020-10-13]</td>\n",
       "      <td>ukb-a-287</td>\n",
       "      <td>ukb-a-287</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000e-308</td>\n",
       "      <td>0.9479</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         trait1.note           trait1._name  \\\n",
       "0  23104: Output from GWAS pipeline using Phesant...  Body mass index (BMI)   \n",
       "1  23104: Output from GWAS pipeline using Phesant...  Body mass index (BMI)   \n",
       "2  23104: Output from GWAS pipeline using Phesant...  Body mass index (BMI)   \n",
       "3  23104: Output from GWAS pipeline using Phesant...  Body mass index (BMI)   \n",
       "4  23104: Output from GWAS pipeline using Phesant...  Body mass index (BMI)   \n",
       "\n",
       "  trait1.year trait1.mr trait1.author trait1.consortium         trait1.sex  \\\n",
       "0      2018.0         1  Ben Elsworth           MRC-IEU  Males and Females   \n",
       "1      2018.0         1  Ben Elsworth           MRC-IEU  Males and Females   \n",
       "2      2018.0         1  Ben Elsworth           MRC-IEU  Males and Females   \n",
       "3      2018.0         1  Ben Elsworth           MRC-IEU  Males and Females   \n",
       "4      2018.0         1  Ben Elsworth           MRC-IEU  Males and Females   \n",
       "\n",
       "  trait1.population trait1.unit trait1.sample_size  ... trait2.sample_size  \\\n",
       "0          European          SD           454884.0  ...           462166.0   \n",
       "1          European          SD           454884.0  ...           336639.0   \n",
       "2          European          SD           454884.0  ...           462166.0   \n",
       "3          European          SD           454884.0  ...           336639.0   \n",
       "4          European          SD           454884.0  ...           331164.0   \n",
       "\n",
       "  trait2.nsnp trait2.build         trait2.trait         trait2._source  \\\n",
       "0     9851867  HG19/GRCh37  Waist circumference  [OpenGWAS-2020-10-13]   \n",
       "1    10894596  HG19/GRCh37  Waist circumference  [OpenGWAS-2020-10-13]   \n",
       "2     9851867  HG19/GRCh37  Waist circumference  [OpenGWAS-2020-10-13]   \n",
       "3    10894596  HG19/GRCh37  Waist circumference  [OpenGWAS-2020-10-13]   \n",
       "4    10894596  HG19/GRCh37  Arm fat mass (left)  [OpenGWAS-2020-10-13]   \n",
       "\n",
       "    trait2.id  trait2._id trait2.category         corr.p corr.rg  \n",
       "0  ukb-b-9405  ukb-b-9405      Continuous  1.000000e-308  0.9027  \n",
       "1   ukb-a-382   ukb-a-382             NaN  1.000000e-308  0.9027  \n",
       "2  ukb-b-9405  ukb-b-9405      Continuous  1.000000e-308  0.9023  \n",
       "3   ukb-a-382   ukb-a-382             NaN  1.000000e-308  0.9023  \n",
       "4   ukb-a-287   ukb-a-287             NaN  1.000000e-308  0.9479  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4.2 Writing custom Cypher queries\n",
    "\n",
    "# Define the target subgraph\n",
    "cypher_query = \"MATCH (trait1:Gwas)-[corr:GEN_COR]-(trait2:Gwas)\"\n",
    "\n",
    "# Add conditions to the query\n",
    "cypher_query += ' WHERE trait1.trait = \"Body mass index (BMI)\"'\n",
    "cypher_query += \" AND corr.rg > 0.9\"\n",
    "\n",
    "# Add which data we want returned\n",
    "cypher_query += \" RETURN trait1, trait2, corr {.rg, .p}\"\n",
    "\n",
    "# Put our query into the correct format for a POST request\n",
    "params = json.dumps({\"query\": cypher_query})\n",
    "\n",
    "# Define the target endpoint and send the request\n",
    "endpoint = \"/cypher\"\n",
    "response_object = requests.post(API_URL + endpoint, data=params)\n",
    "response_object.raise_for_status()\n",
    "\n",
    "# Display the returned data\n",
    "results = response_object.json()[\"results\"]\n",
    "results_df = pd.json_normalize(results)\n",
    "\n",
    "results_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brown-first",
   "metadata": {
    "id": "-IOFxKFHy2RX",
    "papermill": {
     "duration": 0.04074,
     "end_time": "2021-03-22T10:46:14.017605",
     "exception": false,
     "start_time": "2021-03-22T10:46:13.976865",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In our Cypher query, we extracted a subgraph from the database that comprised nodes representing biomedical traits, with edges between them representing their genetic correlation. The subgraph was then filtered to select any node-edge-node triples where the first node had the `.trait` property of \"Body mass index (BMI)\", and where the edge between the nodes had a `.rg` (genetic correlation score) value greater than 0.9. We then asked Neo4j to return us the names of the two traits, as well as the score and p-value of the correlation between the two, for all triples not filtered out by our conditions. Finally, we converted the returned dictionary to a dataframe for ease of viewing.\n",
    "\n",
    "For more detailed information on Cypher queries, please refer to the [official documentation](https://neo4j.com/developer/cypher/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "getting-started-with-epigraphdb.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 7.197531,
   "end_time": "2021-03-22T10:46:15.336987",
   "environment_variables": {},
   "exception": null,
   "input_path": "general-examples/getting-started-with-epigraphdb.ipynb",
   "output_path": "general-examples/getting-started-with-epigraphdb.ipynb",
   "parameters": {
    "API_URL": "https://api.epigraphdb.org"
   },
   "start_time": "2021-03-22T10:46:08.139456",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}