analysis/emergency_admission_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "historic-western",
   "metadata": {},
   "source": [
    "## Identification of patients admitted to hospital due to COVID-19 using emergence care attendance data\n",
    "\n",
    "### Background\n",
    "There is an urgent need to understand which patients are being admitted to hospital with COVID-19.  SUS-APCS data is the 'gold standard' for COVID-19 hospital admission but is only made available once the patient is discharged from hospital which can be many weeks or months after their admission.  Data for ongoing hospital spells at the time of the SUS-APCS extract is therefore not available, creating an ascertainment bias against longer spells and more recent spells. \n",
    "\n",
    "Hospital admission is a crucial outcome for vaccine evaluation.  The absence of rapidly available hospital admissions data means we cannot rapidly evaluate vaccine effectiveness with respect to reducing hospital admission, compare different vaccines' impact on hospital admission or identify changes in vaccine effectives on admission over time.\n",
    "\n",
    "A large proportion of hospital admission comes through A&E attendance. Emergency admission data through ECDS is much more rapidly available than SUS-APCS.  We therefore set out to validate if those patients being admitted to hospital due to COVID-19 could be identified earlier using data made available through SUS-APCS.\n",
    "\n",
    "### Methods\n",
    "\n",
    "Working on behalf of NHS England we used routine clinical data from 24 million patients to conduct a retrospective cohort study of comprehensive electronic health record data in NHS England, using the OpenSAFELY-TPP platform which covers approximately 40% of the general population in England.  Using data between 2020-09-01 and 2021-01-01, we selected a study population that included all those that were registered with a GP practice at the start of the study period, were aged between 18-110 at the start of the study period, had not died before the start of the study period and those who had either attended emergency care or had been admitted to hospital at any point during the study period.  \n",
    "\n",
    "Patients that were admitted to hospital with COVID-19 as their primary diagnosis were identified using a COVID-19 identification codelist made available at [OpenSAFELY Codelists here](https://codelists.opensafely.org/codelist/opensafely/covid-identification/2020-06-03/).  Patients that attended emergency care with COVID-19 were identified using the following SNOMED diagnosis code: 1240751000000100 and their discharge destination is extracted.  Whether patients attending emergency care had had a positive COVID-19 test in the period inclusive of the 2 weeks prior to attendance and 1 weeks after attendance was also extracted, as was whether patients attending emergency care had had COVID-19 confirmed in primary care in the period 2 weeks prior to admission using the following [codelists]().\n",
    "\n",
    "We first assessed how many patients hospitalised for any cause could be identified in those attending emergency care for any reason.  We then explore how many patients hospitalised for any cause could be identified in those attending emergency care who were discharged to either the ward or emergency short stay ward.  \n",
    "\n",
    "We then look at identifying those admitted to hospital for COVID-19 in the emergency care data.  In the emergency care data, patients are defined as being admitted to hospital for COVID-19 if they attend emergency care with subsequent discharge to the ward or emergency short stay ward and they have either had a recent positive COVID-19 test, have had recently confirmed COVID-19 in primary care, or are recorded as hhaving a COVID-19 diagnosis code.  Using this classification we provide a contingency matrix and measures of measures of the predictive ability of identifying COVID-19 hospitalisations using emergency care data.  Variable importance of this classification is then assessed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "mobile-corps",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "from matplotlib.colors import ListedColormap\n",
    "from IPython.display import Markdown as md\n",
    "from IPython.display import HTML, display\n",
    "%matplotlib inline\n",
    "\n",
    "pd.options.display.float_format = '{:.0f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "little-thought",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('../output/input.csv')\n",
    "num_patients = len(df['patient_id'].unique())\n",
    "num_patients_hosp = len(df[df['hospital_admission'].notna()]['patient_id'].unique())\n",
    "num_patients_hosp_prim_covid = len(df[df['primary_covid_hospital_admission'].notna()]['patient_id'].unique())\n",
    "num_patients_hosp_covid = len(df[df['covid_hospital_admission'].notna()]['patient_id'].unique())\n",
    "num_patients_hosp_positive_cov_test = len(df[(df['hospital_admission'].notna()) & (df['positive_covid_test_before_hospital_admission']==1)]['patient_id'].unique())\n",
    "num_patients_hosp_positive_cov_pc = len(df[(df['hospital_admission'].notna()) & (df['covid_primary_care_before_hospital_admission']==1)]['patient_id'].unique())\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "num_patients_ae = len(df[df['ae_attendance_any']==1]['patient_id'].unique())\n",
    "num_patients_ae_hosp_discharge = len(df[df['ae_attendance']==1]['patient_id'].unique())\n",
    "num_patients_ae_pos_covid = len(df[(df['ae_attendance']==1) & (df['ae_attendance_covid_status']==1)]['patient_id'].unique())\n",
    "num_patients_ae_pos_covid_test = len(df[(df['ae_attendance']==1) & (df['positive_covid_test_before_ae_attendance']==1)]['patient_id'].unique())\n",
    "num_patients_ae_pos_covid_pc = len(df[(df['ae_attendance']==1) & (df['covid_primary_care_before_ae_attendance']==1)]['patient_id'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "packed-camera",
   "metadata": {},
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "annual-notion",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Between 2020-09-01 and 2021-09-01 10000 people out of 24 million in our dataset either attended emergency care (4000) or were admitted to hospital (3000). In those attending emergency care, 4000 (100.0%) were discharged to the hospital ward or emergency short stay hospital ward."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md(f'Between 2020-09-01 and 2021-09-01 {num_patients} people out of 24 million in our dataset either attended emergency care ({num_patients_ae}) or were admitted to hospital ({num_patients_hosp}). In those attending emergency care, {num_patients_ae_hosp_discharge} ({(num_patients_ae_hosp_discharge/num_patients_ae)*100}%) were discharged to the hospital ward or emergency short stay hospital ward.')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "unnecessary-adapter",
   "metadata": {
    "code_folding": [],
    "variables": {
     "(num_patients_ae/num_patients_ae)100": "<p><strong>SyntaxError</strong>: invalid syntax (<ipython-input-47-c0558208f3d2>, line 1)</p>\n",
     "(num_patients_ae_hosp_discharge/num_patients_ae)<em>100": "<p><strong>NameError</strong>: name &#39;em&#39; is not defined</p>\n",
     "(num_patients_ae_pos_covid/num_patients_ae)100": "<p><strong>SyntaxError</strong>: invalid syntax (<ipython-input-47-81d1e1460ad3>, line 1)</p>\n",
     "(num_patients_ae_pos_covid_pc/num_patients_ae)100": "<p><strong>SyntaxError</strong>: invalid syntax (<ipython-input-47-eb6d5c87b8ed>, line 1)</p>\n",
     "(num_patients_ae_pos_covid_test/num_patients_ae)<em>100": "<p><strong>NameError</strong>: name &#39;em&#39; is not defined</p>\n",
     "(num_patients_hosp_covid/num_patients_hosp)<em>100": "<p><strong>NameError</strong>: name &#39;em&#39; is not defined</p>\n",
     "(num_patients_hosp_positive_cov_pc/num_patients_hosp)<em>100": "<p><strong>NameError</strong>: name &#39;em&#39; is not defined</p>\n",
     "(num_patients_hosp_positive_cov_test/num_patients_hosp)100": "<p><strong>SyntaxError</strong>: invalid syntax (<ipython-input-47-e6b11a714312>, line 1)</p>\n",
     "(num_patients_hosp_prim_covid/num_patients_hosp)100": "<p><strong>SyntaxError</strong>: invalid syntax (<ipython-input-47-b4660b553bcf>, line 1)</p>\n",
     "num_patients_ae": "4000",
     "num_patients_ae_hosp_discharge": "4000",
     "num_patients_ae_pos_covid": "3578",
     "num_patients_ae_pos_covid_pc": "2011",
     "num_patients_ae_pos_covid_test": "2050",
     "num_patients_hosp_covid": "3000",
     "num_patients_hosp_positive_cov_pc": "1473",
     "num_patients_hosp_positive_cov_test": "1505",
     "num_patients_hosp_prim_covid": "3000"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>Group</th><th>Total</th><th>%</th></tr><tr><th>Hospital Admission</th></tr><tr><td>Admitted</td><td>3000</td><td>100.0</td></tr><tr><td>Admitted with primary COVID-19</td><td>3000</td><td>100.0</td></tr><tr><td>Admitted with COVID-19</td><td>3000</td><td>100.0</td></tr><tr><td>Recent + COVID Test</td><td>1505</td><td>50.16666666666667</td></tr>  <tr><td>Recent + COVID in Primary Care</td><td>1473</td><td>49.1</td></tr><tr><th>Emergence Care Attendance</th></tr><tr><td>Attended</td><td>4000</td><td>100.0</td></tr><tr><td>Attended with hospital discharge</td><td>4000</td><td>100.0</td></tr><tr><td>Attended with COVID</td><td>3578</td><td>89.45</td></tr><tr><td>Recent + COVID Test</td><td>2050</td><td>51.24999999999999</td></tr><tr><td>Recent + COVID in Primary Care</td><td>2011</td><td>50.275000000000006</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "display(HTML(f'<table><tr><th>Group</th><th>Total</th><th>%</th></tr><tr><th>Hospital Admission</th></tr><tr><td>Admitted</td><td>{num_patients_hosp}</td><td>{(num_patients_hosp/num_patients_hosp)*100}</td></tr><tr><td>Admitted with primary COVID-19</td><td>{num_patients_hosp_prim_covid}</td><td>{(num_patients_hosp_prim_covid/num_patients_hosp)*100}</td></tr><tr><td>Admitted with COVID-19</td><td>{num_patients_hosp_covid}</td><td>{(num_patients_hosp_covid/num_patients_hosp)*100}</td></tr><tr><td>Recent + COVID Test</td><td>{num_patients_hosp_positive_cov_test}</td><td>{(num_patients_hosp_positive_cov_test/num_patients_hosp)*100}</td></tr>  <tr><td>Recent + COVID in Primary Care</td><td>{num_patients_hosp_positive_cov_pc}</td><td>{(num_patients_hosp_positive_cov_pc/num_patients_hosp)*100}</td></tr><tr><th>Emergence Care Attendance</th></tr><tr><td>Attended</td><td>{num_patients_ae}</td><td>{(num_patients_ae/num_patients_ae)*100}</td></tr><tr><td>Attended with hospital discharge</td><td>{num_patients_ae_hosp_discharge}</td><td>{(num_patients_ae_hosp_discharge/num_patients_ae)*100}</td></tr><tr><td>Attended with COVID</td><td>{num_patients_ae_pos_covid}</td><td>{(num_patients_ae_pos_covid/num_patients_ae)*100}</td></tr><tr><td>Recent + COVID Test</td><td>{num_patients_ae_pos_covid_test}</td><td>{(num_patients_ae_pos_covid_test/num_patients_ae)*100}</td></tr><tr><td>Recent + COVID in Primary Care</td><td>{num_patients_ae_pos_covid_pc}</td><td>{(num_patients_ae_pos_covid_pc/num_patients_ae)*100}</td></tr></table>'\n",
    "            ))\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unusual-malta",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "outstanding-police",
   "metadata": {},
   "source": [
    "In those admitted to hospital (any cause) how many could be identified looking at emergency care attendance (without appplying discharge filter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "young-perry",
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_patients_sus = df[df['hospital_admission'].notna()]\n",
    "negative_patients_sus = df[~df['hospital_admission'].notna()]\n",
    "\n",
    "positive_patients_ecds = df[(df['ae_attendance_any']==1)]\n",
    "negative_patients_ecds = df[(df['ae_attendance_any']==0)]\n",
    "\n",
    "total_sus_patients_positive = set(list(positive_patients_sus['patient_id']))\n",
    "total_sus_patients_negative = set(list(negative_patients_sus['patient_id']))\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_patients_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_patients_ecds['patient_id']))\n",
    "\n",
    "sus_total_pos_ecds_pos = len(list(set(total_sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_total_pos_ecds_neg = len(list(set(total_sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_total_neg_ecds_pos = len(list(set(total_sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_total_neg_ecds_neg = len(list(set(total_sus_patients_negative) & set(ecds_patients_negative)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "quantitative-clinton",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-total-positive</th>\n",
       "      <th>SUS-total-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ECDS-positive</th>\n",
       "      <td>1219</td>\n",
       "      <td>2781</td>\n",
       "      <td>4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ECDS-negative</th>\n",
       "      <td>1781</td>\n",
       "      <td>4219</td>\n",
       "      <td>6000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               SUS-total-positive  SUS-total-negative  Total\n",
       "ECDS-positive                1219                2781   4000\n",
       "ECDS-negative                1781                4219   6000\n",
       "Total                        3000                7000  10000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[sus_total_pos_ecds_pos, sus_total_neg_ecds_pos, (sus_total_pos_ecds_pos + sus_total_neg_ecds_pos)], [sus_total_pos_ecds_neg, sus_total_neg_ecds_neg, (sus_total_pos_ecds_neg + sus_total_neg_ecds_neg)], [(sus_total_pos_ecds_pos+sus_total_pos_ecds_neg), (sus_total_neg_ecds_pos+sus_total_neg_ecds_neg), (sus_total_pos_ecds_pos + sus_total_pos_ecds_neg + sus_total_neg_ecds_pos + sus_total_neg_ecds_neg)]], columns=[\"SUS-total-positive\", \"SUS-total-negative\", \"Total\"], index=[\"ECDS-positive\", \"ECDS-negative\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "lightweight-jenny",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 40.63%\n",
      "Specificity : 60.27%\n"
     ]
    }
   ],
   "source": [
    "#sensitivity - number of sus identified by ecds\n",
    "#specificity - number of those not in sus who are not in ecds\n",
    "\n",
    "sensitivity = (sus_total_pos_ecds_pos/(sus_total_pos_ecds_pos + sus_total_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_total_neg_ecds_neg/(sus_total_neg_ecds_pos + sus_total_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gross-semiconductor",
   "metadata": {},
   "source": [
    "In those admitted to hospital (any cause) how many could be identified looking at emergency care attendance that results in discharge to ward or emergency short stay ward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "otherwise-evidence",
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_patients_sus = df[df['hospital_admission'].notna()]\n",
    "negative_patients_sus = df[~df['hospital_admission'].notna()]\n",
    "\n",
    "positive_patients_ecds = df[(df['ae_attendance']==1)]\n",
    "negative_patients_ecds = df[(df['ae_attendance']==0)]\n",
    "\n",
    "total_sus_patients_positive = set(list(positive_patients_sus['patient_id']))\n",
    "total_sus_patients_negative = set(list(negative_patients_sus['patient_id']))\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_patients_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_patients_ecds['patient_id']))\n",
    "\n",
    "sus_total_pos_ecds_pos = len(list(set(total_sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_total_pos_ecds_neg = len(list(set(total_sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_total_neg_ecds_pos = len(list(set(total_sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_total_neg_ecds_neg = len(list(set(total_sus_patients_negative) & set(ecds_patients_negative)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dimensional-birthday",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-total-positive</th>\n",
       "      <th>SUS-total-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ECDS-positive</th>\n",
       "      <td>1180</td>\n",
       "      <td>2820</td>\n",
       "      <td>4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ECDS-negative</th>\n",
       "      <td>1820</td>\n",
       "      <td>4180</td>\n",
       "      <td>6000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               SUS-total-positive  SUS-total-negative  Total\n",
       "ECDS-positive                1180                2820   4000\n",
       "ECDS-negative                1820                4180   6000\n",
       "Total                        3000                7000  10000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[sus_total_pos_ecds_pos, sus_total_neg_ecds_pos, (sus_total_pos_ecds_pos + sus_total_neg_ecds_pos)], [sus_total_pos_ecds_neg, sus_total_neg_ecds_neg, (sus_total_pos_ecds_neg + sus_total_neg_ecds_neg)], [(sus_total_pos_ecds_pos+sus_total_pos_ecds_neg), (sus_total_neg_ecds_pos+sus_total_neg_ecds_neg), (sus_total_pos_ecds_pos + sus_total_pos_ecds_neg + sus_total_neg_ecds_pos + sus_total_neg_ecds_neg)]], columns=[\"SUS-total-positive\", \"SUS-total-negative\", \"Total\"], index=[\"ECDS-positive\", \"ECDS-negative\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "interracial-northern",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 39.33%\n",
      "Specificity : 59.71%\n"
     ]
    }
   ],
   "source": [
    "#sensitivity - number of sus identified by ecds\n",
    "#specificity - number of those not in sus who are not in ecds\n",
    "\n",
    "sensitivity = (sus_total_pos_ecds_pos/(sus_total_pos_ecds_pos + sus_total_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_total_neg_ecds_neg/(sus_total_neg_ecds_pos + sus_total_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "heard-chase",
   "metadata": {},
   "source": [
    "In those admitted to hospital as a result of COVID-19, how many could be identified by finding those that attended emergency care, were later discharged to hospital and had had either a previous positive covid test, had recently had covid confirmed in primary care or attended emergency care with a covid diagnosis code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "posted-treasury",
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_covid_patients_sus = df[df['primary_covid_hospital_admission'].notna()]\n",
    "negative_covid_patients_sus = df[~df['primary_covid_hospital_admission'].notna()]\n",
    "\n",
    "positive_covid_patients_ecds = df[(df['ae_attendance']==1) & ((df['ae_attendance_covid_status']==1) | (df['positive_covid_test_before_ae_attendance'] ==1) | (df['covid_primary_care_before_ae_attendance'] ==1))]\n",
    "negative_covid_patients_ecds = df[(df['ae_attendance']==0) | ((df['ae_attendance']==1) & ((df['ae_attendance_covid_status']==0) & (df['positive_covid_test_before_ae_attendance'] ==0) & (df['covid_primary_care_before_ae_attendance'] ==0)))]\n",
    "\n",
    "\n",
    "sus_patients_positive = set(list(positive_covid_patients_sus['patient_id']))\n",
    "ecds_patients_positive = set(list(positive_covid_patients_ecds['patient_id']))\n",
    "\n",
    "sus_patients_negative = set(list(negative_covid_patients_sus['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_covid_patients_ecds['patient_id']))\n",
    "\n",
    "\n",
    "sus_pos_ecds_pos = len(list(set(sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_pos_ecds_neg = len(list(set(sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_neg_ecds_pos = len(list(set(sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_neg_ecds_neg = len(list(set(sus_patients_negative) & set(ecds_patients_negative)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "wireless-benchmark",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-positive</th>\n",
       "      <th>SUS-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ECDS-positive</th>\n",
       "      <td>1193</td>\n",
       "      <td>2697</td>\n",
       "      <td>3890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ECDS-negative</th>\n",
       "      <td>1807</td>\n",
       "      <td>4303</td>\n",
       "      <td>6110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               SUS-positive  SUS-negative  Total\n",
       "ECDS-positive          1193          2697   3890\n",
       "ECDS-negative          1807          4303   6110\n",
       "Total                  3000          7000  10000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[sus_pos_ecds_pos, sus_neg_ecds_pos, (sus_pos_ecds_pos + sus_neg_ecds_pos)], [sus_pos_ecds_neg, sus_neg_ecds_neg, (sus_pos_ecds_neg + sus_neg_ecds_neg)], [(sus_pos_ecds_pos+sus_pos_ecds_neg), (sus_neg_ecds_pos+sus_neg_ecds_neg), (sus_pos_ecds_pos + sus_pos_ecds_neg + sus_neg_ecds_pos + sus_neg_ecds_neg)]], columns=[\"SUS-positive\", \"SUS-negative\", \"Total\"], index=[\"ECDS-positive\", \"ECDS-negative\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "arctic-energy",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 39.77%\n",
      "Specificity : 61.47%\n"
     ]
    }
   ],
   "source": [
    "#sensitivity - number of sus identified by ecds\n",
    "#specificity - number of those not in sus who are not in ecds\n",
    "\n",
    "sensitivity = (sus_pos_ecds_pos/(sus_pos_ecds_pos + sus_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_neg_ecds_neg/(sus_neg_ecds_pos + sus_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "historic-evening",
   "metadata": {},
   "source": [
    "## Variable Breakdown \n",
    "\n",
    "### AE Attendance (and subsequent discharge to ward or icu)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "extreme-magazine",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-positive</th>\n",
       "      <th>SUS-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AE attendance +</th>\n",
       "      <td>1218</td>\n",
       "      <td>2782</td>\n",
       "      <td>4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AE attendance -</th>\n",
       "      <td>1782</td>\n",
       "      <td>4218</td>\n",
       "      <td>6000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 SUS-positive  SUS-negative  Total\n",
       "AE attendance +          1218          2782   4000\n",
       "AE attendance -          1782          4218   6000\n",
       "Total                    3000          7000  10000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "positive_ae_covid_ecds = df[((df['ae_attendance']==1))]\n",
    "negative_ae_covid_ecds = df[((df['ae_attendance']==0))]\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_ae_covid_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_ae_covid_ecds['patient_id']))\n",
    "\n",
    "sus_pos_ecds_pos = len(list(set(sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_pos_ecds_neg = len(list(set(sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_neg_ecds_pos = len(list(set(sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_neg_ecds_neg = len(list(set(sus_patients_negative) & set(ecds_patients_negative)))\n",
    "\n",
    "pd.DataFrame([[sus_pos_ecds_pos, sus_neg_ecds_pos, (sus_pos_ecds_pos + sus_neg_ecds_pos)], [sus_pos_ecds_neg, sus_neg_ecds_neg, (sus_pos_ecds_neg + sus_neg_ecds_neg)], [(sus_pos_ecds_pos+sus_pos_ecds_neg), (sus_neg_ecds_pos+sus_neg_ecds_neg), (sus_pos_ecds_pos + sus_pos_ecds_neg + sus_neg_ecds_pos + sus_neg_ecds_neg)]], columns=[\"SUS-positive\", \"SUS-negative\", \"Total\"], index=[\"AE attendance +\", \"AE attendance -\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "concerned-rapid",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 40.60%\n",
      "Specificity : 60.26%\n"
     ]
    }
   ],
   "source": [
    "sensitivity = (sus_pos_ecds_pos/(sus_pos_ecds_pos + sus_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_neg_ecds_neg/(sus_neg_ecds_pos + sus_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hollywood-keyboard",
   "metadata": {},
   "source": [
    "### AE Attendance + AE Covid Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "otherwise-penguin",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-positive</th>\n",
       "      <th>SUS-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AE Covid +</th>\n",
       "      <td>1095</td>\n",
       "      <td>2483</td>\n",
       "      <td>3578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AE Covid -</th>\n",
       "      <td>1905</td>\n",
       "      <td>4517</td>\n",
       "      <td>6422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            SUS-positive  SUS-negative  Total\n",
       "AE Covid +          1095          2483   3578\n",
       "AE Covid -          1905          4517   6422\n",
       "Total               3000          7000  10000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "positive_ae_covid_ecds = df[(df['ae_attendance']==1) & ((df['ae_attendance_covid_status']==1))]\n",
    "negative_ae_covid_ecds = df[((df['ae_attendance']==0) | ((df['ae_attendance']==1) & (df['ae_attendance_covid_status']==0)))]\n",
    "\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_ae_covid_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_ae_covid_ecds['patient_id']))\n",
    "\n",
    "sus_pos_ecds_pos = len(list(set(sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_pos_ecds_neg = len(list(set(sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_neg_ecds_pos = len(list(set(sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_neg_ecds_neg = len(list(set(sus_patients_negative) & set(ecds_patients_negative)))\n",
    "\n",
    "pd.DataFrame([[sus_pos_ecds_pos, sus_neg_ecds_pos, (sus_pos_ecds_pos + sus_neg_ecds_pos)], [sus_pos_ecds_neg, sus_neg_ecds_neg, (sus_pos_ecds_neg + sus_neg_ecds_neg)], [(sus_pos_ecds_pos+sus_pos_ecds_neg), (sus_neg_ecds_pos+sus_neg_ecds_neg), (sus_pos_ecds_pos + sus_pos_ecds_neg + sus_neg_ecds_pos + sus_neg_ecds_neg)]], columns=[\"SUS-positive\", \"SUS-negative\", \"Total\"], index=[\"AE Covid +\", \"AE Covid -\", \"Total\"])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "settled-insured",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 36.50%\n",
      "Specificity : 64.53%\n"
     ]
    }
   ],
   "source": [
    "sensitivity = (sus_pos_ecds_pos/(sus_pos_ecds_pos + sus_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_neg_ecds_neg/(sus_neg_ecds_pos + sus_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "synthetic-afternoon",
   "metadata": {},
   "source": [
    "###  AE Attendance + Recent Positive Covid Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "increased-screw",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-positive</th>\n",
       "      <th>SUS-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Covid Test +</th>\n",
       "      <td>633</td>\n",
       "      <td>1417</td>\n",
       "      <td>2050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Covid Test -</th>\n",
       "      <td>2367</td>\n",
       "      <td>5583</td>\n",
       "      <td>7950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              SUS-positive  SUS-negative  Total\n",
       "Covid Test +           633          1417   2050\n",
       "Covid Test -          2367          5583   7950\n",
       "Total                 3000          7000  10000"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "positive_cov_test_ecds = df[(df['ae_attendance']==1) & ((df['positive_covid_test_before_ae_attendance']==1))]\n",
    "negative_cov_test_ecds = df[((df['ae_attendance']==0) | ((df['ae_attendance']==1) & (df['positive_covid_test_before_ae_attendance']==0)))]\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_cov_test_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_cov_test_ecds['patient_id']))\n",
    "\n",
    "sus_pos_ecds_pos = len(list(set(sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_pos_ecds_neg = len(list(set(sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_neg_ecds_pos = len(list(set(sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_neg_ecds_neg = len(list(set(sus_patients_negative) & set(ecds_patients_negative)))\n",
    "\n",
    "pd.DataFrame([[sus_pos_ecds_pos, sus_neg_ecds_pos, (sus_pos_ecds_pos + sus_neg_ecds_pos)], [sus_pos_ecds_neg, sus_neg_ecds_neg, (sus_pos_ecds_neg + sus_neg_ecds_neg)], [(sus_pos_ecds_pos+sus_pos_ecds_neg), (sus_neg_ecds_pos+sus_neg_ecds_neg), (sus_pos_ecds_pos + sus_pos_ecds_neg + sus_neg_ecds_pos + sus_neg_ecds_neg)]], columns=[\"SUS-positive\", \"SUS-negative\", \"Total\"], index=[\"Covid Test +\", \"Covid Test -\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "complimentary-opening",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 21.10%\n",
      "Specificity : 79.76%\n"
     ]
    }
   ],
   "source": [
    "sensitivity = (sus_pos_ecds_pos/(sus_pos_ecds_pos + sus_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_neg_ecds_neg/(sus_neg_ecds_pos + sus_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "foreign-inflation",
   "metadata": {},
   "source": [
    "### AE Attendance + Covid Positive Primary Care"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "settled-beast",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SUS-positive</th>\n",
       "      <th>SUS-negative</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Primary Care Covid +</th>\n",
       "      <td>612</td>\n",
       "      <td>1399</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Primary Care Covid -</th>\n",
       "      <td>2388</td>\n",
       "      <td>5601</td>\n",
       "      <td>7989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>3000</td>\n",
       "      <td>7000</td>\n",
       "      <td>10000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      SUS-positive  SUS-negative  Total\n",
       "Primary Care Covid +           612          1399   2011\n",
       "Primary Care Covid -          2388          5601   7989\n",
       "Total                         3000          7000  10000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "positive_cov_pc_ecds = df[(df['ae_attendance']==1) & ((df['covid_primary_care_before_ae_attendance']==1))]\n",
    "negative_cov_pc_ecds = df[((df['ae_attendance']==0) | ((df['ae_attendance']==1) & (df['covid_primary_care_before_ae_attendance']==0)))]\n",
    "\n",
    "\n",
    "ecds_patients_positive = set(list(positive_cov_pc_ecds['patient_id']))\n",
    "ecds_patients_negative = set(list(negative_cov_pc_ecds['patient_id']))\n",
    "\n",
    "sus_pos_ecds_pos = len(list(set(sus_patients_positive) & set(ecds_patients_positive)))\n",
    "sus_pos_ecds_neg = len(list(set(sus_patients_positive) & set(ecds_patients_negative)))\n",
    "sus_neg_ecds_pos = len(list(set(sus_patients_negative) & set(ecds_patients_positive)))\n",
    "sus_neg_ecds_neg = len(list(set(sus_patients_negative) & set(ecds_patients_negative)))\n",
    "\n",
    "pd.DataFrame([[sus_pos_ecds_pos, sus_neg_ecds_pos, (sus_pos_ecds_pos + sus_neg_ecds_pos)], [sus_pos_ecds_neg, sus_neg_ecds_neg, (sus_pos_ecds_neg + sus_neg_ecds_neg)], [(sus_pos_ecds_pos+sus_pos_ecds_neg), (sus_neg_ecds_pos+sus_neg_ecds_neg), (sus_pos_ecds_pos + sus_pos_ecds_neg + sus_neg_ecds_pos + sus_neg_ecds_neg)]], columns=[\"SUS-positive\", \"SUS-negative\", \"Total\"], index=[\"Primary Care Covid +\", \"Primary Care Covid -\", \"Total\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "muslim-debate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 20.40%\n",
      "Specificity : 80.01%\n"
     ]
    }
   ],
   "source": [
    "sensitivity = (sus_pos_ecds_pos/(sus_pos_ecds_pos + sus_pos_ecds_neg))*100\n",
    "print(f\"Sensitivity: {sensitivity:.2f}%\")\n",
    "\n",
    "specificity = (sus_neg_ecds_neg/(sus_neg_ecds_pos + sus_neg_ecds_neg))*100\n",
    "print(f\"Specificity : {specificity:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unsigned-flood",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hazardous-headset",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}