examples/SIR_Posterior_Estimation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bdff5b1d08cd2bca",
   "metadata": {},
   "source": [
    "# Posterior Estimation for SIR-like Models\n",
    "\n",
    "_Author: Stefan T. Radev_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "684f2d7e19d40e09",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# ensure the backend is set\n",
    "import os\n",
    "if \"KERAS_BACKEND\" not in os.environ:\n",
    "    # set this to \"torch\", \"tensorflow\", or \"jax\"\n",
    "    os.environ[\"KERAS_BACKEND\"] = \"tensorflow\"\n",
    "\n",
    "import keras\n",
    "\n",
    "import bayesflow as bf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "576d14f2d1f377b9",
   "metadata": {},
   "source": [
    "## Introduction <a class=\"anchor\" id=\"introduction\"></a>\n",
    "\n",
    "In this tutorial, we will illustrate how to perform posterior inference on simple, stationary SIR-like models (complex models will be tackled in a further notebook). SIR-like models comprise suitable illustrative examples, since they generate time-series and their outputs represent the results of solving a system of ordinary differential equations (ODEs).\n",
    "\n",
    "The details for tackling stochastic epidemiological models with neural networks are described in our corresponding paper, which you can consult for a more formal exposition and a more comprehensive treatment of neural architectures:\n",
    "\n",
    "<em>OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural networks and its application to the COVID-19 pandemics in Germany</em> https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009472"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c25067d290a9a3e4",
   "metadata": {},
   "source": [
    "## Defining the Simulator <a class=\"anchor\" id=\"defining_the_generative\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c104ae86c4f59a",
   "metadata": {},
   "outputs": [],
   "source": [
    "RNG = np.random.default_rng(2025)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39846c15b88eaf8e",
   "metadata": {},
   "source": [
    "As described in our [very first notebook](Quickstart_Amortized_Posterior_Estimation.ipynb), a generative model consists of a prior (encoding suitable parameter ranges) and a simulator (generating data given simulations). Our underlying model distinguishes between susceptible, $S$, infected, $I$, and recovered, $R$, individuals with infection and recovery occurring at a constant transmission rate $\\lambda$ and constant recovery rate $\\mu$, respectively. The model dynamics are governed by the following system of ODEs:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "    \\frac{dS}{dt} &= -\\lambda\\,\\left(\\frac{S\\,I}{N}\\right) \\\\\n",
    "    \\frac{dI}{dt} &= \\lambda\\,\\left(\\frac{S\\,I}{N}\\right) - \\mu\\,I \\\\\n",
    "    \\frac{dR}{dt} &= \\mu\\,I,\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "with $N = S + I + R$ denoting the total population size. For the purpose of forward inference (simulation), we will use a time step of $dt = 1$, corresponding to daily case reports. In addition to the ODE parameters $\\lambda$ and $\\mu$, we consider a reporting delay parameter $L$ and a dispersion parameter $\\psi$, which affect the number of reported infected individuals via a negative binomial disttribution (https://en.wikipedia.org/wiki/Negative_binomial_distribution):\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "    I_t^{(obs)} \\sim \\textrm{NegBinomial}(I^{(new)}_{t-L}, \\psi),\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "In this way, we connect the latent disease model to an observation model, which renders the relationship between parameters and data a stochastic one. Note, that the observation model induces a further parameter $\\psi$, responsible for the dispersion of the noise.\n",
    "Finally, we will also treat the number of initially infected individuals, $I_0$ as an unknown parameter (having its own prior distribution)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36a6e034f7329fdb",
   "metadata": {},
   "source": [
    "### Prior <a class=\"anchor\" id=\"prior\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3654ae4f3f243509",
   "metadata": {},
   "source": [
    "We will place the following prior distributions over the five model parameters, summarized in the table below:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "& \\text {Table 1. Description of model parameters and corresponding prior distributions}\\\\\n",
    "&\\begin{array}{lcl}\n",
    "\\hline \\hline \\text { Description} & \\text { Symbol } & \\text { Prior Distribution } \\\\\n",
    "\\hline \\hline \\text{Initial transmission rate} & \\text{$\\lambda$} & \\text{$\\textrm{LogNormal}(\\log(0.4), 0.5)$} \\\\\n",
    "\\text{Recovery rate of infected individuals} & \\text{$\\mu$} & \\text{$\\textrm{LogNormal}(\\log(1/8), 0.2)$} \\\\\n",
    "\\text{Reporting delay (lag)} & \\text{$L$} & \\text{$\\textrm{LogNormal}(\\log(8), 0.2)$} \\\\\n",
    "\\text{Number of initially infected individuals} & \\text{$I_0$} & \\text{$\\textrm{Gamma}(2, 20)$} \\\\\n",
    "\\text{Dispersion of the negative binomial distribution} & \\text{$\\psi$} & \\text{$\\textrm{Exponential}(5)$} \\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "How did we come up with these priors? In this case, we rely on the domain expertise and previous research  (https://www.science.org/doi/10.1126/science.abb9789). In addition, the new parameter $\\psi$ follows an exponential distribution, which restricts it to positive numbers. Below is the implementation of these priors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be2c630b4527ae40",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prior():\n",
    "    \"\"\"Generates a random draw from the joint prior.\"\"\"\n",
    "\n",
    "    lambd = RNG.lognormal(mean=np.log(0.4), sigma=0.5)\n",
    "    mu = RNG.lognormal(mean=np.log(1 / 8), sigma=0.2)\n",
    "    D = RNG.lognormal(mean=np.log(8), sigma=0.2)\n",
    "    I0 = RNG.gamma(shape=2, scale=20)\n",
    "    psi = RNG.exponential(5)\n",
    "    return {\"lambd\": lambd, \"mu\": mu, \"D\": D, \"I0\": I0, \"psi\": psi}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de749b48ab53d91",
   "metadata": {},
   "source": [
    "### Observation Model (Implicit Likelihood Function) <a class=\"anchor\" id=\"simulator__implicit_likelihood\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e599e5cc2e0ff37",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import nbinom\n",
    "\n",
    "\n",
    "def convert_params(mu, phi):\n",
    "    \"\"\"Helper function to convert mean/dispersion parameterization of a negative binomial to N and p,\n",
    "    as expected by numpy's negative_binomial.\n",
    "\n",
    "    See https://en.wikipedia.org/wiki/Negative_binomial_distribution#Alternative_formulations\n",
    "    \"\"\"\n",
    "\n",
    "    r = phi\n",
    "    var = mu + 1 / r * mu**2\n",
    "    p = (var - mu) / var\n",
    "    return r, 1 - p\n",
    "\n",
    "\n",
    "def stationary_SIR(lambd, mu, D, I0, psi, N=83e6, T=14, eps=1e-5):\n",
    "    \"\"\"Performs a forward simulation from the stationary SIR model given a random draw from the prior.\"\"\"\n",
    "\n",
    "    # Extract parameters and round I0 and D\n",
    "    I0 = np.ceil(I0)\n",
    "    D = int(round(D))\n",
    "\n",
    "    # Initial conditions\n",
    "    S, I, R = [N - I0], [I0], [0]\n",
    "\n",
    "    # Reported new cases\n",
    "    C = [I0]\n",
    "\n",
    "    # Simulate T-1 timesteps\n",
    "    for t in range(1, T + D):\n",
    "        # Calculate new cases\n",
    "        I_new = lambd * (I[-1] * S[-1] / N)\n",
    "\n",
    "        # SIR equations\n",
    "        S_t = S[-1] - I_new\n",
    "        I_t = np.clip(I[-1] + I_new - mu * I[-1], 0.0, N)\n",
    "        R_t = np.clip(R[-1] + mu * I[-1], 0.0, N)\n",
    "\n",
    "        # Track\n",
    "        S.append(S_t)\n",
    "        I.append(I_t)\n",
    "        R.append(R_t)\n",
    "        C.append(I_new)\n",
    "\n",
    "    reparam = convert_params(np.clip(np.array(C[D:]), 0, N) + eps, psi)\n",
    "    C_obs = RNG.negative_binomial(reparam[0], reparam[1])\n",
    "    return dict(cases=C_obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7e1dc9545f948a",
   "metadata": {},
   "source": [
    "As you can see, in addition to the parameters, our simulator requires two further arguments: the total population size $N$ and the time horizon $T$. These are quantities over which we can amortize (i.e., context variables), but for this example, we will just use the population of Germany and the first two weeks of the pandemics (i.e., $T=14$), in the same vein as https://www.science.org/doi/10.1126/science.abb9789."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d3204086674397",
   "metadata": {},
   "source": [
    "### Loading Real Data <a class=\"anchor\" id=\"loading_real_data\"></a>\n",
    "\n",
    "We will define a simple helper function to load the actually reported cases in 2020 for the first two weeks of the Covid-19 pandemic in Germany."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f5305859288ca0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    \"\"\"Helper function to load cumulative cases and transform them to new cases.\"\"\"\n",
    "\n",
    "    confirmed_cases_url = \"https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv\"\n",
    "    confirmed_cases = pd.read_csv(confirmed_cases_url, sep=\",\")\n",
    "\n",
    "    date_data_begin = datetime.date(2020, 3, 1)\n",
    "    date_data_end = datetime.date(2020, 3, 15)\n",
    "    format_date = lambda date_py: f\"{date_py.month}/{date_py.day}/{str(date_py.year)[2:4]}\"\n",
    "    date_formatted_begin = format_date(date_data_begin)\n",
    "    date_formatted_end = format_date(date_data_end)\n",
    "\n",
    "    cases_obs = np.array(\n",
    "        confirmed_cases.loc[confirmed_cases[\"Country/Region\"] == \"Germany\", date_formatted_begin:date_formatted_end]\n",
    "    )[0]\n",
    "    new_cases_obs = np.diff(cases_obs)\n",
    "    return new_cases_obs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260da93b2f20a502",
   "metadata": {},
   "source": [
    "### Stitiching Things Together <a class=\"anchor\" id=\"generative_model\"></a>\n",
    "We can combine the prior $p(\\theta)$ and the observation model $p(x_{1:T}\\mid\\theta)$ into a joint model $p(\\theta, x_{1:T}) = p(\\theta) \\; p(x_{1:T}\\mid\\theta)$ using the `make_simulator` builder.\n",
    "The resulting object can now generate *batches* of simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a4e209c4ac07598",
   "metadata": {},
   "outputs": [],
   "source": [
    "simulator = bf.make_simulator([prior, stationary_SIR])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb375d6690435d31",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "test_sims = simulator.sample(batch_size=2)\n",
    "print(test_sims[\"lambd\"].shape)\n",
    "print(test_sims[\"D\"].shape)\n",
    "print(test_sims[\"cases\"].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "572000649eee6d15",
   "metadata": {},
   "source": [
    "## Prior Checking <a class=\"anchor\" id=\"prior_checking\"></a>\n",
    "\n",
    "Any principled Bayesian workflow requires some prior predictive or prior pushforward checks to ensure that the prior specification is consistent with domain expertise (see https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html). The BayesFlow library provides some rudimentary visual tools for performing prior checking. For instance, we can visually inspect the joint prior in the form of bivariate plots. We can focus on particular parameter combinations, such as $\\lambda$, $\\mu$, and $D$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fbb5ca04e75e47b",
   "metadata": {},
   "outputs": [],
   "source": [
    "prior_samples = simulator.simulators[0].sample(1000)\n",
    "\n",
    "grid = bf.diagnostics.plots.pairs_samples(\n",
    "    prior_samples, variable_keys=[\"lambd\", \"mu\", \"D\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52520d4c6c04fe4c",
   "metadata": {},
   "source": [
    "## Defining the Adapter\n",
    "\n",
    "We need to ensure that the outputs of the forward model are suitable for processing with neural networks. Currently, they are not, since our data $x_{1:T}$ consists of large integer (count) values. However, neural networks like scaled data. Furthermore, our parameters $\\theta$ exhibit widely different scales due to their prior specification and role in the simulator. Finally, BayesFlow needs to know which variables are to be inferred and which ones are to be processed by the summary network before being passed to the inference network. We handle all of these steps using an `Adapter`.\n",
    "\n",
    "Since all of our parameters and observables can only take on positive values, we will apply a log plus one transform to all quantities. Note, that `BayesFlow` expects the following keys to be present in the final outputs of your configured simulations:\n",
    "- `inference_variables`: These are the variables we are inferring.\n",
    "- `summary_variables`: These are the variables that are compressed throgh a summary network and used for inferring the inference variables.\n",
    "\n",
    "Thus, what our approximators are learning is $p(\\text{inference variables} \\mid t(\\text{summary variables}))$, where $t$ is the summary network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50d34e6dc1267048",
   "metadata": {},
   "outputs": [],
   "source": [
    "adapter = (\n",
    "    bf.adapters.Adapter()\n",
    "    .convert_dtype(\"float64\", \"float32\")\n",
    "    .as_time_series(\"cases\")\n",
    "    .concatenate([\"lambd\", \"mu\", \"D\", \"I0\", \"psi\"], into=\"inference_variables\")\n",
    "    .rename(\"cases\", \"summary_variables\")\n",
    "    # since all our variables are non-negative (zero or larger)\n",
    "    # this .apply call ensures that the variables are transformed\n",
    "    # to the unconstrained real space and can be back-transformed under the hood\n",
    "    .apply(forward=np.log1p)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c72020475ed04816",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's check out the new shapes\n",
    "adapted_sims = adapter(simulator.sample(2))\n",
    "print(adapted_sims[\"summary_variables\"].shape)\n",
    "print(adapted_sims[\"inference_variables\"].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33850f0bff73f125",
   "metadata": {},
   "source": [
    "## Defining the Neural Approximator <a class=\"anchor\" id=\"defining_the_neural\"></a>\n",
    "\n",
    "We can now proceed to define our `BayesFlow` neural architecture, that is, combine a summary network with an inference network."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee78608e39b07c5",
   "metadata": {},
   "source": [
    "### Summary Network <a class=\"anchor\" id=\"summary_network\"></a>\n",
    "\n",
    "Since our simulator outputs 3D tensors of shape ``(batch_size, T = 14, 1)``, we need to reduce this three-dimensional tensor into a two-dimensional tensor of shape ``(batch_size, summary_dim)``. Our model outputs are actually so simple that we could have just removed the trailing dimension of the raw outputs and simply fed the data directly to the inference network.\n",
    "\n",
    "However, we demonstrate the use of a simple Gated Recurrent Unit (GRU) summary network. Any `keras` model can interact with `BayesFlow` by inherting from `SummaryNetwork` which accepts an addition `stage` argument indicating the mode the network is currently operating in (i.e., `training` vs. `inference`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e34389285522c53f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GRU(bf.networks.SummaryNetwork):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "\n",
    "        self.gru = keras.layers.GRU(64, dropout=0.1)\n",
    "        self.summary_stats = keras.layers.Dense(8)\n",
    "        \n",
    "    def call(self, time_series, **kwargs):\n",
    "        \"\"\"Compresses time_series of shape (batch_size, T, 1) into summaries of shape (batch_size, 8).\"\"\"\n",
    "\n",
    "        summary = self.gru(time_series, training=kwargs.get(\"stage\") == \"training\")\n",
    "        summary = self.summary_stats(summary)\n",
    "        return summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9169f1e899bf874f",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_net = GRU()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fddf7a519ef6ccfa",
   "metadata": {},
   "source": [
    "### Inference Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43da1858d6d994b9",
   "metadata": {},
   "source": [
    "As inference network we choose a flow matching architecture with some dropout to robustify the inference. Dropout is primarily important when learning from a (small) offline dataset. See below for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcbc90940a36139f",
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_net = bf.networks.CouplingFlow(\n",
    "    subnet_kwargs={\"residual\": True, \"dropout\": 0.1, \"widths\": (128, 128, 128)}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ab78d30046fd2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow = bf.BasicWorkflow(\n",
    "    simulator=simulator,\n",
    "    adapter=adapter,\n",
    "    inference_network=inference_net,\n",
    "    summary_network=summary_net,\n",
    "    inference_variables=[\"lambd\", \"mu\", \"D\", \"I0\", \"psi\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c3456ee103b9c62",
   "metadata": {},
   "source": [
    "## Training <a class=\"anchor\" id=\"training\"></a>\n",
    "\n",
    "Ready to train! Since our simulator is pretty fast, we can safely go with online training. Let's glean the time taken for a batch of $32$ simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c550c08b9efcb0ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "_ = workflow.simulate(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33aca301a75c32a2",
   "metadata": {},
   "source": [
    "Not too bad! However, for the purpose of illustration, we will go with offline training using a fixed data set of simulations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4fae304214cc745",
   "metadata": {},
   "source": [
    "### Generating Offline Data <a class=\"anchor\" id=\"generating_offline_data\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7731d6e78304a722",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data = workflow.simulate(5000)\n",
    "validation_data = workflow.simulate(300)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21880ce8a1643c2e",
   "metadata": {},
   "source": [
    "We are now ready to train. If not provided, the default settings use $100$ epochs with a batch size of $32$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edaf144e3e5a8c81",
   "metadata": {},
   "outputs": [],
   "source": [
    "history = workflow.fit_offline(training_data, epochs=300, batch_size=64, validation_data=validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a4859372b0df84",
   "metadata": {},
   "source": [
    "### Inspecting the Loss <a class=\"anchor\" id=\"inspecting_the_loss\"></a>\n",
    "\n",
    "Following our online simulation-based training, we can quickly visualize the loss trajectory using the `plots.loss` function from the `diagnostics` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b4d998d830cd24",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = bf.diagnostics.plots.loss(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66715891c31e66f9",
   "metadata": {},
   "source": [
    "Great, it seems that our approximator has converged! Before we get too excited and throw our networks at real data, we need to make sure that they meet our expectations <em>in silico</em>, that is, given the small world of simulations the networks have seen during training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f750fb25272c4f17",
   "metadata": {},
   "source": [
    "## Validation Phase\n",
    "\n",
    "When it comes to validating posterior inference, we can either deploy manual diagnostics from the `diagnostics` module, or use the automated functions from the `BasicWorkflow` object. First, we demonstrate manual validation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8eccd7f41072001",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the number of posterior draws you want to get\n",
    "num_datasets = 300\n",
    "num_samples = 1000\n",
    "\n",
    "# Simulate 300 scenarios and extract time series from dict\n",
    "test_sims = workflow.simulate(num_datasets)\n",
    "time_series = test_sims.pop(\"cases\")\n",
    "\n",
    "# Obtain num_samples posterior samples per scenario\n",
    "samples = workflow.sample(conditions={\"cases\": time_series}, num_datasets=num_datasets, num_samples=num_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e696cb368f2072ae",
   "metadata": {},
   "source": [
    "### Simulation-Based Calibration - Rank Histograms\n",
    "\n",
    "As a further <strong>small world</strong> (i.e., before real data) sanity check, we can also test the calibration of the amortizer through simulation-based calibration (SBC). See the corresponding paper for more details (https://arxiv.org/pdf/1804.06788.pdf). Accordingly, we expect to observe approximately uniform rank statistic histograms. In the present case, this is indeed what we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3895cc36a2f3e1bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = bf.diagnostics.plots.calibration_histogram(samples, test_sims)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b73fd0c6f8487943",
   "metadata": {},
   "source": [
    "### Simulation-Based Calibration - Rank ECDF\n",
    "\n",
    "For models with many parameters, inspecting many histograms can become unwieldly. Moreover, the `num_bins` hyperparameter for the construction of SBC rank histograms can be hard to choose. An alternative diagnostic approach for calibration is through empirical cumulative distribution functions (ECDF) of rank statistics. You can read more about this approach in the corresponding paper (https://arxiv.org/abs/2103.10522).\n",
    "\n",
    "In order to inspect the ECDFs of marginal distributions, we will simulate $300$ new pairs of simulated data and generating parameters $(\\boldsymbol{x}, \\boldsymbol{\\theta})$ and use the function `plots.calibration_ecdf` from the `diagnostics` module: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9c77ead431320cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = bf.diagnostics.plots.calibration_ecdf(samples, test_sims, difference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9da20e3a66fd589",
   "metadata": {},
   "source": [
    "### Inferential Adequacy (Global)\n",
    "\n",
    "Depending on the application, it might be interesting to see how well summaries of the full posterior (e.g., means, medians) recover the assumed true parameter values. We can test this <em>in silico</em> via the `plots.recovery` function in the `diagnostics` module. For instance, we can compare how well posterior means recover the true parameter (i.e., posterior z-score, https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81416d87265a1c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = bf.diagnostics.plots.recovery(samples, test_sims)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a680fb4138ae6b3",
   "metadata": {},
   "source": [
    "Interestingly, it seems that the parameters $\\theta_1 = \\mu$ and $\\theta_2 = D$ have not been learned properly as they are estimated roughly the same for every simulated datset used during testing. For some models, this might indicate that the the network training had partially failed; and we would have to train longer or adjust the network architecture. For this specific model, however, the reason is different: From the provided observables, these parameters are actually not identified so cannot be learned consistently, no matter the kind of approximator we would use. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9414bace2440ce9",
   "metadata": {},
   "source": [
    "### Automatic Diagnostics\n",
    "\n",
    "The basic workflow object wraps together a bunch of useful functions that can be called automatically. For instance, we can easily obtain numerical error estimates for the big three: normalized roor mean square error (NRMSE), posterior contraction, and calibration, for $300$ new data sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a80e870b7eac9f29",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = workflow.compute_diagnostics(test_data=300)\n",
    "metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4e86c702256c1d3",
   "metadata": {},
   "source": [
    "We can also obtain the full set of graphical diagnostics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e3e4fc643dd226f",
   "metadata": {},
   "outputs": [],
   "source": [
    "figures = workflow.plot_diagnostics(\n",
    "    test_data=300,\n",
    "    loss_kwargs={\"figsize\": (15, 3), \"label_fontsize\": 12},\n",
    "    recovery_kwargs={\"figsize\": (15, 3), \"label_fontsize\": 12},\n",
    "    calibration_ecdf_kwargs={\"figsize\": (15, 3), \"legend_fontsize\": 8, \"difference\": True, \"label_fontsize\": 12},\n",
    "    z_score_contraction_kwargs={\"figsize\": (15, 3), \"label_fontsize\": 12}    \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3637102a2416b6ca",
   "metadata": {},
   "source": [
    "## Inference Phase <a class=\"anchor\" id=\"inference_phase\"></a>\n",
    "\n",
    "We can now move on to using real data. This is easy, and since we are using an adapter, the same transformations applied during training will be applied during the inference phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aef0e4c163e0ea6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our real-data loader returns the time series as a 1D array\n",
    "obs_cases = load_data()\n",
    "\n",
    "# Note that we transform the 1D array into shape (1, T), indicating one time series\n",
    "samples = workflow.sample(conditions={\"cases\": obs_cases[None, :]}, num_samples=num_samples)\n",
    "\n",
    "# Convert into a nice format 2D data frame\n",
    "samples = workflow.samples_to_data_frame(samples)\n",
    "\n",
    "samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5ba3061de00015",
   "metadata": {},
   "source": [
    "### Posterior Retrodictive Checks <a class=\"anchor\" id=\"posterior_retrodictive_checks\"></a>\n",
    "\n",
    "These are also called <em>posterior predictive checks</em>, but here we want to explicitly highlight the fact that we are not predicting future data but testing the <strong>generative performance</strong> or <strong>re-simulation performance</strong> of the model. In other words, we want to test how well the simulator can reproduce the actually observed data given the parameter posterior $p(\\theta \\mid x_{1:T})$. \n",
    "\n",
    "Here, we will create a custom function which plots the observed data and then overlays draws from the posterior predictive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25c5c5ad1a3e0071",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ppc(samples, obs_cases, logscale=True, color=\"#132a70\", figsize=(12, 6), font_size=18):\n",
    "    \"\"\"\n",
    "    Helper function to perform some plotting of the posterior predictive.\n",
    "    \"\"\"\n",
    "    # Plot settings\n",
    "    plt.rcParams[\"font.size\"] = font_size\n",
    "    f, ax = plt.subplots(1, 1, figsize=figsize)\n",
    "    T = len(obs_cases)\n",
    "\n",
    "    # Re-simulations\n",
    "    sims = []\n",
    "    for i in range(samples.shape[0]):\n",
    "        # Note - simulator returns 2D arrays of shape (T, 1), so we remove trailing dim\n",
    "        sim_cases = stationary_SIR(*samples.values[i])\n",
    "        sims.append(sim_cases[\"cases\"])\n",
    "    sims = np.array(sims)\n",
    "\n",
    "    # Compute quantiles for each t = 1,...,T\n",
    "    qs_50 = np.quantile(sims, q=[0.25, 0.75], axis=0)\n",
    "    qs_90 = np.quantile(sims, q=[0.05, 0.95], axis=0)\n",
    "    qs_95 = np.quantile(sims, q=[0.025, 0.975], axis=0)\n",
    "\n",
    "    # Plot median predictions and observed data\n",
    "    ax.plot(np.median(sims, axis=0), label=\"Median predicted cases\", color=color)\n",
    "    ax.plot(obs_cases, marker=\"o\", label=\"Reported cases\", color=\"black\", linestyle=\"dashed\", alpha=0.8)\n",
    "\n",
    "    # Add compatibility intervals (also called credible intervals)\n",
    "    ax.fill_between(range(T), qs_50[0], qs_50[1], color=color, alpha=0.5, label=\"50% CI\")\n",
    "    ax.fill_between(range(T), qs_90[0], qs_90[1], color=color, alpha=0.3, label=\"90% CI\")\n",
    "    ax.fill_between(range(T), qs_95[0], qs_95[1], color=color, alpha=0.1, label=\"95% CI\")\n",
    "\n",
    "    # Grid and schmuck\n",
    "    ax.grid(color=\"grey\", linestyle=\"-\", linewidth=0.25, alpha=0.5)\n",
    "    ax.spines[\"right\"].set_visible(False)\n",
    "    ax.spines[\"top\"].set_visible(False)\n",
    "    ax.set_xlabel(\"Days since pandemic onset\")\n",
    "    ax.set_ylabel(\"Number of cases\")\n",
    "    ax.minorticks_off()\n",
    "    if logscale:\n",
    "        ax.set_yscale(\"log\")\n",
    "    ax.legend(fontsize=font_size)\n",
    "    return f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b317bb5eaa5a634",
   "metadata": {},
   "source": [
    "We can now go on and plot the re-simulations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5540a4c6db5fe53",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = plot_ppc(samples, obs_cases)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "611b9c9fe8769f0f",
   "metadata": {},
   "source": [
    "That's it for this tutorial! You now know how to use the basic building blocks of `BayesFlow` to create amortized neural approximators. :)\n",
    "\n",
    "<!-- In the [next tutorial](./PriorSensitivity_Covid19_Initial.ipynb), we will go through a <strong>prior sensitivity analysis</strong> with `BayesFlow`, which is as easy to perform as it is important for ascertaining the robustness of our inferences. -->"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bf",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}