docs/demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Demo\n",
    "\n",
    "This notebook demonstrates the basic functionality of the `perfectns` package; for background see the dynamic nested sampling paper [(Higson at al., 2019a)](https://doi.org/10.1007/s11222-018-9844-0).\n",
    "\n",
    "### Running nested sampling calculations\n",
    "\n",
    "The likelihood $\\mathcal{L}(\\theta)$, prior $\\pi(\\theta)$ and calculation settings are specified in a PerfectNSSettings object. For this example we will use a 10-dimensional spherically symmetric Gaussian likelihood with size $\\sigma_\\mathcal{L}=1$ and a Gaussian prior with size $\\sigma_{\\pi}=10$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import perfectns.settings\n",
    "import perfectns.likelihoods as likelihoods\n",
    "import perfectns.priors as priors\n",
    "\n",
    "# Input settings\n",
    "settings = perfectns.settings.PerfectNSSettings()\n",
    "settings.likelihood = likelihoods.Gaussian(likelihood_scale=1)\n",
    "settings.prior = priors.Gaussian(prior_scale=10)\n",
    "settings.n_dim = 10\n",
    "settings.ninit = 10\n",
    "settings.nlive_const = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"dynamic_goal\" setting determines if dynamic nested sampling should be used and, if so, how to split the computational effort between increasing parameter estimation accuracy and evidence calculation accuracy. dynamic_goal=1 optimises purely for parameter estimation and dynamic_goal=0 optimises purely for calculating the Bayesian evidence $\\mathcal{Z}$.\n",
    "\n",
    "Lets try running standard nested sampling and dynamic nested sampling calculation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import perfectns.nested_sampling as nested_sampling\n",
    "\n",
    "# Perform standard nested sampling\n",
    "settings.dynamic_goal = None\n",
    "standard_ns_run = nested_sampling.generate_ns_run(settings, random_seed=0)  # set random_seed for reproducible results\n",
    "# Perform dynamic nested sampling\n",
    "settings.dynamic_goal = 1  # optimise for parameter estimation accuracy\n",
    "dynamic_ns_run = nested_sampling.generate_ns_run(settings, random_seed=0)  # set random_seed for reproducible results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now make posterior inferences using the samples generated by the nested sampling calculations using the utility functions from ``nestcheck``. Here we calculate:\n",
    "\n",
    "1\\. the log Bayesian evidence $\\log \\mathcal{Z}=\\log \\left( \\int \\mathcal{L}(\\theta) \\pi(\\theta) \\mathrm{d}\\theta \\right)$,\n",
    "\n",
    "2\\. the mean of the first parameter $\\theta_1$,\n",
    "\n",
    "3\\. the second moment of the posterior distribution of $\\theta_1$,\n",
    "\n",
    "4\\. the median of $\\theta_1$,\n",
    "\n",
    "5\\. the 84% one-tailed credible interval on $\\theta_1$.\n",
    "\n",
    "For the Gaussian likelihood and prior we can calculate the posterior distribution analytically, so we first calculate the analytic values of each quantity for comparison. The results are displayed in a `pandas` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>$\\mathrm{log} \\mathcal{Z}$</th>\n",
       "      <th>$\\overline{\\theta_{\\hat{1}}}$</th>\n",
       "      <th>$\\overline{\\theta^2_{\\hat{1}}}$</th>\n",
       "      <th>$\\mathrm{median}(\\theta_{\\hat{1}})$</th>\n",
       "      <th>$\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>standard run</th>\n",
       "      <td>-32.289475</td>\n",
       "      <td>-0.029820</td>\n",
       "      <td>0.971677</td>\n",
       "      <td>-0.046113</td>\n",
       "      <td>0.945738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dynamic run</th>\n",
       "      <td>-33.053897</td>\n",
       "      <td>-0.054664</td>\n",
       "      <td>0.931194</td>\n",
       "      <td>-0.093203</td>\n",
       "      <td>0.913430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>true values</th>\n",
       "      <td>-32.264988</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.990080</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.989523</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              $\\mathrm{log} \\mathcal{Z}$  $\\overline{\\theta_{\\hat{1}}}$  \\\n",
       "standard run                  -32.289475                      -0.029820   \n",
       "dynamic run                   -33.053897                      -0.054664   \n",
       "true values                   -32.264988                       0.000000   \n",
       "\n",
       "              $\\overline{\\theta^2_{\\hat{1}}}$  \\\n",
       "standard run                         0.971677   \n",
       "dynamic run                          0.931194   \n",
       "true values                          0.990080   \n",
       "\n",
       "              $\\mathrm{median}(\\theta_{\\hat{1}})$  \\\n",
       "standard run                            -0.046113   \n",
       "dynamic run                             -0.093203   \n",
       "true values                              0.000000   \n",
       "\n",
       "              $\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$  \n",
       "standard run                                  0.945738  \n",
       "dynamic run                                   0.913430  \n",
       "true values                                   0.989523  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import perfectns.estimators as e\n",
    "import nestcheck.ns_run_utils\n",
    "import pandas as pd\n",
    "\n",
    "estimator_list = [e.LogZ(),\n",
    "                  e.ParamMean(),\n",
    "                  e.ParamSquaredMean(),\n",
    "                  e.ParamCred(0.5),\n",
    "                  e.ParamCred(0.84)]\n",
    "estimator_names = [est.latex_name for est in estimator_list]\n",
    "results = pd.DataFrame([nestcheck.ns_run_utils.run_estimators(standard_ns_run, estimator_list),\n",
    "                        nestcheck.ns_run_utils.run_estimators(dynamic_ns_run, estimator_list)],\n",
    "                       columns=estimator_names, index=['standard run', 'dynamic run'])\n",
    "# Add true values for comparison\n",
    "results.loc['true values'] = e.get_true_estimator_values(estimator_list, settings)\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimating sampling errors\n",
    "\n",
    "You can estimate the numerical uncertainties on these results by calculating the standard deviation of the sampling errors distributions each run using the bootstrap resampling approach described in [Higson et al. (2018)](https://doi.org/10.1214/17-BA1075) (implemented in `nestcheck`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>$\\mathrm{log} \\mathcal{Z}$</th>\n",
       "      <th>$\\overline{\\theta_{\\hat{1}}}$</th>\n",
       "      <th>$\\overline{\\theta^2_{\\hat{1}}}$</th>\n",
       "      <th>$\\mathrm{median}(\\theta_{\\hat{1}})$</th>\n",
       "      <th>$\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>standard unc</th>\n",
       "      <td>0.366472</td>\n",
       "      <td>0.031321</td>\n",
       "      <td>0.049564</td>\n",
       "      <td>0.043201</td>\n",
       "      <td>0.050531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dynamic unc</th>\n",
       "      <td>1.204710</td>\n",
       "      <td>0.017033</td>\n",
       "      <td>0.030143</td>\n",
       "      <td>0.020610</td>\n",
       "      <td>0.027900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              $\\mathrm{log} \\mathcal{Z}$  $\\overline{\\theta_{\\hat{1}}}$  \\\n",
       "standard unc                    0.366472                       0.031321   \n",
       "dynamic unc                     1.204710                       0.017033   \n",
       "\n",
       "              $\\overline{\\theta^2_{\\hat{1}}}$  \\\n",
       "standard unc                         0.049564   \n",
       "dynamic unc                          0.030143   \n",
       "\n",
       "              $\\mathrm{median}(\\theta_{\\hat{1}})$  \\\n",
       "standard unc                             0.043201   \n",
       "dynamic unc                              0.020610   \n",
       "\n",
       "              $\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$  \n",
       "standard unc                                  0.050531  \n",
       "dynamic unc                                   0.027900  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import nestcheck.error_analysis\n",
    "np.random.seed(0)\n",
    "results.loc['standard unc'] = nestcheck.error_analysis.run_std_bootstrap(\n",
    "    standard_ns_run, estimator_list, n_simulate=200)\n",
    "results.loc['dynamic unc'] = nestcheck.error_analysis.run_std_bootstrap(\n",
    "    dynamic_ns_run, estimator_list, n_simulate=200)\n",
    "results.loc[['standard unc', 'dynamic unc']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach works for both dynamic and standard nested sampling. In addition as `perfectns` can perform the nested sampling algorithm \"perfectly\" there are no additional errors from implementation-specific effects such as correlated samples (see [Higson et al., 2019b](http://doi.org/10.1093/mnras/sty3090) for a detailed discussion)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating and analysing runs in parallel\n",
    "\n",
    "Multiple nested sampling runs can be generated and analysed in parallel (using `parallel_utils` from `nestcheck`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60a4012481274ffdb94de93b27ee7f9e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4f9639f9c944164b53a9ef0b8892408",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>$\\mathrm{log} \\mathcal{Z}$</th>\n",
       "      <th>$\\overline{\\theta_{\\hat{1}}}$</th>\n",
       "      <th>$\\overline{\\theta^2_{\\hat{1}}}$</th>\n",
       "      <th>$\\mathrm{median}(\\theta_{\\hat{1}})$</th>\n",
       "      <th>$\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>calculation type</th>\n",
       "      <th>result type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">mean</th>\n",
       "      <th>value</th>\n",
       "      <td>-32.139762</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>0.992171</td>\n",
       "      <td>-0.001141</td>\n",
       "      <td>0.992247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.114168</td>\n",
       "      <td>0.001896</td>\n",
       "      <td>0.002937</td>\n",
       "      <td>0.002478</td>\n",
       "      <td>0.002826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">std</th>\n",
       "      <th>value</th>\n",
       "      <td>1.141684</td>\n",
       "      <td>0.018956</td>\n",
       "      <td>0.029367</td>\n",
       "      <td>0.024777</td>\n",
       "      <td>0.028265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.081136</td>\n",
       "      <td>0.001347</td>\n",
       "      <td>0.002087</td>\n",
       "      <td>0.001761</td>\n",
       "      <td>0.002009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              $\\mathrm{log} \\mathcal{Z}$  \\\n",
       "calculation type result type                               \n",
       "mean             value                        -32.139762   \n",
       "                 uncertainty                    0.114168   \n",
       "std              value                          1.141684   \n",
       "                 uncertainty                    0.081136   \n",
       "\n",
       "                              $\\overline{\\theta_{\\hat{1}}}$  \\\n",
       "calculation type result type                                  \n",
       "mean             value                            -0.000028   \n",
       "                 uncertainty                       0.001896   \n",
       "std              value                             0.018956   \n",
       "                 uncertainty                       0.001347   \n",
       "\n",
       "                              $\\overline{\\theta^2_{\\hat{1}}}$  \\\n",
       "calculation type result type                                    \n",
       "mean             value                               0.992171   \n",
       "                 uncertainty                         0.002937   \n",
       "std              value                               0.029367   \n",
       "                 uncertainty                         0.002087   \n",
       "\n",
       "                              $\\mathrm{median}(\\theta_{\\hat{1}})$  \\\n",
       "calculation type result type                                        \n",
       "mean             value                                  -0.001141   \n",
       "                 uncertainty                             0.002478   \n",
       "std              value                                   0.024777   \n",
       "                 uncertainty                             0.001761   \n",
       "\n",
       "                              $\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$  \n",
       "calculation type result type                                            \n",
       "mean             value                                        0.992247  \n",
       "                 uncertainty                                  0.002826  \n",
       "std              value                                        0.028265  \n",
       "                 uncertainty                                  0.002009  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import nestcheck.parallel_utils as pu\n",
    "import nestcheck.pandas_functions as pf\n",
    "\n",
    "# Generate 100 nested sampling runs\n",
    "run_list = nested_sampling.get_run_data(settings, 100, save=False, load=False, random_seeds=list(range(100)))\n",
    "# Calculate posterior inferences for each run\n",
    "values = pu.parallel_apply(nestcheck.ns_run_utils.run_estimators, run_list,\n",
    "                           func_args=(estimator_list,))\n",
    "# Show the mean and standard deviation of the calculation results\n",
    "multi_run_tests = pf.summary_df_from_list(values, estimator_names)\n",
    "multi_run_tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing dynamic and standard nested sampling performance\n",
    "  \n",
    "Lets now compare the performance of dynamic and standard nested sampling, using the 10-dimensional Gaussian likelihood and prior. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dynamic_goal=None n_samples_max=None\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51b9b62f07cf409890a6b982a420bb11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "832da42b1f02429e98b76975c4ab6ad0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "dynamic_goal=0 n_samples_max=2992\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9db0bde8fa84743a8e525b159dea0a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c738a1b32924756bf479c5f82789912",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "dynamic_goal=1 n_samples_max=3007\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b675d3990233435aa74dfe53bd097e3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "862e1e98a4ce41abb103c15d230a73ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>$\\mathrm{log} \\mathcal{Z}$</th>\n",
       "      <th>$\\overline{\\theta_{\\hat{1}}}$</th>\n",
       "      <th>$\\overline{\\theta^2_{\\hat{1}}}$</th>\n",
       "      <th>$\\mathrm{median}(\\theta_{\\hat{1}})$</th>\n",
       "      <th>$\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>calculation type</th>\n",
       "      <th>dynamic settings</th>\n",
       "      <th>result type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">mean</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">standard</th>\n",
       "      <th>value</th>\n",
       "      <td>-32.259097</td>\n",
       "      <td>-0.004415</td>\n",
       "      <td>0.984532</td>\n",
       "      <td>0.001289</td>\n",
       "      <td>0.981203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.040170</td>\n",
       "      <td>0.003470</td>\n",
       "      <td>0.004505</td>\n",
       "      <td>0.004431</td>\n",
       "      <td>0.005580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=0$</th>\n",
       "      <th>value</th>\n",
       "      <td>-32.218098</td>\n",
       "      <td>-0.002052</td>\n",
       "      <td>0.979013</td>\n",
       "      <td>-0.001307</td>\n",
       "      <td>0.978076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.034715</td>\n",
       "      <td>0.004263</td>\n",
       "      <td>0.006417</td>\n",
       "      <td>0.005667</td>\n",
       "      <td>0.007123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=1$</th>\n",
       "      <th>value</th>\n",
       "      <td>-32.143640</td>\n",
       "      <td>-0.000484</td>\n",
       "      <td>0.992710</td>\n",
       "      <td>-0.001467</td>\n",
       "      <td>0.992193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.113207</td>\n",
       "      <td>0.001923</td>\n",
       "      <td>0.002890</td>\n",
       "      <td>0.002494</td>\n",
       "      <td>0.002868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">std</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">standard</th>\n",
       "      <th>value</th>\n",
       "      <td>0.401704</td>\n",
       "      <td>0.034697</td>\n",
       "      <td>0.045052</td>\n",
       "      <td>0.044311</td>\n",
       "      <td>0.055795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.028548</td>\n",
       "      <td>0.002466</td>\n",
       "      <td>0.003202</td>\n",
       "      <td>0.003149</td>\n",
       "      <td>0.003965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=0$</th>\n",
       "      <th>value</th>\n",
       "      <td>0.347150</td>\n",
       "      <td>0.042629</td>\n",
       "      <td>0.064168</td>\n",
       "      <td>0.056671</td>\n",
       "      <td>0.071234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.024671</td>\n",
       "      <td>0.003029</td>\n",
       "      <td>0.004560</td>\n",
       "      <td>0.004027</td>\n",
       "      <td>0.005062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=1$</th>\n",
       "      <th>value</th>\n",
       "      <td>1.132066</td>\n",
       "      <td>0.019231</td>\n",
       "      <td>0.028904</td>\n",
       "      <td>0.024941</td>\n",
       "      <td>0.028678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.080452</td>\n",
       "      <td>0.001367</td>\n",
       "      <td>0.002054</td>\n",
       "      <td>0.001772</td>\n",
       "      <td>0.002038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">std efficiency gain</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=0$</th>\n",
       "      <th>value</th>\n",
       "      <td>1.338991</td>\n",
       "      <td>0.662480</td>\n",
       "      <td>0.492935</td>\n",
       "      <td>0.611363</td>\n",
       "      <td>0.613510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.269147</td>\n",
       "      <td>0.133164</td>\n",
       "      <td>0.099084</td>\n",
       "      <td>0.122889</td>\n",
       "      <td>0.123320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">dynamic $G=1$</th>\n",
       "      <th>value</th>\n",
       "      <td>0.125912</td>\n",
       "      <td>3.255211</td>\n",
       "      <td>2.429505</td>\n",
       "      <td>3.156470</td>\n",
       "      <td>3.785308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.025309</td>\n",
       "      <td>0.654322</td>\n",
       "      <td>0.488349</td>\n",
       "      <td>0.634474</td>\n",
       "      <td>0.760876</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  $\\mathrm{log} \\mathcal{Z}$  \\\n",
       "calculation type    dynamic settings result type                               \n",
       "mean                standard         value                        -32.259097   \n",
       "                                     uncertainty                    0.040170   \n",
       "                    dynamic $G=0$    value                        -32.218098   \n",
       "                                     uncertainty                    0.034715   \n",
       "                    dynamic $G=1$    value                        -32.143640   \n",
       "                                     uncertainty                    0.113207   \n",
       "std                 standard         value                          0.401704   \n",
       "                                     uncertainty                    0.028548   \n",
       "                    dynamic $G=0$    value                          0.347150   \n",
       "                                     uncertainty                    0.024671   \n",
       "                    dynamic $G=1$    value                          1.132066   \n",
       "                                     uncertainty                    0.080452   \n",
       "std efficiency gain dynamic $G=0$    value                          1.338991   \n",
       "                                     uncertainty                    0.269147   \n",
       "                    dynamic $G=1$    value                          0.125912   \n",
       "                                     uncertainty                    0.025309   \n",
       "\n",
       "                                                  $\\overline{\\theta_{\\hat{1}}}$  \\\n",
       "calculation type    dynamic settings result type                                  \n",
       "mean                standard         value                            -0.004415   \n",
       "                                     uncertainty                       0.003470   \n",
       "                    dynamic $G=0$    value                            -0.002052   \n",
       "                                     uncertainty                       0.004263   \n",
       "                    dynamic $G=1$    value                            -0.000484   \n",
       "                                     uncertainty                       0.001923   \n",
       "std                 standard         value                             0.034697   \n",
       "                                     uncertainty                       0.002466   \n",
       "                    dynamic $G=0$    value                             0.042629   \n",
       "                                     uncertainty                       0.003029   \n",
       "                    dynamic $G=1$    value                             0.019231   \n",
       "                                     uncertainty                       0.001367   \n",
       "std efficiency gain dynamic $G=0$    value                             0.662480   \n",
       "                                     uncertainty                       0.133164   \n",
       "                    dynamic $G=1$    value                             3.255211   \n",
       "                                     uncertainty                       0.654322   \n",
       "\n",
       "                                                  $\\overline{\\theta^2_{\\hat{1}}}$  \\\n",
       "calculation type    dynamic settings result type                                    \n",
       "mean                standard         value                               0.984532   \n",
       "                                     uncertainty                         0.004505   \n",
       "                    dynamic $G=0$    value                               0.979013   \n",
       "                                     uncertainty                         0.006417   \n",
       "                    dynamic $G=1$    value                               0.992710   \n",
       "                                     uncertainty                         0.002890   \n",
       "std                 standard         value                               0.045052   \n",
       "                                     uncertainty                         0.003202   \n",
       "                    dynamic $G=0$    value                               0.064168   \n",
       "                                     uncertainty                         0.004560   \n",
       "                    dynamic $G=1$    value                               0.028904   \n",
       "                                     uncertainty                         0.002054   \n",
       "std efficiency gain dynamic $G=0$    value                               0.492935   \n",
       "                                     uncertainty                         0.099084   \n",
       "                    dynamic $G=1$    value                               2.429505   \n",
       "                                     uncertainty                         0.488349   \n",
       "\n",
       "                                                  $\\mathrm{median}(\\theta_{\\hat{1}})$  \\\n",
       "calculation type    dynamic settings result type                                        \n",
       "mean                standard         value                                   0.001289   \n",
       "                                     uncertainty                             0.004431   \n",
       "                    dynamic $G=0$    value                                  -0.001307   \n",
       "                                     uncertainty                             0.005667   \n",
       "                    dynamic $G=1$    value                                  -0.001467   \n",
       "                                     uncertainty                             0.002494   \n",
       "std                 standard         value                                   0.044311   \n",
       "                                     uncertainty                             0.003149   \n",
       "                    dynamic $G=0$    value                                   0.056671   \n",
       "                                     uncertainty                             0.004027   \n",
       "                    dynamic $G=1$    value                                   0.024941   \n",
       "                                     uncertainty                             0.001772   \n",
       "std efficiency gain dynamic $G=0$    value                                   0.611363   \n",
       "                                     uncertainty                             0.122889   \n",
       "                    dynamic $G=1$    value                                   3.156470   \n",
       "                                     uncertainty                             0.634474   \n",
       "\n",
       "                                                  $\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$  \n",
       "calculation type    dynamic settings result type                                            \n",
       "mean                standard         value                                        0.981203  \n",
       "                                     uncertainty                                  0.005580  \n",
       "                    dynamic $G=0$    value                                        0.978076  \n",
       "                                     uncertainty                                  0.007123  \n",
       "                    dynamic $G=1$    value                                        0.992193  \n",
       "                                     uncertainty                                  0.002868  \n",
       "std                 standard         value                                        0.055795  \n",
       "                                     uncertainty                                  0.003965  \n",
       "                    dynamic $G=0$    value                                        0.071234  \n",
       "                                     uncertainty                                  0.005062  \n",
       "                    dynamic $G=1$    value                                        0.028678  \n",
       "                                     uncertainty                                  0.002038  \n",
       "std efficiency gain dynamic $G=0$    value                                        0.613510  \n",
       "                                     uncertainty                                  0.123320  \n",
       "                    dynamic $G=1$    value                                        3.785308  \n",
       "                                     uncertainty                                  0.760876  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import perfectns.results_tables as rt\n",
    "\n",
    "# Input settings\n",
    "settings = perfectns.settings.PerfectNSSettings()\n",
    "settings.likelihood = likelihoods.Gaussian(likelihood_scale=1)\n",
    "settings.prior = priors.Gaussian(prior_scale=10)\n",
    "settings.ninit = 10\n",
    "settings.nlive_const = 100\n",
    "settings.n_dim = 10\n",
    "# Run results settings\n",
    "dynamic_results_table = rt.get_dynamic_results(100, [0, 1], estimator_list, settings, save=False, load=False)\n",
    "dynamic_results_table[estimator_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the `std efficiency gain` rows, you should see that dynamic nested sampling targeted at parameter estimation (dynamic goal=1) has an efficiency gain (equivalent computational speedup) for parameter estimation (columns other than $\\log \\mathcal{Z}$) of factor of around 3 compared to standard nested sampling.\n",
    "\n",
    "Similar results tables for different likelihoods can be found in the dynamic nested sampling paper [(Higson at al., 2019a)](https://doi.org/10.1007/s11222-018-9844-0). For more information about the get_dynamic_results function look at its documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing bootstrap error estimates to observed distributions of results\n",
    "\n",
    "We can check if the bootstrap estimates of parameter estimation sampling errors are accurate, using a 3d Gaussian likelihood and Gaussian prior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a11ca414a82471baca853c56945b650",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=50), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0362a6d056dd4c0b9245f61e9889db04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=50), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10365434da634dec8b70ce651df37cd6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=50), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef3f44e1eade46a5bfad417ff079fc41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=20), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>$\\mathrm{log} \\mathcal{Z}$</th>\n",
       "      <th>$\\overline{\\theta_{\\hat{1}}}$</th>\n",
       "      <th>$\\overline{\\theta^2_{\\hat{1}}}$</th>\n",
       "      <th>$\\mathrm{median}(\\theta_{\\hat{1}})$</th>\n",
       "      <th>$\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>calculation type</th>\n",
       "      <th>result type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">repeats mean</th>\n",
       "      <th>value</th>\n",
       "      <td>-9.625403</td>\n",
       "      <td>0.002816</td>\n",
       "      <td>0.987749</td>\n",
       "      <td>0.007308</td>\n",
       "      <td>0.999308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.036342</td>\n",
       "      <td>0.007468</td>\n",
       "      <td>0.010398</td>\n",
       "      <td>0.009250</td>\n",
       "      <td>0.012202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">repeats std</th>\n",
       "      <th>value</th>\n",
       "      <td>0.256977</td>\n",
       "      <td>0.052807</td>\n",
       "      <td>0.073526</td>\n",
       "      <td>0.065408</td>\n",
       "      <td>0.086281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.025959</td>\n",
       "      <td>0.005334</td>\n",
       "      <td>0.007427</td>\n",
       "      <td>0.006607</td>\n",
       "      <td>0.008716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">bs std / repeats std</th>\n",
       "      <th>value</th>\n",
       "      <td>0.892930</td>\n",
       "      <td>0.873771</td>\n",
       "      <td>0.925045</td>\n",
       "      <td>0.972450</td>\n",
       "      <td>0.920943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.091285</td>\n",
       "      <td>0.089760</td>\n",
       "      <td>0.095384</td>\n",
       "      <td>0.100413</td>\n",
       "      <td>0.096081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">bs estimate % variation</th>\n",
       "      <th>value</th>\n",
       "      <td>11.112636</td>\n",
       "      <td>13.206385</td>\n",
       "      <td>14.631458</td>\n",
       "      <td>15.134609</td>\n",
       "      <td>18.444304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>1.122546</td>\n",
       "      <td>1.334046</td>\n",
       "      <td>1.478000</td>\n",
       "      <td>1.528826</td>\n",
       "      <td>1.863156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">bs 0.95 CI</th>\n",
       "      <th>value</th>\n",
       "      <td>-9.220209</td>\n",
       "      <td>0.070315</td>\n",
       "      <td>1.114863</td>\n",
       "      <td>0.097298</td>\n",
       "      <td>1.127381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uncertainty</th>\n",
       "      <td>0.044147</td>\n",
       "      <td>0.012680</td>\n",
       "      <td>0.018326</td>\n",
       "      <td>0.016915</td>\n",
       "      <td>0.018304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs +-1std % coverage</th>\n",
       "      <th>value</th>\n",
       "      <td>72.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>64.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs 0.95 CI % coverage</th>\n",
       "      <th>value</th>\n",
       "      <td>94.000000</td>\n",
       "      <td>90.000000</td>\n",
       "      <td>96.000000</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>92.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     $\\mathrm{log} \\mathcal{Z}$  \\\n",
       "calculation type        result type                               \n",
       "repeats mean            value                         -9.625403   \n",
       "                        uncertainty                    0.036342   \n",
       "repeats std             value                          0.256977   \n",
       "                        uncertainty                    0.025959   \n",
       "bs std / repeats std    value                          0.892930   \n",
       "                        uncertainty                    0.091285   \n",
       "bs estimate % variation value                         11.112636   \n",
       "                        uncertainty                    1.122546   \n",
       "bs 0.95 CI              value                         -9.220209   \n",
       "                        uncertainty                    0.044147   \n",
       "bs +-1std % coverage    value                         72.000000   \n",
       "bs 0.95 CI % coverage   value                         94.000000   \n",
       "\n",
       "                                     $\\overline{\\theta_{\\hat{1}}}$  \\\n",
       "calculation type        result type                                  \n",
       "repeats mean            value                             0.002816   \n",
       "                        uncertainty                       0.007468   \n",
       "repeats std             value                             0.052807   \n",
       "                        uncertainty                       0.005334   \n",
       "bs std / repeats std    value                             0.873771   \n",
       "                        uncertainty                       0.089760   \n",
       "bs estimate % variation value                            13.206385   \n",
       "                        uncertainty                       1.334046   \n",
       "bs 0.95 CI              value                             0.070315   \n",
       "                        uncertainty                       0.012680   \n",
       "bs +-1std % coverage    value                            62.000000   \n",
       "bs 0.95 CI % coverage   value                            90.000000   \n",
       "\n",
       "                                     $\\overline{\\theta^2_{\\hat{1}}}$  \\\n",
       "calculation type        result type                                    \n",
       "repeats mean            value                               0.987749   \n",
       "                        uncertainty                         0.010398   \n",
       "repeats std             value                               0.073526   \n",
       "                        uncertainty                         0.007427   \n",
       "bs std / repeats std    value                               0.925045   \n",
       "                        uncertainty                         0.095384   \n",
       "bs estimate % variation value                              14.631458   \n",
       "                        uncertainty                         1.478000   \n",
       "bs 0.95 CI              value                               1.114863   \n",
       "                        uncertainty                         0.018326   \n",
       "bs +-1std % coverage    value                              52.000000   \n",
       "bs 0.95 CI % coverage   value                              96.000000   \n",
       "\n",
       "                                     $\\mathrm{median}(\\theta_{\\hat{1}})$  \\\n",
       "calculation type        result type                                        \n",
       "repeats mean            value                                   0.007308   \n",
       "                        uncertainty                             0.009250   \n",
       "repeats std             value                                   0.065408   \n",
       "                        uncertainty                             0.006607   \n",
       "bs std / repeats std    value                                   0.972450   \n",
       "                        uncertainty                             0.100413   \n",
       "bs estimate % variation value                                  15.134609   \n",
       "                        uncertainty                             1.528826   \n",
       "bs 0.95 CI              value                                   0.097298   \n",
       "                        uncertainty                             0.016915   \n",
       "bs +-1std % coverage    value                                  62.000000   \n",
       "bs 0.95 CI % coverage   value                                  92.000000   \n",
       "\n",
       "                                     $\\mathrm{C.I.}_{84\\%}(\\theta_{\\hat{1}})$  \n",
       "calculation type        result type                                            \n",
       "repeats mean            value                                        0.999308  \n",
       "                        uncertainty                                  0.012202  \n",
       "repeats std             value                                        0.086281  \n",
       "                        uncertainty                                  0.008716  \n",
       "bs std / repeats std    value                                        0.920943  \n",
       "                        uncertainty                                  0.096081  \n",
       "bs estimate % variation value                                       18.444304  \n",
       "                        uncertainty                                  1.863156  \n",
       "bs 0.95 CI              value                                        1.127381  \n",
       "                        uncertainty                                  0.018304  \n",
       "bs +-1std % coverage    value                                       64.000000  \n",
       "bs 0.95 CI % coverage   value                                       92.000000  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "settings.likelihood = likelihoods.Gaussian(likelihood_scale=1)\n",
    "settings.prior = priors.Gaussian(prior_scale=10)\n",
    "settings.n_dim = 3\n",
    "bootstrap_results_table = rt.get_bootstrap_results(50, 50, # 100, 200,\n",
    "                                                   estimator_list, settings,\n",
    "                                                   n_run_ci=20,\n",
    "                                                   n_simulate_ci=200,  # n_simulate_ci=1000,\n",
    "                                                   add_sim_method=False,\n",
    "                                                   cred_int=0.95,\n",
    "                                                   ninit_sep=True,\n",
    "                                                   parallel=True)\n",
    "bootstrap_results_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that every second column gives an estimated numerical uncertainty on the values in the previous column.\n",
    "\n",
    "You should see that the ratio of the bootstrap error estimates to bootstrap_results the standard deviation of results (row 4 of bootstrap_results_table) has values close to 1 given the estimated numerical uncertainties. Similar results are given in the appendix of the dynamic nested sampling paper [(Higson, 2019a)](https://doi.org/10.1007/s11222-018-9844-0); see the paper and the get_bootstrap_results function's documentation for more details."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}