jupyter_notebooks/Tutorials/09 Bootstrapped Error Bars.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: this tutorial needs updating and has not been recently tested for basic functionality: use at own risk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import time\n",
    "import json\n",
    "\n",
    "import pygsti\n",
    "\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Load example quantities from files\n",
    "gs_target = pygsti.io.load_gateset(\"tutorial_files/Example_Gateset.txt\")\n",
    "gs_mc2gst = pygsti.io.load_gateset(\"tutorial_files/Example_MC2GST_Gateset.txt\")\n",
    "\n",
    "ds = pygsti.io.load_dataset(\"tutorial_files/Example_Dataset.txt\", cache=True)\n",
    "\n",
    "fiducials = pygsti.io.load_gatestring_list(\"tutorial_files/Example_FiducialList.txt\")\n",
    "germs = pygsti.io.load_gatestring_list(\"tutorial_files/Example_GermsList.txt\")\n",
    "maxLengths = json.load(open(\"tutorial_files/Example_maxLengths.json\",\"r\"))\n",
    "\n",
    "specs = pygsti.construction.build_spam_specs(fiducials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parametric Bootstrapping\n",
    "Here we do parametric bootstrapping, as indicated by the 'parametric' argument below.\n",
    "The output is eventually stored in the \"mean\" and \"std\" GateSets, which hold the mean and standard deviation values of the set of bootstrapped gatesets (after gauge optimization).  It is this latter \"standard deviation Gateset\"\n",
    "which holds the collection of error bars.  Note: due to print setting issues, the outputs that are printed here will not necessarily reflect the true accuracy of the estimates made.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#The number of simulated datasets & gatesets made for bootstrapping purposes.  \n",
    "# For good statistics, should probably be greater than 10.\n",
    "numGatesets=10\n",
    "\n",
    "param_boot_gatesets = pygsti.drivers.make_bootstrap_gatesets(\n",
    "                        numGatesets, ds, 'parametric', fiducials, fiducials, germs, maxLengths,\n",
    "                        inputGateSet=gs_mc2gst, startSeed=0, constrainToTP=False, returnData=False,\n",
    "                        verbosity=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "gauge_opt_pboot_gatesets = pygsti.drivers.gauge_optimize_gs_list(param_boot_gatesets, gs_mc2gst,\n",
    "                                                                 constrainToTP=False, plot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pboot_mean = pygsti.drivers.to_mean_gateset(gauge_opt_pboot_gatesets, gs_mc2gst)\n",
    "pboot_std  = pygsti.drivers.to_std_gateset(gauge_opt_pboot_gatesets, gs_mc2gst)\n",
    "\n",
    "#Summary of the error bars\n",
    "print(\"Parametric bootstrapped error bars, with\", numGatesets, \"resamples\\n\")\n",
    "print(\"Error in rho vec:\") \n",
    "print(pboot_std['rho0'], end='\\n\\n')\n",
    "print(\"Error in E vec:\")\n",
    "print(pboot_std['E0'], end='\\n\\n')\n",
    "print(\"Error in Gi:\")\n",
    "print(pboot_std['Gi'], end='\\n\\n')\n",
    "print(\"Error in Gx:\")\n",
    "print(pboot_std['Gx'], end='\\n\\n')\n",
    "print(\"Error in Gy:\")\n",
    "print(pboot_std['Gy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Non-parametric Bootstrapping\n",
    "Here we do non-parametric bootstrapping, as indicated by the 'nonparametric' argument below.\n",
    "The output is again eventually stored in the \"mean\" and \"std\" GateSets, which hold the mean and standard deviation values of the set of bootstrapped gatesets (after gauge optimization).  It is this latter \"standard deviation Gateset\"\n",
    "which holds the collection of error bars.  Note: due to print setting issues, the outputs that are printed here will not necessarily reflect the true accuracy of the estimates made.\n",
    "\n",
    "(Technical note: ddof = 1 is by default used when computing the standard deviation -- see numpy.std -- meaning that we are computing a standard deviation of the sample, not of the population.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#The number of simulated datasets & gatesets made for bootstrapping purposes.  \n",
    "# For good statistics, should probably be greater than 10.\n",
    "numGatesets=10\n",
    "\n",
    "nonparam_boot_gatesets = pygsti.drivers.make_bootstrap_gatesets(\n",
    "                          numGatesets, ds, 'nonparametric', fiducials, fiducials, germs, maxLengths,\n",
    "                          targetGateSet=gs_mc2gst, startSeed=0, constrainToTP=False, returnData=False,\n",
    "                          verbosity=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gauge_opt_npboot_gatesets = pygsti.drivers.gauge_optimize_gs_list(nonparam_boot_gatesets, gs_mc2gst,\n",
    "                                                                 constrainToTP=False, plot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "npboot_mean = pygsti.drivers.to_mean_gateset(gauge_opt_npboot_gatesets, gs_mc2gst)\n",
    "npboot_std  = pygsti.drivers.to_std_gateset(gauge_opt_npboot_gatesets, gs_mc2gst)\n",
    "\n",
    "#Summary of the error bars\n",
    "print(\"Non-parametric bootstrapped error bars, with\", numGatesets, \"resamples\\n\")\n",
    "print(\"Error in rho vec:\")\n",
    "print(npboot_std['rho0'], end='\\n\\n')\n",
    "print(\"Error in E vec:\")\n",
    "print(npboot_std['E0'], end='\\n\\n')\n",
    "print(\"Error in Gi:\")\n",
    "print(npboot_std['Gi'], end='\\n\\n')\n",
    "print(\"Error in Gx:\")\n",
    "print(npboot_std['Gx'], end='\\n\\n')\n",
    "print(\"Error in Gy:\")\n",
    "print(npboot_std['Gy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "loglog(npboot_std.to_vector(),pboot_std.to_vector(),'.')\n",
    "loglog(np.logspace(-4,-2,10),np.logspace(-4,-2,10),'--')\n",
    "xlabel('Non-parametric')\n",
    "ylabel('Parametric')\n",
    "xlim((1e-4,1e-2)); ylim((1e-4,1e-2))\n",
    "title('Scatter plot comparing param vs. non-param bootstrapping error bars.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}