report_general_main.tex

\documentclass{article}[11pt]
\usepackage{longtable}
\usepackage{graphicx}
\usepackage{fix-cm}
\usepackage[margin=1in,paperwidth=8.5in,paperheight=11in]{geometry}
\usepackage[section]{placeins}
\usepackage{flafter}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{etoolbox}
\usepackage{units}
\usepackage{multirow}
\usepackage{adjustbox}
\usepackage{array}
\usepackage{hyperref}
\usepackage{pdfcomment}
\usepackage{color}

\definecolor{darkgreen}{RGB}{0, 128, 0}

\setcounter{topnumber}{3}
\setcounter{bottomnumber}{3}
\setcounter{totalnumber}{4}
\renewcommand{\topfraction}{0.9}
\renewcommand{\bottomfraction}{0.9}
\renewcommand{\textfraction}{0.1}
\renewcommand{\floatpagefraction}{0.7}

\newcommand{\rrangle}{\rangle\!\rangle} \newcommand{\llangle}{\langle\!\langle}
\newcommand{\ket}[1]{\ensuremath{\left|#1\right\rangle}}
\newcommand{\bra}[1]{\ensuremath{\left\langle#1\right|}}
\newcommand{\braket}[2]{\ensuremath{\left\langle#1|#2\right\rangle}}
\newcommand{\expec}[1]{\ensuremath{\left\langle#1\right\rangle}}
\newcommand{\ketbra}[2]{\ket{#1}\!\!\bra{#2}}
\newcommand{\braopket}[3]{\ensuremath{\bra{#1}#2\ket{#3}}}
\newcommand{\proj}[1]{\ketbra{#1}{#1}}
\newcommand{\sket}[1]{\ensuremath{\left|#1\right\rrangle}}
\newcommand{\sbra}[1]{\ensuremath{\left\llangle#1\right|}}
\newcommand{\sbraket}[2]{\ensuremath{\left\llangle#1|#2\right\rrangle}}
\newcommand{\sketbra}[2]{\sket{#1}\!\!\sbra{#2}}
\newcommand{\sbraopket}[3]{\ensuremath{\sbra{#1}#2\sket{#3}}}
\newcommand{\sproj}[1]{\sketbra{#1}{#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\def\Id{1\!\mathrm{l}}
\newcommand{\Tr}[0]{\mathrm{Tr}}
\providecommand{\e}[1]{\ensuremath{\times 10^{#1}}}

%Command used for python automatic substitution
\newcommand{\putfield}[2]{#2}

\newcommand*{\vcenteredhbox}[1]{\begingroup
\setbox0=\hbox{#1}\parbox{\wd0}{\box0}\endgroup}

\newtoggle{confidences}
\newtoggle{LsAndGermsSet}
%\newtoggle{debuggingaidsappendix}
%\newtoggle{gaugeoptappendix}
%\newtoggle{pixelplotsappendix}
%\newtoggle{whackamoleappendix}
\putfield{settoggles}{}

\hypersetup{
  pdfinfo={ \putfield{pdfinfo}{}  }
}


\begin{document}

\title{\putfield{title}{Report Title Goes Here}}
\date{\vspace{-1cm}\today}
%\author{}

\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup

\tableofcontents

\section{Overview}
This report presents a gate-set tomography (GST) analysis of a dataset called ``\putfield{datasetLabel}{DATASET LABEL HERE}''.  

GST characterizes logic operations on a quantum device (e.g., a qubit), by treating it as a black box.  This black box is equipped with a small set of ``buttons'' that apply quantum \emph{gates} to the quantum system inside.  One button initializes it, a second button triggers an $n$-outcome measurement, and the remaining buttons perform transformations.  %add reference(s) to GST paper/ pygsti document?

GST's primary output is an estimated \emph{gate set} that models or fits the device's observed behavior.  Gate sets are of the form $\{\rho_0,E_0,\{G_k\}\}$, where
\begin{itemize}
\item $\rho_0$ is an estimate of the density matrix in which the device gets initialized,
\item $\{E_0,\Id-E_0\}$ is an estimate of the POVM describing how it gets measured,
\item and each of the $G_k$ is an estimate of the superoperator (quantum process) describing the corresponding gate.
\end{itemize}

This document is organized into three main sections, which address three broad questions.
\begin{itemize}
\item Section \ref{secInput}:  What inputs did you give GST?
\item Section \ref{secOutput}:  What estimate did GST output, and what does it mean?
\item Section \ref{secGoodness}:  How reliable are the results? (How badly was the model violated?)
\end{itemize}
Section \ref{secInput} is primarily useful to verify that the inputs were correct.  Section \ref{secOutput} is the most important:  it presents the raw estimates derived by the GST algorithm, and also provides a variety of derived quantities that may be useful in interpreting what this estimate means.

Section \ref{secGoodness} is dedicated to summarizing how well the model imposed by GST was able to fit the data, relative to what is expected of a ``good'' model.  This is \emph{not} related to ``How close is the GST estimate to the target gates?'', which is addressed in Section \ref{secOutput}.  It is also not the same as ``How large are the error bars on the GST estimate?'', which is a good question that is addressed in section \ref{secOutput} when this report is generated with the confidence intervals option turned on.  Instead, Section \ref{secGoodness} is intended to tell you whether (a) you should take the GST estimate at face value, or (b) it should be treated skeptically because \emph{no} gate set was capable of fitting the data.

%REMOVE Finally, appendices may be present (depending on which options were chosen when this report was generated).  Appendices present more detailed debugging information, elaborating on the goodness-of-fit metrics presented in Section \ref{secGoodness}.

\section{Input Summary\label{secInput}}
The input for this GST analysis comprised: (1) a target gate set (see Tables \ref{targetSpamTable}-\ref{targetGatesTable}); and (2) a dataset called ``\putfield{datasetLabel}{DATASET LABEL HERE}''.

\subsection{Target Gate set}

The target gate set describes the ideal initial state (density matrix), measurement (POVM effect), and gate operations (superoperators).  The density matrices and POVM effects are represented as square $d\times d$ matrices on a Hilbert space $\mathcal{H}$.  These are shown in table \ref{targetSpamTable}.  It is often convenient in compuatations to represent them as $d^2$-element vectors in the Hilbert-Schmidt space $\mathcal{B}(\mathcal{H})$ of linear operators on $\mathcal{H}$.  We don't, however, display this representation here because 1) it's meaning is often less intuitive than the $d\times d$ matrix, and 2) its length can make it unwieldy to write down.  Superoperators are sometimes represented in Choi or Kraus form, but for GST it is more convenient to represent them as square $d^2\times d^2$ matrices that multiply associatively and act on $\mathcal{B}(\mathcal{H})$.  These are shown in Table \ref{targetGatesTable} as heat map images to allow compact representation of large matrices.

These Hilbert-Schmidt space representations require choosing a basis $\{M_i\}$ for $\mathcal{B}(\mathcal{H})$.  We use the \emph{Pauli basis}, comprising the four $2\times2$ Pauli matrices (including the identity $\Id$) when $d=2$.  When $d>2$, we either use Gell-Mann matrices or the set of tensor products of Pauli matrices as a basis (indicated where relevant).  Keep in mind that we want to use an orthonormal basis, so the basis matrices are normalized so that $\sbraket{M_i}{M_j} = \Tr M_i^\dagger M_j = \delta_{ij}$.  In $d=2$, this means that the basis matrices are $M_i = \frac{1}{\sqrt{2}}\sigma_i$.

\begin{table}[h]
\begin{center}
\putfield{targetSpamBriefTable}{Target rho and E vectors table will be placed here}
\caption{\putfield{tt_targetSpamBriefTable}{}\textbf{Target gate set: SPAM (state preparation and measurement) gates}.  These are the \emph{ideal} input state ($\rho_0$) and `plus' POVM effect $E_0$ for the device on which we report.  SPAM gates are given here as $d\times d$ matrices.  See Table \ref{bestGatesetSpamTable} for GST estimates of the actual $\rho_0$ and $E_0$ implemented in this experiment.\label{targetSpamTable}}
\end{center}
\end{table}

The ideal state prepration and measurement (SPAM) operations for your particular case are given in Table \ref{targetSpamTable}.  The ideal \emph{logic gate} operations are given as superoperators in Table \ref{targetGatesTable}.  In most cases, the ideal/target logic gates are unitary maps.  
%REMOVE The corresponding superoperators are orthogonal rotations on $\mathcal{B}(\mathcal{H})$.  For your convenience, Table \ref{targetGatesTable} also lists (for each logic gate) an axis of rotation [as a vector in $\mathcal{B}(\mathcal{H})$] and an angle of rotation.  

\begin{table}[h]
\begin{center}
\putfield{targetGatesBoxTable}{Target gate set overview table will be placed here}
\caption{\putfield{tt_targetGatesBoxTable}{}\textbf{Target gate set: logic gates}.  These are the \emph{ideal} (generally unitary) logic gates.  Each has a name starting with ``G'', and is represented as a $d^2\times d^2$ \emph{superoperator} that acts by matrix multiplication on vectors in $\mathcal{B}(\mathcal{H})$.  Matrices are displayed using a heat map that ranges between 1.0 (red) and -1.0 (blue).\label{targetGatesTable}}
% REMOVE See Table \ref{bestGatesetGatesTable} for GST estimates of the actual logic gates implemented in this experiment.
\end{center}
\end{table}

\subsection{GST Input Data}

The most important input to GST is a \emph{dataset} -- a list of experimental counts or frequencies, each associated with a \emph{gate sequences}.  Gate sequences are also referred to as ``gate strings''.  Each gate sequence defines an experiment, in which you (1) initialize the device, (2) apply the operations specified by the gate sequence, and (3) measure and record the result (e.g.~``plus'' or ``minus'').

Typically, the gate sequences that appear in the dataset are generated by the following process:
\begin{enumerate}
\item A small set of short gate sequences called \emph{germs} are chosen,
\item A small set of short \emph{fiducial sequences} are chosen so that, when applied to $\rho_0$ or $E_0$, they generate an informationally complete set of states or effects.  Note that the set of preparation fiducial sequences does not need to be the same as the set of measurement fiducial sequences.
\item Each germ is concatenated with itself to form \emph{base sequences} of length approximately $1,2,4,8,\ldots L_{max}$.
\item Each base sequence is sandwiched between every possible pair of fiducial sequences.
\end{enumerate}
The dataset comprises all sandwiched base sequences.  A few other short sequences (e.g., those corresponding to the empty base sequence) may also appear.

\iftoggle{LsAndGermsSet}{ The fiducial sequences and germs for \emph{this} dataset are given in Tables \ref{fiducialListTable} and \ref{germListTable}. }{ Fiducial sequence and germ information was not given for this report, and may not be applicable.}  An overview of the information contained in the file you provided for dataset ``\putfield{datasetLabel}{DATASET LABEL HERE}'' is given in Table \ref{datasetOverviewTable}.  

This table also contains one derived quantity, the spectrum of the largest \emph{Gram matrix} that GST could extract from the data.  This is included here rather than in the analysis because it is not useful for predictive purposes, and therefore is not part of the estimate.  It serves, instead, to tell you something about the quality of the data.  More precisely, it tells you about the dimension of the state space that is explored by the fiducial sequences.  This should be $d^2$-dimensional [because the fiducials are intended to explore all of $\mathcal{B}(\mathcal{H})$], and therefore the spectrum listed in Table \ref{datasetOverviewTable} should (ideally) have exactly $d^2$ elements that are large and nonzero.  In practice, you should see $d^2$ large elements, and a rapid drop in magnitude thereafter.  If fewer than $d^2$ elements are large, then the fiducials were poorly chosen and are not exploring the state space effectively.  If more than $d^2$ are large, then the system is experiencing strong non-Markovian effects (e.g., strong coupling to environmental degrees of freedom) or it has a larger Hilbert space dimension than expected.

\iftoggle{LsAndGermsSet}{

\begin{table}[h]
\begin{center}
\putfield{fiducialListTable}{List of fiducials table will be placed here}
\caption{\putfield{tt_fiducialListTable}{}\textbf{Fiducial sequences.}  A list of the preparation and measurement ``fiducial'' gate sequences. See discussion in text.\label{fiducialListTable}}
\end{center}
\end{table}

\begin{table}[h]
\begin{center}
\small
\putfield{germList2ColTable}{List of germs table will be placed here}
\caption{\putfield{tt_germList2ColTable}{}\textbf{Germ sequences.}  A list of the ``germ'' gate sequences.  See discussion in text.\label{germListTable}}
\end{center}
\end{table}

}{}

\begin{table}[h]
\begin{center}
\putfield{datasetOverviewTable}{Dataset overview table will be placed here}
\caption{\putfield{tt_datasetOverviewTable}{}\textbf{General dataset properties}.  See discussion in text.\label{datasetOverviewTable}}
\end{center}
\end{table}

\section{Output from GST\label{secOutput}}

The primary output of GST is an estimated gate set.  This section presents the raw estimate, and then some useful derived quantities of the estimated gates, including comparisons to the target gates.  Some of these quanties are ``gauge-dependent'', meaning they will depend on unphysical gauge degrees of freedom that are a necessary byproduct of estimating an entire gate set at once (akin to a freedom of reference frame).  After finding a best-estimate based on the (physical) data, GST optimizes within the space of all (unphysical) gauge degrees of freedom.  Typically this is done to make the estimated gates and SPAM operations look as close to the ideal target gates as possible.  But tradeoffs can and almost always must be made regarding this optimization.  Table \ref{bestGatesetGaugeParamsTable} lists the relevant parameters used during the gauge optimization step which resulted in the estimates that follow.


\begin{table}[h]
\begin{center}
\putfield{bestGatesetGaugeOptParamsTable}{Table of gauge optimization parameters}
\caption{\putfield{tt_bestGatesetGaugeOptParamsTable}{}\textbf{Gauge Optimization Details}.  A list of the parameters used when performing the gauge optimization that produced the final GST results found in subsequent tables and figures.\label{bestGatesetGaugeParamsTable}}
\end{center}
\end{table}


\subsection{Raw GST estimates}

Table \ref{bestGatesetSpamTable} reports GST's best estimate of the SPAM operations.  The estimated SPAM operations are $d\times d$ matrices that can be expressed as length-$d^2$ vectors in $\mathcal{B}(\mathcal{H})$ by writing each matrix as a linear combinations of $d^2$ basis matrices (e.g.~the four Pauli matrices when $d=2$).  The estimated logic gates are superoperators represented as matrices acting on $\mathcal{B}(\mathcal{H})$, all in the basis specified.   By taking the dot product of state preparation and measurement vectors, estimated SPAM probabilites are computed in Table \ref{bestGatesetSpamParametersTable}.  Because the superoperator matrices can be quite large, we do not display their raw entries because of page-space limitations.  (Plotting them as heat maps similar to Table \ref{targetGatesTable} is unhelpful because the magnitude of the errors is typically so much smaller than the magniude of the elements that errors are invisible on the color scale.) Instead, we display discrepancies between the estimates and the targets using \emph{derived} quantities -- i.e., properties calculated from the gate matrices and SPAM vectors -- in the section \ref{derivedQtySection}.
\iftoggle{confidences}{
Table \ref{bestGatesetSpamTable} reports \putfield{confidenceLevel}{X}\% confidence intervals for each of the gate matrix and SPAM vector elements.  A confidence region is obtained by approximating the log-likelihood (see below) as being quadratic about its minimum, and determining the ellipsoid where this approximation equals a value, $C$, defined below.  For a given parameter (e.g. gate or SPAM vector element) $x$, a confidence interval is obtained by projecting the ellipsiodal region onto that $x$'s axis.  This computes a 1-dimensional \putfield{confidenceLevel}{X}\% confidence interval for the profile log-liklihood for $x$, and for this reason the value of $C$ used above is chosen such that $\mathrm{CDF}[\chi^2_1](C) = \putfield{confidenceLevel}{X}\%$ (that is, at the value $C$ the cumulative density function of a $\chi^2_1$ distribution reaches \putfield{confidenceLevel}{X}\%).  If, instead, the interval corresponding to a projection of the \putfield{confidenceLevel}{X}\% multi-dimensional confidence region (defined by $C$ s.t.~$\mathrm{CDF}[\chi^2_n](C) = \putfield{confidenceLevel}{X}\%$, where $n=\putfield{confidenceIntervalNumNonGaugeParams}{X}$ is the number of non-gauge gate set parameters) is desired, then the all the interval widths reported here should be multiplied by \putfield{confidenceIntervalScaleFctr}{X}.  The resulting confidence interval is always symmetric about the estimated value, and we report the half-width of the intervals in the tables.  In table \ref{bestGatesetSpamParametersTable} and those in the following section, we specify the \putfield{confidenceLevel}{-1}\% confidence intervals of derived quantities in using \emph{value} $\pm$ \emph{half-width} notation.  The derived-quantity confidence intervals in section \ref{derivedQtySection} are computed by finding the minimum and maximum values of the linearization of the derived quantity (e.g.~fidelity).
}{}


\begin{table}[h]
\begin{center}
\putfield{bestGatesetSpamBriefTable}{Best gate set rho and E vectors table will be placed here}
\caption{\putfield{tt_bestGatesetSpamBriefTable}{}\textbf{The GST estimate of the SPAM operations}.  Compare to Table \ref{targetSpamTable}.\label{bestGatesetSpamTable}}
\end{center}
\end{table}

\begin{table}[h]
\begin{center}
\putfield{bestGatesetSpamParametersTable}{Best gate set spam parametesr table will be placed here}
\caption{\putfield{tt_bestGatesetSpamParametersTable}{}\textbf{GST estimate of SPAM probabilities}.  Computed by taking the dot products of \emph{vectorized} SPAM operations in Table \ref{bestGatesetSpamTable}.  The symbol $E_C$, when it appears, refers to the ``complement'' effect given by subtracting each of the other effects from the identity.\label{bestGatesetSpamParametersTable}}
\end{center}
\end{table}

%, and Table \ref{bestGatesetGatesTable} reports the logic gate operations

%\begin{table}[h]
%\begin{center}
%XXXputfield{bestGatesetGatesTable}{Best gate set's gates table will be placed here}
%\caption{XXXputfield{tt_bestGatesetGatesTable}{}\textbf{The GST estimate of the logic gate operations}.  Compare to Table \ref{targetGatesTable}.\label{bestGatesetGatesTable}}
%\end{center}
%\end{table}

%REMOVE The estimated SPAM operations can be compared directly to the target gate set given in Section \ref{secInput}.  Ideally, they would match.  In practice, of course, they won't.  One of the best ways we have found to evaluate the significance of discrepancies is to compare \emph{derived} quantities -- i.e., certain properties calculated from the gate matrices and SPAM vectors.  Deriving quantities from these raw outputs occupies the remainder of this section.
\clearpage

\subsection{Derived quantities\label{derivedQtySection}}

Generally, the first thing that you want to know is ``How far from ideal are the gates?''.  To answer this, the report tabulates several well-known definitions of distance.  Table \ref{bestGatesetVsTargetTable} lists the discrepancy from each estimated gate to its corresponding target, as measured by:
\begin{enumerate}
\item \textbf{Process infidelity}.  Infidelity is simply $1-F$, where $F$ is the \emph{fidelity}.  The process fidelity between quantum processes $G_a$ and $G_b$ is given by $F = \Tr\left( \sqrt{ \sqrt{\chi_a} \chi_b \sqrt{\chi_a} } \right)^2$, where $\chi_a$ and $\chi_b$ are the Jamiolkowski states (normalized Choi process matrices) corresponding to gate matrices $G_a$ and $G_b$ respectively.  If the target is unitary (as is often the case), $F = \Tr\left( \chi_a \chi_b \right)$.  Process infidelity is roughly what is measured in randomized benchmarking protocols; it quantifies the \emph{incoherent} error rate if coherent errors (e.g. over-rotations) are not allowed to accumulate.
\item \textbf{Trace distance}.  This is the \emph{Jamiolkowski trace distance} between the Jamiolkowski states corresponding to the two processes:  $d_{tr} = \vert\chi_a - \chi_b\vert_1 = \Tr\left(\sqrt{(\chi_a-\chi_b)^2}\right)$.  This distance is useful primarily as a proxy for the \emph{diamond norm distance}, because $d_{tr} \leq d_{\diamond} \leq \mathrm{dim}(\mathcal{H}) d_{tr}$.
\item \textbf{Diamond Norm}.  The diamond norm between two quantum processes $G_a$ and $G_b$ is given by $\norm{G_a - G_b}_\Diamond = \sup_\rho \norm{(G_a \otimes I_k)(\rho) - (G_b \otimes I_k)(\rho)}_1$, where $I_k$ is the $k$-dimensional identity operation, $\norm{\cdot}_1$ denotes the trace norm, and the supremum is taken over all $k \ge 1$ and density matrices $\rho$ of dimension $nk$, with $n$ the dimension of $G_a$ and $G_b$.  The diamond norm is also called the \emph{completely bounded trace norm}, and plays the analogous role for quantum process distinguishability that the trace norm plays for density matrices.  Specifically, the optimal probability of distinguishing $G_a$ from $G_b$ after a \emph{single evaluation} is given by $\frac{1}{2} + \frac{1}{4}\norm{G_a - G_b}_\Diamond$.  The diamond norm distance is an upper bound on the rate of error under any possible circumstance (including coherent accumulation of errors) and is often used in proofs of fault tolerance.  For gates dominated by coherent/unitary error, it is common to see $d_{\diamond} \approx \sqrt{1-F}$.  For gates dominated by incoherent error, $d_{\diamond} \approx 1-F$.
%\item \textbf{Frobenius-norm distance}.  The Frobenius norm distance between two gates $G_a$ and $G_b$ is simply $d_F = \sqrt{\Tr\left[\left(G_a-G_b\right)^2\right]}$.  It has no known \emph{operational} interpretation, but is very convenient as a rough measure of inaccuracy.  It is also equal to the sum of the RMS errors in the individual matrix elements of the gates.
\end{enumerate}

Table \ref{bestGatesetSpamVsTargetTable} shows a similar comparision using the information using the standard state infidelity and trace distance between the estimated and ideal state preparations and POVM effects.

It's also useful to know \emph{how} the real gates (or, more precisely, GST's estimates of the real gates) differ from the targets.  There are several ways we could represent this, but the most useful involves an \emph{error generator}.  These are given in Table \ref{bestGatesetErrGenTable}.  The table displays, in heat map form for compactness, a Lindbladian superoperator $\mathbb{L}$ for each gate.  It is defined by the equation $\hat{G} = G_{\mathrm{target}}e^{\mathbb{L}}$, where $\hat{G}$ is the estimate and $G_{\mathrm{target}}$ is the ideal gate.  This Lindbladian would be zero if the gates were perfect, and its overall magnitude is approximately equal to the diamond distance (or Jamiolkowski trace distance) between the target gate and the estimate.

\begin{table}[h]
\begin{center}
\putfield{bestGatesetVsTargetTable}{Best gate set overview table will be placed here}
\caption{\putfield{tt_bestGatesetVsTargetTable}{}\textbf{Comparison of GST estimated gates to target gates}.  This table presents, for each of the gates, three different measures of distance or discrepancy from the GST estimate to the ideal target operation.  See text for more detail.\label{bestGatesetVsTargetTable}}
\end{center}
\end{table}


\begin{table}[h]
\begin{center}
\putfield{bestGatesetSpamVsTargetTable}{Best gate set overview table will be placed here}
\caption{\putfield{tt_bestGatesetSpamVsTargetTable}{}\textbf{Comparison of GST estimated SPAM to target SPAM}.  This table presents, for each state preparation and POVM effect, two different measures of distance or discrepancy from the GST estimate to the ideal target operation.  See text for more detail.\label{bestGatesetSpamVsTargetTable}}
\end{center}
\end{table}


\begin{table}[h]
\begin{center}
\putfield{bestGatesetErrGenBoxTable}{Best gate set error generator table will be placed here}
\caption{\putfield{tt_bestGatesetErrGenBoxTable}{}\textbf{The GST estimate of the logic gate operation generators}. A heat map of the ``Error Generator'' for each gate, which is the Lindbladian $\mathbb{L}$ that describes \emph{how} the gate is failing to match the target.  This error generator is defined by the equation $\hat{G} = G_{\mathrm{target}}e^{\mathbb{L}}$. When all elements of these matrices is zero, the estimated gates match the target gates (Table \ref{targetGatesTable}).  Note that the range of the color scale is variable, being determined by the data.  In the final column, these generators are projected onto each of the ``Pauli-product generators'' given by the action of commutation with each Pauli-product basis element.  Columns and rows correspond to Pauli operators on the first and second (if present) qubit.\label{bestGatesetErrGenTable} }
\end{center}
\end{table}


It's usually useful to understand \emph{how} gates fail.  The error generators in Table \ref{bestGatesetErrGenTable} provide one view on this, but they are not necessarily intuitive.   For example, you might want to know whether your gate suffers depolarizing, dephasing, or over-rotation errors.  In Tables \ref{bestGatesetEigenvalueTable}, the eigenvalues of the estimated gates are shown and plotted on the complex disc (which can be helpful for visualizing the different types of errors, e.g., over- or under-rotation vs.~depolarization).  Table \ref{bestGatesetRelativeEigenvalueTable} shows the same quantities but for the eigenvalues of $G_{\mathrm{target}}^{-1}\hat{G} = e^{\mathbb{L}}$, which would all be $1.0$ if the gate were ideal.  %Table \ref{bestGatesetPolarEigvalTable} plots these eigenvalues on the complex disc, which can be helpful for visualizing the different types of errors (e.g.~over- or under-rotation vs.~depolarization). 
%TODO: add hamiltonian generator projections figure?

\begin{table}[h]
\small
\begin{center}
\putfield{bestGatesetEvalTable}{Best gate set eigevalue-decomp table will be placed here}
\caption{\putfield{tt_bestGatesetEvalTable}{}\textbf{Eigenvalues of estimated gates}.  The spectrum (Eigenvalues column) of each estimated gate.  The second column displays these eigenvalues over the complex disc.  \textbf{Black} and \textcolor{cyan}{\textbf{cyan}} circles plot the spectrum of the \textbf{ideal target gate} and \textcolor{cyan}{\textbf{estimated gate}}, respectively.\label{bestGatesetEigenvalueTable}}
\end{center}
\end{table}

\begin{table}[h]
\small
\begin{center}
\putfield{bestGatesetRelEvalTable}{Best gate set relative-eigevalue-decomp table will be placed here}
\caption{\putfield{tt_bestGatesetEvalTable}{}\textbf{Relative Eigenvalues of estimated gates}.  For each gate, the spectrum of $G_{\mathrm{target}}^{-1}\hat{G}$, defined as its \emph{relative eigenvalues} (see text for details).  The second column displays these eigenvalues over the complex disc.  Small \textcolor{red}{\textbf{red}} circles plot the \textcolor{red}{\textbf{relative eigenvalues}}, and \textcolor{darkgreen}{\textbf{green}} circles plot the relative eigenvalues of the gate \textcolor{darkgreen}{\textbf{repeated 10 times}} (to increase visibility).\label{bestGatesetRelativeEigenvalueTable}}
\end{center}
\end{table}


%\begin{figure}
%\begin{center}
%\XXputfield{bestEstimatePolarEvalPlots}{Polar eigenvalue plots for gates placed here}
%\caption{\XXputfield{tt_bestEstimatePolarEvalPlots}\textbf{Polar plots of gate eigenvalues}.  The eigenvalues from Table \ref{bestGatesetEigenvalueTable} plotted over the complex disc.  Large \textbf{black} and \textcolor{cyan}{\textbf{cyan}} circles plot the spectrum of the \textbf{ideal target gate} and \textcolor{cyan}{\textbf{estimated gate}}, respectively.  Small \textcolor{red}{\textbf{red}} circles plot the \textcolor{red}{\textbf{``relative'' eigenvalues}} (see Table \ref{bestGatesetEigenvalueTable}), and \textcolor{darkgreen}{\textbf{green}} circles plot the relative eigenvalues of the gate \textcolor{darkgreen}{\textbf{repeated 10 times}} (to increase visibility).\label{bestGatesetPolarEigvalTable}}
%\end{center}
%\end{figure}


%REMOVE It might be useful to know the closest \emph{unitary} operation to the estimated gate, and how close it is.  Usually, you were trying to implement a unitary.  If the closest unitary to $G$ was indeed $G_{\mathrm{target}}$, then all errors are incoherent; if not, you might be able to tweak the gate parameters to get closer relatively easily.  Also, implementing a particular unitary may be less important than just achieving \emph{some} set of mutually independent unitaries.  In these and other cases, the distance from an estimated gate to its closest unitary approximation is of interest.

%REMOVE Table \ref{bestGatesetClosestUnitaryTable} lists, for each estimated gate, the properties of its closest unitary approximation.  The table defines the closest unitary, in terms of an axis and angle (in $\mathcal{B}(\mathcal{H})$) of rotation.  It also presents the process fidelity and Jamiolkowski trace distance between the estimated gate and its closest unitary approximation.  A sanity check is computed by comparing the fidelity of the obtained closest unitary with a theoretical upper bound (if a value greater than one appears in this column then the other values in that row may be inaccurate).  If these numbers are similar to those in Table \ref{bestGatesetVsTargetTable}, then the gates are as close to the targets as they are to \emph{any} unitary.

%\begin{table}[h]
%\begin{center}
%XXXputfield{bestGatesetClosestUnitaryTable}{Best gate set overview table will be placed here}
%\caption{XXXputfield{tt_bestGatesetClosestUnitaryTable}{}Information pertaining to the closest unitary gate to each of the estimated gates.\label{bestGatesetClosestUnitaryTable}}
%\end{center}
%\end{table}


Finally, Table \ref{bestGatesetChoiTable} presents the spectrum of each estimated gate's \emph{Choi matrix}.  The Choi matrix (sometimes ambiguously referred to as the ``process matrix'') is an alternative way to describe a process.  We usually prefer the ``superoperator representation'', which has the very useful property that the process matrix corresponding to applying $G_a$ and then $G_b$ is simply $G_bG_a$.  This is completely false for the Choi representation.  Nonetheless, the Choi representation is often useful, so we present it here -- but without a detailed discussion of its properties (see, e.g. the textbook by Nielsen and Chuang).

The Choi matrix $\chi(G)$ for a gate $G$ can be simply understood in either of two ways.  First, it is equivalent (up to choice of basis) to the \emph{Jamiolkowski state} defined by applying $G$ to one half of a maximally entangled bipartite state.  Second, it is the general (non-diagonal) form of the well-known Kraus representation, $G[\rho] = \sum_i{K_i\rho K_i^\dagger}$.  The Choi matrix behaves in many ways like a quantum state, and appears naturally in expressions for the process fidelity and Jamiolkowsi trace distance just as density matrices would enter these expressions when computing differences between states.  

Additionally, the condition of \emph{complete positivity} or CP (which all real quantum processes must satisfy) is simply the positivity of the Choi matrix.  Thus, negative eigenvalues in Table \ref{bestGatesetChoiTable} indicate that the estimate violates complete positivity.  If they are very small, they may simply indicate statistical fluctuations (unitary gates have $\chi$ matrices with zero eigenvalues, so any small fluctuation is likely to violate CP).  If they are large, they serve as a warning that (1) the model of CPTP maps is probably violated (usually because of non-Markovian behavior), and (2) this estimate may produce negative or greater-than-unity probabilities.  GST does \emph{not} generally impose complete positivity, precisely because violation of CP is a warning flag for non-Markovian behavior (which is very common in experimental qubits).

\begin{table}[h]
\begin{center}
\putfield{bestGatesetChoiEvalTable}{Best gate set's choi matrix table will be placed here}
\caption{\putfield{tt_bestGatesetChoiEvalTable}{}\textbf{Choi matrix spectrum of the GST estimated gate set}.  The eigenvalues of the Choi representation of each estimated gate.  In the third column, magnitudes of \emph{negative} values are plotted using \textcolor{red}{red} bars. Unitary gates have a spectrum $(1,0,0\ldots)$, just like pure quantum states.  Negative eigenvalues are non-physical, and may represent either statistical fluctuations or violations of the CPTP model used by GST.\label{bestGatesetChoiTable}}
\end{center}
\end{table}



\section{Goodness-of-model Analysis\label{secGoodness}}

The previous section presented the estimated gate set, and compared it to the target gate set.  This section is concerned with a mostly orthogonal analysis which seeks to explain how much the estimated gate set can be trusted -- i.e., how well it fits the data.

To understand the goal of this section, consider the simple problem of fitting a line to a set of points.  For any set of points, there is \emph{always} a best-fit line -- but this doesn't mean that the best-fit line is a \emph{good} fit!  The data points may trace out a parabola, a square, or even something more complicated.  It is essential to understand not just what the best-fit line was (and perhaps how close it was to some desired line), but also \textbf{how well that linear model was able to fit all the data}.  Of course, we do not expect it to fit every data point perfectly.  The critical question is ``Did the linear model fit \emph{as well as we would expect it to} if the data really were generated by a linear process?''

In this analogy, GST's estimated gate set is like the best-fit line, and the target gate set like the desired line.  This section asks the question ``How well was GST able to fit all of the data -- and did it fit well enough to suggest that its model is valid?''. A central tool used to do this is the \emph{likelihood function}, which we denote $\mathcal{L}$, which formally is the probability of the observed data given a set of model parameters.  The basic idea is that we maximize the likelihood function to obtain the best set of model parameters (i.e.~gate set), and by looking at the value of this maximum we can determine the model's goodness-of-fit.  We will actually deal primarily with the logarithm of the likelihood function, $\log(\mathcal{L})$, which is simliarly maximized.

\subsection{Aggregated $\log(\mathcal{L})$}

The log-likelihood for an $n$-outcome system with predicted probabilities $p_i$ and observed frequencies $f_i$ ($i=1\ldots n$) is given by:
\begin{equation}
\log(\mathcal{L}) = \sum_i N f_i \log(p_i).
\end{equation}
where $N$ is the total number of counts. In \emph{this} analysis, $\log(\mathcal{L})$ is used to compare the set of probabilities predicted by a gate set ($p_s$) and the frequencies obtained from a dataset ($f_s$).  Each experiment (or gate sequence) $s$ is associated to two probabilities:  ``plus'' has probability $p_s$ and ``minus'' has probability $1-p_s$.  The $\log(\mathcal{L})$ contribution of a single gate string $s$ is
\begin{equation}
\log(\mathcal{L})_s = N f_s \log(p_s) + N (1-f_s) \log(1-p_s),\label{eqGateStringLogL}
\end{equation}
where $N$ is the number of times the experiment $s$ was performed, $p_s$ is the probability of a ``plus'' outcome as predicted by the gate set, and $f_s$ is the observed frequency of ``plus''.  The total log-likelihood for an entire dataset is just the sum 
\begin{equation}
\log(\mathcal{L}) = \sum_{s\in\mathcal{S}}{ \log(\mathcal{L})_s}.\label{eqDatasetLogL}
\end{equation}
A theoretical upper bound on the log-likelihood can be found by replacing $p_s$ with $f_s$ in Eq.~\ref{eqGateStringLogL} and evaluating Eq.~\ref{eqDatasetLogL}.  We will refer to this quantity as $\log(\mathcal{L})_{ub}$.

Statistical theory has quite a lot to say about the likelihood function (see any of the major textbooks).  Using some of these results, we can predict that if there are $N_p$ free parameters in the gate set that GST is fitting, and GST fits a dataset containing $N_s > N_p$ distinct experiments (gate sequences), then \emph{if the gate set model is correct}, then two times the difference between $\log(\mathcal{L})_{ub}$ and the maximum $\log(\mathcal{L})$ obtained is a random variable with a $\chi^2_{k}$ distribution, where 
$$k \equiv N_s - N_p.$$
Its expected value is $\expec{\chi^2}=k$, and its standard deviation is $\sqrt{2k}$.  Thus, if the fit is ``good'', then twice $\Delta\log(\mathcal{L}) \equiv \log(\mathcal{L})_{ub} - \max(\log(\mathcal{L}))$ should lie roughly within the interval $[k-\sqrt{2k},k+\sqrt{2k}]$.
Thus, by comparing the difference $2\Delta\log(\mathcal{L}) - k$ to $\sqrt{2k}$, one can determine how well the GST estimate was able to fit the data in dataset ``\putfield{datasetLabel}{DATASET LABEL HERE}''.

\iftoggle{LsAndGermsSet}{
The MLEGST algorithm used to generate this estimate is iterative.  It starts by fitting only data from the shortest gate sequences (which are easy to fit \emph{and} insensitive to most non-Markovian noise), then successively adds longer and longer sequences (with base sequence length $L\leq 1,2,4,8,\ldots$) to the mix.  Since we get an estimate at each intermediate $L$, it is possible to quantify not just the goodness of the \emph{best} fit (presented in the previous section), but how the goodness-of-fit behaves as longer and longer sequences are added in.

This data is presented in Table \ref{progressTable}.  What you should be looking for here is whether -- at each value of $L$ -- the $2\Delta\log(\mathcal{L})$ quantity is roughly the same as $k$.  More precisely, is $|2\Delta\log(\mathcal{L})-k|$ less than or equal to $\sqrt{2k}$?  If not, then the model is not fitting as well as it should, which usually indicates non-Markovian noise (or, rarely, that the GST algorithm has simply failed to find a good fit even though one exists).  

As a rough rule of thumb, for GST experiments involving relatively long sequences (e.g. $L\geq100$):
%\begin{tabular}{rp{5.5in}}
%$\bigstar\bigstar\bigstar\bigstar\bigstar$ & ``Incredibly good'' experiments have $\chi^2 \approx k$, as predicted by theory (and seen in simulations). \\
%$\bigstar\bigstar\bigstar\bigstar$ & ``Great'' experiments have $\chi^2 \leq 2k$ or so. \\
%$\bigstar\bigstar\bigstar$ & ``Good'' experiments have $\chi^2 \leq 5k$ or so. \\
%$\bigstar\bigstar$ & ``Okay'' experiments have $\chi^2 \leq 10k$. \\
%$\bigstar$ & Experiments in which $\chi^2 > 10k$ have very significant non-Markovian noise, and the results in the previous section should be viewed very cautiously.
%\end{tabular}
\begin{itemize}
\item ``Incredibly good'' ($\bigstar\bigstar\bigstar\bigstar\bigstar$) experiments have $2\Delta\log(\mathcal{L}) \approx k$, as predicted by theory (and seen in simulations).
\item ``Great'' ($\bigstar\bigstar\bigstar\bigstar$) experiments have $2\Delta\log(\mathcal{L}) \leq 2k$ or so.
\item ``Good'' ($\bigstar\bigstar\bigstar$) experiments have $2\Delta\log(\mathcal{L}) \leq 5k$ or so.
\item ``Okay'' ($\bigstar\bigstar$) experiments have $2\Delta\log(\mathcal{L}) \leq 10k$.
\item Experiments in which $2\Delta\log(\mathcal{L}) > 10k$ ($\bigstar$) have very significant non-Markovian noise, and the results in the previous section should be viewed very cautiously.
\end{itemize}

\begin{table}[h]
\begin{center}
\putfield{progressTable}{logL progress table will be placed here}
\caption{\putfield{tt_progressTable}{}\textbf{Comparison between the computed and expected maximum $\log(\mathcal{L})$ for different values of $L$}.  $N_S$ and $N_p$ are the number of gate strings and parameters, respectively.  The quantity $2\Delta\log(\mathcal{L})$ measures the goodness of fit of the GST model (small is better) and is expected to lie within $[k-\sqrt{2k},k+\sqrt{2k}]$ where $k = N_s-N_p$. $p$ is the p-value derived from a $\chi^2_k$ distribution.(For example, if $p=0.05$, then the probability of observing a $\chi^{2}$ value as large as, or larger than, the one indicated in the table is 5\%, assuming the GST model is valid.) The rating from 1 to 5 stars gives a very crude indication of goodness of fit as explained in the text.\label{progressTable}}
\end{center}
\end{table}

}{
\textbf{NOTE: Not enough information was given at the time of this report generation to know provide a goodness-of-model analysis}
}

\FloatBarrier

\iftoggle{LsAndGermsSet}{
\subsection{Detailed likelihood analysis}

The aggregated $2\Delta\log(\mathcal{L})$ numbers presented in Table \ref{progressTable} tell you how well the GST estimate fits the \emph{entire} dataset.  If they are in line with theory ($2\Delta\log(\mathcal{L}) \approx k$), then there is little more to be said.  But if the best fit to the data is not good, we can debug it by identifying \emph{which} experiments are inconsistent with the fit.

%s \ref{bestEstimateSummedColorBoxPlot} and
%In Fig.~\ref{bestEstimateColorBoxPlot}, each
%Figure \ref{bestEstimateSummedColorBoxPlot} shows the same $2\Delta\log(\mathcal{L})$ contributions on a coarser level by summing together the values of all the boxes in a each block of Fig.~\ref{bestEstimateColorBoxPlot}.  Thus, in Fig.~\ref{bestEstimateSummedColorBoxPlot}, each box represents a single germ-power gate sequence.
Figure \ref{bestEstimateColorBoxPlot} displays the $2\Delta\log(\mathcal{L})$ contribution from individual gate sequence (Eq.~\ref{eqGateStringLogL}).  Each gate sequence corresponds to a single colored ``pixel'' in the plot.  Each block of pixels corresponds to a single base sequence (i.e., a germ power), and the individual pixels within a block correspond to the various fiducial sequence pairs between which that base sequence was sandwiched, as indicated in Figure \ref{colorBoxPlotKeyPlot}.  Base sequences are arranged in a grid; different rows correspond to different germs, while different columns correspond to different maximum lengths $L$.  Pixels are labeled with the $2\Delta\log(\mathcal{L})$ contribution for that sequence, and colored appropriately. 

Sequences whose observed frequencies are consistent with a Markovian gate set are shown in gray, with darker shades indicating greater inconsistency with the estimated gate set.  Data shown in red are \emph{not} consistent with a Markovian gate set.  It may appear contradictory to say that (a) gray is ``consistent" with Markovian, but (b) darker shades indicate ``greater inconsistency".  The resolution is that the $\chi^2$ values quantify inconsistency with the model, \emph{but} they themselves are also subject to random fluctuations.  Therefore, even if the data are perfectly consistent with the model, we expect to see (for example) a single $\chi^2_s \geq 10$ once per each 638 experiments.  Observing $\chi^2_s \geq 10$ for any given sequence does suggest that the data from
$s$ were relatively surprising, but we also expect to see one such fluctuation if there are more than about 600 experiments.  The gray/red threshold is chosen based on the total number of sequences so that \emph{if} the data are perfectly Markovian, then the probability of one or more experiments being colored red is only \putfield{linlg_pcntle}{X}\%.

Identifying patterns and trends within such ``pixel plots'' can aid in identifying specific sources and types of non-Markovian noise which may be to blame if the GST algorithms are unable to produce a ``good'' estimate.  For example, it is often the case that all the short sequences [$L = O(1)$] can be fit reasonably well, but the right-hand side of Figure \ref{bestEstimateColorBoxPlot} becomes a sea of red.  This indicates that non-Markovian behavior (potentially due to slow drift of gate set parameters) is becoming more significant for longer experiments.  In other cases, a single row may be particularly bad, indicating that a particular gate or germ is especially problematic (e.g., was not stabilized using dynamical decoupling techniques).  Be cautious in debugging, however -- sometimes bad $\log(\mathcal{L})$ values for a particular gate or germ can result \emph{not} from faults in that operation, but because another operation failed so badly that it distorted the entire fit (e.g., in trying to fit catastrophically non-Markovian data at Point A, GST ended up failing to fit perfectly good data at Point B).

%REMOVE Similar pixel plots for the intermediate estimates whose total $2\Delta\log(\mathcal{L})$ is listed in Table \ref{progressTable} \iftoggle{pixelplotsappendix}{can be found in Appendix \ref{appendix_logL_pixelplots}.}{are included when the ``pixel plots'' appendix is enabled.}

%\begin{figure}
%\begin{center}
%\XXputfield{bestEstimateSummedColorBoxPlot}{Box plot of best gate set logL}
%\caption{\XXputfield{tt_bestEstimateSummedColorBoxPlot}{}\textbf{$2\Delta\log(\mathcal{L})$ contributions for every gate sequence up to fiducials}.   Each box shows the aggregate (summed) $2\Delta\log(\mathcal{L})$ for the set of experiments whereby a single germ power sequence is sandwiched between every combination of fiducial strin pairs.  Box color indicates whether GST was able to fit the corresponding frequencies well.  Shades of white/gray are typical. Red squares represent statistically significant evidence for model violation (non-Markovianity), and should appear with probability at most \putfield{linlg_pcntle}{32}\% if the data really are Markovian.  See text for further details.\label{bestEstimateSummedColorBoxPlot}}
%\end{center}
%\end{figure}


\begin{figure}
\begin{center}
\putfield{colorBoxPlotKeyPlot}{Key for color box plot sub grids}
\caption{\putfield{tt_colorBoxPlotKeyPlot}{}\textbf{Sub-block key for subsequent plots.} Shows how elements of the sub-blocks in Figure \ref{bestEstimateColorBoxPlot} correspond to preparation and measurement fiducial sequences.  Note that the column indicates the fiducial adjacent to state preparation, while the row indicates the fiducial adjacent to measurement.\label{colorBoxPlotKeyPlot}}
\end{center}
\end{figure}


\begin{figure}
\begin{center}
\putfield{bestEstimateColorBoxPlotPages}{Box plot of best gate set logL}
\caption{\putfield{tt_bestEstimateColorBoxPlotPages}{}\textbf{$2\Delta\log(\mathcal{L})$ contributions for every individual experiment in the dataset}.   Each pixel represents a single experiment (gate sequence), and its color indicates whether GST was able to fit the corresponding frequency well.  Shades of white/gray are typical. Red squares represent statistically significant evidence for model violation (non-Markovianity), and should appear with probability at most \putfield{linlg_pcntle}{32}\% if the data really are Markovian. Blocks of pixels correspond to base sequences (arranged vertically by germ and horizontally by length); each pixel within a block corresponds to a specific choice of pre- and post-fiducial sequences as shown in Figure \ref{colorBoxPlotKeyPlot}.  See text for further details.\label{bestEstimateColorBoxPlot}}
\end{center}
\end{figure}

%Figure \ref{invertedBestEstimateColorBoxPlot} shows exactly the same $\log(\mathcal{L})$ analysis as Figure \ref{bestEstimateColorBoxPlot}, but arranged differently.  Here, blocks (not square) all correspond to a single fiducial pair (e.g., pre- and post-fiducial), and pixels within a block correspond to different base sequences.  This can be useful for diagnosing a single bad fiducial sequence.
%
%\begin{figure}
%\begin{center}
%XXXputfield{invertedBestEstimateColorBoxPlot}{Box plot of best gate set logL with inverted box order}
%\caption{XXXputfield{tt_invertedBestEstimateColorBoxPlot}{}\textbf{$2\Delta\log(\mathcal{L})$ contributions for each experiment, arranged differently.}  This figure shows the same data as Figure \ref{bestEstimateColorBoxPlot}, but arranged differently.  Each block now corresponds to a particular pair of fiducial sequences, while pixels within the block correspond to different base sequences sandwiched between those fiducials.\label{invertedBestEstimateColorBoxPlot}}
%\end{center}
%\end{figure}

}{}

%XXXputfield{appendices}{Appendices start here}

\end{document}