Target & True Models Should Be Static

[pcwysoc]

Describe the bug
The target model (and additionally any underlying true model used for simulated data generation) should be static and have N_p=0 parameters. This causes a discrepancy with the model violation being too high for the true model.

To Reproduce

1. Simulate data using a noisy model
2. Run gst_protocol = pygsti.protocols.StandardGST(['CPTPLND', 'true', 'full TP', 'Target'], models_to_test={'true': noisy_model})
3. Observe that N_p should be 0 for target and true models on the model overview tab

Expected behavior
N_p should be 0 for target and true models on the model overview tab. This can be achieved by copying the models to test and setting them to static parameterizations. Similarly, this should be done for the target model. This problem is not blocking and can be worked around by creating a noisy_model with a static parameterization.

Environment (please complete the following information):

• pyGSTi version qiskit-feature-mcm
• python version 3.10
• OS Ventura 13.6.2

[coreyostrove]

Hi @pcwysoc, thanks for reporting this. I think I've found why this is happening. Since you already have code on your side that produces this error (and I don't have a minimal (non)working example handy) can you try applying the following change and report back on whether it fixes this issue?

On line 1852 of pygsti/protocols/gst.py change initial_model = target_model to initial_model = target_model.copy().

[pcwysoc]

@coreyostrove that didn't seem to work. I've generated a working example:
NumGaugeParamsBug.ipynb.zip.

@kmrudin mentioned to me that there were/are reasons that target wasn't static, since in the creation of the edesign/protocol objects if not parameterizations for the fit(s) are specified then it defaults to fitting the target's parameterization. Additionally, there can be instances were you want to test another model from, say, another GST fit. In that instance N_p=0 wouldn't be appropriate. This is all largely verbatim.

[sserita]

I'm not sure this is an actual bug... Maybe we should have a discussion, @kmrudin. There are definitely cases where you want to test models that are not static, and it seems troublesome to automatically cast those as static.

I think setting the parameterization to static for the true model is the most pyGSTi-y way to do this currently, and there is likely a similar way to set the Target parameterization of StandardGST to generate a static model as well. However, I take your point @pcwysoc that this is probably something that you may want to do on a regular basis without having to cast the models manually.

Maybe there can be a flag that is set that would do this casting for you? I'm not sure what the best solution here looks like.
Maybe something like cast_models_to_static=None as a default and then you could provide cast_models_to_static=["Target", "true"]?

[coreyostrove]

Chiming in here with a couple of thoughts to piggy back off of @sserita.

But first:

in the creation of the edesign/protocol objects if not parameterizations for the fit(s) are specified then it defaults to fitting the target's parameterization

StandardGST does not inherit any parameterization from the target model for fitting. It uses whichever parameterization have been specified as a mode string. In order to fit nonstandard parameterizations you need to use the GateSetTomography protocol, which does inherit the parameterization for its fit from the initial_model passed in.

On the conceptual side for model testing, when would we want to use a static model for testing purposes? As in, what does that mean statistically? Even setting that aside, a lot of the things from statistics we often rely on for evaluating the quality of our fits likely break down already in the context of model testing. For example, @pcwysoc in your original post you mentioned getting larger than expected values for the model violation. But statements about model violation are (in pyGSTi) most often based upon application of Wilks’ theorem. But Wilks’ theorem is a statement about the distribution of log-likelihood ratio statistics (LLRS) for maximum likelihood estimators. When we’re doing model testing we are not necessarily at, or even close to, the MLE, so reported N_sigmas may or may not make sense taken literally (my noncommittalness is a representation of the fact that I am genuinely unsure). LLRSs in general still make sense and are useful, but the statements about the expected probability distribution they follow wouldn’t automatically hold. All of this is to say, we should think about this carefully.

@kmrudin, @robinbk, @enielse: You were almost certainly either responsible for, or present during the crafting of the model testing framework in pyGSTi. Do you have any thoughts or historical insights? I have not done a deep dive into this part of the codebase so don’t have too much more to add before having done so.

[robinbk]

First time trying to chime in on one of these conversations, so crossing my fingers and hoping I know what I'm doing here! I've lost track of the context for this, so I may be off base. However, I'm going to focus in on Corey's comment that "But Wilks’ theorem is a statement about the distribution of log-likelihood ratio statistics (LLRS) for maximum likelihood estimators." This undersells Wilks' Theorem a little. By invoking a "maximal model" in the way that we usually do, we can also use Wilks' Theorem to ask "Does Data A look like it could have been generated by Model B?" with Model B not being an MLE at all. So if Model B was just *defined* from theory grounds (the way that, e.g., the target model usually is), we can use Wilks' Theorem to determined whether Data A are consistent with the model, or (if not) how much the model is violated. Does this justify the use of a "static model" in this context? I freely admit I don't know exactly all of what "static model" means in this context. RBK

…

On Mon, Feb 19, 2024 at 6:04 PM coreyostrove ***@***.***> wrote: Chiming in here with a couple of thoughts to piggy back off of @sserita <https://github.com/sserita>. But first: in the creation of the edesign/protocol objects if not parameterizations for the fit(s) are specified then it defaults to fitting the target's parameterization StandardGST does not inherit any parameterization from the target model for fitting. It uses whichever parameterization have been specified as a mode string. In order to fit nonstandard parameterizations you need to use the GateSetTomography protocol, which does inherit the parameterization for its fit from the initial_model passed in. On the conceptual side for model testing, when would we want to use a static model for testing purposes? As in, what does that mean statistically? Even setting that aside, a lot of the things from statistics we often rely on for evaluating the quality of our fits likely break down already in the context of model testing. For example, @pcwysoc <https://github.com/pcwysoc> in your original post you mentioned getting larger than expected values for the model violation. But statements about model violation are (in pyGSTi) most often based upon application of Wilks’ theorem. But Wilks’ theorem is a statement about the distribution of log-likelihood ratio statistics (LLRS) for maximum likelihood estimators. When we’re doing model testing we are not necessarily at, or even close to, the MLE, so reported N_sigmas may or may not make sense taken literally (my noncommittalness is a representation of the fact that I am genuinely unsure). LLRSs in general still make sense and are useful, but the statements about the expected probability distribution they follow wouldn’t automatically hold. All of this is to say, we should think about this carefully. @kmrudin <https://github.com/kmrudin>, @robinbk <https://github.com/robinbk>, @enielse <https://github.com/enielse>: You were almost certainly either responsible for, or present during the crafting of the model testing framework in pyGSTi. Do you have any thoughts or historical insights? I have not done a deep dive into this part of the codebase so don’t have too much more to add before having done so. — Reply to this email directly, view it on GitHub <#383 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNUZ5JCFDN4X5U6RS3PIQDYUQAD5AVCNFSM6AAAAABAKFVE2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGM3TEMBVGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>