-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Target & True Models Should Be Static #383
Comments
Hi @pcwysoc, thanks for reporting this. I think I've found why this is happening. Since you already have code on your side that produces this error (and I don't have a minimal (non)working example handy) can you try applying the following change and report back on whether it fixes this issue? On line 1852 of |
@coreyostrove that didn't seem to work. I've generated a working example: @kmrudin mentioned to me that there were/are reasons that target wasn't static, since in the creation of the edesign/protocol objects if not parameterizations for the fit(s) are specified then it defaults to fitting the target's parameterization. Additionally, there can be instances were you want to test another model from, say, another GST fit. In that instance N_p=0 wouldn't be appropriate. This is all largely verbatim. |
I'm not sure this is an actual bug... Maybe we should have a discussion, @kmrudin. There are definitely cases where you want to test models that are not static, and it seems troublesome to automatically cast those as static. I think setting the parameterization to static for the true model is the most pyGSTi-y way to do this currently, and there is likely a similar way to set the Target parameterization of StandardGST to generate a static model as well. However, I take your point @pcwysoc that this is probably something that you may want to do on a regular basis without having to cast the models manually. Maybe there can be a flag that is set that would do this casting for you? I'm not sure what the best solution here looks like. |
Chiming in here with a couple of thoughts to piggy back off of @sserita. But first:
StandardGST does not inherit any parameterization from the target model for fitting. It uses whichever parameterization have been specified as a mode string. In order to fit nonstandard parameterizations you need to use the GateSetTomography protocol, which does inherit the parameterization for its fit from the initial_model passed in. On the conceptual side for model testing, when would we want to use a static model for testing purposes? As in, what does that mean statistically? Even setting that aside, a lot of the things from statistics we often rely on for evaluating the quality of our fits likely break down already in the context of model testing. For example, @pcwysoc in your original post you mentioned getting larger than expected values for the model violation. But statements about model violation are (in pyGSTi) most often based upon application of Wilks’ theorem. But Wilks’ theorem is a statement about the distribution of log-likelihood ratio statistics (LLRS) for maximum likelihood estimators. When we’re doing model testing we are not necessarily at, or even close to, the MLE, so reported N_sigmas may or may not make sense taken literally (my noncommittalness is a representation of the fact that I am genuinely unsure). LLRSs in general still make sense and are useful, but the statements about the expected probability distribution they follow wouldn’t automatically hold. All of this is to say, we should think about this carefully. @kmrudin, @robinbk, @enielse: You were almost certainly either responsible for, or present during the crafting of the model testing framework in pyGSTi. Do you have any thoughts or historical insights? I have not done a deep dive into this part of the codebase so don’t have too much more to add before having done so. |
First time trying to chime in on one of these conversations, so crossing my
fingers and hoping I know what I'm doing here!
I've lost track of the context for this, so I may be off base. However,
I'm going to focus in on Corey's comment that "But Wilks’ theorem is a
statement about the distribution of log-likelihood ratio statistics (LLRS)
for maximum likelihood estimators." This undersells Wilks' Theorem a
little. By invoking a "maximal model" in the way that we usually do, we
can also use Wilks' Theorem to ask "Does Data A look like it could have
been generated by Model B?" with Model B not being an MLE at all. So if
Model B was just *defined* from theory grounds (the way that, e.g., the
target model usually is), we can use Wilks' Theorem to determined whether
Data A are consistent with the model, or (if not) how much the model is
violated.
Does this justify the use of a "static model" in this context? I freely
admit I don't know exactly all of what "static model" means in this context.
RBK
…On Mon, Feb 19, 2024 at 6:04 PM coreyostrove ***@***.***> wrote:
Chiming in here with a couple of thoughts to piggy back off of @sserita
<https://github.com/sserita>.
But first:
in the creation of the edesign/protocol objects if not parameterizations
for the fit(s) are specified then it defaults to fitting the target's
parameterization
StandardGST does not inherit any parameterization from the target model
for fitting. It uses whichever parameterization have been specified as a
mode string. In order to fit nonstandard parameterizations you need to use
the GateSetTomography protocol, which does inherit the parameterization for
its fit from the initial_model passed in.
On the conceptual side for model testing, when would we want to use a
static model for testing purposes? As in, what does that mean
statistically? Even setting that aside, a lot of the things from statistics
we often rely on for evaluating the quality of our fits likely break down
already in the context of model testing. For example, @pcwysoc
<https://github.com/pcwysoc> in your original post you mentioned getting
larger than expected values for the model violation. But statements about
model violation are (in pyGSTi) most often based upon application of Wilks’
theorem. But Wilks’ theorem is a statement about the distribution of
log-likelihood ratio statistics (LLRS) for maximum likelihood estimators.
When we’re doing model testing we are not necessarily at, or even close to,
the MLE, so reported N_sigmas may or may not make sense taken literally (my
noncommittalness is a representation of the fact that I am genuinely
unsure). LLRSs in general still make sense and are useful, but the
statements about the expected probability distribution they follow wouldn’t
automatically hold. All of this is to say, we should think about this
carefully.
@kmrudin <https://github.com/kmrudin>, @robinbk
<https://github.com/robinbk>, @enielse <https://github.com/enielse>: You
were almost certainly either responsible for, or present during the
crafting of the model testing framework in pyGSTi. Do you have any thoughts
or historical insights? I have not done a deep dive into this part of the
codebase so don’t have too much more to add before having done so.
—
Reply to this email directly, view it on GitHub
<#383 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACNUZ5JCFDN4X5U6RS3PIQDYUQAD5AVCNFSM6AAAAABAKFVE2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGM3TEMBVGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Describe the bug
The target model (and additionally any underlying true model used for simulated data generation) should be static and have N_p=0 parameters. This causes a discrepancy with the model violation being too high for the true model.
To Reproduce
Expected behavior
N_p should be 0 for target and true models on the model overview tab. This can be achieved by copying the models to test and setting them to static parameterizations. Similarly, this should be done for the target model. This problem is not blocking and can be worked around by creating a noisy_model with a static parameterization.
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: