-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fit class: what to do about changing the stats class? #2063
Comments
I think it's a design problem and should be fixed, if possible with low effort. |
The issue at hand is that if you *change* the stat method of the Fit structure then you will get a mis-match with the actual fitted statistic, which remains using the _iterfit.stat method (i.e. the original statistic). We only have two test cases that expose this difference, so take advantage of the new "write the fit results to StingIO" capability and show that the stat value has changed (i.e. all but the last row use the original statistic, in this case Chi2DataVar, so the stat value is ~ 8500, and only the last row uses the Cash or CStat value). This is intended as a regression test.
We can make sure that changing the Fit stat also changes the associated _iterfit.stat field, which seems like it should be the correct thing to do. There are only two tests where this is an issue, and it's unclear whether the changes are good, bad, or not significant, since a) the tests appear to be regression tests b) we don't know how the test data was created, so we don't know what the true values are Note that the parameter values do not appear to have changed "hugely", but I haven't looked into the results too deeply. The tests do show the change - in that the statistic value reported in the "fit(outfile=...)" option now remains consistent (i.e. matches the expected value, assuming our new interpretation of the stat field of the Fit object is correct [which it should be, I claim]).
We can just call the callback routne after updating the model values, rather than manually setting the thawed pars, calculating the stat, and writing out the values to a file. This is ony possible now that sherpa#2063 has been addressed.
I ended up with a "simple" fix - when you change the I made the changes to #2054 |
The issue at hand is that if you *change* the stat method of the Fit structure then you will get a mis-match with the actual fitted statistic, which remains using the _iterfit.stat method (i.e. the original statistic). We only have two test cases that expose this difference, so take advantage of the new "write the fit results to StingIO" capability and show that the stat value has changed (i.e. all but the last row use the original statistic, in this case Chi2DataVar, so the stat value is ~ 8500, and only the last row uses the Cash or CStat value). This is intended as a regression test.
We can make sure that changing the Fit stat also changes the associated _iterfit.stat field, which seems like it should be the correct thing to do. There are only two tests where this is an issue, and it's unclear whether the changes are good, bad, or not significant, since a) the tests appear to be regression tests b) we don't know how the test data was created, so we don't know what the true values are Note that the parameter values do not appear to have changed "hugely", but I haven't looked into the results too deeply. The tests do show the change - in that the statistic value reported in the "fit(outfile=...)" option now remains consistent (i.e. matches the expected value, assuming our new interpretation of the stat field of the Fit object is correct [which it should be, I claim]).
We can just call the callback routne after updating the model values, rather than manually setting the thawed pars, calculating the stat, and writing out the values to a file. This is ony possible now that sherpa#2063 has been addressed.
This has come out of the discussion at #2054 (comment)
It could be considered a bug, a design decision, a new enhancement, or ...
The issue is that when directly using the
Fit
class you can change the statistic in use, but an internally-controlled object doesn't know about this change:You don't need to have called the
fit
method to show this. The_iterfit
attribute is a "hidden" part ofFit
.Now, I can change the
stat
attributeHowever, note that
_iterfit
has not been updated. I believe that it should be updated too (ideally we'd be able to source the information from the same location but the current design makes that hard, as would - I think - making_iterfit.stat
a read-only attribute to make sure users can't sayf.stat = CStat(); f._iterstat.stat = Chi2DataVar();
).However, I am not 100% sure about this (I note that we only have two tests which would be affected by this behaviour).
I do not believe this is a problem for the UI layer since in the UI layer when we need a
Fit
object we create one on the fly, via the internal_get_fit
method, so we never get into this situation where the stat field can get changed.The text was updated successfully, but these errors were encountered: