-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
calibration to hospital occupancy and waste in calibration_ww.R not working as expected #94
Comments
Can you post a link to a reproducible example here please ... ? |
I have just pushed misc/experiments/calibration/calibration_ww.R, that if sourced should give a very good hospital occupancy fit. If you uncomment lines 213 and 214, you should be calibrating to both hospital occupancy and waste, which gives a very bad fit. |
Thank you @mayaearn this is very helpful. |
I do not have time to dig into this at the moment, but when i do have time this is what I will do.
|
Sounds great! Thank you! Just as something to look out for, I realized recently that the Champredon paper was not splitting hospital occupancy into the 4 compartments that macpan does, so I combined the compartments H, H2, ICUd, and ICUs for the calibration to hospital occupancy. This is a change from what I was doing previously and I wanted to mention it to avoid any confusion (also this could be a possible location of new technical issues). |
I have a couple of questions. @mayaearn: the suggested lines to uncomment (213-214) use a Poisson log-likelihood for wastewater (although there are other commented lines lying around that use log-Normal instead): was that intentional? Seems to me that could be problematic/add unrelated difficulties into the mix. @stevencarlislewalker: it seems a little odd to me that we create complex |
@bbolker Yes, this was intentional. The log-normal and poisson seemed to be giving very similar fits for just wastewater, so I assumed that using the poisson log-likelihood, which Steve had improved with various clamping fixes was the better option. I will look into whether switching back to log-normal improves anything right now (given I've changed a few parts of the calibration since making this switch), and get back to you! |
@bbolker, one may always use The idea of the other ones is for developers of higher level interfaces to be able to swap out optimizers without needing to change interface code. So for example, I have three goals:
Did you have a particular design in mind? I'm certainly open to adjusting the approach as long as we hit these two goals. I have a preference to avoid writing an optimization function that takes many arguments that interact, but then again I have a stronger preference to make anyone who is interested in the project happy. |
I have just pushed a new version of calibration_ww.R (and opt_parameters.csv) so that log-normal can be used for waste calibration. This does work somewhat better for waste calibration specifically, but the fit to both hospital occupancy and waste (by uncommenting lines 203 to 208) is still very bad. |
FWIW I took a brief crack at using One potential quick-and-dirty fix would be to set up a range of randomly generated starting values (all reasonable, but covering a range) and look at the variation among the results. |
@bbolker More support for your suspicion: Late last week, I tried to set up this calibration in MacPan1.5 to compare issues and though I wasn't facing exactly the same errors due to a few things I wasn't replicating perfectly, I was facing similar issues with calibration / volatile breakpoints. I can likely set up the fix you suggest. When you say starting values, do you mean state values or simulation starting dates (or something else)? |
@stevencarlislewalker , I should probably spawn another issue. What
|
@mayaearn : by starting values I meant the initial values of the parameters in the objective function. (As @stevencarlislewalker mentioned, sometimes starting state values are also parameters ...) |
@bbolker This took longer than expected, but I have just pushed a document: misc/experiments/calibration/issues/CalibrationIssues.Rmd that produces a bunch of calibrations with different starting values. It seems that maybe the answer to the unrealistic fitted values during calibration issue is simply to not fit mu (proportion of symptomatic infections that are mild). The only reason I was fitting mu was because I was doing this when calibrating with macpan1.5. I want to add a few more examples to this document but have pushed what I have so far. |
Thanks @mayaearn . I believe that the reason for fitting mu was because when fitting to both case and hospital data you need some way to explain the proportion of cases that go to hospital and mu is a reasonable parameter for doing that. In my non-ww work for pho I absolutely needed to fit mu or something like it to include all of these data streams in the fit. I can't dig in now but just wanted to share my hot take in case it helps. |
I have pushed a version with some amendments and comments. Some issues:
Another way to think about this: there is some unknown relationship between infections and wastewater concentrations (determined by viral shedding, dilution in the waste stream, etc etc) and some slightly less obscure relationship between infections and hospitalizations (determined by proportion symptomatic/proportion of symptomatic cases that are severe/hospital capacity etc.). If we only have WW and hospitalization, we can fit at most one of the parameters that govern these relationships. I haven't looked to see which parameters you are fitting/allowing to vary, but my guess is that you have more than one of the ones described above ... |
Thank you! I will take the documentation advice that you added in the comments, and I'm currently trying some increases in max iterations. That makes sense; I am currently fitting both the shedding rate and mu, so going forward, I will not fit mu when fitting to WW and hospitalization. |
Out of curiosity, was this scenario (fitting to H and WW while estimating both shedding rate and symptom severity prob) something that was working with Macpan 1.5? It would be somewhat surprising ... |
You are correct; this scenario doesn't seem to work in MacPan 1.5 either. When I was fitting mu last year using MacPan 1.5, I was also calibrating with report data. When I calibrate only with hospitalizations and waste, mu fits okay but the waste parameters fit very badly (which makes sense given your explanation above). |
Calibration to hospital occupancy and wastewater data seems to be volatile. Slight changes in breakpoints (slight time difference or adding / removing breakpoints) causes large changes in calibration or failure to fit at all.
The text was updated successfully, but these errors were encountered: