-
-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cross-validation of CAR models cannot be done with stratified folds #637
Comments
The problem is that CAR structures cannot handle new locations (regions in your case) that were not present when fitting the model but are present when predicting new data. Stratified folds should work if you have enough observations per region so that each region is present in every fold. I will take a closer look at your example to see whats wrong. |
Thanks for getting back to me! I had previously constructed the holdout datasets so that each region is present in every fold, but a reviewer argued that this does not correctly address spatial dependence and gives artificially low RMSE. So I wanted to address the comment by holding out entire regions one by one. I guess what you are saying is that the CAR structure inherently cannot support that? |
Not in the current implementation. I don't know enough of CAR models to tell you if it is possible or not in general. So if you can point me to some ideas to make this possible I am happy to take a look and see if we can get make this possible eventually. |
Thanks so much! The paper cited by the reviewer was this one by Roberts et al.: https://onlinelibrary.wiley.com/doi/full/10.1111/ecog.02881 |
An additional resource which describes prediction in CAR models is a paper by Ver Hoef et al. that came out last year in Ecological Monographs: https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1283 |
Thanks! I will take a look, but I don't know when I will be able to implement any satisfying solution. |
Hi there Paul -- first off thanks so much for all your work! I appreciate the improvements to the cross-validation options but I came across an issue 😕
I am trying to implement "block" cross validation of a CAR model by sequentially holding out multiple regions for each fold of the CV. However this gives the error message: "Error: Cannot handle new locations in CAR models." I cannot determine what is going on in the traceback.
Here is a minimal example:
The text was updated successfully, but these errors were encountered: