WAIC on lognormal Poisson models

[rburner]

Hi,

A quick question about calculating WAIC - the computeWAIC() function works fine on my probit models, but throws an NaN on my lognormal poisson models. Is this a bug, or does the WAIC function not work on these models?

Thanks again for developing the HMSC package and all your work improving it and answering questions.

Best,

Ryan

[ovaskain]

It should work for lognormal poisson as well. This can be because lognormal poisson is often quite difficult to fit, can lead to numerical instability. In the last update to CRAN, we changed the default prior for lognormal poisson, that hopefully solves some of the issues. Best, Otso

…

On 14.4.2020 19.32, Ryan Burner wrote: Hi, A quick question about calculating WAIC - the computeWAIC() function works fine on my probit models, but throws an NaN on my lognormal poisson models. Is this a bug, or does the WAIC function not work on these models? Thanks again for developing the HMSC package and all your work improving it and answering questions. Best, Ryan — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#44>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIYMZTREK6G5Z3S7ZSATC3RMSF2PANCNFSM4MH4ED6Q>.

[rburner]

@ovaskain Thanks Otso, I'll give some simple models a try with the new CRAN version and see what happens.
Best,
Ryan

[rburner]

I seem to still be getting the same problem...would it be possible for me to replace any infinities and -Inf's with some arbitrarily large numbers to make the calculations work? How bad of an idea is this?

Any tips on where in the fitted model object I can find those numbers? Thanks!

[ovaskain]

Hi Ryan, Sorry to hear that this was not fixed. Can you send me some test problem that runs preferably in few seconds and reproduces the error. That would give a starting point for exploring what is the issue - and how to solve it. Best, O2

…

On 20.4.2020 12.13, Ryan Burner wrote: I seem to still be getting the same problem...would it be possible for me to replace any infinities and -Inf's with some arbitrarily large numbers to make the calculations work? How bad of an idea is this? Any tips on where in the fitted model object I can find those numbers? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIYMZSQH77PAMQ65A2RW4TRNQG3PANCNFSM4MH4ED6Q>.

[rburner]

Hi Otso,

Thanks, that would be great! I'll work on getting a fast example to send.

Best,
Ryan

[rburner]

Hi Otso,

Any luck with that example file I sent? I just reran a model with more samples and am still getting NaN for the WAIC.

Thanks,

Ryan

[hmsc-r]

I studied your example. The main problem is that log-normal Poisson model is very difficult to fit (e.g. in terms of MCMC convergence) and model fit can be bad. With log-normal Poisson, it is very important to study model fit, by e.g. comparing the data to predicted values. If predicted values are 100 times greater than the data, then something is obviously not well (and this can easily happen). I added to computeWAIC na.rm = TRUE, so now it should give a number. However, that number will not be meaningful if the MCMC has not converged and/or the model fit is not good.

[rburner]

Otso,

Thanks so much for the help - yes, that's an important reminder to always check model fit. I will take a look and see what is going on.

Best,

Ryan

[rburner]

Now most of my models are giving me 'Inf' as a WAIC...I guess that is just another indication of poor fit? But in some cases couldn't it be just a small subset of species that were responsible for the problems?

[ovaskain]

I added the option "byColumn" to WAIC. If setting=TRUE, it computes WAIC separately for each species. Seemed to work well for some simple simulated data tests with normal and probit models that I did (=returned lowest WAIC for those species where the covariates had an effect). With lognormal poisson the use of WAIC (both for entire model as well as for each species) separately does not seem equally stable. Otso

…

On 27.4.2020 15.48, Ryan Burner wrote: Now most of my models are giving me 'Inf' as a WAIC...I guess that is just another indication of poor fit? But in some cases it couldn't it be just a small subset of species that were responsible for the problems? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIYMZSE7IAVTGYVNAF2JBLROV5JJANCNFSM4MH4ED6Q>.

[rburner]

Great, thanks so much for that! I'll use it with due caution.

So I guess the sum of those species-specific WAIC values would be the WAIC value that was previously reported by the function? What would be the most appropriate summary of those values then when comparing models? Maybe median, to reduce the effect of outlier species?

Incidentally, I also compared the absolute values of species' responses to an influential covariate in one of my models with their WAIC and also found a relationship, which is good:

[ovaskain]

So I guess the sum of those species-specific WAIC values would be the WAIC value that was previously reported by the function? Not exactly but probably close of it. Is because log-likelihoods, variances etc. are not exactly additive. What would be the most appropriate summary of those values then when comparing models? Maybe median, to reduce the effect of outlier species? I don't have opinion here. But I would compare these to cross-validation results, especially for lognormal poisson models, as with cross-validation you can also evaluate many aspects of model fit more critically than with WAIC that is just a number.

[rburner]

Ok, thank you!