Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error with lme4 models with offset and re.form = NULL #335

Closed
florianhartig opened this issue May 23, 2022 · 6 comments
Closed

Error with lme4 models with offset and re.form = NULL #335

florianhartig opened this issue May 23, 2022 · 6 comments
Labels
Milestone

Comments

@florianhartig
Copy link
Owner

Reported by a user. Reproducible example:

library(DHARMa)
dat <- createData()

fit <- glmer(observedResponse ~ (1|group) + offset(Environment1), data = dat, family = "poisson")
summary(fit)
simulateResiduals(fit, plot = T, re.form = NULL)
@florianhartig florianhartig added this to the 0.4.6 milestone May 23, 2022
@florianhartig
Copy link
Owner Author

OK, I assume that this is an lme4 problem, because if I check lme4 S3 functions for the model object

  • simulate(fit, re.form = NULL) --> doesn't work
  • predict(fit, re.form = NULL) --> works

I don't see why simulate shouldn't work if predict works. I will probably have to send this to the lme4 crew. @bbolker - could you maybe have a look?

For the user - as a quick fix, you could use predict and then simulate by hand using the appropriate distribution.

@bbolker
Copy link
Contributor

bbolker commented May 23, 2022

As the error message shows, this is a problem with trying to evaluate the offset term in an environment where it can't be found (the bane of my lme4-developing existence):

simulate(fit, re.form = NULL)

Error in offset(Environment1) : object 'Environment1' not found

PS it looks like offset is getting included in the random effects term when simulating, for some reason?? That might not be too hard to fix ...

@florianhartig
Copy link
Owner Author

Regarding the environment: OK, but I assume there is no fundamental reason for this to occur, so it's a programming bug on lme4 side, right? Sorry for adding to your pile / plight ;)

About the offset included in the REs - my understanding / expectation would have been that regardless of the re.form setting, the offset always needs to be included, because it's part of the model, so re.form only affects whether the REs should be included, but offset + fixed effects will always be included?

@bbolker
Copy link
Contributor

bbolker commented May 23, 2022

I think lme4/lme4@8c8ab5c should fix this ... although (a) I'm not sure it's 100% bulletproof (although I couldn't break it as easily as I thought I'd be able to) and (b) can't give a guarantee at the moment about when this will make it to CRAN (although I have my arm twisted; last CRAN submission was 7 May 2022, so should probably wait a couple of weeks at least)

@florianhartig
Copy link
Owner Author

Great, thanks! I'll leave this open for a moment to check with the next CRAN version!

@florianhartig
Copy link
Owner Author

This was fixed in glmmTMB 1.1.4 -> closed!

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Jun 14, 2023
CHANGES IN VERSION 1.1-33:

  BUG FIXES:

         * a boundary check could fail occasionally when large data
           produced an NA value in a computed gradient; now warns
           instead (GH #719, Mathias Ambuehl)

         * 'allFit' now works better when 'optimx' and 'dfoptim'
           packages are not installed (GH #724)

         * 'refit' reset internal degrees of freedom component
           incorrectly for REML fits (resulted in incorrect reported
           REML criteria, but otherwise harmless: side effect of GH
           #678)

  NEW FEATURES:

         * 'dotplot' and 'qqmath' methods gain a 'level' argument to
           set the width of confidence intervals

         * 'dotplot' method is now more flexible, using ".v" options
           ('lty.v', 'col.line.v', 'lwd.v') to set appearance of
           vertical lines (Iago Gin<c3><a9> V<c3><a1>zquez)

         * 'refit' gains a 'newweights' argument (GH #678)

CHANGES IN VERSION 1.1-32 (2023-03-14):

  USER-VISIBLE CHANGES:

         * 'formatVC()' gets a new optional argument 'corr' indicating
           if correlations or covariances should be used for vector
           random effects; this corresponds to 'print(<merMod>,
           ranef.corr = ...)' .  By default, it is FALSE for 'comp =
           "Variance"', fixing (GH #707).

         * 'qqmath.merMod' adds a (useless) 'data' argument for S3
           compatibility. Going forward, the 'id' and 'idLabels'
           arguments should always be specified by name.  We have added
           code to try to detect/warn when this is not done.

  BUG FIXES:

         * 'nobars' now retains the environment of its formula argument
           (GH #713, Mikael Jagan)

CHANGES IN VERSION 1.1-31 (2022-11-01):

  BUG FIXES:

         * 'confint(fm, <single string>)' now works (after years of
           being broken) again.

         * simulating from binomial model with a factor response, when
           the simulated response contains only a single factor level,
           now works (Daniel Kennedy)

CHANGES IN VERSION 1.1-30 (2022-07-08):

  USER-VISIBLE CHANGES:

         * 'nl' (term names) component added to output list of
           'mkReTrms (GH #679)'

         * eliminate partial-matching of 'eta' (for 'etastart') (GH
           #686: not actually "user-visible" unless
           'getOption("warnPartialMatchDollar")' is 'TRUE')

         * 'summary' method doesn't break for GLMMs other than
           binomial/Poisson when 'merDeriv''s 'vcov.glmerMod' method is
           attached (GH #688)

  BUG FIXES:

         * better handling of 'simulate(., re.form = NULL)' when model
           frame contains derived components (e.g. 'offset()',
           'log(x)')
           (<florianhartig/DHARMa#335>)

         * 'bootMer' works with 'glmmTMB' again (broken in 1.1-29)

         * 'maxfun' argument to 'allFit' controls max function
           evaluations for every optimizer type (GH#685)

CHANGES IN VERSION 1.1-29 (2022-04-07):

  USER-VISIBLE CHANGES:

         * prediction with new levels (when not allowed) returns a more
           informative error message (displays a list of unobserved
           levels)

  BUG FIXES:

         * 'glmer.nb' now works when 'lme4' is not loaded (GH #658,
           @brgew)

         * tests for singularity ('check.conv.singular') now run
           independently of derivative computation (e.g., when
           'calc.derivs=FALSE') (GH #660, @palday)

         * 'influence.merMod' now works when data were originally
           specified as a tibble

         * fixed bug in 'cooks.distance' method for 'influence.merMod'
           (i.e., objects created via 'influence(fitted_model)') (John
           Fox) (GH #672)

         * 'predict' works for formulas containing . when 'newdata' is
           specified (GH #653)

         * 'bootMer' now correctly inherits control settings from
           original fit

CHANGES IN VERSION 1.1-28 (2022-02-04):

  USER-VISIBLE CHANGES:

         * construction of interacting factors (e.g. when 'f1:f2' or
           'f1/f2' occur in random effects terms) is now more efficient
           for partially crossed designs (doesn't try to create all
           combinations of 'f1' and 'f2') (GH #635 and #636)

         * 'mkNewReTrms' is exported

         * singular-fit message now refers to 'help("isSingular")'
           rather than '?isSingular'

  TESTS:

         * fix 'all.equal(p1,p2,p3)' and similar 'expect_equal()'
           thinkos

         * fix some tests only run when 'lme4:::testLevel() > 1'; adapt
           tests for upcoming Matrix 1.4-1 which has
           'names(diag(<sparse>))'

  BUG FIXES:

         * 'reOnly' preserves environment (GH #654, Mikael Jagan)

         * backward-compatibility hooks changed to evaluate at run-time
           (i.e., in '.onLoad()') rather than at build time (GH #649)

         * 'lmList' no longer warns when 'data' is a tibble (GH #645)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants