glmer memory gets out of hand

[themeo]

I am comparing a lot of ME models (backward regression), and I use the 'doParallel' package to compute many reduced models for comparison at the same time. When I use lmer() everything is fine, but when I use glmer() with logit link functinon and bobyqa optimizer things quickly get out of hand: memory used by the master process running all the parallelization doubles with each iteration of the stepwise backward reduction process.
Aside from using glmer against lmer, the code is identical.
At each step I am calling gc() to facilitate garbage collection, but this doesn't help.

[bbolker]

Just so this doesn't go completely ignored ... I sympathize, but diagnosing and solving this could take a while. What exactly do you mean by "each iteration of the stepwise backward reduction process"? Do you mean that with lmer the total memory doesn't increase when you fit a new, reduced model (via lmer() or update())? Can you provide a little more detail about what you're doing?

[themeo]

Ok, I did some testing and simplified the problem.

Both for lmer() and glmer() I ran the same code snippet:

print (gc())
for (i in 1:10) {
  print (i)
#  lm = lmer(f, data=asum, REML=T)
#  lm = glmer(f, data=asum, family=binomial(link="logit"), control=glmerControl(optimizer="bobyqa"))
  print(gc())
}

In the first run the first line was uncommented, in the second run the second line. Of course, I reset R before each run. Formula was the same in both runs (except the dependent variable), and included 9 fixed effects, a few by-item and by-subject random effects + some random interactions. Data included 12031 data-points.

For lm(), memory usage did not increase with each run (or rather it stabilized after the 2nd run):

          used (Mb) gc trigger  (Mb) max used (Mb)
Ncells 1200734 64.2    2164898 115.7  1442291 77.1
Vcells 4320660 33.0    6405392  48.9  6405035 48.9
[1] 1
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1244975 66.5    2164898 115.7  1902825 101.7
Vcells 5606259 42.8    9399764  71.8  7762496  59.3
[1] 2
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1244994 66.5    2164898 115.7  1934326 103.4
Vcells 5606277 42.8   11359716  86.7 10525185  80.4

(...)

[1] 10
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1244994 66.5    2164898 115.7  1934326 103.4
Vcells 5606277 42.8   11359716  86.7 11359712  86.7

However, with glmer() the situation was different:

          used (Mb) gc trigger  (Mb) max used (Mb)
Ncells 1192130 63.7    2164898 115.7  1442291 77.1
Vcells 4318188 33.0    6405392  48.9  6402591 48.9
[1] 1
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1273026   68    2164898 115.7  2164898 115.7
Vcells 6550325   50   11359716  86.7 11359704  86.7
[1] 2
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1274637 68.1    2164898 115.7  2164898 115.7
Vcells 7847738 59.9   13711659 104.7 13709824 104.6
[1] 3
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 1276027 68.2    2164898 115.7  2164898 115.7
Vcells 9145128 69.8   13711659 104.7 13711644 104.7
[1] 4
           used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1277417 68.3    2164898 115.7  2164898 115.7
Vcells 10442518 79.7   16533990 126.2 16528545 126.2
[1] 5
           used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1278807 68.3    2164898 115.7  2164898 115.7
Vcells 11739908 89.6   19920788 152.0 19917826 152.0
[1] 6
           used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1280197 68.4    2164898 115.7  2164898 115.7
Vcells 13037298 99.5   19920788 152.0 19920785 152.0
[1] 7
           used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1281587  68.5    2164898 115.7  2164898 115.7
Vcells 14334688 109.4   23984945 183.0 23981242 183.0
[1] 8
           used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1282977  68.6    2164898 115.7  2164898 115.7
Vcells 15632078 119.3   23984945 183.0 23984933 183.0
[1] 9
           used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1284367  68.6    2164898 115.7  2164898 115.7
Vcells 16929468 129.2   28861934 220.2 23984933 183.0
[1] 10
           used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  1285433  68.7    2164898 115.7  2164898 115.7
Vcells 18226684 139.1   28861934 220.2 28858516 220.2

.. and in the course of the 10 runs Vcells memory usage rose from 49Mb to 220Mb. What is interesting, the memory increase was not deterministic -- sometimes it occurred, sometimes not. So it looks like there is a memory leak in glmer/bobyqa. Please let me know if you need any more information.

Aha, this is sessionInfo():

> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 8 (jessie)

locale:
 [1] LC_CTYPE=pl_PL.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=pl_PL.UTF-8        LC_COLLATE=pl_PL.UTF-8
 [5] LC_MONETARY=pl_PL.UTF-8    LC_MESSAGES=pl_PL.UTF-8
 [7] LC_PAPER=pl_PL.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=pl_PL.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

other attached packages:
 [1] scales_0.4.0      RePsychLing_0.0.4 remef_1.0.6.9000  doParallel_1.0.10
 [5] iterators_1.0.8   foreach_1.4.3     lme4_1.1-12       Matrix_1.2-7.1
 [9] ggplot2_2.1.0     stringr_1.1.0     data.table_1.9.7  knitr_1.14

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.7      magrittr_1.5     splines_3.3.1    MASS_7.3-45
 [5] munsell_0.4.3    colorspace_1.2-6 lattice_0.20-34  minqa_1.2.4
 [9] plyr_1.8.4       tools_3.3.1      grid_3.3.1       gtable_0.2.0
[13] nlme_3.1-128     nloptr_1.0.4     codetools_0.2-14 stringi_1.1.1
[17] chron_2.3-47

thanks,
Jakub

[bbolker]

OK, I will try with valgrind (sigh) and see what I can find out.