Running 2nd map2stan() model with rstan 2.14.1 leads to segfault #65

kmiddleton · 2017-01-05T22:31:29Z

After the rstan 2.14.1 (it might have been 2.14, but at least by 2.14.1), running a second map2stan() model in the same R session causes a segfault in my setup. Example:

library(rethinking)

data(chimpanzees)

# don't want any variables with NAs
d <- list( 
  pulled_left = chimpanzees$pulled_left ,
  prosoc_left = chimpanzees$prosoc_left ,
  condition = chimpanzees$condition ,
  actor = as.integer( chimpanzees$actor ) ,
  blockid = as.integer( chimpanzees$block )
)

# RStan fit
m2 <- map2stan(
  alist(
    pulled_left ~ dbinom(1,theta),
    logit(theta) <- a + bp*prosoc_left + bpc*condition*prosoc_left ,
    a ~ dnorm(0,10),
    bp ~ dnorm(0,10),
    bpc ~ dnorm(0,10)
  ) ,
  data=d, chains=1, cores=1 )

m2_2 <- map2stan(
  alist(
    pulled_left ~ dbinom(1,theta),
    logit(theta) <- a + bp*prosoc_left + bpc*condition*prosoc_left ,
    a ~ dnorm(0,10),
    bp ~ dnorm(0,10),
    bpc ~ dnorm(0,10)
  ) ,
  data=d, chains=1, cores=1 )

Gives:

 *** caught segfault ***
address 0x1c, cause 'memory not mapped'

Traceback:
 1: FUN(X[[i]], ...)
 2: lapply(name, function(id) {    v <- .Internal(getSymbolInfo(as.character(id), PACKAGE, as.logical(withRegistrationInfo)))  ...
 3: getNativeSymbolInfo(name, PACKAGE)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch(getNativeSymbolInfo(name, PACKAGE), error = function(e) e)
 8: Module(module, mustStart = TRUE)
 9: .getModulePointer(x)
10: <S4 object of class "Module">$stan_fit4model11743559f5a88_pulled_left___dbinom_1__theta_
11: eval(expr, envir, enclos)
12: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
13: object@mk_cppmodule(object)
14: .local(object, ...)
15: sampling(sm, data, pars, chains, iter, warmup, thin, seed, init,     check_data = TRUE, sample_file = sample_file, diagnostic_file = diagnostic_file,   ...
16: sampling(sm, data, pars, chains, iter, warmup, thin, seed, init,     check_data = TRUE, sample_file = sample_file, diagnostic_file = diagnostic_file,   ...
17: stan(model_code = model_code, model_name = modname, data = d,     init = initlist, iter = iter, warmup = warmup, chains = chains,   ...
18: doTryCatch(return(expr), name, parentenv, handler)
19: tryCatchOne(expr, names, parentenv, handlers[[1L]])
20: tryCatchList(expr, classes, parentenv, handlers)
21: tryCatch(expr, error = function(e) {    call <- conditionCall(e)  ...
22: try(stan(model_code = model_code, model_name = modname, data = d,     init = initlist, iter = iter, warmup = warmup, chains = chains,   ...
23: map2stan(alist(pulled_left ~ dbinom(1, theta), logit(theta) <- a +     bp * prosoc_left + bpc * condition * prosoc_left, a ~ dnorm(0,   ...

I am running R 3.3.2 Patched and reinstalled rethinking and all dependencies from source. I'm not sure if this is reproducible by anyone else.

My sessionInfo():

R version 3.3.2 Patched (2017-01-03 r71888)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rethinking_1.59    rstan_2.14.1       StanHeaders_2.14.0 ggplot2_2.2.1      fortunes_1.5-4     devtools_1.12.0   

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.8        MASS_7.3-45        munsell_0.4.3      colorspace_1.3-2   lattice_0.20-34    plyr_1.8.4        
 [7] tools_3.3.2        grid_3.3.2         gtable_0.2.0       loo_1.0.0          coda_0.19-1        withr_1.0.2       
[13] matrixStats_0.51.0 lazyeval_0.2.0     assertthat_0.1     digest_0.6.11      tibble_1.2         gridExtra_2.2.1   
[19] codetools_0.2-15   memoise_1.0.0      inline_0.3.14      scales_0.4.1       stats4_3.3.2       mvtnorm_1.0-5

The text was updated successfully, but these errors were encountered:

rmcelreath · 2017-01-06T08:28:43Z

Thanks for the report. This happens only when the model definitions are the same? If so, maybe related to how Stan reuses the compiled model binary.

rmcelreath · 2017-01-06T11:43:54Z

A little experimentation later, still not sure what is causing the problem. But using the previous map2stan result seems to bypass it:

m2 <- map2stan(
  alist(
    pulled_left ~ dbinom(1,theta),
    logit(theta) <- a + bp*prosoc_left + bpc*condition*prosoc_left ,
    a ~ dnorm(0,10),
    bp ~ dnorm(0,10),
    bpc ~ dnorm(0,10)
  ) ,
  data=d, chains=1, cores=1 )

m2_2 <- map2stan( m2 ,data=d, chains=1, cores=1 )

Hopefully that's a useful workaround for others, until I can figure out the core issue.

kmiddleton · 2017-01-06T14:53:14Z

I think you are correct that stan reusing the compiled model leads to the failure. If you delete the relevant files from the tmp directory, the same model will run twice.

library(rethinking)

data(chimpanzees)

# don't want any variables with NAs
d <- list( 
  pulled_left = chimpanzees$pulled_left ,
  prosoc_left = chimpanzees$prosoc_left ,
  condition = chimpanzees$condition ,
  actor = as.integer( chimpanzees$actor ) ,
  blockid = as.integer( chimpanzees$block )
)

# RStan fit
m2 <- map2stan(
  alist(
    pulled_left ~ dbinom(1,theta),
    logit(theta) <- a + bp*prosoc_left + bpc*condition*prosoc_left ,
    a ~ dnorm(0,10),
    bp ~ dnorm(0,10),
    bpc ~ dnorm(0,10)
  ) ,
  data=d, chains=1, cores=1 )

# Clean up tmpdir ########
tmp_dir <- tempdir()
model_code <- m2@stanfit@stanmodel@model_code
model_name2 <- attr(model_code, "model_name2")
unlink(paste0(tmp_dir, "/", model_name2, c(".rds", ".stan")))
# Seems to work ok without removing the dso, but just in case
dso_name <- m2@stanfit@stanmodel@dso@dso_filename
unlink(paste0(tmp_dir, "/", model_name2, c(".cpp", ".o", ".so")))

m2_2 <- map2stan(
  alist(
    pulled_left ~ dbinom(1,theta),
    logit(theta) <- a + bp*prosoc_left + bpc*condition*prosoc_left ,
    a ~ dnorm(0,10),
    bp ~ dnorm(0,10),
    bpc ~ dnorm(0,10)
  ) ,
  data=d, chains=1, cores=1 )

rmcelreath · 2017-01-13T10:40:32Z

I haven't had any luck figuring out the root cause. I thought perhaps it was a mismatch between the model name that map2stan uses and the model name internal to the stanfit object and module. But making those identical didn't change anything.

It's perhaps related to this general bug with stan, in which it keeps links to removed objects:
https://groups.google.com/forum/#!category-topic/stan-users/general/Bg8HhyB6qG4

rmcelreath · 2017-01-13T13:26:25Z

I can reliably prevent the segfault by simply making a copy of the stanfit object in the global environment:

library(rethinking)

flist <- alist(
  y ~ dnorm(mu,1), 
  mu <- a + b*x,
  a ~ dnorm(0,10),
  b ~ dnorm(0,1)
)

set.seed(1)
x <- rnorm(100)
y <- rnorm(100,-0.1*x,1)

d <- list(x=x,y=y)

m1 <- map2stan( flist , data=d , chains=1 )
fit <- m1@stanfit
m2 <- map2stan( flist , data=d , chains=1 )

As soon as the fit copy is removed, it'll segfault again:

rm(fit)
m2 <- map2stan( flist , data=d , chains=1 )

No idea what this means.

kmiddleton · 2017-01-13T13:54:35Z

Not surprising, because it's a Windows fix, but this commit stan-dev/rstan@6a61721 does not fix the crash on OS X.

Your code above leads to crash with recompiled rstan from the develop tree:

devtools::install_github("stan-dev/rstan", ref = "develop", subdir = "rstan/rstan", dependencies = TRUE)

j-d-miller · 2017-01-21T02:21:58Z

I've been working through the excellent Rethinking book and have encountered the same problem.

Everything worked well until I got to the map2stan examples in Chapter 10 and then R started to crash whenever I run a given map2stan more than once. (When I call map2stan with a 'map' model as the input R crashes the first time I run the code.)

(Also, as an aside, the function logistic doesn't seem to exist in the rethinking package. Or at least I can't find it.)

rmcelreath · 2017-02-21T14:48:18Z

I just pushed to the Experimental branch a solution to this issue. It's not a long-term solution, because it prevents Stan from reusing the compiled model. It forces recompilation. But it does stop the segfaults.

https://github.com/rmcelreath/rethinking/tree/Experimental

j-d-miller · 2017-02-21T19:32:18Z

Thanks for letting me know. I've really enjoyed the book. Jeff Miller

…

Sent from my iPhone

On Feb 21, 2017, at 7:48 AM, Richard McElreath ***@***.***> wrote: I just pushed to the Experimental branch a solution to this issue. It's not a long-term solution, because it prevents Stan from reusing the compiled model. It forces recompilation. But it does stop the segfaults. https://github.com/rmcelreath/rethinking/tree/Experimental — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

kmiddleton · 2017-02-22T01:10:14Z

Works well. I know it's not an ideal fix, but for me, recompiling models is less aggravating than starting over completely after a segfault.

rmcelreath · 2017-03-29T17:54:47Z

Experimental branch work-around is working for people:

library(devtools)
install_github( "rmcelreath/rethinking" , ref="Experimental" )

rmcelreath · 2017-04-12T09:15:23Z

Kinda fixed for now, through workaround. Closing the issue, but still worrying about the underlying cause.

kmiddleton changed the title ~~Running 2nd map2stan() model leads to segfault~~ Running 2nd map2stan() model with rstan 2.14.1 leads to segfault Jan 5, 2017

rmcelreath added the bug label Jan 6, 2017

rmcelreath closed this as completed Apr 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running 2nd map2stan() model with rstan 2.14.1 leads to segfault #65

Running 2nd map2stan() model with rstan 2.14.1 leads to segfault #65

kmiddleton commented Jan 5, 2017

rmcelreath commented Jan 6, 2017

rmcelreath commented Jan 6, 2017

kmiddleton commented Jan 6, 2017

rmcelreath commented Jan 13, 2017

rmcelreath commented Jan 13, 2017

kmiddleton commented Jan 13, 2017

j-d-miller commented Jan 21, 2017

rmcelreath commented Feb 21, 2017

j-d-miller commented Feb 21, 2017 via email

kmiddleton commented Feb 22, 2017

rmcelreath commented Mar 29, 2017

rmcelreath commented Apr 12, 2017

Running 2nd map2stan() model with rstan 2.14.1 leads to segfault #65

Running 2nd map2stan() model with rstan 2.14.1 leads to segfault #65

Comments

kmiddleton commented Jan 5, 2017

rmcelreath commented Jan 6, 2017

rmcelreath commented Jan 6, 2017

kmiddleton commented Jan 6, 2017

rmcelreath commented Jan 13, 2017

rmcelreath commented Jan 13, 2017

kmiddleton commented Jan 13, 2017

j-d-miller commented Jan 21, 2017

rmcelreath commented Feb 21, 2017

j-d-miller commented Feb 21, 2017 via email

kmiddleton commented Feb 22, 2017

rmcelreath commented Mar 29, 2017

rmcelreath commented Apr 12, 2017