stan_model error #382
Comments
|
Can you force it to run with 1 core? Often parts of error messages are
swallowed when running in parallel.
…On Tue, Jan 24, 2017 at 12:59 PM, Rob Trangucci ***@***.***> wrote:
Summary:
calling stan_model on a .stan file results in an an error.
Of note: I can't reproduce this error on my laptop, this is only
reproducible on a RHEL server.
Description:
If I call stan_model on a stan file, I get:
starting worker pid=1470 on localhost:11117 at 12:46:59.214
Error in checkForRemoteErrors(lapply(cl, recvResult)) :
one node produced an error: unused argument (allow_undefined = allow_undefined)
Reproducible Steps:
Here is the stan file I'm trying to compile, called test.stan:
data {
}parameters {
real y;
}model {
y ~ normal(0, 1);
}
Calling:
library(rstan)
> model <- stan_model('~/test.stan')
leads to the error.
Current Output:
The only output is:
starting worker pid=1649 on localhost:11117 at 12:51:03.204
Error in checkForRemoteErrors(lapply(cl, recvResult)) :
one node produced an error: unused argument (allow_undefined = allow_undefined)
Expected Output:
A compiled model.
RStan Version:
2.14.1
rstan (Version 2.14.1, packaged: 2016-12-28 14:55:41 UTC, GitRev: 5fa1e80
<5fa1e80>
)
R Version:
[1] "R version 3.2.3 (2015-12-10)"
Session info:
R version 3.2.3 (2015-12-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.2 (Maipo)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rstan_2.14.1 StanHeaders_2.14.0-1 ggplot2_2.2.1
loaded via a namespace (and not attached):
[1] Rcpp_0.12.7 assertthat_0.1 grid_3.2.3 plyr_1.8.3 gtable_0.2.0 stats4_3.2.3 scales_0.4.1 lazyeval_0.2.0 tools_3.2.3
[10] munsell_0.4.3 parallel_3.2.3 inline_0.3.14 colorspace_1.2-6 gridExtra_2.2.1 tibble_1.2
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#382>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHb4Q-V40gR3Zo1JrKL3CuVJDGGuIaFmks5rVjv1gaJpZM4Lslel>
.
|
|
I've tried both: library(rstan)
options(mc.cores = 1)
model <- stan_model('test.stan')and library(rstan)
model <- stan('test.stan', cores=1)But I still get for each call. |
|
It works fine for me in the R GUI on Mac OS X with
RStan 2.14.1.
- Bob
… On Jan 24, 2017, at 1:53 PM, Rob Trangucci ***@***.***> wrote:
I've tried both:
library(rstan
)
options(
mc.cores = 1
)
model <- stan_model('test.stan')
and
library(rstan
)
model <- stan('test.stan', cores=1)
But I still get
starting worker pid=5012 on localhost:11240 at 13:51:03.638
Error in checkForRemoteErrors(lapply(cl, recvResult)) :
one node produced an error: unused argument (allow_undefined = allow_undefined)
for each call.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Rob, I think the rstan package could incompletely installed on RHEL. Can
you do remove.packages("rstan"), make double sure the whole directory is
gone on the disk, and reinstall rstan (making sure not to load any old
stanfit objects at startup that are often buried in a .RData file)
On Tue, Jan 24, 2017 at 4:20 PM, Bob Carpenter <notifications@github.com>
wrote:
… It works fine for me in the R GUI on Mac OS X with
RStan 2.14.1.
- Bob
> On Jan 24, 2017, at 1:53 PM, Rob Trangucci ***@***.***>
wrote:
>
> I've tried both:
>
> library(rstan
> )
> options(
> mc.cores = 1
> )
>
> model <- stan_model('test.stan')
> and
>
> library(rstan
> )
>
> model <- stan('test.stan', cores=1)
> But I still get
>
> starting worker pid=5012 on localhost:11240 at 13:51:03.638
> Error in checkForRemoteErrors(lapply(cl, recvResult)) :
> one node produced an error: unused argument (allow_undefined =
allow_undefined)
>
> for each call.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#382 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADOrqmwdbZEyJnmK6yiVPR2N-0bTIlJfks5rVmsXgaJpZM4Lslel>
.
|
|
Thanks, @bgoodri we'll try this and report back. I'm going to close the issue, it does seem to be localized to our machine |
|
Same thing on fresh install for stan and all dependencies. data {
}
parameters {
real y;
}
model {
y ~ normal(0, 1);
}
knitr::opts_chunk$set(comment=".")
.libPaths('lib')
library(rstan)
. Loading required package: ggplot2
. Loading required package: StanHeaders
. rstan (Version 2.14.1, packaged: 2016-12-28 14:55:41 UTC, GitRev: 5fa1e80eb817)
. For execution on a local, multicore CPU with excess RAM we recommend calling
. rstan_options(auto_write = TRUE)
. options(mc.cores = parallel::detectCores())
options(mc.cores = 1)
model <- try(stan_model('test.stan'))
model
. [1] "Error in checkForRemoteErrors(lapply(cl, recvResult)) : \n one node produced an error: unused argument (allow_undefined = allow_undefined)\n"
. attr(,"class")
. [1] "try-error"
. attr(,"condition")
. <simpleError in checkForRemoteErrors(lapply(cl, recvResult)): one node produced an error: unused argument (allow_undefined = allow_undefined)>
sessionInfo()
. R version 3.2.3 (2015-12-10)
. Platform: x86_64-pc-linux-gnu (64-bit)
. Running under: Ubuntu precise (12.04.2 LTS)
.
. locale:
. [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
. [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
. [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
. [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
. [9] LC_ADDRESS=C LC_TELEPHONE=C
. [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
.
. attached base packages:
. [1] stats graphics grDevices utils datasets methods base
.
. other attached packages:
. [1] rstan_2.14.1 StanHeaders_2.14.0-1 ggplot2_2.1.0
.
. loaded via a namespace (and not attached):
. [1] Rcpp_0.12.3 digest_0.6.8 grid_3.2.3 plyr_1.8.3
. [5] gtable_0.2.0 stats4_3.2.3 magrittr_1.5 evaluate_0.8
. [9] scales_0.4.0 stringi_1.0-1 rmarkdown_0.9.5 tools_3.2.3
. [13] stringr_1.0.0 munsell_0.4.3 |
|
I think this may be unique to rstan 2.14.1 running in RStudio on linux. The code that prompts the error appears to be the following if-block in stanc:
In particular it happens on the clusterEvalQ call. The code runs fine on the same Linux system if I run it in batch (R CMD BATCH). It also runs in the Rstudio environment when I delete the offending if-block. |
|
I have also encountered this error on Rstudio Server in Linux. |
|
This error persists in Stan 2.15 on RStudio Server Linux. Can be reproduced quickly with the latest docker-stan install (https://github.com/jrnold/docker-stan). > testfit <- stan(m, data=data_list, iter=1, chains = 1, init_r = 1)
starting worker pid=910 on localhost:11093 at 23:54:40.766
*** caught illegal operation ***
address 0x7f3e3271ab0c, cause 'illegal operand'
Traceback:
1: .Call("CPP_stanc280", model_code, model_cppname, allow_undefined)
2: rstan::stanc(file, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)
3: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) || is.pairlist(envir)) parent.frame() else baseenv()) .Internal(eval(expr, envir, enclos)))(quote(rstan::stanc(file, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)), env = quote(<environment>))
4: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) || is.pairlist(envir)) parent.frame() else baseenv()) .Internal(eval(expr, envir, enclos)))(quote(rstan::stanc(file, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)), env = quote(<environment>))
5: do.call(msg$data$fun, msg$data$args, quote = TRUE)
6: doTryCatch(return(expr), name, parentenv, handler)
7: tryCatchOne(expr, names, parentenv, handlers[[1L]])
8: tryCatchList(expr, classes, parentenv, handlers)
9: tryCatch(do.call(msg$data$fun, msg$data$args, quote = TRUE), error = handler)
10: doTryCatch(return(expr), name, parentenv, handler)
11: tryCatchOne(expr, names, parentenv, handlers[[1L]])
12: tryCatchList(expr, classes, parentenv, handlers)
13: tryCatch({ msg <- recvData(master) if (msg$type == "DONE") { closeNode(master) break } else if (msg$type == "EXEC") { success <- TRUE handler <- function(e) { success <<- FALSE structure(conditionMessage(e), class = c("snow-try-error", "try-error")) } t1 <- proc.time() value <- tryCatch(do.call(msg$data$fun, msg$data$args, quote = TRUE), error = handler) t2 <- proc.time() value <- list(type = "VALUE", value = value, success = success, time = t2 - t1, tag = msg$data$tag) msg <- NULL sendData(master, value) value <- NULL }}, interrupt = function(e) NULL)
14: slaveLoop(makeSOCKmaster(master, port, timeout, useXDR))
15: parallel:::.slaveRSOCK()
An irrecoverable exception occurred. R is aborting now ...
Error in unserialize(node$con) : error reading from connection > sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 8 (jessie)
Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.12.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rstan_2.15.1 StanHeaders_2.15.0-1 dplyr_0.5.0
[4] purrr_0.2.2.2 readr_1.1.0 tidyr_0.6.2
[7] tibble_1.3.0 ggplot2_2.2.1 tidyverse_1.1.1
[10] MASS_7.3-47
loaded via a namespace (and not attached):
[1] Rcpp_0.12.10 compiler_3.4.0 cellranger_1.1.0 plyr_1.8.4
[5] forcats_0.2.0 tools_3.4.0 jsonlite_1.4 lubridate_1.6.0
[9] gtable_0.2.0 nlme_3.1-131 lattice_0.20-35 psych_1.7.5
[13] DBI_0.6-1 parallel_3.4.0 haven_1.0.0 gridExtra_2.2.1
[17] xml2_1.1.1 stringr_1.2.0 httr_1.2.1 hms_0.3
[21] stats4_3.4.0 grid_3.4.0 inline_0.3.14 R6_2.2.1
[25] readxl_1.0.0 foreign_0.8-68 reshape2_1.4.2 modelr_0.1.0
[29] magrittr_1.5 scales_0.4.1 assertthat_0.2.0 mnormt_1.5-5
[33] rvest_0.3.2 colorspace_1.3-2 stringi_1.1.5 lazyeval_0.2.0
[37] munsell_0.4.3 broom_0.4.2 |
|
@bgoodri --- any idea what's going on with this R/Linux combo? |
|
In stanc try replacing parallel::makePSOCKcluster with parallel::makeForkCluster. That works in our environment using RStudio Server on Linux. |
|
On Linux, it shouldn't even be calling makePSOCKcluster() unless you call
it from RStudio.
…On Mon, May 15, 2017 at 12:52 PM, Bill Gillespie ***@***.***> wrote:
In stanc try replacing parallel::makePSOCKcluster with
parallel::makeForkCluster. That works in our environment using RStudio
Server on Linux.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#382 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADOrqu6POvz2XymEdJjfDzWUDLZxHjJxks5r6ILEgaJpZM4Lslel>
.
|
|
I am referring to the situation where stan() is called from RStudio Server on Linux. I'm guessing something prevents passing of needed environment info when makePSOCKcluster is used in our system. |
|
Replacing Error in unserialize(node$con) : error reading from connection
11. unserialize(node$con)
10. recvData.SOCK0node(con)
9. recvData(con)
8. FUN(X[[i]], ...)
7. lapply(cl, recvResult)
6. checkForRemoteErrors(lapply(cl, recvResult))
5. clusterCall(cl, eval, substitute(expr), env = .GlobalEnv)
4. parallel::clusterEvalQ(cl, rstan::stanc(file, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)) at <tmp>#19
3. stanc(file = file, model_code = model_code, model_name = model_name, verbose = verbose, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)
2. stan_model(file, model_name = model_name, model_code = model_code, stanc_ret = NULL, boost_lib = boost_lib, eigen_lib = eigen_lib, save_dso = save_dso, verbose = verbose)
1. stan(m, data = data_list, iter = 1, chains = 1, init_r = 1) Any further suggestions? |
|
Yes, on RStudio it should call makePSOCKcluster() but otherwise on Linux
servers it is just calling mclapply(). I think there could be security
settings or something that prevent makePSOCKcluster() from working.
…On Mon, May 15, 2017 at 5:31 PM, Bill Gillespie ***@***.***> wrote:
I am referring to the situation where stan() is called from RStudio Server
on Linux. I'm guessing something prevents passing of needed environment
info when makePSOCKcluster is used in our system.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#382 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADOrqg_kLOE6yv2mVZYP5YXyC6HUEDAfks5r6MRAgaJpZM4Lslel>
.
|
|
Does it work on your server if you call your R code from the terminal
rather than from RStudio?
…On Mon, May 15, 2017 at 8:03 PM, aornugent ***@***.***> wrote:
Replacing parallel::makePSOCKcluster with parallel::makeForkCluster using
trace() results in:
Error in unserialize(node$con) : error reading from connection 11. unserialize(node$con) 10. recvData.SOCK0node(con) 9. recvData(con) 8. FUN(X[[i]], ...) 7. lapply(cl, recvResult) 6. checkForRemoteErrors(lapply(cl, recvResult)) 5. clusterCall(cl, eval, substitute(expr), env = .GlobalEnv) 4. parallel::clusterEvalQ(cl, rstan::stanc(file, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)) at <tmp>#193. stanc(file = file, model_code = model_code, model_name = model_name, verbose = verbose, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined) 2. stan_model(file, model_name = model_name, model_code = model_code, stanc_ret = NULL, boost_lib = boost_lib, eigen_lib = eigen_lib, save_dso = save_dso, verbose = verbose) 1. stan(m, data = data_list, iter = 1, chains = 1, init_r = 1)
Any further suggestions?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#382 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADOrqp7ZvukZbjg7WAfK5qjaGkS3Cu3Mks5r6OfUgaJpZM4Lslel>
.
|
|
R from the terminal results in: testfit <- stan(m, data=data_list, iter=1, chains = 1, init_r = 1)
*** caught illegal operation ***
address 0x7f070f723b0c, cause 'illegal operand'
Traceback:
1: .Call("CPP_stanc280", model_code, model_cppname, allow_undefined)
2: stanc(file = file, model_code = model_code, model_name = model_name, verbose = verbose, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)
3: stan_model(file, model_name = model_name, model_code = model_code, stanc_ret = NULL, boost_lib = boost_lib, eigen_lib = eigen_lib, save_dso = save_dso, verbose = verbose)
4: stan(m, data = data_list, iter = 1, chains = 1, init_r = 1)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace |
|
Maybe try with -O2 instead of -O3 in ~/.R/Makevars? Or the reverse if you
are doing -O2 or -Os now.
…On Wed, May 17, 2017 at 1:02 AM, aornugent ***@***.***> wrote:
R from the terminal results in:
testfit <- stan(m, data=data_list, iter=1, chains = 1, init_r = 1)
*** caught illegal operation ***address 0x7f070f723b0c, cause 'illegal operand'
Traceback:
1: .Call("CPP_stanc280", model_code, model_cppname, allow_undefined)
2: stanc(file = file, model_code = model_code, model_name = model_name, verbose = verbose, obfuscate_model_name = obfuscate_model_name, allow_undefined = allow_undefined)
3: stan_model(file, model_name = model_name, model_code = model_code, stanc_ret = NULL, boost_lib = boost_lib, eigen_lib = eigen_lib, save_dso = save_dso, verbose = verbose)
4: stan(m, data = data_list, iter = 1, chains = 1, init_r = 1)
Possible actions:1: abort (with core dump, if enabled)2: normal R exit3: exit R without saving workspace4: exit R saving workspace
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#382 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADOrqh72aula26NsvmolrzIRTkKaZdiZks5r6n9dgaJpZM4Lslel>
.
|
|
Tweaking jrnold's dockerfile to use the -O2 tags causes the build to hang when compiling on several different computers. Key excerpt below, the full logs can be seen here: https://hub.docker.com/r/aornugent/docker-stan/builds/bbunfihmsgcvbuge3cyepna/ |
|
I'm so joyful to see the issue hasn't been fixed yet (stan version: 2.26)! True R style magic, good job devs! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary:
calling
stan_modelon a.stanfile results in an an error.Of note: I can't reproduce this error on my laptop, this is only reproducible on an RHEL server.
Description:
If I call stan_model on a stan file, I get:
Reproducible Steps:
Here is the
stanfile I'm trying to compile, calledtest.stan:Calling:
leads to the error.
Current Output:
The only output is:
Expected Output:
A compiled model.
RStan Version:
2.14.1
rstan (Version 2.14.1, packaged: 2016-12-28 14:55:41 UTC, GitRev: 5fa1e80)
R Version:
[1] "R version 3.2.3 (2015-12-10)"
Session info:
The text was updated successfully, but these errors were encountered: