Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Able to install and load rstan on RHEL7, but error on compile #829

Closed
leungi opened this issue Aug 15, 2020 · 18 comments
Closed

Able to install and load rstan on RHEL7, but error on compile #829

leungi opened this issue Aug 15, 2020 · 18 comments

Comments

@leungi
Copy link

leungi commented Aug 15, 2020

Summary:

rstan installed and able to load, but error on compile.

Description:

  • Installed rstan via CLI - scl enable devtoolset-7 "/opt/R/4.0.2/bin/Rscript -e 'install.packages(\"rstan\", repos=\"https://cloud.r-project.org\")'"
  • installed fine, and library able to load in R session
  • tried to compile Eight Schools example, but error arises

Reproducible Steps:

> Sys.getenv("LD_LIBRARY_PATH")
[1] "/usr/bin/ld:/usr/local/lib:/usr/lib:/usr/local/lib64:/usr/lib64:/opt/R/4.0.2/lib/R/lib::/lib:/usr/local/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/lib/amd64/server:/usr/lib/oracle/12.2/client64/lib"

> library("rstan")
Loading required package: StanHeaders
Loading required package: ggplot2
rstan (Version 2.21.2, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)

> stan_code <- "
 // saved as schools.stan
 data {
   int<lower=0> J;         // number of schools 
   real y[J];              // estimated treatment effects
   real<lower=0> sigma[J]; // standard error of effect estimates 
 }
 parameters {
   real mu;                // population treatment effect
   real<lower=0> tau;      // standard deviation in treatment effects
   vector[J] eta;          // unscaled deviation from mu by school
 }
 transformed parameters {
   vector[J] theta = mu  tau * eta;        // school treatment effects
 }
 model {
   target = normal_lpdf(eta | 0, 1);       // prior log-density
   target = normal_lpdf(y | theta, sigma); // log-likelihood
 }
 "
> schools_dat <- list(J = 8,
                      y = c(28,  8, -3,  7, -1,  1, 18, 12),
                      sigma = c(15, 10, 16, 11,  9, 11, 10, 18))

> fit <- stan(model_code = stan_code, data = schools_dat)
Error in open.connection(con, open = mode) : HTTP error 403.
Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! /usr/bin/ld: /home/martinj1/R/x86_64-pc-linux-gnu-library/4.0/rstan/lib//libStanServices.a(stan_fit.o): unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
make: *** [fileddae3d9ad70d.so] Error 1
In addition: Warning message:
In system(cmd, intern = !verbose) :
  running command '/opt/R/4.0.2/lib/R/bin/R CMD SHLIB fileddae3d9ad70d.cpp 2> fileddae3d9ad70d.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection

Current Output:

Refer to Reproducible Steps section.

Expected Output:

fit model from Eight Schools example

RStan Version:

> packageVersion("rstan")
[1] ‘2.21.2

R Version:

> R.version.string
[1] "R version 4.0.2 (2020-06-22)"

Operating System:

Red Hat Enterprise Linux Server 7.3 (Maipo)

@jgabry
Copy link
Member

jgabry commented Aug 15, 2020

Sorry you're getting an error. We've been seeing some Mac and Windows errors but not so many errors on Linux. Tagging @bgoodri who will hopefully have some ideas.

@bgoodri
Copy link
Contributor

bgoodri commented Aug 15, 2020 via email

@leungi
Copy link
Author

leungi commented Aug 15, 2020

Appreciate the prompt response!

@jgabry: no need for apologies 🙂 I'm sure everyone noticed that the Stan team has been actively addressing these kinds of issue lately.

@bgoodri : I've tried several options (from existing Github issue, SO, and my previous successes with Stan installation); the one that works for current installation is:

CXX14FLAGS=-O3 -march=native -mtune=native -fPIC
CXX14FLAGS+= -Wno-unused-variable -Wno-unused-function
CXX14 = g++ -std=c++1y

@jgabry
Copy link
Member

jgabry commented Aug 15, 2020

@bgoodri Were you implying that -fPIC should or should not be in Makevars? Looks like it's there.

the one that works for current installation is:

@leungi Just to clarify, by "works for current installation" do you mean it works ok during the installation process but still does not work when compiling models?

I'm sure everyone noticed that the Stan team has been actively addressing these kinds of issue lately.

Yeah unfortunately there have been a lot of difficulties with the latest RStan release.

@leungi
Copy link
Author

leungi commented Aug 15, 2020

@jgabry: you're correct; the current Makevars allow successful installation, but fail on compilation.

I believe @bgoodri meant to add -fPIC; removing it resulted in failed installation (see below), with a suggestion to recompile with -fPIC.

...
...
...
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld:chains.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: pointer-tools.o: relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan_fit_base.o: relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: init.o: relocationR_X86_64_32 against `.rodata' can not be used when making a shared object; recompilewith -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stanc.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: Module.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan_fit_rccp.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be usedwhen making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: sparse_extractors.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/ast_def.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/program_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/expression_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/statement_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/expression07_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/statement_2_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/block_var_decls_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/indexes_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/local_var_decls_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/semantic_actions_def.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/whitespace_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/bare_type_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/term_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: stan/lang/grammars/functions_grammar_inst.o: relocation R_X86_64_32 against undefined symbol `__pthread_key_create' can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: final link failed:Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make: *** [/opt/R/4.0.2/lib/R/share/make/shlib.mk:6: rstan.so] Error 1
ERROR: compilation failed for package ârstanâ
* removing â/opt/R/4.0.2/lib/R/library/rstanâ
* restoring previous â/opt/R/4.0.2/lib/R/library/rstanâ

The downloaded source packages are in
â/tmp/RtmppzGf5F/downloaded_packagesâ
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages("rstan", repos = "https://cloud.r-project.org",:
installation of package ârstanâ had non-zero exit status

@bgoodri
Copy link
Contributor

bgoodri commented Aug 15, 2020 via email

@leungi
Copy link
Author

leungi commented Aug 15, 2020

Thanks for suggestion @bgoodri.

Here is stdout from example(stan_model, package = "rstan", run.dontrun = TRUE) - rstan-example-stdout.txt

Sessioninfo

R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.3 (Maipo)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblasp-r0.3.3.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rstan_2.21.2         ggplot2_3.3.2        StanHeaders_2.21.0-5

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5         magrittr_1.5       munsell_0.5.0      colorspace_1.4-1
 [5] R6_2.4.1           rlang_0.4.7        fansi_0.4.1        tools_4.0.2
 [9] parallel_4.0.2     pkgbuild_1.1.0     grid_4.0.2         gtable_0.3.0
[13] loo_2.3.1          cli_2.0.2          withr_2.2.0        matrixStats_0.56.0
[17] ellipsis_0.3.1     RcppParallel_5.0.2 assertthat_0.2.1   tibble_3.0.3
[21] lifecycle_0.2.0    crayon_1.3.4       processx_3.4.3     gridExtra_2.3
[25] callr_3.4.3        codetools_0.2-16   vctrs_0.3.2        ps_1.3.4
[29] curl_4.3           inline_0.3.16      glue_1.4.1         V8_3.2.0
[33] compiler_4.0.2     pillar_1.4.6       prettyunits_1.1.1  scales_1.1.1
[37] stats4_4.0.2       jsonlite_1.7.0     pkgconfig_2.0.3

@bgoodri
Copy link
Contributor

bgoodri commented Aug 15, 2020 via email

@leungi
Copy link
Author

leungi commented Aug 15, 2020

@bgoodri:

Makevars which allowed successful install as such:

CXX14FLAGS=-O3 -march=native -mtune=native -fPIC
CXX14FLAGS+= -Wno-unused-variable -Wno-unused-function
CXX14 = g++ -std=c++1y

The system compiler is indeed old, hence I use scl enable devtoolset-7 "/opt/R/4.0.2/bin/R" to ensure session is on newer compiler.

@bgoodri
Copy link
Contributor

bgoodri commented Aug 16, 2020 via email

@leungi
Copy link
Author

leungi commented Aug 16, 2020

Whoaaa...it seems to work 🤩

🙏 both for the guidance!

Steps taken:

  1. Add back -fPIC into CXX14FLAGS args in Makevars
  2. run session with scl enable devtoolset-7 "/opt/R/4.0.2/bin/R" to ensure compiler is up to date, in this case 7.3.1
(python) [root@ohylpyt2-d bin]# scl enable devtoolset-7 "g++ --version"
g++ (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Here is stdout from example(stan_model, package = "rstan", run.dontrun = TRUE) - rstan-example-v.txt

One persistent concern I have is this warning message as model starts to compile - Error in open.connection(con, open = mode) : HTTP error 403, though it doesn't seem to affect compilation.

I'll raise another ticket for this.

@leungi leungi closed this as completed Aug 16, 2020
@bgoodri
Copy link
Contributor

bgoodri commented Aug 16, 2020 via email

@jgabry
Copy link
Member

jgabry commented Aug 16, 2020

@bgoodri I didn't realize it tries to download even if there's no connection. I think the download should really be wrapped in try() or something like that because a user shouldn't have to see error messages just because they're not connected to the internet.

@bgoodri
Copy link
Contributor

bgoodri commented Aug 16, 2020 via email

@jgabry
Copy link
Member

jgabry commented Aug 16, 2020

Ah ok, but there has to be a better option than just letting the user see irrelevant error messages. If we can't suppress those error messages then I guess stanc_beta should be skipped entirely if there's no internet connection instead of wrapped in try().

Or maybe wrapping more lines in stanc_beta in try() would work. E.g., the line sourcing the file from GitHub:

ctx$source("https://github.com/stan-dev/stanc3/releases/download/nightly/stanc.js")

@leungi
Copy link
Author

leungi commented Aug 17, 2020

@jgabry: will this work; if so, I can do a PR.

> ctx <- V8::v8()

# original
> ctx$source("https://github.com/stan-dev/stanc3/releases/download/nightly/stanc.js")
Error in open.connection(con, open = mode) : HTTP error 403.

# edit
> tryCatch({
   ctx$source("https://github.com/stan-dev/stanc3/releases/download/nightly/stanc.js")
   },
   error = function(e) {
     cat("")
   }
 )
>

@jgabry
Copy link
Member

jgabry commented Aug 17, 2020

Not sure off the top of my head if that will work, but yeah it looks like the right idea.

@drezap
Copy link

drezap commented Dec 22, 2022

Hey guys!

Getting something similar, but instead of Rstan, I'm using building the stan math library standalone. I'm trying to add the following flag to one of the make files, to see if this will compile my stan program: CXX14=g++.

This is a centOS server. What I'm doing is looking for the string to detect $OS within a makefile, by using echo $OS, and nothings coming up.

How do I check the value of this string?

I've also just added CXX14=g++ immediately after the block that starts with the comment ## Set default compiler, with no luck.

Any suggestions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants