package compilation failure

[mansillo]

This is perhaps an esoteric approach to installing the RcppArmadillo package update but here goes. I routinely update my R packages in RStudio with the check for updates function. Every single time that the RcppArmadillo package appears in that list it fails to compile. I restart the session, restart R, terminate R, restart the computer etc and various iterations of these and the package stubbornly does not compile. Then typically a day or so later I have found in the past without an explanation I can identify the package updates. This drives me utterly mad.

I run the code ...
install.packages("RcppArmadillo")

If anyone can identify why this occurs locally, I'd be interested to know:

installing source package ‘RcppArmadillo’ ...
** package ‘RcppArmadillo’ successfully unpacked and MD5 sums checked
** using staged installation
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C++... yes
checking whether clang++ -mmacosx-version-min=10.13 -std=gnu++14 accepts -g... yes
checking for clang++ -mmacosx-version-min=10.13 -std=gnu++14 option to enable C++11 features... none needed
checking how to run the C++ preprocessor... clang++ -mmacosx-version-min=10.13 -std=gnu++14 -E
checking whether the compiler supports GNU C++... (cached) yes
checking whether clang++ -mmacosx-version-min=10.13 -std=gnu++14 accepts -g... (cached) yes
checking for clang++ -mmacosx-version-min=10.13 -std=gnu++14 option to enable C++11 features... (cached) none needed
checking whether we have a suitable tempdir... /var/folders/gm/lg9j9k791ms06zsbp6r9zk500000gn/T
checking whether R CMD SHLIB can already compile programs using OpenMP... yes
checking LAPACK_LIBS... R-supplied partial LAPACK found
configure: WARNING: Some complex-valued LAPACK functions may not be available
configure: creating ./config.status
config.status: creating inst/include/RcppArmadillo/config/RcppArmadilloConfigGenerated.h
config.status: creating src/Makevars
** libs
clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include -I../inst/include -fPIC -Wall -g -O2 -c RcppArmadillo.cpp -o RcppArmadillo.o
In file included from RcppArmadillo.cpp:21:
In file included from ../inst/include/RcppArmadillo.h:29:
../inst/include/RcppArmadillo/interface/RcppArmadilloForward.h:68:51: error: no template named 'subview_cols' in namespace 'arma'; did you mean 'subview_col'?
template SEXP wrap ( const arma::subview_cols& ) ;
~~~~~~^~~~~~~~~~~~
subview_col
/usr/local/include/armadillo_bits/subview_bones.hpp:359:7: note: 'subview_col' declared here
class subview_col : public subview
^
In file included from RcppArmadillo.cpp:21:
In file included from ../inst/include/RcppArmadillo.h:35:
../inst/include/RcppArmadillo/interface/RcppArmadilloWrap.h:57:38: error: no template named 'subview_cols' in namespace 'arma'; did you mean 'subview_col'?
SEXP arma_subview_wrap( const arma::subview_cols& data, int nrows, int ncols ){
~~~~~~^~~~~~~~~~~~
subview_col
/usr/local/include/armadillo_bits/subview_bones.hpp:359:7: note: 'subview_col' declared here
class subview_col : public subview
^
In file included from RcppArmadillo.cpp:21:
In file included from ../inst/include/RcppArmadillo.h:35:
../inst/include/RcppArmadillo/interface/RcppArmadilloWrap.h:99:50: error: no template named 'subview_cols' in namespace 'arma'; did you mean 'subview_col'?
template SEXP wrap( const arma::subview_cols& data ){
~~~~~~^~~~~~~~~~~~
subview_col
/usr/local/include/armadillo_bits/subview_bones.hpp:359:7: note: 'subview_col' declared here
class subview_col : public subview
^
3 errors generated.
make: *** [RcppArmadillo.o] Error 1
ERROR: compilation failed for package ‘RcppArmadillo’
removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo’
restoring previous ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo’
Warning in install.packages :
installation of package ‘RcppArmadillo’ had non-zero exit status

[eddelbuettel]

Briefly:

1. The package is in great standing at CRAN; if it fails to compile for you can in fact download binary for macOS
2. The package was updated today at CRAN. Did you fail to compile the newest version too?
3. Moreover you did not even tell us what version you tried to compile. Which one was it?
4. I am truly sorry but I do not work on macOS. The very first issue you see on the Issue page is this pinned post by @coatless titled "Guide for macOS installation of RcppArmadillo/using Compiled Code" . Did you read it or the references therein?

I cannot really help here so I would suggest we close this. This appears to be a local issue at your end. Truly sorry.

[kevinushey]

It looks like you have a "system" copy of Armadillo that's being picked up at installation time, at /usr/local/include. For example:

/usr/local/include/armadillo_bits/subview_bones.hpp:359:7: note: 'subview_col' declared here

Presumedly, that's coming from R CMD config CPPFLAGS; I think R likes to set that by default for Intel builds of R on macOS.

kevin@MBP-P2MQ:/Library/Frameworks/R.framework/Versions/4.2/Resources/etc
$ cat Makeconf | grep '^CPPFLAGS ='
CPPFLAGS = -I/usr/local/include

Your best bet is to unset / override that temporarily in your ~/.R/Makevars, or to remove your system installation of Armadillo.

[eddelbuettel]

Well spotted @kevinushey !

(Or upgrade the system version to one as current as the CRAN version, but that may still lead to "surprises". Best to reign R in here then. Or remove the system one from a default path.)

[mansillo]

Briefly:

1. The package is in great standing at CRAN; if it fails to compile for you can in fact download binary for macOS
2. The package was updated today at CRAN. Did you fail to compile the newest version too?
3. Moreover you did not even tell us what version you tried to compile. Which one was it?
4. I am truly sorry but I do not work on macOS. The very first issue you see on the Issue page is this pinned post by @coatless titled "Guide for macOS installation of RcppArmadillo/using Compiled Code" . Did you read it or the references therein?

I cannot really help here so I would suggest we close this. This appears to be a local issue at your end. Truly sorry.

I today failed to install 0.11.4.4.0 - on the 16th January I tweeted you when I had difficulty with 0.11.4.3.1 and that resolved itself after I did many attempts and referring to that @coatless article. So I have read it and followed the steps.

I went to homebrew and did a lot of updating, cleaning and linking - the package was installed! I just reinstalled/updated a lot of things. I think around Big Sur Apple added an extra barrier to changing directories.
I'm not a computer scientist but a social scientist; my attempts to follow the @coatless guide which I recall worked like a treat on High Sierra might have been me reinstalling gcc rather than updating it? I've no idea really.
Thank you for your time and @kevinushey you were on the money with your recommendations.

[mansillo]

I don't know if this is something I should comment about here or on another R forum or R package github for this general matter. I mentioned earlier that I am not a computer scientist but I am a social scientist that happens to use statistics in my work so I use R. There are many packages that require R to interact with packages that deal with languages C, C++, Stan etc etc and packages that manage software such has proj@7 that homebrew supplies etc.
Having a more user-friendly guide to set up/diagnose issues with RcppArmadillo to access RStan in the R universe would be really useful.

[coatless]

@mansillo thanks for the kind words.

The issue at the moment is twofold regarding an "automatic" setup wizard:

1. My developer credentials for Apple expired last year, which means I can't release a new installer package; and,
2. I have yet to upgrade from my Intel mac to an M1/M2-powered mac.

That said, it's easy for me to work around both 1 & 2.

For 1, I would need a macOS community member that is part of the developer program (maybe @hrbrmstr or someone at Stan though, they weren't hot on the idea before) or a donation to cover the license fee (good for a year). The cost of the developer subscription for newly married and recently minted PhD that doesn't do heavy mac development anymore isn't something that passes the wife's sniff test.

On the second note, I likely can find a willing person to test the installer package. This will be slower, but more predictable. In Q3 of 2023, I'll likely even have access to the M1 chip via GitHub action runners (per GitHub Roadmap)

[eddelbuettel]

All this infrastructure business is news to me. I do carry the hat of an R Foundation Board member so if there is a general usability issue we could (and should) support it besides relying solely on @s-u (and effectively monopolosing @s-u) I would be all ears.

But I can't drive this, and I am (quite frankly) also not all that interested as I work on an OS where all this "just works", without paying anybody. So if you all choose something different, maybe you all can work out how to make it work while minimising impact on other volunteers like myself who, frankly, are really fully orthogonal to all your headaches.

[s-u]

@eddelbuettel Honestly, I have no idea what you are referring to. The way I read it is the user has messed up their system with Homebrew and tried (unnecessarily) to install it from sources. The guides mentioned above are terribly outdated and outright dangerous to follow making the toolchains unusable. This is not a fault of the guides, they were meant well at the time, but they do not apply to today's R as none of it is now necessary as we are now using standard Apple tools. In short, I suspect this issue can be trivially solved by deleting Homebrew (and any custom modifications from ~/.R) if the user really insists on compiling from sources. But, as you say, just running install.packages("RcppArmadillo") in CRAN version of R will "just work" without the necessity to compile anything so for normal R users none of the above applies.

[kevinushey]

Perhaps this could be resolved in RcppArmadillo by having it set the include paths in PKG_CPPFLAGS. Since R uses:

ALL_CPPFLAGS =  -I"$(R_INCLUDE_DIR)" -DNDEBUG $(PKG_CPPFLAGS) $(CLINK_CPPFLAGS) $(CPPFLAGS) 

Setting the include paths in PKG_CPPFLAGS would give RcppArmadillo a chance to "override" the include paths set via the R installation's CPPFLAGS. I'll prepare a PR.

[eddelbuettel]

@s-u I was making a specific request to consider helping with the creation of the installer @coatless provided in the past, but no longer can as he no developer credentials for the walled garded that macOS is. Note the comment directly preceding mine and its near-identical timestamp.

[s-u]

@eddelbuettel I still don't see installer of what - as I said nothing needs to be installed anymore, just regular Apple Xcode or command line tools. If there is anything I'd be happy to create it. One that I plan to do is the Fortran installer for the 4.3.0 release once we finalise the binary, but that's all I'm aware of (and I don't think that's needed here).

[mansillo]

@eddelbuettel Honestly, I have no idea what you are referring to. The way I read it is the user has messed up their system with Homebrew and tried (unnecessarily) to install it from sources. The guides mentioned above are terribly outdated and outright dangerous to follow making the toolchains unusable. This is not a fault of the guides, they were meant well at the time, but they do not apply to today's R as none of it is now necessary as we are now using standard Apple tools. In short, I suspect this issue can be trivially solved by deleting Homebrew (and any custom modifications from ~/.R) if the user really insists on compiling from sources. But, as you say, just running install.packages("RcppArmadillo") in CRAN version of R will "just work" without the necessity to compile anything so for normal R users none of the above applies.

There was a time before standard Apple tools contained all that was required. The upgrade to Catalina was a major irritant to many. There are legacies, errors were incurred and the point I was making was that it would be helpful to have a clear troubleshooting resource because the past did happen and many would like to address the past mistakes. How else are non-computer science background folk who require applied statistical programming deep knowledge to diagnose and address their issues?

[kevinushey]

TIL this is documented in R-exts, from https://cran.r-project.org/doc/manuals/R-exts.html#Using-Makevars:

N.B.: Include paths are preprocessor options, not compiler options, and must be set in PKG_CPPFLAGS as otherwise platform-specific paths (e.g. ‘-I/usr/local/include’) will take precedence. PKG_CPPFLAGS should contain ‘-I’, ‘-D’, ‘-U’ and (where supported) ‘-include’ and ‘-pthread’ options: everything else should be a compiler flag. The order of flags matters, and using ‘-I’ in PKG_CFLAGS or PKG_CXXFLAGS has led to hard-to-debug platform-specific errors.

[coatless]

@mansillo still a WIP, but consider checking out macrtools. I opted to forgo the installer route in-favor of an R package. This removes the need for point 1 (finding someone to sign or pay for the apple developer subscription ~$105 per year).

If you have an m1/m2 mac, I would appreciate some testing help. Sample setting up gfortran, which is required for RcppArmadillo:

Standard disclaimer: This is an unofficial project to provide a convenient toolchain setup for macOS. It is not officially approved by R Core, CRAN, or the R Foundation.

[eddelbuettel]

Just wanted to say that I really appreciate this @coatless!

[coatless]

We have a nice ~2 minutes-ish video using the package to install everything: Xcode CLI, gfortran, and necessary r-base-dev binaries.

https://www.youtube.com/watch?v=_fckF0fefXQ

Note: I've only really been able to test it heavily on x86_64, but I'm pretty confident that there shouldn't be an issue with arm64/aarch64 macs.

I'll ping Simon privately to poke at the package and circle back here later next week.

This is the best I can do without having a developer subscription

[coatless]

An update of sorts:

Bob tested out an early version of the package and found a few issues with arm64 (as to be expected). Thanks to Adam, we were able to work out the hiccups on arm64. Pinged bob to re-test so that we have at least 2 arm64 testers.

I haven't heard back from Simon. I sent out a friendly bump e-mail.

[FrankMaiwald]

I cannot compile RcppArmadillo or install it in R either.
I seems that my g++ uses -std gnu++17 and does not recognize llapack, lblas, lgfortran flags.
Can I set a different compiler / compiler options in R CMD INSTALL?
The issue hampers the installation of standard libraries like edgeR or DESeq2 (from Bioconductor).

Ubuntu 22.0.4
R version 4.3.1 (2023-06-16) -- "Beagle Scouts"

R CMD INSTALL RcppArmadillo_0.12.4.1.0.tar.gz
... all good ...
using C++ compiler: ‘g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0’
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/prg/R_libs_user/Rcpp/include' -fopenmp -fpic -g -O2 -ffile-prefix-map=/build/r-base-1upgAf/r-base-4.3.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -c RcppArmadillo.cpp -o RcppArmadillo.o
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/prg/R_libs_user/Rcpp/include' -fopenmp -fpic -g -O2 -ffile-prefix-map=/build/r-base-1upgAf/r-base-4.3.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/prg/R_libs_user/Rcpp/include' -fopenmp -fpic -g -O2 -ffile-prefix-map=/build/r-base-1upgAf/r-base-4.3.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -c fastLm.cpp -o fastLm.o
g++ -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o RcppArmadillo.so RcppArmadillo.o RcppExports.o fastLm.o -fopenmp -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
/usr/bin/ld: cannot find -llapack: No such file or directory
/usr/bin/ld: cannot find -lblas: No such file or directory
/usr/bin/ld: cannot find -lgfortran: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [/usr/share/R/share/make/shlib.mk:10: RcppArmadillo.so] Error 1
ERROR: compilation failed for package ‘RcppArmadillo’

• removing ‘/usr/prg/R_libs_user/RcppArmadillo’

same errors of course with install.packages("RcppArmadillo") in R

[eddelbuettel]

@FrankMaiwald

• for those who cannot (or maybe do not want to) compile RcppArmadillo we offer the package as a binary in the distribution so please just do sudo apt install r-cran-rcpparmadillo
• when compilation fails, it can be enlightening to read the error messages: here they tell you that 'lapack' and 'blas' and 'gfortran' are missing -- which is actually somewhat common so offer the r-base-dev package ensure they are present; for this you need to do sudo apt install r-base-dev

So in short, not an error in RcppArmadillo but a simple matter of you not having all ducks in the right rows. Good luck.

[FrankMaiwald]

Dear Dirk,
thank you for the very quick response. I did not want to blame RcppArmadillo but just hoped to get compilation / installation help. After all those years I am still a R noob...
Installing the r-base-dev package did help.