Installing from source fails for arm64-based Macs (M1)

[jankowtf]

Running `renv::install("scrypt") fails with the following error message

[...]
config.status: executing depfiles commands
** libs
clang -I"/opt/homebrew/Cellar/r/4.1.1_1/lib/R/include" -DNDEBUG  -I'/Users/jankothyson/Code/sandbox/renv/staging/1/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include  -I./scrypt-1.1.6/ -I./scrypt-1.1.6/lib -I./scrypt-1.1.6/lib/util -DHAVE_CONFIG_H -fPIC  -Wno-implicit-function-declaration  -c scrypt-1.1.6/lib/crypto/sha256.c -o scrypt-1.1.6/lib/crypto/sha256.o
clang -I"/opt/homebrew/Cellar/r/4.1.1_1/lib/R/include" -DNDEBUG  -I'/Users/jankothyson/Code/sandbox/renv/staging/1/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include  -I./scrypt-1.1.6/ -I./scrypt-1.1.6/lib -I./scrypt-1.1.6/lib/util -DHAVE_CONFIG_H -fPIC  -Wno-implicit-function-declaration  -c scrypt-1.1.6/lib/crypto/crypto_scrypt-ref.c -o scrypt-1.1.6/lib/crypto/crypto_scrypt-ref.o
clang++ -std=gnu++14 -I"/opt/homebrew/Cellar/r/4.1.1_1/lib/R/include" -DNDEBUG  -I'/Users/jankothyson/Code/sandbox/renv/staging/1/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include  `/opt/homebrew/Cellar/r/4.1.1_1/lib/R/bin/Rscript -e "Rcpp:::CxxFlags()"` -I./scrypt-1.1.6/ -I./scrypt-1.1.6/lib -I./scrypt-1.1.6/lib/util -DHAVE_CONFIG_H -fPIC  -g -O2  -c RcppExports.cpp -o RcppExports.o
clang: error: no such file or directory: 'Support/renv/cache/v5/R-4.1/aarch64-apple-darwin20.6.0/Rcpp/1.0.7/dab19adae4440ae55aa8a9d238b246bb/Rcpp/include'
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for package ‘scrypt’

I also tried it outside of renv via install.packages("scrypt", type = "source") and R CMD install --build scrypt and both works

Update 2021-11-11 15:50

I switched my R version from the one installable via brew install r to the latest RC from https://mac.r-project.org/#nightly (4.1.2 RC) and now `renv::install("scrypt") runs smoothly

> renv::install("scrypt")
Retrieving 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.1/scrypt_0.1.3.tgz' ...
	OK [downloaded 295 Kb in 0.6 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.1/Rcpp_1.0.7.tgz' ...
	OK [downloaded 3.2 Mb in 0.2 secs]
Installing Rcpp [1.0.7] ...
	OK [installed binary]
Moving Rcpp [1.0.7] into the cache ...
	OK [moved to cache in 0.4 milliseconds]
Installing scrypt [0.1.3] ...
	OK [installed binary]
Moving scrypt [0.1.3] into the cache ...
	OK [moved to cache in 0.3 milliseconds]

System info

• MBP M1 with Max Chip
• macOS Monterey 12.0.1
• R versions
  • The issue occured when using R as installed with Homebrew via brew install r (as of 2021-11-06)
  • After switching to the arm64-specific nightly build (4.1.2 RC) the install works. Will try the stable 4.1.2 version for arm64 next)
• GCC/GFortran compilers installed as instructed on https://mac.r-project.org/tools/ (https://mac.r-project.org/libs-arm64/gfortran-f51f1da0-darwin20.0-arm64.tar.gz in particular)
• No particular ~/.R/Makevars used (as was still necessary a couple of weeks ago for other packages; see this issue)

[kevinushey]

I think this is arguably an issue in Rcpp, that only occurs if your package happens to be installed into a path that contains spaces. What is the output of:

Rcpp:::CxxFlags()

for you?

For what it's worth, the latest versions of renv will have migrated the cache to a new (CRAN-compliant) location, so I see:

> Rcpp:::CxxFlags()
-I/Users/kevinushey/Library/Caches/org.R-project.R/R/renv/cache/v5/R-4.1/x86_64-apple-darwin17.0/Rcpp/1.0.7/dab19adae4440ae55aa8a9d238b246bb/Rcpp/include

which likely would work fine in this case.

[kevinushey]

I've filed an issue + PR in Rcpp that I think will resolve this issue. In the meantime, I think you'll need to migrate the renv cache from its current location to the new (with no spaces) location.

[jankowtf]

@kevinushey thanks a lot for addressing this so quickly (as you always do - really appreciating it!)

I've checked and seems like you're right about the spaces:

> Rcpp:::CxxFlags()
-I/Users/jankothyson/Library/Application Support/renv/cache/v5/R-4.1/aarch64-apple-darwin20/Rcpp/1.0.7/dab19adae4440ae55aa8a9d238b246bb/Rcpp/include

Still using renv@0.13.2 as I was restoring an old project. But I'll upgrade to the latest version ASAP.

Just to mention this once more, though: the problem did in fact NOT occur anymore after switching to the latest arm64-based version of R as I outlined above.

[jankowtf]

@kevinushey stupid question: how do I actually make renv migrate the cache from the previous location to the new one once I switched to 0.14?

[kevinushey]

I think one solution would be to move the cache to the "new" location, and leave a symlink at the "old" location pointing to the new one. For example, in R:

old <- "~/Library/Application Support/renv"
new <- "~/Library/Caches/org.R-project.R/R/renv"

dir.create(dirname(new), recursive = TRUE, showWarnings = FALSE)
renv:::renv_file_move(old, new)
renv:::renv_file_link(new, old)

This way, older projects using the old cache location could still "see" it using their existing symlinks, but paths would resolve to the "new" path without spaces.

[kevinushey]

As an aside, the reason why installation of rscrypt succeeded in other cases is because a binary was downloaded, instead of installing from sources.

[jankowtf]

@kevinushey

I ran your code from above and it went through just fine.

But do I need to tell renv (I'm using the latest 0.14 CRAN version) to use the new cache somehow?

Because when I run renv::install("Rcpp") in a fresh project after moving the cache, I get the following

Welcome to renv!

It looks like this is your first time using renv. This is a one-time message,
briefly describing some of renv's functionality.

renv maintains a local cache of data on the filesystem, located at:

  - "~/Library/Application Support/renv"

This path can be customized: please see the documentation in `?renv::paths`.

renv will also write to files within the active project folder, including:

  - A folder 'renv' in the project directory, and
  - A lockfile called 'renv.lock' in the project directory.

In particular, projects using renv will normally use a private, per-project
R library, in which new packages will be installed. This project library is
isolated from other R libraries on your system.

In addition, renv will update files within your project directory, including:

  - .gitignore
  - .Rbuildignore
  - .Rprofile

Please read the introduction vignette with `vignette("renv")` for more information.
You can browse the package documentation online at https://rstudio.github.io/renv/.
Do you want to proceed? [y/N]: 

And renv::paths$cache() gives me this:

> renv::paths$cache()
[1] "~/Library/Application Support/renv/cache/v5/R-4.1/aarch64-apple-darwin20"

Update 2021-11-12 11:10

I read through https://rstudio.github.io/renv/reference/paths.html and went with setting RENV_PATHS_ROOT = "~/Library/Caches/org.R-project.R/R/renv" in my ~/.Renviron. Two questions on this:

1. I guess this is the best one can do when not wanting to delete the old location to not break projects/packages depending on it, right?
2. What would happen if I went ahead and deleted the old location? Would old projects simply switch to using the new cache instead?

[jankowtf]

@kevinushey re: scrypt:
Yeah, you're totally right. I thought I explicitly called renv::install("scrypt", rebuild = TRUE, type = "source") but when I double checked, I got the error.

[kevinushey]

That above issue is unexpected to me -- let me see if I can figure out why that is happening.

I guess this is the best one can do when not wanting to delete the old location to not break projects/packages depending on it, right?

This is the other alternative, and is the generally recommended approach if you need finer-grained control over where renv stores its cache of data / packages and so on.

What would happen if I went ahead and deleted the old location? Would old projects simply switch to using the new cache instead?

I believe you'd see renv report that all of the project's package symlinks appear to be broken, and you would be prompted to repair those symlinks (e.g. via renv::restore()). Assuming the lockfile and package library were in sync, that call to renv::restore() should automagically find and use the packages in the new cache location.

[kevinushey]

I think I've fixed the above issue in the development version of renv. We were inadvertently using the "old" cache location if it existed, even if the "new" cache location existed as well. Could you let me know if that appears to be the case for you as well?

[kathrynnapier]

I just wanted to thank you @rappster - was able to use your instructions to set up R/R studio and my project after a frustrating morning setting up my new Mac.