pak often forgets how to build from source

[hadley]

remove.packages("ellipsis")
pak::pkg_install("r-lib/ellipsis")
#> ℹ Checking for package metadata updates
#> ✔ All 10 metadata files are current.                                            
#> ✔ Using session cached package metadata
#> ✔ Using cached package metadata                                                 
#>                                                                                 
#> → Will install 1 packages:
#>   r-lib/ellipsis
#>   
#> → Will not update 1 packages.
#> 
#> → Will download 1 packages with unknown size.
#> 
#> ℹ Building ellipsis 0.1.0.9000                                                  
#> Error: Could not find tools necessary to compile a package

After running

pak:::pkg_data$remote$run(function() pkgbuild::has_build_tools())
#> [1] TRUE

The problem appears to go away. Running pak_setup() also seems to fix the problem. I don't know why it fails in the first place (since I should always be able to compile packages). I can't reliably trigger it, but it happens to me pretty frequently.

[hadley]

pak:::pkg_data$remote$run_with_output(function() pkgbuild::has_build_tools(TRUE))

[hadley]

Ok here it is

[master] > pak:::pkg_data$remote$run_with_output(function() pkgbuild::has_build_tools(TRUE))
$code
[1] 200

$message
[1] "done file1837474d2bd7f"

$result
[1] FALSE

$stdout
[1] "Running /Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB foo.c\nccache clang -Qunused-arguments  -I\"/Library/Frameworks/R.framework/Resources/include\" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c foo.c -o foo.o\n\033[31m/bin/sh: ccache: command not found\n\033[39m\033[31mmake: *** [foo.o] Error 127\n\033[39m"

$stderr
[1] "Trying to compile a simple C file\n"

$error
NULL

attr(,"class")
[1] "callr_session_result"

[gaborcsardi]

Aha! ccache!

[gaborcsardi]

@hadley how did you install and set up ccache? Is it from brew? Do you have sg in your user Makevars? It is possible that it is because of this: r-lib/pkgbuild#76

[hadley]

My ~/.R/Makevars is:

CC=ccache clang -Qunused-arguments
CXX=ccache clang++ -Qunused-arguments

And which ccache gives /usr/local/bin/ccache.

[gaborcsardi]

@hadley One last question. Do you have /usr/local/bin in /etc/paths? Or do you change the PATH somewhere else, e.g. your shell profile? I don't quite understand why R CMD SHLIB would not find ccache.

[hadley]

I change it in my bash profile:

export PATH="/usr/local/bin:$PATH:/usr/local/sbin:$HOME/bin:/usr/local/ggobi/bin:/Users/hadley/node_modules/.bin:/Library/TeX/texbin"

[gaborcsardi]

I think that profile does not run when you start RStudio from the GUI.

But still, /usr/local/bin should be on the PATH by default. What do you have in /etc/paths? Or /private/etc/paths?

Also, can you pls run these from an RStudio that you started from a GUI:

Sys.getenv("PATH")
callr::r(function() Sys.getenv("PATH"))

Thanks!

[hadley]

By hand, from console

Sys.getenv("PATH")
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS"
callr::r(function() Sys.getenv("PATH"))
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS"

And as reprex, just to check they're the same:

Sys.getenv("PATH")
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS"
callr::r(function() Sys.getenv("PATH"))
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS"

^{Created on 2019-06-25 by the reprex package (v0.2.1.9000)}

[hadley]

Run pak:::pkg_data$remote$run_with_output(function() Sys.getenv("PATH")) next time it happens

[hadley]

callr::r(function() Sys.getenv("PATH"))
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS"
pak:::pkg_data$remote$run_with_output(function() Sys.getenv("PATH"))$result
#> [1] "/usr/bin:/bin:/usr/sbin:/sbin"

[gaborcsardi]

So for record, this is what happens.

1. When RStudio starts up, it starts running the .Rprofile immediately, before its setup is complete.
2. On macOS, if you start RStudio from the GUI, PATH is not set "correctly" at the startup time. Rstudio reads /etc/paths as part of the startup process, and sets PATH, but that's too late for the subprocesses started from .Rprofile.
3. pak is loaded in .Rprofile, and it immediately starts a background process.
4. ~/.R/Makevars has CC=ccache clang and ccache is on /usr/local/bin.

So when RStudio starts up and runs the .Rprofile which starts the pak subprocess, the PATH does not include /usr/local/bin, and then the build fails, because ccache is not available.

[nlbjan1]

@gaborcsardi Just saw your userR! presentation, regarding the question about caching compiled packages, I wanted to add on: Would it make sense to suggest installation of ccache and integrate it directly into pak so it will always be used and users don't have to bother with Makevars? Maybe not worth the extra complexity in pak itself?

[gaborcsardi]

@nlbjan1 maybe we could suggest it in the documentation. I would probably not install it from pak, because that's a system task.

[bobjansen]

I didn't realize that my reference in the commit and editing it would spam this thread :( In any case, I've added a paragraph on this in the README.