Cannot install knitr from source on macOS

[jdblischak]

My package is currently failing to build on macOS because install_deps() (on Travis) and install_local() (on rhub) cannot install knitr and stringi from source. It attempts to build knitr first, which fails because knitr depends on stringi (via stringr).

Here are the relevant parts of the Travis log:

> devtools::install_deps(dependencies = TRUE)
...
  There are binary versions available but the source versions are later:
        binary source needs_compilation
knitr     1.20   1.21             FALSE
stringi  1.2.4  1.3.1              TRUE
...
* installing *source* package ‘knitr’ ...
** package ‘knitr’ successfully unpacked and MD5 sums checked
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
  there is no package called ‘stringi’
Error : unable to load R code in package ‘knitr’
ERROR: lazy loading failed for package ‘knitr’
* removing ‘/Users/travis/R/Library/knitr’
Error in i.p(...) : 
  (converted from warning) installation of package ‘knitr’ had non-zero exit status
Calls: <Anonymous> ... with_rprofile_user -> with_envvar -> force -> force -> i.p
Execution halted

• Travis error with R 3.5.2
• Travis error with R 3.4.4
• rhub::check_on_macos() error from install_local()
• I searched for similar issues. The closest I could find was Issue install_deps() fails on travis on mac: 'missing value where TRUE/FALSE needed' #289, however they were able to build with rhub::check_on_macos()

[gaborcsardi]

I think this is because the installation order is wrong, remotes tries to install knitr before stringi. I am pretty sure that I have already seen an issue for this, but can't find it now....

[jdblischak]

A quick update. I verified that install.packages() installs the packages from source in the correct order.

> remove.packages(c("stringi", "stringr", "knitr"))
> install.packages("knitr")
also installing the dependencies ‘stringi’, ‘stringr’


  There are binary versions available but the source versions are later:
        binary source needs_compilation
stringi  1.1.6  1.3.1             FALSE
stringr  1.2.0  1.4.0             FALSE
knitr     1.18   1.21             FALSE

installing the source packages ‘stringi’, ‘stringr’, ‘knitr’
...
> packageVersion("stringi")
[1] ‘1.3.1’
> packageVersion("stringr")
[1] ‘1.4.0’
> packageVersion("knitr")
[1] ‘1.21’

[jdblischak]

I think this is because the installation order is wrong, remotes tries to install knitr before stringi.

Agreed.

I am pretty sure that I have already seen an issue for this, but can't find it now....

I also checked the travis-ci commmunity:r label, but couldn't find anything.

[jimhester]

This situation only happens where there is a source package without a built binary, if you force source packages for everything or binary packages for everything then it will work again. Or once the binary is built by CRAN then again it would work.

[jdblischak]

This situation only happens where there is a source package without a built binary, if you force source packages for everything or binary packages for everything then it will work again. Or once the binary is built by CRAN then again it would work.

@jimhester I agree it's an edge case and that there are ways around it when you control the environment (e.g. I fixed it for Travis by specifying r_packages: stringi). However, if possible, I still think it should be fixed for the following reasons:

1. install.packages() works in this situation
2. The user doesn't always have control of the computing environment, e.g. with rhub::check_on_macos()
3. It's difficult to anticipate this situation. For example, my R packages were cached on Travis for testing my dev branch. When I merged to master, the build on macOS unexpectedly failed.

For setting getOption("pkgType") on Travis, would it be possible to add a field to control this like is already done for other options like repos?

https://docs.travis-ci.com/user/languages/r#miscellaneous

And I guess an alternative option would be to customize the installation step:

install:
- options(pkgType = "binary")
- R -e 'devtools::install_deps(dep = T)'

[jdblischak]

I re-ran rhub::check_on_macos(). Now the devtools package also cannot be installed because a new version of R6 was released to CRAN:

https://builder.r-hub.io/status/workflowr_1.2.0.9000.tar.gz-b0be12486a934aab846e02f48cf8c022#L379

There are binary versions available but the source versions are later:
binary source needs_compilation
devtools 1.13.6 2.0.1 FALSE
knitr 1.20 1.21 FALSE
R6 2.3.0 2.4.0 FALSE
...
* installing *source* package ‘devtools’ ...
** package ‘devtools’ successfully unpacked and MD5 sums checked
** R
** inst
** byte-compile and prepare package for lazy loading
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
there is no package called ‘R6’
ERROR: lazy loading failed for package ‘devtools’
* removing ‘/Users/userCzAlV6UP/R/devtools’

So for packages like devtools that have a lot of dependencies, this could be a common issue on macOS.

[gaborcsardi]

Actually, I suspect that this is an install.packages() bug. #296 (comment) is not a good control experiment, because that's a direct install. I think it fails if multiple packages fall back to source when doing a binary install.

But this is just theory right now, this bug is quite hard to reproduce....

[jdblischak]

I suspect that this is an install.packages() bug

@gaborcsardi I don't understand. For two different scenarios where binary packages were unavailable (stringi+knitr and R6+devtools), install.packages() was able to install them in the correct order, but remotes::install_deps() tried to install knitr before stringi and devtools before R6.

But this is just theory right now, this bug is quite hard to reproduce....

Yes, I certainly appreciate that this is a moving target that is hard to isolate. Thanks for investigating!

[jimhester]

I am going to re-open this, as the fix in 800ca64 was not sufficient to fix it.

[IndrajeetPatil]

I am facing exactly the same issue but in the context of GitHub actions on a Windows platform:
https://community.rstudio.com/t/github-actions-failing-on-windows/52150

[jimhester]

I finally tracked this down and was able to consistently reproduce it. I believe that is should be fixed in the general case by 88f302f

[lcolladotor]

Hi,

Sorry to bug you here. We are actively discussing this issue at https://stat.ethz.ch/pipermail/bioc-devel/2020-April/016679.html and earlier messages on Bioc-devel.

I do get the same problem of missing source dependencies (with other packages) using BiocManager::install() instead of remotes::install_deps(), so like Martin Morgan mentioned at https://stat.ethz.ch/pipermail/bioc-devel/2020-April/016676.html, this could be an upstream issue in R. Hence why I'm not opening a new issue here. But given your expertise in the topic, maybe you want to chime in.

Best,
Leo

[tjfarrar]

I think I'm experiencing the same issue with this OSX build of my package.

There are binary versions available but the source versions are later:
                   binary      source needs_compilation
broom               0.5.6       0.7.0             FALSE
vctrs               0.3.1       0.3.2              TRUE
RcppArmadillo 0.9.900.1.0 0.9.900.2.0              TRUE

Because it opts for binary versions of these packages, it installs them in the wrong order which causes an error in lazy loading of broom since it seems to depend indirectly on vctrs.

I presume the problem will resolve itself when the updated binaries are available on CRAN. For now I fixed it by installing these three packages from Github using

r_github_packages:
        - tidymodels/broom
        - RcppCore/RcppArmadillo
        - r-lib/vctrs

[gaborcsardi]

This is a bug in install.packages(), it cannot install a mix of binary and source packages in the right order. We'll work around it in remotes, by working out the order manually. I'll re-open this.