[R][Linux] Make R Binary Install Friendlier

[asfimport]

When R install tries to run a binary install, it looks for an exact match on the binary version, say "0.17.1.zip" from https://dl.bintray.com/ursalabs/arrow-r/libarrow/bin/ubuntu-18.04/.

The problem is that even though "0.17.1" is pushed to CRAN as an official release, there is a time period (like right now) where bintray does not have an official binary build, just a date stamped build:

 

arrow-0.17.0.20200516.zip
arrow-0.17.0.20200517.zip
arrow-0.17.0.20200518.zip
arrow-0.17.0.zip
arrow-0.17.1.20200517.zip
arrow-0.17.1.20200519.zip
arrow-0.17.1.20200520.zip

 

I'd like to suggest adding a new environment variable trigger that would allow for the scanning of bintray for a recent timestamped version, if the specific release number is not present.

I'd like to suggest enhancing the linux code:

arrow/r/tools/linuxlibs.R

Lines 39 to 47 in 02f7be3

	try_download <- function(from_url, to_file) {
	try(
	suppressWarnings(
	download.file(from_url, to_file, quiet = quietly)
	),
	silent = quietly
	)
	file.exists(to_file)
	}

with scanning functionality:

try_download <- function(from_url, to_file, scan_dates = FALSE) {
    try(
            suppressWarnings(
                  download.file(from_url, to_file, quiet = quietly)
            ),
            silent = quietly
      )
{color:#c1c7d0}      if (!file.exists(to_file)) {{color}
{color:#c1c7d0}                if (scan_dates) {{color}
{color:#0747a6}                      scan_dates <- format(Sys.Date()-(0:10),"%Y%m%d"){color}
{color:#0747a6}                       for (scan_date in scan_dates) {{color}
{color:#0747a6}                              base_url <- tools::file_path_sans_ext(from_url){color}
{color:#0747a6}                              ext <- tools::file_ext(from_url){color}
{color:#0747a6}                              scan_url <- sprintf("%s.%s.%s", base_url, scan_date, ext){color}
{color:#0747a6}                              if (try_download(from_url = scan_url, to_file, scan_dates = FALSE)) {{color}
{color:#0747a6}                                      return(TRUE){color}
{color:#0747a6}                              }{color}
{color:#0747a6}                        }{color}
{color:#0747a6}                 }{color}
{color:#0747a6}                 return(FALSE){color}

{color:#c1c7d0}   } else {{color}
{color:#c1c7d0}        return(TRUE){color}
{color:#c1c7d0}   }{color}
}

And then augment the calling function:

arrow/r/tools/linuxlibs.R

Line 55 in 02f7be3

if (try_download(binary_url, libfile)) {

 

with:

binary_scan_ok <- !identical(tolower(Sys.getenv("LIBARROW_BINARY_SCAN", "false")), "false")
if (try_download(binary_url, libfile, scan_dates = binary_scan_ok)) {

 

This would allow automated builds to set the scan option, and then find and install the most recent daily build in lieu of an official binary build being in place.

 

 

 

 

 

 

 

 

 

 

 

 

 

Environment: Linux (Ubuntu)
Reporter: Brian Schultheiss
Assignee: Neal Richardson / @nealrichardson

_{Note: This issue was originally created as ARROW-8880. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
It's an interesting idea, though it wouldn't have actually worked in the case of 0.17.1 because the patch release was not built off of master, so arrow-0.17.0.20200518.zip C++ actually would have failed with 0.17.1 R due to ABI incompatibility.

The real problem here is that the 0.17.1 binaries weren't published correctly, it turns out. So I'll fix that. Thanks for the alert.

If you want to try out how your proposal would work, you're welcome to make a PR and we can discuss there. I could see it being interesting for the nightly builds, as they can occasionally fail to build for spurious reasons. I wouldn't want to do it for official releases though--those should match exactly, and should exist.

[asfimport]

Neal Richardson / @nealrichardson:
C++ binaries for 0.17.1 now exist in the right place. Thanks again for the heads up about that.

[asfimport]

Brian Schultheiss:
Thanks, verified that the 0.17.1.zip is now found during install of CRAN package.

 

Also, thanks for the clarification on the timestamped files as being nightly's and not source compatible with the CRAN release. With this insight, I am happy for you to close this request as my thought process was wrong on the compatibility with CRAN releases.