ARROW-5134: [R][CI] Run nightly tests against multiple R versions

Nightly R builds on multiple R versions are now running on https://github.com/ursa-labs/arrow-r-nightly on Travis and Appveyor. https://issues.apache.org/jira/browse/ARROW-6258 facilitated the macOS portion of this; Windows required no changes to apache/arrow to set up. As a side benefit of the Travis/Appveyor setup, binary packages for macOS and Windows are being built and pushed to https://dl.bintray.com/ursalabs/arrow-r, which R users can use as package repository.

This patch fixes one line of R code (offending R change [here](wch/r-source@aeed807
)) to enable passing builds going back to R 3.2. That's 6 total R versions supported with binary builds: 3.2 - 3.6, plus R-devel (to be 3.7) on Windows. It also adds some badges to the R README indicating status of the nightly builds, plus a note pointing intrepid users to the (very unofficial) binaries on Bintray.

Closes #5121 from nealrichardson/nightly-r and squashes the following commits:

ac6b0b0 <Neal Richardson> Don't 'Suggest' pkgdown and roxygen2 and instead suggest that developers install them (they're not needed for package checks and don't enhance the use of the package itself
c8b6722 <Neal Richardson> Advertise the bintray on the R readme
adfc2a9 <Neal Richardson> Add LOCAL_AUTOBREW argument to configure
d781981 <Neal Richardson> Add nightly badges to R README
d0adf22 <Neal Richardson> Revise write_arrow() to work in R 3.4 and before

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>

r/.Rbuildignore

@@ -18,3 +18,5 @@ clang_format.sh
 ^Makefile$
 ^.*\.orig$
 ^.*\.cmd$
+^autobrew$
+^apache-arrow.rb$

r/DESCRIPTION

@@ -41,9 +41,7 @@ Suggests:
     covr,
     hms,
     lubridate,
-    pkgdown,
     rmarkdown,
-    roxygen2,
     testthat,
     tibble,
     vctrs

r/R/write_arrow.R

@@ -71,7 +71,13 @@ write_arrow <- function(x, stream, ...) {
   file_stream <- FileOutputStream(stream)
   on.exit(file_stream$close())
   file_writer <- RecordBatchFileWriter(file_stream, x$schema)
-  on.exit(file_writer$close(), add = TRUE, after = FALSE)
+  on.exit({
+    # Re-set the exit code to close both connections, LIFO
+    file_writer$close()
+    file_stream$close()
+  })
+  # Available on R >= 3.5
+  # on.exit(file_writer$close(), add = TRUE, after = FALSE)
   write_arrow(x, file_writer, ...)
 }
 

r/README.Rmd

@@ -16,7 +16,7 @@ knitr::opts_chunk$set(
 ```
 # arrow
 
-[![cran](https://www.r-pkg.org/badges/version-last-release/arrow)](https://cran.r-project.org/package=arrow) [![conda-forge](https://img.shields.io/conda/vn/conda-forge/r-arrow.svg)](https://anaconda.org/conda-forge/r-arrow)
+[![cran](https://www.r-pkg.org/badges/version-last-release/arrow)](https://cran.r-project.org/package=arrow) [![conda-forge](https://img.shields.io/conda/vn/conda-forge/r-arrow.svg)](https://anaconda.org/conda-forge/r-arrow) [![Nightly macOS Build Status](https://travis-ci.org/ursa-labs/arrow-r-nightly.png?branch=master)](https://travis-ci.org/ursa-labs/arrow-r-nightly) [![Nightly Windows Build Status](https://ci.appveyor.com/api/projects/status/ume8udm5r26u2c9l/branch/master?svg=true)](https://ci.appveyor.com/project/nealrichardson/arrow-r-nightly-yxl55/branch/master) [![codecov](https://codecov.io/gh/ursa-labs/arrow-r-nightly/branch/master/graph/badge.svg)](https://codecov.io/gh/ursa-labs/arrow-r-nightly)
 
 [Apache Arrow](https://arrow.apache.org/) is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication.
 
@@ -62,20 +62,15 @@ as.data.frame(tab)
 
 ## Installing a development version
 
-To use the development version of the R package, you'll need to install it from source, which requires the additional C++ library setup. On macOS, you may install the C++ library using [Homebrew](https://brew.sh/):
+Binary R packages for macOS and Windows are built daily and hosted at https://dl.bintray.com/ursalabs/arrow-r/. To install from there:
 
-```shell
-# For the released version:
-brew install apache-arrow
-# Or for a development version, you can try:
-brew install apache-arrow --HEAD
+```r
+install.packages("arrow", repos="https://dl.bintray.com/ursalabs/arrow-r")
 ```
 
-On Windows, you can download a .zip file with the arrow dependencies from the [rwinlib](https://github.com/rwinlib/arrow/releases) project, and then set the `RWINLIB_LOCAL` environment variable to point to that zip file before installing the `arrow` R package. That project contains released versions of the C++ library; for a development version, Windows users may be able to find a binary by going to the [Apache Arrow project's Appveyor](https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow), selecting an R job from a recent build, and downloading the `build\arrow-*.zip` file from the "Artifacts" tab.
-
-Linux users can get a released version of the library from our PPAs, as described above. If you need a development version of the C++ library, you will likely need to build it from source. See "Development" below.
+These daily package builds are not official Apache releases and are not recommended for production use. They may be useful for testing bug fixes and new features under active development.
 
-Once you have the C++ library, you can install the R package from GitHub using the [`remotes`](https://remotes.r-lib.org/) package. From within an R session,
+Linux users will need to build the Arrow C++ library from source. See "Development" below. Once you have the C++ library, you can install the R package from GitHub using the [`remotes`](https://remotes.r-lib.org/) package. From within an R session,
 
 ```r
 # install.packages("remotes") # Or install "devtools", which includes remotes
@@ -92,6 +87,17 @@ You can specify a particular commit, branch, or [release](https://github.com/apa
 
 ## Developing
 
+Windows and macOS users who wish to contribute to the R package and don't need to alter the Arrow C++ library may be able to obtain a recent version of the library without building from source. On macOS, you may install the C++ library using [Homebrew](https://brew.sh/):
+
+```shell
+# For the released version:
+brew install apache-arrow
+# Or for a development version, you can try:
+brew install apache-arrow --HEAD
+```
+
+On Windows, you can download a .zip file with the arrow dependencies from the [rwinlib](https://github.com/rwinlib/arrow/releases) project, and then set the `RWINLIB_LOCAL` environment variable to point to that zip file before installing the `arrow` R package. That project contains released versions of the C++ library; for a development version, Windows users may be able to find a binary by going to the [Apache Arrow project's Appveyor](https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow), selecting an R job from a recent build, and downloading the `build\arrow-*.zip` file from the "Artifacts" tab.
+
 If you need to alter both the Arrow C++ library and the R package code, or if
 you can't get a binary version of the latest C++ library elsewhere, you'll need
 to build it from source too.
@@ -108,7 +114,7 @@ dependencies, along with additional dev dependencies, from the git checkout:
 
 ```shell
 cd ../../r
-R -e 'install.packages("devtools"); devtools::install_dev_deps()'
+R -e 'install.packages(c("devtools", "roxygen2", "pkgdown")); devtools::install_dev_deps()'
 R CMD INSTALL .
 ```
 
@@ -160,7 +166,7 @@ devtools::load_all() # Load the dev package
 devtools::test(filter="^regexp$") # Run the test suite, optionally filtering file names
 devtools::document() # Update roxygen documentation
 rmarkdown::render("README.Rmd") # To rebuild README.md
-pkgdown::build_site(run_dont_run=TRUE) # To preview the documentation website
+pkgdown::build_site() # To preview the documentation website
 devtools::check() # All package checks; see also below
 ```
 

r/README.md

@@ -5,6 +5,11 @@
 
 [![cran](https://www.r-pkg.org/badges/version-last-release/arrow)](https://cran.r-project.org/package=arrow)
 [![conda-forge](https://img.shields.io/conda/vn/conda-forge/r-arrow.svg)](https://anaconda.org/conda-forge/r-arrow)
+[![Nightly macOS Build
+Status](https://travis-ci.org/ursa-labs/arrow-r-nightly.png?branch=master)](https://travis-ci.org/ursa-labs/arrow-r-nightly)
+[![Nightly Windows Build
+Status](https://ci.appveyor.com/api/projects/status/ume8udm5r26u2c9l/branch/master?svg=true)](https://ci.appveyor.com/project/nealrichardson/arrow-r-nightly-yxl55/branch/master)
+[![codecov](https://codecov.io/gh/ursa-labs/arrow-r-nightly/branch/master/graph/badge.svg)](https://codecov.io/gh/ursa-labs/arrow-r-nightly)
 
 [Apache Arrow](https://arrow.apache.org/) is a cross-language
 development platform for in-memory data. It specifies a standardized
@@ -88,36 +93,22 @@ as.data.frame(tab)
 
 ## Installing a development version
 
-To use the development version of the R package, you’ll need to install
-it from source, which requires the additional C++ library setup. On
-macOS, you may install the C++ library using
-[Homebrew](https://brew.sh/):
+Binary R packages for macOS and Windows are built daily and hosted at
+<https://dl.bintray.com/ursalabs/arrow-r/>. To install from there:
 
-``` shell
-# For the released version:
-brew install apache-arrow
-# Or for a development version, you can try:
-brew install apache-arrow --HEAD
+``` r
+install.packages("arrow", repos="https://dl.bintray.com/ursalabs/arrow-r")
 ```
 
-On Windows, you can download a .zip file with the arrow dependencies
-from the [rwinlib](https://github.com/rwinlib/arrow/releases) project,
-and then set the `RWINLIB_LOCAL` environment variable to point to that
-zip file before installing the `arrow` R package. That project contains
-released versions of the C++ library; for a development version, Windows
-users may be able to find a binary by going to the [Apache Arrow
-project’s
-Appveyor](https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow),
-selecting an R job from a recent build, and downloading the
-`build\arrow-*.zip` file from the “Artifacts” tab.
-
-Linux users can get a released version of the library from our PPAs, as
-described above. If you need a development version of the C++ library,
-you will likely need to build it from source. See “Development” below.
+These daily package builds are not official Apache releases and are not
+recommended for production use. They may be useful for testing bug fixes
+and new features under active development.
 
-Once you have the C++ library, you can install the R package from GitHub
-using the [`remotes`](https://remotes.r-lib.org/) package. From within
-an R session,
+Linux users will need to build the Arrow C++ library from source. See
+“Development” below. Once you have the C++ library, you can install
+the R package from GitHub using the
+[`remotes`](https://remotes.r-lib.org/) package. From within an R
+session,
 
 ``` r
 # install.packages("remotes") # Or install "devtools", which includes remotes
@@ -138,6 +129,29 @@ installed.
 
 ## Developing
 
+Windows and macOS users who wish to contribute to the R package and
+don’t need to alter the Arrow C++ library may be able to obtain a
+recent version of the library without building from source. On macOS,
+you may install the C++ library using [Homebrew](https://brew.sh/):
+
+``` shell
+# For the released version:
+brew install apache-arrow
+# Or for a development version, you can try:
+brew install apache-arrow --HEAD
+```
+
+On Windows, you can download a .zip file with the arrow dependencies
+from the [rwinlib](https://github.com/rwinlib/arrow/releases) project,
+and then set the `RWINLIB_LOCAL` environment variable to point to that
+zip file before installing the `arrow` R package. That project contains
+released versions of the C++ library; for a development version, Windows
+users may be able to find a binary by going to the [Apache Arrow
+project’s
+Appveyor](https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow),
+selecting an R job from a recent build, and downloading the
+`build\arrow-*.zip` file from the “Artifacts” tab.
+
 If you need to alter both the Arrow C++ library and the R package code,
 or if you can’t get a binary version of the latest C++ library
 elsewhere, you’ll need to build it from source too.
@@ -158,7 +172,7 @@ checkout:
 
 ``` shell
 cd ../../r
-R -e 'install.packages("devtools"); devtools::install_dev_deps()'
+R -e 'install.packages(c("devtools", "roxygen2", "pkgdown")); devtools::install_dev_deps()'
 R CMD INSTALL .
 ```
 
@@ -210,7 +224,7 @@ devtools::load_all() # Load the dev package
 devtools::test(filter="^regexp$") # Run the test suite, optionally filtering file names
 devtools::document() # Update roxygen documentation
 rmarkdown::render("README.Rmd") # To rebuild README.md
-pkgdown::build_site(run_dont_run=TRUE) # To preview the documentation website
+pkgdown::build_site() # To preview the documentation website
 devtools::check() # All package checks; see also below
 ```
 

r/configure

@@ -39,6 +39,14 @@ if [ "$ARROW_R_DEV" == "TRUE" ]; then
   ${R_HOME}/bin/Rscript data-raw/codegen.R
 fi
 
+if [ "$LOCAL_AUTOBREW" == "TRUE" ]; then
+  # LOCAL_AUTOBREW means use the formula in tools/
+  # FORCE_AUTOBREW without LOCAL_AUTOBREW means to pull from jeroen.github.io
+  cp tools/autobrew .
+  cp tools/apache-arrow.rb .
+  FORCE_AUTOBREW="TRUE"
+fi
+
 # Note that cflags may be empty in case of success
 if [ "$INCLUDE_DIR" ] || [ "$LIB_DIR" ]; then
   echo "Found INCLUDE_DIR and/or LIB_DIR!"