ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor

This is very much a work-in-progress for running the R package tests on Appveyor. It's an attempt to bring in the "standard" R Appveyor build tooling (https://github.com/krlmlr/r-appveyor, Apache 2.0 licensed) into Arrow's configuration.

I currently have the C++ library building and installing, and R and all R package dependencies installing, and R checks beginning, but they fail. This is because https://github.com/apache/arrow/blob/master/r/tools/winlibs.R is configured only to build with a released [sic] version of the arrow C++ binaries pre-built and hosted on rwinlib. Two immediate to-dos are:

1. get the package to install and check without libarrow at all (as apache/arrow#4471 now allows)
2. get the package to find the libarrow we built locally from source

After that, I can reverse-engineer what appveyor does and document it for any Windows developers (https://issues.apache.org/jira/browse/ARROW-3758).

There's obviously some mess I've made of appveyor.yml that I'll clean up before I'm done, and I'll do my best to document and refactor some of the non-obvious features of the setup that I've discovered (e.g. that `ci/appveyor-cpp-build-mingw.bat` also installs GLib and Ruby and runs their test suites too) while I'm at it so that the next person might be able to pick things up faster.

cc @romainfrancois @javierluraschi

Author: Neal Richardson <neal.p.richardson@gmail.com>

Closes #4538 from nealrichardson/r-appveyor and squashes the following commits:

6d6a74e31 <Neal Richardson> PR feedback
84eef1b2d <Neal Richardson> Merge upstream/master
328bc376f <Neal Richardson> Merge remote-tracking branch 'upstream/master' into r-appveyor
53c3a2084 <Neal Richardson> Makevars.win is dynamically generated now, no need to tweak it in the release script
567f04142 <Neal Richardson> Cleanups
f4cf5c3dd <Neal Richardson> Some appveyor.yml cleanup
19b60df07 <Neal Richardson> Enable R package to install on windows even if rwinlib isn't found
fb199110a <Neal Richardson> Try this name hack
15c0342d1 <Neal Richardson> Try building first, see where that goes
18803c353 <Neal Richardson> cd this way
b6f6c8de7 <Neal Richardson> No popd?
1ba4839e2 <Neal Richardson> let the debugging begin
d5e8f4ed4 <Neal Richardson> Bootstrap within the package dir
f7327ed23 <Neal Richardson> Find shell script up a level
8c3e898f8 <Neal Richardson> nm we already fast finish
39236f3f7 <Neal Richardson> Put mine first and fast-finish
94c9a42df <Neal Richardson> Debug with fewer jobs in the matrix
af1852e79 <Neal Richardson> Throw this at appveyor and see how far off we are

appveyor.yml

@@ -19,7 +19,7 @@
 os: Visual Studio 2015
 
 only_commits:
-  # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
+  # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
   files:
     - appveyor.yml
     - c_glib/
@@ -29,6 +29,7 @@ only_commits:
     - format/
     - go/
     - python/
+    - r/
     - ruby/
     - rust/
 
@@ -102,6 +103,8 @@ environment:
       GO111MODULE: on
       GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
       PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
+    - JOB: "R without libarrow"
+      USE_CLCACHE: false
 
   MSVC_DEFAULT_OPTIONS: ON
   APPVEYOR_SAVE_CACHE_ON_ERROR: true
@@ -119,6 +122,57 @@ before_build:
 build_script:
   - call ci\appveyor-build.bat
 
+for:
+-
+  matrix:
+    only:
+      - JOB: "R without libarrow"
+  init:
+    ps: |
+          $ErrorActionPreference = "Stop"
+          Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
+          Import-Module '..\appveyor-tool.ps1'
+
+  before_build:
+    - cd r
+    - ps: Bootstrap
+    - cd %APPVEYOR_BUILD_FOLDER%
+
+  build_script:
+    # TODO: for the version of the R tests that builds, probably can't use this
+    # ci script because we need to build with the Rtools toolchain
+    # - call ci\appveyor-build.bat
+    - pushd r
+    - travis-tool.sh install_deps
+
+  test_script:
+    - travis-tool.sh run_tests
+
+  on_failure:
+    - travis-tool.sh dump_logs
+
+  environment:
+    NOT_CRAN: true
+
+  artifacts:
+    - path: 'r\*.Rcheck\**\*.log'
+      name: Logs
+
+    - path: 'r\*.Rcheck\**\*.out'
+      name: Logs
+
+    - path: 'r\*.Rcheck\**\*.fail'
+      name: Logs
+
+    - path: 'r\*.Rcheck\**\*.Rout'
+      name: Logs
+
+    - path: 'r\*_*.tar.gz'
+      name: Bits
+
+    - path: 'r\*_*.zip'
+      name: Bits
+
 # Disable test discovery
 test: off
 

ci/appveyor-filter-changes.bat

@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
         echo ===
         appveyor exit
     )
+) else if "%JOB:~,2%" == "R " (
+    if "%ARROW_CI_R_AFFECTED%" == "0" (
+        echo ===
+        echo === No C++ or R changes, exiting job
+        echo ===
+        appveyor exit
+    )
 ) else (
     if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
         echo ===

dev/release/00-prepare-test.rb

@@ -130,13 +130,6 @@ def test_update_version_pre_tag
                         "+Version: #{@release_version}"],
                      ],
                    },
-                   {
-                     path: "r/src/Makevars.win",
-                     hunks: [
-                       ["-VERSION = #{@previous_version}.9000",
-                        "+VERSION = #{@release_version}"],
-                     ],
-                   },
                    {
                      path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
                      hunks: [
@@ -279,13 +272,6 @@ def test_update_version_post_tag
                         "+Version: #{@release_version}.9000"],
                      ],
                    },
-                   {
-                     path: "r/src/Makevars.win",
-                     hunks: [
-                       ["-VERSION = #{@release_version}",
-                        "+VERSION = #{@release_version}.9000"],
-                     ],
-                   },
                    {
                      path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
                      hunks: [

dev/release/00-prepare.sh

@@ -101,14 +101,6 @@ update_versions() {
   git add DESCRIPTION
   cd -
 
-  cd "${SOURCE_DIR}/../../r/src"
-  sed -i.bak -E -e \
-    "s/^VERSION = .+/VERSION = ${r_version}/" \
-    Makevars.win
-  rm -f Makevars.win.bak
-  git add Makevars.win
-  cd -
-
   cd "${SOURCE_DIR}/../../ruby"
   sed -i.bak -E -e \
     "s/^  VERSION = \".+\"/  VERSION = \"${version}\"/g" \