Travis CI .yml Update for Build Matrices

[coatless]

The CI testing via Travis needs a gentle update to the latest standards (c.f. Jim Hester's post on the RStudio blog) per:

+echo '\033[0;31m
r-travis is DEPRECATED!
Please see https://docs.travis-ci.com/user/languages/r/ for info, or
https://github.com/craigcitro/r-travis/wiki/Porting-to-native-R-support-in-Travis
for information on porting to native R support in Travis.\033[0m'
\033[0;31m

https://travis-ci.org/RcppCore/Rcpp#L657-L669

During this update, I think it would be good to also add in testing on:

1. Different R versions.
2. Operating system
   • Nice write up on test matrices by Jeroen Ooms.

Ideal use case: RMySQL's .travis.yml

Travis-CI documentation for R: https://docs.travis-ci.com/user/languages/r

[eddelbuettel]

I am sorry, but that is probably going to be a no.

I was one the earlier three people building the first gen of Travis support for R, and I still use it everywhere, and prefer it. I am not opposed to mucking with the new container-based setup for mini projects, but I am very much opposed to using for Rcpp at this point (and possibly future points).

Lastly, there is no urgency. They have been spewing those messages for a very long time.

[coatless]

@eddelbuettel Nothing to be sorry about nor is there any rush to implement. Just wanted to bring it up and provide possible implementations due to the NORET bit in #512.

Feel free to close this out.

[kevinushey]

If updating Travis means catching errors that are only seen on certain mixtures or R / gcc / otherwise, I think it's worth doing. What are the downsides to updating?

[eddelbuettel]

@coatless Point taken. Had we had 'old version' tests we might not have been bitten by NORET.

@kevinushey In theory having more test setups is of course a good idea. In practice they tend to randomly break, and having more just means there is more chance of breakage.

A pragmatic compromise is IMHO ideal. What we have now mostly works, completes in reasonable time, and catches most things CRAN is after (with current versions) whereas I have seen a number of other repos with partially borked CI setups. If they worked more reliably eg appveyor could help with Windows and the new matrix may help with OS X. So far I am still not convinced it is there. Of course I may well be wrong here and would be happy to be convinced otherwise. The last rushed conversion was by Jeroen himself, not coordinated within the repo (ie master, not branch, no prior discussion) and I reverted after four rounds all of which ran way longer than before and did not complete. That moves the needle the wrong way.

[coatless]

Per a little test with gmwm, it seems as if the build matrices with multiple r versions are not well supported. In the sense that, for each OS specified the build seemed to not be expanded outside of the array, e.g. R: oldrel,release,devel.

Though, the following does provide dual coverage of multiple OS + R versions:

# Use the R testing scheme
language: r

# Test on multiple versions (note requirement of R > 3.0)
r:
  - oldrel
  - release
  - devel

# Enable package cache
cache: packages

# Disable sudo to ensure container infrasturcture
sudo: false

# To CRAN, warnings **are** errors.
warnings_are_errors: true

# Build on OS
os:
  - linux
  - osx

See travis-ci/travis-ci#6353 for feature clarification

[eddelbuettel]

Gabor also has something along those lines as a sub-project of R-hub.

And Gitlab allows you to bring your own Docker container, which is something somewhat interesting too.

[coatless]

Aye, I've been monitoring the r-hub project for a while, it looked like it stalled out around December as the monthly reports stopped. Glad to see it picking up steam. Lots of potential if it meets the initial spec.

Though, right now, it seems as if they only have a manual package builder active.

[eddelbuettel]

I had to dig a little as there are so many sub-projects by Gabor -- what I had in mind was r-builder which appears to offer multiple R versions for tests.

[coatless]

@eddelbuettel

Seems like everything Gabor offers via r-builder was baked into Travis-CI support by Jim Hester. The only feature difference is the addition of Semaphore as an alternative test platform. Neither provides build support for windows.

Per a mistake I made (addressed in travis-ci/travis-ci/issues/6353), the test matrix could be deployed with multiple R versions across multiple OSes in the form of:

Number of R versions * Number of OSes = Total Number of Matrix Entries

For total coverage offered by travis, the .travis.yml would then need:

3 * 5 = 15 (this covers all the different options available on travis!)

Sample matrix for supported linux distros (Precise & Trusty ubuntu)

# Test package on multiple platforms
matrix:
  include:
    - os: linux
      dist: precise
      sudo: false
      r: oldrel
    - os: linux
      dist: precise
      sudo: false
      r: release
    - os: linux
      dist: precise
      sudo: false
      r: devel
    - os: linux
      dist: trusty
      sudo: true
      r: oldrel
    - os: linux
      dist: trusty
      sudo: true
      r: release
    - os: linux
      dist: trusty
      sudo: true
      r: devel

[eddelbuettel]

Maybe we can have this much easier / simpler. I just had success with

$ docker run --rm -ti r-base:3.2.5 R -e 'install.packages("Rcpp")'

whereas I got the (now expected) immediate fail with this

$ docker run --rm -ti r-base:3.1.3 R -e 'install.packages("Rcpp")'

We have a drat repo so relying on install.packages() is not really a constraint. And for those who cannot work without devtools, I guess they could always pre-load it first (or one of the lighter-weight direct-from-git installers) as we can sequence R commands in the argument to -e ....

[coatless]

Per dcdillon/RcppHoney#7, PMassicotte/gtrendsR#105, and slack, the resolution is:

keep travis 1.0 via eddelbuettel/r-travis (albeit with the deprecation message removed)

There is _no_ rush nor any immediate need to fix something that isn't broken.

Thus, I'm going to shift my focus to knocking off the remainder of issues I've identified in the Rcpp issue tracker and start working on a feature development roadmap to knock off the more time intensive features. (e.g. crossprod() and improved sparse matrix wrapping support for RcppArmadillo !!)

[eddelbuettel]

Yay.

Given that I don't have OS X, and that we have no 'mac-builder' in the sense of the 'win-builder' box it would actually be nice to have an option. Maybe we drop another, or extended, or different, .travis.yml into a branch one in a while? Or use some other service that offers OS X? I just really, really, really don't want to make the main .travis.yml more fragile or slower. And all attempts at build matrices that I have seen do both.

[coatless]

Looks like eddelbuettel/r-travis@d1cfe4e is live.

[eddelbuettel]

Yes, was about to poke you but then pulled that back. Short write-up / cover here. Too much silly heat in that discussion already and too little signal -- but I am slowly converting packages. It is a little quieter than the original, and still let's me sudo, and apt-get from non-blessed sources and whatnot. I get to Rcpp in a few days or so and will just do a quick PR for review.