Native R support in Travis and containers

[craigcitro]

hi all -- opening a discussion ticket, hoping folks will pipe in.

@hadley @eddelbuettel @jimhester @cbarraford -- cc away if you know other interested folks.

currently, we're all hitting some pain because the native R support in travis makes use of sudo, which means we can't use their container-based infrastructure.

Problem

We need to install a number of prereqs (notably R and LaTeX), which takes time. On top of that, we'd like to save as much time as possible installing user packages, which we do by using marutter's PPA. All of these require sudo; we could (hopefully?) move the first two into the pre-container step, but the third is still a real problem.

A second constraint is that we want to support multiple R versions (including R-devel).

Option 1: Use our own containers

I think this is what we all want, but Travis doesn't yet support this. So we wait.

Option 2: Build our own image

Here we'd add our own Chef cookbook to travis-ci/travis-cookbooks that installed R and LaTeX, along with whitelisting a large number of packages from the marutter PPA for sudo-less install. Then we'd fall back to source installs for non-whitelisted binary packages, and be in OK shape.

Our biggest constraint here is that we don't have any feedback from the Travis folks about whether or not they'll actually accept a PR with a new cookbook.

Option 3: Whitelist a large number of packages

We could try adding a large number of R-related packages to their apt whitelist for sudo-less install. @jimhester seems to be starting down this road?

---

other thoughts? comments?

[jimhester]

FWIW it looks like pandoc and texlive are already whitelisted packages on ubuntu-precise.

I am also unclear if once you have a whitelisted source if you need to also whitelist the individual packages (I would hope not).

[eddelbuettel]

I got so bloody disenchanted with the new setup that I decided to move my build-depends to my own PPA on launchpad. So I very much need sudo to install from there.

I may just sit on the sidelines til all you good folks have sorted out what to do with the new containers.

[rmflight]

just to clarify, I thought it was possible to install R packages on Ubuntu in a user directory without using sudo??

[craigcitro]

@rmflight for source packages, no sudo required. binary packages need apt-get.

[rmflight]

Ah, thanks @craigcitro, my bad.

[yihui]

I'd say don't worry about marutter's PPA at the moment. The top priority is to whitelist the CRAN Ubuntu repo so we can install r-base-dev without sudo. Once we get there, I think we are 80% done. The rest 20% is more difficult and has less gain.

The marutter's PPA will become less important with Travis' caches: one can install packages to the home directory (e.g. ~/R/somewhere), and cache that directory. That will save a lot of time when the R packages need to be compiled. It will certainly be a problem if these packages have external system dependencies, but I think this problem has to be sorted out case by case since Travis will have to whitelist apt packages case by case. There isn't much you can do for this problem.

[eddelbuettel]

I am partial to own my PPA too as it is so damn convenient and easy to create (especially when you, as a I do, already have Debian package sources around -- essentially just tar up and push to launchpad). With that I will probably always need sudo 😞

But yes, the caching is quite impressive as I saw today when having Travis CI run over a non-R-but-C++ project which build itself CMake and fetched g++ and clang. The second pass was very quick.

[craigcitro]

@yihui based on user feedback, at least in the short term, getting R without sudo is far from 80% of the way there, unfortunately. every single question outside "why is this broken" has been "how do i avoid building my deps every time".

i agree that caching might be a nice way forward here with a longer view. but i think it'll take some work to make it smooth.

[yihui]

@craigcitro Fair enough. Let me change it to "80% done for Yihui, instead of most other users" :) I know reasonably well about how to use Ubuntu, Travis, and build R packages, so I can take care of the rest of stuff by myself as long as I'm able to install R in the container. This may not apply to other users.

Anyway, I guess we have to get the CRAN repo and r-base-dev whitelisted first before we talk about anything else, right? I will truly appreciate it if someone can do this. 🙇

[jimhester]

@yihui I opened travis-ci/travis-ci#3729 (comment) which should whitelist what you are looking for once someone from travis takes a look at it.

[yihui]

@jimhester Awesome. Thanks so much! I will up-vote there (if it helps at all).

[mllg]

The whitelists are set, it is possible to install r-base and r-base-dev w/o sudo.

Here is a minimal example for a container-based build: https://github.com/mllg/travis-test

[eddelbuettel]

Looks interesting, thanks.

It does the caching so subsequent runs on unchanged dependencies are fast?

Now I need a version without devtools :) Any chance you could whitelist littler which is also in Debian/Ubuntu, on the CRAN mirrors as backport, and heavily used in our Rocker containers for R with Docker?

[mllg]

I don't work for Travis -- but as far as I can tell littler is already whitelisted (see https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise#L7217).

The directory containing the R user library is cached, so yes, the install process is pretty fast in subsequent runs.

[eddelbuettel]

Nice.

Now do you know which reposititory they are hitting? Because if it ain't the CRAN backport then we're stuck with the versions originally in precise -- and may as well mash up stone tablets for that matter.

[mllg]

They use the apt sources listed in this json: https://github.com/travis-ci/apt-source-whitelist/blob/master/ubuntu.json.
So "r-packages-precise" points to the Rstudio mirror which should provide backports (https://cran.rstudio.com/bin/linux/debian/#backports-on-cran).

[eddelbuettel]

Thanks so much, Michel -- very helpful.