Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with… #3079

bmcgough · 2016-05-19T16:20:18Z

… required patches for icc/icpc

This is my first (of hopefully many) easyconfig PRs. Please let me know if I've done this in the best way, and how I can improve future PRs.

… required patches for icc/icpc

boegel · 2016-05-19T17:34:58Z

@bmcgough Thanks a lot, looking forward to your many contributions in the future!

The PR looks good at first sight, especially since you went through the trouble of updating the versions of the R packages too, fantastic!
You're really taking quite a bit of load away from several people with this, including myself. ;-)

The first thing you should try to do is make Travis (which runs the unit tests on the whole easyconfigs collection) happy.
It currently complains with this (see the Details link):

EasyBuildError: 'Irresolvable dependencies encountered: GDAL/2.1.0-foss-2016a, GDAL/2.1.0-intel-2016a'

So, basically, you're missing easyconfigs for GDAL 2.1.0 in this PR.
If they're available somewhere else, please provide a pointer to that PR.

A next thing you could do is submit a test report, if you're willing to look into that.
The documentation on this is a bit hidden right now, we (still) need to port it to http://easybuild.readthedocs.io:

see https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#automated-testing-of-easyconfigs-pull-requests
you may want to use eb --install-github-token rather than manually install your GitHub token, since that'll validate your token for use as well
run eb --check-github to see whether you've set things up correctly for uploading test reports

I'll take a closer look now at things, and may include some inline remarks/questions too.
Don't hesitate to let us know if something is not clear!

boegel · 2016-05-19T17:36:18Z

easybuild/easyconfigs/r/R/R-3.3.0-foss-2016a.eb

+
+preconfigopts = 'BLAS_LIBS="$LIBBLAS" LAPACK_LIBS="$LIBLAPACK"'
+configopts = "--with-lapack --with-blas --with-pic --enable-threads --with-x=no --enable-R-shlib --with-tcltk "
+configopts += '--with-tcl-config="$EBROOTTCL"/lib/tclConfig.sh --with-tk-config="$EBROOTTK"/lib/tkConfig.sh '


this is new, we don't seem to have this in other R easyconfigs

why did you (have to) add it?

is it related to #1863 at all?

(Assuming you mean the TCL/TK configopts?)

The CRAN package tkrplot couldn't find the config files until I added this.

I am a bit confused about TCL/TK in R - is it even worth configuring with -noX? What does Tk do without X?

@boegel I also do the same for Tcl/Tk. See https://github.com/hpcugent/easybuild-easyconfigs/pull/3069/files#diff-b782c64de427c0b1534a81b8f2bd6789R21

For me enabling X is mandatory. Bioinformaticians need it to plot images

Actually, the -no-X11 is kind of a misnomer, see also #2261...

@pescobar:

For me enabling X is mandatory. Bioinformaticians need it to plot images

+1

More broadly, the more that get enabled here the better:

> capabilities() jpeg png tiff tcltk X11 aqua TRUE FALSE TRUE TRUE FALSE FALSE http/ftp sockets libxml fifo cledit iconv TRUE TRUE TRUE TRUE TRUE TRUE NLS profmem cairo ICU long.double libcurl TRUE FALSE TRUE TRUE TRUE TRUE

The default (non-versionsuffixed) R easyconfigs should be as fat as possible, probably, but that may mean pulling in a whole X11 stack if we want to be independent of the OS, cfr. #2624

bmcgough · 2016-05-19T19:07:29Z

I am just now working with travis-ci on another repo, so once I have that figured out, I will work toward getting the testing working.

We have a script that I will add to the framework repo shortly called ezupdate.py (should we change the name?). It works for R and Python packages and uses the CRAN/PYPI APIs to take an existing package list, update each package to the newest version, and include any new dependent packages. We've tinkered with it for a while, and it seems to work pretty well at this point. The fiddly stuff comes from parsing the EB, which I assume we would not have to do if we integrated into EB fully.

So I used that script to update the packages, but forgot that it would not update dependencies, which is why java was not updated. Also, probably why openssl was pulled in w/o me knowing it.

It is our goal to keep R and Python (and maybe Perl in the future) up to date on every release - within a few days of release. We have that goal in-house here, and by using EB, we hope to benefit others with that work.

I have the GDAL packages and will include them - I guess with a new PR that I will reference here? Also, I have R-bundle-Bioconductor-3.3 easyconfigs to commit, but I thought I would wait until this PR is resolved.

pescobar · 2016-05-19T19:47:54Z

@bmcgough maybe you are interested in this. I create all my R easyconfigs using it

https://github.com/fgeorgatos/easybuild.experimental/tree/master/users/pneerincx

The procedure I follow to update my R installation to a newer version is:

Load current R module. In current R version run this to fetch list of installed libs:

    save.pkg.list <- installed.packages()[is.na(installed.packages()[ , "Priority"]), 1]
    save(save.pkg.list, file="pkglist.Rdata")

I install a easyconfig for the new R version without any R library (the bare version) to my home folder. Load new R module. In new R version in my $HOME I run this to clone the libraries and upgrade to latest versions of the libs:

    source("http://bioconductor.org/biocLite.R")
    chooseCRANmirror()
    chooseBioCmirror()
    load("pkglist.Rdata")
    biocLite(save.pkg.list)

If I need any extra R library I use biocLite() to install it and then I use the "EasyConfig generator for R" to generate the easyconfig

boegel · 2016-05-19T20:08:09Z

@bmcgough Since the number of easyconfigs is likely going to remain limited (and easy to review via eb --review-pr), it's fine to add additional easyconfig to this PR rather than via new PRs.

That being said, either way is fine by me. A simple PR with newer versions of GDAL should b easy to get merged, and we can make Travis retest this PR on top of that quite easily.

Contributing back the update script would be very useful. I'll leave the renaming up to you (maybe refer to 'extensions' in some way), but we can also look into integrating it into the EasyBuild framework itself (I can help out there, since this sort of functionality is very relevant for us too).

Thanks a lot for engaging yourself in keeping the easyconfigs with extensions up to date, that would be very valuable to a lot of people indeed! We (at HPC-UGent) tend to do it mostly by user request.

The Bioconductor easyconfig should definitely be a separate PR; feel free to already open a PR for that (that way it can already go through review and maybe even testing), just mention in the description that it depends on this PR by mentioning the PR ID.

boegel · 2016-05-19T20:10:44Z

easybuild/easyconfigs/r/R/R-3.3.0-foss-2016a.eb

+    ('libjpeg-turbo', '1.4.2'),  # for plottting in R
+    ('Java', '1.8.0_72', '', True),  # Java bindings are built if Java is found, might as well provide it
+    ('Tcl', '8.6.4'),  # for tcltk
+    ('Tk', '8.6.4', '-no-X11'),  # for tcltk


there's a Tcl/Tk 8.6.5, do we want to jump to that too?

bmcgough · 2016-05-19T23:57:56Z

Ah, such fun.

cURL 7.47.0 is used by netCDF-4.4.0. There is no new version of netCDF, so if I move to cURL-7.49.0 I'll have to use a versionsuffix. Is it worth upgrading cURL?

boegel · 2016-05-20T06:45:22Z

@bmcgough in that case, stick to cURL 7.47.0, it's probably not worth the trouble

verdurin · 2016-05-21T09:12:17Z

@bmcgough - the update script would indeed be very useful.

…sPer to GDAL as it was pulling from OS and those easyconfigs already existed

boegel · 2016-06-03T19:08:01Z

@bmcgough ping on both this PR and the script for updating extension versions?

bmcgough · 2016-06-03T21:24:16Z

One of my colleagues will submit the extensions update script. It is being re-worked a bit to interface with pipy via XML.

As for the R easyconfigs, I am lost in dependency hell trying to get support enabled for cairo and the rest of the graphics options. I think I am near done, but am finding lots of libraries pulled in from our build host's OS (not a clean build host I'm sorry to say). I was going to try running travis on our fork of the easyconfigs repo in a minute to see if it is able to build these R packages. Any advice on doing that? I see there is a travis.yml file in the repo, but I imagine you have some config in the travis UI (ENV variables)?

boegel · 2016-06-03T21:32:49Z

@bmcgough Travis will not actually build the easyconfigs, it will just run the unit tests, basically meaning parsing easyconfig files, checking whether all dependencies & patches are there, checking whether there are no version conflicts between dependencies, etc.

I'm happy to give it a full build test on my end and submit a test report, but then you'll need to update the PR to have all required easyconfigs...

boegel · 2016-06-03T21:42:10Z

@bmcgough oh boy, this is getting rather big... :)

bmcgough · 2016-06-03T22:22:33Z

Sorry, didn't intend to push all the way up. These are testing in travis now in our fork. I have been working on R-3.3.0-foss-2016a-bare.eb so I was unprepared for the full packages and intel version dependencies, thus the additional pushes. Sorry.

boegel · 2016-06-03T23:15:21Z

Test report by @boegel
FAILED
Build succeeded for 17 out of 23 (23 easyconfigs in this PR)
node2542.golett.os - Linux centos linux 7.2.1511, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/9b6e674792e7b976930430799ab42269 for a full test report.

hajgato · 2016-06-08T10:40:58Z

just one little remark - if we update X component frequently (and not only by toolchain version), then we will end up infinite amount of X packages. This is why we have to solve the issue what I started in #2579 (to make one install for X headers, and X libs (not yet in PR)

boegel · 2016-06-08T20:53:15Z

@bmcgough seems like you've introduced conflicts with the current develop branch, you'll need to resolve those first... Let me know if you need help with that.

bmcgough · 2016-06-08T21:58:53Z

I'm not sure what I can do to help with the X headers/libs issue - if there is something specific I should do in this PR, let me know.

As for getting out of sync with develop, I can find libXft-2.3.2-intel-2016a-fontconfig-2.11.95.eb and netCDF-4.4.0-foss-2016a.eb, but I'm not sure I'm looking for the diffs correctly.

I should have the dependencies worked out for R-3.3 in my next push, which will be within the next 24 hours.

bedroge · 2016-06-27T09:25:24Z

@bmcgough did you perhaps make any progress with this? I am also looking into the installation of R 3.3 (3.3.1, released last week) and it would be nice if I could somehow (re)use your easyconfigs.

boegel · 2016-07-07T14:08:35Z

@bmcgough ping on this? I'm very keen on including easyconfigs for the latest R 3.3.x with the updated 2016b toolchains...

boegel · 2016-07-08T16:38:17Z

(close/open to trigger Travis)

boegel · 2016-07-08T16:41:07Z

ah, no, this needs a sync with develop...

@bmcgough see FredHutch#2

bedroge · 2016-07-11T07:27:45Z

I updated all extensions in my easyconfig for R 3.3.1 to the latest version, see:
https://github.com/rug-cit-ris/easybuild-easyconfigs/blob/R_3.3.1_foss2016a/easybuild/easyconfigs/r/R/R-3.3.1-foss-2016a.eb
Maybe it's useful to include those in this PR as well?

Update:
Ah, never mind, just found that @boegel already did this in #3314

boegel · 2016-07-12T13:42:41Z

#3314 was merged with easyconfigs for R 3.3.1 with updates dependencies and extensions, for both intel/2016b and foss/2016b (to be part of upcoming EasyBuild v2.8.2 release)

boegel · 2016-09-19T12:29:08Z

@bmcgough do you think it's still useful to try and get this merged?

sync with develop & resolve conflict

migueldiascosta · 2018-07-30T04:11:35Z

@bmcgough, this PR is being closed for the following reason(s): no activity for > 6 months, obsoleted by more recent PRs.
Please don't hesitate to reopen this PR or add a comment if you feel this contribution is still relevant.
For more information on our policy w.r.t. closing PRs, see https://easybuild.readthedocs.io/en/latest/Contributing.html#why-a-pull-request-may-be-closed-by-a-maintainer

Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with…

73548b9

… required patches for icc/icpc

boegel added this to the v2.9.0 milestone May 19, 2016

boegel reviewed May 19, 2016
View reviewed changes

boegel mentioned this pull request May 21, 2016

{lang} [foss-2016a} R-3.2.5 #3094

Closed

Added internal libgeotiff, libffi, LibTIFF,PCRE, gnutls, HDF5, and Ja…

f35caee

…sPer to GDAL as it was pulling from OS and those easyconfigs already existed

pushing R and dependent packages to test travis build

6475619

Benjamin McGough added 2 commits June 3, 2016 14:56

missing dependencies for intel GDAL

b76b719

missing dependencies for intel still

fce446b

Benjamin McGough added 3 commits June 3, 2016 15:33

Missed Tk, and added Tk for intel

06adba3

forgot libXft for intel

447b0b9

Missed changes to Python for intel

697514a

Adding SQLite easyconfig for intel-2016a R 3.3.0

17ee492

stachon mentioned this pull request Jun 16, 2016

tcl tk failure with R extension tkrplot #2749

Closed

boegel mentioned this pull request Jul 8, 2016

update extension versions in R 3.2.5 easyconfig verdurin/easybuild-easyconfigs#3

Merged

boegel closed this Jul 8, 2016

boegel reopened this Jul 8, 2016

Merge branch 'develop' into bmcgough-R-3.3

8006145

boegel mentioned this pull request Jul 8, 2016

sync with develop & resolve conflict FredHutch/easybuild-easyconfigs#2

Merged

verdurin mentioned this pull request Jul 19, 2016

generateEasyConfig aborting fgeorgatos/easybuild.experimental#16

Open

boegel modified the milestones: 3.0, v2.9.0 Sep 19, 2016

Merge pull request #2 from boegel/bmcgough-R-3.3

7e1a898

sync with develop & resolve conflict

boegel modified the milestones: 3.0, v3.x Nov 14, 2016

migueldiascosta closed this Jul 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with… #3079

Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with… #3079

bmcgough commented May 19, 2016

boegel commented May 19, 2016

boegel May 19, 2016

bmcgough May 19, 2016

pescobar May 19, 2016 •

edited

boegel May 19, 2016

fgeorgatos May 19, 2016

boegel May 20, 2016

bmcgough commented May 19, 2016

pescobar commented May 19, 2016

boegel commented May 19, 2016

boegel May 19, 2016

bmcgough commented May 19, 2016

boegel commented May 20, 2016

verdurin commented May 21, 2016

boegel commented Jun 3, 2016

bmcgough commented Jun 3, 2016

boegel commented Jun 3, 2016

boegel commented Jun 3, 2016

bmcgough commented Jun 3, 2016

boegel commented Jun 3, 2016

hajgato commented Jun 8, 2016

boegel commented Jun 8, 2016

bmcgough commented Jun 8, 2016

bedroge commented Jun 27, 2016

boegel commented Jul 7, 2016

boegel commented Jul 8, 2016

boegel commented Jul 8, 2016

bedroge commented Jul 11, 2016 •

edited

boegel commented Jul 12, 2016

boegel commented Sep 19, 2016

migueldiascosta commented Jul 30, 2018

Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with… #3079

Adding updated easyconfigs for R 3.3 foss and intel 2016a, along with… #3079

Conversation

bmcgough commented May 19, 2016

boegel commented May 19, 2016

boegel May 19, 2016

Choose a reason for hiding this comment

bmcgough May 19, 2016

Choose a reason for hiding this comment

pescobar May 19, 2016 • edited

Choose a reason for hiding this comment

boegel May 19, 2016

Choose a reason for hiding this comment

fgeorgatos May 19, 2016

Choose a reason for hiding this comment

boegel May 20, 2016

Choose a reason for hiding this comment

bmcgough commented May 19, 2016

pescobar commented May 19, 2016

boegel commented May 19, 2016

boegel May 19, 2016

Choose a reason for hiding this comment

bmcgough commented May 19, 2016

boegel commented May 20, 2016

verdurin commented May 21, 2016

boegel commented Jun 3, 2016

bmcgough commented Jun 3, 2016

boegel commented Jun 3, 2016

boegel commented Jun 3, 2016

bmcgough commented Jun 3, 2016

boegel commented Jun 3, 2016

hajgato commented Jun 8, 2016

boegel commented Jun 8, 2016

bmcgough commented Jun 8, 2016

bedroge commented Jun 27, 2016

boegel commented Jul 7, 2016

boegel commented Jul 8, 2016

boegel commented Jul 8, 2016

bedroge commented Jul 11, 2016 • edited

boegel commented Jul 12, 2016

boegel commented Sep 19, 2016

migueldiascosta commented Jul 30, 2018

pescobar May 19, 2016 •

edited

bedroge commented Jul 11, 2016 •

edited