add Emacs-25.3

[michaelkarlcoleman]

Add latest emacs, more or less in the prior style (X/png/jpeg).

The existing Emacs configs all slightly vary in minor ways--this is intended to be a reasonable compromise of those.

[michaelkarlcoleman]

Argh. Apparently toolchain GCC implicitly satisfies the binutils dep while GCCcore does not. No luck finding doc on the difference between them.

Is there a way to run these consistency checks before submitting?

[boegel]

@tutufan The GCC toolchain is basically just a bundle of GCCcore + binutils, there's nothing more to it.
So the question boils down to when to use GCCcore as a toolchain (+ binutils as build dependency, which is enforced by the test suite since otherwise the build is likely going to fail with weird errors) rather than GCC.

GCCcore was introduced mainly for sites that use a hierarchical module naming scheme, where modules are organised in a tree-like fashion, with different "levels". On the lowest level you mainly find compilers which are installed with the dummy toolchain (i.e. which were built with the system compiler). One level up, you have tools that are built with a compiler-only toolchain (and higher levels have stuff built with compiler+MPI toolchain, or a full toolchain, i.e. compiler+MPI+BLAS/LAPACK/FFTW).
For more information on a setup like this, see http://easybuilders.github.io/easybuild/files/hust14_paper.pdf .

Easyconfigs that use the GCCcore toolchain result in modules on the compiler-only level of a hierarchy, which means those modules can be used to resolve dependencies for easyconfigs that use several toolchains, i.e. both the common foss and intel toolchains (which both have GCCcore as a 'base'), among others.
Modules that were installed with the GCC toolchain are not compatible with the intel toolchain because GCC is not a subtoolchain of intel (while GCCcore is).

For Emacs specifically, it may make sense to use GCC rather than GCCcore, since it's quite unlikely that Emacs is a dependency for something else. Modules installed with the GCC toolchain are compatible with both foss and intel (assuming the versions match up, of course), meaning that you can load those modules together.

I hope this clarifies things a bit...

[michaelkarlcoleman]

This was indeed helpful--thanks for your reply. Most of this makes sense, although I'm tripping over these two sentences a bit:

Modules that were installed with the GCC toolchain are not compatible with the intel toolchain...

vs

Modules installed with the GCC toolchain are compatible with both foss and intel...

Is the second GCC just a typo? (i.e., should be GCCcore) Or am I missing a further subtlety?

[verdurin]

@tutufan - might be worth mentioning in the description that this is a text-only build, in case we differentiate between this and a full X11 build in future.

[boegel]

@tutufan The wording is a bit confusing...

In the first sentence you highlighted, with 'compatible' I actually mean "will be considered when resolving dependencies". So, when resolving dependencies for something built with the intel toolchain, modules(/easyconfigs) that use the GCC toolchain are not considered to resolve dependencies, while modules installed with GCCcore are considered (because GCCcore is a subtoolchain of intel, while GCC is not).

In the 2nd sentence, I mean that you can load a module built with the GCC module together with modules built with the intel toolchain, as long as they use the same GCCcore version as a base, hence they're compatible.

[michaelkarlcoleman]

@verdurin Actually, this build does include the X client, but not built with the X toolkit. That is, the user interface looks like the classic emacs 1990s GUI. I don't use the GUI myself, so don't really care. We could pull in GTK and all of that (a bit expensive), but thought I'd leave that for someone who cares.

[michaelkarlcoleman]

@boegel Tried switching to GCC toolchain just now, and quickly realized that the dependencies (e.g., zlib, libpng) are nowhere near as up-to-date for GCC as they are for GCCcore. There doesn't even seem to be one for libpng right now. For expediency, I'm tempted to just stick with GCCcore and introduce the binutils build dep.

Alternatively, I could add up-to-date GCC easyconfigs for all of the deps. Which is better?

Also, if I were to do the latter, would you want to see one big PR, or does that need to be split into multiple PRs?

[boegel]

@tutufan When using the GCC toolchain, EasyBuild will pick up the easyconfigs using GCCcore to resolve the dependencies, since GCCcore is a subtoolchain of GCC.

So, if you switch to GCC/6.3.0-2.28 as toolchain, Travis should be happy (it makes sure that easyconfigs that can resolve the dependencies are available in the central repo or included in the PR).

W.r.t. your question on including easyconfigs for dependencies: that's usually OK if the total amount of easyconfigs in a single PR is small enough (say, <= 10). Especially if it's mainly version/toolchain updates for stuff we have already it's OK, since that's easy to review via eb --review-pr (see http://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html#reviewing-easyconfig-pull-requests-review-pr).

[michaelkarlcoleman]

@boegel Somehow GCC doesn't seem to be picking up the available GCCcore dependencies, or I'm still doing something wrong.
As a specific example, zlib/1.2.11-GCC-6.3.0 isn't getting picked up here, even though it seems to be available. I thought maybe --robot wasn't smart enough to build it, so I did so manually, but the Emacs build still isn't using it.
What am I missing?

(I'm using toolchain = {'name': 'GCC', 'version': '6.3.0'}. Is the -2.28 you mention above somehow needed here?)

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot    --robot-paths=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28 --rebuild  Emacs-25.3-GCC-6.3.0.eb
== temporary log file in case of crash /tmp/eb-Ql6iSg/easybuild-PgYpeE.log
== resolving dependencies ...
ERROR: Irresolvable dependencies encountered: zlib/1.2.11-GCC-6.3.0, libpng/1.6.29-GCC-6.3.0, libjpeg-turbo/1.5.2-GCC-6.3.0, ncurses/6.0-GCC-6.3.0, GCC/6.3.0

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot    --robot-paths=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28 --rebuild  zlib-1.2.11-GCCcore-6.3.0.eb
== temporary log file in case of crash /tmp/eb-L3XNwJ/easybuild-Q4veKF.log
== resolving dependencies ...
== processing EasyBuild easyconfig /projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/z/zlib/zlib-1.2.11-GCCcore-6.3.0.eb
== building and installing zlib/1.2.11-GCCcore-6.3.0...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /projects/eb-racs/p/software/zlib/1.2.11-GCCcore-6.3.0/easybuild/easybuild-zlib-1.2.11-20180111.102713.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-L3XNwJ/easybuild-Q4veKF.log* have been removed.
== Temporary directory /tmp/eb-L3XNwJ has been removed.

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot    --robot-paths=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28 --rebuild  Emacs-25.3-GCC-6.3.0.eb
== temporary log file in case of crash /tmp/eb-9CLIoz/easybuild-JhOLCg.log
== resolving dependencies ...
ERROR: Irresolvable dependencies encountered: zlib/1.2.11-GCC-6.3.0, libpng/1.6.29-GCC-6.3.0, libjpeg-turbo/1.5.2-GCC-6.3.0, ncurses/6.0-GCC-6.3.0, GCC/6.3.0

[michaelkarlcoleman]

Hmm. The last sentence of this section suggests that this ought to work: using minimal toolschains for dependencies

[boegel]

@tutufan The problem is that the robot can't find all necessary easyconfig files anymore, which is currently required to resolve dependencies via subtoolchains (we should loosen that up a bit though, it's now a bit too strict; see easybuilders/easybuild-framework#2375).

Here's the issue in your example, I think:

--robot    --robot-paths=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/

There's usually no reason to overwrite --robot-paths; if you want to specify an additional location for easyconfig files, just pass it to --robot, and leave --robot-paths as is. The location specified to --robot has preference.

[michaelkarlcoleman]

@boegel Looking back at the doc for --robot and --robot-paths, I'm recalling that I was explicitly trying to disable all easyconfigs except for my development repo. The singularity image I'm using for dev has a "pip install" version of easybuild in it (currently 3.5.0), but I don't want to pick up any easyconfigs from there, because if there's something wrong with my dev repo, I don't want anything from there to bleed through. Does that make sense?

I'd forgotten, though, that these options affect dependency resolution, at least according to the doc. It's not clear what happens if both options are specified, but in any case, I'm still failing to get subtoolchain dependency resolution to work. Here are some failures:

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --robot-paths=    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28       --trace    --debug    --disable-cleanup-builddir    --disable-cleanup-tmpdir Emacs-25.3-GCC-6.3.0.eb
== temporary log file in case of crash /tmp/eb-3pjGWV/easybuild-w1mM0U.log
== resolving dependencies ...
ERROR: Irresolvable dependencies encountered: zlib/1.2.11-GCC-6.3.0, libpng/1.6.29-GCC-6.3.0, libjpeg-turbo/1.5.2-GCC-6.3.0, ncurses/6.0-GCC-6.3.0, GCC/6.3.0

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28       --trace    --debug    --disable-cleanup-builddir    --disable-cleanup-tmpdir Emacs-25.3-GCC-6.3.0.eb
== temporary log file in case of crash /tmp/eb-LEzZoz/easybuild-n3z1sL.log
== resolving dependencies ...
ERROR: Irresolvable dependencies encountered: zlib/1.2.11-GCC-6.3.0, libpng/1.6.29-GCC-6.3.0, libjpeg-turbo/1.5.2-GCC-6.3.0, ncurses/6.0-GCC-6.3.0, GCC/6.3.0

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --robot-paths=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --prefix=/projects/eb-racs/p --buildpath=/tmp/eb-racs    --parallel=28       --trace    --debug    --disable-cleanup-builddir    --disable-cleanup-tmpdir Emacs-25.3-GCC-6.3.0.eb
== temporary log file in case of crash /tmp/eb-MIbgos/easybuild-IhgxA1.log
== resolving dependencies ...
ERROR: Irresolvable dependencies encountered: zlib/1.2.11-GCC-6.3.0, libpng/1.6.29-GCC-6.3.0, libjpeg-turbo/1.5.2-GCC-6.3.0, ncurses/6.0-GCC-6.3.0, GCC/6.3.0

Also, this case of a nonexistent config (note "-xxx") suggests that something is putting "." in the path, which seems like unexpected/undesirable behavior to me.

Singularity centos-7-easybuild-3.5.0.simg:/projects/eb-racs> eb    --robot=/projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/    --robot-paths=    --prefix=/projects/eb-racs/p    --buildpath=/tmp/eb-racs    --parallel=28       --trace    --debug    --disable-cleanup-builddir    --disable-cleanup-tmpdir Emacs-25.3-GCC-6.3.0-xxx.eb
== temporary log file in case of crash /tmp/eb-9ziZfd/easybuild-pl8XxW.log
ERROR: Can't find path /projects/eb-racs/Emacs-25.3-GCC-6.3.0-xxx.eb

(The Emacs easyconfig I'm using is exactly the same as the one in this PR, except that GCCcore has been changed to GCC.)

Any ideas? Am I still missing something? Or should I just give up on subtoolchain resolution and go back to GCCcore here?

[michaelkarlcoleman]

This version just uses GCCcore after all.

[boegel]

@tutufan So, you are pointing --robot to the location where the easyconfig file from develop are, while trying to avoid that the easyconfigs from the 3.5.0 installation you are using are being picked up, is that right?

That should work, and the easiest way to do that is to override the location via --robot-paths ... and just using --robot (without arguments) to enable dependency resolution.

Your last attempt (providing the location to both --robot and --robot-paths) is pretty much equivalent with that though, so I'm not sure why it's still failing for you...

You could try enabling --debug, and then diving into the log file to see which location EasyBuild considered for finding easyconfigs for the dependencies...

[boegel]

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in this PR)
node2016.delcatty.os - Linux centos linux 7.4.1708, Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, Python 2.7.5
See https://gist.github.com/1cf0a14c19b5c1cf2a6e2eaec0994d37 for a full test report.

[boegel]

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in this PR)
node2482.golett.os - Linux centos linux 7.4.1708, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/dba35acd16f3222cd49f312bfc947024 for a full test report.

[boegel]

@tutufan If it's OK with you, this PR looks good to go for me.

Maybe we should follow up on your issue with --robot and --robot-paths to get dependencies resolved via subtoolchains in a separate issue (in https://github.com/easybuilders/easybuild-framework/issues), since it's likely not specific to this particular easyconfig file?

[michaelkarlcoleman]

Yes, I think pushing this out now makes sense. Emacs-25.3 is a significant improvement over the existing.

Regarding your questions, yes, that's my intent. My intent is that only my tree is searched for easyconfigs, and the pip install tree is searched for easyblocks and the framework (since I'm not yet having to worry about making changes in those trees). And yes, I've done all of my dev except for those experiments with just --robot --robot-paths=/my/tree/only.

Looking at the debug log for this, it appears that actually a lot of directories are being searched (due to $PYTHONPATH?). Here's an excerpt:

== 2018-01-16 16:41:21,890 easyblock.py:1557 WARNING Easyconfig does not specify an EasyBuild-version (key 'easybuild_version')! Assuming the latest version
== 2018-01-16 16:41:21,891 filetools.py:343 INFO Command eb found at /usr/bin/eb
== 2018-01-16 16:41:21,891 tools.py:263 DEBUG Also considering installation prefix /usr...
== 2018-01-16 16:41:21,891 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /projects/eb-racs/easybuild-easyconfigs/easybuild/easyconfigs/easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib/python2.7/site-packages/git/ext/gitdb/easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python27.zip/easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/plat-linux2/easybuild/easyconfigs
== 2018-01-16 16:41:21,892 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/lib-tk/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/lib-old/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/lib-dynload/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/site-packages/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib/python2.7/site-packages/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/lib64/python2.7/site-packages/gitdb/ext/smmap/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:268 DEBUG Checking for easybuild/easyconfigs at /usr/easybuild/easyconfigs
== 2018-01-16 16:41:21,893 tools.py:272 DEBUG Added /usr/easybuild/easyconfigs to list of paths for easybuild/easyconfigs
== 2018-01-16 16:41:21,894 easyblock.py:646 INFO Found file emacs-25.3.tar.gz at /projects/eb-racs/p/sources/e/Emacs/emacs-25.3.tar.gz

As far as I can see, the log does not actually indicate the specific location where the easyconfig was actually found. (That would be useful!)

[boegel]

Going in, thanks @tutufan!