Spackenv: Script to create environments based on spack module loads

[citibeth]

The Problem

I've been able to build my stuff with Spack, but was still having problems assembling a proper set of spack module loads commands to use it. The reason is because I need to support multiple configurations / subsets of my overall system. For example, they might be:

1. Just the climate model
2. Just the ice model and coupler
3. Just the Python stuff needed for data processing
4. Any combination of (1), (2) and (3)
5. The kitchen sink
6. Potentially develop or release versions of the above (where some packages in the develop version are set up by hand using spack setup).

This PR makes it easier to create environment scripts --- that is, files full of module load commands --- based on this variety of needs. It is built as a separate Python executable program that calls Spack multiple times (to ensure robustness, so I don't hit bugs like #2666).

One serious problem up till now is that spack load, spack module loads, etc. do not have the benefit of concretization; they are resolved completely differently from spack install (although concretization is slow, so doing it in spack load would be a mixed blessing). The problem is that once you've used a spec for spack install, there is no easy / foolproof / automatable way to subsequently issue a spack load command for that module. If you have many similar installed versions in your Spack, that could be a real problem. This PR gets around that issue by tracking the hash that Spack uses to install the original spec, and then issuing a spack module loads command for that hash.

@tgamblin @eschnett @mathstuf @adamjstewart @becker33 @hartzell
Comments welcome, this software is "hot off the press." I'm especially interested in design ideas at this point. My guess is this script is a little "far out," and will need some iteration on design. I'm also hoping to get better / more robust file formats, but am not sure of what would be best.

Brief Docs

The user creates environment files (.env) in a src directory. spackenv then turns them into environment scripts (no extension) in an env directory. Users can then source those scripts in their Bash shell.

Here is a sample environment file:

envsrc/test.env
----------------
# `spack install` args : `spack module loads` args
#
# spack=spack --config ~/config-scopes/icebin-develop

--only dependencies libpciaccess
emacs : --dependencies

This environment consists of emacs (and all its dependencies), plus the dependencies for libpciaccess (but not their dependencies). Syntactically, it works as so:

1. The spack= directive gives the Spack command that spackenv is to use. This is to allow for environments to use different configurations (eg packages.yaml files; see Command Line Scopes #2686).

2. Each line consists of two parts, separated by a colon. The first part is command line arguments to be sent to spack install. The second part (optional) is command line arguments to be sent to spack module loads.

spackenv has two sub-commands: spackenv install and spackenv loads.

spackenv install

spackenv install runs spack install once per line in the .env file. It creates a .log files with Spack's output, and tees it to STDOUT as well. Here is the log for a typical spackenv install command; the contents of ~/env/test.log is also displayed to STDOUT here:

$ spackenv --src ~/envsrc --env ~/env install test
[+]  a6hzhtg  libpciaccess@0.13.4%gcc@4.9.3 arch=linux-centos7-x86_64
[+]  q5ztpkl      ^libtool@2.4.6%gcc@4.9.3 arch=linux-centos7-x86_64
[+]  kmufpwm          ^m4@1.4.17%gcc@4.9.3+sigsegv arch=linux-centos7-x86_64
[+]  ytewamh              ^libsigsegv@2.10%gcc@4.9.3 arch=linux-centos7-x86_64
[+]  zmirozy      ^pkg-config@0.29.1%gcc@4.9.3+internal_glib arch=linux-centos7-x86_64
[+]  36ec63z      ^util-macros@1.19.0%gcc@4.9.3 arch=linux-centos7-x86_64

SPACKENV BEGIN libpciaccess
==> util-macros is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/util-macros-1.19.0-36ec63zxfqtarj4wzuztzeckh6th53kp
SPACKENV INSTALLED util-macros/36ec63zxfqtarj4wzuztzeckh6th53kp
==> libtool is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/libtool-2.4.6-q5ztpklzt3emeq3p7rqguoqyht5lac36
SPACKENV INSTALLED libtool/q5ztpklzt3emeq3p7rqguoqyht5lac36
==> pkg-config is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/pkg-config-0.29.1-zmirozyvyf5jfalrz5vgge2uydgjtviq
SPACKENV INSTALLED pkg-config/zmirozyvyf5jfalrz5vgge2uydgjtviq
[+]  xmvbkdg  emacs@25.1%gcc@4.9.3~X toolkit=gtk arch=linux-centos7-x86_64
[+]  h66uwdb      ^ncurses@6.0%gcc@4.9.3 arch=linux-centos7-x86_64

SPACKENV BEGIN emacs
==> emacs is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/emacs-25.1-xmvbkdgpwvy4sedr2tl4a2yrx4togbiu
SPACKENV INSTALLED emacs/xmvbkdgpwvy4sedr2tl4a2yrx4togbiu

This is just typical Spack output. The extra lines starting with SPACKENV are parsed later by spackenv. Note that if stuff was already installed (for real or via spack setup), spackenv install won't build anything.

spackenv loads

The script of module load commands is generated by spackenv loads. Once spackenv install has run successfully, the user can run spackenv loads, as follows:

$ spackenv --src ~/envsrc --env ~/env loads test
spack --config ~/config-scopes/icebin-develop module loads util-macros/36ec63zxfqtarj4wzuztzeckh6th53kp
spack --config ~/config-scopes/icebin-develop module loads libtool/q5ztpklzt3emeq3p7rqguoqyht5lac36
spack --config ~/config-scopes/icebin-develop module loads pkg-config/zmirozyvyf5jfalrz5vgge2uydgjtviq
spack --config ~/config-scopes/icebin-develop module loads --dependencies emacs/xmvbkdgpwvy4sedr2tl4a2yrx4togbiu

Here, spackenv shows you what commands it's running. It also produces the file ~/env/test, which looks like:

$ cat ~/env/test
# util-macros@1.19.0%gcc@4.9.3 arch=linux-centos7-x86_64
module load util-macros-1.19.0-gcc-4.9.3-36ec63z
# libtool@2.4.6%gcc@4.9.3 arch=linux-centos7-x86_64
module load libtool-2.4.6-gcc-4.9.3-q5ztpkl
# pkg-config@0.29.1%gcc@4.9.3+internal_glib arch=linux-centos7-x86_64
module load pkg-config-0.29.1-gcc-4.9.3-zmirozy
# ncurses@6.0%gcc@4.9.3 arch=linux-centos7-x86_64
module load ncurses-6.0-gcc-4.9.3-h66uwdb
# emacs@25.1%gcc@4.9.3~X toolkit=gtk arch=linux-centos7-x86_64
module load emacs-25.1-gcc-4.9.3-xmvbkdg

This file may now be used to load up the specified environment in a Bash shell.

[This PR depends on #2664...]

[citibeth]

BTW... this needs a better name, so it's not confused with spack env, which is entirely different. I just don't know what that better name should be. Maybe call "Spack Profiles" (as long as that doesn't conflict with #643 which eventually got implemented as "Filesystem Views").

[alalazo]

Closed as depending on a closed PR #2664