[WIP] CLI usage improvements

[certik]

In this PR I will be implementing CLI usage improvements. I will list below what you can do now.

TODO before merging

• Rename list-profiles to just profile and make it behave just like git branch, i.e. list all profiles with the currently enabled one highlighted
• We need to decide whether to use @ or / below as a separator between the package and a hash. Conclusion: use /.
• The code accessing the list of profiles should not go via the hashstack directory, but using the hashdist database directly. This requires that the json files there contain the name of the profile as one of the fields.
• Merge show and show-profile into one command. There is already a show command in hit, so they all need to be merged.
• hit load should save the name (possibly link, whatever is needed) of the currently loaded profile into an environment variable, so that hit profile can then access it to see which profile is active and hit unload could unload it. Also hit load should unload any previously loaded profile. So a profile behaves like a git branch, only one (or none) active at a given time.

Here is what it can do:

(In a terminal, it prints the hashes nicely in color.)

1. List the installed profiles (name + hashes):

$ hit list-profiles
List of installed profiles (profile_name/profile_hash):
    basis/k4g2r7wjqcth
    truchas/ws2qspqle76r
    py32/loglud5apymb
    py3/DELETED
    py34/htvz74mbenv5
    xx/d34x65pdbsxt
    d2/2t547icnuhph
    hd_base/bto56eplrk4v
    b/DELETED
    test/vndgqtgsmdth
    rhel6/5vq4exlhbnuj
    python3.4.linux/yt6dksb4bzp3
    temp/ehcfjfeibvh5
    gtk/mgz74vgzjtdp
    py33/opqanpz7bwzd
    default/lcbhm255ymgh
    py27/pcnu7kh736at
    numpy/kkhizx2z6ea4
    truchas_bootstrap/frxnfq2suzik
    g5/4aktmgneyxgi

2. Load a particular profile

$ hit load gtk
This command requires hit's shell integration.

To initialize hit's shell commands, you must run in Bash:

    . /home/certik/repos/hashdist/share/hashdist/setup-env.sh

You can execute this line by hand or add it to your '.bashrc'. After that,
rerun your last command.

and just do what it says:

$ . /home/certik/repos/hashdist/share/hashdist/setup-env.sh
$ hit load gtk
Exporting HASHSTACK=/local/certik/bld/profile/mgz74vgzjtdp
Adding ${HASHSTACK}/bin to PATH
Profile gtk/mgz74vgzjtdp loaded.

3. List packages and other info about a particular profile

$ hit show py32
Information about the py32/loglud5apymb profile:
Path: /local/certik/bld/profile/loglud5apymb
Full profile hash: loglud5apymbhz3esdvq5tvb4js26hqq
List of packages:
    berkeleydb-5/ezpn7nogtzm4rt7hdhujxhn3o7nyhlaq
    blas/xsvwemqbi4dtutp326zzjpoldt62lqpa
    bzip2/yhn7t7sdxdfdkhyie6hg36hqvgccj7rj
    cmake/ugrko5ribtzlnijeygdngjxzk774ea6w
    gdbm/xqni5hcjx43andearscwqprb54wxnopq
    lapack/enbo6mjc6pkeoaoxnykphtro67cnqwqe
    launcher/wvmh5uvayzv3bxqk5kirztzn3rckjyv7
    mpi/hbd47qrzmwyecvdbonfalbn5xtqurzi3
    ncurses/viognkgus4njgat77hcakdedqccl3wjy
    nose/v7xodqh7kz4drb45ylswzazeqgem4hel
    numpy/oomyf2lggbc3bpl3bpdkp3teh5nfqyxc
    openssl/m6pttxckahscjt6jdev5acigozwa6ths
    patchelf/k3rloj265ogtl4dmmmbmyt34dnffryka
    pcre/2dpe5reczy3rt2jpx33hs2v675tofarr
    perl/xskj3irapqvxclfj5vnb4kgqcygpbpmt
    python/3dy43mxwqukrufgdrrlf3jng3ik7yaml
    readline/5tdfuoei3z6ektgh7v7lh3ra36s32ssp
    scipy/2mbg5u5xdskh2iezre6vewrsbalnfyci
    setuptools/4cd55p423ojcptjk5wugz7kiakxhcour
    sqlite/m5jo67qgu6zfrjydvg3fj3c5zvguflsx
    swig/qlr7obcxibc6erl74unyuactjlta7pec
    sympy/zglxlpatnl6e4omr22rj65spkm6u7vfg
    zlib/3el5ccejre7bcjqgld5gp6iym4ccd5oe

4. List dependencies and other info about a particular package

$ hit show-package python/3dy43mxwqukrufgdrrlf3jng3ik7yaml
Information about the python/3dy43mxwqukr package:
Path: /local/certik/bld/python/3dy43mxwqukr
Full package hash: 3dy43mxwqukrufgdrrlf3jng3ik7yaml
List of sources:
    files:ut3iptrelcmvcl3tmyu5z67vcyza3d5d
    tar.gz:7qpecklofhkhn5uwga5mvyutvz5cgehq
List of dependencies:
    bzip2/yhn7t7sdxdfdkhyie6hg36hqvgccj7rj
    launcher/wvmh5uvayzv3bxqk5kirztzn3rckjyv7
    ncurses/viognkgus4njgat77hcakdedqccl3wjy
    openssl/m6pttxckahscjt6jdev5acigozwa6ths
    patchelf/k3rloj265ogtl4dmmmbmyt34dnffryka
    readline/5tdfuoei3z6ektgh7v7lh3ra36s32ssp
    sqlite/m5jo67qgu6zfrjydvg3fj3c5zvguflsx
    zlib/3el5ccejre7bcjqgld5gp6iym4ccd5oe

[dagss]

Wonderful!

I don't think accessing Hashdist datastructured instead of filesystem is that important; in fact I don't believe there is much code for accessing profile database (only build artifact database, profiles are managed at the file system level if I remember correctly, which I may not).

As for the @ vs /, the / is already in use internally in Hashdist. If people feel that @ is much more readable we should change it to a @ internally too (which will change all hashes and cause full Hashstack rebuild everywhere, though that's not that important I guess). I think the options look about equally good and thus vote for / since it's less work..

[certik]

The argument for @ is that this is what Spack uses: http://scalability-llnl.github.io/spack/basic_usage.html#specs-dependencies, but now when I think about it, I think @ should only be used for a version (that's what Spack does), but version is not unique. You also have a compiler, that's %, i.e. %gcc, or %gcc4.9.2 or %intel14.1.1, which is also not unique. Our hash, on the other hand, is unique (but you do not know it ahead of time, only after you actually build). So now I think that if a hash is specified, one should use /. If a version is specified, one should use @ and if compiler is specified, one should use %. If you use /, then this already specifies both a version and a compiler, because our hash is unique. But you can also reference a package just by python@2.6.1, or you can be more specific as python@2.6.1%gcc or even more specific python@2.6.1%gcc4.9.2, or even more specific (following Spack) as mpileaks @1.2:1.4 %gcc@4.7.5 +debug -qt =bgqos_0 ^callpath @1.1 %gcc@4.7.2 (this is pretty specific already for mpileaks, specifying some of the dependencies and other configuration) ---- but none of these are unique, that's the point. Hashdist or Spack will make some choices, and then Hashdist will give it a hash, once all the build specs are fully known. Only then the package specification is unique, e.g. python/3dy43mxwqukrufgdrrlf3jng3ik7yaml.

Conclusion: we should use / for a hash, @ for a version and % for a compiler.

Thanks @dagss for the feedback. Extremely important to get this right.

@tgamblin, what do you think?

[tgamblin]

I like the use of / for the hash. I might actually consider adding that to Spack, because I'd like to be able to refer to specs by hash sometimes (hey look -- more convergence!). Would you allow hashes to be truncated like git does? e.g. could I put '/abc123', which is not a full hash, but might uniquely identify something already built? That would be a great feature.

For the other stuff, obviously I like the way Spack does it :). If you guys want to adopt the syntax, that's great and sets us up for an easier merge someday down the road when we build the next super-distribution tool.

The place I see this not being consistent with what hit does at the moment is that I don't think you guys always have all the parameters. If you implement this spec, so that people can customize it not he command line, each build needs to know the meaning of the various parameters. In Spack, every build has some sort of version associated with it (for repo checkouts we currently make them up but I'd like Spack to be smart enough to canonicalize a repo version to a git commit or svn revision, even if it is "named"). I get the impression that in hashdist a build might just have an implicit version, and that you don't always consider it to be parameterized. In that case I don't know what @ means. What happens if someone asks for numpy@1.4 and the numpy build doesn't have a version?

Same for compiler -- if you want % (and I think compiler is important enough to promote to first-class status), then the builds need to know their compiler somehow. In Spack the build always knows the compiler and its version (although I also think some package should just have None for compiler, which I don't currently support). How do you enforce this in hit? Or do you just exclude the ones that don't have the parameter from querying?

Do you also plan to support architecture? That one currently uses =, though I've thought about repurposing that to allow more sophisticated command line parameters, e.g. spack install myapp +dimensions=2, and using something else for arch. I am running out of characters, though.

Finally, what about the version range syntax? Spack has a fairly well fleshed out version class that supports ranges lists, intersections, etc: https://github.com/scalability-llnl/spack/blob/master/lib/spack/spack/version.py That is needed for concretization, when you apply all the different constraints to the package and settle on a set of build parameters. AFAIK hit has point versions and doesn't do any of that stuff -- are you supporting that syntax?

[tgamblin]

@certik: one more thing -- you wrote %intel14.1.1 but in the spec syntax the compiler version has another @ in there. Spack actually uses a recursive descent parser for this, esp. because the syntax is recursive -- you can apply all the same constraints to deps.

[tgamblin]

@certik, @dagss: I think having an in-memory representation of the DAG is helpful if you want to figure out how to satisfy all the constraints that the user gives vs. the ones in the package files. Does hit have an in-memory DAG representation?

[certik]

Hi @tgamblin thanks for the input. Yes, we already support shorter hashes in the file system and we definitely want to allow that, just like git. I meant to write %intel@14.1.1. For version and compiler support, see #327, #328, it will work exactly as in Spack, unless I am missing something, in which case, please let me know. I mentioned my plan in https://groups.google.com/d/msg/spack/9280IcPs0VI/xabY2Hk6ogUJ, where I posted links to all these open issues. Is there any feature that you think is still missing (after the issues are implemented)?

[certik]

Here is a bug that I found:

ondrej@eagle:~$ hit load -h
usage:: command not found
Loads: command not found
Example:: command not found
$: command not found
This: command not found
bash: syntax error near unexpected token `you'
ondrej@eagle:~$ hit load 
usage: hit load [-h] [--print-bash-commands] [-v] profile
hit load: error: too few arguments

I think the hit command doesn't return non zero exit code, but failed to return the commands (since it printed help instead).

[certik]

I extracted the hit load stuff from this PR and posted it in #340, so that we can merge it and people can start using hit load. I can then polish this PR properly, at the same time we can keep improving hit load once it is merged.