Getting unique identifier of installed package

[samfux84]

Hi all,

Installing a package, for instance

spack install m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=linux-centos7-x86_64_avx
will generate a unique identifier as part of the name of the installation directory. I would like to get this identifier after installing a package and save it in a variable. Up to now I was parsing the output of spack find:

[spackapps@lo-login-02 scripts]$ spack find --path m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=linux-centos7-x86_64_avx
==> 1 installed packages.
-- linux-centos7-x86_64_avx / gcc@4.8.5 -------------------------
    m4@1.4.18  /cluster/spack/apps/linux-centos7-x86_64_avx/gcc-4.8.5/m4-1.4.18-vdq5a4jub3w4dkb4fqe62f5eqh353ior
[spackapps@lo-login-02 scripts]$ LABEL=`spack find --path m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=linux-centos7-x86_64_avx | grep "/cluster/spack/apps" | cut -d "/" -f 7`
[spackapps@lo-login-02 scripts]$ echo $LABEL
m4-1.4.18-vdq5a4jub3w4dkb4fqe62f5eqh353ior

If I have two packages, which only differ by the CPU target, then specifying the "arch" when running spack find resolves the conflict and returns only the package I am searching for:

[spackapps@lo-login-02 scripts]$ spack find m4
==> 2 installed packages.
-- linux-centos7-x86_64_avx / gcc@4.8.5 -------------------------
m4@1.4.18

-- linux-centos7-x86_64_avx2 / gcc@4.8.5 ------------------------
m4@1.4.18

[spackapps@lo-login-02 scripts]$ spack find --path m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=linux-centos7-x86_64_avx==> 1 installed packages.
-- linux-centos7-x86_64_avx / gcc@4.8.5 -------------------------
    m4@1.4.18  /cluster/spack/apps/linux-centos7-x86_64_avx/gcc-4.8.5/m4-1.4.18-vdq5a4jub3w4dkb4fqe62f5eqh353ior

For some reason, this does not properly work for ncurses:

[spackapps@lo-login-02 scripts]$ spack find --path ncurses
==> 2 installed packages.
-- linux-centos7-x86_64_avx / gcc@4.8.5 -------------------------
    ncurses@6.0  /cluster/spack/apps/linux-centos7-x86_64_avx/gcc-4.8.5/ncurses-6.0-72fhqqtiayl4uaqqdep4xslahf2wqcex

-- linux-centos7-x86_64_avx2 / gcc@4.8.5 ------------------------
    ncurses@6.0  /cluster/spack/apps/linux-centos7-x86_64_avx2/gcc-4.8.5/ncurses-6.0-q7gctrew5lyulpoii2g6ztwfcxehn5ee

[spackapps@lo-login-02 scripts]$ spack find --path ncurses@6.0 ^pkg-config@0.29.2 %gcc@4.8.5 arch=linux-centos7-x86_64_avx
==> 2 installed packages.
-- linux-centos7-x86_64_avx / gcc@4.8.5 -------------------------
    ncurses@6.0  /cluster/spack/apps/linux-centos7-x86_64_avx/gcc-4.8.5/ncurses-6.0-72fhqqtiayl4uaqqdep4xslahf2wqcex

-- linux-centos7-x86_64_avx2 / gcc@4.8.5 ------------------------
    ncurses@6.0  /cluster/spack/apps/linux-centos7-x86_64_avx2/gcc-4.8.5/ncurses-6.0-q7gctrew5lyulpoii2g6ztwfcxehn5ee
[spackapps@lo-login-02 scripts]$

Even when specifying the arch parameter, spack returns the path for both installations instead of just for the specified arch.

Therefore I have two questions:

1.) Is there a better way to "extract" the unique identifier of a package installation ?

2.) @alalazo Do you think the problem described above could be related to $SPACK_TARGET_TYPE ? I would rather think that this is not the case, as the output of spack find for the patch package was correct.

Best regards and thank you in advance.

[adamjstewart]

Try spack find -l. This will print the hash that uniquely identifies the installation. You can then use this on the command line like so:

$ spack install foo ^/hash
$ spack activate /hash
$ spack uninstall /hash
...

[samfux84]

@adamjstewart Thank you for the hint about "spack -l". It does only return part of the hash. I would rather need the entire hash, as I use a script, which generates some symbolic links for installed spack packages.

My plan is to start from a minimal Linux installation and then to also build system tools (in principle I am rebuilding a part of /usr/bin). Since spack has a different installation directory for each package, I create kind of a fake "/usr/bin" and create symbolic links to the system tools installed by spack.

A user needs then to only add a single directory to his PATH and can use all the system tools (I think nobody wants to load a module for using tools like "ls" or "mv"):

[spackapps@lo-login-02 bin]$ pwd
/cluster/spack/sfos/linux-centos7-x86_64_avx2/bin

[spackapps@lo-login-02 bin]$ ls -ltr tar 
lrwxrwxrwx 1 spackapps spackapps-group 105  2. Mai 09:49 tar -> /cluster/spack/apps/linux-centos7-x86_64_avx2/gcc-4.8.5/tar-1.29-ddgpf3dkswbzaqawvyzjcvrrdb663zmo/bin/tar

[spackapps@lo-login-02 bin]$ ls
'['                                   fc-list              msgattrib          sort
 aclocal                              fc-match             msgcat             splain
 aclocal-1.15                         fc-pattern           msgcmp             split
 addr2line                            fc-query             msgcomm            sqlite3
 ar                                   fc-scan              msgconv            sqlite3_analyzer
 as                                   fc-validate          msgen              srconv
 autoconf                             flex                 msgexec            stat
 autoheader                           flex++               msgfilter          stdbuf
 autom4te                             fmt                  msgfmt             strings
 automake                             fold                 msggrep            strip
 automake-1.15                        fonttosfnt           msginit            stty
 autopoint                            freetype-config      msgmerge           sum
 autoreconf                           funzip               msgunfmt           sync
 autoscan                             gdbm_dump            msguniq            sysdump
 autoupdate                           gdbm_load            mv                 tabs
 b2sum                                gdbmtool             nasm               tac
 base32                               gettext              ncurses6-config    tail
 base64                               gettextize           ncursesw6-config   tar
 basename                             gettext.sh           ndisasm            tclsh
 bash                                 gif2tiff             ngettext           tclsh8.6
 bashbug                              git                  nice               tee
 bdftruncate                          git-cvsserver        nl                 test
 bison                                gitk                 nlmconv            thumbnail
 bmp2tiff                             git-receive-pack     nm                 tic
 bunzip2                              git-shell            nohup              tiff2bw
 bzcat                                git-upload-archive   nproc              tiff2pdf
 bzcmp                                git-upload-pack      numfmt             tiff2ps
 bzdiff                               gmake                objcopy            tiff2rgba
 bzegrep                              gperf                objdump            tiffcmp
 bzfgrep                              gpg-error            od                 tiffcp
 bzgrep                               gpg-error-config     pal2rgb            tiffcrop
 bzip2                                gprof                paste              tiffdither
 bzip2recover                         groups               patch              tiffdump
 bzless                               h2ph                 patchelf           tiffinfo
 bzmore                               h2xs                 pathchk            tiffmedian
 c2ph                                 head                 pcre-config        tiffset
 captoinfo                            help2man             pcregrep           tiffsplit
 cat                                  hmac256              pcretest           timeout
 c++filt                              hostid               perl               toe
 chcon                                iconv                perl5.24.1         touch
 chgrp                                id                   perlbug            tput
 chmod                                ifnames              perldoc            tr
 chown                                infocmp              perlivp            true
 chroot                               infotocap            perlthanks         truncate
 cjpeg                                install              piconv             tset
 cksum                                instmodsh            pinky              tsort
 clear                                join                 pkg-config         tty
 coffdump                             jpegtran             pl2pm              ucs2any
 comm                                 json_pp              pngfix             uname
 corelist                             kill                 png-fix-itxt       unexpand
 cp                                   ld                   pod2html           uniq
 cpan                                 ld.bfd               pod2man            unlink
 cpanm                                ld.gold              pod2text           unlz4
 csplit                               libgcrypt-config     pod2usage          unlzma
 curl                                 libnetcfg            podchecker         unxz
 curl-config                          libpng16-config      podselect          unzip
 cut                                  libpng-config        ppm2tiff           unzipsfx
 date                                 libtool              pprof              uptime
 dbus-cleanup-sockets                 libtoolize           pr                 users
 dbus-daemon                          link                 printenv           vdir
 dbus-launch                          ln                   printf             vsyasm
 dbus-monitor                         logname              prove              wc
 dbus-run-session                     ls                   pstruct            wget
 dbus-send                            lua                  ptar               who
 dbus-test-tool                       luac                 ptardiff           whoami
 dbus-update-activation-environment   luarocks             ptargrep           windmc
 dbus-uuidgen                         luarocks-5.3         ptx                windres
 dd                                   luarocks-admin       pwd                wrjpgcom
 df                                   luarocks-admin-5.3   ranlib             x86_64-unknown-linux-gnu-pkg-config
 dir                                  lz4                  ras2tiff           xgettext
 dircolors                            lz4c                 raw2tiff           xml2-config
 dirname                              lz4cat               rdjpgcom           xmlcatalog
 djpeg                                lzcat                readelf            xmllint
 dlltool                              lzcmp                readlink           xmlwf
 dllwrap                              lzdiff               realpath           xsubpp
 du                                   lzegrep              recode-sr-latin    xz
 dumpsexp                             lzfgrep              reset              xzcat
 dwp                                  lzgrep               rgb2ycbcr          xzcmp
 echo                                 lzless               rm                 xzdec
 elfedit                              lzma                 rmdir              xzdiff
 enc2xs                               lzmadec              runcon             xzegrep
 encguess                             lzmainfo             seq                xzfgrep
 env                                  lzmore               sha1sum            xzgrep
 envsubst                             m4                   sha224sum          xzless
 expand                               make                 sha256sum          xzmore
 expr                                 mawk                 sha384sum          yacc
 factor                               md5sum               sha512sum          yasm
 false                                mkdir                shasum             yes
 fax2ps                               mkfifo               shred              ytasm
 fax2tiff                             mknod                shuf               zipdetails
 fc-cache                             mktemp               size               zipgrep
 fc-cat                               mpicalc              sleep              zipinfo
[spackapps@lo-login-02 bin]$

In order to make this setup simpler, I created a script that sets up the system tools and creates the symbolic links automatically, therefore I need the full hash. Please find below an excerpt from the script, which contains m4 as example:

spack install m4@1.4.18+sigsegv ^libsigsegv@2.11 arch=$OS_ARCH
LABEL=`spack find --path m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=$OS_ARCH | grep "/cluster/spack/apps" | cut -d "/" -f 7`
echo -e "LABEL=$LABEL"
for tool in /cluster/spack/apps/$OS_ARCH/$STDCOMP/$LABEL/bin/*; do btool=$(basename $tool);ln -s $tool $btool; done

[adamjstewart]

You can get the full hash with spack find -L. But have you heard of spack view? It already does what you're trying to do.

[samfux84]

@adamjstewart Thank you for pointing me to spack view, this sounds exactly like what I was looking for. Do you know where I can find some documentation about it ? I was browsing the spack documentation but could not find information part about "spack view". It is also not listed in the command reference part of the documentation:

https://spack.readthedocs.io/en/latest/command_index.html

When trying to use it to create symbolic links, I run into the same problem as before with spack find:

[spackapps@lo-login-02 bin]$ spack view symlink add m4@1.4.18+sigsegv ^libsigsegv@2.11 %gcc@4.8.5 arch=x86_64_avx
==> Error: m4@1.4.18+sigsegv matches multiple packages.
  Matching packages:
    vdq5a4j m4@1.4.18%gcc@4.8.5 arch=linux-centos7-x86_64_avx 
    iugfemj m4@1.4.18%gcc@4.8.5 arch=linux-centos7-x86_64_avx2 
  Use a more specific spec.
[spackapps@lo-login-02 bin]$

Best regards and thank you for helping me.

[adamjstewart]

Does this work?

$ spack view symlink add /vdq5a4j

This is the only documentation I could find on spack view:
http://spack.readthedocs.io/en/latest/workflows.html?highlight=view#filesystem-views

I'll see if I can add that to the Command Index.

[samfux84]

@adamjstewart Thank you for your help. The command that you propose creates some links:

[spackapps@lo-login-02 test]$ spack view symlink add /vdq5a4j
[spackapps@lo-login-02 test]$ ls */*
add/bin:
m4

add/include:
sigsegv.h

add/lib:
libsigsegv.a  libsigsegv.la  libsigsegv.so  libsigsegv.so.2  libsigsegv.so.2.0.4

add/share:
info  man
[spackapps@lo-login-02 test]$

[adamjstewart]

Any time you need to pass a spec to a command, you should be able to use a hash instead. If you encounter any commands that this doesn't work with, it's a bug.

[citibeth]

@scheibelp

@serbanmaerean spack find is fundamentally broken, for the reason you mentioned. After Spack installs a package, it currently "forgets" what it just installed, hoping that you find it again with spack find for inclusion (for example) in a Spack view. We are working on fixing this with an up-coming plan for Spack Environments.

In the meantime, you can (have to) parse the output of spack install, if you want to know the hash of what you just installed. I've made this even easier with the --report option in #2698, which you might wish to try. Note that the spackenv script in that PR is oriented toward generating a script of module load commands, which is similar in functionality to a Spack view. If you really want a Spack view, extending the script to generate views instead of module load commands would not be hard either. (i.e. it would be a one-line kind of change).

[citibeth]

PS: We know that #2698 is not going to get merged. But boy do I depend on it in the meantime, you might find it useful too.

[samfux84]

@citibeth Thank you for pointing to #2698. I will have a look at it.

With a combination of "spack find", "grep" and "cut" I can meanwhile consistently get the unique identifier (even for the case shown above, where it did not work in the beginning).