Why does spack install new version of package instead of using existing one?

[dlukes]

I have currently two versions of perl installed:

$ spack find -dv perl
==> 2 installed packages.
-- linux-ubuntu12.04-x86_64 / gcc@6.4.0 -------------------------
    perl@5.24.1+cpanm+shared
        ^gdbm@1.13
            ^readline@7.0
                ^ncurses@6.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks

    perl@5.24.1+cpanm+shared
        ^gdbm@1.13
            ^readline@7.0
                ^ncurses@6.0~symlinks

The second one was installed manually some time ago, the first one was installed as a dependency when trying to install r^gettext~libxml2. Is there a way to find out why, instead of reusing the existing installation?

I tried to examine spack spec r^gettext~libxml2, but that got me even more confused, because the concretized perl dependency seems to be the second one (i.e. the one installed first, manually):

^perl@5.24.1%gcc@6.4.0+cpanm+shared arch=linux-ubuntu12.04-x86_64

[scheibelp]

Was the manual installation added as an entry to packages.yaml?

[dlukes]

Thanks for getting back to me and sorry for the slow response, I was offline yesterday :)

Was the manual installation added as an entry to packages.yaml?

No, my only customization of packages.yaml is the following:

packages:
  all:
    # prefer a reasonably recent but not too recent gcc
    compiler: ["gcc@6:6.99", "gcc@5:5.99"]

[dlukes]

the concretized perl dependency seems to be the second one

OK, scratch this, I don't know why I thought so. Looking at the concretized spec, I think I see why this is happening. r depends on perl and gettext, among other things. This is the part of the concretized spec tree for spack spec r^gettext~libxml2 (or spack spec r, it doesn't matter) under gettext~libxml2:

^gettext@0.19.8.1%gcc@6.4.0+bzip2+curses+git~libunistring~libxml2 patches=9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8 +tar+xz arch=linux-ubuntu12.04-x86_64
    ^ncurses@6.0%gcc@6.4.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-ubuntu
12.04-x86_64
    ^tar@1.29%gcc@6.4.0 patches=08921fcbd732050c74ddf1de7d8ad95ffdbc09f8b4342456fa2f6a0dd02a957c,5c314db58d005043bf407abaf25eb9823b9032a22fd12a0b142d4bf548130fa4,d428578be7fb99b831eb61e53b8d88
a859afe08b479a21238180899707d79ce4,e34d2470461037933737c0a60fbcc5d5ea905fefb6b414b4a2e88a440fcae23a arch=linux-ubuntu12.04-x86_64

Notice the ncurses dep with patches. This is the part of the concretized spec tree under perl:

^perl@5.24.1%gcc@6.4.0+cpanm+shared arch=linux-ubuntu12.04-x86_64
    ^gdbm@1.13%gcc@6.4.0 arch=linux-ubuntu12.04-x86_64
    ^readline@7.0%gcc@6.4.0 arch=linux-ubuntu12.04-x86_64

However, perl also depends on ncurses, only here, that dependency is already satisfied via gettext above. With spack spec perl, it's listed explicitly:

perl@5.24.1%gcc@6.4.0+cpanm+shared arch=linux-ubuntu12.04-x86_64
    ^gdbm@1.13%gcc@6.4.0 arch=linux-ubuntu12.04-x86_64
        ^readline@7.0%gcc@6.4.0 arch=linux-ubuntu12.04-x86_64
            ^ncurses@6.0%gcc@6.4.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-ubuntu12.04-x86_64
                ^pkg-config@0.29.2%gcc@6.4.0+internal_glib arch=linux-ubuntu12.04-x86_64

So an explicit install of perl should currently also depend on patched ncurses, which is consistent. What I'm still wondering about is the following:

1. I'm not really sure why the perl I explicitly installed on Sep 26th didn't depend on patched ncurses -- as far as I can see, these patches are nothing new...
2. Even so, why did a new perl have to be installed if the previous explicit install was available? Is it because spack requires the concretization to be fully consistent, i.e. if r requires gettext to be installed, which in turn triggers an installation of a patched version of ncurses, then any other package which also depends on ncurses has to depend against this specific patched version as well?

If it is so, I'm guessing there's no easy way around it, because spack (probably?) needs this determinism to guarantee reproducibility (and for the dep hashes to make sense). Theoretically, allowing different versions of a build/link (not run) dependency should be possible (i.e. there's no reason why r couldn't be built with a gettext that was built against one version of ncurses, and a perl that was built against another one, is there?), but I assume it would introduce additional complexity into the concretization algorithm...?

[dlukes]

Still, it is annoying that you end up with a new version of perl/python etc. whenever you install a package that needs them to build, and also shares a dependency with them which happens to have been updated in the meantime :( But as noted above, I've no idea whether a fix is even theoretically possible given spack's architecture, and even if possible, whether it's doable in practice.

[scheibelp]

I'm not really sure why the perl I explicitly installed on Sep 26th didn't depend on patched ncurses -- as far as I can see, these patches are nothing new...

#5476 was merged on 9/30 and changed the hashes of specs where hashes are applied (even if the same hashes are applied as before)

if r requires gettext to be installed, which in turn triggers an installation of a patched version of ncurses, then any other package which also depends on ncurses has to depend against this specific patched version as well?

Generally speaking - yes. Whether or not Spack could get away with doing otherwise is a nuance not yet appreciated by the current concretizer. See for example: #3768 (which gets into when it makes sense to build separate instances of a given package)

Still, it is annoying that you end up with a new version of perl/python etc. whenever you install a package that needs them to build, and also shares a dependency with them which happens to have been updated in the meantime

This issue is documented at #646

I think this either covers existing questions or has overlap with existing issues that are linked, so I'm going to close this. Feel free to reopen if I missed something and there is a remaining issue that is unaddressed and not related to the issue I linked here (to be clear: feel free to reopen if your remaining problems aren't covered by #646 - you don't have to search around extensively for other issues that may be related).

[dlukes]

Thank you for taking the time to post such a complete reply, including pointers to existing issues :) I think that covers all my questions and misgivings, I've subscribed to the relevant issues to keep track of progress. Thanks again!