concretizer: reuse existing packages

[tgamblin]

Draft PR because it still needs some work, but it's in a state that @adamjstewart could start playing with.

Fixes #311.
Fixes #22613.
Fixes #20429.

This PR makes the concretizer aware of already-installed packages, and it tries to optimize the solve by reusing them instead of building when it can.

--reuse option

Currently, this adds a --reuse option to spack solve. You can try it out with spack solve -Il --reuse <spec>. I think the interface is TBD; we probably want to think about what to call this vs. the old solve, and about which one should be default.

Minimizing builds

The way this works is:

1. Figure out which installed packages are possible dependencies of the things to be installed.
2. Add information about installed hashes that are possible dependencies to the ASP program.
   It looks kind of like this openssl example, but for everything that is installed:

   installed_hash("openssl","lwatuuysmwkhuahrncywvn77icdhs6mn").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node","openssl").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","version","openssl","1.1.1g").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_platform","openssl","darwin").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_os","openssl","catalina").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_target","openssl","x86_64").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_compiler","openssl","apple-clang").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_compiler_version","openssl","apple-clang","12.0.0").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","hash","openssl","lwatuuysmwkhuahrncywvn77icdhs6mn").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","depends_on","openssl","zlib","build").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","depends_on","openssl","zlib","link").
   imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","hash","zlib","x2anksgssxsxa7pcnhzg5k3dhgacglze").

   You can see from the example that if we use this hash for openssl, it also forces a bunch of constraints and a dependency on its particular zlib installation on the solve. i.e., it's concrete -- we have to use it the way it was built and we cannot twiddle things like variant values on things we already installed.
3. Try to minimize the number of packages that are built. That looks like of like:

   % the solver is free to choose at most one installed hash for each package in the solution
   { hash(Package, Hash) : installed_hash(Package, Hash) } 1 :- node(Package).
   
   % if a hash is selected, we impose all of its constraints
   impose(Hash) :- hash(Package, Hash).
   
   % if we haven't selected a hash for a package, and if it is not external, we build it
   build(Package) :- not hash(Package, _), not external(Package), node(Package).
   #minimize { 1@18,Package : build(Package) }.

OS handling

The solver was still handling OS defaults through hard constraints (this is how we did it before we knew how to do it the right way -- with optimization), so this reworks OS handling in the solver. I believe this is the last part of the spec that needed converting -- targets and compilers were already changed in past PRs.

We now optimize for OS matches in the Spec, but we allow you to depend on a package for an older OS, with some limits. Specifically, I noticed when testing this that I wanted to reuse still-working Catalina installations on Big Sur. This adds the ability to specify, e.g.: os_compatible("catalina", "bigsur") so that the solver knows that it can safely reuse certain packages. I think we can also leverage this for, e.g., reusing CentOS binaries on RHEL.

Relaxing existing constraints

Many of the existing constraints in the solver are meant for things we plan to build. When reusing concrete specs, We have to take the installed specs as they are. We can't do things like disallow unknown variant names (the variants may not exist) or compiler version OS support (we only know about the compilers on the current system, not the ones the concrete specs were built with). So, these types of constraints have been restricted to only apply to specs we intend to build. If we reuse something concrete, we still have to consider things like dependency version constraints (e.g. if you want to reuse openssl you should still check if the version that's installed satisfies your depends_on("openssl@:1.0") constraint).

Example

Here's a comparison of reusing vs. not reusing with llvm:

With reuse:

`spack solve -Il --reuse llvm` output

(spackle):spack> spack solve -Il --reuse llvm
==> Best of 7 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       5
  2         deprecated versions used                      2
  3         version weight                                0
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    5
  8         preferred providers (non-roots)              10
  9         compiler mismatches                           0
  10        os mismatches                                 8
  11        version badness                              51
  12        count of non-root multi-valued variants      -5
  13        non-preferred compilers                       0
  14        target mismatches                             0
  15        non-preferred targets                       253

 -   ikfi254  llvm@12.0.1%apple-clang@12.0.0~all_targets+clang~code_signing+compiler-rt~cuda~flang~gold+internal_unwind~ipo+libcxx+lld~lldb~llvm_dylib~mlir~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf build_type=Release cuda_arch=none arch=darwin-bigsur-x86_64
 -   xtkcrrd      ^cmake@3.20.5%apple-clang@12.0.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-x86_64
[+]  eli4aax          ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib arch=darwin-catalina-x86_64
[+]  lwatuuy          ^openssl@1.1.1g%apple-clang@12.0.0+systemcerts arch=darwin-catalina-x86_64
[+]  x2anksg              ^zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-x86_64
 -   oc2m54j      ^hwloc@2.5.0%apple-clang@12.0.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~pci+shared arch=darwin-bigsur-x86_64
[+]  dqzlvh6          ^libxml2@2.9.10%apple-clang@12.0.0~python arch=darwin-catalina-x86_64
[+]  xrjlxcv              ^libiconv@1.16%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]  gfa5h76              ^xz@5.2.5%apple-clang@12.0.0~pic arch=darwin-catalina-x86_64
 -   ozoef2w          ^pkgconf@1.7.4%apple-clang@12.0.0 arch=darwin-bigsur-x86_64
 -   tzk3evu      ^perl-data-dumper@2.173%apple-clang@12.0.0 arch=darwin-bigsur-x86_64
[+]  jjgcc6o          ^perl@5.30.3%apple-clang@12.0.0+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]  df3hxmr              ^gdbm@1.18.1%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]  4d3hjqd                  ^readline@8.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]  24lf5pr      ^python@3.7.8%apple-clang@12.0.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 arch=darwin-catalina-x86_64
[+]  6ntaixw          ^bzip2@1.0.8%apple-clang@12.0.0+shared arch=darwin-catalina-x86_64
[+]  2jdnmfp          ^expat@2.2.9%apple-clang@12.0.0~libbsd arch=darwin-catalina-x86_64
[+]  g5lf2co          ^gettext@0.20.2%apple-clang@12.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]  dc675hk              ^tar@1.32%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]  5ye4c7r          ^libffi@3.3%apple-clang@12.0.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]  nwrcpot          ^sqlite@3.31.1%apple-clang@12.0.0+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]  de6nj46      ^z3@4.8.7%apple-clang@12.0.0+python arch=darwin-catalina-x86_64
[+]  mjlsl7o          ^py-setuptools@49.2.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64

Without reuse:

`spack solve -Il llvm` output

(spackle):spack> spack solve -Il llvm
==> Best of 7 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)      16
  2         deprecated versions used                      0
  3         version weight                                9
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)   10
  8         preferred providers (non-roots)              20
  9         compiler mismatches                           0
  10        os mismatches                                 0
  11        version badness                               9
  12        count of non-root multi-valued variants       0
  13        non-preferred compilers                       0
  14        target mismatches                             0
  15        non-preferred targets                         0

 -   2p6rpiv  llvm@8.0.1%apple-clang@12.0.0~all_targets+clang~code_signing+compiler-rt~cuda~flang~gold+internal_unwind~ipo+libcxx+lld~lldb~llvm_dylib~mlir~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf build_type=Release cuda_arch=none patches=3fa900e31763cc7ff604430b86faa4eb650048c1796ebd39db2dcf84f6168a8b arch=darwin-bigsur-skylake
 -   rs5nwwn      ^cmake@3.20.5%apple-clang@12.0.0~doc+ncurses~openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-skylake
 -   xrklbl5          ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib abi=none arch=darwin-bigsur-skylake
 -   ztt3pgs              ^pkgconf@1.7.4%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   hjw2gx7      ^hwloc@2.5.0%apple-clang@12.0.0~cairo~cuda~gl~libudev~libxml2~netloc~nvml~pci+shared arch=darwin-bigsur-skylake
 -   zipu7bi      ^perl-data-dumper@2.173%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   raet62c          ^perl@5.34.0%apple-clang@12.0.0+cpanm+shared+threads arch=darwin-bigsur-skylake
 -   akjl2di              ^berkeley-db@18.1.40%apple-clang@12.0.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-bigsur-skylake
 -   l7gigul              ^bzip2@1.0.8%apple-clang@12.0.0~debug~pic+shared arch=darwin-bigsur-skylake
 -   vazus7p                  ^diffutils@3.7%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   vtlbzpn                      ^libiconv@1.16%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   xkdly6s              ^gdbm@1.19%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   gmhw2jt                  ^readline@8.1%apple-clang@12.0.0 arch=darwin-bigsur-skylake
[+]  xflsoui              ^zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-bigsur-skylake
 -   inxzdsq      ^python@3.9.6%apple-clang@12.0.0+bz2~ctypes+dbm~debug~libxml2~lzma~nis~optimizations+pic~pyexpat+pythoncmd+readline+shared~sqlite3~ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-bigsur-skylake
 -   m6ef5no          ^apple-libuuid@1353.100.2%apple-clang@12.0.0 arch=darwin-bigsur-skylake
 -   otyemkg          ^gettext@0.21%apple-clang@12.0.0+bzip2+curses+git~libunistring~libxml2~tar~xz arch=darwin-bigsur-skylake

You can see we reuse a lot more with the reuse option on -- whereas without it Spack wants to build clean versions of everything but zlib. You can see this in the optimization criteria as well -- the difference between using --reuse and not is 16 built packages. In the case without --reuse, we're just not telling the solver about the installed packages so it has no hashes to swap in for different builds.

Optimization criteria

There are still some tricky things in here -- specifically around the priority of these optimizations. You can see in the solve calls above that we're optimizing by minimizing the number of packages built. That has some weird consequences for things that aren't installed yet, like:

(spackle):spack> spack solve -Il --reuse cmake
[...]
 -   fw3sb6l  cmake@3.20.5%apple-clang@12.0.0~doc+ncurses~openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-skylake
[+]  ywtu57g      ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib abi=none arch=darwin-catalina-skylake

Here, the concretizer decided to disable openssl on a new cmake install, to avoid installing more things. So it did exactly what we told it to, but not what we likely want. We probably need to add some nuance to the optimization criterion. In particular, sometimes you care more about default variants, and sometimes you'd rather it reuse what's already installed.

Currently, I am thinking we want to only minimize builds if something that is installed could be reused. If there's some package we already have to build, we should just respect its defaults and build dependencies as we normally would. I haven't worked out how best to model that yet or if it'll end up fitting peoples' intuition, so please provide suggestions.

On a similar note, you can see that we also used a couple of deprecated versions of things because they were already installed. Seems like maybe we should put the reuse optimization below the "don't use deprecated versions" optimization so that this won't happen. That would mean that you'd need to be explicit on the command line about using deprecated versions to get them into the solve. I suspect that might surprise some people so I'm not sure whether we shouldn't just warn instead.

Tasks

• Add information about installed dependencies to the solve
• Rework OS semantics in the solver to allow for certain OS's to be compatible
• Make certain integrity constraints only apply to specs we intend to build and not to reused specs
• Resolve issues with semantics and ordering of optimization criteria (i.e. do the right thing when we're not reusing)
• Figure out whether this should be the default and what the option should be called (e.g., this is --reuse -- should the old way be --deterministic? --upgrade? --latest?)
• add options to spack install
• enable reuse from binary caches and upstreams too (so that if you register them, you automatically reuse prebuilt things)
• add --reuse to docs

That last task will make build caches and upstreams significantly easier to use the way we intended them, and it'll mean that you can both pull from develop and easily reuse binaries from older versions of Spack. This will reduce the perception of "hashes changing" that people have with the old, more deterministic concretizer.

[adamjstewart]

Thanks @tgamblin, you have no idea how happy this makes me! Gave this a quick spin, here are my thoughts.

CLI

I definitely think this should be configurable from the command-line. Can we move the option into common_arguments since multiple commands need to share it? Specifically, can you add the flag to concretize and install so I can play around with an environment and really give this PR a good test?

P.S. When are we merging spack solve into spack spec? I still want to be able to use spack spec --concretizer=clingo or spack spec --concretizer=original so I can see how things differ, which isn't possible with spack spec or spack solve if the default concretizer is set to clingo already. Would also love to merge spack concretize into spack spec too. But I can open an issue for that at some point, don't want to derail this PR.

Configuration

Since most Spack users will want to install things many times, I think this flag should be configurable in config.yaml. At SC a few years ago (has it really been that long already?) we talked about how users of the new concretizer could configure it to prefer certain behaviors, like:

• minimize reinstalls
• maximize variants enabled
• minimize variants enabled
• minimize dependencies
• etc.

Is this still something you think we should do? I would actually love to make the entire optimization criteria API public so that I can configure the order in my config.yaml.

On this thread, the behavior I would personally like doesn't exactly fall under --reuse or --deterministic. What I would personally like to use would be something in between. Let's say I have an environment with a list of root specs. These are likely packages that I care about, and would like to be able to test with the latest version whenever they get updated. However, non-root specs (or dependencies of specs I ask for on the command-line) are something I don't care about. So I would like to --reuse but only for dependencies of root specs.

Default

Since we're planning on switching the default concretizer in 0.17, and since concretization will change as a result anyway, I think we have a rare opportunity to change default concretization behavior without worrying too much about backwards compatibility. I think --reuse being on by default is the correct way to go, and works very similarly to literally every other PM in existence.

Bugs

This PR seems to drastically change concretization behavior. That's expected, since --reuse will tell the concretizer to prefer existing installations even if they aren't exactly what were expected. What isn't expected is that this PR drastically changes concretization behavior when --reuse isn't being used! For example:

develop, no reuse

$ spack solve -I python
==> Best of 10 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)              20
  8         compiler mismatches                           0
  9         version badness                              -1
  10        count of non-root multi-valued variants      -2
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                      -600

[+]  python@3.8.11%apple-clang@12.0.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^apple-libuuid@1353.100.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^bzip2@1.0.8%apple-clang@12.0.0~debug~pic+shared arch=darwin-catalina-x86_64
[+]          ^diffutils@3.7%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]              ^libiconv@1.16%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^expat@2.4.1%apple-clang@12.0.0~libbsd arch=darwin-catalina-x86_64
[+]      ^gdbm@1.19%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]          ^readline@8.1%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]              ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]                  ^pkgconf@1.7.4%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^gettext@0.21%apple-clang@12.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]          ^libxml2@2.9.10%apple-clang@12.0.0~python arch=darwin-catalina-x86_64
[+]              ^xz@5.2.5%apple-clang@12.0.0~pic libs=shared,static arch=darwin-catalina-x86_64
[+]              ^zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-x86_64
[+]          ^tar@1.34%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^libffi@3.3%apple-clang@12.0.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]      ^openssl@1.1.1k%apple-clang@12.0.0~docs+systemcerts arch=darwin-catalina-x86_64
[+]          ^perl@5.34.0%apple-clang@12.0.0+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]              ^berkeley-db@18.1.40%apple-clang@12.0.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-catalina-x86_64
[+]      ^sqlite@3.35.5%apple-clang@12.0.0+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64

this branch, no reuse

$ spack solve -I python
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       4
  2         deprecated versions used                      0
  3         version weight                                0
  4         number of non-default variants (roots)       10
  5         multi-valued variants                         0
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    4
  8         preferred providers (non-roots)              20
  9         compiler mismatches                           0
  10        OS mismatches                                 0
  11        non-preferred OSes                            0
  12        version badness                              -1
  13        count of non-root multi-valued variants       0
  14        non-preferred compilers                       0
  15        target mismatches                             0
  16        non-preferred targets                      -150

 -   python@3.8.11%apple-clang@12.0.0~bz2~ctypes~dbm~debug~libxml2~lzma~nis~optimizations+pic~pyexpat+pythoncmd~readline+shared~sqlite3~ssl~tix~tkinter~ucs4+uuid~zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^apple-libuuid@1353.100.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
 -       ^gettext@0.21%apple-clang@12.0.0~bzip2~curses+git~libunistring~libxml2~tar~xz arch=darwin-catalina-x86_64
[+]          ^libiconv@1.16%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^pkgconf@1.7.4%apple-clang@12.0.0 arch=darwin-catalina-x86_64

this branch, reuse

$ spack solve -I --reuse python
==> Best of 3 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       0
  2         deprecated versions used                      0
  3         version weight                                0
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    1
  8         preferred providers (non-roots)               0
  9         compiler mismatches                           0
  10        OS mismatches                                 0
  11        non-preferred OSes                            0
  12        version badness                              -1
  13        count of non-root multi-valued variants      -3
  14        non-preferred compilers                       0
  15        target mismatches                             0
  16        non-preferred targets                      -480

[+]  python@3.8.11%apple-clang@12.0.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^apple-libuuid@1353.100.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^bzip2@1.0.8%apple-clang@12.0.0~debug~pic+shared arch=darwin-catalina-x86_64
[+]      ^expat@2.4.1%apple-clang@12.0.0~libbsd arch=darwin-catalina-x86_64
[+]      ^gdbm@1.19%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]          ^readline@8.1%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]              ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]      ^gettext@0.21%apple-clang@12.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]          ^libiconv@1.16%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]          ^libxml2@2.9.10%apple-clang@12.0.0~python arch=darwin-catalina-x86_64
[+]              ^xz@5.2.5%apple-clang@12.0.0~pic libs=shared,static arch=darwin-catalina-x86_64
[+]              ^zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-x86_64
[+]          ^tar@1.34%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^libffi@3.3%apple-clang@12.0.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]      ^openssl@1.1.1k%apple-clang@12.0.0~docs+systemcerts arch=darwin-catalina-x86_64
[+]      ^sqlite@3.35.5%apple-clang@12.0.0+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64

It seems that even when --reuse isn't being used, the concretizer is still trying to minimize the number of packages that are built. I think that's a useful feature to have (see configuration section above), but seems to be orthogonal to the feature we're trying to implement here.

[tgamblin]

It seems that even when --reuse isn't being used, the concretizer is still trying to minimize the number of packages that are built.

Yep this was true. It would not have any installations to consider, but it would minimize the number of built packages -- and everything was built so this boiled down to minimizing nodes as the top priority. Pushed a fix for just this problem.

I would still like to make the reuse a bit smarter -- basically I would like it to aggressively reuse when possible, without minimizing builds quite so aggressively. I am not sure I really know how to structure the optimization criteria to get that result, but I'm thinking about it.

[tldahlgren]

Will this resolve multiple dependency spec issues such as the one reported for font-util at https://gitlab.spack.io/spack/spack/-/jobs/619732?

...
==> [2021-08-09-20:06:17.817550] Error: font-util@1.3.2%gcc@7.5.0 fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64 ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^automake@1.16.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^bdftopcf@1.0.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu18.04-x86_64 ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64 ^diffutils@3.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^fontsproto@2.1.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^freetype@2.10.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^libfontenc@1.1.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^libpng@1.6.37%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^libxfont@1.5.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^m4@1.4.19%gcc@7.5.0+sigsegv arch=linux-ubuntu18.04-x86_64 ^mkfontdir@1.0.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^mkfontscale@1.1.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64 ^perl@5.34.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64 ^pkgconf@1.7.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^xproto@7.0.31%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^xtrans@1.3.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64 ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64 matches multiple specs in the environment /builds/spack/spack/jobs_scratch_dir/concrete_environment: 
Dependency spec
  skmlfuy  font-util@1.3.2%gcc@7.5.0 fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64
Dependency spec
  skmlfuy  font-util@1.3.2%gcc@7.5.0 fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64
...

[alalazo]

@tldahlgren @tgamblin Fyi, I moved all the pipelines to clingo in #25502 and I encounter several errors similar to #25310 (comment), see e.g. https://gitlab.spack.io/spack/spack/-/jobs/659647

[becker33]

I haven't looked into whether it's feasible yet, but I think what we want for the optimization criteria is a heuristic for how unlikely we are to care about something -- for example, we could score each node by the minimal depth to traverse from the root to the node, and then put the optimization "cost" of building that node at magic_weight * depth and minimize on that criterion.

We may also want to prioritize default variants for root packages above minimizing builds, but default variants for non-root packages below minimizing builds. That seems to track more with what we think we're doing when we install a package "unconstrained" (I want the default for the package, and then as efficient as possible to build deps).

I'll start looking into how feasible this is in case people like it.

[becker33]

I also agree with @adamjstewart that we want to make these options as configurable as possible. I don't think we want to expose the API for this because it's probably not sufficiently user friends to make a good user interface, but we should basically create a config option for every decision that we feel is non-obvious what people will want, because that probably means different people will want different things.

[becker33]

Another thought -- if we want to minimize the extent to which turning off default variants improves the score without preventing the concretizer from toggling variants to get better reuse, we could combine the variant issue into the criterion. The "score" for the reuse optimization criterion could be something like sum(depth * weight1) * sum(weight2 * non_default_variants / depth) or something equally insane that gets the behavior that users will find intuitive. The variant default criterion could then be handled independently at a different priority.

In general I think some of our optimization criteria may be too complicated to be modeled at discrete priorities, and we may need to write more complicated fitness functions to explore the trade-offs in the space. Those fitness functions could well be named and selected by config options, to get back to the point Adam was making.

[becker33]

Thinking about this more, I think the default behavior probably does not require a complicated priority-mixing fitness function, because I think my earlier idea about alternating the priority root_variants > reuse > non_root_variants is probably close enough to a good default. But we may need complicated functions for some of the other concretization options we may consider.

[becker33]

Partial review

[becker33]

This should include everything from platform.operating_systems.values(). That's really the list of operating systems we could target from this build node.

[becker33]

I don't think this was addressed

[tgamblin]

This worked with platform.operating_sys.keys()

[becker33]

I think we need a much more formal model of what a compatible os is before we can try to mix operating systems in a meaningful way -- I think we currently get away with it because we put the OS optimization at such a high priority that the algorithm never chooses something else, but if we really want to allow centos7 binaries to be selected for rhel7 nodes we need a module for operating system compatibility which we can call into to compare any two operating systems and determine whether binaries for the first can run on the second. Then we could populate our possible_oses list with something like all the operating systems that we know could run on any of the operating systems in the buildable list.

[tgamblin]

@adamjstewart @alalazo I still need to clean this up a bit, rework how spack solve displays opt criteria, and address @becker33's comments but can you see if the version here fixes the issue that minimizing nodes was drastically affecting the way we concretize built specs?

I'll write up a better description of how this works later, but short story is that now the opt criteria are structured like this:

1. criteria for specs to be built (all minimize -- i.e. "best" value is zero)
2. number of built (minimize)
3. criteria for reused specs (exactly the same criteria as 1 above, but for the already installed stuff)

The idea is that if we build nothing, all the criteria in (1) are zero. If we do build something, then the top priority is making the build specs look like we've come to expect (i.e., criteria in (1)), but we still prioritize reusing to satisfy their dependencies. This will still reuse as much as possible, as reusing will always be "as good" as the best possible build configuration (because all the criteria are minimizing and their best value is zero). No build configuration will be inherently better than reusing because reusing will not worsen any criteria in (1).

It's a bit hard to think about at first but once you get the idea that the reused specs can't make the criteria in (1) worse, and (2) dominates the optimization process until we determine that something has to be built, it starts to make sense.

[tgamblin]

@haampie can you give this a try as well?

[haampie]

Hey Todd, I'll have a look on Monday!

[adamjstewart]

@tgamblin with the latest commit, all calls to spack spec/solve result in the following warning:

/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

Also, I'm not seeing any dependency reuse anymore. spack spec -I results in the same thing on develop as it does on your branch, even though I have all of these dependencies installed somewhere:

$ spack solve -I py-torch
/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

==> Best of 12 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                         Value
  1         (build) deprecated versions used                      0
  2         (build) version weight                                0
  3         (build) number of non-default variants (roots)        0
  4         (build) preferred providers for roots                 0
  5         (build) preferred providers for roots                 0
  6         (build) number of non-default variants (non-roots)    0
  7         (build) preferred providers (non-roots)               0
  8         (build) compiler mismatches                           0
  9         (build) OS mismatches                                 0
  10        (build) non-preferred OSes                            0
  11        (build) version badness                               5
  12        (build) non-preferred compilers                       0
  13        (build) target mismatches                             0
  14        (build) non-preferred targets                     -1230
  15        number of packages to build (vs. reuse)               0
  16        deprecated versions used                              0
  17        version weight                                        0
  18        number of non-default variants (roots)                0
  19        preferred providers for roots                         0
  20        preferred providers for roots                         0
  21        number of non-default variants (non-roots)            0
  22        preferred providers (non-roots)                       0
  23        compiler mismatches                                   0
  24        OS mismatches                                         0
  25        version badness                                       0
  26        non-preferred compilers                               0
  27        target mismatches                                     0
  28        non-preferred targets                                 0

 -   py-torch@1.9.1%apple-clang@12.0.0+caffe2~cuda~cudnn~distributed~fbgemm~gloo+kineto~magma+metal+mkldnn~mpi~nccl+nnpack~numa+numpy+onnx_ml+openmp+qnnpack~rocm~tensorpipe~test~valgrind+xnnpack cuda_arch=none patches=e37afffe45cf7594c22050109942370e49983ad772d12ebccf508377dc9dcfc9 arch=darwin-catalina-x86_64
[+]      ^cmake@3.21.3%apple-clang@12.0.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-catalina-x86_64
[+]          ^ncurses@6.2%apple-clang@12.0.0~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]              ^pkgconf@1.8.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]          ^openssl@1.1.1l%apple-clang@12.0.0~docs certs=system arch=darwin-catalina-x86_64
[+]              ^perl@5.34.0%apple-clang@12.0.0+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]                  ^berkeley-db@18.1.40%apple-clang@12.0.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-catalina-x86_64
[+]                  ^bzip2@1.0.8%apple-clang@12.0.0~debug~pic+shared arch=darwin-catalina-x86_64
[+]                      ^diffutils@3.8%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]                          ^libiconv@1.16%apple-clang@12.0.0 libs=shared,static arch=darwin-catalina-x86_64
[+]                  ^gdbm@1.19%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]                      ^readline@8.1%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]                  ^zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-x86_64
[+]      ^eigen@3.4.0%apple-clang@12.0.0~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^fxdiv@2020-04-17%apple-clang@12.0.0~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]          ^ninja@1.10.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]              ^python@3.8.11%apple-clang@12.0.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=darwin-catalina-x86_64
[+]                  ^apple-libuuid@1353.100.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]                  ^expat@2.4.1%apple-clang@12.0.0~libbsd arch=darwin-catalina-x86_64
[+]                  ^gettext@0.21%apple-clang@12.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]                      ^libxml2@2.9.12%apple-clang@12.0.0~python arch=darwin-catalina-x86_64
[+]                          ^xz@5.2.5%apple-clang@12.0.0~pic libs=shared,static arch=darwin-catalina-x86_64
[+]                      ^tar@1.34%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]                  ^libffi@3.3%apple-clang@12.0.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]                  ^sqlite@3.36.0%apple-clang@12.0.0+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]      ^intel-mkl@2020.0.166%apple-clang@12.0.0~ilp64+shared threads=none arch=darwin-catalina-x86_64
[+]      ^llvm-openmp@12.0.1%apple-clang@12.0.0~ipo~multicompat build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^protobuf@3.18.0%apple-clang@12.0.0+shared build_type=Release arch=darwin-catalina-x86_64
[+]      ^psimd@2020-05-17%apple-clang@12.0.0~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^pthreadpool@2021-04-13%apple-clang@12.0.0~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^py-future@0.18.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
 -           ^py-setuptools@57.4.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64
 -       ^py-numpy@1.21.2%apple-clang@12.0.0+blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604 arch=darwin-catalina-x86_64
 -           ^py-cython@0.29.24%apple-clang@12.0.0 arch=darwin-catalina-x86_64
 -       ^py-protobuf@3.17.3%apple-clang@12.0.0~cpp arch=darwin-catalina-x86_64
[+]          ^py-six@1.16.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^py-pybind11@2.6.2%apple-clang@12.0.0~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^py-pyyaml@5.3.1%apple-clang@12.0.0+libyaml arch=darwin-catalina-x86_64
[+]          ^libyaml@0.2.5%apple-clang@12.0.0 arch=darwin-catalina-x86_64
 -       ^py-tqdm@4.62.3%apple-clang@12.0.0~notebook~telegram arch=darwin-catalina-x86_64
 -           ^py-setuptools-scm@6.0.1%apple-clang@12.0.0+toml arch=darwin-catalina-x86_64
[+]              ^py-toml@0.10.2%apple-clang@12.0.0 arch=darwin-catalina-x86_64
[+]      ^py-typing-extensions@3.10.0.0%apple-clang@12.0.0 arch=darwin-catalina-x86_64

[tgamblin]

@adamjstewart can you test with spack solve --reuse? Looks like you didn’t give the option.

the provider weight thing needs to be fixed with a #defined but it’s minor.

addressed

[alalazo]

I checked the last commits and gave this PR another fast test-drive. LGTM.

[psakievich]

Congrats all for getting this in. I followed the dev conversation from a far. I'm excited to try this out. 🥳