concretize fails in develop environment when dependencies are also under development

[simonpintarelli]

Steps to reproduce

For the following develop environment with the linear chain of dependencies sirius -> nlcglib -> kokkos (all under development):

spack:
  concretization: together
  specs:
    - sirius
  develop:
    kokkos:
      path: /path/to/kokkos
      spec: kokkos@develop%clang
    nlcglib:
      path: /path/to/nlcglib
      spec: nlcglib@develop%clang
    sirius:
      path: /path/to/SIRIUS
      spec: sirius@develop+nlcglib%gcc

Concretization fails with the following error (apparently coming from nlcglib, the first dependency of the root package sirius):

$ spack -e . concretize
=> Error: 0.9 does not satisfy develop

If ^nlcglib@develop is appended to the root spec of sirius, it will complain with the same message about the latest version of kokkos not matching develop.

Error message

lib/spack/spack/main.py:800 ==> [2021-09-28-11:51:09.616421] UnsatisfiableVersionSpecError: 0.9 does not satisfy develop
lib/spack/spack/error.py:54 ==> [2021-09-28-11:51:09.623945] Error: 0.9 does not satisfy develop
Traceback (most recent call last):
  File "/home/simon/spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/simon/spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/home/simon/spack/lib/spack/spack/cmd/concretize.py", line 37, in concretize
    concretized_specs = env.concretize(force=args.force, tests=tests)
  File "/home/simon/spack/lib/spack/spack/environment.py", line 1147, in concretize
    return self._concretize_together(tests=tests)
  File "/home/simon/spack/lib/spack/spack/environment.py", line 1186, in _concretize_together
    concrete_specs = spack.concretize.concretize_specs_together(
  File "/home/simon/spack/lib/spack/spack/concretize.py", line 725, in concretize_specs_together
    return _concretize_specs_together_new(*abstract_specs, **kwargs)
  File "/home/simon/spack/lib/spack/spack/concretize.py", line 730, in _concretize_specs_together_new
    result = spack.solver.asp.solve(abstract_specs)
  File "/home/simon/spack/lib/spack/spack/solver/asp.py", line 1691, in solve
    return driver.solve(setup, specs, dump, models, timers, stats, tests)
  File "/home/simon/spack/lib/spack/spack/solver/asp.py", line 396, in solve
    answers = builder.build_specs(tuples)
  File "/home/simon/spack/lib/spack/spack/solver/asp.py", line 1641, in build_specs
    _develop_specs_from_env(s, ev.active_environment())
  File "/home/simon/spack/lib/spack/spack/solver/asp.py", line 1665, in _develop_specs_from_env
    spec.constrain(dev_info['spec'])
  File "/home/simon/spack/lib/spack/spack/spec.py", line 3115, in constrain
    raise UnsatisfiableVersionSpecError(self.versions, other.versions)
spack.spec.UnsatisfiableVersionSpecError: 0.9 does not satisfy develop

Information on your system

• Spack: 0.16.3-4489-56d4c21c75
• Python: 3.9.7
• Platform: linux-arch-skylake
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[simonpintarelli]

ping @haampie

[haampie]

@becker33 once more I would propose to make the specs in the develop section selectors/filters on the concrete environment, and not have them influence concretization. The current behavior is hard to explain to users: all root specs get constrained with develop specs before concretization -- but if you fail to mention any of them in the root specs, you run into this issue. So should we advise users to write

spack:
  specs:
    - sirius ^kokkos ^nlcglib
  develop:
    kokkos:
      path: /path/to/kokkos
      spec: kokkos@develop%clang
    nlcglib:
      path: /path/to/nlcglib
      spec: nlcglib@develop%clang
    sirius:
      path: /path/to/SIRIUS
      spec: sirius@develop+nlcglib%gcc

as a workaround? That doesn't look great to me :(

If you translate this environment literally into the syntax from the develop-specs-not-packages PR #22087 and run concretize, you get this behavior:

spack:
  specs:
    - sirius
  develop:
  - spec: kokkos@develop%clang
    path: /path/to/kokkos
  - spec: nlcglib@develop%clang
    path: /path/to/nlcglib--wt-gpu-memory
  - spec: sirius@develop+nlcglib%gcc
    path: /path/to/SIRIUS--wt-nlcg-gpu-memory

from the warnings it's clear that the specs act as filters, so you'd have to move those to the specs section of the environment:

spack:
  specs:
    - sirius@develop+nlcglib%gcc ^kokkos@develop%clang ^nlcglib@develop%clang
  develop:
  - spec: kokkos
    path: /path/to/kokkos
  - spec: nlcglib
    path: /path/to/nlcglib--wt-gpu-memory
  - spec: sirius
    path: /path/to/SIRIUS--wt-nlcg-gpu-memory

and then it concretizes without warning to

 -   icf4ftu  sirius@develop%gcc@10.3.0+nlcglib dev_path=/path/to/SIRIUS--wt-nlcg-gpu-memory
 -   wf4dmj7      ^nlcglib@develop%clang@9.0.1 dev_path=/path/to/nlcglib--wt-gpu-memory
 -   wp7eota          ^kokkos@develop%clang@9.0.1 dev_path=/path/to/kokkos

which to me seems much easier to understand

[haampie]

From a discussion with @psakievich:

You could also concretize the devs specs like concretize_together([root spec, dev specs...]), which would make concretization less surprising.

However, in the same discussion we agreed that it'd be nice to be able to compare a dev spec with a stable non-dev spec in one and the same environment, which would be impossible when the dev spec is always imposed as a constraint to the root specs.

[psakievich]

Yes I think that would be going in the wrong direction because we end up constraining what develop specs can do and limit opportunities for future growth. Spack has all the infrastructure to do multiple develop builds with variants of the same spec in one environment, and a scalable build system to boot. It would be a shame to handicap develop specs with a change like this.

[psakievich]

Another idea that follows the spirit of #22087 is to allow users to put the dev_path variant directly in the specs.
This is clunkier for spec specifications but removes the ambiguity users are experiencing. Not sure I endorse this but I thought I'd mention it. I could support it as an optional way to specify develop specs, but wouldn't want to use it myself. Probably not a good sign 😬

[simonpintarelli]

Here is a closely related example which fails and for which the above described workaround doesn't help:

spack:
  view: true
  concretization: together
  specs:
  - q-e-sirius ^sirius
  - sirius@develop%gcc@10.3.0~python+cuda cuda_arch=75 build_type=RelWithDebInfo ^py-h5py~mpi
    ^hdf5~mpi ^openblas threads=openmp
  develop:
    sirius:
      path: /path/to/sirius
      spec: sirius@develop
    q-e-sirius:
      path: /path/to/qe-sirius
      spec: q-e-sirius@develop

fails when using clingo with Error: Unsatisfiable spec., together with a list of unsatisfiable constraints (39'400 lines). The same spec works when setting concretizer: original in config.yaml