ASP solver does not handle missing packages during --reuse

[mpbelhorn]

Steps to reproduce

The clingo concretizer fails when attempting to --reuse existing builds when a package that at exists in the builtin package repo was previously installed using a custom fork of the package (with site-specific modifications) located in a higher-priority local package repo has since been removed from the local repo so that a newer builtin-repo version of the package is preferred for future builds.

# add modified package with same name as a builtin package to a local repo
spack install <package>
# remove modified package from local repo so builtin version is used
spack -vd --stacktrace concretize --reuse -f

Ideally, a spack.repo.UnknownPackageError would mark the previous build as ineligible for re-use and concertize specs using only packages still existing in available repos, even under --reuse.

While it's certainly possible to restore the missing package to the local repo as a direct clone of the upstream package, my hope is that when upstream packages are updated to handle issues that previously required site-specific hacks applied through a fork of the package existing in a private repo, I would be able to drop my private fork of the package.

Error message

spack -vd --stacktrace concretize --reuse -f

[PID=3602251] Traceback (most recent call last):                                                                                                                                                                                               
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/util/parallel.py", line 53, in __call__                                                                                                      
    value = self.func(*args, **kwargs)                                                                                                                                                                                                         
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2057, in _concretize_task                                                                                  
    return _concretize_from_constraints(spec_constraints, tests, reuse)                                                                                                                                                                        
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2035, in _concretize_from_constraints                                                                      
    return s.concretized(tests=tests, reuse=reuse)                                                                                                                                                                                             
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2692, in concretized                                                                                                          
    clone.concretize(tests=tests, reuse=reuse)                                                                                                                                                                                                 
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2648, in concretize                                                                                                           
    self._new_concretize(tests, reuse=reuse)                                                                                                                                                                                                   
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2619, in _new_concretize                                                                                                      
    result = spack.solver.asp.solve([self], tests=tests, reuse=reuse)                                                                                                                                                                          
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 2054, in solve                                                                                                          
    setup, specs, dump, models, timers, stats, tests, reuse                                                                                                                                                                                    
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 621, in solve                                                                                                           
    answers = builder.build_specs(tuples)                                                                                                                                                                                                      
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 1995, in build_specs                                                                                                    
    spack.spec.Spec.inject_patches_variant(root)                                                                                                                                                                                               
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2534, in inject_patches_variant                                                                                               
    pkg_deps = dspec.parent.package_class.dependencies                                                                                                                                                                                         
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 1305, in package_class                                                                                                        
    return spack.repo.path.get_pkg_class(self.fullname)                                                                                                                                                                                        
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 647, in get_pkg_class                                                                                                         
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)                                                                                                                                                                                 
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 1107, in get_pkg_class                                                                                                        
    module = self._get_pkg_module(pkg_name)                                                                                                                                                                                                    
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 1067, in _get_pkg_module                                                                                                      
    raise UnknownPackageError(pkg_name, self)                                                                                                                                                                                                  
spack.repo.UnknownPackageError: Package 'ncl' not found in repository '/ccs/home/belhorn/development/olcf-spack-environments/share/spack/repos/olcf'

Information on your system

$ spack debug report

• Spack: 0.17.1
• Python: 3.6.8
• Platform: linux-rhel8-zen2
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[alalazo]

Reproduced by creating a repo with just zlib:

myrepo$ tree
.
├── packages
│   └── zlib
│       ├── configure-cc.patch
│       ├── package.py
│       └── w_patch.patch
└── repo.yaml

2 directories, 4 files

then:

$ spack repo add .
==> Added repo with namespace 'myrepo'.
$ spack install zlib
[ ... ]
$ spack repo remove .
$ spack solve -I --reuse zlib
==> [2022-04-15-15:49:22.931162] Imported solve from built-in commands
==> [2022-04-15-15:49:22.932430] Imported solve from built-in commands
==> [2022-04-15-15:49:22.933327] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:22.942795] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:22.960173] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:22.973288] DATABASE LOCK TIMEOUT: 3s
==> [2022-04-15-15:49:22.973331] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-04-15-15:49:22.973628] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:22.981017] DATABASE LOCK TIMEOUT: 3s
==> [2022-04-15-15:49:22.981057] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-04-15-15:49:22.981105] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2022-04-15-15:49:22.981383] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/culpo/PycharmProjects/spack/etc/spack/defaults
==> [2022-04-15-15:49:22.981413] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/culpo/PycharmProjects/spack/etc/spack/defaults/linux
==> [2022-04-15-15:49:22.981436] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/home/culpo/.spack/bootstrap/config
==> [2022-04-15-15:49:22.981457] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/home/culpo/.spack/bootstrap/config/linux
==> [2022-04-15-15:49:22.981662] Reading config file /home/culpo/.spack/bootstrap/config/linux/compilers.yaml
==> [2022-04-15-15:49:23.002132] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
==> [2022-04-15-15:49:23.002191] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2022-04-15-15:49:23.002360] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:23.114100] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:23.195629] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.8 /shqedxgvjnhiwdcdrvjhbd73jaevv7wt" provides the "clingo" Python module
==> [2022-04-15-15:49:23.195884] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/concretizer.yaml
==> [2022-04-15-15:49:23.197623] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/repos.yaml
==> [2022-04-15-15:49:23.198864] Reading config file /home/culpo/.spack/repos.yaml
==> [2022-04-15-15:49:23.228975] Reading config file /home/culpo/.spack/linux/compilers.yaml
==> [2022-04-15-15:49:23.236692] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:23.249716] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/packages.yaml
==> [2022-04-15-15:49:23.271535] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/mirrors.yaml
==> [2022-04-15-15:49:24.034166] Fetching index from https://mirror.spack.io/build_cache/index.json
==> [2022-04-15-15:49:24.913261] UnknownNamespaceError: Unknown namespace: myrepo
==> [2022-04-15-15:49:24.913312] Error: Unknown namespace: myrepo
Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 915, in main
    return _main(argv)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 870, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 898, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 550, in _invoke_command
    return_val = command(parser, args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/solve.py", line 111, in solve
    result = solver.solve(
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 2141, in solve
    return self.driver.solve(
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 641, in solve
    answers = builder.build_specs(tuples)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 2038, in build_specs
    repo = spack.repo.path.repo_for_pkg(spec)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/repo.py", line 933, in repo_for_pkg
    raise UnknownNamespaceError(namespace)
spack.repo.UnknownNamespaceError: Unknown namespace: myrepo