Upcoming high-impact changes (perma-pin)

[tgamblin]

This issue is meant to track upcoming changes to Spack that are likely to impact users. We'll put links to issues here that will either break existing workflows or cause major disruptions when you pull from develop.

develop is the main development branch of Spack, but many users like to stay at the bleeding edge, and this is intended to provide advance warning for those users. Note that we still recommend using either a release (more here)or a fixed commit from develop for production/large site deployments.

High-impact changes to hit before v0.18:

• concretizer: make --reuse the default behavior #29111: We will be making --reuse the default concretizer behavior. This will be configurable so you can revert back using Add a top-level concretizer config scope and --fresh option #28468.
• Switch to full hash everywhere #28504: After --reuse is default, we will be switching to a "full hash" as the default package id in Spack. Spack has long used a coarser package hash that it probably should, in order to prevent redundant rebuilds. Specifically, we omit build dependencies, the hash of package.py, and the hashes of resources (tarballs, etc.) from our build identifiers. This was mainly to prevent every update of cmake or other build tools from triggering a rebuild of the full stack. With --reuse as the default, we don't have this concern any more, and we will prioritize detailed provenance for package builds. The full hash will provide this and will also avoid hash collisions that we currently see in CI.
• Allow for multiple dependencies/dependents from the same package #28673: This PR allows a given spec to depend on multiple instances of the same package, differentiated by dependency type; recording the dependency type will change the underlying storage representation
• PythonPackage: install packages with pip #27798: This PR completely changes how we install Python packages to run pip install instead of python setup.py install. It also improves bootstrapping of frontend tools (pip/build/installer) and backend tools (setuptools/flit/poetry).
• Revert "Deprecate Python 2 installations" #28411: We got some volunteers and will be reverting the change below. Spack will continue to support Python 2.7 and 3.5 for reproducibility.
• Deprecate Python 2 installations #28003: This PR deprecates versions of Python that have reached EOL (Python 2.7-3.5) and packages that require them. Note that this PR does not affect which versions of Python can be used to run Spack, it only affects which versions of Python Spack can install.

High-impact changes for Spack v0.17:

• specs: move to new spec.json format with build provenance #22845: This PR changes the spec.yaml format that Spack uses for hashes to spec.json. We're doing this for performance -- YAML is too slow for large operations, like reindexing the Spack database or creating indexes for large build caches. We expect this to speed up Spack in many ways, but it will cause the hashing algorithm to change.
• Make clingo the default solver #25502: This PR makes clingo the default solver, in preparation for 0.17.0. You may see different concretizations for some packages, and the new concretizer will need to be bootstrapped (see https://spack.io/changes-spack-v017/). You can switch back by setting concretizer: original in config.yaml, but note that the original concretizer will be deprecated in 0.17 and removed in a later version.
• Remove support for Python 2.6 #27256: Python 2.6 was deprecated in Spack v0.17.0 and support has been removed in develop -- it will not be supported in v0.18.

[haampie]

@tgamblin can you be a bit more precise about "hit develop soon"? Before or after 0.18 branches off?