Python: improve site_packages_dir handling

[adamjstewart]

This PR includes the following changes. I'll do my best to provide the background and motivation behind these changes. Some of these changes were extracted from #27798, which will be rebased after this is merged.

• Query sysconfig instead of distutils.sysconfig

In Python 3.10, the distutils module has been deprecated. In Python 3.12, it will be removed entirely. To future-proof our Python build system, we should use sysconfig instead (added in Python 2.7 and 3.2). Also, the Debian/Ubuntu system Python contains a bug where distutils.sysconfig and sysconfig return different purelib/platlib paths. Since build backends like setuptools use sysconfig, we also need to use sysconfig for compatibility.

• Add both purelib and platlib to the PYTHONPATH

On the majority of systems, purelib and platlib are identical, providing the location where third-party Python libraries are installed. Historically, we've been calling this site_packages_dir. However, when using the system Python on RHEL/CentOS/Fedora, purelib uses lib while platlib uses lib64. Different packages will end up in different directories depending on whether they contain C/C++ code or are written in pure Python. Since Spack has no way of knowing this, we should always add both purelib and platlib to the PYTHONPATH.

• Fix bug in which dependencies are added to PYTHONPATH

Previously we were recursively adding all build/run/test dependencies. However, this caused build deps of build deps to be added. The correct solution is to add all direct build/run/test deps, and all recursive run deps of those.

External references

• https://discuss.python.org/t/understanding-site-packages-directories/12959
• https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/
• https://docs.python.org/3/library/sysconfig.html#installation-paths
• How does setuptools determine the appropriate site-packages directory? pypa/setuptools#2813

Fixes #15304
Fixes #20043
Fixes #22299
Fixes #24076
Fixes #26546
Fixes #27497

[spackbot-app]

Hi @adamjstewart! I noticed that the following package(s) don't yet have maintainers:

• hoomd-blue
• py-pyqt5
• qscintilla

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers = ['adamjstewart']

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame hoomd-blue

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

[becker33]

Some comments, may or may not be worth addressing but I'll approve and defer to you on whether to make any changes before merging.

[adamjstewart]

@becker33 ready for another round of review. All usage of site_packages_dir has been replaced by either python_purelib or python_platlib. I rather arbitrarily decided between purelib and platlib based on whether the package had non-Python dependencies or not. As to whether these packages follow that rule or not, only CI will tell...

[adamjstewart]

@spackbot run pipeline

[spackbot-app]

I've started that pipeline for you!

[becker33]

Assuming the pipeline comes back successful, I think the platlib vs purelib decisions look reasonable.

[adamjstewart]

Failing build systems check in GitLab CI will be fixed by #28360.

[adamjstewart]

@becker33 @scheibelp is this good to go?