Environment activation is slow

[adamjstewart]

Activating/deactivating Spack environments is incredibly slow:

$ time spack env activate .

real	2m13.037s
user	1m25.584s
sys	0m43.654s
$ time spack env deactivate

real	2m30.974s
user	1m38.090s
sys	0m49.781s

For comparison, for a similarly sized Conda environment:

$ time conda activate azureml_py38

real	0m0.099s
user	0m0.081s
sys	0m0.018s

Unfortunately, pyinstrument doesn't work for bash functions (which are required for env support), and spack --profile doesn't work either.

May be related to #25541, #25306

[adamjstewart]

This is with a fairly large environment:

spack:
  specs:

  # Shells
  - bash

  # Spack dependencies
  - clingo
  - gnupg
  - graphviz
  - kcov

  # Linux tools
  - watch
  - wget

  # Compression
  - p7zip
  - unrar

  # Software installation tools
  - automake
  - autoconf
  - m4
  - cmake
  - ninja
  - patchelf
  - scons

  # Version control systems
  - mercurial
  - subversion

  # Python libraries
  - python
  - py-azureml-sdk
  - py-black
  - py-cartopy
  - py-cmocean
  - py-codecov
  - py-dask
  - py-fiona
  - py-flake8
  - py-geocube
  - py-geopandas
  - py-geoplot
  - py-inference-schema+numpy
  - py-ipywidgets
  - py-isort+colors
  - py-joblib
  - py-jupyterlab
  - py-matplotlib
  - py-metpy
  - py-mypy
  - py-numpy
  - py-openpyxl
  - py-pandas
  - py-pycocotools
  - py-pygeos
  - py-pyinstrument
  - py-pytest
  - py-pytest-cov
  - py-pytest-mock
  - py-pytorch-sphinx-theme
  - py-pyyaml
  - py-radiant-mlhub
  - py-rarfile
  - py-rasterio
  - py-scikit-learn
  - py-scipy
  - py-seaborn
  - py-setuptools
  - py-shapely
  - py-sphinx
  - py-sphinx-rtd-theme
  - py-sphinxcontrib-programoutput
  - py-statsmodels
  - py-tables
  - py-torch
  - py-torchvision
  - py-twine
  - py-vermin
  - py-wheel
  - py-xarray
  - py-xgboost

  # Research libraries
  - gdal
  - opencv+imgcodecs+python3+tiff+jpeg+png  # no idea why it won't take this from packages.yaml

  concretization: together

[adamjstewart]

From running spack -d, it looks like the issue is that we are running the code I added in #24095 for every Python package. Coupled with #25306, we are running this recursively for every package.

In the short term, I think we can just cache that result. In the long term, we need to do something differently in that code anyway because distutils is being removed.

@alalazo

[alalazo]

In the long term, we need to do something differently in that code anyway because distutils is being removed.

I was thinking that it might be difficult to settle on a single strategy for all the Python versions we support and all the Python packages, so maybe we should:

1. Design an approach that could account for different strategies (distutils + ?)
2. Have some way to dispatch this call to the right strategy depending on a) Python version in the DAG b) Metadata in the package