Modules: /opt/homebrew should be in list of system paths on macOS - or not?

[climbfuji]

Steps to reproduce

On macOS, specifically on the M1 architecture, /opt/homebrew should be in the system paths:

diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
index da6832e4dd..8d60f108e8 100644
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -36,7 +36,7 @@
     ]
     SUFFIXES = []
 else:
-    SYSTEM_PATHS = ["/", "/usr", "/usr/local"]
+    SYSTEM_PATHS = ["/", "/usr", "/usr/local", "/opt/homebrew"]
     SUFFIXES = ["bin", "bin64", "include", "lib", "lib64"]

 SYSTEM_DIRS = [os.path.join(p, s) for s in SUFFIXES for p in SYSTEM_PATHS] + SYSTEM_PATHS

The reason is that if you have things like git-lfs, mysql installed there (which are a nightmare to build on macOS in spack), then the current spack develop code puts the path /opt/homebrew/bin in front of other paths when the spack-generated modules are loaded. If one has a python3 in /opt/homebrew/bin, then suddenly that python3 gets found again, even though the spack python3 module was loaded (first). The above change fixes that.

Error message

See description above

Information on your system

* **Spack:** 0.21.0.dev0 (98957cf32aa809e831bee4af2bf642e2bd038c04)
* **Python:** 3.8.9
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[climbfuji]

I am finally coming back to this - one thought that I had based on our discussion in the spack user meeting a few weeks ago is that we could change the spack logic for appending/prepending to the various PATH variables as follows: if it’s a spack-built package, prepend; if it’s an external dependency, append. This way, spack-built packages would always take precedence over homebrew and otherwise installed packages. It’s likely that I am missing something in my thinking, though.

[climbfuji]

There is also PR #35737 ("Improve setup/build/run environment") that could consider and solve this problem.

[haampie]

Can you give more details? There shouldn't be modules for externals, so what's adding this into PATH?

[haampie]

FWIW, I'd rather see fewer than more system paths. At the end of the day it's a matter of prioritizing Spack prefixes over external prefixes, Spack doesn't have to know what a system path is to do that.

[climbfuji]

Can you give more details? There shouldn't be modules for externals, so what's adding this into PATH?

I talked with @alalazo and @scheibelp about this issue during our weekly Wednesday meetings, it might be more efficient if you have a chat with them. I don't understand "There shouldn't be modules for externals", though. If I ask spack to create modules, tcl or lmod, it is creating modules for external packages from homebrew unless I explicitly exclude each of them in the config:modules:tcl|lmod:exclude list. Or is there a switch that I can make that suppresses module generation for all external modules?

[haampie]

it is creating modules for external packages

Hm okay, I thought we excluded those by default.

In that case it's a load order issue, which is a bit annoying cause that's determined by the module command.

[climbfuji]

it is creating modules for external packages

Hm okay, I thought we excluded those by default.

In that case it's a load order issue, which is a bit annoying cause that's determined by the module command.

Correct, and therefore my (probably naive) thinking that appending for externals and prepending for spack-built packages would work. In our case, not creating modules for any external package would work, but maybe that's not the case for everyone.

[haampie]

Yeah, that could work too.

[alalazo]

Prepending has issues too (but might be better than the current state?). The fact here is that both system paths and /opt/homebrew contain multiple, unrelated software. So, no matter the order, we might have some executable in the path that has precedence mask an executable we'd like to use in the lower priority path. 🤔

[haampie]

prepend-path SPACK_PATH /spack/pkg-1.0.0-hash/bin
append-path SPACK_PATH /opt/homebrew/bin

if [module that is being loaded]
   prepend-path PATH $SPACK_PATH

;p that's ugly, but should be roughly equivalent to how we partition externals in spack load / env activate ... if that branch is expressable...

[alalazo]

To explain better what I mean by #39046 (comment) Assume that two applications, say python3 and tar are both in /usr/bin and in /opt/homebrew/bin. There is no way we can set the PATH to get one from the former directory and the other from the latter, without creating additional dirs with symlinks.

[haampie]

sure, but that's unsolvable

[scheibelp]

I've updated the title and tags to reflect the fact that this is only an issue with generated module files.

The environment that Spack sets up for Spack installs generally avoids these types of issues (e.g. making sure that Spack-built prefixes appear first, followed by externals). Perhaps the modules generation could use the same approach.

I think that depending on module load order, there might still be an issue. IIRC, one possibility discussed for modules was to always prepend Spack-built prefixes and always append external prefixes, which might resolve that.

[haampie]

The environment that Spack sets up for Spack installs generally avoids these types of issues (e.g. making sure that Spack-built prefixes appear first, followed by externals). Perhaps the modules generation could use the same approach.

#39046 (comment) this is likely closest, like PATH=$spack_prefixes:$external_prefixes:$PATH instead of PATH=$spack_prefixes:$PATH:$external_prefixes.