Spack not really usable by non-root users

[m4r1k]

Describe the bug

In a freshly built environment, when I try to install new software through Spack, I need to run it with sudo (just like any other package manager) otherwise the I get some basic filesystem permission issues

==> Waiting for intel-mpi-2018.4.274-7d33ducjgmxpdqaksszmp3bplefkug6x
==> Error: Failed to acquire a write lock for intel-mpi-2018.4.274-7d33ducjgmxpdqaksszmp3bplefkug6x due to LockROFileError: Can't take write lock on read-only file: /sw/spack/opt/spack/.spack-db/prefix_lock
==> Error: Can't take write lock on read-only file: /sw/spack/opt/spack/.spack-db/prefix_lock

I guess, then the question is: is Spack supposed to be consumed by regular end-users?

• if the answer is yes, then we need to fix the permission of the entire /sw/spack directory (consistently I'd like to add)
• if the answer is no, then, 1st we need to document how to run Spack (likely sudo -E) and then try to add it to the path while sudoing without forcing users to use full paths

[admin@internalhp-login-4sses53j-001 ~]$ which spack
/sw/spack/bin/spack
[admin@internalhp-login-4sses53j-001 ~]$ spack --version
0.18.1
[admin@internalhp-login-4sses53j-001 ~]$ sudo spack --version
sudo: spack: command not found
[admin@internalhp-login-4sses53j-001 ~]$ sudo -E spack --version
sudo: spack: command not found
[admin@internalhp-login-4sses53j-001 ~]$ sudo env|grep -i spack
[admin@internalhp-login-4sses53j-001 ~]$ sudo -E env|grep ^PATH
PATH=/sbin:/bin:/usr/sbin:/usr/bin
[admin@internalhp-login-4sses53j-001 ~]$ sudo -E env|grep -i spack
SPACK_ROOT=/sw/spack
__LMOD_REF_COUNT_MODULEPATH=/sw/spack/share/spack/modules/linux-centos7-x86_64:1;/sw/spack/share/spack/modules/linux-centos7-x86_64_v3:1;/etc/modulefiles:1;/usr/share/modulefiles:1;/opt/apps/modulefiles:1
SPACK_LD_LIBRARY_PATH=/usr/local/cuda/lib64:
MODULEPATH=/sw/spack/share/spack/modules/linux-centos7-x86_64:/sw/spack/share/spack/modules/linux-centos7-x86_64_v3:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
SPACK_PYTHON=/usr/bin/python3
[admin@internalhp-login-4sses53j-001 ~]$ sudo -E /sw/spack/bin/spack compilers
==> Available compilers
-- clang centos7-x86_64 -----------------------------------------
clang@3.4.2

-- gcc centos7-x86_64 -------------------------------------------
gcc@12.1.0  gcc@4.8.5

Please note that from sudo -E, spack is missing from the PATH due to the sudo enforcing the default secure_path

WDYT?

[cboneti]

Hi Federico,

Historically, we have been running spack as root, but we understand that may not be the best / recommended way.
I will do some more research internally and come back to you this week.

[nick-stroud]

There are two different issues here.

1. Spack is not in your path. The intention is for the spack binary to be in your path even when not acting as root. These lines are meant to accomplish this. The problem is that if Spack is installed from one machine and used from a different machine these lines do not run on user login and the spack setup never happens. When setup properly you should be able to use spack as non-root users (just not install new packages). A short term solution is to add a startup script to the machine on which you want to use spack (example):

      - type: data
        destination: /etc/profile.d/spack.sh
        content: |
          #!/bin/sh
          if [ -f /sw/spack/share/spack/setup-env.sh ]; then
                  . /sw/spack/share/spack/setup-env.sh
          fi

This should allow any user to use installed spack packages

In the near term we can add functionality to the spack-install module to provide the above runner as an output from the spack-install module. Longer term we would like to have a solution that does not require adding a startup script, but this will take some design and has not yet been committed to.

2. General permissions: Currently you must be root to install packages. We may consider adding configurable options to the spack-install module that would allow 1. all users to install packages or 2. allow the user to provide an oslogin user group for users that are allowed to install packages. These options require some design and have not yet been committed to.

[nick-stroud]

With #586, the docs have been updated and the near term solution for "Spack not in path" has been implemented on develop. We will leave this issue open until the update has made it into a formal release.

[tpdownes]

Related to this issue, I have observed some time delays between when I deactivate one environment and when I am able to activate the next. They generate filesystem errors for approx. 15 seconds until suddenly I am able to.

(At this point the "openfoam" environment is active)
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env deactivate 
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env activate openfoam2
==> Error: [Errno 13] Permission denied: '/sw/spack/var/spack/environments/openfoam2/.spack-env'
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env activate openfoam2
==> Error: [Errno 13] Permission denied: '/sw/spack/var/spack/environments/openfoam2/.spack-env'
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env activate openfoam2
==> Error: [Errno 13] Permission denied: '/sw/spack/var/spack/environments/openfoam2/.spack-env'
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env activate openfoam2
==> Error: [Errno 13] Permission denied: '/sw/spack/var/spack/environments/openfoam2/.spack-env'
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ ls -lh /sw/spack/var/spack/environments/openfoam2
total 80K
-rw-r--r-- 1 root root 75K Oct  6 15:44 spack.lock
-rw-r--r-- 1 root root 455 Oct  6 15:44 spack.yaml
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ ls -lha /sw/spack/var/spack/environments/openfoam2
total 92K
drwxr-sr-x 3 root root 4.0K Oct  6 15:44 .
drwxr-sr-x 4 root root 4.0K Oct  6 13:57 ..
drwxr-sr-x 5 root root 4.0K Oct  6 15:44 .spack-env
-rw-r--r-- 1 root root  75K Oct  6 15:44 spack.lock
-rw-r--r-- 1 root root  455 Oct  6 15:44 spack.yaml
[ext_tpdownes_google_com@amdv5-login-xvv0qv7o-001 ~]$ spack env activate openfoam2
(success)

[tpdownes]

Very likely related to my comment and probably to the original issue raised:

spack/spack#17407

Perhaps in particular:

spack/spack#17407 (comment)

[tpdownes]

If you follow the issue above, it appears to have been resolved on the Spack development branch for inclusion in a tagged release. I have asked whether that will be the next patch release of v0.18 or a not-yet-planned v0.19.

[tpdownes]

The team discussed this issue and we believe that the issue I mention in #539 (comment) is probably unrelated.

We feel that this issue has been resolved in release v1.7.0 through the following changes:

• the addition of a Toolkit runner which sets up /etc/profile.d to include a script that sources the Spack initialization script if it is present
  • See example usage

The primary use of this runner is to enable non-root users to use Spack on a shared filesystem. However, this enables the execution of spack commands by root if you invoke sudo -i spack .... Additionally, you can explicitly include the sourcing in custom shell scripts.