spack ci rebuild: Don't install in a login shell

[blue42u]

On some systems (e.g. OpenSUSE Leap 15) running the shell in login mode wipes important parts of the environment, such as PATH. Currently spack ci rebuild runs it's generated install.sh script in a login shell, which causes all the build-jobs to fail with a mysterious spack: command not found error.

This hotpatch changes to run the install.sh script in a normal shell without any extra modifications.

[blue42u]

/cc @scottwittenburg as original author for this code

(I don't know why GitHub randomly cancels my checks sometimes...)

[scottwittenburg]

@blue42u as I recall, we used the login shell when we were working on making failed gitlab builds reproducible outside of gitlab. When you're in your docker reproducer container, you'll run the install.sh script and that will be happening in a login shell, and we wanted the two environments to be as similar as possible.

@becker33 originally suggested this approach, so maybe he'll have some suggestion to add.

But at least one other person has complained about how the install is run in a login shell, so maybe we can do without it. However, unless this PR actually rebuilds some specs, the gitlab pipeline won't help us know whether the change is going to work. So maybe a WIP commit to add a new spec to one of the tested stacks, or some WIP change to one of the package files will be needed.

[blue42u]

@scottwittenburg I pulled this commit out of my other WIP PR where I'm adding hpctoolkit-* stacks. The latest pipeline there successfully rebuilt a stack with OpenSUSE Leap 15.

[blue42u]

we used the login shell when we were working on making failed gitlab builds reproducible outside of gitlab. When you're in your docker reproducer container, you'll run the install.sh script and that will be happening in a login shell, and we wanted the two environments to be as similar as possible.

Thinking about this a bit more, I think using a login shell actually makes this "reproducibility" harder in practice. In a full CI job the final spack install is run in an environment formed by scripts running in this order:

export AWS_SECRET=...                  # 1. Load environment from GitLab project variables
source spack/share/spack/setup-env.sh  # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env     # 3. Activate the concrete environment
source /etc/profile                    # 4. Bash login shell (from -l)
spack install ...

Whereas when a user launches their own container with (docker|podman) run -it, they end up running spack install in an environment formed in this order:

source /etc/bash.bashrc                # (not 4). Bash interactive shell (default with TTY)
export AWS_SECRET=...                  #~1. Manually load environment from GitLab project variables
source spack/share/spack/setup-env.sh  # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env     # 3. Activate the concrete environment
spack install ...

The big problem being that (4) has a completely different position and content (on Leap 15 and possibly other containers).

So in context, this PR removes (4) from the CI job case, leaving us with the simpler:

export AWS_SECRET=...                  # 1. Load environment from GitLab project variables
source spack/share/spack/setup-env.sh  # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env     # 3. Activate the concrete environment
spack install ...

[becker33]

I find this argument convincing that I was wrong to put it in a login shell to begin with.

[becker33]

@scottwittenburg I'll leave this to you to merge when you're convinced that this is suitably tested from #31598

[blue42u]

Rebased onto latest develop, no functional change.

[blue42u]

#31598 was also rebased onto this PR last night, and the latest pipeline from it rebuilt (part of) the HPCToolkit stacks successfully on a range of OSs: https://gitlab.spack.io/spack/spack/-/pipelines/147102

[becker33]

Thanks @blue42u