Unit testing for setup scripts

[adamjstewart]

We should add unit tests for our setup scripts (setup-env.sh and setup-env.csh). These tests would serve 2 purposes:

• Check for POSIX compliance (does the setup-env.sh script work for Bash, Zsh, Ksh, Dash, etc.)
• Check for uniformity (does setup-env.sh work the same was as setup-env.csh, do they work as intended)

A lot of PRs introduce new features that seem fine to me, but I don't know whether they work in other shells (#7475). #4048 adds Dash support but no unit tests to see if it works. Other PRs add awesome new features to setup-env.sh but forget to add them to setup-env.csh (#5955).

Recently, we have seen requests for support for other shells like Fish (#7311). It will be important to have a testing script that works regardless of shell before we add a setup-env.fish file.

Thoughts on where to start? #4048 (comment) recommends ShellCheck for testing POSIX compliance. I would also want a function that tests that it is not only compliant but also works as intended. This should work the same for setup-env.sh, setup-env.csh, and setup-env.fish.

Another step we can take is to make lib/spack/env/cc POSIX compliant. If we replace the shebang with #!/bin/sh, we could see a significant speed boost on Ubuntu which uses Dash instead of Bash. But of course, this requires unit tests to keep it from breaking.

We may also need to consider unit tests for our tab completion scripts (spack-completion.bash), especially if we add one for Zsh (#2356).

[adamjstewart]

Found an interesting unit testing framework that supports all POSIX shells: https://github.com/kward/shunit2

[JBlaschke]

@adamjstewart Is there any progress on this? #9279 would really benefit from some unit tests that I can run against my attempt at setup-env.fish

[tgamblin]

@JBlaschke: FWIW, we already test some of the shell support in share/spack/qa/setup-env-test.sh. It's just for bourne-based shells, but you could copy the types of tests that are there for fish.

[JBlaschke]

@tgamblin Thanks. I've built a share/spack/qa/setup-env-test.fish. It does the same tests as the bash version. Since fish handles the user-prompt differently the test for SPACK_OLD_PS1 fails, so I removed it. Otherwisesetup-env-test.fish is a fishy mirror of setup-env-test.sh. If you ran it with fish, you should get:

Cleanup
---------------------------------
Removing test packages before exiting.
==> Successfully uninstalled a@2.0%clang@10.0.0-apple+bvv foo=bar foobar=bar arch=darwin-mojave-x86_64/sdnxztz
==> Successfully uninstalled b@1.0%clang@10.0.0-apple arch=darwin-mojave-x86_64/wzg4pti

44 tests succeeded.
0 tests failed.
Exited due to an error.
SUCCESS