treewide: `set -u` everywhere #72347

Ericson2314 · 2019-10-31T02:33:42Z

2 years or so ago I made stdenv and cc-wrapper use set -u, which enormously helped with debugging. But I didn't want to change all of nixpkgs so I added lots of temporary set +u-ing.

Well, I think it's now time to bring the benefit to all of nixpkgs. Why? Because we have another big repo-wide refactor going on #72074. This will make that much easier to debug, and hopefully get some momentum going for stdenv tech debt fixing. Everyone chip in!

In particular, I think this is the easiest stdenv refactor we'd like to make, Why? This has the nice property that programs that continue to not fail have the same behavior. [Actually, this is not true with trap to catch failures or no set -e, but we use set -e and still fail after trap-ing.] That means it's very easy to debug; no getting a bunch of things working only to find some deeper dependency was built wrong and needing to backtrack, and no needing to debug complex failures.

I swear there was an issue for thus, but I couldn't find it.

Hydra job: https://hydra.nixos.org/jobset/nixpkgs/bash-no-undef-vars#tabs-evaluations

TODO:

Fix the broken things
Write release notes

Mic92 · 2019-10-31T09:36:22Z

This should be also mentioned in the release notes as it can break out-of-tree packages.

Ericson2314 · 2019-10-31T12:56:39Z

Wow, is there no way to mark a PR draft after the fact?!?! Embarrising, GitHub.

Ericson2314 · 2019-11-01T23:00:45Z

Hydra reached the point where the legitimate failures were buried amidst all sorts of spurious failures. This makes it harder to use it. I think it might be ready to go to staging then, at least after this these newer evals get further? @globin @matthewbauer What do you think?

Ericson2314 · 2019-11-02T00:18:37Z

https://hydra.nixos.org/eval/1552118?compare=1551936&full=1#tabs-now-fail here is a comparison to the base commit, which is a nixpkgs-unstable eval.

My earlier sed missed this because it already had `{..}`.

pkgs/build-support/setup-hooks/multiple-outputs.sh

I don't want to just rebase this away because the original commit is also in #72074.

FRidh · 2019-11-05T15:23:46Z

Darwin fails on the bash-no-undef-vars branch https://hydra.nixos.org/build/105765822

Ericson2314 · 2019-11-05T19:17:25Z

@FRidh that is a commit pushed post merge. The job is now just for #72812 . Do ping me with any/all set -u related failures on staging though, and I will try to watch it too.

nixos-discourse · 2019-11-13T11:26:17Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-weekly-14-hercules-ci-launch-performance-improvements-in-nixpkgs-single-dependency-kubernetes-clusters/4769/1

Mic92 · 2019-11-19T10:57:42Z

qt needs a fix: #73586 (comment) in pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh

untested patch:

diff --git a/pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh b/pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh
index 35982e86628..0ad185a16eb 100644
--- a/pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh
+++ b/pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh
@@ -77,6 +77,8 @@ ecmUnseenHostPath() {
 ecmHostPathHook() {
     ecmUnseenHostPath "$1" || return 0
 
+    declare -a qtWrapperArgs
+
     local xdgConfigDir="$1/etc/xdg"
     if [ -d "$xdgConfigDir" ]
     then

FRidh · 2019-11-19T13:37:02Z

qtWrapperArgs is fixed with ef43c5f. Now there are issues left with the setup hooks of both plasma5 packages and kdeFrameworks packages.
pkgs/desktops/plasma-5/default.nix
pkgs/development/libraries/kde-frameworks/default.nix
Looked into this briefly, #73756

Ericson2314 · 2019-11-20T01:17:20Z

Actually my commit could not be enough, but @Mic92's won't work either. The issue is bash says an undefined variable and and undefined variable are the same thing (which is terrible!!!).

The direct workaround is awful ${foo+${foo[@]}}, but thankfully there are other ways like ${#foo[@]} to get the same result.

Bash takes an assignment of a string to an array variable: local -a user_args user_args="(foo bar)" to mean appending the string to the array, not parsing the string into an array as is the case when on the same line as the declaration: local -a user_args="(foo bar)" b063340 extracted the declaration before the newly branched code block, causing string makeWrapperArgs being added to the array verbatim. Since local is function scoped, it does not matter if we move it inside each of the branches so we fix it this way.

jtojnar · 2019-12-29T18:04:59Z

Hmm, it appears this is not enough. For example wrapPython setup hook is running with +o nounset: a6bb2ed#commitcomment-36605469

In fact, when I comment out

nixpkgs/pkgs/stdenv/generic/setup.sh

Lines 1307 to 1308 in 4c7a9a0

    
           # Disable nounset for nix-shell. 
        
           set +u

and try to build python3.pkgs.setuptools, it fails on trying to build xz:

/nix/store/aknix5zw9cj7hd1m3h1d6nnmncl1vkvn-multiple-outputs.sh: line 55: shareDocName: unbound variable

Ericson2314 · 2019-12-29T18:41:43Z

Indeed the work isn't done! https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh#L1307-L1308 needs to be guarded by IN_NIX_SHELL so it doesn't mess up normal builds.

I should have made an issue earlier when I realized this, but didn't have the time to immediately start fixing it.

Ericson2314 requested review from matthewbauer and ttuegel as code owners October 31, 2019 02:33

Ericson2314 changed the base branch from master to staging October 31, 2019 02:34

Ericson2314 mentioned this pull request Oct 31, 2019

makeWrapper doesn't work with set +u #35164

Closed

ofborg bot added 6.topic: qt/kde 6.topic: stdenv Standard environment labels Oct 31, 2019

Ericson2314 changed the title ~~treewide: set -u everywhere~~ treewide: set -u everywhere Oct 31, 2019

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 10.rebuild-linux-stdenv This PR causes stdenv to rebuild 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Oct 31, 2019

This comment has been minimized.

Sign in to view

Ericson2314 changed the title ~~treewide: set -u everywhere~~ WIP: treewide: set -u everywhere Oct 31, 2019

c0bw3b added the 2.status: work-in-progress This PR isn't done label Oct 31, 2019

Ericson2314 requested review from alyssais, FRidh, jonringer and zimbatm as code owners October 31, 2019 18:05

ofborg bot added 6.topic: python 6.topic: ruby labels Oct 31, 2019

Ericson2314 force-pushed the bash-no-undef-vars branch from 7fe5b45 to b27cf98 Compare October 31, 2019 20:25

Ericson2314 requested review from LnL7 and Mic92 as code owners November 1, 2019 15:10

Ericson2314 force-pushed the bash-no-undef-vars branch from 182ce64 to 7a97e90 Compare November 1, 2019 15:27

ofborg bot added 6.topic: rust 2.status: merge conflict This PR has merge conflicts with the target branch labels Nov 1, 2019

ofborg bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation This PR adds or changes documentation labels Nov 1, 2019

Ericson2314 changed the title ~~WIP: treewide: set -u everywhere~~ treewide: set -u everywhere Nov 2, 2019

qtbase-setup-hook: postPhases may be undefined

9b274e2

My earlier sed missed this because it already had `{..}`.

matthewbauer reviewed Nov 4, 2019

View reviewed changes

pkgs/build-support/setup-hooks/multiple-outputs.sh Outdated Show resolved Hide resolved

matthewbauer approved these changes Nov 4, 2019

View reviewed changes

multiple-outputs setup hook: Remove accidentally added tab

c5cf009

I don't want to just rebase this away because the original commit is also in #72074.

Ericson2314 merged commit acd2d19 into staging Nov 5, 2019

Ericson2314 mentioned this pull request Nov 5, 2019

cc-wrapper, bintools-wrapper: Remove now unneeded set {+,-}u #72812

Merged

10 tasks

veprbl removed the 2.status: work-in-progress This PR isn't done label Nov 19, 2019

This was referenced Nov 24, 2019

nix-shell with coq+compcert will not start #74083

Closed

applgrid: fix build #74092

Merged

eraserhd mentioned this pull request Nov 25, 2019

java classpath hook: allow unbound $CLASSPATH #74093

Merged

10 tasks

peterhoeg mentioned this pull request Jun 5, 2020

Setting -u in stdenv to catch invalid variables #29296

Closed

veprbl mentioned this pull request Nov 9, 2020

darwin.adv_cmds: fix build #103180

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

treewide: `set -u` everywhere #72347

treewide: `set -u` everywhere #72347

Ericson2314 commented Oct 31, 2019 •

edited

Loading

This comment has been minimized.

Mic92 commented Oct 31, 2019 •

edited

Loading

Ericson2314 commented Oct 31, 2019

Ericson2314 commented Nov 1, 2019 •

edited

Loading

Ericson2314 commented Nov 2, 2019

FRidh commented Nov 5, 2019

Ericson2314 commented Nov 5, 2019 •

edited

Loading

nixos-discourse commented Nov 13, 2019

Mic92 commented Nov 19, 2019 •

edited

Loading

FRidh commented Nov 19, 2019

Ericson2314 commented Nov 20, 2019

jtojnar commented Dec 29, 2019

Ericson2314 commented Dec 29, 2019

treewide: set -u everywhere #72347

treewide: set -u everywhere #72347

Conversation

Ericson2314 commented Oct 31, 2019 • edited Loading

This comment has been minimized.

Mic92 commented Oct 31, 2019 • edited Loading

Ericson2314 commented Oct 31, 2019

Ericson2314 commented Nov 1, 2019 • edited Loading

Ericson2314 commented Nov 2, 2019

FRidh commented Nov 5, 2019

Ericson2314 commented Nov 5, 2019 • edited Loading

nixos-discourse commented Nov 13, 2019

Mic92 commented Nov 19, 2019 • edited Loading

FRidh commented Nov 19, 2019

Ericson2314 commented Nov 20, 2019

jtojnar commented Dec 29, 2019

Ericson2314 commented Dec 29, 2019

treewide: `set -u` everywhere #72347

treewide: `set -u` everywhere #72347

Ericson2314 commented Oct 31, 2019 •

edited

Loading

Mic92 commented Oct 31, 2019 •

edited

Loading

Ericson2314 commented Nov 1, 2019 •

edited

Loading

Ericson2314 commented Nov 5, 2019 •

edited

Loading

Mic92 commented Nov 19, 2019 •

edited

Loading