pythonPackages.Babel: Fix use of glibcLocales, skip musl tests

[nh2]

Fixes #74904.

The test failure:

File "/build/Babel-2.7.0/babel/messages/pofile.py", line 325, in _invalid_pofile
  print(u"WARNING: Problem on line {0}: {1}".format(lineno + 1, line))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 84: ordinal not in range(128)

occurs in either of these cases:

• glibc is used and glibcLocales is not a buildInput
• musl is used and tests are run

python3Packages.Babel does not have this problem, on either glibc
or musl.

I also found that the preCheck export LC_ALL="en_US.UTF-8" is not
necessary in any case.

With this commit,

python27Packages.Babel
python3Packages.Babel
pkgsMusl.python27Packages.Babel
pkgsMusl.python3Packages.Babel

all build.

Motivation for this change

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

No maintainers listed.

FYI @dtzWill from #74904.

[nh2]

@GrahamcOfBorg build python27Packages.Babel python3Packages.Babel pkgsMusl.python27Packages.Babel pkgsMusl.python3Packages.Babel

Can somebody please ofborg-build this on OSX for me?

[nh2]

Looks like Babel triggers a mass rebuild, will change to staging.

[nh2]

@GrahamcOfBorg build python27Packages.Babel python3Packages.Babel pkgsMusl.python27Packages.Babel pkgsMusl.python3Packages.Babel

[nh2]

I can't figure it out

OK I could repro it at least on one of my machines now -.-

[jonringer]

could also be:

  preCheck = lib.optionalString isPy27 ''
    export LC_ALL=C.UTF-8
  '';

[nh2]

I have no opinion on that; I wanted to express that "versions < 3" get this wrong, and isPy27 seems more specific than that. But at the same time, is the only < 3 version available in nixpkgs and no 2.8 is supposed to be released, so it may as well be equivalent.

Whichever you think is better!

[jonringer]

I don't have a strong opinion either

[nh2]

I suspect the unicode error difference occurs depending on the version of nix!

With nix 2.0.4 on NixOS 18.03 in the nixpkgs checkout of this PR, I can reproduce:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 84: ordinal not in range(128)

But if I build with nix 2.3.1 by running, on the same machine:

NIX_PATH=nixpkgs=. nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel'

then the build of the same .drv magically succeeds (bot nix versions produce the same .drv).

@edolstra Are you aware of any changes to nix that might induce this?

[nh2]

The difference is likely somewhere between nix 2.2.2 and nix 2.3:

# fails: current 19.03
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1dff0bcd9f8cd5cf8fca1ab8f08d55dff5c9c57.tar.gz nix-shell -p pkgs.nix --pure --run 'nix-build --version'
nix-build (Nix) 2.2.2
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1dff0bcd9f8cd5cf8fca1ab8f08d55dff5c9c57.tar.gz nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel'

# works: current 19.09
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/69ed29f5f41107aca5fb674218a4f75619c22207.tar.gz nix-shell -p pkgs.nix --pure --run 'nix-build --version'
nix-build (Nix) 2.3
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/69ed29f5f41107aca5fb674218a4f75619c22207.tar.gz nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel

[nh2]

Preliminary bisection results yield:

# possible first bad commit: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
# possible first bad commit: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
# possible first bad commit: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
# possible first bad commit: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
# possible first bad commit: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal

Full bisection log (initial skips generated with the technique shown in https://blog.smart.ly/2015/02/03/git-bisect-debugging-with-feature-branches/, using for rev in $(git rev-list 2.2.2..2.3 --merges --first-parent); do git rev-list $rev^2 --not $rev^; done | xargs git bisect skip):

# git bisect log
git bisect start
# good: [0baf7dc25e0a5fcd25dcd9d9cb737c682d01e4e7] Bump version
git bisect good 0baf7dc25e0a5fcd25dcd9d9cb737c682d01e4e7
# bad: [22d4ea7a989d26b86fc27706dfea0abd2fb52c52] Tweak release notes
git bisect bad 22d4ea7a989d26b86fc27706dfea0abd2fb52c52
# skip: [87c604c1f04ff31a68f2fa73fe5a2ed9f6ec7ac7] Fix launchd program args
git bisect skip 87c604c1f04ff31a68f2fa73fe5a2ed9f6ec7ac7
# skip: [800fba1037ad97ff5e6d9acf9e820a30f0922f59] Use wait4path on org.nixos.nix-daemon.plist
git bisect skip 800fba1037ad97ff5e6d9acf9e820a30f0922f59
# skip: [693e68e09c9a17ca72b074bd2e575bf435647b45] Set maximum name length in Nix
git bisect skip 693e68e09c9a17ca72b074bd2e575bf435647b45
# skip: [5bdac86be2eb6d541784c4f149bfa06b59ef63a2] Reword to clarify newer generations are left alone
git bisect skip 5bdac86be2eb6d541784c4f149bfa06b59ef63a2
# skip: [1dbaf119486fecbdf57ecc2f877307030b665f69] Fix nix-env documentation for --delete-generations
git bisect skip 1dbaf119486fecbdf57ecc2f877307030b665f69
# skip: [5011a52cf3f291c22d784aff6f5bbf8b99ae186a] Just enable hack on macOS
git bisect skip 5011a52cf3f291c22d784aff6f5bbf8b99ae186a
# skip: [4a3e96281d6b9ae3dbc20e638f389677df9a649e] Handle SIGWINCH in main thread
git bisect skip 4a3e96281d6b9ae3dbc20e638f389677df9a649e
# skip: [d854e7dfd6d512c8ed687b61d7aae3358eda71cd] install-multi-user: Detect and fail lack of systemd separately
git bisect skip d854e7dfd6d512c8ed687b61d7aae3358eda71cd
# skip: [0463d5e36ffdbb60f851bd3b34756eff84151b28] Allow empty /nix directory in multi-user installer
git bisect skip 0463d5e36ffdbb60f851bd3b34756eff84151b28
# skip: [177e5742fa732856f358cfb1b22b7575e94e9bb3] Merge pull request #3056 from grahamc/operators
git bisect skip 177e5742fa732856f358cfb1b22b7575e94e9bb3
# skip: [171d784404fefa7c91e2082f92ec92c39cc1aa8e] docs: operators: Make OR and AND capitalized
git bisect skip 171d784404fefa7c91e2082f92ec92c39cc1aa8e
# skip: [92ddce4f463dc3cf2327c832d45f62d985d5e09f] operators: document exact precedenc, split up similar operators
git bisect skip 92ddce4f463dc3cf2327c832d45f62d985d5e09f
# skip: [5c06a8d3283139140e765b5f10ad7102a6a3e964] Reset tmpDirInSandbox for unsandboxed
git bisect skip 5c06a8d3283139140e765b5f10ad7102a6a3e964
# skip: [9a0855bbb6546e792848e551e79f8efc40782eeb] Don’t rely on EPERM
git bisect skip 9a0855bbb6546e792848e551e79f8efc40782eeb
# skip: [11d853462925d0b57fe956962e07edf5751fd4c3] Use sandbox fallback when cloning fails in builder
git bisect skip 11d853462925d0b57fe956962e07edf5751fd4c3
# skip: [d171090530f4a2a79efec2c385bee1a10844c706] Disable CLONE_NEWUSER when it’s unavailable
git bisect skip d171090530f4a2a79efec2c385bee1a10844c706
# skip: [d45922472434783bdc29610479a735dd236229a3] nix-daemon.service: add install section.
git bisect skip d45922472434783bdc29610479a735dd236229a3
# skip: [057af1dbd802987df7990d965e08db185820a193] docs: document the installer's use of proxy env vars
git bisect skip 057af1dbd802987df7990d965e08db185820a193
# skip: [6dab42a551a769f304c2a0e66e8771d7289502f2] installer: handle network proxy in systemd install
git bisect skip 6dab42a551a769f304c2a0e66e8771d7289502f2
# skip: [b226b5cd976ca71abb3c0861b56d5e5940430924] nix-store: fix out of sync protocol
git bisect skip b226b5cd976ca71abb3c0861b56d5e5940430924
# skip: [91b00b145f0c50e346d0250168cbcbcba7aa3b40] libutil: add SizedSource
git bisect skip 91b00b145f0c50e346d0250168cbcbcba7aa3b40
# skip: [653c40778462b5921fbbe2abffd1a2822f08ec57] Expanded documentation for .nix-defexpr
git bisect skip 653c40778462b5921fbbe2abffd1a2822f08ec57
# skip: [ee9c988a1b2e3c511b8613e698a0f9632ab1538f] Track function start and ends for flame graphs
git bisect skip ee9c988a1b2e3c511b8613e698a0f9632ab1538f
# skip: [a02457db715a80de7630d823fd8d7230a4bd451f] conf: stalled-download-timeout: make tunable
git bisect skip a02457db715a80de7630d823fd8d7230a4bd451f
# skip: [c3fefd1a6efec457395a187d15f435447dee6f3b] pathlocks: add include to fcntl.h for O_CLOEXEC
git bisect skip c3fefd1a6efec457395a187d15f435447dee6f3b
# skip: [363a2f68261af73aefe4edead9c0526030751a27] post-build-hook: docs fixup
git bisect skip 363a2f68261af73aefe4edead9c0526030751a27
# skip: [7c5596734f65b30b250ea73a423b40a4ce513fdf] Add a post-build-hook
git bisect skip 7c5596734f65b30b250ea73a423b40a4ce513fdf
# skip: [cd933b22d2041b7efc348dcc09ff255967ffc663] Add pname and version to nix-env -q --json
git bisect skip cd933b22d2041b7efc348dcc09ff255967ffc663
# skip: [ee1e3132cab9d7de14837f2b04b115e544ae1622] Disable findRuntimeRoots on darwin when running tests because lsof is slow
git bisect skip ee1e3132cab9d7de14837f2b04b115e544ae1622
# skip: [89865144c3ba0162cd37bcbe49b3095e9bab4164] Allow builtins.pathExists to check the existence of /nix/store paths
git bisect skip 89865144c3ba0162cd37bcbe49b3095e9bab4164
# skip: [03addc3b0a79cb7a58ae5651d1915af7383a220c] Use $HOME instead of $USER
git bisect skip 03addc3b0a79cb7a58ae5651d1915af7383a220c
# skip: [c82a856b36bcd008edeb9c4c981b974ae642afea] Add default for USER when unset
git bisect skip c82a856b36bcd008edeb9c4c981b974ae642afea
# skip: [9031a6838ccf10b0db0d91a89e14d71a1adf6235] Remove .github/FUNDING.yml
git bisect skip 9031a6838ccf10b0db0d91a89e14d71a1adf6235
# skip: [cf6172f05e9446b2b85dd7fa2e73cb93f56620e2] docs: document balancing cores and max-jobs
git bisect skip cf6172f05e9446b2b85dd7fa2e73cb93f56620e2
# skip: [ec0087df0a2da5c68363b3f4509d4545deed97a0] Don’t use entire /etc/nsswitch.conf file
git bisect skip ec0087df0a2da5c68363b3f4509d4545deed97a0
# skip: [648bdf153d2e4d3c6687d8e1780b0dfd0aa98cda] tarball-ttl: document
git bisect skip 648bdf153d2e4d3c6687d8e1780b0dfd0aa98cda
# skip: [82b7f0e840983879a510245903ff7c917276f65d] autoconf: Implement release tarball detection. Fixes #257.
git bisect skip 82b7f0e840983879a510245903ff7c917276f65d
# skip: [cd8bc06e8786018ddb16cea4cb10971b63d0efd2] autoconf: Add comment on use of `false`.
git bisect skip cd8bc06e8786018ddb16cea4cb10971b63d0efd2
# skip: [c3db9e6f8fd06d691be04cdd95a6bb21a400481d] autoconf: Check if --nonet works. Fixes #967 #506.
git bisect skip c3db9e6f8fd06d691be04cdd95a6bb21a400481d
# skip: [a96006d97fc87c5073f9a39db841625bdb7c401c] Get BOOST_LDFLAGS from autoconf, fix Ubuntu 16.04 build.
git bisect skip a96006d97fc87c5073f9a39db841625bdb7c401c
# skip: [d203c554faa00fec55377d6640c8fb335a611a09] Fix C++ compatibility with older editline versions.
git bisect skip d203c554faa00fec55377d6640c8fb335a611a09
# skip: [b49c3a9db516a87d38476966eff9994c13c5acf6] Makefile.config.in: Remove HAVE_READLINE.
git bisect skip b49c3a9db516a87d38476966eff9994c13c5acf6
# skip: [717e821b99797845e1bef47d862f8cb0fb69cfc9] autoconf: Allow overriding CFLAGS/CXXFLAGS from outside.
git bisect skip 717e821b99797845e1bef47d862f8cb0fb69cfc9
# skip: [20129bd83d57316cc0b69bec7abeec43011c56d5] autoconf: Fix AC_STRUCT_DIRENT_D_TYPE being used before AC_PROG_CC.
git bisect skip 20129bd83d57316cc0b69bec7abeec43011c56d5
# skip: [fe068eca00d6a2dd2ea9ee08924dc3898c37fb6c] mk: add support for passing LDFLAGS to libs and bins
git bisect skip fe068eca00d6a2dd2ea9ee08924dc3898c37fb6c
# skip: [57daa860e8ed8432937aeecdcf6b9e952b0481b1] autoconf: Fix C++17 detection not working on Ubuntu 16.04.
git bisect skip 57daa860e8ed8432937aeecdcf6b9e952b0481b1
# skip: [1f97b16b1d9d93e083a4f1436ba002e073cbe379] autoconf: Work around editline not being found on Ubuntu 16.04.
git bisect skip 1f97b16b1d9d93e083a4f1436ba002e073cbe379
# skip: [00a450026f71306bef6acc150af4aa5ffe75801c] autoconf: Detect boost, require version, set CXXFLAGS.
git bisect skip 00a450026f71306bef6acc150af4aa5ffe75801c
# skip: [96cd3d607374cb4bab27095a62a6f3a832518c5a] autoconf: Change quotes in description.
git bisect skip 96cd3d607374cb4bab27095a62a6f3a832518c5a
# skip: [68bdd83dc88ec55c6c51fa92e84e7d7d408c554a] timeout: test for error code
git bisect skip 68bdd83dc88ec55c6c51fa92e84e7d7d408c554a
# skip: [a52c331edba6e8c67469f53dda0be2903d18fa8c] build: replace 100 offset for build exit codes
git bisect skip a52c331edba6e8c67469f53dda0be2903d18fa8c
# skip: [1ac399dd115d5c86629389e0cdfefa0d654fc90a] nix-store: document exit codes
git bisect skip 1ac399dd115d5c86629389e0cdfefa0d654fc90a
# skip: [99ee3755dd12e18f7085b8319a27798b0e5b7de9] build: add tests for --check status codes
git bisect skip 99ee3755dd12e18f7085b8319a27798b0e5b7de9
# skip: [cbf84bcce7d3fddb353d3aa0f012f5cbdbb77629] build: use binary mask for build status flags
git bisect skip cbf84bcce7d3fddb353d3aa0f012f5cbdbb77629
# skip: [97baf32fbca13101f58d30bb3a601302c3d560f3] build: add exit code for hash and check mismatches
git bisect skip 97baf32fbca13101f58d30bb3a601302c3d560f3
# skip: [17d3ec3405eb114f89dc42d065afa887161f6149] checkStoreName: give more precise/verbose error information
git bisect skip 17d3ec3405eb114f89dc42d065afa887161f6149
# skip: [a3c77c15369703975f7c3e65ebb829580754bc1e] nix-store: document --add-fixed
git bisect skip a3c77c15369703975f7c3e65ebb829580754bc1e
# skip: [c8205a3413217ccf8a6a1f7e064b06a5b86c3253] builtins.fetchGit: document absolute ref support
git bisect skip c8205a3413217ccf8a6a1f7e064b06a5b86c3253
# skip: [7e35e914c1aa24957107c666c76f1d834ebae90a] fetchGit: allow fetching explicit refs
git bisect skip 7e35e914c1aa24957107c666c76f1d834ebae90a
# skip: [ec58ba38c55a3ab61fdb1da7d746251e4887ea2c] Fix `http2 = false` having no effect. Fixes #2971.
git bisect skip ec58ba38c55a3ab61fdb1da7d746251e4887ea2c
# skip: [74a65d313f5c9f4d504ea9e2caadfe0ff83776d5] Nix uses the CPP SDK, not Java
git bisect skip 74a65d313f5c9f4d504ea9e2caadfe0ff83776d5
# skip: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
git bisect skip 3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2
# skip: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
git bisect skip 2743bf0bb1727056ce6d6d6be4e38c3251aac1f8
# skip: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
git bisect skip 82ca6ef390b752faf1bf27b22daa34b29a4b00d4
# skip: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal
git bisect skip e84c26564507388f2e8b7c7f2b00eb2b57856da1
# skip: [07d9981f34c8423e74a8dc1b4f978fe58b421aee] install-multi-user: remove unneeded settings from nix.conf
git bisect skip 07d9981f34c8423e74a8dc1b4f978fe58b421aee
# skip: [dbe4c043d75f90f30932d326d2c80fd030b1f06d] install-multi-user: reduce max-jobs from 32 to 1
git bisect skip dbe4c043d75f90f30932d326d2c80fd030b1f06d
# skip: [9e0f5f803f6cbfe9925cef69a0e58cbf5375bfaf] Daemon: warn when an untrusted user cannot override a setting
git bisect skip 9e0f5f803f6cbfe9925cef69a0e58cbf5375bfaf
# skip: [4b0d6133836113572418f383ddd14c92bfebc768] Minor typo
git bisect skip 4b0d6133836113572418f383ddd14c92bfebc768
# skip: [2d34028b1e4e46d600926d4ad0c687c4d1d58c75] Add .github/FUNDING.yml
git bisect skip 2d34028b1e4e46d600926d4ad0c687c4d1d58c75
# skip: [3b1cc8b0cbc0a2530e4671231257a497db8eede2] release-common: fix build with `config.allowAliases = false`
git bisect skip 3b1cc8b0cbc0a2530e4671231257a497db8eede2
# skip: [a8251ba2ed9e833ef3e8b20d233064aba74446db] Replace `type` with `command -v` in install script
git bisect skip a8251ba2ed9e833ef3e8b20d233064aba74446db
# skip: [b2f3a7411a509d5df5a189066870a0f02b6523a9] nix-lang: Add deep nested mixed attrs test case.
git bisect skip b2f3a7411a509d5df5a189066870a0f02b6523a9
# skip: [00584bb0915ee21fb01e9390a901161bdd988197] parser: Allow mixed nested and top-level attrpaths
git bisect skip 00584bb0915ee21fb01e9390a901161bdd988197
# skip: [10d33452e289ded93e192c7d99c1da08a52448e1] nix-lang parser: Add mixed nested attrs tests.
git bisect skip 10d33452e289ded93e192c7d99c1da08a52448e1
# skip: [b502b6682b9c73ae5760967eaf7161b3e8523e9e] doc: clarify that optional attrs in a function argument will be ignored unless specified
git bisect skip b502b6682b9c73ae5760967eaf7161b3e8523e9e
# skip: [6ade7ec022c836b7d1f9bd06be45e2c07835ec8c] progress-bar: hide expected if expected is 0 (unknown)
git bisect skip 6ade7ec022c836b7d1f9bd06be45e2c07835ec8c
# skip: [92f461e4f46b595e3712b41c30b8403905dd8673] Don’t set NIX_REMOTE=daemon in daemon profile
git bisect skip 92f461e4f46b595e3712b41c30b8403905dd8673
# skip: [7c20ee448fa924d898bcebf84bd0a7caf368a656] Sync NIX_PROFILES between single-user and multi-user modes
git bisect skip 7c20ee448fa924d898bcebf84bd0a7caf368a656
# skip: [66b8a62101cb1dfe2e368346cf99efd32e9328ae] nix: Add --print-build-logs flag
git bisect skip 66b8a62101cb1dfe2e368346cf99efd32e9328ae
# skip: [ce02fc74b2db35e45906865c8a3ce2e98871eeb8] build: make needsHashRewrite a method
git bisect skip ce02fc74b2db35e45906865c8a3ce2e98871eeb8
# skip: [d75bdb5793e5ebf9e480f5a0012d141347725801] build: add test for sandboxed --check
git bisect skip d75bdb5793e5ebf9e480f5a0012d141347725801
# skip: [ff6867ab94cbe9ddcb4ba18d68a4a2dcb79b865d] build: move needsHashRewrite initialization to startBuilder
git bisect skip ff6867ab94cbe9ddcb4ba18d68a4a2dcb79b865d
# skip: [f1b8e9efe77014655f059b44afa05c38990dc4aa] runProgram: Uncomment chdir support
git bisect skip f1b8e9efe77014655f059b44afa05c38990dc4aa
# skip: [73b797c207e1c7a0fd9059d2cf1e3479502f8f1b] handleDiffHook: stop passing allowVfork
git bisect skip 73b797c207e1c7a0fd9059d2cf1e3479502f8f1b
# skip: [a5efe617862484ab7dd234a495d315e7b08aa519] Clarify where output from the diff hook goes.
git bisect skip a5efe617862484ab7dd234a495d315e7b08aa519
# skip: [b4a05edbfe49f87555fd284dfb0d6c56ed43217d] runProgram: support gid, uid, chdir
git bisect skip b4a05edbfe49f87555fd284dfb0d6c56ed43217d
# skip: [dde8eeb39ae9fb73011462c74e5fa6405e432147] chdir, setgroups
git bisect skip dde8eeb39ae9fb73011462c74e5fa6405e432147
# skip: [6df61db0600ca73ccd51e3e5bec5312a04e99da1] diff hook: execute as the build user, and pass the temp dir
git bisect skip 6df61db0600ca73ccd51e3e5bec5312a04e99da1
# skip: [c78686e411e0a14cff51836fe6c35d7584171df3] build: run diff-hook under --check and document diff-hook
git bisect skip c78686e411e0a14cff51836fe6c35d7584171df3
# skip: [82f054d7d5fc8f9de45afa7107557644d1514c98] installer: update macOS version check to 10.12.2
git bisect skip 82f054d7d5fc8f9de45afa7107557644d1514c98
# skip: [288f93cec0efeba5a767c09e4d58203c67704a6f] manual: "Nix Package collection" -> "Nixpkgs package collection".
git bisect skip 288f93cec0efeba5a767c09e4d58203c67704a6f
# skip: [fcd766097636943ff11b84747d11a4e52d3d8e38] repl: Restore CTRL-C behaviour
git bisect skip fcd766097636943ff11b84747d11a4e52d3d8e38
# skip: [cbc7d9a4124343c7ba68b695b5e9b3c8188fb267] findRootsNoTemp: fixes comment about findRuntimeRoots
git bisect skip cbc7d9a4124343c7ba68b695b5e9b3c8188fb267
# skip: [3f192ac80ca421888c668896b63485486e1397ae] Add builtins.hashFile
git bisect skip 3f192ac80ca421888c668896b63485486e1397ae
# skip: [b9567aa8b6f935522d1037377e0e5205a7529fd8] install script: don't abort when "nix-channel --update" fails
git bisect skip b9567aa8b6f935522d1037377e0e5205a7529fd8
# skip: [caa76c369a6d77eb3ccaa14bd8144581c8cd50a7] docs: Mention `--max-jobs 0` to build remotely only
git bisect skip caa76c369a6d77eb3ccaa14bd8144581c8cd50a7
# skip: [b614e0e53da1f825e198996f88935a1d1a89725f] build: only skip hash rewriting for sandboxing on linux
git bisect skip b614e0e53da1f825e198996f88935a1d1a89725f
# skip: [75ec68f93ac98956552c06a7fdf4dc8da6841a65] manual: mention the "channel:" shorthand for NIX_PATH
git bisect skip 75ec68f93ac98956552c06a7fdf4dc8da6841a65
# skip: [514b3c7f8345cfcbbe166981214497ed9d93ae18] Add isPath primop
git bisect skip 514b3c7f8345cfcbbe166981214497ed9d93ae18
# skip: [d7a7a029ff9c1fc6c5a22b91718e28c4c1062d9f] scripts: remove default 'sandbox = false' from multi-user installer
git bisect skip d7a7a029ff9c1fc6c5a22b91718e28c4c1062d9f
# skip: [2aa89daab3a8351a03f6d57d3e1f737cccd59850] eval: improve type description for primops and applied primops
git bisect skip 2aa89daab3a8351a03f6d57d3e1f737cccd59850
# skip: [6f093073b6ff38fb2c4e22331e73930baa1b30e7] manual: include builtins.* for globally available builtins
git bisect skip 6f093073b6ff38fb2c4e22331e73930baa1b30e7
# skip: [684c7fff80b575da8055015c6b87693fd1d858ca] Link to `builders-use-substitutes` in chapter on distributed builds
git bisect skip 684c7fff80b575da8055015c6b87693fd1d858ca
# skip: [81a23fa7e2eb4d58d5edf2a145af5c8bbe4c331e] Add a 5 seconds of timeouts to connect to S3.
git bisect skip 81a23fa7e2eb4d58d5edf2a145af5c8bbe4c331e
# skip: [34fade478a84d822646a8daa334014134c0ec59e] docs: fix typo in AllowDirectReads
git bisect skip 34fade478a84d822646a8daa334014134c0ec59e
# skip: [fff8db205c393c53e42326f274b322eb70259250] canBuildLocally: check for features
git bisect skip fff8db205c393c53e42326f274b322eb70259250
# skip: [8ac1130cc2f0d717fa49adf18465ecf5613d295c] nix-daemon: add variable to disable fork safety
git bisect skip 8ac1130cc2f0d717fa49adf18465ecf5613d295c
# skip: [06d633598727763c54b4b049dbc213106474d10c] fix indentation
git bisect skip 06d633598727763c54b4b049dbc213106474d10c
# skip: [ac200c3678c0f569cd962d8bbd22bb74b223d748] Apply param values on S3Helper initialization
git bisect skip ac200c3678c0f569cd962d8bbd22bb74b223d748
# skip: [56c18c67d98078dbed1d05ac68663cc52d2cb543] Extend S3 URL parsing with parameters extraction
git bisect skip 56c18c67d98078dbed1d05ac68663cc52d2cb543
# skip: [07f992a74b64f4376d5b415d0042babc924772f3] Extract and expose splitUriAndParams function
git bisect skip 07f992a74b64f4376d5b415d0042babc924772f3
# skip: [b402148d8f584144b80fe159e76e553a1792ecb7] remove noop uses of nix-store --init
git bisect skip b402148d8f584144b80fe159e76e553a1792ecb7
# skip: [5f1891b795c4c4daac5336d1ec60c94f521ede1d] nix-store: make --dump-db take a list of paths to dump
git bisect skip 5f1891b795c4c4daac5336d1ec60c94f521ede1d
# skip: [7ce1fae59f6cb4341f61a48a5216cb252e4eca54] Support --disable-shared flag.
git bisect skip 7ce1fae59f6cb4341f61a48a5216cb252e4eca54
# skip: [b30be6b450f872f8be6dc8afa28f4b030fa8d1d1] Add builtins.appendContext.
git bisect skip b30be6b450f872f8be6dc8afa28f4b030fa8d1d1
# skip: [1d757292d0cb78beec32fcdfe15c2944a4bc4a95] Add builtins.getContext.
git bisect skip 1d757292d0cb78beec32fcdfe15c2944a4bc4a95
# skip: [087be7281a02a6d0e548ae56b910771837023d21] Treat plain derivation paths in context as normal paths.
git bisect skip 087be7281a02a6d0e548ae56b910771837023d21
# skip: [15efd5437377aa7054bca753abff5451396bfdca] manual: "sandbox" option default is "true" on Linux
git bisect skip 15efd5437377aa7054bca753abff5451396bfdca
# skip: [2ae5624b2f32e3c84c57d164f2c7fef205b458cc] Update instructions for Haskell script using nix-shell interpreter
git bisect skip 2ae5624b2f32e3c84c57d164f2c7fef205b458cc
# good: [bbaf865655cff51f30be254f94c22407e368cb96] Add 2.2 release notes
git bisect good bbaf865655cff51f30be254f94c22407e368cb96
# bad: [f27e53f77e39d1247c47b48acb63af3d24ca74bf] Cleanup
git bisect bad f27e53f77e39d1247c47b48acb63af3d24ca74bf
# good: [abdedcdb387628aaa4d9dc362b7e588bbc62baae] bump cpptoml to v0.1.1
git bisect good abdedcdb387628aaa4d9dc362b7e588bbc62baae
# bad: [b5ae85f0882f4857f550ed68b23af260af5000a0] Merge pull request #2882 from grahamc/docs/1115-tarball-ttl
git bisect bad b5ae85f0882f4857f550ed68b23af260af5000a0
# bad: [5600b070a766b83200a68d3632793917cf19a550] Add "warning" verbosity level
git bisect bad 5600b070a766b83200a68d3632793917cf19a550
# bad: [7b9c68766d513260d5262d5782b46384834cdb33] Add '--no-net' convenience flag
git bisect bad 7b9c68766d513260d5262d5782b46384834cdb33
# good: [b693029ca076abf8a4084a053a122ffcdf4d61b2] Style fix
git bisect good b693029ca076abf8a4084a053a122ffcdf4d61b2
# bad: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
git bisect bad 4b214e6e4566a52fa073166d8164b79627eb5601
# good: [38a4d38bc33d9943bdee4de8c77f5161bcc3fbb0] Merge pull request #2746 from bjornfor/install-multi-user-defaults
git bisect good 38a4d38bc33d9943bdee4de8c77f5161bcc3fbb0
# only skipped commits left to test
# possible first bad commit: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
# possible first bad commit: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
# possible first bad commit: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
# possible first bad commit: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
# possible first bad commit: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal

Run builds in a pseudo-terminal sounds possibly relevant.

[nh2]

I have checked manually and found that this nix commit NixOS/nix@e84c265 is the one that makes the Python test failure go away:

commit e84c26564507388f2e8b7c7f2b00eb2b57856da1
Author: Eelco Dolstra <edolstra@gmail.com>
Date:   Fri May 17 22:29:15 2019 +0200

    Run builds in a pseudo-terminal
    
    This allows many programs (e.g. gcc, clang, cmake) to print colorized
    log output (assuming $TERM is set to a value like "xterm").
    
    There are other ways to get colors, in particular setting
    CLICOLOR_FORCE, but they're less widely supported and can break
    programs that parse tool output.

@grahamc confirmed on IRC that ofborg runs nix-2.3pre6631_e58a7144 which is slightly too old to have the above commit, which is why ofborg fails here.

---

What I'm not sure on is: Is this now a nix bug, or a Python / Babel bug? Should the test work even when not in a pseudo-terminal? Should I set that locale?

[grahamc]

TBH I think the bug is squarely in Babel's side: builds shouldn't depend upon a build-time PTY and it is useful for builds to continue working in older versions of Nix. That said, ofborg needs a system upgrade anyway, which may happen today (or next month...).

[nh2]

builds shouldn't depend upon a build-time PTY

I buy this argument. What should we do to ensure that though, given that Nix 2.3, and soon ofborg, may hide these problems?

[nh2]

looks like export LC_ALL="en_US.UTF-8" is still needed for python27:

@jonringer I've confirmed now that adding any LC_ALL does not help if an older version of Nix is used:

Even before this PR, on current master (commit 5aae6ee), does the build fail with NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel --check if e.g. Nix 2.0.4 is used.

-    export LC_ALL="C.UTF-8"
+    export LC_ALL=C.UTF-8

The former fails (with the usual error), the latter succeeds.

Why is that? I thought these should be equivalent in bash.

I have most likely cleared this up as well now: I used remote --builders with 2 machines, one of them having Nix 2.3 and the other one Nix 2.0.4 on them. The time the removal of quotes made a difference, it probably happened to pick the 2.0.4 machine

[jonringer]

this is now passing on master

[jonringer]

going to close if no one has any reservations

[FRidh]

I pushed 296b2ab which (partially) fixed this.

[jonringer]

also, conflicts

[nh2]

TBH I think the bug is squarely in Babel's side: builds shouldn't depend upon a build-time PTY and it is useful for builds to continue working in older versions of Nix.

I have figured out what the precise difference is.

From https://stackoverflow.com/questions/17430168/python-encoding-issue-when-using-linux/17430280#17430280:

When redirecting the output, sys.stdout is not connected to a terminal and Python cannot determine the output encoding. When not directing the output, Python can detect that sys.stdout is a TTY and will use the codec configured for that TTY when printing unicode.

To verify it, I printed In the Babel test suite (here):

        import sys, locale, os
        print(('sys.stdout.encoding', sys.stdout.encoding))
        print(('sys.stdout.isatty()', sys.stdout.isatty()))
        print(('locale.getpreferredencoding()', locale.getpreferredencoding()))
        print(('sys.getfilesystemencoding()', sys.getfilesystemencoding()))
        print(('os.environ["PYTHONIOENCODING"]', os.environ.get("PYTHONIOENCODING")))

and got:

# nix 2.2.2 with Python 2.7

('sys.stdout.encoding', None)
('sys.stdout.isatty()', False)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'UTF-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.3.2 with Python 2.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', True)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'UTF-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.2.2 with Python 3.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', False)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'utf-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.3.2 with Python 3.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', True)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'utf-8')
('os.environ["PYTHONIOENCODING"]', None)

As you can see:

• The nix 2.2 -> 2.3 output, as expected, switches isatty() to True.
• Python 3 defaults sys.stdout.encoding to 'UTF-8' if it's not a TTY, and Python 2 defaults it to None.

When on Python 2.7 with Nix 2.2 this line from the tests containing the str type contents

            Pour toute question, veuillez communiquer avec Fulano  à nadie@blah.com

is run through pofile.read_po(), the à in the line becomes (repr(invalid_po) reveals it`):

Pour toute question, veuillez communiquer avec Fulano  \\xc3\\xa0 nadie@blah.com\\n

and that cannot be print()ed when sys.stdout.encoding is None.

---

The issue can be reproduced on Ubuntu 18.04 by running python setup.py test | cat, where the cat makes sure that there is no TTY.

(I used Babel commit 4fa749b91.)

So it's a Babel test runner pofile.py bug.

[nh2]

So it's a Babel pofile.py bug.

@grahamc I've PR'd a fix to Babel: python-babel/babel#691

[nh2]

I've also added a comment explaining the above in #78644.

I have confirmed that musl makes no difference in this, only the nix version is relevant.

I may still want to make a PR to apply the upstream patch I submitted to fix the test, because I use some remote builders that use Nix < 2.2.

[dtzWill]

❤️ 🚀 Thanks for the great work chasing this down!!