glibc 2.27 breaks locale support

[peti]

I recently updated to the latest version of master @ 546696e, and now my perl binary emits the following errors when I run non-trivial scripts (like mosh or mr):

$ mosh example.org
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_CTYPE = "en_US.UTF-8",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_CTYPE = "en_US.UTF-8",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

The locale requested by LC_CTYPE=en_US.UTF-8 isn't available here.
Running `locale-gen en_US.UTF-8' may be necessary.

mosh-client needs a UTF-8 native locale to run.

Unfortunately, the client's environment (LC_CTYPE=en_US.UTF-8) specifies
the character set "US-ASCII".

locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I have

$ ls -Ll $LOCALE_ARCHIVE
-r--r--r-- 4 root root 114228912 Jan  1  1970 /run/current-system/sw/lib/locale/locale-archive

properly configured (as per nixos-unstable @ ea145b6).

[jtojnar]

I am getting the following warning when I run any GTK application from master.

(process:9154): Gtk-WARNING **: 12:00:47.472: Locale not supported by C library.
	Using the fallback 'C' locale.

Possibly something to do with recent glibc bump?

[dezgeg]

Are the failing programs also compiled against glibc 2.27 or are they using the old glibc? I wonder if glibc made an incompatible change to the locale file format again?

[peti]

According to git bisect, the commit that causes these issues is 273e58e:

273e58ebd9f8ef04948a89d496c7cb23dab8cbe8 is the first bad commit
commit 273e58ebd9f8ef04948a89d496c7cb23dab8cbe8
Author: Shea Levy <shea@shealevy.com>
Date:   Sun Feb 18 16:52:44 2018 -0500

    glibc: Make 2.27 the default.

:040000 040000 3eedda0d7ed484b1be7d3336999f28dba8c3c89f c3824531b2620a536e18f8749f3dd76fa0b7a686 M      pkgs
bisect run success

Ping @shlevy.

[shlevy]

https://sourceware.org/glibc/wiki/Release/2.27#Statically_compiled_applications_using_locales seems relevant...

[shlevy]

5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

[shlevy]

Bah... Looks like localedef just creates broken archives.

[shlevy]

Ah! Figured it out. If you have LOCALE_ARCHIVE pointing to a pre-2.27 archive, you get this issue. We do have version-specific variants of LOCALE_ARCHIVE, so there are a few options I can see

• Do nothing. glibc 2.27 broke locale backwards compat and we just need users to run in a consistent environment
• Switch NixOS on release-18.03 to set LOCALE_ARCHIVE_2_11 (which is the version-specific env var from 2.11 through 2.26)
• Have NixOS on release-18.03 set LOCALE_ARCHIVE_2_27, which will require building glibc 2.27 locales there.

[shlevy]

@peti thoughts?

[coreyoconnor]

Trying to work this through. Adding my observations.

I have a machine using 2.27. When connecting via mosh the errors in the bug report are printed. I can reproduce that part.

If you have LOCALE_ARCHIVE pointing to a pre-2.27 archive, you get this issue

When ssh'ing to a machine using 2.27 I see:

[coconnor@grr:~]$ echo $LOCALE_ARCHIVE
/run/current-system/sw/lib/locale/locale-archive

[coconnor@grr:~]$ file -d /run/current-system/sw/lib/locale
/run/current-system/sw/lib/locale: symbolic link to /nix/store/8kw71xqs5bz6s68dylv6y13082zn0023-glibc-locales-2.27/lib/locale

Which, in that case, has a LOCALE_ARCHIVE not pointing to pre-2.27.

Does mosh reference a different LOCALE_ARCHIVE than the ssh session? Why is there different behavior here?

Figuring it might be perl specific I tried

[coconnor@grr:~]$ perl
print "hmm..."
hmm...

I also recall seeing messages during a nixos-rebuild of a system related to locale but don't have them on hand.

[shlevy]

@coreyoconnor Hmm, do you get the bug with ssh or just mosh?

[shlevy]

(e.g. if you run perl after connecting over ssh)

[coreyoconnor]

@shlevy

I see the error messages using just mosh. I do not see the error messages running perl after connecting over ssh. (The above session was all over ssh)

[shlevy]

Strange... Are you connecting to an old screen session or something? Or maybe mosh in your PATH is linked against glibc 2.27 or something?

[coreyoconnor]

Indeed! mosh is in .nix-profile and references pre-2.27:

uninstalling and including in system environment worked as expected.

[mguentner]

Ref: #38738

[shlevy]

OK, my current thoughts are:

1. NixOS should use the version specific env vars always
2. We should only bump the version specific env vars when there is a locale archive format change (which seems to be pretty rare...)
3. (the controversial piece) In the release before we bump to an incompatible glibc, we should also set the next version-specific env var, requiring us to build the next version's glibcLocales.

So, concretely this would mean:

1. Change master to set only LOCALE_ARCHIVE_2_27
2. Change 18.03 to set LOCALE_ARCHIVE_2_11
3. Change 18.03 to set LOCALE_ARCHIVE_2_27 to point to a glibc 2.27 locale archive

Since this is a somewhat sweeping change, I'd appreciate some additional thoughts on this. @peti @edolstra thoughts?

[peti]

My understanding is that this issue will disappear when nixos-unstable catches up with the master branch to contain glibc-2.27. For me personally, that is good enough, i.e. I'd be happy to "just wait".

As far as the versioned variable suggestion goes, I think it sounds good, but I haven't really thought about it and therefore don't feel like I have a qualified opinion on the matter.

[shlevy]

@peti It will still exist for anyone running nixos-18.03 and developing on master, but yes, you're generally correct.

[coreyoconnor]

One aspect of this will resolve once unstable catches up with the master branch.

The other aspect of this is similar to the "qt version mismatch" issue: A users profile may contain software incompatible with the services provided by the OS.

[dezgeg]

Changing the environment variable name sounds a bit annoying. What about changing the locale archive patch to instead try loading from "$LOCALE_ARCHIVE.$ABIVER" (where ABIVER is some compile-time constant we pick and change each time the incompatible change happens) first if such file exists. Then there is no need to do any preparation like "(the controversial piece) In the release before we bump to an incompatible glibc, we should also set the next version-specific env var, requiring us to build the next version's glibcLocales."

[shlevy]

@dezgeg Why wouldn't the controversial piece be needed on that model? We still wouldn't have a new locale archive version available to programs, so someone running current stable NixOS but testing master after a bump will see locale issues.

[shlevy]

(it's also not clear to me why a separate extension is better or worse than a separtae env var)

[dezgeg]

We still wouldn't have a new locale archive version available to programs, so someone running current stable NixOS but testing master after a bump will see locale issues.

Right, I didn't think it through completely. Yes that needs to be done.

(it's also not clear to me why a separate extension is better or worse than a separtae env var)

Because there are several places in Nixpkgs that for some reason or another do stuff like:

nixos/modules/installer/tools/nixos-enter.sh:LOCALE_ARCHIVE=$system/sw/lib/locale/locale-archive chroot "$mountPoint" "$system/activate" >&2 || true
nixos/modules/services/misc/dictd.nix:      environment = { LOCALE_ARCHIVE = "/run/current-system/sw/lib/locale/locale-archive"; };
n
pkgs/development/haskell-modules/generic-builder.nix:      LOCALE_ARCHIVE = optionalString stdenv.isLinux "${glibcLocales}/lib/locale/locale-archive";

If you got rid of the plain LOCALE_ARCHIVE variable and kept only the version-specific ones you'd be constantly updating all such usages.

[shlevy]

There'd be no need to get rid of the old one, and we already have the version specific one.

[vcunat]

Just en_US.UTF-8 is only 1.7 MiB, so that approach really seems appealing to me over long term (glibc has 26 MiB), but:

• combining with env-var locale archives might be tricky, and
• this in principle can't help all those binaries that people already have on their systems.

[vandenoever]

I'm on a fresh 18.09 NixOS system with a 122M locale-archive for glibc-locales-2.27 and yet applications like Firefox and gkrellm claim that 'locale not supported by C library' even after setting LOCALE_ARCHIVE_2_27 in a bash session.

[vcunat]

@vandenoever: on 18.09 you have 2.27 locale. If you want to support older ones on there, you need _2_11 with glibc version e.g. from 18.03 (2.26).

[bfortz]

Is it just about the closure size? Or are there other reasons to prefer system locales?
Currently on non-nixos systems, there's no good workaround. The options are:
(a) overriding individual packages which are particularly broken, or
(b) manually setting LOCALE_ARCHIVE_2_27 somewhere in your system configuration

I have been scratching my head on this all day. I have nothing against setting an env variable (but it is not clear where to do it properly, and it makes things ugly for non-specialist user), but this is not the only problem: you are not even sure the right version of glibc-locale is even installed in nix (if you are on a non-nixos box). IMO, this goes against the philosophy of nix where dependencies should be automatically fetched. I know the wrapper solution is not ideal because it increases the closure size, but at least it is safe and works out-of-the-box without users having to manually tweak their environment. Otherwise, if e.g. I want to use perl, I must manually install glibc-locale and set the env var. Too much trouble. I agree with @timbertson that I would prefer a larger closure size to all this mess.

[bfortz]

And as a side note, while upgrading mixxx (#a847dcc), I also messed up my database (happy I had a backup...) because of the mismatch between nixos-18.03 and master. Again, the wrapper was the cleanest way to solve the problem.

[coretemp]

Shouldn't this have been a release critical error?

Module `/etc/nixos/configuration.nix' has an unsupported attribute `boot'. This is caused by assignments to the top-level attributes `config' or `options'.

when I set config.glibc.installLocales = true;

When you suggest a work around can you please be specific as to where you apply the workaround?

[coretemp]

Why are two different versions of glibc in the output below on NixOS-18.09?

root # ldd `which htop`
        linux-vdso.so.1 (0x00007ffde8ff5000)
        libncursesw.so.5 => /nix/store/pk8fj5jcwvl6mb7rm34yhhga1vagh15v-ncurses-5.9/lib/libncursesw.so.5 (0x00007f96923cd000)
        libm.so.6 => /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/libm.so.6 (0x00007f96920cc000)
        libc.so.6 => /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/libc.so.6 (0x00007f9691d1f000)
        /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/ld-linux-x86-64.so.2 => /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f969262f000)

[coretemp]

I figured it out myself:

The solution is to remove old programs that use older glibc versions that have been installed via nix-env from your profile (via nix-env -e).

I do think it's a bug that previously installed programs break regardless of what the glibc maintainers did or did not do.

[peti]

I would like to close this chapter, er, bug. I realize some people still have open questions about the stae of affairs, but from a Nixpkgs development POV there's nothing left to do. A grand sweeping solution that solves all problems does clearly not exist and even if it did then I am unaware of anyone actively pursuing it. Therefore, I don't consider this problem an "open issue" that's actively being worked on and am happy to lay it at rest. Obviously, this ticket remains open for people to comment and/or ask questions.

[justinwoo]

For those coming to this thread later:

One workaround for non-NixOS is described here: #38991 (comment) (and a gist here: https://gist.github.com/peti/2c818d6cb49b0b0f2fd7c300f8386bc3)

So if you want to fix your Travis CI setup:

sudo: required
dist: trusty
language: nix
script:
  - glibcLocales=$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)
  - echo $glibcLocales
  - export LOCALE_ARCHIVE_2_27="${glibcLocales}/lib/locale/locale-archive"
  - # other things you do in your script

[tbenst]

For ubuntu, I added the following to my .bashrc:

export LOCALE_ARCHIVE_2_11="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"
export LOCALE_ARCHIVE_2_27="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"
export LOCALE_ARCHIVE="/usr/bin/locale"

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/conflict-between-glibc-and-glibclocales-workaround-inside/7608/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/bad-locale-support-perl-warning-setting-locale-failed/8261/1