Try to set LC_CTYPE to something UTF-8 capable

[faho]

When fish is started with LC_CTYPE=C (even just effectively, often via
LC_ALL=C!), it's basically broken. There's no way to handle non-ASCII
characters with a C locale unless we want to write our
locale-independent replacements for all of the system functions.

Since we're not going to do that, let's try to find some locale for
LC_CTYPE.

We already do that in __fish_setlocale, but that's

• a bit of a weird thing that reads unstandardized system
  configuration files
• allows setting locale to C explicitly

So it's still easily possible to end up in a broken configuration.

Now, the issue with this is that there is (AFAICT) no portable way to
get a list of all allowed locales and C.UTF-8 is not standardized, so
we have no one locale to fall back on and are forced to try a few. The
list we have here is quite arbitrary, but it's a start.

Python does something similar and only tries C.UTF-8, C.utf8 and
"UTF-8".

Once C.UTF-8 is (hopefully) standardized, that will just start
working (tm).

Note that we do not export the fixed LC_CTYPE variable, so external
programs still have to deal with the C locale, but we have no real
business messing with the user's environment.

TODOs:

• Changes to fish usage are reflected in user documentation/manpages.
• Tests have been added for regressions fixed
• User-visible changes noted in CHANGELOG.rst

[faho]

Oh, also importantly, there is no opt-out of this, because I don't think there is anything that is fixed by having fish use a singlebyte locale. It's possible I'm wrong there.

[ridiculousfish]

Please relnote this of course.

init_locale runs at startup, and also when any of the locale variables change. Do we want to do this even if the user modifies LC_ALL via a set command? If not we could add a param to init_locale indicating if it's being run at startup or not.

[faho]

init_locale runs at startup, and also when any of the locale variables change. Do we want to do this even if the user modifies LC_ALL via a set command?

The idea here is that an LC_CTYPE of C is broken.

LC_MESSAGES is a matter of which language you speak - do you want your messages in slovenian or chinese? LC_COLLATE is also a matter of regional preference - do you sort "Ä" after "Z" or equivalent to "A"?

By contrast, LC_CTYPE isn't a preference - if you set it to C, we can strictly work with less, and then providing non-ASCII input doesn't work. This is an issue because people tend to default to "LC_ALL=C" as a simple "reset my locale", and then continue providing emoji and such, even in filenames, and we can't work with them because functions like iswalnum just break. E.g. in UTF-8 mode you can set a variable called ö, and in ASCII mode you can't because it doesn't detect "ö" as a valid variable name character!

(this is of course assuming that there's two encodings worth caring about - 8-bit ASCII and UTF-8, and one is a strict subset of the other. On unix that's broadly true and there isn't really a good argument for anything else. I've also not seen anyone work with anything else unless it was a misconfiguration)

So, if a C locale is broken, the logical thing to do is to try to avoid it, even if the user indicated otherwise - and frequently they didn't, actually! They just set LC_ALL=C because that's the reset button. Some bug trackers even suggest reproducing with that to get messages in english, because they can't rely on en_US or en_GB or other variants being available.

[faho]

Also compare python's PEP 538, which does something quite similar to this.

[krobelus]

Great idea! This means that fish should work in a container even without setting a multibyte locale.
Python only does this when the locale is the default C, but this seems fine as well.

[faho]

I'm not entirely sure what to do about __fish_set_locale. It's slow and a bit of a hack (we read a bunch of config files we have no business reading), but it does a few things this doesn't, ATM:

1. It gives a full locale, with language and everything, in case that's passed wrong (or removed like that awful getty hack)
2. It gives a proper LC_CTYPE in case none from the list exist, e.g. if you only have fr_FR.UTF-8 built (and no C.UTF-8)
3. It actually sets the variables instead of just fixing fish

So I'm not sure if we should remove it or keep it. We could extend the list of locales this PR uses to everything glibc provides, which should fix it, to a first approximation, everywhere (but just for fish). Removing __fish_set_locale would be faster and allow us to add a warning in case of a misconfigured and unfixable locale (e.g. LC_ALL set to C, but no C.UTF-8 or anything available).

[faho]

Alright, I have now added a way to turn it off - setting $fish_allow_singlebyte_locale to 1 (strictly speaking something "bool_from_string" interprets as true).

This is mostly so we can actually test a C locale.