Warnings and errors when not run from a UTF-8 locale

[jbicha]

meson 0.42.0-2
Debian unstable

meson is emitting warnings when building projects on Debian's infrastructure.

Excerpt from nautilus 3.25.91-1

Warning: You are using 'ANSI_X3.4-1968' which is not a a
Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as
strings, or as file contents.
Please switch to a UTF-8 locale for your platform.

Fatal error when building geocode-glib 3.25.4.1-1

Running custom install script '/usr/bin/python3 /usr/share/meson/meson
  --internal gettext install --subdir=po --localedir=share/locale
  --pkgname=geocode-glib-1.0'
Running custom install script '/usr/bin/python3 /usr/share/meson/meson
  --internal gtkdoc --sourcedir=/<<PKGBUILDDIR>>
 --builddir=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu --subdir=docs
 --headerdirs=/<<PKGBUILDDIR>>/geocode-glib@@/
 <<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib
 --mainfile=geocode-glib-docs.xml --modulename=geocode-glib --mode=auto
 --scanargs=--rebuild-types@@--ignore-decorators=GEOCODE_EXTERN@@--ignore-headers=config.h
 geocode-glib-private.h geocode-enum-types.h geocode-nominatim-test.h
 --gobjects-types-file=geocode-glib.types
 --fixxrefargs=--html-dir=/usr/share/gtk-doc/html@@--extra-dir=/usr/share/
gtk-doc/html/glib@@--extra-dir=/usr/share/gtk-doc/html/gobject@@--extra-dir=/usr/
share/gtk-doc/html/gio
 --cflags=-I/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib/..
 -I/<<PKGBUILDDIR>>/geocode-glib/.. -pthread -I/usr/include/glib-2.0
 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/json-glib-1.0
 -I/usr/include/libsoup-2.4 -I/usr/include/libxml2
 --ldflags=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib
 -Wl,-rpath,/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib -lgeocode-glib
 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -ljson-glib-1.0 -lsoup-2.4 -lm --cc=cc
 --ld=cc'Warning: You are using 'ANSI_X3.4-1968' which is not a a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
Building documentation for geocode-glib
Traceback (most recent call last):
  File "/usr/share/meson/meson", line 37, in <module>
    sys.exit(main())
  File "/usr/share/meson/meson", line 34, in main
    return mesonmain.run(sys.argv[1:], launcher)
  File "/usr/share/meson/mesonbuild/mesonmain.py", line 297, in run
    sys.exit(run_script_command(args[1:]))
  File "/usr/share/meson/mesonbuild/mesonmain.py", line 264, in run_script_command
    return cmdfunc(cmdargs)
  File "/usr/share/meson/mesonbuild/scripts/gtkdochelper.py", line 218, in run
    options.mode)
  File "/usr/share/meson/mesonbuild/scripts/gtkdochelper.py", line 166, in build_gtkdoc
    gtkdoc_run_check(fixref_cmd, abs_out)
  File "/usr/share/meson/mesonbuild/scripts/gtkdochelper.py", line 51, in gtkdoc_run_check
    p, out = Popen_safe(cmd, cwd=cwd, stderr=subprocess.STDOUT)[0:2]
  File "/usr/share/meson/mesonbuild/mesonlib.py", line 522, in Popen_safe
    o, e = p.communicate(write)
  File "/usr/lib/python3.5/subprocess.py", line 790, in communicate
    stdout = self.stdout.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1094: ordinal not in range(128)
Warning: You are using 'ANSI_X3.4-1968' which is not a a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.

Failed to run install script '/usr/bin/python3 /usr/share/meson/meson --internal
 gtkdoc --sourcedir=/<<PKGBUILDDIR>>
 --builddir=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu --subdir=docs
 --headerdirs=/<<PKGBUILDDIR>>/geocode-glib@@/<<PKGBUILDDIR>>/obj
-x86_64-linux-gnu/geocode-glib --mainfile=geocode-glib-docs.xml
 --modulename=geocode-glib
 --mode=auto --scanargs=--rebuild-types@@--ignore-decorators=GEOCODE
_EXTERN@@--ignore-headers=config.h geocode-glib-private.h
 geocode-enum-types.h geocode-nominatim-test.h --gobjects-types-file=geocode-glib.types 
--fixxrefargs=--html-dir=/usr/share/gtk-doc/html@@--extra-dir=/usr/share/gtk-doc/
html/glib@@--extra-dir=/usr/share/gtk-doc/html/gobject@@--extra-dir=/usr/share
/gtk-doc/html/gio --cflags=-I/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib/.. 
-I/<<PKGBUILDDIR>>/geocode-glib/.. -pthread -I/usr/include/glib-2.0 -I/usr/
lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/json-glib-1.0
 -I/usr/include/libsoup-2.4 -I/usr/include/libxml2
 --ldflags=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib
 -Wl,-rpath,/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/geocode-glib
 -lgeocode-glib -lgio-2.0 -lgobject-2.0 -lglib-2.0 -ljson-glib-1.0 -lsoup-2.4 -lm --cc=cc --ld=cc'
FAILED: meson-install 

Suggestion

An earlier issue here suggested that meson should coerce a UTF-8 locale.

[jpakkane]

Duplicating information here from the Debian bug on this: maybe Debian should change the default package build locale to UTF-8 instead. Having it set to plain ansi is a bit stupid since UTF-8 has been the default user locale for 10+ years already.

[jbicha]

But if it's possible for meson to take care of that detail itself, shouldn't it?

[jpakkane]

In the latter case it is not really possible. The system has explicitly specified that it is an ANSI environment but then Meson encounters non-ascii characters from the output of a command it executes. Our design goals have stated that in case of ambiguity fail immediately and loudly as it is better than potential silent data corruption.

Python 3.7 will eventually do the policy decision of turning ANSI into UTF-8. Once that happens Meson will inherit it automatically. We don't want to do this choice by ourselves because experience has shown that trying to be clever with locales breaks things silently and makes people sad (especially me, because in the end I'm the one who has to debug said issues).

[jbicha]

For reference, the initial Debian bug is 873831

[AdrianBunk]

We don't want to do this choice by ourselves because experience has shown that trying to be clever with locales breaks things silently and makes people sad (especially me, because in the end I'm the one who has to debug said issues).

There is no need to do anything clever with locales, Meson just shouldn't use the locale in places where the encoding does not depend on the locale.

open()/Popen() use locale.getpreferredencoding(False) as default encoding. This default is correct when the data is supposed to be in the encoding of the locale, but when the encoding is fixed to UTF-8 this should be set by using encoding="utf-8".

[eli-schwartz]

So this is fixed on python 3.7+, which most places now have, and we're not touching this otherwise, right?