[BUG] complains of no UTF-8 while locale is UTF-8 #52

joske · 2021-09-28T07:32:36Z

Describe the bug

Running on RHEL 7.7 at work, using the statically compiled binary 1.0.9, it complains of no UTF-8 locale detected. While locale is:
[jdehaes@berx771 Snape]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

When using the --utf-force flag, btop works perfectly. We have the same on other RHEL 7.x boxes.
I can not compile/debug btop on this machine (as its compilers are too old - even with gcc 8.3 -std=c++20 is not available). On a RHEL 8 system this just works. I can compile/debug there (after installing gcc 10 toolchain), but useless as the problem doesn't occur there. The locale settings are identical on that system.

To Reproduce

Get a RHEL 7.7 (probably any 7.x will do) system (via docker?) and copy the btop binary. It will complain although locale is UTF-8.

Expected behavior

btop can run with the --utf-force flag.

[jdehaes@berx771 Snape]$ btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

[If applicable, add screenshots to help explain your problem.]

Info (please complete the following information):

btop++ version: 1.0.9
Binary: static binary from release
Architecture: x86_64
Platform: Linux
(Linux) Kernel: 3.10.0-1062.4.1.el7.x86_64
Terminal used: mate-terminal
Font used: Noto Mono for Powerline Regular

Additional context

contents of ~/.config/btop/btop.log -> nothing there other than no UTF-8 locale detected.

The text was updated successfully, but these errors were encountered:

daniejstriata · 2021-09-28T08:20:14Z

I'm getting this too on Ubuntu 20.04.3 LTS and Centos 7.9.
Seems like my ubuntu has a valid issue. After updating my locale locale-gen "en_ZA.UTF-8" the issue went away.
I am not seeing this happen on my Centos 8 servers.

Centos 7.9

→ locale
LANG=en_ZA.UTF-8
LC_CTYPE="en_ZA.UTF-8"
LC_NUMERIC="en_ZA.UTF-8"
LC_TIME="en_ZA.UTF-8"
LC_COLLATE="en_ZA.UTF-8"
LC_MONETARY="en_ZA.UTF-8"
LC_MESSAGES="en_ZA.UTF-8"
LC_PAPER="en_ZA.UTF-8"
LC_NAME="en_ZA.UTF-8"
LC_ADDRESS="en_ZA.UTF-8"
LC_TELEPHONE="en_ZA.UTF-8"
LC_MEASUREMENT="en_ZA.UTF-8"
LC_IDENTIFICATION="en_ZA.UTF-8"
LC_ALL=

Ubuntu 20.04

LANG=C
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

aristocratos · 2021-09-28T10:43:11Z

@joske
Run locale -a to check if you actually have a UTF-8 locale generated.

As @daniejstriata pointed out above, you can generate one with locale-gen or something equivalent for your distro.

daniejstriata · 2021-09-28T10:44:45Z

@aristocratos Why would we get the error if the locale is UTF8?

aristocratos · 2021-09-28T10:47:19Z

@daniejstriata
You wouldn't, and as you pointed out, when you generated one it went away.

The output when running just locale is just the environment variables of the locales you want to use.
The output of locale -a is the actually available locales.

daniejstriata · 2021-09-28T11:03:35Z

On the Centos host I still get the error.

Output from locale:

LANG=en_ZA.UTF-8
LC_CTYPE="en_ZA.UTF-8"
LC_NUMERIC="en_ZA.UTF-8"
LC_TIME="en_ZA.UTF-8"
LC_COLLATE="en_ZA.UTF-8"
LC_MONETARY="en_ZA.UTF-8"
LC_MESSAGES="en_ZA.UTF-8"
LC_PAPER="en_ZA.UTF-8"
LC_NAME="en_ZA.UTF-8"
LC_ADDRESS="en_ZA.UTF-8"
LC_TELEPHONE="en_ZA.UTF-8"
LC_MEASUREMENT="en_ZA.UTF-8"
LC_IDENTIFICATION="en_ZA.UTF-8"
LC_ALL=

output from locale -a

→ locale -a | grep -i en_ZA.utf8
en_ZA.utf8

output from localectl status

aristocratos · 2021-09-28T11:10:55Z

@daniejstriata
Try with latest commit, made some changes if LANG locale fails to be set.

daniejstriata · 2021-09-28T11:46:21Z

@aristocratos I have tried to compile statically on Fedora 34 but I get a failure. Dynamic is fine:

vs

I have the development tools installed. Regrettably I'm not sure what I'm missing.

$ make STATIC=true


██████╗ ████████╗ ██████╗ ██████╗
██╔══██╗╚══██╔══╝██╔═══██╗██╔══██╗   ██╗    ██╗
██████╔╝   ██║   ██║   ██║██████╔╝ ██████╗██████╗
██╔══██╗   ██║   ██║   ██║██╔═══╝  ╚═██╔═╝╚═██╔═╝
██████╔╝   ██║   ╚██████╔╝██║        ╚═╝    ╚═╝
╚═════╝    ╚═╝    ╚═════╝ ╚═╝      Makefile v1.2
PLATFORM   ?| Linux
ARCH       ?| x86_64
CXX        ?| g++ (11.2.1)
THREADS    :| 12
REQFLAGS   !| -std=c++20
WARNFLAGS  :| -Wall -Wextra -pedantic
OPTFLAGS   :| -O2 -ftree-loop-vectorize -flto=12
LDCXXFLAGS :| -pthread -D_FORTIFY_SOURCE=2 -D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector -fstack-clash-protection -fcf-protection -static -static-libgcc -static-libstdc++
CXXFLAGS   +| $(REQFLAGS) $(LDCXXFLAGS) $(OPTFLAGS) $(WARNFLAGS)
LDFLAGS    +| $(LDCXXFLAGS) $(OPTFLAGS) $(WARNFLAGS)

Building btop++ (v1.0.9)

Linking and optimizing binary...
/usr/bin/ld: cannot find -lstdc++
/usr/bin/ld: cannot find -lm
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
make: *** [Makefile:178: btop] Error 1

aristocratos · 2021-09-28T12:02:02Z

@daniejstriata
I'm not very well versed in fedora. But you can look for libstdc++ in your package manager and see if you find any missing packages, libstc++-dev or similar.

Also not sure if you just re-ran the make command just for the ouput, but make sure to make clean when changing any compiler flags if your not already doing it.

daniejstriata · 2021-09-28T12:09:22Z

@aristocratos I'll try some more. I already had libstdc++.x86_64 and libstdc++-devel.x86_64 installed. What is suspicious is that dynamic does not fail so should that not indicate I have all the relevant packages installed?

aristocratos · 2021-09-28T12:20:28Z

@daniejstriata
Since static compilation builds the needed parts from libstdc++ and libgcc into the binary, some additional sources for the libraries might be needed.

You can try without STATIC=true and just add ADDFLAGS=-static and see if that makes any difference, but I'm not sure how well it will work on other systems without the other flags.

daniejstriata · 2021-09-28T12:31:12Z

On Fedora 34 I had to install:

libstdc++-static
glibc-static
libxcrypt-static

daniejstriata · 2021-09-28T12:51:04Z

@aristocratos When I run the statically compiled binary I created just now from the latest git repo on a Centos 7 host it quits without any output/errors but. It still works normally on Centos 8 or the Fedora host I built the binary on.

$ git describe
v1.0.9-47-g8296c1a

→ /usr/local/bin/btop --version
btop version: 1.0.9

Version I built
btop.zip

aristocratos · 2021-09-28T15:12:18Z

@daniejstriata
Not sure why that is.
But there is new static binaries in v1.0.10 release you can try.

daniejstriata · 2021-09-28T18:13:29Z

@aristocratos I installed the version you compiled and is doing the same thing as mine. The exit code is 1

 ██████╗ ████████╗ ██████╗ ██████╗
 ██╔══██╗╚══██╔══╝██╔═══██╗██╔══██╗   ██╗    ██╗
 ██████╔╝   ██║   ██║   ██║██████╔╝ ██████╗██████╗
 ██╔══██╗   ██║   ██║   ██║██╔═══╝  ╚═██╔═╝╚═██╔═╝
 ██████╔╝   ██║   ╚██████╔╝██║        ╚═╝    ╚═╝
 ╚═════╝    ╚═╝    ╚═════╝ ╚═╝      Installer

Installing binary to: /usr/local/bin/btop
Installing doc to: /usr/local/share/btop
Installing themes to: /usr/local/share/btop/themes
File: /usr/local/bin/btop
Setting owner root:root
Setting SUID bit
Installed btop Version v1.0.10.

(01:09:15) ± [root@server1] ~/ (master ✓) 
→ btop
(01:09:23) ± [root@server1] ~/ (master ✓) 
→ echo $?
1

--debug also made no difference


(01:11:53) ± [root@server1] ~/ (master ✓) 
→ btop --debug
(01:12:00) ± [root@server1] ~/ (master ✓) 
→ echo $?
1

I was able to use btop --utf-force to execute it.

with gdb

→ gdb -ex run /usr/local/bin/btop
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/btop...done.
Starting program: /usr/local/bin/btop 
[Inferior 1 (process 57902) exited with code 01]
(gdb) quit

aristocratos · 2021-09-28T20:07:20Z

@daniejstriata
If you run with gdb again and before you quit out enter bt what output do you get?

daniejstriata · 2021-09-29T05:23:57Z

@aristocratos There is no stack.

→  gdb -ex run /usr/local/bin/btop
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/btop...done.
Starting program: /usr/local/bin/btop 
[Inferior 1 (process 8447) exited with code 01]
(gdb) bt
No stack.
(gdb)

aristocratos · 2021-09-29T06:19:07Z

@daniejstriata
Forgot that I moved the signal handler init, so needed to change the function for exiting on errors.
Also made some changes to UTF8 detection in latest commit.
See if that works for you.

joske · 2021-09-29T06:55:51Z

@aristocratos Sorry, was busy day at work yesterday. Yes locales are installed:

[jdehaes@berx771 AMH-4.2]$ locale -a | grep en_US
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8

But I guess you already know by know from the others. Can not build on this machine as the most recent GCC I have is 8.3. @daniejstriata how did you get working compilers on RHEL 7.x?? They are not in EPEL or RHEL satellite.

aristocratos · 2021-09-29T07:04:58Z

@joske
Static binary you can try.
btop.zip

joske · 2021-09-29T07:09:25Z

@aristocratos Unfortunately, same:

[jdehaes@berx771 tmp]$ chmod 755 btop
2021-09-29 09:08:53
[jdehaes@berx771 tmp]$ ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

aristocratos · 2021-09-29T07:11:43Z

@joske
Any difference if you run LANG= LC_ALL= ./btop or LANG=en_US.utf8 LC_ALL=en_US.utf8 ./btop ?

joske · 2021-09-29T07:15:14Z

@aristocratos No.

[jdehaes@berx771 tmp]$ LANG= LC_ALL= ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.
2021-09-29 09:14:40
[jdehaes@berx771 tmp]$ LANG=en_US.utf8 LC_ALL=en_US.utf8 ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

aristocratos · 2021-09-29T07:22:38Z

@joske
Try also LC_ALL=en_US.UTF-8 ./btop

joske · 2021-09-29T07:23:22Z

@aristocratos Same:
[jdehaes@berx771 tmp]$ LC_ALL=en_US.UTF-8 ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

aristocratos · 2021-09-29T07:45:22Z

@joske
Ok, relaxed the constraints to no try to set the locale if it's already reporting it's valid. Test with this one
btop.zip

joske · 2021-09-29T07:47:27Z

@aristocratos Nope:
[jdehaes@berx771 tmp]$ ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

aristocratos · 2021-09-29T07:56:04Z

@joske
And if you run LC_ALL=en_US.UTF-8 ./btop and LANG=LC_ALL=en_US.UTF-8 LC_ALL=en_US.UTF-8 ./btop now?

joske · 2021-09-29T07:56:59Z

@aristocratos Both fail:
[jdehaes@berx771 tmp]$ LC_ALL=en_US.UTF-8 ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.
2021-09-29 09:56:34
[jdehaes@berx771 tmp]$ LANG=LC_ALL=en_US.UTF-8 LC_ALL=en_US.UTF-8 ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

aristocratos · 2021-09-29T08:04:26Z

@joske
Oops, should have been LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 ./btop

aristocratos · 2021-09-29T08:05:11Z

@joske
Can also try the lower case variation again LANG=en_US.utf8 LC_ALL=en_US.utf8 ./btop

joske · 2021-09-29T08:05:13Z

@aristocratos same. Edit: lower case also same.

aristocratos · 2021-09-29T08:12:29Z

@joske
Ok, try this one, will show your current locale on fail.
btop.zip

joske · 2021-09-29T08:24:44Z

@aristocratos
[jdehaes@berx771 tmp]$ ./btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.
Current locale: C

Strange as locale, when run in terminal returns something else entirely:
[jdehaes@berx771 tmp]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

joske · 2021-09-29T08:26:43Z

Environment does not have all those LC_ vars, but it does have LANG:
[jdehaes@berx771 tmp]$ env | grep LC
2021-09-29 10:26:05
[jdehaes@berx771 tmp]$ env | grep LANG
LANG=en_US.UTF-8

aristocratos · 2021-09-29T08:44:28Z

@joske
Another try, if it starts this time check your btop.log for warnings after.
btop.zip

joske · 2021-09-29T08:47:21Z

@aristocratos This is even worse ;-) :
[jdehaes@berx771 tmp]$ ./btop
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted

snip of btop.log (previous attempt and current):
2021/09/29 (10:23:40) | ===> btop++ v.1.0.10
2021/09/29 (10:23:40) | ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.
Current locale: C

2021/09/29 (10:45:38) | ===> btop++ v.1.0.10
2021/09/29 (10:45:38) | WARNING: Failed to set locale en_US.UTF-8 continuing anyway.

aristocratos · 2021-09-29T08:56:02Z

@joske
That's an easy fix, but now we know why it kept failing on your system, it finds a valid locale but aren't allowed to set it. That you're gonna have to look into yourself. Permission errors I can't do anything about from my end.

This one should have the string error fixed:
btop.zip

joske · 2021-09-29T08:58:23Z

@aristocratos This one works (although it complained my terminal was one character too narrow (79x24 by accident). When I resized to 80x24 it worked without any arguments :-).

aristocratos · 2021-09-29T09:02:03Z

@joske
You can try the running the old release binary as superuser or just sudo btop, if that works there's definitvly some weird permissions set for your user. Not sure why setting the terminal locale would be restricted otherwise.

joske · 2021-09-29T09:12:42Z

@aristocratos Same with old with sudo:
[jdehaes@berx771 AMH-4.1]$ sudo /usr/local/bin/btop
ERROR: No UTF-8 locale detected!
Use --utf-force argument to force start if you're sure your terminal can handle it.

RHEL has strange defaults, and our company messes with lots of things in the images used for our VMs (although I disable a lot of those already). But the fact that other users report the same, I'm inclined to think it's RHEL and not our company that messes this up. Running btop under strace did not yield any clues as to what is failing exactly. Any ideas on how to troubleshoot (if you still care as it's working now).

Edit: the last version you attached here does not give any warning when ran with sudo (while without it still logs the warning) :
[jdehaes@berx771 tmp]$ rm ~/.config/btop/btop.log
2021-09-29 11:15:11
[jdehaes@berx771 tmp]$ sudo ./btop
2021-09-29 11:15:36
[jdehaes@berx771 tmp]$ cat ~/.config/btop/btop.log
cat: /home/jdehaes/.config/btop/btop.log: No such file or directory
2021-09-29 11:15:37
[jdehaes@berx771 tmp]$ ./btop
2021-09-29 11:16:32
[jdehaes@berx771 tmp]$ cat ~/.config/btop/btop.log

2021/09/29 (11:16:32) | ===> btop++ v.1.0.10
2021/09/29 (11:16:32) | WARNING: Failed to set locale en_US.UTF-8 continuing anyway.

aristocratos · 2021-09-29T09:41:51Z

@joske
I'd rather not get in to the mess of how different distros handles locales. As long as it's working.
This fix does take away some safety since it ignores failures of setting locale, but at least the warnings should give some idea to whats going wrong in case of weird output.

Will include the fix in release v1.0.11 tonight along with new static binaries.

daniejstriata · 2021-09-29T10:47:59Z

@aristocratos Sorry, was busy day at work yesterday. Yes locales are installed:

[jdehaes@berx771 AMH-4.2]$ locale -a | grep en_US en_US en_US.iso88591 en_US.iso885915 en_US.utf8

But I guess you already know by know from the others. Can not build on this machine as the most recent GCC I have is 8.3. @daniejstriata how did you get working compilers on RHEL 7.x?? They are not in EPEL or RHEL satellite.

@joske I built on my Fedora install.

daniejstriata · 2021-09-29T10:53:52Z

@aristocratos The last zip uploaded worked on my Centos 7 servers.

Here is the output from running it with debug.

2021/09/29 (11:52:42) | ===> btop++ v.1.0.10
2021/09/29 (11:52:42) | DEBUG: Starting in DEBUG mode!
2021/09/29 (11:52:42) | INFO: Logger set to DEBUG
2021/09/29 (11:52:42) | WARNING: Failed to set locale en_ZA.UTF-8 continuing anyway.
2021/09/29 (11:52:42) | DEBUG: Setting LC_ALL=en_ZA.UTF-8
2021/09/29 (11:52:42) | INFO: Running on /dev/pts/4
2021/09/29 (11:52:44) | INFO: Quitting! Runtime: 00:00:02

aristocratos · 2021-09-29T19:31:53Z

@joske @daniejstriata
Fixed in release v1.0.11

joske added the bug Something isn't working label Sep 28, 2021

joske assigned aristocratos Sep 28, 2021

aristocratos closed this as completed Sep 29, 2021

[BUG] complains of no UTF-8 while locale *is* UTF-8 #52

[BUG] complains of no UTF-8 while locale *is* UTF-8 #52

Comments

joske commented Sep 28, 2021

daniejstriata commented Sep 28, 2021 • edited

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021 • edited

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021 • edited

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021

daniejstriata commented Sep 28, 2021 • edited

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 28, 2021 • edited

aristocratos commented Sep 28, 2021

daniejstriata commented Sep 29, 2021 • edited

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021 • edited

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021

aristocratos commented Sep 29, 2021

joske commented Sep 29, 2021 • edited

aristocratos commented Sep 29, 2021

daniejstriata commented Sep 29, 2021 • edited

daniejstriata commented Sep 29, 2021 • edited

aristocratos commented Sep 29, 2021

[BUG] complains of no UTF-8 while locale is UTF-8 #52

[BUG] complains of no UTF-8 while locale is UTF-8 #52

daniejstriata commented Sep 28, 2021 •

edited

daniejstriata commented Sep 28, 2021 •

edited

daniejstriata commented Sep 28, 2021 •

edited

daniejstriata commented Sep 28, 2021 •

edited

daniejstriata commented Sep 28, 2021 •

edited

daniejstriata commented Sep 29, 2021 •

edited

joske commented Sep 29, 2021 •

edited

joske commented Sep 29, 2021 •

edited

daniejstriata commented Sep 29, 2021 •

edited

daniejstriata commented Sep 29, 2021 •

edited