backspace, arrow keys, etc broken on FreeBSD with QTerminal after 7954b95

[asomers]

• Have you checked if problem occurs with fish 2.3.1?
• Tried fish without third-party customizations (check sh -c 'env HOME=$(mktemp -d) fish')?

fish version installed (fish --version): fish, version 2.3.1

OS/terminal used: FreeBSD 12.0-CURRENT, QTerminal 0.6

The backspace key has no effect. The Home, Delete, End, Page Up/Down and arrow keys all insert garbage. Curiously, fish-2.3.1 works fine in FreeBSD's virtual terminals, and Bash works fine in QTerminal. I have bisected the problem and found that it was introduced in f2246df (reduce number of Unicode private-use characters)

Reproduction steps

1. start fish
2. type backspace. Nothing will happen
3. type Right Arrow Key. "[C" will appear.

[krader1961]

We've had several people complain about weird behavior on various FreeBSD variants (e.g., DragonFly BSD). None of the core developers use BSD on a regular basis. We've made attempts to get to root cause (e.g., by installing DragonFly BSD in a VM) but threw up our hands in defeat. There is something distinctly odd about the FreeBSD handling of unicode and UTF-8 locales.

When you say "works fine in FreeBSD's virtual terminals" are you referring to a console VT? What is the output of locale in a terminal where things work correctly and in a terminal where they do not? What happens if you do env LC_ALL=C fish in both terminals? What about env LC_ALL=en_US.UTF-8 fish?

I'll try to find the time to setup FreeBSD in a VM and reproduce but we're probably going to be dependent on a knowledgeable person like yourself to help us find root cause.

[asomers]

Good guess @krader1961 . I did mean the console when I said "virtual terminal". In the console, where fish works, we have:

LANG=
LC_CTYPE=C
LC_COLLATE=C
LC_TIME=C
LC_NUMERIC=C
LC_MONETARY=C
LC_MESSAGES=C
LC_ALL=

In QTerminal, where fish-2.3.1 doesn't work (but fish-2.2.0 does) we have

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8

In QTerminal, if I do env LC_ALL=C fish fish works. And in the VT, if I do LC_ALL=en_US.UTF-8 fish then fish breaks.

[krader1961]

Okay, that's consistent with the other problem reports. Note that no one running fish on macOS (another BSD flavor) is seeing problems (and that includes me) or MS Windows (both Cygwin and the new Windows Linux Subsystem). To be determined is whether fish is doing something wrong that just happens to work on Linux, macOS, etc. or whether there is a bug in FreeBSD. Any assistance or insights you can provide is welcomed.

[krader1961]

I'm pretty confident this has the same root cause as issue #3302.

There's also issue #3050 which also probably has the same root cause. Since that issue is still open and is older I'm closing this as a duplicate.

[asomers]

I agree that this looks like a duplicate of #3050. I'm not convinced about #3302 though; I can't even reproduce that one.

[krader1961]

FYI, I installed FreeBSD 12 and can reproduce this problem. The reason none of the keys works is that the default key bindings aren't setup in a UTF-8 locale:

Reverting to default bindings
The function call stack limit has been exceeded. Do you have an accidental infinite loop?
fish: __fish_reload_key_bindings VARIABLE SET fish_key_bindings
      ^
in event handler: handler for variable 'fish_key_bindings'

There are other errors as well such as alias: Name cannot be empty. I also confirmed that building from git checkout f2246df~ does not exhibit the problem. Which is very surprising to me as the author of that change.

Discussion to be continued in issue #3050 as I learn more.

[bapt]

Fixed on FreeBSD as well https://svnweb.freebsd.org/base?view=revision&revision=306782