Linux vconsole with utf-8 character broke when autosuggesting

[pickfire]

Hi, after a few updates, it seems that fish shows something weird when autosuggesting (grey characters). Step to reproduce:

1. In a terminal, touch 'asdf — test'
2. Make sure that the next time you type touch in a terminal, 'asdf — test' shows up behind it.
3. ^D to close the terminal
4. Ctrl+Alt+F2 to open framebuffer
5. Type touch

As you type:

• prompt is backspaced by 2 characters
• 4 is showed up at the end of line

[krader1961]

I can't reproduce this problem. Albeit I didn't use Ctrl+Alt+F2 to open a "framebfuffer". I executed your step one then opened a new terminal and typed touch. I'm using a fish shell built from the most recent GitHub repository as I type this so it is possible the problem has been fixed after whatever version of fish you are running. It's also possible the problem is due to an incorrect $TERM variable or something else unique to your environment. In particular I know what a "framebuffer" is in the context of the X11 window system but have no idea what you mean by that term since a framebuffer is irrelevant to fish.

You also specify that it is necessary to close the existing fish process by typing \cD to close the terminal and then open a new terminal to the same host. What happens if you type touch in the original terminal? I would expect the same problem to occur.

[pickfire]

@krader1961 If I Am Correct, framebuffer isn't X11 and it can't display utf8 characters.

I executed your step one then opened a new terminal and typed touch.

A new terminal would just works fine, there is no reason it wouldn't work.

What happens if you type touch in the original terminal? I would expect the same problem to occur.

It wouldn't have any problems as it can display the character.

I don't think it is $TERM but you could always check if it is framebuffer by check if $TERM is 'linux'.

[faho]

Works for me. I tried both with your text here and the unicode snowman (☃). The glyph itself isn't displayed correctly (because IIRC kernel vts are limited to 256 characters), but it doesn't harm the prompt.

@krader1961: He means the linux kernel terminals (you know, getty and stuff) - though I've never found an unmistakeable term for just those.

[terlar]

I think usually they are called virtual console or vconsole for short. There is a term that supports UTF-8 and more goodies. See https://github.com/dvdhrm/kmscon. I spawn that one for my extra TTY:s.

[krader1961]

As you implied, @faho, the OP is probably talking about "linux kernel terminals"; aka virtual consoles. the linux framebuffer driver provides an abstraction for the graphics hardware and is used by the X11 subsystem. The Ctrl+Alt+F2 step mentioned in the original problem statement suggests the author is talking about switching console ttys using the virtual console driver (see http://www.tldp.org/HOWTO/Keyboard-and-Console-HOWTO-7.html). So, as is all too often the case, we don't have a sufficiently detailed and accurate description of the problem and the environment it occurs within to really do more than guess.

@pickfire We really need more information such as the Linux distro, that you are in fact running your commands on the linux console, the contents of set before and after you open the new console, why you're using the term "framebuffer", and probably other things I haven't thought of.

The linux virtual console driver sort of supports Unicode: see http://askubuntu.com/questions/23610/how-to-enable-unicode-support-in-a-tty for one article on this topic. I'm guessing that the default, non-UTF-8 font, used by the linux virtual console is the source of the problem. Which is to say it's probably garbling the UTF-8 sequences output by fish. Does running unicode_start fix the problem @pickfire?

FWIW, My previous job was as a Linux level 3 support engineer at IBM with a specialization in kernel crash dump analysis for almost a decade. And the decade before that supporting Sequent's unique flavor of UNIX known as Dynix/3 then Dynix/PTX. In those two decades the biggest challenge in solving customer problems was usually getting a clear and concise problem statement :-)

[pickfire]

@faho, that steps isn't real, I just reproduce something similar based on my situation. This is a real:

1. In a terminal, type `touch 'hello ☃ world', and exit.
2. In a framebuffer, type t o u...

[pickfire]

@krader1961 There is a lot of questions, I will brake it down into small pieces, ask me anything if I had not.

I am using getty without any add-on (eg. tmux) for framebuffer in Linux alarmpi 4.1.13-1-ARCH #1 SMP PREEMPT Fri Nov 13 20:11:44 MST 2015 armv7l GNU/Linux.

the contents of set before and after you open the new console

It should be the same, I didn't not set anything.

why you're using the term "framebuffer"

Because some called it framebuffer and some called it linux console.

I need to use framebuffer at that moment because I want to watch video without triggering the screen locker (slock) as I am using omxplayer to watch video in rpi2.

Does running unicode_start fix the problem @pickfire?

No, it doesn't.

The linux virtual console driver sort of supports Unicode: see http://askubuntu.com/questions/23610/how-to-enable-unicode-support-in-a-tty for one article on this topic.

I know that by somehow ugly hack can allow framebuffer to display unicode support but as I am not using it very frequently, I don't think that is required.

My fish_version is fish, version 0918ad6.

[krader1961]

It should be the same, I didn't not set anything.

I agree it "should be the same". However in my 20+ years of experience working problems like this assumptions are frequently wrong. So it is important to actually collect data and confirm whether or not the assumption is correct.

Based on your other answers I conclude you're using a Raspberry Pi, https://www.raspberrypi.org/. You're probably therefore running X11 on top of the framebuffer driver on top of the linux virtual console driver. Presumably fish is being launched inside a terminal emulator of some sort. But again, I'm just making a SWAG, (silly wild assed guess) because I can't really tell from your description what the environment is that fish is running within. Perhaps a screen-shot would help.

[pickfire]

[zanchey]

I can't reproduce this on 4.5.0 using the text mode console (Console: colour VGA+ 80x25 in dmesg) or 3.16.7 using the framebuffer console (Console: switching to colour frame buffer device 128x48`).

https://en.wikipedia.org/wiki/Linux_console has a good disambiguation of the difference between the virtual terminal and the underlying device (text-mode or framebuffer).

[faho]

@pickfire: What's your locale and $LANGUAGE? You're using Arch Linux ARM, right? That would mean you would also likely hit #3092 - systemd unsets the locale variables for the getty service, so fish inherits no locale, so it defaults to "POSIX" - which uses ASCII, not UTF-8, as encoding.

If you set $LANG or LC_CTYPE to something with UTF-8 in your config.fish (e.g. "en_US.UTF-8") does that help?

[pickfire]

@faho, @zanchey I need to recheck this, I will see if I could reproduce it with fish 2.3. My locale is:

LANG=en_GB.UTF-8
LC_CTYPE=en_GB.UTF-8
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES=C
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

$LANGUAGE is unset and yeah, I am using ALARM

[pickfire]

On another system (arch with fish 2.3), it showed a block character (maybe indicating that it can't display it, but then when I deleted those character, the characters that I had deleted turned red (maybe it's expected). I think this is solved now, closing