Looping History with Unicode Items

[mitnk]

When adding history and fetch them back with Ctrl+P, Unicode item would result wrong buffer rendering.

demo $ echo hi
hi
demo $ ls 世界
ls: 世界: No such file or directory
demo $ echo yoo
yoo

Then, let's fetch items back with Ctrl+P:

demo $ echo yoo       # <-- first `Ctrl+P` got "echo yoo", good!
demo $ ls 世界      # <-- second, we got "ls 世界", good!
demo $ lsecho hi    # <-- now a third `Ctrl+P` fetched something weird

Though, when you hit ENTER now, you will get output "hi", which means this is only a rendering issue. It is actually "echo hi" in buffer, not "lsecho hi".

[murarth]

Yes, there appears to be a mistake in how multibyte characters are counted. I'm looking into it now.

[murarth]

Nope. I was mistaken. Multibyte characters are handled properly as far as moving the internal cursor around.

The issue is actually caused by fullwidth characters and how they are treated by terminals. Solving this problem will be a bit more interesting. Fortunately, there already exists a crate for computing the width of a string, so the most confusing part is already done.

[mitnk]

Linefeed seems know the total chars (i.e. 5 for "ls 世界") in buffer. But when moving cursor, it seems cursor moves over bytes (two moves on surface of char "界") instead of chars (or boundary), compare to bash.

[murarth]

That was my first thought, as well, but it turned out to be wrong. Other multibyte characters such as "ü" and "•" were being treated correctly.

What's actually happening is that the terminal application considers characters such as "世" (called fullwidth characters) to be occupying two columns. linefeed was erroneously assuming that any character could occupy only one column and so, when it would signal to the terminal to move from the end of the line to the beginning of the line "ls 世界", it would signal to the terminal to move five columns left. Instead, linefeed must signal to the terminal to move seven columns left.

I've corrected by this checking for fullwidth characters (using the simple and wonderful unicode-width crate) and giving such characters two columns, which implicitly corrects terminal cursor movement operations.

I've just committed this change. Please let me know if it solves the problem or if you encounter any other issues.

[mitnk]

Just checked with master, it solved the issue! Awesome! Thanks a lot!

[murarth]

You're welcome and thanks for filing the issue. Also, I've just published the fix on crates.io as version 0.2.3.