Qemu headless curses: charset issues

[easyaspi314]

Problem description

I honestly don't know if this is a bug with Termux's terminal, iconv, QEMU, or ncurses.

In qemu-system's curses backend, most low ASCII characters (0x00-0x1f) show up as random CJK characters:

Steps to reproduce

The thing this most clearly affects is FreeDOS's edit program in color mode. It is 100% unusable.

However, for a standalone BIOS program, unzip the attached file (which has the source code as well for nasm and definitely not a modified Hello World example I found online), and run it like so:

qemu-system-{x86_64/i386} -display curses boot.bin

Either qemu-system-x86_64 or qemu-system-i386 will work, and headless is fine.

This will print bytes 0-255 to the console and hang. Hit Alt+2 and type quit to exit as it tells you to do.
boot.zip

Expected behavior

Something like what the -nographic displays:

Additional information

Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://termux.org/packages/ stable main
# science-repo (sources.list.d/science.list)
deb https://dl.bintray.com/grimler/science-packages-24 science stable
# game-repo (sources.list.d/game.list)
deb https://dl.bintray.com/grimler/game-packages-24 games stable
# x11-repo (sources.list.d/x11.list)
deb https://dl.bintray.com/xeffyr/x11-packages x11 main
# unstable-repo (sources.list.d/unstable.list)
deb https://dl.bintray.com/xeffyr/unstable-packages unstable main
# sources.list.d/pointless.list
deb https://its-pointless.github.io/files/24 termux extras
Updatable packages:
libcairo/stable 1.16.0-7 aarch64 [upgradable from: 1.16.0-2]
pulseaudio/stable 14.2-2 aarch64 [upgradable from: 12.2-21]
Android version:
11
Kernel build information:
Linux localhost 4.4.223-g52750b8f2138-ab6846512 #1 SMP PREEMPT Fri Sep 18 09:49:34 UTC 2020 aarch64 Android
Device manufacturer:
Google
Device model:
Pixel 2 XL

[easyaspi314]

I assume it might have something to do with the code here but I haven't tested it.

[ghost]

That's libiconv I guess. QEMU previously had error "Could not convert font glyphs from UCS-2" but it gone at some version update.

Ctrl-A X doesn't work

Works for me...

[easyaspi314]

Ctrl-A X doesn't work

Works for me...

Yeah I take it back. It is working now.

It might have been a bad ncurses shutdown blocking keystrokes.

[easyaspi314]

Oh ho ho I found a clue...

It seems that iconv is trying to convert from UCS-2 big endian.

Why? IDK.

I am going to go boot up WSL and see if this is actually a termux bug, because this is a pretty important discovery.

[easyaspi314]

Nah, it appears to be a Termux issue.

QEMU 4.2.1, glibc 2.31, Ubuntu Focal on WSL

However, one thing to note is that it appears that iconv's default for UTF-16 is little endian.

[easyaspi314]

If I manually replace UCS-2 with UCS-2LE in ui/curses.c, it works!

[easyaspi314]

So, AFAIK, the issue is that glibc's iconv will treat "UCS-2" as native endianness, while libiconv will treat it as big endian. What a blatant inconsistency.

[ghost]

I guess this issue can be closed now. Package is now patched and seems working correctly.