[wincon] What is the status of Unicode support in this project?

[chawyehsu]

I would like to get a general idea of ​​the current Unicode support in this project, particularly for the wincon backend. To provide some context: my nano-for-windows project relies on PDCursesMod. This project evolved from an earlier version, okibcn/nano-for-windows, which also used PDCursesMod. IIUC, the original author (okibcn) had previously discussed Unicode support (specifically regarding emojis) and submitted several patches¹. And according to the PDCursesMod CHANGELOG, Unicode support for both input and output, including the Supplementary Multilingual Plane (SMP), was introduced in version 4.3.7.

However, in my own testing with nano-for-windows, emojis still fail to display.

UTF-8.txt: GNU nano with PDCursesMod [4.5.4, wincon]

UTF-8.txt: Windows Terminal

I am using the latest version (4.5.4) of PDCursesMod without any custom patches. This differs from the okibcn/nano-for-windows project, which utilized specific patches²; while those patches appeared to target earlier versions of PDCursesMod, I have observed that it seems some of them have since been upstreamed into the main PDCursesMod repository.

I am not very familiar with this project or other curses-like projects, so please accept my apologies if I have overlooked anything obvious or made an inappropriate inquiry.

Footnotes

1. https://github.com/Bill-Gray/PDCursesMod/issues?q=author%3Aokibcn ↩

2. https://github.com/okibcn/nano-for-windows/blob/e6ff29c5e65580683ac4643e47ef68444d042e04/.github/workflows/Xbuild4Windows.yml#L387-L390 ↩

[GitMensch]

In any case I'd highly suggest to recheck with the WinGUI port (if you got PDCM from MSYS2 you have the dll ready and just need to rename it) - as that will commonly show the "upper limits" of what is supported.
If you don't use the pdcm dll(s) from MSYS2, then it is important to specify how you built PDCM / where you got it from - as unicode support heavily depends on the compile options.

[GitMensch]

I see, a WIDE=Y UTF8=Y build. Testing should be fine with just replacing "wincon" by "wingui" in some branch.

Note: I'd highly suggest to make that defines explicit in the using environment by changing your header install - untested PR chawyehsu/conda-recipes#25 - as you get strange behaviour if it doesn't match.

[chawyehsu]

Thanks, I'll test with that.

[chawyehsu]

Seems worse with wingui backend (btw the performance with wingui was terrible... guess I won't build and use nano with wingui again, resize took seconds)

And with vt backend, what's that? D:

[GitMensch]

Haven't tried vt in PowerShell, but both there and in cmd you'd have to ensure with a UTF8-force built that its encoding is UTF8 chcp 65001)

[chawyehsu]

Sure, the default encoding in PowerShell 7+ is already UTF-8 though (ASCII/UTF-16 LE in Windows PowerShell 5).

[Bill-Gray]

@chawyehsu , thanks for bringing this up. The various demonstrations of "unusual" text display (wide characters, combining characters, SMP) didn't include emojis. I've added a commit to fix this; tuidemo will now test that out.

When I try it with both WinGUI and WinCon on Wine, the emojis show up as garbage (but not the same garbage you got). I suspect it's a Wine problem, though; I've never really gotten SMP or combining characters to work on that platform. Will have to dust off my remaining "genuine Windows®" machine and see what happens there.

In the past, we've had problems because on Windows (and no other platform), wchar_t is 16 bits, and SMP is handled via Unicode surrogates. I thought those problems were all fixed. But I could be wrong about that.

An important test, though : I gather the emojis show up Just Fine if you copy/paste them on the console, or run type UTF-8.txt? I've also had characters not show up, only to (eventually) realize that the font didn't include them. (Though if I recall correctly, that tended to get blanks, not garbage.)

[chawyehsu]

Thank you for your reply and for being involved in this!

... we've had problems because on Windows (and no other platform), wchar_t is 16 bits...

I've just noticed and it seems to be related to #286 (if I didn't misunderstand).

I gather the emojis show up Just Fine if you copy/paste them on the console, or run type UTF-8.txt?

They do show up corrcetly on the terminal, I've updated the screenshots.