Unicode encoding issues on Windows

[jazzl0ver]

Windows [Version 10.0.19042.1466]

[o-sdn-o]

It looks like the code page is not set to UTF-8. Could it be?

[o-sdn-o]

This should help:

chcp 65001 && vtm

[jazzl0ver]

yeah, chcp did the job, thank you! should I create a bat file to run vtm or there's a way to set up unicode by default somewhere in the system properties (didn't find it) ? the screenshot i posted i got after running vtm from inside the archive from windows explorer

[o-sdn-o]

Update: As of release v0.9.9i this is not required.

"All Settings" -> "Time & Language" -> "Language" (side menu) -> "Administrative language settings" -> "Change system locale..." and set the checkbox "Beta: Use Unicode UTF-8 for worldwide language support". Restart your system.

[jazzl0ver]

thank you!

[cmplstofB]

Note:
Some user building vtm from source code by Visual Studio may get a warning C4189 from cl.exe and could fail to compile it in a chain reaction; even after set codepage to UTF-8.
One of the solution of this is to add /utf-8 option to cl.exe.

chcp 65001
cmake ./src -DCMAKE_BUILD_TYPE=Release "-GVisual Studio 17 2022"
cmake -E env "CXXFLAGS=/utf-8" cmake --build . --config Release

[o-sdn-o]

Thanks for the note! I added this flag to CMakeLists.txt, also added it to CMakeSettings.json just in case, and added chcp 65001 to the Building from Source section in readme.md.

CMakeLists.txt:

...
if (CMAKE_SYSTEM_NAME STREQUAL "Windows")
    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /DWIN32 /D_WINDOWS /W3 /GR /EHsc /bigobj /utf-8")
...

readme.md:
Visual Studio 2019:

git clone https://github.com/netxs-group/vtm.git
cd ./vtm
chcp 65001
cmake ./src -DCMAKE_BUILD_TYPE=Release "-GVisual Studio 16 2019"
cmake --build . --config Release
cd Release
vtm

Visual Studio 2022:

git clone https://github.com/netxs-group/vtm.git
cd ./vtm
chcp 65001
cmake ./src -DCMAKE_BUILD_TYPE=Release "-GVisual Studio 17 2022"
cmake --build . --config Release
cd Release
vtm

[o-sdn-o]

Question: Should we support for code pages other than 65001(UTF-8) on Windows?

Encodings popularity (www): https://en.wikipedia.org/wiki/Popularity_of_text_encodings

• UTF-8
• ISO-8859-1 https://ru.wikipedia.org/wiki/ISO_8859-1
• CP437
• Windows-1251
• Windows-1252
• CP850
• CP866

pros:

• Applications and libraries system-wide based on 8.3 names.
• Support for old archives with filenames packed in an environment with national encoding.
• Maintaining compatibility with older applications that have been discontinued by the developer.
• Windows uses national code pages by default.
• CP65001 in the console breaks things if enabled. E.g., .NET sees CP65001 and starts outputting the UTF-8 BOM everywhere, breaking applications that don't even care about the character encoding. Fixed in PowerShell 7.1
  • System.Console unexpectedly uses a UTF-8 encoding *with BOM* on Windows dotnet/runtime#28929
  • Make console windows fully UTF-8 by default on Windows, in line with the behavior on Unix-like platforms - character encoding, code page PowerShell/PowerShell#7233

cons:

• The presence of support for legacy text encodings slows down the transition of software to UTF-8 encoding.
• Excessive code base growth
  • See https://en.wikipedia.org/wiki/Windows_code_page
  • See https://learn.microsoft.com/en-us/windows/win32/intl/code-page-identifiers
• Waste of resources for implementation.

pros > cons. I think it needs to be implemented. At least one-byte code pages.

[o-sdn-o]

Affected win32 console api functions:

Set/GetConsoleCP - Input code page (keyboard and titles only)

• ReadConsole
• ReadConsoleInput
• PeekConsoleInput
• WriteConsoleInput
• SetConsoleTitle
• GetConsoleTitle
• SetConsoleOriginalTitle
• GetConsoleOriginalTitle

Set/GetConsoleOutputCP - Output code page

• WriteConsole
• WriteConsoleOutput
• FillConsoleOutputCharacter
• ReadConsoleOutput
• ReadConsoleOutputCharacter
• GetConsoleLangId