Unreadable non-UTF-8 output on localized MSVC

[Boddlnagg]

A localized (e.g. German) MSVC can produce non-UTF-8 output, which can become almost unreadable in the way it's currently forwarded to the user.

This can be seen in an (otherwise unrelated) issue comment: rust-lang/cc-rs#87 (comment)
Note especially this line:

note: Non-UTF-8 output: LINK : fatal error LNK1104: Datei \"ucrt.lib\" kann nicht ge\xf6ffnet werden.\r\n

That output might be a lot longer for multiple LNK errors (one line per error, but the lines are not properly separated in the output, because they are converted to \r\n) and become really hard to read.
If possible, the output should be converted to Unicode in this case.

(previously reported as rust-lang/cargo#3012)

---

NOTE from @crlf0710: Please install Visual Studio English Language Pack side by side with your favorite language pack for UI

[nagisa]

I’m surprised we try to interpret any output in Windows as UTF-8 as opposed to UTF-16, like we should.

[petrochenkov]

I suspect that the MSVC compiler's raw output is encoded as Windows-1252 or Windows-1250 (for German) depending on the current console code page and not as UTF-16.

@Boddlnagg
Does cmd /c "chcp 65001" solve the encoding problem?

[Boddlnagg]

@petrochenkov

Does cmd /c "chcp 65001" solve the encoding problem?

No, neither in cmd nor in powershell (usually I'm working with the latter). It prints Aktive Codepage: 65001. but the encoding problems persist.

[retep998]

Is it just MSVC which is localized, or is your system codepage itself different? If we know that a certain program's output is the system codepage we could use MultiByteToWideChar with CP_OEMCP to convert it. Perhaps we could check whether the output is UTF-8, and attempt to do the codepage conversion if it isn't.

[Boddlnagg]

@retep998 I am using a German localized Windows. How can I find out the system codepage? The default CP for console windows is "850 (OEM - Multilingual Lateinisch I)".

[retep998]

@Boddlnagg You can call GetCPInfoEx with CP_ACP or CP_OEMCP.

[Boddlnagg]

@retep998 That returns codepage numbers 1252 for CP_ACP and 850 for CP_OEMCP.

[retep998]

@Boddlnagg In which case the output from the linker appears to be CP_OEMCP, so now someone just has to add code to rustc which detects when the linker output isn't utf-8 on windows and use MultiByteToWideChar with CP_OEMCP instead.

[danyx23]

I ran into this problem as well (also on a German Windows 10). As a workaround it helps if you go to in Win 10 to Settings -> Region and Language -> Language and add English as a language and make it default ( you may have to log out and back in again). After that, programs like link.exe should output using a locale that works with rust as it is right now.

It would still be great if this could be fixed in rust :)

[michaelfairley]

Just saw another instance of this in Rust-SDL2/rust-sdl2#783, except with GBK/CP936/GB2312 (a common encoding in China), where SDL2.lib : warning LNK4272:\xbf\xe2\xbc\xc6\xcb\xe3\xbb\xfa\xc0\xe0\xd0\xcd\xa1\xb0x86\xa1\xb1\xd3\xeb\xc4\xbf\xb1\xea\xbc\xc6\xcb\xe3\xbb\xfa\xc0\xe0\xd0\xcd\xa1\xb0x64\xa1\xb1\xb3\xe5\xcd\xbb got printed instead of the desired SDL2.lib : warning LNK4272:库计算机类型“x86”与目标计算机类型“x64”冲突.

[crlf0710]

Using nightly 2019-07-04, we can confirm that this issue is fixed. (I'm using Chinese version of Visual Studio).

[duester]

For me, it still does not work even with nightly build (1.38 x86_64-pc-windows-msvc, Russian version of VS). I try to compile a Hello world project in Intellij Idea but always get the exception:

error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "link.exe" "/NOLOGO" "/NXCOMPAT" "/LIBPATH:C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.3jz4wzfxl06xy1jf.rcgu.o" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.3km2wbb9jydj43df.rcgu.o" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.42ky0m2plnm34s09.rcgu.o" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.52jmj1740mcbmdz0.rcgu.o" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.5509wc46fhf0uffc.rcgu.o" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.9lo4gj875er9ymo.rcgu.o" "/OUT:Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.exe" "Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps\\Test-63f2382baaf4d784.1id7vce7itpr8ejr.rcgu.o" "/OPT:REF,NOICF" "/DEBUG" "/NATVIS:C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\etc\\libcore.natvis" "/LIBPATH:Q:\\Documents\\workspace\\Rust\\Test\\target\\debug\\deps" "/LIBPATH:C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-cf34d7d652c49148.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-fdc10a5672b0a944.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libbacktrace-a3d3ff900124b0c2.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-cab590b2458496ef.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-d57237973d04b75e.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-9351200acecb2a9f.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a54b9ab1de8667f4.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-917c86f963486d9f.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-9ce934c7b0e2a66b.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-cf7a654a78004adf.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-14df46fe02585934.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-2506cbc701568648.rlib" "C:\\Program Files\\Rust nightly MSVC 1.38\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-504270631fad550c.rlib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "msvcrt.lib"
  = note: Non-UTF-8 output: LINK : fatal error LNK1181: \xed\xe5 \xf3\xe4\xe0\xe5\xf2\xf1\xff \xee\xf2\xea\xf0\xfb\xf2\xfc \xe2\xf5\xee\xe4\xed\xee\xe9 \xf4\xe0\xe9\xeb \"advapi32.lib\"\r\n

error: aborting due to previous error

[crlf0710]

@duester Could you execute rustc --version and see what that outputs?

[duester]

@crlf0710

C:\Users\Maxim>rustc --version
rustc 1.38.0-nightly (dfd52ba6a 2019-07-06)

[crlf0710]

@duester
That's a little strange. Could you open "x64 Native Tools Command Prompt for Visual Studio 2019" from start menu, and execute the following command:

set VSLANG=1033
link

And see whether the output is English?

If the output's still not in English, would you mind open "Visual Studio Installer" from start menu, choose "Modify" for the corresponding Visual Studio edition, and in the Language packs tab, install the English language pack. And try the instructions above again?

[duester]

@crlf0710, thanks! The hint about installing the English language pack has made the previous error disappear. But now I have another message while running in Idea:

"C:/Program Files/Rust nightly MSVC 1.38/bin/cargo.exe" build --color=always --all --all-targets
   Compiling mytest v0.1.0 (C:\Users\m.dyuster.TCSBANK\workspace\Rust\mytest)
error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "link.exe" "/NOLOGO" "/NXCOMPAT" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.1hnlpgryprc9p9xi.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.2gxvjdcoooxv0ejr.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.38hf6hk8x3wljiz2.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.3di4574vy9u099e6.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.579lyju0fbv2vqal.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.lqc1y36ppxvcbvb.rcgu.o" "/OUT:C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.exe" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-ff31f611ae19598e.2716pn6rggtx9414.rcgu.o" "/OPT:REF,NOICF" "/DEBUG" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-c418a1c7c96bdbae.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-740e55ec30ec9b42.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-4e3d4b2c73be2130.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-e03c95365300773b.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-9b63ec27f8df7123.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-cc811f306714ba24.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-dee84a63f4b90027.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-7d070a58cfb70fe0.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-10ec18f1dfe36e70.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-e0e69d50f48d4667.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-45eb0e961cc324b2.rlib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "msvcrt.lib"
  = note: LINK : fatal error LNK1181: cannot open input file 'advapi32.lib'
          

error: aborting due to previous error

error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "link.exe" "/NOLOGO" "/NXCOMPAT" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.35mzi2ybnl9qcopy.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.4kad2mi3myvxcejq.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.4y3895lz3cmibp9r.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.acgdiy44rmzthg8.rcgu.o" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.kah10no9rc6ivrw.rcgu.o" "/OUT:C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.exe" "C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps\\mytest-3a8239d26fb8b726.3v405rv7zkctyu.rcgu.o" "/OPT:REF,NOICF" "/DEBUG" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\workspace\\Rust\\mytest\\target\\debug\\deps" "/LIBPATH:C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libtest-0a151bd0e20c2999.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libterm-f1a6f540f469eecb.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libgetopts-cc579f183f059136.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunicode_width-786eb0999ad22e42.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-c418a1c7c96bdbae.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-740e55ec30ec9b42.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-4e3d4b2c73be2130.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-e03c95365300773b.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-9b63ec27f8df7123.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-cc811f306714ba24.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-dee84a63f4b90027.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-7d070a58cfb70fe0.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-10ec18f1dfe36e70.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-e0e69d50f48d4667.rlib" "C:\\Users\\m.dyuster.TCSBANK\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-45eb0e961cc324b2.rlib" "kernel32.lib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "msvcrt.lib"
  = note: LINK : fatal error LNK1181: cannot open input file 'kernel32.lib'
          

error: aborting due to previous error

error: Could not compile `mytest`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `mytest`.

I haven't changed the path to the std library, it's looks like C:\Users\m.dyuster.TCSBANK\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src

[crlf0710]

Yes, this is the same error as before, it's because you didn't install windows sdk. Choose the newest one in the visual studio installer too.

[duester]

@crlf0710, thank you again. Now it's compiling and running. I'm happy ;)
Just out of curiosity: was I supposed to know in advance that I would need an English lang pack and a Windows SDK when installing Visual Studio?

[retep998]

Please feel free to point out places where Rust tells you to install Visual Studio but does not tell you to install the Windows SDK, so that we can fix those places.

It might still be worth implementing the text encoding version of the fix for people who don't have the English language pack, though I really wish we could just get unicode output out of VS without having to specify a language.

[duester]

@retep998, sorry, seems I was a bit inattentive. Now I double-checked: after starting rustup-init.exe the screen info says, I should assure that I have Windows SDK installed. So my fault, nothing to fix there :) Thanks.

[lygstate]

This is not a fix anymore.

[lygstate]

refer to rust-lang/cc-rs#552

[crlf0710]

@lygstate: Please make sure you have installed Visual Studio English Language Pack side by side with your favorite language pack for UI for this to work.

[ChrisDenton]

I'm not sure this problem is really fixed. The PR that closed this kinda works around the issue because English is usually ASCII which is the same in UTF-8. However, as noted above, this relies on the Visual Studio English language pack being installed which it isn't always. And that there's no non-ascii output (e.g. file paths, etc).

The suggestion to use MultiByteToWideChar with CP_OEMCP seems like a better fix, imho.