tool: support UTF-16 command line on Windows

[MarcelRaad]

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests.

Out of the box, this currently only works with the Visual Studio
project files, which default to Unicode. I'll look into building
Unicode versions with the other build tools later.

Ref: #3747

[jay]

Refer to #637 and #345 particularly this post. The problem is basically the Microsoft C runtime does not support UTF-8 locale. Lets say you have an outfile name extracted from the UTF-8 encoded URL or a separate UTF-8 encoded argument then you can't pass that to fopen. We're left with a problem where we have to somehow keep track of the encoding of strings because we can't set the locale to UTF-8.

Anyway I think this is possible to support I just feel like it's going down a rabbit hole.

[MarcelRaad]

Thanks @jay , I hadn't thought about that! Reading these issues now. I wish there were char8_t in C.

[bagder]

hi @MarcelRaad, is this something you still intend to finalize?

[MarcelRaad]

Sorry for the long delay - yes, I plan to resume work on this when I have a couple of days, hopefully soon.

[MarcelRaad]

If there are no objections, I'd like to merge this at the beginning of the upcoming feature window. It could still be improved to handle non-Unicode builds later as done by @jay and @captain-caveman2k in their branches. But I don't see any reason for non-Unicode builds on Windows (pre-Windows 95 compatibility and compatibility for Windows 9x without UnicoWS isn't an issue, is it?), and I'm going to look into adding options for supporting Unicode Windows builds to the CMake and autotools builds after this has landed.

[mback2k]

I would like to see the CI build failures fixed before a merge, e.g.:
https://curlbuild.uxnr.de/builders/curl_winssl_cross_x64/builds/16906/steps/compile/logs/stdio

vtls/schannel.c: In function 'schannel_connect_step1':
vtls/schannel.c:636:9: error: implicit declaration of function 'Curl_unicodefree' [-Werror=implicit-function-declaration]
         Curl_unicodefree(cert_path);
         ^~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors

[jay]

curl_free now that it's a curlx function. libcurl may be using a CRT different from the application. If curlx.h is available publicly for any application I don't know if including this file will work like this, because it is an internal lib file.

[MarcelRaad]

Good point! I think it's the other way around though: curlx functions should never use curl's memory functions as they're not included in curlx. That's what the previous curlx functions dealing with memory did, like curlx_strdup before commit b387560.

Note that the curlx functions are not public, but instead the tool and tests build their own copies of them, so using the curl memory functions would result in linker errors.

Therefore, I changed the macro to always use the non-macro version of free as that's what's used for malloc in curl_multibyte.c.

[jay]

Note that the curlx functions are not public, but instead the tool and tests build their own copies of them, so using the curl memory functions would result in linker errors.

Fair enough, I did not know that. I think @bagder could review the curlx handling since I'm not entirely grokking it.

[bagder]

I'm with @MarcelRaad here. curlx functions are used in the tool's domain so they would not use libcurl's CRT.

[jay]

this is another instance where we are passing something opened from libcurl possibly to an application using a different CRT

[jay]

Aside from the curlx_ issues I think should be carefully examined (I just posted some comments; admittedly I don't understand curlx that well), I don't have an objection if you are going to follow up with maintenance on this. I just don't think it's going to be this easy, but let's find out.

[MarcelRaad]

@mback2k

I would like to see the CI build failures fixed before a merge

Sure! They're new after today's rebase. Admittedly, I haven't compiled the branch locally between rebasing and pushing.

[MarcelRaad]

Any idea how I can see what went wrong with the "CI / macos normal (pull_request)" job? All I see there is a big white cross and "This check failed" without any visible link or button.

Also, it seems like 148534d broke the AppVeyor tests when both OpenSSL and WinSSL are off. 36 tests are failing with "curl: (48) An unknown option was passed in to libcurl".

[mback2k]

Any idea how I can see what went wrong with the "CI / macos normal (pull_request)" job? All I see there is a big white cross and "This check failed" without any visible link or button.

This happens randomly with macOS builds on GitHub Actions (similar to the random failures on travis-ci before), because all CIs use MacStadium for hosting their macOS runners. If there is an issue with the runner it will show up as a failed CI run, but GitHub Action will retry running it. This is why you can see a 2nd result of "CI / macos normal (pull_request)" showing up as successful.