Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDN tests failing on Windows / MSYS2 #3747

Closed
MarcelRaad opened this issue Apr 7, 2019 · 1 comment
Closed

IDN tests failing on Windows / MSYS2 #3747

MarcelRaad opened this issue Apr 7, 2019 · 1 comment

Comments

@MarcelRaad
Copy link
Member

MarcelRaad commented Apr 7, 2019

I did this

See for example Marc's 6DY-SHZ-5KMWP2---- autobuilds, like this one:
https://curl.haxx.se/dev/log.cgi?id=20190407081751-30470

Note this:

=== Start of file stderr165
curl: (3) Failed to convert www.[EF][BF][BD][EF][BF][BD][EF][BF][BD].se to ACE;
curl: (3) Failed to convert www.gro[EF][BF][BD]e.de to ACE;
=== End of file stderr165

which doesn't happen when run in gdb, which is a native MSYS program, via ./runtests.pl -g (and the original URLs are printed correctly then).
So it seems like MSYS2 does some UTF-8-to-something-else conversion for Windows compatibility.

I expected the following

The tests to either succeed or not be executed (but preferably succeed as the MSYS2 terminal itself is UTF-8 capable).

curl/libcurl version

post-7.64.1 master 2f44e94

operating system

MSYS2 with MinGW-w64 on Windows 10

@MarcelRaad MarcelRaad self-assigned this Apr 8, 2019
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Apr 13, 2019
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8
- make a few char pointers point to const char instead

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Apr 14, 2019
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes
@MarcelRaad
Copy link
Member Author

The problem is that curl receives its command-line arguments with ANSI encoding even though the perl precheck reports a UTF-8 locale. Using Windows's native UTF-16 for them as done in #3784 fixes this issue.

Only test 165 is then still failing because of ß handling in IDNA 2003 vs. 2008.

@bagder bagder closed this as completed in b8ea432 Oct 9, 2019
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 2, 2020
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
- use `wmain` instead of `main` when `_UNICODE` is defined [0]
- define `argv_item_t` as `wchar_t *` in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
@lock lock bot locked as resolved and limited conversation to collaborators Jan 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

2 participants