argv is ANSI on Windows, so non-ASCII URLs and paths are lossy (charset, proxyurl, idna self-tests)

[xroche]

Following #565, three more self-tests fail on Windows, and they share one root cause: argv is ANSI on Windows, not UTF-8, so any non-ASCII command-line argument is transcoded to the local codepage before the engine ever sees it.

httrack.exe is an MBCS build, so the CRT converts the UTF-16 command line down to the ANSI codepage to produce char **argv. Bytes outside that codepage do not survive.

Evidence (traces from an MSVC x64 build, identical on Win32)

-#test=charset — UTF-8 in, CP1252 out:

++ httrack -O /dev/null -#test=charset utf-8 $'caf\303\251'      # café, UTF-8 (2 bytes)
+  test $'caf\351' == $'caf\303\251'                             # got 1 byte 0xE9 = CP1252

-#test=proxyurl:

++ httrack -O /dev/null -#test=proxyurl $'socks5://na\303\257ve:pass@h\303\251llo:1080'
+  test $'name=socks5://na\357ve:pass@h\351llo port=1080 host=h\351llo' == $'...na\303\257ve...h\303\251llo...'

-#test=idna-encode — the mangled bytes are then not valid UTF-8, so IDNA rejects them:

++ httrack -O /dev/null -#test=idna-encode $'www.caf\303\251.com'
invalid string 'www.caf?.com'
+  test '' == www.xn--caf-dma.com

Why this is not just a test problem

The tests pass UTF-8 because on Linux argv is UTF-8. On Windows it is CP1252 (or CP932, CP936, … depending on the machine), so:

• a non-ASCII URL or output path given to httrack.exe on the command line is lossy, and
• anything outside the machine's ANSI codepage is destroyed outright — a Cyrillic or CJK URL on a Western-codepage box becomes ?.

HTTrack has a lot of users mirroring non-Latin sites, so this looks worth fixing rather than papering over in the tests.

The usual remedy is to take the command line as UTF-16 on Windows (wmain, or GetCommandLineW + CommandLineToArgvW) and convert to UTF-8 before handing it to the engine, so argv has the same encoding on every platform.

Reproducing

Build src/httrack.vcxproj (VS2022 v143, MBCS) and run the three commands above with UTF-8 bytes. From the httrack-windows CI: https://github.com/xroche/httrack-windows/actions/runs/29345824840 (step "Run the engine test suite (offline tests)"; failing tests are re-run under bash -x, and the artifact engine-tests-x64-Release has the full traces).

Separately

01_engine-filter.test fails because httrack -#test=filterbounds prints nothing and exits without output — it is registered unconditionally, so it should run. That self-test drives the matcher deep on purpose, and Windows gives a thread 1 MB of stack where Linux gives 8 MB, so a stack overflow is my first suspicion — but I have not confirmed it and am currently capturing the exit code. I will follow up here rather than guess.

01_engine-ftp-line.test and 01_engine-filelist.test are test-portability issues, not engine bugs: the ftp-line self-test is #ifndef _WIN32 (htsselftest.c) so the test can never pass on Windows and needs a skip, and filelist relies on chmod 000 denying reads, which Windows ignores. Happy to send a PR for both.

[xroche]

Decision (Xavier): go with the command-line conversion, not the manifest.

For the record, <activeCodePage>UTF-8</activeCodePage> in the app manifest would fix this in one line — but it needs Windows 10 1903+ and is silently ignored below that. HTTrack's Windows floor is Windows 7 SP1 precisely because a lot of users are on very old Windows, and those are the same machines running CP1251/CP936/CP1256 where a non-Latin URL is destroyed today. So the manifest route would fix it for the users who need it least. It also changes the ANSI codepage process-wide, which is a large blast radius for the MBCS MFC GUI and its Latin-1 resources.

The targeted fix is also the smaller one, because the engine has already committed to "char* is UTF-8 on Windows" everywhere else:

#define FOPEN  hts_fopen_utf8
#define STAT   hts_stat_utf8
#define UNLINK hts_unlink_utf8

Those convert UTF-8 -> UTF-16 at the syscall boundary themselves and do not depend on the ANSI codepage. argv is the one thing that never got the memo. Converting it at the entry point (GetCommandLineW + CommandLineToArgvW + WideCharToMultiByte(CP_UTF8, ...)) works on Windows 7, needs no OS gate, and touches one function.

Happy to send the PR if it is not already in hand — say the word and I will, otherwise it is yours.