curl can't open Unicode filenames in Windows

[z0hm]

WinXP SP2, cURL 7.43.
cURL can't open a file for transmission when the file name contains characters from a different code page in OS. cURL - don't support unicode?

[dfandrich]

Can you give an example including logs?

[jay]

I know I've heard of this issue before but I can't find it. For example a UTF-8 encoded batch file like this won't work:

chcp 65001
curl -F filedata=@И.txt http://website
curl: (26) couldn't open file "?.txt"

65001 is the UTF-8 codepage and the И is UTF-8 encoded there. Same thing with cyrillic code page. In Process Monitor I can see that the error is "NAME INVALID" but I don't know why that error.

"19","ntdll.dll","NtCreateFile + 0x12","0x773b0112","C:\Windows\SysWOW64\ntdll.dll"
"20","KernelBase.dll","CreateFileW + 0x35e","0x758ac5fd","C:\Windows\SysWOW64\KernelBase.dll"
"21","kernel32.dll","CreateFileW + 0x4a","0x759c3f56","C:\Windows\SysWOW64\kernel32.dll"
"22","kernel32.dll","CreateFileA + 0x36","0x759c53b4","C:\Windows\SysWOW64\kernel32.dll"
"23","msvcrt.dll","clearerr_s + 0x75b","0x76a1a310","C:\Windows\SysWOW64\msvcrt.dll"
"24","msvcrt.dll","sopen_s + 0x79","0x76a1a789","C:\Windows\SysWOW64\msvcrt.dll"
"25","msvcrt.dll","sopen_s + 0x1b","0x76a1a72b","C:\Windows\SysWOW64\msvcrt.dll"
"26","msvcrt.dll","remove + 0x137","0x76a1a628","C:\Windows\SysWOW64\msvcrt.dll"
"27","msvcrt.dll","fsopen + 0x6a","0x76a1a6c1","C:\Windows\SysWOW64\msvcrt.dll"
"28","msvcrt.dll","fopen + 0x12","0x76a1b2d6","C:\Windows\SysWOW64\msvcrt.dll"

The CRT specific locale or something might need to be changed, or use the version of the command line that's UTF-16 encoded and work with that and _wfopen, maybe.

[z0hm]

WinXPSP2 RUS (CP for non unicode app -1251).

lua script in utf8 1.lua:
---------------------------
local function fread(f) local h,x = io.open(f,"rb"),nil if h then x=h:read("*all"); io.close(h) end return x end 
local s1=fread("1.txt") 
local s2=fread("2.txt") 
curl ... -T '"'..s1..'"' ...
curl ... -T '"'..s2..'"' ...

---------------------------

1.txt in utf8 (65001)
--------------
Read Me.txt

--------------

2.txt in utf8 (65001)
--------------
Første Pucambù.txt

--------------

command line for run: lflua.exe 1.lua

lflua.exe -- lua 5.1 interpreter from unicode FAR3 (http://www.farmanager.com/index.php?l=en)

Read Me.txt -- transfer good
Første Pucambù.txt -- don't transfer, curl answer: don't open file

[dbyron0]

I agree that WinMain (or perhaps GetCommandLineW) and _wfopen look like the way to address this. If it's helpful to include the filename in logging/debug output, it may also mean changing the logic here: https://github.com/bagder/curl/blob/master/lib/curl_multibyte.c#L25 to get these functions whenever we build for windows.

I imagine this is a big enough blob to bite off at once, but I am tempted to mention that if we want support for long (~> 260 characters) file names on windows, _wfopen isn't sufficient and we need to drop down to CreateFile/ReadFile/WriteFile and deal with the joy of HANDLEs. See https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath for some of the details.

[vszakats]

Not only GetCommandLineW(), but all string (filename) interactions via Windows API should be done using the "WIDE" variant of said API functions. F.e. in above case it'd mean calling CreateFileW() instead of CreateFileA(). And because this one is called via C RTL function fopen(), using _wfopen() may be used in this case. I would opt to go with the direct API calls to avoid messing with C RTL codepage and compatibility issues altogether. Another question to consider here is what should be the UNICODE encoding expected by libcurl API functions that expect or return strings. For portability and to retain ABI compatibility, probably UTF-8 would be best.

Another issue to tackle (if this is a concern) is how to stay compatible with Windows versions that do not natively support UNICODE ("WIDE") API variants. These Windows versions are Win95/Win98/WinME and these will either need a special build that keeps using non-WIDE ("ANSI") API variants, or use the unicows.dll layer to make them work transparently with WIDE ones. Not sure how to handle this when dealing with C RTL functions. Supporting these old versions for certain needs a C compiler that is also compatible with them; MinGW is, and MS Visual Studio 2005 or older are (plus most other 3rd party C compilers).

Another potentially interesting note is that WinCE OS only supports the "WIDE" APIs (with some minor exceptions probably not relevant in context of libcurl). Such differences should be hidden by the C RTL layer, if used.

[DemiMarie]

The Microsoft-recommended method of handling Unicode/ASCII compatibility is to

#define UNICODE 1
#define _UNICODE 1

in a header included by every source file. Then use the un-suffixed versions of the functions.

Other notes:

• As mentioned above, never call fopen() -- use _wfopen(). This probably needs to go in a compatibility shim layer that abstracts over this.
• Any write to a file not opened by libcurl must be considered to potentially be a Windows console handle. In this case, the CRT functions must not be used, as they do not support Unicode -- one must use WriteConsoleW() and do all buffering manually. This is especially serious for the command-line curl utility.

[bagder]

I'll welcome a patch from someone that has been tested with a fair degree of success (on Windows).

[jay]

I was kind of hoping someone would pick up on this but it's about that time it goes in the TODO. To see how this might work I wrote a draft with the idea to convert the command line arguments to UTF-8 (see the discussion in #637) so we can continue to pass the user input around as char *. When files are opened or statused then convert from UTF-8 to UTF-16 encoding. There is no way to set the locale as UTF-8 so fopen will open as UTF-8, afaict.

Things like this work:

-v --output спасти.txt http://россия.net
-v -F filedata=@спасти.txt http://example.com/

The first one the host is an IDN and it's converted to UTF-8 and if a WinIDN build is used it's later converted to punycode (xn--h1alffa9f.net).

Has some problems though like the whole URL is now UTF-8. output to the screen is not UTF-8 so any command line input that is output to the screen is incorrect:

* Connection #0 to host Ñ?D_Ñ?Ñ?D,Ñ?.net left intact

The way I did it was sloppy, just to get it working to see how it might work. I'm using a global variable in the DLL g_curl_tool_args_are_utf8 and also I have some duplicated code like the fopen and stat wrappers. But it seems to work in the case of filenames and hostnames.

The draft is here:
https://github.com/curl/curl/compare/master...jay:win-utf8-test?expand=1