[Windows] Long-filename problem

[gvanem]

There is an issue with access to filenames with >= 260 characters in the Windows version of curl.
I used this <curl_root>\upload-long-file.bat file to verify:

@echo off
setlocal
::
:: File to upload: 261 bytes long (446 bytes) [1]
::
set LONG_FILE=f:\MingW32\src\GUI\Harlinn\2022.01.11\Tests\3rdParty\gdal\data\gdrivers\sentinel2\fake_l2a_MSIL2Ap\S2A_MSIL2A_20170823T094031_N0205_R036_T34VFJ_20170823T094252.SAFE\GRANULE\L2A_T34VFJ_A011330_20170823T094252\IMG_DATA\R60m\L2A_T34VFJ_20170823T094031_AOT_60m.jp2

::
:: A copy of the above, but 260 bytes long
::
set SHORT_FILE=f:\MingW32\src\GUI\Harlinn\2022.01.11\Tests\3rdParty\gdal\data\gdrivers\sentinel2\fake_l2a_MSIL2Ap\S2A_MSIL2A_20170823T094031_N0205_R036_T34VFJ_20170823T094252.SAFE\GRANULE\L2A_T34VFJ_A011330_20170823T094252\IMG_DATA\R60m\L2A_T34VFJ_20170823T094031_AOT_60m.jp

set URL=ftp://ftp.xxx.yyy.com

if %1. == long. (
  echo on
  %~dp0\src\curl.exe -vT %LONG_FILE% %URL%
) else (
  echo on
  %~dp0\src\curl.exe -vT %SHORT_FILE% %URL%
)

With a upload-long-file.bat long, curl fails to open the %LONG_FILE:

curl: Can't open 'f:\MingW32\src\GUI\Harlinn\2022.01.11\Tests\3rdParty\gdal\data\gdrivers\sentinel2\fake_l2a_MSIL2Ap\
S2A_MSIL2A_20170823T094031_N0205_R036_T34VFJ_20170823T094252.SAFE\GRANULE\
L2A_T34VFJ_A011330_20170823T094252\IMG_DATA\R60m\L2A_T34VFJ_20170823T094031_AOT_60m.jp2'!
curl: try 'curl --help' or 'curl --manual' for more information
curl: (26) Failed to open/read local data from file/application

Seems the problem is here which does not support such long-filenames AFAICS.
According to MSDN, this needs CreateFileW() as "\\?\" + per->uploadfile.

curl/libcurl version

curl 7.82.0-DEV (i386-pc-win32) libcurl/7.82.0-DEV wolfSSL/5.0.1 (GnuTLS/3.7.2) (mbedTLS/2.25.0) (OpenSSL/3.1.0)
(Schannel) (rustls-ffi/0.8.2/rustls/0.20.2) zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 c-ares/1.18.1 WinIDN libpsl/0.21.1
(no IDNA support) libssh2/1.9.0_DEV librtmp/2.3
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb
smbs smtp smtps telnet tftp
Features: AsynchDNS Debug HSTS IDN Kerberos Largefile MultiSSL NTLM PSL SPNEGO SSL SSPI TLS-SRP TrackMemory
UnixSockets alt-svc brotli libz zstd

operating system

Windows-10, 21H1, 19043.1503.

[1] From the "Geospatial Data Abstraction Library" project.

[bagder]

Changing from open() to CreateFileW() for Windows looks like it can cause decent "ripples" elsewhere in the code too.

[gvanem]

We can still use similar code like (untested):

int curlx_open (const char *file, int flags)
{
#ifdef _WIN32
  if (strlen(file) >= MAX_PATH) {
     wchar_t w_file [32*1024 + 1];
     snwprintf(w_file sizeof(w_file)/2, L"\\\\?\\%S", file);
     return _wopen(w_file, flags);
  }
#elif defined(__VMS)
   ...
#endif
  return open(file, flags);
}

But how to report this true file-name (w_file) in case that fails too?

[jay]

Windows has never worked right with long paths 260+ characters created with \\?\. Maybe you can create them but some random OS function called by some other random OS function can't read them or says they don't exist, or other nonsense and then your application goes bad in ways you didn't expect. MS seems to be aware of this and added some better Windows API long path support without \\?\ in Windows 10, if a registry key is set and the application is manifested for it. Who's doing that yet, I don't know. The support may also apply to CRT functions (fopen, wfopen etc) which eventually call the Windows API, however MAX_PATH continues to be defined as 260.

\\?\ is a prefix to tell Windows do not do any interpretation of the path. It may allow for long paths depending on the API call but --aside from the better support option-- suffers from the nonsense problem I mentioned. The user can specify \\?\ on their own if they have a fully qualified path, and if the file is longer than MAX_PATH possibly curl will be able to create the file and complete successfully. We know \\?\ + long path works with CreateFileW but will it work with other Windows API calls the same way? Not guaranteed.

Most of the time the user is going to pass something informal like -o foo or -o ..\bar or even -o c:/baz/qux. None of those will be understood at the lower levels without OS interpretation. In other words \\?\ + foo = \\?\foo isn't going to work. There may be functions that we can use to imitate retrieving a fully qualified path to some degree, but we may break other things in the process and I'm not convinced it's worth it.

I tried curl -o \\?\longpath with two curl builds (one Unicode and one not) in Windows 7 Enterprise and both of them worked:

curl 7.82.0-DEV (i386-pc-win32) libcurl/7.82.0-DEV OpenSSL/1.1.1m nghttp2/1.46.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug HSTS HTTP2 HTTPS-proxy IPv6 Largefile NTLM SSL Unicode UnixSockets alt-svc


curl 7.82.0-DEV (i686-w64-mingw32) libcurl/7.82.0-DEV mbedTLS/2.28.0 zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libssh2/1.10.0 nghttp2/1.46.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli Debug HSTS HTTP2 IPv6 Kerberos Largefile libz NTLM SPNEGO SSL SSPI TrackMemory zstd

int curlx_open (const char *file, int flags)

There's already Windows curlx file functions for opening Unicode file names. My gripe with the MS CRT is we should not have to make workarounds for long paths or UTF-8 it should just support it. Otherwise we end up with a bunch of things we constantly have to maintain with ever more hacks as users report problems and I just don't like that.

[gvanem]

I already had the HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled set to 1 as mentioned in your
MSDN link.

So just changing my .bat file to say %~dp0\src\curl.exe -vT \\?\%LONG_FILE% %URL%, the open() works fine.
I did not change the manifest in curl.rc to specify this XML non-sense mentioned in that link:
<ws2:longPathAware>true</ws2:longPathAware>

So I agree, it's a messy situation with many such APIs in Windows.

[ghost]

Long Path works on windows 10 v1607 and upwards with opt-in.
Benefit : No need for \\? or UAC Path Handling Complexities Nonsense.

---

build curl with following embedded manifest file.

File name must be app.manifest

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>
</assembly>

---

Users must Set Long Path Aware On system wide through Registry : Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem make a DWORD value LongPathsEnabled and set the value to 1

Why System Wide Long Path is not on by default (?) according to a MS engineer : microsoft/WindowsAppSDK#875 (comment)

---

Following APIs change behaviors after opt-in :

CopyFileW, CopyFile2, CopyFileExW, CreateFileW, CreateFile2, CreateHardLinkW, CreateSymbolicLinkW, DeleteFileW, FindFirstFileW, FindFirstFileExW, FindNextFileW, GetFileAttributesW, GetFileAttributesExW, SetFileAttributesW, GetFullPathNameW, GetLongPathNameW, MoveFileW, MoveFileExW, MoveFileWithProgressW, ReplaceFileW, SearchPathW, FindFirstFileNameW, FindNextFileNameW, FindFirstStreamW, FindNextStreamW, GetCompressedFileSizeW, GetFinalPathNameByHandleW.

Ref:
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd

[bagder]

Did this end up in any conclusion? Can/should we change anything or do we live with it?

[jay]

Fixing this would mean changing the code that uses MAX_PATH and setting long path aware in the manifest. And as I mentioned earlier it wouldn't actually be supported unless the user has set LongPathsEnabled in the registry, and even then there's no guarantee it will work with CRT functions. I'd prefer to document it as a known bug and explain the user may be able to use the \\?\ prefix as a workaround. I'm curious what others think.

[ghost]

Feel free to ignore everything written below.

---

Fixing this would mean changing the code that uses MAX_PATH

No need. MAX_PATH const in the code will remain untouched that's the whole point of manifest.

Using manifest won't help only when a library hardcodes 256 char instead of MAX_PATH, neither will using \\? in that case.

Windows itself will change APIs behaviors upon seeing the manifest for that particular executable. ( that's why the manifest file has to be embedded per executable.)

Python, CMD, Windows PowerShell, whole .NET runtime supports long path using the manifest in that way. https://docs.microsoft.com/en-us/archive/blogs/jeremykuhne/net-4-6-2-and-long-paths-on-windows-10#enabling-win32-long-path-support

even then there's no guarantee it will work with CRT functions.

All CRT functions maps to WinAPI. so the guarantee is there. Python does that. https://stackoverflow.com/questions/5604699/crt-and-win32-api

I mentioned earlier it wouldn't actually be supported unless the user has set LongPathsEnabled in the registry
I'd prefer to document it as a known bug and explain the user may be able to use the \\?\ prefix as a workaround.

it might be better to add a line somewhere in the docs that says-

"As per microsoft recommended way, Users who wish to use long path must use windows 10 and enable it system wide by going to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem make a DWORD value LongPathsEnabled and set the value to 1"

User may use \\?\ workaround on windows Vista, 7 , 8 platforms. after all manifest won't break the app on those platforms.

I will prefer avoiding \\?\ simply because not all WinAPI support it especially device paths(\\.\) , hence the additional complexities of using \\?\ paths.

---

Not sure relevant or not, even upcoming cygwin v3.4 will be the last supported version on vista, 7, 8 platforms. https://cygwin.com/pipermail/cygwin-announce/2022-January/010438.html

[jay]

Fixing this would mean changing the code that uses MAX_PATH

No need. MAX_PATH const in the code will remain untouched that's the whole point of manifest.

We use PATH_MAX (MAX_PATH) in tool_doswin for some path buffers when not \\?\ so that code would need to be changed and the manifest changed as well.

All CRT functions maps to WinAPI. so the guarantee is there. Python does that.

There is no guarantee. The restriction has been removed from "common" Windows API functions and that's it. It may work.

[ghost]

in case it boils down to Manifest not supporting all WinAPI such as CreateProcessW and friends (well \\?\ is not supported in these functions either i.e pretty much any APIs that takes lpCommandLine parameter alone without lpApplicationName) , as more and more apps use Manifest, those functions will likely support long path through manifest in future windows versions as opposed to \\?\ which didn't happen in decades and likely never ;).

with Manifest, when these function supports beyond MAX_PATH buffer, application authors won't have to do anything and they will get benefit automatically in future windows versions.

therefore I conclude with :
for Short Term : yes continue using \\?\.
for Long term : use manifest. no hassle in future.

[bagder]

Let's document this as a known bug/limitation, especially since nobody is working on addressing it. Anyone feels inclined to write up a description ?

[jay]

I'd prefer to document it as a known bug and explain the user may be able to use the \\?\ prefix as a workaround.

This issue has been documented as known bug 5.13.