curl -O and wrong parsing of URL

[7-904-001-67-28]

I did this

In Command Prompt:

1. curl.exe -I --url "https://github.com/PowerShell/vscode-powershell/releases/download/v2021.8.2/powershell-2021.8.2.vsix"
   HTTP/2 302
   …
   location: …
   …
   Redirection needed.
2. Find string started with "location: ", get the new URL and restart cURL:
   curl.exe -I --url "https://github-releases.githubusercontent.com/42131201/f648ac7e-7de8-4a00-851e-f7a1631391d2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210825T124435Z&X-Amz-Expires=300&X-Amz-Signature=2cd843328ab928a1df7a9075151c8ec42c92331985beb6108d81871729143af2&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=42131201&response-content-disposition=attachment%3B%20filename%3Dpowershell-2021.8.2.vsix&response-content-type=application%2Foctet-stream"
   HTTP/2 200
   …
   The URL is good.
3. Change cURL option «-I» to «-O» only, NOT change the URL:
   curl.exe -O --url "https://github-releases.githubusercontent.com/42131201/f648ac7e-7de8-4a00-851e-f7a1631391d2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210825T124435Z&X-Amz-Expires=300&X-Amz-Signature=2cd843328ab928a1df7a9075151c8ec42c92331985beb6108d81871729143af2&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=42131201&response-content-disposition=attachment%3B%20filename%3Dpowershell-2021.8.2.vsix&response-content-type=application%2Foctet-stream"
   curl: (3) URL using bad/illegal format or missing URL

I expected the following

The downloading, not error.

curl/libcurl version

Build: 7.78.0_1
[curl -V output]
curl 7.78.0 (x86_64-pc-win32) libcurl/7.78.0 OpenSSL/1.1.1l (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.2 libssh2/1.9.0 nghttp2/1.44.0 libgsasl/1.10.0
Release-Date: 2021-07-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP UnixSockets zstd

operating system

MS Windows 10 21H1

[bagder]

I can't reproduce on Linux. Here I get a failure because "File name too long" ...

[7-904-001-67-28]

NOT ONLY this URL. Sometimes doubling the ampersands («&») helped. But not always.
Not only in Command Prompt, but also in «MS PowerShell».

[vszakats]

It breaks in the (Windows-specific) call sanitize_file_name() when trying to determine the output filename from the URL in get_url_file_name(). The reason is that the filename part of the URL is longer than 255 characters.

#!/bin/sh

url="$(curl --disable --silent --write-out '%{redirect_url}' --output /dev/null \
  'https://github.com/curl/curl/releases/download/curl-7_78_0/curl-7.78.0.tar.xz.asc')"

curl='curl.exe'

wine64 "${curl}" --disable "${url}" --head  # OK
wine64 "${curl}" --disable "${url}" --remote-name  # fail: 3
wine64 "${curl}" --disable "${url}" --output out.bin  # OK

output:

HTTP/2 200
x-amz-id-2: ********
x-amz-request-id: ********
last-modified: Wed, 21 Jul 2021 07:06:56 GMT
etag: "1c76c2080e3ea486d3756585448249e3"
content-disposition: attachment; filename=curl-7.78.0.tar.xz.asc
content-type: application/octet-stream
server: AmazonS3
via: 1.1 varnish, 1.1 varnish
accept-ranges: bytes
date: Wed, 25 Aug 2021 **:**:** GMT
age: 0
x-served-by: cache-********, cache-********
x-cache: MISS, MISS
x-cache-hits: 0, 0
strict-transport-security: max-age=31536000
x-fastly-request-id: ********
content-length: 488

curl: (3) URL using bad/illegal format or missing URL

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   488  100   488    0     0   5809      0 --:--:-- --:--:-- --:--:--  5809

[bagder]

It breaks in the (Windows-specific) call sanitize_file_name()

Aha! That explains why this only happens on Windows, since that function is only used there.

So perhaps we can help out and clarify this sitation a little bit better? Maybe like this:

diff --git a/src/tool_operate.c b/src/tool_operate.c
index 74221599d..960f3020a 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -948,12 +948,15 @@ static CURLcode single_transfer(struct GlobalConfig *global,
            */
 
           if(!per->outfile) {
             /* extract the file name from the URL */
             result = get_url_file_name(&per->outfile, per->this_url);
-            if(result)
+            if(result) {
+              errorf(global, "Failed to extract a sensible file name"
+                     " from the URL to use for storage!\n");
               break;
+            }
             if(!*per->outfile && !config->content_disposition) {
               errorf(global, "Remote file name has no length!\n");
               result = CURLE_WRITE_ERROR;
               break;
             }

[7-904-001-67-28]

Quote: «when trying to determine the output filename from the URL».
Sorry, but with option «-J» (i.e. curl.exe -J -O …) – the same error. Syntax error number 3, not error of writing to file number 23.
I was trying to simplify the situation, remove EXTRA options.

[bagder]

-J won't help here since there's no Content-Disposition header in there.

This is simply not a URL that -O works well with, and -J doesn't make it better.

[vszakats]

I can confirm the similar error with -J. It appeared to be a separate issue, though I did not investigate further.

On a second look it seems get_url_file_name() is called even when -J is specified, and on Windows (and MS-DOS) this will bail out early if sanitization fails. On other platforms the only similar error condition is out of memory, so it's much less likely to unexpectedly trigger. Ideally, it'd be best not to look at the URL at all with -J.

[vszakats]

@bagder: In my self-contained example there is a content-disposition: attachment; filename=curl-7.78.0.tar.xz.asc header. But even if there wouldn't be one, shouldn't -OJ rather result in curl: Remote file name has no length!?

[7-904-001-67-28]

Here is the information for the option «-J»:
&response-content-disposition=attachment%3B%20filename%3Dpowershell-2021.8.2.vsix&response-content-type=application%2Foctet-stream

Sorry, what header are you writing about? Syntax error. There was no attempts to communicate with the server.

Quote: «not to look at the URL at all with «-J»».
I agree. Or change description of «-J».

[vszakats]

@7-904-001-67-28: After the GET request is actually made (e.g. on non-Windows, or by omitting -O on Windows), and before serving the payload, the server will return a content-disposition response header, with a usable filename in it. That filename is normally used by -J.

[7-904-001-67-28]

Quote: «the server will return a content-disposition response header, with a usable filename in it. That filename is normally used by -J.»
Sorry, error number 3 is a syntax error. SYNTAX. «bad/illegal format». There can be no request to the server. Stop execution immediately.
Or, if there was a query, where is the server response? WHERE?! Nonsense.