curl --head file:// no longer reports the headers

[sjackman]

With curl 7.43.0, curl --head file://… outputs headers.
With curl 7.61.1, curl --head file://… outputs the contents of the file.

I did this

curl --head file:///usr/share/dict/words

I expected the following

Content-Length: 2493109
Accept-ranges: bytes
Last-Modified: Wed, 16 Sep 2015 03:00:17 GMT

I saw the following

a
aa
aal
aalii
aam
…

curl/libcurl version

curl 7.61.1 (x86_64-apple-darwin15.6.0) libcurl/7.61.1 SecureTransport zlib/1.2.5
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 

operating system

macOS 10.11.6 El Capitan

❯❯❯ /usr/bin/curl --head file:///usr/share/dict/words
Content-Length: 2493109
Accept-ranges: bytes
Last-Modified: Wed, 16 Sep 2015 03:00:17 GMT
❯❯❯ curl 7.43.0 (x86_64-apple-darwin15.0.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

[jay]

bisected to c1c2762 which added bold header support but removed setting CURLOPT_HEADER on config->no_body.

This should fix it but I'm not sure what it would break, also tests would need to be adjusted:

diff --git a/src/tool_operate.c b/src/tool_operate.c
index 5a72b6a..1e5743c 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -864,8 +864,15 @@ static CURLcode operate_do(struct GlobalConfig *global,
           my_setopt(curl, CURLOPT_INFILESIZE_LARGE, uploadfilesize);
         my_setopt_str(curl, CURLOPT_URL, this_url);     /* what to fetch */
         my_setopt(curl, CURLOPT_NOPROGRESS, global->noprogress?1L:0L);
-        if(config->no_body)
+        if(config->no_body) {
           my_setopt(curl, CURLOPT_NOBODY, 1L);
+          my_setopt(curl, CURLOPT_HEADER, 1L);
+        }
+         /* If --metalink is used, we ignore --include (headers in
+            output) option because mixing headers to the body will
+            confuse XML parser and/or hash check will fail. */
+        else if(!config->use_metalink)
+          my_setopt(curl, CURLOPT_HEADER, config->show_headers?1L:0L);

         if(config->oauth_bearer)
           my_setopt_str(curl, CURLOPT_XOAUTH2_BEARER, config->oauth_bearer);

And file does not handle CURLOPT_NOBODY on its own which is why the body is being shown

curl/lib/file.c

Lines 416 to 419 in 432eb5f

	/* If we have selected NOBODY and HEADER, it means that we only want file
	information. Which for FILE can't be much more than the file size and
	date. */
	if(data->set.opt_no_body && data->set.include_header && fstated) {

which seems to me should be

  if(data->set.opt_no_body) {
    if(fstated) {
      if(data->set.include_header) {
        ...
      }
      if(!result)
        /* set the file size to make it available post transfer */
        Curl_pgrsSetDownloadSize(data, expected_size);
    }
    return result;
  }

[bagder]

I think the latter part is what needs to be fixed, as I believe the upper part breaks other things...

[jay]

The latter as I wrote it should only correct the issue of the body being shown when it shouldn't be, the headers would still not be shown unless CURLOPT_HEADER is set.

[bagder]

curl -I localhost shows the headers so I would probably say that curl -I file:///usr/share/dict/words should also show the headers. (without --include I mean)

[jay]

Right, that is the regression, it is documented that way. If the bold code now goes through the header callback though I may have broken something else by setting CURLOPT_HEADER, I didn't try this out yet.

[bagder]

Right, that's exactly what I'm thinking. The header output logic was slightly changed with that bolding...

[sjackman]

I've just noticed that macOS curl 7.43.0 had a different bug. curl -I works as expected, but curl -i does not report both the headers and the content as expected.

Observed

❯❯❯ curl -I file://$PWD/hello
Content-Length: 14
Accept-ranges: bytes
Last-Modified: Wed, 03 Oct 2018 17:31:42 GMT
❯❯❯ curl -i file://$PWD/hello
Hello, world!

Expected

❯❯❯ curl -i file://$PWD/hello
Content-Length: 14
Accept-ranges: bytes
Last-Modified: Wed, 03 Oct 2018 17:31:42 GMT

Hello, world!

[jay]

-i is not documented to work with file. there may be some undocumented behavior that varies depending on the version.

[sjackman]

Feature request then: could -i and -I work with file://?

[jay]

-I not working for file is a bug which we'll fix but -i iirc was really intended just for HTTP headers. In the short description it shows as "Include protocol response headers in the output" which could imply other protocols, so I'm on the fence on that one.

[sjackman]

curl -i for file:// is not terribly important for me. It's just for consistency. I'd also be okay with documenting that it doesn't work. curl -I for file:// is important to me.