--etag-save failing with error "(23) Failed writing header"

[bhartshorn]

I did this

Against a specific server (get.helm.sh), curl returns an error when using the --etag-save flag. I have used this feature with other servers (specifically github releases) without issue. I'm not familiar enough with http to troubleshoot this further. I tested on two machines: latest MacOS and Arch Linux, each with fully updated curl versions (7.71 & 7.70).

MacOS (installed via homebrew) output:

> uname -a
Darwin [masked hostname] 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

> /usr/local/opt/curl/bin/curl -V
curl 7.71.0 (x86_64-apple-darwin19.5.0) libcurl/7.71.0 SecureTransport zlib/1.2.11
Release-Date: 2020-06-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

> /usr/local/opt/curl/bin/curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: sni114fcgl.wpc.edgecastcdn.net
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/1.1
> Host: get.helm.sh
> User-Agent: curl/7.71.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 43
< Content-MD5: LntYfUTjxG3ILF3WkCeKzQ==
< Content-Type: application/x-tar
< Date: Thu, 25 Jun 2020 15:24:11 GMT
< Etag: 0x8D8117F40DD72AC
* Failed writing header
* Closing connection 0
curl: (23) Failed writing header

Arch Linux output:

[brandon@[masked hostname] ~]$ uname -a
Linux [masked hostname] 5.7.5-arch1-1 #1 SMP PREEMPT Mon, 22 Jun 2020 08:10:02 +0000 x86_64 GNU/Linux

[brandon@[masked hostname] ~]$ curl --version
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1g zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.41.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

[brandon@[masked hostname] ~]$ curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [88 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [155 bytes data]                                                                                                                                              [26/47]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2971 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Los Angeles; O=Verizon Digital Media Services, Inc.; OU=SecOps; CN=sni114fcgl.wpc.edgecastcdn.net
*  start date: May  8 00:00:00 2019 GMT
*  expire date: May 12 12:00:00 2021 GMT
*  subjectAltName: host "get.helm.sh" matched cert's "get.helm.sh"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55b2b261b8b0)
} [5 bytes data]
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/2
> Host: get.helm.sh
> user-agent: curl/7.70.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< accept-ranges: bytes
< content-md5: LntYfUTjxG3ILF3WkCeKzQ==
< content-type: application/x-tar
< date: Thu, 25 Jun 2020 15:40:59 GMT
< etag: 0x8D8117F40DD72AC
* Failed writing header
} [5 bytes data]
* stopped the pause stream!
* Connection #0 to host get.helm.sh left intact
curl: (23) Failed writing header

I expected the following

I would expect curl to succesfully download the file, and write the etag value to the specified file.

curl/libcurl version

Found in output above ^^

operating system

Found in output above ^^

Other

Happy to help troubleshoot further if needed.

[jay]

You've silenced the warnings which is why you don't see:

< Etag: 0x8D8117F40DD72AC
Warning:
Received header etag is missing double quote/s

curl/src/tool_cb_hdr.c

Lines 115 to 128 in e9db32a

	/*
	* if server side messed with the etag header and doesn't include
	* double quotes around the etag, kindly exit with a warning
	*/
	if(!first) {
	warnf(per->config->global,
	"\nReceived header etag is missing double quote/s\n");
	return 1;
	}
	else {
	/* discard first double quote */
	first++;
	}

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

/cc @s-3ntinel

[jay]

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

I fixed that but I think we should change the behavior to not fail in this case and instead ignore the invalid etag (or ignore the double quotes if there's some precedent for it).

[bagder]

Yes, I too think it should rather ignore it when invalid (but probably say something in the verbose output).

[sterchelen]

To start playing with the code, can I take care of your proposal?

[bagder]

@sterchelen please go ahead, this could certainly be a good entry step into the code. This is entirely cmdline tool-side code. My advice: submit the first take on your PR early to solicit feedback on your approach!

[bhartshorn]

You've silenced the warnings which is why you don't see:

D'oh. I figured there was some PEBCAK at play here also. Thanks for looking at this.

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

[sterchelen]

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

In fact, as the rfc specify:

An entity-tag consists of an opaque quoted string