Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--etag-save failing with error "(23) Failed writing header" #5610

Closed
bhartshorn opened this issue Jun 25, 2020 · 7 comments
Closed

--etag-save failing with error "(23) Failed writing header" #5610

bhartshorn opened this issue Jun 25, 2020 · 7 comments

Comments

@bhartshorn
Copy link

@bhartshorn bhartshorn commented Jun 25, 2020

I did this

Against a specific server (get.helm.sh), curl returns an error when using the --etag-save flag. I have used this feature with other servers (specifically github releases) without issue. I'm not familiar enough with http to troubleshoot this further. I tested on two machines: latest MacOS and Arch Linux, each with fully updated curl versions (7.71 & 7.70).

MacOS (installed via homebrew) output:

> uname -a
Darwin [masked hostname] 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

> /usr/local/opt/curl/bin/curl -V
curl 7.71.0 (x86_64-apple-darwin19.5.0) libcurl/7.71.0 SecureTransport zlib/1.2.11
Release-Date: 2020-06-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

> /usr/local/opt/curl/bin/curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: sni114fcgl.wpc.edgecastcdn.net
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/1.1
> Host: get.helm.sh
> User-Agent: curl/7.71.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 43
< Content-MD5: LntYfUTjxG3ILF3WkCeKzQ==
< Content-Type: application/x-tar
< Date: Thu, 25 Jun 2020 15:24:11 GMT
< Etag: 0x8D8117F40DD72AC
* Failed writing header
* Closing connection 0
curl: (23) Failed writing header

Arch Linux output:

[brandon@[masked hostname] ~]$ uname -a
Linux [masked hostname] 5.7.5-arch1-1 #1 SMP PREEMPT Mon, 22 Jun 2020 08:10:02 +0000 x86_64 GNU/Linux

[brandon@[masked hostname] ~]$ curl --version
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1g zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.41.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

[brandon@[masked hostname] ~]$ curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [88 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [155 bytes data]                                                                                                                                              [26/47]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2971 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Los Angeles; O=Verizon Digital Media Services, Inc.; OU=SecOps; CN=sni114fcgl.wpc.edgecastcdn.net
*  start date: May  8 00:00:00 2019 GMT
*  expire date: May 12 12:00:00 2021 GMT
*  subjectAltName: host "get.helm.sh" matched cert's "get.helm.sh"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55b2b261b8b0)
} [5 bytes data]
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/2
> Host: get.helm.sh
> user-agent: curl/7.70.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< accept-ranges: bytes
< content-md5: LntYfUTjxG3ILF3WkCeKzQ==
< content-type: application/x-tar
< date: Thu, 25 Jun 2020 15:40:59 GMT
< etag: 0x8D8117F40DD72AC
* Failed writing header
} [5 bytes data]
* stopped the pause stream!
* Connection #0 to host get.helm.sh left intact
curl: (23) Failed writing header

I expected the following

I would expect curl to succesfully download the file, and write the etag value to the specified file.

curl/libcurl version

Found in output above ^^

operating system

Found in output above ^^

Other

Happy to help troubleshoot further if needed.

@jay
Copy link
Member

@jay jay commented Jun 26, 2020

You've silenced the warnings which is why you don't see:

< Etag: 0x8D8117F40DD72AC
Warning:
Received header etag is missing double quote/s

curl/src/tool_cb_hdr.c

Lines 115 to 128 in e9db32a

/*
* if server side messed with the etag header and doesn't include
* double quotes around the etag, kindly exit with a warning
*/
if(!first) {
warnf(per->config->global,
"\nReceived header etag is missing double quote/s\n");
return 1;
}
else {
/* discard first double quote */
first++;
}

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

/cc @s-3ntinel

jay added a commit to jay/curl that referenced this issue Jun 26, 2020
- Return 'failure' on failure, to follow the existing style.

- Put Warning: and the warning message on the same line.

Ref: curl#5610
jay added a commit that referenced this issue Jun 26, 2020
- Return 'failure' on failure, to follow the existing style.

- Put Warning: and the warning message on the same line.

Ref: #5610

Closes #5612
@jay
Copy link
Member

@jay jay commented Jun 26, 2020

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

I fixed that but I think we should change the behavior to not fail in this case and instead ignore the invalid etag (or ignore the double quotes if there's some precedent for it).

@bagder
Copy link
Member

@bagder bagder commented Jun 26, 2020

Yes, I too think it should rather ignore it when invalid (but probably say something in the verbose output).

@sterchelen
Copy link
Contributor

@sterchelen sterchelen commented Jun 27, 2020

To start playing with the code, can I take care of your proposal?

@bagder
Copy link
Member

@bagder bagder commented Jun 27, 2020

@sterchelen please go ahead, this could certainly be a good entry step into the code. This is entirely cmdline tool-side code. My advice: submit the first take on your PR early to solicit feedback on your approach!

@bhartshorn
Copy link
Author

@bhartshorn bhartshorn commented Jun 29, 2020

You've silenced the warnings which is why you don't see:

D'oh. I figured there was some PEBCAK at play here also. Thanks for looking at this.

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

@sterchelen
Copy link
Contributor

@sterchelen sterchelen commented Jun 29, 2020

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

In fact, as the rfc specify:

An entity-tag consists of an opaque quoted string

@bagder bagder linked a pull request that will close this issue Aug 24, 2020
@bagder bagder closed this in 98c9459 Aug 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.