curl-7.63.0 not sending POST body when using --negotiate

[cstivers78]

I did this

We use kerberos (GSSAPI/SPNEGO) for authenticating against web services. It seems something broke in 7.63.0, where POST bodies are not being transmitted when --negotiate is being used.

Example command use:

curl --negotiate -u ':' -H "Content-Type: application/json" -d '{"audiences":["people"]}' -XPOST https://some.api --trace-ascii trace.log

When I checked the trace.log, I saw:

785: User-Agent: curl/7.63.0
179e: Accept: */*
17ab: Content-Type: application/json
17cb: Content-Length: 0
17de:
=> Send SSL data, 5 bytes (0x5)
0000: ....!
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv header, 13 bytes (0xd)
0000: HTTP/2 400
<= Recv header, 37 bytes (0x25)
0000: date: Mon, 17 Dec 2018 19:41:10 GMT
<= Recv header, 41 bytes (0x29)
0000: content-type: text/plain; charset=utf-8
<= Recv header, 20 bytes (0x14)
0000: content-length: 49
<= Recv header, 62 bytes (0x3e)
0000: www-authenticate: Negotiate <<OMITTED>>
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 49 bytes (0x31)
0000: {"status":400,"message":"Invalid request body."}.
<= Recv SSL data, 5 bytes (0x5)
0000: ....!

Notice Content-Length: 0 and No Body.

I expected the following

I expected to see the body transmitted, with a computed Content-Length header.

For example, the same command ran in curl-7.54.0 yields:

1761: User-Agent: curl/7.54.0
177a: Accept: */*
1787: Content-Type: application/json
17a7: Content-Length: 24
17bb:
=> Send data, 24 bytes (0x18)
0000: {"audiences":["people"]}
== Info: We are completely uploaded and fine
<= Recv header, 13 bytes (0xd)
0000: HTTP/2 200
<= Recv header, 37 bytes (0x25)
0000: date: Mon, 17 Dec 2018 19:34:06 GMT
<= Recv header, 47 bytes (0x2f)
0000: content-type: application/json; charset=utf-8
<= Recv header, 21 bytes (0x15)
0000: content-length: 750
<= Recv header, 62 bytes (0x3e)
0000: www-authenticate: Negotiate <<OMITTED>>
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 750 bytes (0x2ee)
0000:  <<OMITTED>>
== Info: Connection #0 to host some.api left intact

curl/libcurl version

[curl -V output]

curl --version
curl 7.63.0 (x86_64-pc-linux-gnu) libcurl/7.63.0 OpenSSL/1.1.1a zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.4) libssh2/1.8.0 nghttp2/1.34.0
Release-Date: 2018-12-12
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

operating system

Occurrect on both Linux and Mac

[bagder]

Does it work if you use --anyauth instead of --negotiate ?

[gavinhungry]

@bagder,

Using --anyauth in place of --negotiate with 7.63.0 gives me the exact same results as described by @cstivers78 above.

[bagder]

But surely using --anyauth did send a body in the initial request and not a zero content-length one?

[bagder]

something broke in 7.63.0

Do you know the last version in which it worked?

[gavinhungry]

But surely using --anyauth did send a body in the initial request and not a zero content-length one?

@bagder: Yes, I do see a non-zero Content-Length for the initial request, followed by Content-Length: 0 after the server responds.

I'm on Linux, and 7.62.0 works.

[scottgreenup]

Can confirm with @gavinhungry , works in 7.62.0 on Arch Linux and the brew version of curl on macOS. Both of them break in 7.63.0.

[bagder]

There's only a single change done to lib/http_negotiate.c since 7.62.0 and that's 07ebaf8, but it seems innocuous enough...

Would you be able to bisect your way to spot/verify the exact change that broke this functionality? See https://github.com/curl/curl/wiki/how-to-git-bisect on how to do it.

[gavinhungry]

@bagder:

I bisected curl-7_62_0 (good) with curl-7_63_0 (bad) on Arch Linux.

07ebaf837843124ee670e5b8c218b80b92e06e47 is the first bad commit
commit 07ebaf837843124ee670e5b8c218b80b92e06e47
Author: Elia Tufarolo <elia.tufarolo@hcl.com>
Date:   Tue Nov 13 18:30:56 2018 +0100

    http_negotiate: do not close connection until negotiation is completed
    
    Fix HTTP POST using CURLAUTH_NEGOTIATE.
    
    Closes #3275

:040000 040000 84a440504f00b9345a68585a5640ecea5fda9e75 b947fa1d37f0012693c21f1ccd984e7377cea18f M  lib

[michael-o]

Sigh, I just have been hit by this on FreeBSD.

7.62.0 is working:

osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$  $HOME/curl-7.62.0/bin/curl --version
curl 7.62.0-DEV (i386-unknown-freebsd11.2) libcurl/7.62.0-DEV OpenSSL/1.0.2q zlib/1.2.11 libidn2/2.0.5 nghttp2/1.35.1
Release-Date: [unreleased]
Protocols: file http https smtp smtps
Features: AsynchDNS IDN Largefile GSS-API Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy
osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.62.0/bin/curl -k --negotiate -u : --upload-file target/ld-docgen-webapp-0.0.5-SNAPSHOT-backend-dev.war --verbose 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010'
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.62.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 10:14:35 GMT
< Connection: close
<
* Excess found in a non pipelined read: excess = 1034 url = /manager/text/deploy (zero-length body)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* SSL re-using session ID
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* old SSL session ID is stale, removing
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Negotiate with user ''
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> Authorization: Negotiate YIISIgYGKwYBBQUCoIISFj...
> User-Agent: curl/7.62.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 100
* We are completely uploaded and fine
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYH1MIH...
< X-Remote-User: osipovmi@AD001.SIEMENS.NET
< X-Remote-AuthType: SPNEGO
< X-Content-Type-Options: nosniff
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Date: Fri, 04 Jan 2019 10:14:46 GMT
<
OK - Deployed application at context path [/ld-docgen-backend-dev##010]
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):

and now broken in 7.63.0:

osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.63.0/bin/curl --version
curl 7.63.0-DEV (i386-unknown-freebsd11.2) libcurl/7.63.0-DEV OpenSSL/1.0.2q zlib/1.2.11 libidn2/2.0.5 nghttp2/1.35.1
Release-Date: [unreleased]
Protocols: file http https smtp smtps
Features: AsynchDNS IDN Largefile GSS-API Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy
osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.63.0/bin/curl -k --negotiate -u : --upload-file target/ld-docgen-webapp-0.0.5-SNAPSHOT-backend-dev.war --verbose 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'             *   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 09:56:45 GMT
< Connection: close
<
* Excess found in a non pipelined read: excess = 1034 url = /manager/text/deploy (zero-length body)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Negotiate with user ''
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> Authorization: Negotiate YIISIgYGKwYBBQU...
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 0
>
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYH1MIHyoAMKAQC...
< X-Remote-User: osipovmi@AD001.SIEMENS.NET
< X-Remote-AuthType: SPNEGO
< X-Content-Type-Options: nosniff
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Date: Fri, 04 Jan 2019 09:56:46 GMT
<
* Ignoring the response-body
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 09:56:46 GMT
< Connection: close
<
* Closing connection 2
* TLSv1.2 (OUT), TLS alert, close notify (256):
<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/8.5.35</h3></body></html>

I bisected down to:

osipovmi@blnn719x:~/var/Projekte/curl ((07ebaf837...)|BISECTING)
$ git bisect bad
07ebaf837843124ee670e5b8c218b80b92e06e47 is the first bad commit
commit 07ebaf837843124ee670e5b8c218b80b92e06e47
Author: Elia Tufarolo <elia.tufarolo@hcl.com>
Date:   Tue Nov 13 18:30:56 2018 +0100

    http_negotiate: do not close connection until negotiation is completed

    Fix HTTP POST using CURLAUTH_NEGOTIATE.

    Closes #3275

:040000 040000 84a440504f00b9345a68585a5640ecea5fda9e75 b947fa1d37f0012693c21f1ccd984e7377cea18f M       lib

@bagder @kdudka
Can we drop this commit for now until the cause has been identified? This is a showstopper for us.