Curl never completes SPNEGO security context over HTTP #5235

michael-o · 2020-04-14T16:32:29Z

While working on few issues in brandond/requests-negotiate-sspi which I haven't reported yet, I wanted to reproduce them with curl too and noticed that I can't because there is a severe bug.

I expect curl to signal me similar:

pywintypes.error: (-2146893048, 'InitializeSecurityContext', 'Das Token, das der Funktion übergeben wurde, ist ungültig.')

Which is bascially SEC_E_INVALID_TOKEN. I don't know the reason for the error yet, but this is not important for now.

Next steps: Downloaded release tarball 7.69.1 and compiled on Windows 10 with MinGW.org GCC Build-20200227-1) 9.2.0 by passing: mingw32-make mingw32-winssl-sspi.
FWIW: The backend is running Java 8 with JGSS.

Now run the curl command:

src\curl -s --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUCoIIS...fAO4Rgg
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYHzMIHwoAMKA..oGTI44w+aL6
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:21:30 GMT
<

Why doesn't this fail? So I printed right after

curl/lib/vauth/spnego_sspi.c

Lines 242 to 252 in a6f7b2f

    
           nego->status = s_pSecFn->InitializeSecurityContext(nego->credentials, 
        
                                                              chlg ? nego->context : 
        
                                                                     NULL, 
        
                                                              nego->spn, 
        
                                                              ISC_REQ_CONFIDENTIALITY, 
        
                                                              0, SECURITY_NATIVE_DREP, 
        
                                                              chlg ? &chlg_desc : NULL, 
        
                                                              0, nego->context, 
        
                                                              &resp_desc, &attrs, 
        
                                                              &expiry);

the context status, and see "Context status: 0x90312" (SEC_I_CONTINUE_NEEDED), but I never see the continuation.

The bug is right here:

curl/lib/http.c

Lines 4061 to 4064 in a6f7b2f

    
           else if((checkprefix("WWW-Authenticate:", k->p) && 
        
                    (401 == k->httpcode)) || 
        
                   (checkprefix("Proxy-authenticate:", k->p) && 
        
                    (407 == k->httpcode))) {

libcurl assumes that *-Authenticate can only occur on 401/407. After removing the status code check, I see:

*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x90312
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUC...r/8C+h9Y3Fht1
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x80090308
* InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - Das Token, das der Funktion übergeben wurde, ist ungültig.
< WWW-Authenticate: Negotiate oYHzMIHwo...fKT+CMJ
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:26:52 GMT
<
{ [8367 bytes data]
* schannel: failed to decrypt data, need more data
{ [9000 bytes data]
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
{ [8775 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

This one matches better:

    else if((checkprefix("WWW-Authenticate:", k->p) &&
             (conn->http_negotiate_state == GSS_AUTHSENT)) ||
            (checkprefix("Proxy-authenticate:", k->p) &&
             (conn->proxy_negotiate_state == GSS_AUTHSENT))) {

Unfortuntely, this does not work on Un*x, there must be some other quirk in the code as well. Doing header prefix only, I get from MIT Kerberos:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null   -sq -k
Context status: 0x1
Context status: 0x0

But still authentication errors do not bubble up the chain so I never know that the authentication has failed for a severe reason.

This looks like a severe bug to me. What now?

PS: I am quite certain that the completion does not happen with MIT Kerberos too.

The text was updated successfully, but these errors were encountered:

jay · 2020-04-14T19:15:58Z

I'm not sure if that's expected behavior or not. @captain-caveman2k

michael-o · 2020-04-14T19:16:38Z

If that is expected behavior, then GSS-API based auth is pointless with curl.

jay · 2020-04-14T20:13:24Z

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

michael-o · 2020-04-14T20:23:03Z

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

Taking the status code into account doesn't make it secure. The status comes from application code while the response token from server code. The response token can basically appear in any status code >= 200 && <= 599.

bagder · 2020-04-14T20:51:12Z

200 means OK and thus no authentication was needed to get the content.

michael-o · 2020-04-14T21:07:39Z

200 means OK and thus no authentication was needed to get the content.

That's not correct (in this case). curl performs preemptive authentication. It never sees the 401 in this case. I can give you any status you want from application code, it won't change the auth loop. Since the client initiated it (preemptively) the server will complete. This is what you see.

See here:

PS C:\Entwicklung\Projekte\curl-7.69.1> src\curl -sq  https://deblndw024v.ad001.siemens.net:8444/manager/html  -o toll -
-verbose
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.69.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:03:47 GMT
<
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

Same URL, no preemptive authentication. Let's do --anyauth:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --anyauth -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null  --verbose  -sq -k
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
...
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<
* Ignoring the response-body
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact
* Issue another request to this URL: 'https://deblndw024v.ad001.siemens.net:8444/manager/html'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
* Found bundle for host deblndw024v.ad001.siemens.net: 0x800e78360 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host deblndw024v.ad001.siemens.net
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x1
* Server auth using Negotiate with user ''
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIIRf...HE=
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x0
< WWW-Authenticate: Negotiate oYGFM...4DDw==
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<

dsrink · 2020-05-29T15:27:35Z

I see a similar problem. Not sure if it is the same.

calling: curl -svL --negotiate -u : http://foo/
response on first request(s) will be one or more 301/302
after following those redirects, the chain reaches the actual to-be-authorized URL
when the Authorization header is sent, greatly differs

curl version	with --trust-location	without --trust-location
7.70.0	in every request	(Y) only on the first request
7.29.0	(X) only after receiving a 401	(Y) never

Further notes:

7.29.0 might seem fairly old, but I had it at hand, it is the system default on RHEL7.7.
I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.
(X): this is the behaviour which I would expect, sending Authorization header only after a 401 (or at least only after receiving a WWW-Authenticate)
(Y): obviously, this approach will not succeed to retrieve the requested data, but that is not my point.

michael-o · 2020-05-29T18:20:29Z

I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.

This has been introduced a couple of months ago, I expressed my negative opinion about. RFC 72xx does not forbid preemptive authentication, but I see no point doing to if you hve Expectation header and early responses.

bagder · 2020-05-29T21:21:37Z

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

michael-o · 2020-05-29T21:24:55Z

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

We aren't talking about Basic here. It is solely SPNEGO. I can't remember the ticket by id, but there is one.

stale · 2020-12-25T14:27:11Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

michael-o · 2020-12-25T14:46:33Z

Still valid.

Closes #5235

jay added the authentication label Apr 14, 2020

bagder added the help wanted label May 29, 2020

stale bot added the stale label Dec 25, 2020

stale bot removed the stale label Dec 25, 2020

bagder added the KNOWN_BUGS material label Dec 25, 2020

bagder added a commit that referenced this issue Dec 25, 2020

KNOWN_BUGS: 6.10 curl never completes Negotiate over HTTP

0d5a5e4

Closes #5235

bagder mentioned this issue Dec 25, 2020

KNOWN_BUGS: 6.10 curl never completes Negotiate over HTTP #6370

Closed

bagder closed this as completed in ec424f3 Dec 26, 2020

michael-o mentioned this issue Feb 4, 2022

Unclear/incorrect understanding of mutual auth and context continuation pythongssapi/requests-gssapi#39

Open

Curl never completes SPNEGO security context over HTTP #5235

Curl never completes SPNEGO security context over HTTP #5235

michael-o commented Apr 14, 2020 •

edited

jay commented Apr 14, 2020

michael-o commented Apr 14, 2020

jay commented Apr 14, 2020

michael-o commented Apr 14, 2020

bagder commented Apr 14, 2020

michael-o commented Apr 14, 2020

dsrink commented May 29, 2020

michael-o commented May 29, 2020

bagder commented May 29, 2020

michael-o commented May 29, 2020

stale bot commented Dec 25, 2020

michael-o commented Dec 25, 2020

Curl never completes SPNEGO security context over HTTP #5235

Curl never completes SPNEGO security context over HTTP #5235

Comments

michael-o commented Apr 14, 2020 • edited

jay commented Apr 14, 2020

michael-o commented Apr 14, 2020

jay commented Apr 14, 2020

michael-o commented Apr 14, 2020

bagder commented Apr 14, 2020

michael-o commented Apr 14, 2020

dsrink commented May 29, 2020

michael-o commented May 29, 2020

bagder commented May 29, 2020

michael-o commented May 29, 2020

stale bot commented Dec 25, 2020

michael-o commented Dec 25, 2020

michael-o commented Apr 14, 2020 •

edited