Curl never completes SPNEGO security context over HTTP

[michael-o]

While working on few issues in brandond/requests-negotiate-sspi which I haven't reported yet, I wanted to reproduce them with curl too and noticed that I can't because there is a severe bug.

I expect curl to signal me similar:

pywintypes.error: (-2146893048, 'InitializeSecurityContext', 'Das Token, das der Funktion übergeben wurde, ist ungültig.')

Which is bascially SEC_E_INVALID_TOKEN. I don't know the reason for the error yet, but this is not important for now.

Next steps: Downloaded release tarball 7.69.1 and compiled on Windows 10 with MinGW.org GCC Build-20200227-1) 9.2.0 by passing: mingw32-make mingw32-winssl-sspi.
FWIW: The backend is running Java 8 with JGSS.

Now run the curl command:

src\curl -s --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUCoIIS...fAO4Rgg
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYHzMIHwoAMKA..oGTI44w+aL6
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:21:30 GMT
<

Why doesn't this fail? So I printed right after

curl/lib/vauth/spnego_sspi.c

Lines 242 to 252 in a6f7b2f

	nego->status = s_pSecFn->InitializeSecurityContext(nego->credentials,
	chlg ? nego->context :
	NULL,
	nego->spn,
	ISC_REQ_CONFIDENTIALITY,
	0, SECURITY_NATIVE_DREP,
	chlg ? &chlg_desc : NULL,
	0, nego->context,
	&resp_desc, &attrs,
	&expiry);

the context status, and see "Context status: 0x90312" (SEC_I_CONTINUE_NEEDED), but I never see the continuation.

The bug is right here:

curl/lib/http.c

Lines 4061 to 4064 in a6f7b2f

	else if((checkprefix("WWW-Authenticate:", k->p) &&
	(401 == k->httpcode)) ||
	(checkprefix("Proxy-authenticate:", k->p) &&
	(407 == k->httpcode))) {

libcurl assumes that *-Authenticate can only occur on 401/407. After removing the status code check, I see:

*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x90312
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUC...r/8C+h9Y3Fht1
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x80090308
* InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - Das Token, das der Funktion übergeben wurde, ist ungültig.
< WWW-Authenticate: Negotiate oYHzMIHwo...fKT+CMJ
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:26:52 GMT
<
{ [8367 bytes data]
* schannel: failed to decrypt data, need more data
{ [9000 bytes data]
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
{ [8775 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

This one matches better:

    else if((checkprefix("WWW-Authenticate:", k->p) &&
             (conn->http_negotiate_state == GSS_AUTHSENT)) ||
            (checkprefix("Proxy-authenticate:", k->p) &&
             (conn->proxy_negotiate_state == GSS_AUTHSENT))) {

Unfortuntely, this does not work on Un*x, there must be some other quirk in the code as well. Doing header prefix only, I get from MIT Kerberos:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null   -sq -k
Context status: 0x1
Context status: 0x0

But still authentication errors do not bubble up the chain so I never know that the authentication has failed for a severe reason.

This looks like a severe bug to me. What now?

PS: I am quite certain that the completion does not happen with MIT Kerberos too.

[jay]

I'm not sure if that's expected behavior or not. @captain-caveman2k

[michael-o]

If that is expected behavior, then GSS-API based auth is pointless with curl.

[jay]

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

[michael-o]

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

Taking the status code into account doesn't make it secure. The status comes from application code while the response token from server code. The response token can basically appear in any status code >= 200 && <= 599.

[bagder]

200 means OK and thus no authentication was needed to get the content.

[michael-o]

200 means OK and thus no authentication was needed to get the content.

That's not correct (in this case). curl performs preemptive authentication. It never sees the 401 in this case. I can give you any status you want from application code, it won't change the auth loop. Since the client initiated it (preemptively) the server will complete. This is what you see.

See here:

PS C:\Entwicklung\Projekte\curl-7.69.1> src\curl -sq  https://deblndw024v.ad001.siemens.net:8444/manager/html  -o toll -
-verbose
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.69.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:03:47 GMT
<
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

Same URL, no preemptive authentication. Let's do --anyauth:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --anyauth -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null  --verbose  -sq -k
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
...
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<
* Ignoring the response-body
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact
* Issue another request to this URL: 'https://deblndw024v.ad001.siemens.net:8444/manager/html'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
* Found bundle for host deblndw024v.ad001.siemens.net: 0x800e78360 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host deblndw024v.ad001.siemens.net
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x1
* Server auth using Negotiate with user ''
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIIRf...HE=
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x0
< WWW-Authenticate: Negotiate oYGFM...4DDw==
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<

[dsrink]

I see a similar problem. Not sure if it is the same.

• calling: curl -svL --negotiate -u : http://foo/
• response on first request(s) will be one or more 301/302
• after following those redirects, the chain reaches the actual to-be-authorized URL
• when the Authorization header is sent, greatly differs

curl version	with --trust-location	without --trust-location
7.70.0	in every request	(Y) only on the first request
7.29.0	(X) only after receiving a 401	(Y) never

Further notes:

• 7.29.0 might seem fairly old, but I had it at hand, it is the system default on RHEL7.7.
• I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.
• (X): this is the behaviour which I would expect, sending Authorization header only after a 401 (or at least only after receiving a WWW-Authenticate)
• (Y): obviously, this approach will not succeed to retrieve the requested data, but that is not my point.

[michael-o]

I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.

This has been introduced a couple of months ago, I expressed my negative opinion about. RFC 72xx does not forbid preemptive authentication, but I see no point doing to if you hve Expectation header and early responses.

[bagder]

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

[michael-o]

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

We aren't talking about Basic here. It is solely SPNEGO. I can't remember the ticket by id, but there is one.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michael-o]

Still valid.