Skip to content

Curl never completes SPNEGO security context over HTTP #5235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michael-o opened this issue Apr 14, 2020 · 12 comments
Closed

Curl never completes SPNEGO security context over HTTP #5235

michael-o opened this issue Apr 14, 2020 · 12 comments

Comments

@michael-o
Copy link
Contributor

michael-o commented Apr 14, 2020

While working on few issues in brandond/requests-negotiate-sspi which I haven't reported yet, I wanted to reproduce them with curl too and noticed that I can't because there is a severe bug.

I expect curl to signal me similar:

pywintypes.error: (-2146893048, 'InitializeSecurityContext', 'Das Token, das der Funktion übergeben wurde, ist ungültig.')

Which is bascially SEC_E_INVALID_TOKEN. I don't know the reason for the error yet, but this is not important for now.

Next steps: Downloaded release tarball 7.69.1 and compiled on Windows 10 with MinGW.org GCC Build-20200227-1) 9.2.0 by passing: mingw32-make mingw32-winssl-sspi.
FWIW: The backend is running Java 8 with JGSS.

Now run the curl command:

src\curl -s --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUCoIIS...fAO4Rgg
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYHzMIHwoAMKA..oGTI44w+aL6
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:21:30 GMT
<

Why doesn't this fail? So I printed right after

nego->status = s_pSecFn->InitializeSecurityContext(nego->credentials,
chlg ? nego->context :
NULL,
nego->spn,
ISC_REQ_CONFIDENTIALITY,
0, SECURITY_NATIVE_DREP,
chlg ? &chlg_desc : NULL,
0, nego->context,
&resp_desc, &attrs,
&expiry);
the context status, and see "Context status: 0x90312" (SEC_I_CONTINUE_NEEDED), but I never see the continuation.

The bug is right here:

curl/lib/http.c

Lines 4061 to 4064 in a6f7b2f

else if((checkprefix("WWW-Authenticate:", k->p) &&
(401 == k->httpcode)) ||
(checkprefix("Proxy-authenticate:", k->p) &&
(407 == k->httpcode))) {

libcurl assumes that *-Authenticate can only occur on 401/407. After removing the status code check, I see:

*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x90312
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIITBwYGKwYBBQUC...r/8C+h9Y3Fht1
> User-Agent: curl/7.69.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x80090308
* InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - Das Token, das der Funktion übergeben wurde, ist ungültig.
< WWW-Authenticate: Negotiate oYHzMIHwo...fKT+CMJ
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 16:26:52 GMT
<
{ [8367 bytes data]
* schannel: failed to decrypt data, need more data
{ [9000 bytes data]
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
{ [8775 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

This one matches better:

    else if((checkprefix("WWW-Authenticate:", k->p) &&
             (conn->http_negotiate_state == GSS_AUTHSENT)) ||
            (checkprefix("Proxy-authenticate:", k->p) &&
             (conn->proxy_negotiate_state == GSS_AUTHSENT))) {

Unfortuntely, this does not work on Un*x, there must be some other quirk in the code as well. Doing header prefix only, I get from MIT Kerberos:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --negotiate -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null   -sq -k
Context status: 0x1
Context status: 0x0

But still authentication errors do not bubble up the chain so I never know that the authentication has failed for a severe reason.

This looks like a severe bug to me. What now?

PS: I am quite certain that the completion does not happen with MIT Kerberos too.

@jay
Copy link
Member

jay commented Apr 14, 2020

I'm not sure if that's expected behavior or not. @captain-caveman2k

@michael-o
Copy link
Contributor Author

If that is expected behavior, then GSS-API based auth is pointless with curl.

@jay
Copy link
Member

jay commented Apr 14, 2020

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

@michael-o
Copy link
Contributor Author

To be clear I mean that it's returning 200 succeeded and an authenticate header. Which does not make it pointless it just doesn't work for that case.

Taking the status code into account doesn't make it secure. The status comes from application code while the response token from server code. The response token can basically appear in any status code >= 200 && <= 599.

@bagder
Copy link
Member

bagder commented Apr 14, 2020

200 means OK and thus no authentication was needed to get the content.

@michael-o
Copy link
Contributor Author

200 means OK and thus no authentication was needed to get the content.

That's not correct (in this case). curl performs preemptive authentication. It never sees the 401 in this case. I can give you any status you want from application code, it won't change the auth loop. Since the client initiated it (preemptively) the server will complete. This is what you see.

See here:

PS C:\Entwicklung\Projekte\curl-7.69.1> src\curl -sq  https://deblndw024v.ad001.siemens.net:8444/manager/html  -o toll -
-verbose
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.69.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:03:47 GMT
<
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact

Same URL, no preemptive authentication. Let's do --anyauth:

$ LD_LIBRARY_PATH=/tmp/curl/lib /tmp/curl/bin/curl --anyauth -u : https://deblndw024v.ad001.siemens.net:8444/manager/html -o /dev/null  --verbose  -sq -k
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
*   Trying 147.54.65.24:8444...
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
...
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 669
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<
* Ignoring the response-body
{ [669 bytes data]
* Connection #0 to host deblndw024v.ad001.siemens.net left intact
* Issue another request to this URL: 'https://deblndw024v.ad001.siemens.net:8444/manager/html'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
* Found bundle for host deblndw024v.ad001.siemens.net: 0x800e78360 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host deblndw024v.ad001.siemens.net
* Connected to deblndw024v.ad001.siemens.net (147.54.65.24) port 8444 (#0)
Context status: 0x1
* Server auth using Negotiate with user ''
} [5 bytes data]
> GET /manager/html HTTP/1.1
> Host: deblndw024v.ad001.siemens.net:8444
> Authorization: Negotiate YIIRf...HE=
> User-Agent: curl/7.70.0-DEV
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Context status: 0x0
< WWW-Authenticate: Negotiate oYGFM...4DDw==
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Date: Tue, 14 Apr 2020 21:06:10 GMT
<

@dsrink
Copy link

dsrink commented May 29, 2020

I see a similar problem. Not sure if it is the same.

  • calling: curl -svL --negotiate -u : http://foo/
  • response on first request(s) will be one or more 301/302
  • after following those redirects, the chain reaches the actual to-be-authorized URL
  • when the Authorization header is sent, greatly differs
curl version with --trust-location without --trust-location
7.70.0 in every request (Y) only on the first request
7.29.0 (X) only after receiving a 401 (Y) never

Further notes:

  • 7.29.0 might seem fairly old, but I had it at hand, it is the system default on RHEL7.7.
  • I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.
  • (X): this is the behaviour which I would expect, sending Authorization header only after a 401 (or at least only after receiving a WWW-Authenticate)
  • (Y): obviously, this approach will not succeed to retrieve the requested data, but that is not my point.

@michael-o
Copy link
Contributor Author

I am surprised that 7.70.0 sends the Authorization header unsolicited even on the first request. I consider this to be a bug.

This has been introduced a couple of months ago, I expressed my negative opinion about. RFC 72xx does not forbid preemptive authentication, but I see no point doing to if you hve Expectation header and early responses.

@bagder
Copy link
Member

bagder commented May 29, 2020

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

@michael-o
Copy link
Contributor Author

This has been introduced a couple of months ago

curl has done unsolicited Authorization headers on first request since we first introduced support for Basic auth some two decades ago or so. That's not new. This particular bug might be new though, I don't know.

We aren't talking about Basic here. It is solely SPNEGO. I can't remember the ticket by id, but there is one.

@stale
Copy link

stale bot commented Dec 25, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Dec 25, 2020
@michael-o
Copy link
Contributor Author

Still valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

4 participants