Schannel: Failure when receiving data from the peer

[jeroen]

A user of the R bindings has reported a bug that happens in the schannel backend for this particular server:

curl "https://exoplanetarchive.ipac.caltech.edu/cgi-bin/nstedAPI/nph-nstedAPI?table=exoplanets&select=pl_hostname,ra,dec" > out.txt

Which does succceed in retrieving data, but in the end always errors like this:

{ [8000 bytes data]
* schannel: failed to decrypt data, need more data
{ [8000 bytes data]
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
{ [8000 bytes data]
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
{ [8000 bytes data]
* schannel: server closed abruptly (missing close_notify)
* Closing connection 0
* schannel: shutting down SSL/TLS connection with exoplanetarchive.ipac.caltech.edu port 443

This is with libcurl 7.64.1, haven't been able to test yet with 7.66. The problem does not appear when using the openssl backend.

[jay]

I tested using master. There has to be a known termination point otherwise an error will occur. (to be clear, close_notify is an acceptable termination point)

< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Content-type: text/plain
* no chunk, no close, no size. Assume close to signal end
* Marked for [closure]: HTTP: No end-of-message indicator

I walked through the code from the point of * schannel: server closed the connection and I don't see anything amiss. I can see in Wireshark an encrypted alert is sent, which could be close notify. Unfortunately I don't have a way to decrypt schannel traffic, but apparently it's possible. I also tried with OpenSSL (which I can decrypt) and I see in that case the encrypted alert is a close notify, so it's possible there's a bug somewhere in the schannel code. It's hard for me to say for sure.

[jay]

I also tried with OpenSSL (which I can decrypt) and I see in that case the encrypted alert is a close notify, so it's possible there's a bug somewhere in the schannel code. It's hard for me to say for sure.

It looks like I confused myself, I have to walk that back there's no bug that I can see. In Wireshark while running curl w/openssl the close notify I see is sent by the client (libcurl), and curl w/schannel the encrypted alert is sent by the client (libcurl). In either case no alert is received (definitively with OpenSSL no close_notify is received before FIN; in schannel no encrypted alert is received before FIN therefore no close_notify is received). Therefore there is no known termination point.

I have confirmed close_notify alerts work properly as a solo termination point when I test on other websites. I tested by doing this:

curld -v --ignore-content-length -H "Connection: Close" https://httpbin.org/get

By ignoring content length here and telling the server to close the connection on a server that both sends content length and honors client close request we can simulate * no chunk, no close, no size. Assume close to signal end and close_notify as the termination point. It works fine on the sites that return close_notify in that case (not all do).

I've since tried your URL in Windows 7 8 10 and the result is the same. I've walked through the code again and watched all calls to DecryptMessage and it never returns SEC_I_CONTEXT_EXPIRED (close_notify received).

Based on this I think it's a server issue, sorry for the confusion.

[jeroen]

Thanks for the debugging! Is there anything that can be done to make libcurl robust against such server behavior?

[jay]

libcurl is robust in handling missing HTTP response length. It will keep reading until TCP close if there is no found or allowed HTTP termination point. If the response is over SSL then libcurl must ensure the integrity which includes length. TCP close is not an acceptable indicator of length for the SSL layer because it may be manipulated by an attacker. As a last resort libcurl may treat the SSL layer termination point, aka close_notify alert, as an indicator of length.

However, as you've seen on the contrary, other backends used by libcurl may treat TCP close as an indicator of length when the HTTP response is over SSL and no other termination point was given. That is for legacy reasons (like your server is 15 years old or misconfigured), is not robust, not healthy and I'm wholly against it. I hope one day it will be possible to fix it (starting in openssl.c) without breaking anything for anyone.

Conceivably we could add a flag like CURLSSLOPT_ALLOW_TRUNCATION, CURLSSLOPT_ALLOW_ABRUPT_CLOSURE etc to allow the possibility of the attack similar to what we did for BEAST. At least with schannel I don't think it would see very much use since I can only recall few reports of this issue, despite Microsoft adding curl w/ schannel to Windows 10 at least a year? ago.

[bagder]

Should we then move this to become a well phrased section in the TODO document instead? I don't think having this issue lingering open here will help us much.

[jeroen]

Yes that is fine by me. This problem appears is very rarerly and I don't understand it well enough to suggest a solution myself.

[bigben386]

I just ran across this issue today. We use schannel because we use an ssl inspection proxy. It was easier to enable schannel so git would use the windows cert store. Changing back to openssl and specifying the ca file manually resolved our issue as well.

[gwd999]

same issue here as well ... when downloading pacman tools for R source builds on WINdows

[akanieski]

I've found this same scenario happening.. we noticed that it happens to correlate with this message about HTTP 1.0 that shows only in scenarios it fails.. in scenarios it succeeds it uses HTTP 1.1:

09:26:32.592218 http.c:689              == Info: HTTP 1.0, assume close after body

We've also found that it only shows when repositories either grow large enough (20mb+ perhaps) or that they contain binary files.. (usually this coincides of course with size).

So is it possible in some scenarios Libcurl negotiates down to HTTP 1.0 and subsequently the schannel: server closed abruptly (missing close_notify) shows up? Are the specific scenarios it pivots between HTTP 1.0 and 1.1?

[jay]

So is it possible in some scenarios Libcurl negotiates down to HTTP 1.0 and subsequently the schannel: server closed abruptly (missing close_notify) shows up? Are the specific scenarios it pivots between HTTP 1.0 and 1.1?

Not unless the app set CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_1_0. If you're only seeing it sometimes I think that's pretty unlikely.

We've also found that it only shows when repositories either grow large enough

You can use GIT_CURL_VERBOSE=1 git ... and git will turn on CURLOPT_VERBOSE so you can see what version is being sent/received for each transfer. Note git may do multiple transfers per command.