Can't connect to Fiddler proxy if using WinSSL backend

[jay]

zenden2k reported curl with WinSSL backend can't connect to Fiddler proxy if HTTPS intercept is enabled.

>curl.exe -v -x 127.0.0.1:8888 https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.40.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection Established
< FiddlerGateway: Direct
< StartTime: 22:42:05.973
< Connection: close
<
* Proxy replied OK to CONNECT request
* schannel: SSL/TLS connection with google.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 179 bytes...
* schannel: sent initial handshake data: sent 179 bytes
* schannel: SSL/TLS connection with google.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 1173 length 4096
* schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) -
The revocation function was unable to check revocation for the certificate.
* Closing connection 0
* schannel: shutting down SSL/TLS connection with google.com port 443
* schannel: clear security context handle
* schannel: clear credential handle
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x800
92012) - The revocation function was unable to check revocation for the certificate.

schannel is returning CRYPT_E_NO_REVOCATION_CHECK likely because we pass SCH_CRED_REVOCATION_CHECK_CHAIN in schannel_connect_step1 and in that case schannel requires all root CAs to have some sort of revocation detail when that flag is passed. Fiddler doesn't, so it fails? @ericlaw1979

Assuming that's the case it should be sufficient to advise in the FAQ or wiki or somewhere that if curl with WinSSL backend is used to connect to a Fiddler proxy with HTTPS intercept enabled that the -k option should be passed to curl so it does not attempt to verify Fiddler's certificate. A caution could be added to only do that if the Fiddler proxy is on localhost and only if HTTPS intercept is enabled.

Another way to go would be offer a way to disable revocation checking for WinSSL and advise Fiddler users pass that switch.

Thoughts?

[ericlaw1979]

FWIW, typically "SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT" is used by most things in Windows, as revocation checking the root itself isn't very useful (roots get revoked by WindowsUpdates, not by CRL/OCSP).

Fiddler's CertMaker Addon has a preference (set fiddler.certmaker.bc.AddCRL to true) that allows you to point it at a CRL (the fiddler.certmaker.bc.CRLURL preference points to the garbage "http://www.fiddler2.com/revocationlist.crl" by default), but note that this will only add a CRL reference to the EE certificates, not the root.

[jay]

If I use one of the other mutually exclusive flags like SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT I get the same result. However if I set fiddler.certmaker.bc.AddCRL to True I can get it working with SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT | SCH_CRED_IGNORE_REVOCATION_OFFLINE since the revocation server doesn't exist. So it looks as though a revocation check is always required by schannel when one of the _REVOCATION_CHECK flags is passed even if there's no resource specified for a revocation check.

[mback2k]

I would advise to keep the default as is, meaning SCH_CRED_REVOCATION_CHECK_CHAIN. Maybe we need an option to enable the use of SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT for specific use cases like this.

[jay]

Marc unfortunately using those options is not going to solve this problem. If HTTPS intercept is enabled then by default there's no revocation resource, and even if the revocation resource is enabled it's fake and only for the server certificate. As Eric noted the resource location can be changed, but I don't think most users will go to the trouble of setting up fake replies (assuming that's possible here).

Assume that in the best case revocation checking will fail unless we ignore offline servers which is kind of defeating the point if you ask me. I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

[ericlaw1979]

A couple things--

To Marc's point: As far as I know, Windows itself uses SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT in most places, as roots are customarily revoked via other means, and checking the root entails a performance cost that just doesn't make sense.

Now, to Jay's point "ignore offline servers which is kind of defeating the point if you ask me"-- that's partially true, and what led Chrome to remove revocation checks from Chrome. In IE, revocation offline failures are silent (http://blogs.msdn.com/b/ieinternals/archive/2011/04/07/enabling-certificate-revocation-check-failure-warnings-in-internet-explorer.aspx)

I argue that there are scenarios where revocation checks still have value (e.g. misissuance, as in the case of a phishing attack) even when silent in the case of CRL offline (which an active MITM could obviously simulate).

On Fiddler's side of things, there's the option to specify a revocation URL (and I can have this add that URL to the root as well as the EE). I will have a look at having this generate a fake CRL as well, but that may indeed prove more complex than is worthwhile for this case.

I will note that Fiddler has been used by millions of Windows users for over a decade and this is the first time I'm aware of where a CRL-offline check has been a blocking issue. So, kudos for that I guess. :-)

[jay]

I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

Ping... any watchers have an opinion about this? I could go either way.

[ericlaw1979]

@jay When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

[mback2k]

I think the only valid way to go is to set SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT by default or using an option, and add options for SCH_CRED_IGNORE_NO_REVOCATION_CHECK and SCH_CRED_IGNORE_REVOCATION_OFFLINE. Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

[jay]

When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

@ericlaw1979 help me out here because I thought that's what we'd want to do. Basically we'd stop passing SCH_CRED_REVOCATION_CHECK_CHAIN. I was taking a cue from how we disable revocation when peer verification is disabled. When I said entirely I meant no check for any certificate in the chain not entirely as in system-wide.

Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

@mback2k Agree. I will get in before the feature window closes. I'll change the default to SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT and add an option to disable revocation checking.

[ericlaw1979]

While having an option to disable revocation checking entirely may be useful (for performance reasons), disabling revocation checking entirely would not match the behavior of the majority of components on a Windows system. The majority of components on a Windows system either use WinINET or respect its behavior here-- namely, they perform a revocation check but they ignore offline revocation servers or missing revocation URLs; it becomes a "best-effort" type of thing whereby if a responder is available it is checked and if it returns "revoked" then the connection is blocked.

Some people have called this a "seatbelt that breaks in a wreck" but that's a bit misleading-- "best effort" revocation checks are useful for certain scenarios (e.g. phishing sites like "BankOfTheVVest.com") even while it's true that such an effort would fail in the presence of an active MITM.

[jay]

I've added an option to disable revocation checking for WinSSL in 172b2be.

New tool option --ssl-no-revoke.
New value CURLSSLOPT_NO_REVOKE for CURLOPT_SSL_OPTIONS.

The commit may be seen as a departure from some of the comments I made earlier. Specifically I've eschewed some talked about changes. Now:

• Not exposing the more precise revocation options (revocation unavailable; revocation offline) to the user. We should strive for options that can be implemented for multiple backends if possible. --ssl-no-revoke is simple enough, either the user leaves WinSSL revocation checks on by default or turns them off with that switch at the command line or in their curlrc.
• Not changing the default behavior to exclude the root certificate from the revocation check. I'm loathe to change this default behavior, even if there is not much of a gain in safety from the extra check. I'm concerned that I don't really have a good measurement of "not much". There may be root certificates that Microsoft takes no part in not being removed or blacklisted when they are revoked, and we'd be responsible for disabling that expected revocation check. I think it is what it is at this point whether it's helping or not.

If anyone wants to revisit either of those my suggestion would be exposing a callback, preferably if possible like the schannel equivalent of a CTX function the way we do for OpenSSL so a libcurl user would have some way to make more detailed changes on their own.

@zenden2k thanks for your report.

@ericlaw1979 @mback2k thanks for your help and feedback.