New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't connect to Fiddler proxy if using WinSSL backend #264

Closed
jay opened this Issue May 13, 2015 · 11 comments

Comments

Projects
None yet
3 participants
@jay
Member

jay commented May 13, 2015

zenden2k reported curl with WinSSL backend can't connect to Fiddler proxy if HTTPS intercept is enabled.

>curl.exe -v -x 127.0.0.1:8888 https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.40.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection Established
< FiddlerGateway: Direct
< StartTime: 22:42:05.973
< Connection: close
<
* Proxy replied OK to CONNECT request
* schannel: SSL/TLS connection with google.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 179 bytes...
* schannel: sent initial handshake data: sent 179 bytes
* schannel: SSL/TLS connection with google.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 1173 length 4096
* schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) -
The revocation function was unable to check revocation for the certificate.
* Closing connection 0
* schannel: shutting down SSL/TLS connection with google.com port 443
* schannel: clear security context handle
* schannel: clear credential handle
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x800
92012) - The revocation function was unable to check revocation for the certificate.

schannel is returning CRYPT_E_NO_REVOCATION_CHECK likely because we pass SCH_CRED_REVOCATION_CHECK_CHAIN in schannel_connect_step1 and in that case schannel requires all root CAs to have some sort of revocation detail when that flag is passed. Fiddler doesn't, so it fails? @ericlaw1979

Assuming that's the case it should be sufficient to advise in the FAQ or wiki or somewhere that if curl with WinSSL backend is used to connect to a Fiddler proxy with HTTPS intercept enabled that the -k option should be passed to curl so it does not attempt to verify Fiddler's certificate. A caution could be added to only do that if the Fiddler proxy is on localhost and only if HTTPS intercept is enabled.

Another way to go would be offer a way to disable revocation checking for WinSSL and advise Fiddler users pass that switch.

Thoughts?

@ericlaw1979

This comment has been minimized.

Show comment
Hide comment
@ericlaw1979

ericlaw1979 May 13, 2015

FWIW, typically "SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT" is used by most things in Windows, as revocation checking the root itself isn't very useful (roots get revoked by WindowsUpdates, not by CRL/OCSP).

Fiddler's CertMaker Addon has a preference (set fiddler.certmaker.bc.AddCRL to true) that allows you to point it at a CRL (the fiddler.certmaker.bc.CRLURL preference points to the garbage "http://www.fiddler2.com/revocationlist.crl" by default), but note that this will only add a CRL reference to the EE certificates, not the root.

ericlaw1979 commented May 13, 2015

FWIW, typically "SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT" is used by most things in Windows, as revocation checking the root itself isn't very useful (roots get revoked by WindowsUpdates, not by CRL/OCSP).

Fiddler's CertMaker Addon has a preference (set fiddler.certmaker.bc.AddCRL to true) that allows you to point it at a CRL (the fiddler.certmaker.bc.CRLURL preference points to the garbage "http://www.fiddler2.com/revocationlist.crl" by default), but note that this will only add a CRL reference to the EE certificates, not the root.

@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay May 13, 2015

Member

If I use one of the other mutually exclusive flags like SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT I get the same result. However if I set fiddler.certmaker.bc.AddCRL to True I can get it working with SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT | SCH_CRED_IGNORE_REVOCATION_OFFLINE since the revocation server doesn't exist. So it looks as though a revocation check is always required by schannel when one of the _REVOCATION_CHECK flags is passed even if there's no resource specified for a revocation check.

Member

jay commented May 13, 2015

If I use one of the other mutually exclusive flags like SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT I get the same result. However if I set fiddler.certmaker.bc.AddCRL to True I can get it working with SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT | SCH_CRED_IGNORE_REVOCATION_OFFLINE since the revocation server doesn't exist. So it looks as though a revocation check is always required by schannel when one of the _REVOCATION_CHECK flags is passed even if there's no resource specified for a revocation check.

@mback2k

This comment has been minimized.

Show comment
Hide comment
@mback2k

mback2k May 19, 2015

Member

I would advise to keep the default as is, meaning SCH_CRED_REVOCATION_CHECK_CHAIN. Maybe we need an option to enable the use of SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT for specific use cases like this.

Member

mback2k commented May 19, 2015

I would advise to keep the default as is, meaning SCH_CRED_REVOCATION_CHECK_CHAIN. Maybe we need an option to enable the use of SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT or SCH_CRED_REVOCATION_CHECK_END_CERT for specific use cases like this.

@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay May 19, 2015

Member

Marc unfortunately using those options is not going to solve this problem. If HTTPS intercept is enabled then by default there's no revocation resource, and even if the revocation resource is enabled it's fake and only for the server certificate. As Eric noted the resource location can be changed, but I don't think most users will go to the trouble of setting up fake replies (assuming that's possible here).

Assume that in the best case revocation checking will fail unless we ignore offline servers which is kind of defeating the point if you ask me. I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

Member

jay commented May 19, 2015

Marc unfortunately using those options is not going to solve this problem. If HTTPS intercept is enabled then by default there's no revocation resource, and even if the revocation resource is enabled it's fake and only for the server certificate. As Eric noted the resource location can be changed, but I don't think most users will go to the trouble of setting up fake replies (assuming that's possible here).

Assume that in the best case revocation checking will fail unless we ignore offline servers which is kind of defeating the point if you ask me. I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

@ericlaw1979

This comment has been minimized.

Show comment
Hide comment
@ericlaw1979

ericlaw1979 May 19, 2015

A couple things--

To Marc's point: As far as I know, Windows itself uses SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT in most places, as roots are customarily revoked via other means, and checking the root entails a performance cost that just doesn't make sense.

Now, to Jay's point "ignore offline servers which is kind of defeating the point if you ask me"-- that's partially true, and what led Chrome to remove revocation checks from Chrome. In IE, revocation offline failures are silent (http://blogs.msdn.com/b/ieinternals/archive/2011/04/07/enabling-certificate-revocation-check-failure-warnings-in-internet-explorer.aspx)

I argue that there are scenarios where revocation checks still have value (e.g. misissuance, as in the case of a phishing attack) even when silent in the case of CRL offline (which an active MITM could obviously simulate).

On Fiddler's side of things, there's the option to specify a revocation URL (and I can have this add that URL to the root as well as the EE). I will have a look at having this generate a fake CRL as well, but that may indeed prove more complex than is worthwhile for this case.

I will note that Fiddler has been used by millions of Windows users for over a decade and this is the first time I'm aware of where a CRL-offline check has been a blocking issue. So, kudos for that I guess. :-)

ericlaw1979 commented May 19, 2015

A couple things--

To Marc's point: As far as I know, Windows itself uses SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT in most places, as roots are customarily revoked via other means, and checking the root entails a performance cost that just doesn't make sense.

Now, to Jay's point "ignore offline servers which is kind of defeating the point if you ask me"-- that's partially true, and what led Chrome to remove revocation checks from Chrome. In IE, revocation offline failures are silent (http://blogs.msdn.com/b/ieinternals/archive/2011/04/07/enabling-certificate-revocation-check-failure-warnings-in-internet-explorer.aspx)

I argue that there are scenarios where revocation checks still have value (e.g. misissuance, as in the case of a phishing attack) even when silent in the case of CRL offline (which an active MITM could obviously simulate).

On Fiddler's side of things, there's the option to specify a revocation URL (and I can have this add that URL to the root as well as the EE). I will have a look at having this generate a fake CRL as well, but that may indeed prove more complex than is worthwhile for this case.

I will note that Fiddler has been used by millions of Windows users for over a decade and this is the first time I'm aware of where a CRL-offline check has been a blocking issue. So, kudos for that I guess. :-)

@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay Jun 18, 2015

Member

I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

Ping... any watchers have an opinion about this? I could go either way.

Member

jay commented Jun 18, 2015

I see two possibles, either we document the workaround (allow insecure connections) or add an option to disable revocation entirely. I could implement it for schannel and maybe others here could one day extend it to other SSL backends where revocation checking is enabled by default?

Ping... any watchers have an opinion about this? I could go either way.

@ericlaw1979

This comment has been minimized.

Show comment
Hide comment
@ericlaw1979

ericlaw1979 Jun 18, 2015

@jay When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

ericlaw1979 commented Jun 18, 2015

@jay When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

@mback2k

This comment has been minimized.

Show comment
Hide comment
@mback2k

mback2k Jun 18, 2015

Member

I think the only valid way to go is to set SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT by default or using an option, and add options for SCH_CRED_IGNORE_NO_REVOCATION_CHECK and SCH_CRED_IGNORE_REVOCATION_OFFLINE. Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

Member

mback2k commented Jun 18, 2015

I think the only valid way to go is to set SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT by default or using an option, and add options for SCH_CRED_IGNORE_NO_REVOCATION_CHECK and SCH_CRED_IGNORE_REVOCATION_OFFLINE. Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay Jun 20, 2015

Member

When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

@ericlaw1979 help me out here because I thought that's what we'd want to do. Basically we'd stop passing SCH_CRED_REVOCATION_CHECK_CHAIN. I was taking a cue from how we disable revocation when peer verification is disabled. When I said entirely I meant no check for any certificate in the chain not entirely as in system-wide.


Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

@mback2k Agree. I will get in before the feature window closes. I'll change the default to SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT and add an option to disable revocation checking.

Member

jay commented Jun 20, 2015

When you say "disable revocation entirely" do you mean "set SCH_CRED_IGNORE_NO_REVOCATION_CHECK | SCH_CRED_IGNORE_REVOCATION_OFFLINE like most Windows callers"?

That's different than disabling entirely...

@ericlaw1979 help me out here because I thought that's what we'd want to do. Basically we'd stop passing SCH_CRED_REVOCATION_CHECK_CHAIN. I was taking a cue from how we disable revocation when peer verification is disabled. When I said entirely I meant no check for any certificate in the chain not entirely as in system-wide.


Revocation checking should be enabled and required by default, just like it was before, but I can understand the arguments to disable revocation checking for root certificates (PCAs). I don't want to disable a feature that has been enabled since the first curl release with SChannel support. There may be software that already depends on this.

@mback2k Agree. I will get in before the feature window closes. I'll change the default to SCH_CRED_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT and add an option to disable revocation checking.

@ericlaw1979

This comment has been minimized.

Show comment
Hide comment
@ericlaw1979

ericlaw1979 Jun 22, 2015

While having an option to disable revocation checking entirely may be useful (for performance reasons), disabling revocation checking entirely would not match the behavior of the majority of components on a Windows system. The majority of components on a Windows system either use WinINET or respect its behavior here-- namely, they perform a revocation check but they ignore offline revocation servers or missing revocation URLs; it becomes a "best-effort" type of thing whereby if a responder is available it is checked and if it returns "revoked" then the connection is blocked.

Some people have called this a "seatbelt that breaks in a wreck" but that's a bit misleading-- "best effort" revocation checks are useful for certain scenarios (e.g. phishing sites like "BankOfTheVVest.com") even while it's true that such an effort would fail in the presence of an active MITM.

ericlaw1979 commented Jun 22, 2015

While having an option to disable revocation checking entirely may be useful (for performance reasons), disabling revocation checking entirely would not match the behavior of the majority of components on a Windows system. The majority of components on a Windows system either use WinINET or respect its behavior here-- namely, they perform a revocation check but they ignore offline revocation servers or missing revocation URLs; it becomes a "best-effort" type of thing whereby if a responder is available it is checked and if it returns "revoked" then the connection is blocked.

Some people have called this a "seatbelt that breaks in a wreck" but that's a bit misleading-- "best effort" revocation checks are useful for certain scenarios (e.g. phishing sites like "BankOfTheVVest.com") even while it's true that such an effort would fail in the presence of an active MITM.

jay added a commit that referenced this issue Jul 17, 2015

SSL: Add an option to disable certificate revocation checks
New tool option --ssl-no-revoke.
New value CURLSSLOPT_NO_REVOKE for CURLOPT_SSL_OPTIONS.

Currently this option applies only to WinSSL where we have automatic
certificate revocation checking by default. According to the
ssl-compared chart there are other backends that have automatic checking
(NSS, wolfSSL and DarwinSSL) so we could possibly accommodate them at
some later point.

Bug: #264
Reported-by: zenden2k <zenden2k@gmail.com>
@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay Jul 17, 2015

Member

I've added an option to disable revocation checking for WinSSL in 172b2be.

New tool option --ssl-no-revoke.
New value CURLSSLOPT_NO_REVOKE for CURLOPT_SSL_OPTIONS.

The commit may be seen as a departure from some of the comments I made earlier. Specifically I've eschewed some talked about changes. Now:

  • Not exposing the more precise revocation options (revocation unavailable; revocation offline) to the user. We should strive for options that can be implemented for multiple backends if possible. --ssl-no-revoke is simple enough, either the user leaves WinSSL revocation checks on by default or turns them off with that switch at the command line or in their curlrc.
  • Not changing the default behavior to exclude the root certificate from the revocation check. I'm loathe to change this default behavior, even if there is not much of a gain in safety from the extra check. I'm concerned that I don't really have a good measurement of "not much". There may be root certificates that Microsoft takes no part in not being removed or blacklisted when they are revoked, and we'd be responsible for disabling that expected revocation check. I think it is what it is at this point whether it's helping or not.

If anyone wants to revisit either of those my suggestion would be exposing a callback, preferably if possible like the schannel equivalent of a CTX function the way we do for OpenSSL so a libcurl user would have some way to make more detailed changes on their own.

@zenden2k thanks for your report.

@ericlaw1979 @mback2k thanks for your help and feedback.

Member

jay commented Jul 17, 2015

I've added an option to disable revocation checking for WinSSL in 172b2be.

New tool option --ssl-no-revoke.
New value CURLSSLOPT_NO_REVOKE for CURLOPT_SSL_OPTIONS.

The commit may be seen as a departure from some of the comments I made earlier. Specifically I've eschewed some talked about changes. Now:

  • Not exposing the more precise revocation options (revocation unavailable; revocation offline) to the user. We should strive for options that can be implemented for multiple backends if possible. --ssl-no-revoke is simple enough, either the user leaves WinSSL revocation checks on by default or turns them off with that switch at the command line or in their curlrc.
  • Not changing the default behavior to exclude the root certificate from the revocation check. I'm loathe to change this default behavior, even if there is not much of a gain in safety from the extra check. I'm concerned that I don't really have a good measurement of "not much". There may be root certificates that Microsoft takes no part in not being removed or blacklisted when they are revoked, and we'd be responsible for disabling that expected revocation check. I think it is what it is at this point whether it's helping or not.

If anyone wants to revisit either of those my suggestion would be exposing a callback, preferably if possible like the schannel equivalent of a CTX function the way we do for OpenSSL so a libcurl user would have some way to make more detailed changes on their own.

@zenden2k thanks for your report.

@ericlaw1979 @mback2k thanks for your help and feedback.

@jay jay closed this Jul 17, 2015

jgsogo added a commit to jgsogo/curl that referenced this issue Oct 19, 2015

SSL: Add an option to disable certificate revocation checks
New tool option --ssl-no-revoke.
New value CURLSSLOPT_NO_REVOKE for CURLOPT_SSL_OPTIONS.

Currently this option applies only to WinSSL where we have automatic
certificate revocation checking by default. According to the
ssl-compared chart there are other backends that have automatic checking
(NSS, wolfSSL and DarwinSSL) so we could possibly accommodate them at
some later point.

Bug: curl#264
Reported-by: zenden2k <zenden2k@gmail.com>

@lock lock bot locked as resolved and limited conversation to collaborators May 7, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.