Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent 'Unable to connect' error when using HTTPS endpoint #671

Closed
crusader-mike opened this issue Sep 22, 2017 · 0 comments
Closed

Intermittent 'Unable to connect' error when using HTTPS endpoint #671

crusader-mike opened this issue Sep 22, 2017 · 0 comments

Comments

@crusader-mike
Copy link

I was chasing this one for 3 days. Found the cause and workaround, leaving some info here for those who run into this problem. Or maybe you will find a way to incorporate workaround into AWS SDK code.

Symptoms
Relatively rare Unable to connect to endpoint (code 99) failure with associated Send request failed: A security error occurred in the logs. Also, accompanied SEC_E_MESSAGE_ALTERED and/or SEC_E_BUFFER_TOO_SMALL(0x80090321) errors in captured WinHTTP logs. Verified to happen on Windows 7 and Windows Server 2012 R2 with StorageGrid. CleverSafe seems to be immune (it doesn't support cipher suites mentioned below), not sure about Amazon S3,

Cause
Can happen if during TLS handshake server chooses TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 or TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 cipher suites (which had a problem when they were initially introduced). This problem was supposed to be fixed by KB3018238 (which is part of KB2992611), but it seems that it didn't work and for some reason you can't install KB2992611 manually (it claims 'update is not applicable to your system').

Workaround
Remove these suites from client list via group policy (Computer Configuration\Administrative Templates\Network\SSL Configuration Settings) or disable them in registry (on Windows Server 2008+ can be done with IISCrypto).

A note for AWS SDK team
You may want to consider intercepting some of WinHTTP logging (and redirecting it to AWS SDK logging system) via WinHttpSetStatusCallback() -- I had to hack AWS SDK a bit to do it. It would be nice to have this ability 'out of the box'. :)

More info

  • How to capture WinHTTP logs: here

  • error manifest in two ways (SEC_E_MESSAGE_ALTERED and SEC_E_BUFFER_TOO_SMALL(0x80090321)) in WinHTTP logs. Only one of those cases generates SChannel error in 'System' section of windows logs (accessible via Event Viewer)

  • using wireshark to figure out cipher suite selected for given SSL conneciton: here

  • MS article (see 'Known issue 2')

  • more on botched KB2992611

  • maybe related
pks-t added a commit to pks-t/libgit2 that referenced this issue Mar 29, 2018
@pks-t
appveyor: disable DHE to avoid spurious failures
723e1e9 
Our CI builds have intermittent failures in our online tests, e.g. with
the message "A provided buffer was too small". This is not a programming
error in libgit2 but rather an error in the SChannel component of
Windows. Under certain circumstances involving Diffie-Hellman key
exchange, SChannel is unable to correctly handle input from the server.
This bug has already been fixed in recent patches for Windows 10 and
Windows Server 2016, but they are not yet available for AppVeyor.

Manually pamper over that issue by disabling all ciphersuites using DHE
via the registry. While this disables more ciphers than necessary, we
really don't care for that at all but just want to avoid build failures
due to that bug.

See [1], [2] or [3] for additional information.

1: aws/aws-sdk-cpp#671
2: https://github.com/dotnet/corefx/issues/7812
3: https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n
@pks-t pks-t mentioned this issue Mar 29, 2018
Merged
pks-t added a commit to pks-t/libgit2 that referenced this issue Apr 20, 2018
@pks-t
appveyor: disable DHE to avoid spurious failures
06f7825 
Our CI builds have intermittent failures in our online tests, e.g. with
the message "A provided buffer was too small". This is not a programming
error in libgit2 but rather an error in the SChannel component of
Windows. Under certain circumstances involving Diffie-Hellman key
exchange, SChannel is unable to correctly handle input from the server.
This bug has already been fixed in recent patches for Windows 10 and
Windows Server 2016, but they are not yet available for AppVeyor.

Manually pamper over that issue by disabling all ciphersuites using DHE
via the registry. While this disables more ciphers than necessary, we
really don't care for that at all but just want to avoid build failures
due to that bug.

See [1], [2] or [3] for additional information.

1: aws/aws-sdk-cpp#671
2: https://github.com/dotnet/corefx/issues/7812
3: https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n
pks-t added a commit to pks-t/libgit2 that referenced this issue May 30, 2018
@pks-t
appveyor: disable DHE to avoid spurious failures
1cc6cc9 
Our CI builds have intermittent failures in our online tests, e.g. with
the message "A provided buffer was too small". This is not a programming
error in libgit2 but rather an error in the SChannel component of
Windows. Under certain circumstances involving Diffie-Hellman key
exchange, SChannel is unable to correctly handle input from the server.
This bug has already been fixed in recent patches for Windows 10 and
Windows Server 2016, but they are not yet available for AppVeyor.

Manually pamper over that issue by disabling all ciphersuites using DHE
via the registry. While this disables more ciphers than necessary, we
really don't care for that at all but just want to avoid build failures
due to that bug.

See [1], [2] or [3] for additional information.

1: aws/aws-sdk-cpp#671
2: https://github.com/dotnet/corefx/issues/7812
3: https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n
pks-t added a commit to pks-t/libgit2 that referenced this issue Oct 26, 2018
@pks-t
appveyor: disable DHE to avoid spurious failures
cd14fca 
Our CI builds have intermittent failures in our online tests, e.g. with
the message "A provided buffer was too small". This is not a programming
error in libgit2 but rather an error in the SChannel component of
Windows. Under certain circumstances involving Diffie-Hellman key
exchange, SChannel is unable to correctly handle input from the server.
This bug has already been fixed in recent patches for Windows 10 and
Windows Server 2016, but they are not yet available for AppVeyor.

Manually pamper over that issue by disabling all ciphersuites using DHE
via the registry. While this disables more ciphers than necessary, we
really don't care for that at all but just want to avoid build failures
due to that bug.

See [1], [2] or [3] for additional information.

1: aws/aws-sdk-cpp#671
2: https://github.com/dotnet/corefx/issues/7812
3: https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n
(cherry picked from commit 723e1e9)
@webbnh webbnh mentioned this issue Jan 10, 2019
Closed
@franccan franccan mentioned this issue Sep 9, 2020
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@crusader-mike