SNI: Update SNI related log messages #1240

Closed
wants to merge 1 commit into
from

Projects

None yet

6 participants

@JDepooter
Contributor

Update debug log messages related to SNI:
schannel: Turning off host validation does not turn off SNI
darwinssl: Turning off host validation does turn off SNI

I recently came across a case which worked on Windows using SChannel but not on a Mac using DarwinSSL. It turns out the request was accessing a machine on AWS which required SNI in the TLS handshake. The request was setup to turn off host validation.

Apparently, turning off host validation on DarwinSSL also turns off SNI, which is why the request failed on the Mac. I have added a debug message when this happens, as I think it will be useful for future debugging.

With SChannel , the debug messages claimed that turning off host validation also turned off SNI, but this is not the case. I have updated that debug message as well. See the attached screenshots for verification. It is possible that this behaviour differs between the various windows versions. I only have access to Windows 10.

Both OpenSSL and GnuTLS have the same behaviour as SChannel, SNI is still used when the verify host option is disabled.

curl_output

wireshark_output

@mention-bot

@JDepooter, thanks for your PR! By analyzing the history of the files in this pull request, we identified @mback2k, @nickzman and @bagder to be potential reviewers.

@jay jay added the SSL/TLS label Feb 3, 2017
@jay
Member
jay commented Feb 3, 2017 edited

With SChannel , the debug messages claimed that turning off host validation also turned off SNI, but this is not the case. I have updated that debug message as well. See the attached screenshots for verification. It is possible that this behaviour differs between the various windows versions. I only have access to Windows 10.

Git blame puts the "Also disables SNI" comment in schannel at 1394cad, which has a commit message that says in part:

As a side effect of switching off the RFC2818 related servername checks with SCH_CRED_NO_SERVERNAME_CHECK (http://msdn.microsoft.com/en-us/library/aa923430.aspx) the SNI feature is being disabled. This effect is not documented in MSDN, but Wireshark output clearly shows the effect (details on the libcurl maillist).

I found a bunch of e-mails from Nov 2012 written by the author but not the one that discusses this issue.

I can confirm Windows 7 SP1 SNI is still sent when SCH_CRED_NO_SERVERNAME_CHECK is added to the flags. Test command curld -vk https://www.test.com

/cc @okoeroo

@okoeroo
Contributor
okoeroo commented Feb 4, 2017

Hi,
At the time I had all the libraries on my machines to test with, except the Windows API.

On the maillist archive I dug up this message:
https://curl.haxx.se/mail/lib-2012-11/0036.html

i was amazed about the ill documentation if this side effect really happens. Now you mention that this effect seems to be in line with the MSDN documentation on the Schannel API. I would recommend to follow this and have the message be corrected in libcurl to match the documentation and observation made by @JDepooter and pulled up by @jay.

I did not verify the SNI (side-) effects at the time. I focussed on the peer-jacking attack only exclusively based on the paper which only tested libcurl with the OpenSSL backend. :-)

@jay
Member
jay commented Feb 4, 2017 edited

Thanks for your input @okoeroo. I tested in Windows XP and the SNI is not sent regardless of whether the server name check is enabled. IE which also uses WinSSL shows the same behavior. We could make an <= XP specific message by Curl_verify_windows_version and then if verbose. (Also ref 6842afb)

schannel: WinSSL version is too old to connect to some servers (missing SNI, algorithms)
schannel: WinSSL version is old and may not be able to connect to some servers due to lack of SNI, algorithms, etc.
or something like that and brief. @JDepooter do you want to take a crack at it

Also can we hear from someone with ability to test on Mac versions like @nickzman to know if there is similar inconsistency there.

@JDepooter JDepooter SNI: Update SNI related log messages
darwinssl: Turning off host validation does turn off SNI

schannel: Turning off host validation does not turn off SNI. However, SNI
is not available which SChannel on Windows XP machines, regardless of the
host validation setting.
a9e12a0
@JDepooter
Contributor

@jay I have updated the pull request with a Windows XP specific message which is logged at the beginning of schannel_connect_step1.

@jay jay added a commit that referenced this pull request Feb 7, 2017
@JDepooter @jay JDepooter + jay schannel: Remove incorrect SNI disabled message
- Remove the SNI disabled when host verification disabled message
  since that is incorrect.

- Show a message for legacy versions of Windows <= XP that connections
  may fail since those versions of WinSSL lack SNI, algorithms, etc.

Bug: #1240
18495ec
@jay
Member
jay commented Feb 7, 2017

Thanks. I've split your schannel changes off since it's sort of a different issue and landed it in 18495ec. The darwinssl change I think we should wait for someone who can test new and old Mac versions to check for inconsistency, like maybe old versions disabled SNI and new versions don't? Also typo s/validatation/validation

@jay
Member
jay commented Mar 1, 2017

ping. Anyone with Mac experience can give this a review so I can close it out?

I found a modified version of darwinssl used by the maintainer of Git OS X Installer where he says:

TLS SNI is only available under SecureTransport when running on OS X 10.5.6 or later and SSLSetPeerDomainName has been called.

@mackyle would you mind weighing in on the darwinssl part of this PR, ie that as long as SSLSetPeerDomainName is disabled SNI is disabled as well, that this holds true even in the latest Mac versions?

@nickzman
Collaborator
nickzman commented Mar 1, 2017

Yeah, that's still true; there's no way to disable the peer domain check without also disabling SNI.

@jay jay added a commit that closed this pull request Mar 2, 2017
@JDepooter @jay JDepooter + jay darwinssl: Warn that disabling host verify also disables SNI
In DarwinSSL the SSLSetPeerDomainName function is used to enable both
sending SNI and verifying the host. When host verification is disabled
the function cannot be called, therefore SNI is disabled as well.

Closes #1240
0966ab5
@jay jay closed this in 0966ab5 Mar 2, 2017
@jay
Member
jay commented Mar 2, 2017

Ok. Thanks guys

@mackyle
Contributor
mackyle commented Mar 9, 2017

@jay sorry, just saw this. But yeah, if you never call SSLSetPeerDomainName (I haven't checked the very latest API to see if there are any new functions to set an SNI name independently, but there weren't the last time I looked), then SecureTransport never sends SNI -- you can see this in the code on opensource.apple.com if you dig around in it.

The whole section discussing this can be found in the comments at:

https://github.com/mackyle/git-osx-installer/blob/master/patches/curl/curl_darwinssl_macosx.c#L34

There's a table in the comments:

VERIFYPEER  VERIFYHOST  Result
----------  ----------  ----------------------------------------------------
disabled    disabled    SNI still sent for DNS host names
disabled    enabled     Peer's first cert must match the host DNS or IP name
enabled     disabled    DNS host name matching still occurs, IP does not
enabled     enabled     Peer's first cert must match the host DNS or IP name

It's only the third entry in the table above that ends up with surprising behavior.

Just because you call SSLSetPeerDomainName does not mean you have to verify the peer. The darwinssl version I include in the Git OS X Installer always calls SSLSetPeerDomainName for non-IPv4/IPv6 names but never for IPv4/IPv6 (because SecureTransport is stupid and will send an IPv4 as an SNI if you do and that's standards violating behavior). It also supports CURLOPT_PINNEDPUBLICKEY, works with 10.4 or later and dynamically adjusts to the available SecureTransport features at runtime based on what's actually present in the MacOSX version it runs on (this support was crucial in order to create a universal, single-version installer).

Other than for literal IPv4/IPv6 names, the SSLSetPeerDomainName call should never be skipped because you can end up fetching the wrong site for name-based TLS virtual hosts and that, to me, is far more surprising than having VERIFYPEER force VERIFYHOST.

TLDR; Always call SSLSetPeerDomainName for non-IPv4/IPv6 literal names (otherwise SNI will not be sent) which makes VERIFYPEER force VERIFYHOST for non-IPv4/IPv6 names but avoids fetching the wrong TLS name-based virtual host.

@JDepooter JDepooter deleted the JDepooter:sni_logging_updates branch Mar 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment