Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.

Add server-side SNI support #28278

Merged
merged 20 commits into from Mar 30, 2018
Merged

Add server-side SNI support #28278

merged 20 commits into from Mar 30, 2018

Conversation

krwq
Copy link
Member

@krwq krwq commented Mar 20, 2018

Related issue: https://github.com/dotnet/corefx/issues/24553
This is for overview only so that it is easier to discuss the API shape.

Changes to original proposal:

  • delegate is defined in the namespace and not in the SslStream (to match others)
  • name has changed to ServerCertificateSelectionCallback
  • removed X509CertificateCollection from return values as currently do not see what it would be used for (if other implementations require it will change it)

Left:

cc @Drawaes

@krwq krwq added the * NO MERGE * The PR is not ready for merge yet (see discussion for detailed reasons) label Mar 20, 2018
@@ -0,0 +1,111 @@
using System;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(pending) missing copyright header

@@ -32,6 +32,8 @@ public enum EncryptionPolicy
RequireEncryption = 0,
}
public delegate System.Security.Cryptography.X509Certificates.X509Certificate LocalCertificateSelectionCallback(object sender, string targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection localCertificates, System.Security.Cryptography.X509Certificates.X509Certificate remoteCertificate, string[] acceptableIssuers);
public delegate X509Certificate ServerCertificateSelectionCallback(object sender, string hostName);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's treat this file as informal API design discussion and other files as reference

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref definitions need fully qualified type name: "System.Security.Cryptography.X509Certificates.X509Certificate" like the delegate definition above.

@@ -0,0 +1 @@
schannel.dll!SslGetServerIdentity
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might be missing this somewhere - UAP perhaps?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

Copy link
Contributor

@davidsh davidsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work so far!

We can talk offline about how to set up a working end-to-end test (manual only....not for CI).

@@ -23,6 +23,7 @@ internal static partial class Libraries
internal const string PerfCounter = "perfcounter.dll";
internal const string RoBuffer = "api-ms-win-core-winrt-robuffer-l1-1-0.dll";
internal const string Secur32 = "secur32.dll";
internal const string SChannel = "schannel.dll";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls investigate if you can use 'secur32.dll' for this p/invoke.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like only in schannel.dll, getting EntryPointNotFoundException with secur32

public void ReadFrame(bool server, out byte[] buffer)
{
if (_connectionBroken)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: style - we usually use braces around the body of 'if' statements unless the whole thing is on a single line.

public void BreakConnection()
{
_connectionBroken = true;
_serverDataAvailable.Release(1000000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you could use "_" in numeric literals to make this easier to read.

@@ -32,6 +32,8 @@ public enum EncryptionPolicy
RequireEncryption = 0,
}
public delegate System.Security.Cryptography.X509Certificates.X509Certificate LocalCertificateSelectionCallback(object sender, string targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection localCertificates, System.Security.Cryptography.X509Certificates.X509Certificate remoteCertificate, string[] acceptableIssuers);
public delegate X509Certificate ServerCertificateSelectionCallback(object sender, string hostName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref definitions need fully qualified type name: "System.Security.Cryptography.X509Certificates.X509Certificate" like the delegate definition above.

@@ -179,6 +182,7 @@ public partial class SslStream : AuthenticatedStream
public virtual void AuthenticateAsServer(System.Security.Cryptography.X509Certificates.X509Certificate serverCertificate) { }
public virtual void AuthenticateAsServer(System.Security.Cryptography.X509Certificates.X509Certificate serverCertificate, bool clientCertificateRequired, System.Security.Authentication.SslProtocols enabledSslProtocols, bool checkCertificateRevocation) { }
public virtual void AuthenticateAsServer(System.Security.Cryptography.X509Certificates.X509Certificate serverCertificate, bool clientCertificateRequired, bool checkCertificateRevocation) { }
public void AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions) { }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't add new sync-only APIs. so, pls remove this.

@@ -334,14 +351,16 @@ public virtual void AuthenticateAsServer(X509Certificate serverCertificate, bool
AuthenticateAsServer(options);
}

private void AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions)
public void AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be left as 'private'

return null;
}

return Encoding.UTF8.GetString(serverIdentityBytes, serverIdentitySize);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls verify RFC for TLS/SSL that it is UTF8 encoding.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still pending

Copy link
Member Author

@krwq krwq Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidsh
RFC 4366 defines UTF-8 (should only be used with host names which are defined to be UTF-8)
RFC 6066 (App. A) overrides it to ASCII encoding for both host name and SNI. We should stick to whatever we do for host name (ASCII? - need to check this)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slightly confused on the code -not seeing any other encoding than utf-8 in our code base and targetName for client side seems to be delegated to SspiCli - we should check somehow what that does

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that it is correct at one stage utf8 was okay. But as ASCII is the one true way now for DNS names (with internationalization being supported via App A) and this isn't an existing feature you shouldn't provide backwards compat. Although I do wonder if you should just surface a Span for checking anyway

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the string returned from the API should be "decoded" meaning it should undo any Punnycode if it was transmitted that way on the wire.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tratcher: I have mixed feelings about this. From dev perspective it is always easier to do less. From perf perspective it is better to not create a copy of the string (the name is already allocated inside client hello) and comparing can be done with byte comparison (assuming names are in the same encoding which should be the case). From TLS spec standpoint it probably deoesn't matter as it is simply a byte array and length and encoding is an implementation detail (probably that's why they are being vague about what should implementation do about it)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're not avoiding any work in the end, you're only punting it to the caller.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You kinda are. If you have a server name you would eacape it 1x on the server. Then when a name comes in from the client you do a byte compare. You would likely do the initial escape once and the compare for n number of connections. If you have to depunifiy© the client to compare to a string everytime then you are doing n units of work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, but it requires the caller to be IDN aware.

[Theory]
[InlineData("a")]
[InlineData("test")]
[InlineData("aaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be easier to read and maintain if this used memberdata and a "yield return" with constructing the large strings as "new String(a, 20) + etc." for repetitions.

[InlineData("a")]
[InlineData("test")]
[InlineData("aaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc")]
public void ClientSendsSNIServerReceivesIt(string hostName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls use current test naming convention (3 parts with underscores). Pls see other tests for examples.

@@ -22,6 +22,7 @@
<ItemGroup>
<Compile Include="NotifyReadVirtualNetworkStream.cs" />
<Compile Include="DummyTcpServer.cs" />
<Compile Include="SslStreamSNITest.cs" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls move this line lower into the rest of the SslStream includes.

Copy link
Contributor

@davidsh davidsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.

};

var cts = new CancellationTokenSource();
Assert.ThrowsAsync<NotSupportedException>(async () => {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question is what should we throw when callback returns null - currently there is a code handling null case and I have not added anything new but this is something we might consider changing.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does SslStream throw today if you try to AuthenticateAsServerAsync with a null certificate? whatever that does is probably going to make most sense?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this scenario is being detected early - in this case we don't know about null until we receive hello from the client

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a test for this to make sure that the exception we might throw if catch-able by the developer's code. We don't want, for instance, to throw an exception on a threadpool thread.

Copy link
Member Author

@krwq krwq Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidsh this is a test for that - I will likely change the exception once I figure out which one is the most appropriate here.

@krwq krwq changed the title [WIP] Add server sni windows [WIP] Add server-side SNI support (currently windows only) Mar 20, 2018
int ret = Interop.SChannel.SslGetServerIdentity(clientHelloBytes, clientHello.Length, out serverIdentityBytes, out serverIdentitySize, 0);
if (ret != 0 || serverIdentitySize == 0)
{
return null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the callback will be invoked with a null host name if SNI was not provided by the client?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a null string as input to the developer's callback seems reasonable in this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tratcher correct

@krwq krwq dismissed davidsh’s stale review March 26, 2018 19:54

please review new iteration

// - NameType name_type (1 byte) => 0x00 is host_name
// - opaque HostName
// Per spec:
// If the hostname labels contain only US-ASCII characters, then the
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this part is TODO. We will likely decide on either returning bytes or IDN unmap but let's wait for API design discussion - for now treating as UTF-8 string

private readonly ConcurrentQueue<byte[]> _clientWriteQueue = new ConcurrentQueue<byte[]>();
private readonly ConcurrentQueue<byte[]> _serverWriteQueue = new ConcurrentQueue<byte[]>();

private readonly SemaphoreSlim _clientDataAvailable = new SemaphoreSlim(0);
private readonly SemaphoreSlim _serverDataAvailable = new SemaphoreSlim(0);

public bool DisableConnectionBreaking { get; set; } = false;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reconsider if this option is needed (I think there was 1 test case on linux failing because of lack of this but I think I accidentally reverted the fix - let's wait for linux results)

@krwq krwq changed the title [WIP] Add server-side SNI support (currently windows only) [WIP] Add server-side SNI support Mar 26, 2018
Copy link
Contributor

@davidsh davidsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.


namespace System.Net.Security
{
internal class SNIHelper
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: SNIHelper -> SniHelper
Style guideline uses normal Pascal casing when acronym is more than 2 characters. Change filename as well to match.

{
using Configuration = System.Net.Test.Common.Configuration;

public class SslStreamSNITest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "SslStreamSNITest" -> "SslStreamSniTest".

..change all other "SNI" patterns to "Sni" in other file(s).

// - opaque fragment[SSLPlaintext.length]
private static string GetSniFromSslPlainText(ReadOnlySpan<byte> sslPlainText)
{
// Is SSL 3 handshake? SSL 2 does not support extensions - skipping as well
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently don't "allow" Ssl2 or Ssl3 in .NET Core. It's marked as Obsolete.
https://github.com/dotnet/corefx/blob/master/src/System.Net.Primitives/ref/System.Net.Primitives.cs#L494

But I agree we should probably be correct in parsing it in case it comes from the server.

}

[Fact]
public void SslStream_UnknownHostName_Fails()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be better to rename this test to indicate that it is testing that no SNI data was sent from client to server and thus the callback returns null.

So, perhaps this is a better name: "SslStream_NoSniFromClient_CallbackReturnsNull"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidsh, thanks, I thought the last part is always Fails or Ok 😄

Copy link
Contributor

@davidsh davidsh Mar 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last part is usually "Success" in cases where it makes sense. Some folks use "Succeeds". But "Success" is used most often in the repo especially for network tests.

For failure cases, it always better to use terms like "Throws" or "ThrowsSocketException", or in this case, "CallbackReturnsNull". It is more descriptive and helps us understand the test case better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks will update!

// Opaque type is of structure:
// - length (minimum number of bytes to hold the max value)
// - data (length bytes)
// We will only use opaque bytes which are of max size: 255 (length = 1) or 2^16-1 (length = 2).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: opaque types (not bytes)

}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static ushort ReadUint16(ReadOnlySpan<byte> bytes)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually we could leave it as int since we have to double convert ushort otherwise (here and for comparison)

@krwq
Copy link
Member Author

krwq commented Mar 26, 2018

cc: @davidfowl

@karelz karelz added this to the 2.1.0 milestone Mar 27, 2018
return null;
}

return GetSniFromSslHandshake(sslHandshake);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, might be simpler to just combine the methods

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you don't need to check sslHandshake.Length < 4 I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danmosemsft, could you expand? I think I will still need to check - note that sslHandshake is sliced sslPlaintext. The lengths are fairly redundant in the SSL spec which makes this code slightly confusing but allow to skip parts implementation doesn't understand

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. I misread.

@@ -10,6 +10,7 @@
using System.Security.Authentication.ExtendedProtection;
using System.Security.Cryptography.X509Certificates;
using System.Security.Principal;
using System.Text;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Copy link
Member

@GrabYourPitchforks GrabYourPitchforks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor feedback, still approved.

@@ -15,6 +14,7 @@ internal class SniHelper
private const int UInt24Size = 3;
private const int RandomSize = 32;
private static IdnMapping s_idnMapping = CreateIdnMapping();
private static Encoding s_encoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: private readonly static.

@krwq
Copy link
Member Author

krwq commented Mar 29, 2018

Thank you @GrabYourPitchforks! I've removed the dependency I added by creating Utf8Encoding differently (thank you @tarekgh!) and added a test to make sure we do not regress that in the future

@krwq krwq removed the * NO MERGE * The PR is not ready for merge yet (see discussion for detailed reasons) label Mar 29, 2018
@karelz
Copy link
Member

karelz commented Mar 30, 2018

@stephentoub @bartonjs @Drawaes @Tratcher any additional feedback before we merge the PR?

@@ -66,6 +67,7 @@ internal SslAuthenticationOptions(SslServerAuthenticationOptions sslServerAuthen
internal bool CheckCertName { get; set; }
internal RemoteCertValidationCallback CertValidationDelegate { get; set; }
internal LocalCertSelectionCallback CertSelectionDelegate { get; set; }
internal ServerCertCallback ServerCertSelectionDelegate { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed this one?

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

CI's being a little slow...

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

test OSX x64 Debug Build
test Linux x64 Release Build
test CROSS Check

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

@karelz seems like all PRs are timing out on these 3 builds. CI was passing on almost identical version such as 40f8447 - do you think it is ok to merge?

@karelz
Copy link
Member

karelz commented Mar 30, 2018

Let's wait on CI to come back for now. Having the fix in master just few hours earlier is not worth the risk of breaking master just before last integration to release/2.1 branch.
We will have to come up with overall strategy for CI impact on ZBB and our integration plans.


private static Encoding CreateEncoding()
{
return Encoding.GetEncoding("utf-8", new EncoderExceptionFallback(), new DecoderExceptionFallback());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to use GetEncoding("utf-8", ...) rather than new UTF8Encoding() or Encoding.UTF8?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephentoub Utf8Encoding is in different assembly and it would introduce new dependency (System.Text.Encoding.Extensions) where we can create this exact type without introducing it by doing above

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that a dependency we care about? The implementation of UTF8Encoding is in corelib. And using Encoding.GetEncoding(string) here would seem to a) be more expensive, and b) not necessarily produce a UTF8Encoding as (if I'm understanding the code correctly) it'll go through any registered providers before falling back to looking up what encoder should be used by internal tables.

(And I assume Encoding.UTF8 can't be used because the fallbacks it needs are different?)

cc: @tarekgh, @weshaggard

Anyway, don't block on this one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Encoding.UTF8 does not throw on invalid utf-8 byte sequences.

The more expensive part I'm not concerned about since this is once per process cost.

Getting non-utf8 here would be a concern here though. I'm not sure if there are any concerns with new dependency but it seemed not appropriate to add it just before 2.1 release.

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

test CROSS Check

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

@karelz CROSS Check shows as "Started." but CI console text says it has succeeded: https://ci.dot.net/job/dotnet_corefx/job/master/job/CROSS_check_prtest/14394/consoleText I think this is ok to merge.

@karelz
Copy link
Member

karelz commented Mar 30, 2018

Sounds good, go ahead. Thanks for pushing it through!

@stephentoub
Copy link
Member

Was there an answer to my question at #28278 (comment)?

@krwq
Copy link
Member Author

krwq commented Mar 30, 2018

Merging this since https://github.com/dotnet/corefx/issues/28657 is marked as 2.1. Thanks everyone!

@krwq krwq merged commit 2d39212 into dotnet:master Mar 30, 2018
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
* add initial implementation for server side SNI support

* remove analyzerdata

* add braces around single line if statements and inline simple functions

* fix deadlock on linux

* add test with special characters

* IDN decoding

* idn unmapping (with fallback)

* apply feedback, fix netfx/uwp

* rename the SNIHelper to SniHelper

* rename test file as well (git does not like renames on windows)

* change file casing in csproj

* test should expect AuthenticationException

* add SniHelper tests

* apply review feedback

* shorten SNI (limit is 63 bytes)

* replace remaining constants

* test behavior on truncated client hello

* add structures descriptions

* apply review feedback

* get rid of the new dependency and add a test for invalid utf-8 bytes


Commit migrated from dotnet/corefx@2d39212
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
9 participants