Failed assert in NegotiateStream_StreamToStream_Authentication_TargetName_Success

[stephentoub]

I hit this assert when I try to run the System.Net.Security tests (outerloop) locally:

System.Net.Security.Tests.NegotiateStreamStreamToStreamTest.NegotiateStream_StreamToStream_Authentication_TargetName_Success [FAIL]
        System.Diagnostics.Debug+DebugAssertException : Unknown Interop.SecurityStatus value: -2146892976

           at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
           at System.Environment.get_StackTrace()
           at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage)
           at System.Net.SecurityStatusAdapterPal.GetSecurityStatusPalFromInterop(SecurityStatus win32SecurityStatus) in c:\Users\stoub\Source\repos\corefx\src
  \System.Net.Security\src\System\Net\SecurityStatusAdapterPal.Windows.cs:line 74
           at System.Net.Security.NegotiateStreamPal.InitializeSecurityContext(SafeFreeCredentials credentialsHandle, SafeDeleteContext& securityContext, Strin
  g spn, ContextFlagsPal requestedContextFlags, SecurityBuffer[] inSecurityBufferArray, SecurityBuffer outSecurityBuffer, ContextFlagsPal& contextFlags) in c:\
  Users\stoub\Source\repos\corefx\src\System.Net.Security\src\System\Net\NegotiateStreamPal.Windows.cs:line 166
           at System.Net.NTAuthentication.GetOutgoingBlob(Byte[] incomingBlob, Boolean throwOnError, SecurityStatusPal& statusCode) in c:\Users\stoub\Source\re
  pos\corefx\src\System.Net.Security\src\System\Net\NTAuthentication.cs:line 344
           at System.Net.Security.NegoState.GetOutgoingBlob(Byte[] incomingBlob, Exception& e) in c:\Users\stoub\Source\repos\corefx\src\System.Net.Security\sr
  c\System\Net\SecureProtocols\NegoState.cs:line 787
           at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult) in c:\Users\stoub\Source\repos\corefx\src\System.Net.Secu
  rity\src\System\Net\SecureProtocols\NegoState.cs:line 443
           at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult) in c:\Users\stoub\Source\repos\corefx\src\System.Net.Security\src
  \System\Net\SecureProtocols\NegoState.cs:line 356
           at System.Net.Security.NegotiateStream.BeginAuthenticateAsClient(NetworkCredential credential, ChannelBinding binding, String targetName, Protection
  Level requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel, AsyncCallback asyncCallback, Object asyncState) in c:\Users\stoub\Source\re
  pos\corefx\src\System.Net.Security\src\System\Net\SecureProtocols\NegotiateStream.cs:line 101
           at System.Net.Security.NegotiateStream.BeginAuthenticateAsClient(NetworkCredential credential, String targetName, AsyncCallback asyncCallback, Objec
  t asyncState) in c:\Users\stoub\Source\repos\corefx\src\System.Net.Security\src\System\Net\SecureProtocols\NegotiateStream.cs:line 60
           at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg
  2, Object state, TaskCreationOptions creationOptions)
           at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
           at System.Net.Security.NegotiateStream.AuthenticateAsClientAsync(NetworkCredential credential, String targetName) in c:\Users\stoub\Source\repos\cor
  efx\src\System.Net.Security\src\System\Net\SecureProtocols\NegotiateStream.cs:line 183
           at System.Net.Security.Tests.NegotiateStreamStreamToStreamTest.NegotiateStream_StreamToStream_Authentication_TargetName_Success() in c:\Users\stoub\
  Source\repos\corefx\src\System.Net.Security\tests\FunctionalTests\NegotiateStreamStreamToStreamTest.cs:line 90

According to https://msdn.microsoft.com/en-us/library/windows/desktop/dd542646(v=vs.85).aspx, that status code is SEC_E_DOWNGRADE_DETECTED ("The system cannot contact a domain controller to service the authentication request. Please try again later."). Do we need to add this error and others to the table in https://github.com/dotnet/corefx/blob/2ff9b2a1e367a9694af6bdaf9856ea12f9ae13cd/src/System.Net.Security/src/System/Net/SecurityStatusAdapterPal.Windows.cs?

[Priya91]

@davidsh @CIPop Should this be addressed for 2.0?

[CIPop]

We need to investigate this for 2.0 and answer the question: Is the error still being treated correctly by NegotiateStream internally?
If the answer is yes, we can postpone adding the error to the PAL error code adapter.

[caesar-chen]

@stephentoub Do you remember which version of Windows you used to produce this issue? I want to setup a VM and investigate further.

I have looked into the code and compared the old commit with current code. Both will eventually call into a unmanaged code, Interop.SspiCli.InitializeSecurityContextW(), which will return the correct SEC_I_CONTINUE_NEEDED in current code, and return SEC_E_DOWNGRADE_DETECTED in the old code. The method is imported from the SspiCli.dll, which could be modified by Windows security updates.

InitializeSecurityContext is not documented with the DOWNGRADE_DETECTED error https://msdn.microsoft.com/en-us/library/windows/desktop/aa375506(v=vs.85).aspx#return_value, but it's possible to return this error, according to this article: https://support.microsoft.com/en-us/help/2522623/initializesecuritycontext-function-might-not-fall-back-to-ntlm-authentication-in-windows-7-or-in-windows-server-2008-r2-when-kerberos-fails-and-has-the-status-no-logon-servers-status. So my suggestion is adding this error to table, and next time it occurs, we will know that we hit a known issue.

I don't have any evidence yet for the above analysis, because I am not able to repro this issue. I wrote a batch script to run the System.Net Security tests for 100 times, and there is no failure. My next step is to contact sspi experts about this issue.

[stephentoub]

Do you remember which version of Windows you used to produce this issue?

@shen381, sorry, I don't. Most likely Windows 10 Anniversary Update, as that's what would have been on my machine at about the time this issue was opened.

So my suggestion is adding this error to table

Sounds good to me.

[caesar-chen]

Now the question can be answered.

In .NET Core:

I have researched into this specific scenario: Interop.SspiCli.InitializeSecurityContextW called by AuthenticateAsClientAsync(), and the function fails and returns an error. This is how our code (NegotiateStream) handles it internally:

1. Release the credential passed in
2. Set the out context handle to invalid value
3. Look up error code in the table, if can’t find error code, throw Debug.Fail("Unknown Interop.SecurityStatus value: " + win32SecurityStatus)
4. If the error code can be found in the table, return the error to the upper level function. That function will eventually reset authentication state, and wrap the error with AuthenticationException, as inner exception.
5. The exception will be returned when EndProcessAuthentication() get called

So now for SEC_E_DOWNGRADE_DETECTED, step 3 is performed. After I add it to table, step 4, 5 will be performed.

Details of what I did:

The method I adopted is, set breakpoints at the entry API call (AuthenticateAsClientAsync) and where the error code get returned by SspiCli.dll (InitializeSecurityContextW). When the second BP get hit, I manually changed its error code to -2146892976 (SEC_E_DOWNGRADE_DETECTED). https://msdn.microsoft.com/en-us/library/windows/desktop/dd542646(v=vs.85).aspx and observe our code flow.

In .NET Framework:

I wrote a program in .NET Framework which uses System.Net.Security.NegotiateStream.AuthenticateAsClientAsync(), and used WinDBG to debug into it. AuthenticateAsClientAsync() will call in to the unmanaged Win32 API, Interop.SspiCli.InitializeSecurityContextW(). I set a breakpoint at InisitalizeSecurityContextW, step until just before it returns. Then change return register(EAX) register value to the error code SEC_E_DOWNGRADE_DETECTED, and let it execute.

This is what I observe:

0:000> g
(1b0c.14ec): CLR exception - code e0434352 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0811f1b8 ebx=00000005 ecx=00000005 edx=00000000 esi=0811f278 edi=00000001
eip=765b4ff2 esp=0811f1b8 ebp=0811f210 iopl=0 nv up ei pl nz ac pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216
KERNELBASE!RaiseException+0x62:
765b4ff2 8b4c2454 mov ecx,dword ptr [esp+54h] ss:002b:0811f20c=66147986

0:013> !pe
Exception object: 051385f8
Exception type: System.Security.Authentication.AuthenticationException
Message: A call to SSPI failed, see inner exception.
InnerException: System.ComponentModel.Win32Exception, Use !PrintException 1481b82b to see more.
StackTrace (generated):

StackTraceString:
HResult: 80131501

0:013> !PrintException /d 05137fc0
Exception object: 05137fc0
Exception type: System.ComponentModel.Win32Exception
Message: The system cannot contact a domain controller to service the authentication request. Please try again later
InnerException:
StackTrace (generated):

StackTraceString:
HResult: 80004005

Conclusion:

I have verified in .Net Framework, which matches the description for what will happen if adding the error code in the table for .Net core.

I will submit a PR for this issue.

@stephentoub @davidsh @CIPop