Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ASP.NET core 2.2.4 impersonation not working with SocketsHttpHandler #29351

Closed
Cynthia-Jiang opened this issue Apr 23, 2019 · 29 comments
Closed
Assignees
Labels
blocked Issue/PR is blocked on something - see comments bug
Milestone

Comments

@Cynthia-Jiang
Copy link

environment:
DC: windows server 2012R2
server1: windows server 2016, ASP.NET core 2.2.4 web API on IIS with windows authentication enabled
server2: windows server 2012R2, ASP.NET core 2.2 web api on IIS with windows authentication enabled
client: any browsers
client -> server1 -> server2.
the ASP.NET core app in server1 will impersonate the authenticated user then call server 2 web api.
i have configured app pools on both servers to use service account, and have configured SPN accordingly, added delegation setting for the service account of server1.

sample code of how i do impersonate:

var windowsIdentity = (WindowsIdentity)this.HttpContext.User.Identity;

await (WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, async () =>
{
	
    using (HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, "http://winsvr12r2-1.jiangya1.lab:4000/api/values"))
    {
		
        string user_name = WindowsIdentity.GetCurrent().Name;
        
		try
        {
			
            var response = await _client.SendAsync(requestMessage);
            
			var responseString = await response.Content.ReadAsStringAsync();
            
			result = ($"Called API as: {WindowsIdentity.GetCurrent().Name}.\n\n" +
                               $"{responseString}\n");
        
		}
        
		catch (Exception ex)
        {
        
			result = ex.ToString();
        }
    }
}));
return result;

expected scenario: the impersonation process would succeed,and server2 gets requests from the initial credential.
real scenario: server1 would throw following exception:

System.Net.Http.HttpRequestException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server ---> System.Net.Sockets.SocketException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at WebApplication2.Controllers.PassThroughController.<>c__DisplayClass5_0.<<Get>b__0>d.MoveNext() in C:\Users\jiangya\source\repos\WebApplication2\WebApplication2\Controllers\PassThroughController.cs:line 92

further troubleshooting:
while doing debugging, i found that if i turn off SocketsHttpHandler following this
and use HTTPClientHandler, the issue will get resolved. which seems indicate that ASP.NET core impersonation is not working with SocketsHttpHandler.
i also attached full callstack here:
working-callstack.txt
non-working-callstack.txt

I understand the new socket handler is designed to eliminate platform dependencies, however can you suggest how to make it work with impersonation?

non-working callstack with SocketsHttpHandler:

0:030> !\\**********\shared\Microsoft.NETCore.App\2.2.4\sos.clrstack
OS Thread Id: 0xe00 (30)
        Child SP               IP Call Site
000000d9e19fbf78 00007ffca5a60160   Interop+Winsock.GetAddrInfoExW(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
000000d9e19fbf78 00007ffc343c65a4   Interop+Winsock.GetAddrInfoExW(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
000000d9e19fbf20 00007ffc343c65a4 DomainBoundILStubClass.IL_STUB_PInvoke(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
000000d9e19fc040 00007ffc8aaafba8 System.Net.NameResolutionPal.GetAddrInfoAsync(System.Net.DnsResolveAsyncResult)  
000000d9e19fc110 00007ffc8aaac27b System.Net.Dns.HostResolutionBeginHelper(System.String, Boolean, Boolean, System.AsyncCallback, System.Object)  
000000d9e19fc190 00007ffc8aaacea8 System.Net.Dns.BeginGetHostAddresses(System.String, System.AsyncCallback, System.Object)  
000000d9e19fc1e0 00007ffc8ab1bcc2 System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(System.Net.Sockets.SocketAsyncEventArgs, System.Net.DnsEndPoint)  
000000d9e19fc250 00007ffc8ab08f23 System.Net.Sockets.Socket.ConnectAsync(System.Net.Sockets.SocketType, System.Net.Sockets.ProtocolType, System.Net.Sockets.SocketAsyncEventArgs)  
000000d9e19fc2b0 00007ffc343c5d98 System.Net.Http.ConnectHelper+d__2.MoveNext()  
000000d9e19fc3a0 00007ffc343c5759 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fc410 00007ffc8a95726e System.Net.Http.ConnectHelper.ConnectAsync(System.String, Int32, System.Threading.CancellationToken)
000000d9e19fc4d0 00007ffc8a95afae System.Net.Http.HttpConnectionPool+d__44.MoveNext()  
000000d9e19fc6e0 00007ffc343c51e9 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fc750 00007ffc8a95950e System.Net.Http.HttpConnectionPool.CreateConnectionAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
000000d9e19fc850 00007ffc343c4d29 System.Net.Http.HttpConnectionPool.GetConnectionAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)  
000000d9e19fc9f0 00007ffc8a95a662 System.Net.Http.HttpConnectionPool+d__39.MoveNext()  
000000d9e19fcaf0 00007ffc343c3ff9 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fcb60 00007ffc8a959225 System.Net.Http.HttpConnectionPool.SendWithRetryAsync(System.Net.Http.HttpRequestMessage, Boolean, System.Threading.CancellationToken)
000000d9e19fcc00 00007ffc8a9593ee System.Net.Http.HttpConnectionPool.SendWithProxyAuthAsync(System.Net.Http.HttpRequestMessage, Boolean, System.Threading.CancellationToken)  
000000d9e19fcc60 00007ffc8a9457a2 System.Net.Http.AuthenticationHelper+d__16.MoveNext()  
000000d9e19fcd60 00007ffc343c3cf9 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fcdd0 00007ffc8a9443fb System.Net.Http.AuthenticationHelper.SendWithAuthAsync(System.Net.Http.HttpRequestMessage, System.Uri, System.Net.ICredentials, Boolean, Boolean, Boolean, System.Net.Http.HttpConnectionPool, System.Threading.CancellationToken)
000000d9e19fce90 00007ffc8a95946c System.Net.Http.HttpConnectionPool.SendAsync(System.Net.Http.HttpRequestMessage, Boolean, System.Threading.CancellationToken)  
000000d9e19fcef0 00007ffc8a95c79c System.Net.Http.HttpConnectionPoolManager.SendAsyncCore(System.Net.Http.HttpRequestMessage, System.Uri, Boolean, Boolean, System.Threading.CancellationToken)  
000000d9e19fcfd0 00007ffc8a95c93a System.Net.Http.HttpConnectionPoolManager.SendAsync(System.Net.Http.HttpRequestMessage, Boolean, System.Threading.CancellationToken)  
000000d9e19fd030 00007ffc8a958cea System.Net.Http.HttpAuthenticatedConnectionHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)  
000000d9e19fd060 00007ffc8a95f388 System.Net.Http.RedirectHandler+d__4.MoveNext()  
000000d9e19fd100 00007ffc343c2729 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fd170 00007ffc8a95efba System.Net.Http.RedirectHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
000000d9e19fd1f0 00007ffc8a95ead2 System.Net.Http.SocketsHttpHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)  
000000d9e19fd240 00007ffc8a93a259 System.Net.Http.HttpClientHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)  
000000d9e19fd290 00007ffc8a93ce42 System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)  
000000d9e19fd2e0 00007ffc8a936954 System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage, System.Net.Http.HttpCompletionOption, System.Threading.CancellationToken)  
000000d9e19fd350 00007ffc343bf2fd Error: Failed to load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.4\coreclr.dll
WebApplication2.Controllers.PassThroughController+c__DisplayClass5_0+b__0>d.MoveNext()
000000d9e19fd3c0 00007ffc343bf109 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fd430 00007ffc343bf098 WebApplication2.Controllers.PassThroughController+c__DisplayClass5_0.b__0()
000000d9e19fd4a0 00007ffc8ba09446 System.Security.Principal.WindowsIdentity+c__DisplayClass60_0`1 .b__0()
000000d9e19fd4d0 00007ffc8ba09517 System.Security.Principal.WindowsIdentity+c__DisplayClass64_0.b__0(System.Object)  
000000d9e19fd510 00007ffc8d3c2019 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  
000000d9e19fd590 00007ffc8ba07993 System.Security.Principal.WindowsIdentity.RunImpersonatedInternal(Microsoft.Win32.SafeHandles.SafeAccessTokenHandle, System.Action)  
000000d9e19fd5e0 00007ffc8ba077df System.Security.Principal.WindowsIdentity.RunImpersonated 
000000d9e19fd630 00007ffc343be842 WebApplication2.Controllers.PassThroughController+d__5.MoveNext()
000000d9e19fd690 00007ffc343be5d9 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start 
000000d9e19fd700 00007ffc343be558 WebApplication2.Controllers.PassThroughController.Get()

working callstack with HttpClientHandler:

0:014> k
Child-SP          RetAddr           Call Site
00 000000b3`80a7e140 00007ffc`a5a5ec39 
ws2_32!WinsockThreadpool_CreateWorkContext+0x9b   
01 000000b3`80a7e180 00007ffc`a5a59684 ws2_32!WinsockThreadpool_SubmitWork+0x45   
02 000000b3`80a7e1d0 00007ffc`a5a58009 ws2_32!NSQUERY::LookupServiceNextAsync+0x154   
03 000000b3`80a7e250 00007ffc`a5a56b67 ws2_32!WSALookupServiceNextAsyncW+0x169   
04 000000b3`80a7e2b0 00007ffc`a5a57070 ws2_32!NSSUBJOB::LookupAsyncAndHandleResultsQDNS4F+0xb7   
05 000000b3`80a7e340 00007ffc`a5a5717e ws2_32!NSSUBJOB::TrySubmitQDNS4F+0x6c   
06 000000b3`80a7e390 00007ffc`a5a56814 ws2_32!NSSUBJOB::TrySubmit+0x8e   
07 000000b3`80a7e3f0 00007ffc`a5a55f06 ws2_32!NSJOB::SubmitChild+0x44   
08 000000b3`80a7e440 00007ffc`a5a56329 ws2_32!NSJOB::ForEachChild+0xd6   
09 (Inline Function) --------`-------- ws2_32!NSJOB::ForEachChild+0x12   
0a 000000b3`80a7e490 00007ffc`a5a57201 ws2_32!NSJOB::TrySubmit+0x89   
0b 000000b3`80a7e4f0 00007ffc`a5a56814 ws2_32!NSSUBJOB::TrySubmit+0x111   
0c 000000b3`80a7e550 00007ffc`a5a55f06 ws2_32!NSJOB::SubmitChild+0x44   
0d 000000b3`80a7e5a0 00007ffc`a5a56329 ws2_32!NSJOB::ForEachChild+0xd6   
0e (Inline Function) --------`-------- ws2_32!NSJOB::ForEachChild+0x12   
0f 000000b3`80a7e5f0 00007ffc`a5a55913 ws2_32!NSJOB::TrySubmit+0x89   
10 000000b3`80a7e650 00007ffc`a5a6145c ws2_32!NSMASTERJOB::TrySubmit+0x53   
11 000000b3`80a7e690 00007ffc`99109ac1 ws2_32!GetAddrInfoExW+0x12fc   
12 000000b3`80a7e8d0 00007ffc`991099d1 webio!WapDnsResolveGeneric+0xd9   
13 000000b3`80a7e9c0 00007ffc`99108e3e webio!WapStartDnsIoContext+0x31   
14 000000b3`80a7e9f0 00007ffc`99108a6e webio!WaStartDnsQuery+0x21e   
15 000000b3`80a7eab0 00007ffc`99108fa4 webio!WapTcpStartDnsQuery+0x38e   
16 000000b3`80a7eb40 00007ffc`991090b7 webio!WaTcpConnectRequest+0x54   
17 000000b3`80a7eb90 00007ffc`99102f44 webio!WapHttpConnect+0x107   
18 000000b3`80a7ec20 00007ffc`99102da8 webio!WaQueueHttpRequest+0x144   
19 (Inline Function) --------`-------- webio!WapStartQueueRequest+0x1c   
1a (Inline Function) --------`-------- webio!WapGetConnectionCompletionRoutine+0x5c   
1b 000000b3`80a7ecc0 00007ffc`99102c88 webio!WapStartGetConnection+0xf8   
1c 000000b3`80a7ed10 00007ffc`99102b16 webio!WapStartGetEndpoints+0x44   
1d 000000b3`80a7ed40 00007ffc`99105008 webio!WapAsynchronousSendHttpRequest+0x266   
1e (Inline Function) --------`-------- webio!WapSendHttpRequest+0x2f   
1f 000000b3`80a7ee20 00007ffc`9f001c65 webio!WebSendHttpRequest+0x38   
20 (Inline Function) --------`-------- winhttp!WEBIO_SENDER::SendRequest+0x21d   
21 000000b3`80a7ee80 00007ffc`9f007528 winhttp!WEBIO_REQUEST::SendRequest+0x415   
22 000000b3`80a7ef70 00007ffc`9f007b60 winhttp!HTTP_USER_REQUEST::_SysSendRequest+0x1028   
23 000000b3`80a7f170 00007ffc`9f00783b winhttp!HTTP_USER_REQUEST::_SendRequestWithDrainComplete+0x2d0   
24 000000b3`80a7f200 00007ffc`9f0023fb winhttp!HTTP_USER_REQUEST::_SendRequestWithProxyInitialized+0x6b   
25 000000b3`80a7f230 00007ffc`9efe3a57 winhttp!HTTP_USER_REQUEST::OnProxyResolvedOnSend+0xbf   
26 000000b3`80a7f280 00007ffc`9efe3710 winhttp!UserRequestProxyCompletion::OnProxyResolved+0x27   
27 000000b3`80a7f2b0 00007ffc`9efe351e winhttp!WxProxyManager::WxProxyFailoverCompletionRoutine+0x90   
28 000000b3`80a7f2e0 00007ffc`a60dec59 winhttp!RefCountWorkItemThread+0x1e   
29 000000b3`80a7f310 00007ffc`a60c328a ntdll!RtlpTpWorkCallback+0x129   
2a 000000b3`80a7f3f0 00007ffc`a4e98364 ntdll!TppWorkerThread+0x4aa   
2b 000000b3`80a7f7f0 00007ffc`a60f7091 KERNEL32!BaseThreadInitThunk+0x14   
2c 000000b3`80a7f820 00000000`00000000 ntdll!RtlUserThreadStart+0x21   
@davidsh
Copy link
Contributor

davidsh commented Apr 23, 2019

@Tratcher

@Tratcher
Copy link
Member

await (WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, async () =>
RunImpersonated does not support async, you need to block inside to prevent the impersonation from terminating early.
https://github.com/dotnet/corefx/issues/24977

@davidsh
Copy link
Contributor

davidsh commented Apr 23, 2019

Duplicate of #24009.

@davidsh davidsh closed this as completed Apr 23, 2019
@Cynthia-Jiang
Copy link
Author

This doesn't sound like the story for me..
when i'm checking 30275 no one comment that it doesn't support async.
Besides, how to explain that if i use HTTPClientHandler, it works perfectly? i can easily share my lab to you if you like, through Microsoft corporate network.

@davidsh
Copy link
Contributor

davidsh commented Apr 23, 2019

Besides, how to explain that if i use HTTPClientHandler, it works perfectly? i can easily share my lab to you if you like, through Microsoft corporate network.

HttpClientHandler is either SocketsHttpHandler or WinHttpHandler. By default it is SocketsHttpHandler unless you use the AppContext switch to turn that off.

Does your scenario work with SocketsHttpHandler if you remove async from your repro code?

This code is invalid:

await (WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, async () =>

RunImpersonated() is sync only. So, you can not use 'async' or 'await' modifiers in your code. If you need to call async calls within the RunImpersonated() you need to wait for them to complete with .Result or something.

If you can reproduce the impersonation problem using SocketsHttpHandler (which is the default of HttpClientHandler) when you fix the repro code by removing the 'await' and 'async' code, then there would be something to investigate as a possible bug.

The fact that this works with the invalid async repro code by disabling SocketsHttpHandler (and thus uses WinHttpHandler) might be due to race conditions on the underlying async calls completing fast enough before the RunImpersonated call finishes.

@Cynthia-Jiang
Copy link
Author

as you suggested, here is my new code, but i'm still experiencing the same issue, unless i disable SocketsHttpHandler, and use winHttpHandler instead by adding environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER as 0.

           WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, async () =>
            {
                HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, URL);

                string user_name = WindowsIdentity.GetCurrent().Name;
                result += ($"\n\nimpersonated as: {user_name}\n\n");
                try
                {
                    var response =  _client.SendAsync(requestMessage).Result;
                    var responseString = response.Content.ReadAsStringAsync().Result;

                    result += ($"Called API as: {WindowsIdentity.GetCurrent().Name}.\n\n" +
                           $"{responseString}\n");
                }
                catch (Exception ex)
                {
                    result += ex.ToString();
                }
            });

this is the expcetion thrown, and through further debugging, it's the same callstack with the initial one.

System.AggregateException: One or more errors occurred. (This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server) ---> System.Net.Http.HttpRequestException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server ---> System.Net.Sockets.SocketException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at WebApplication2.Controllers.PassThroughController.<>c__DisplayClass5_0.<<Get>b__0>d.MoveNext() in C:\Users\jiangya\source\repos\WebApplication2\WebApplication2\Controllers\PassThroughController.cs:line 92
---> (Inner Exception #0) System.Net.Http.HttpRequestException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server ---> System.Net.Sockets.SocketException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)<---

@davidsh
Copy link
Contributor

davidsh commented Apr 24, 2019

WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, async () =>

Can you remove the 'async' keyword from this code? Using the 'async' keyword will still result in the body of the method potentially running async.

@davidsh davidsh reopened this Apr 24, 2019
@Cynthia-Jiang
Copy link
Author

This is the new code

WindowsIdentity.RunImpersonated(windowsIdentity.AccessToken, () =>
            {
                HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, URL);

                string user_name = WindowsIdentity.GetCurrent().Name;
                result += ($"\n\nimpersonated as: {user_name}\n\n");
                try
                {
                    var response =  _client.SendAsync(requestMessage).Result;
                    var responseString = response.Content.ReadAsStringAsync().Result;

                    result += ($"Called API as: {WindowsIdentity.GetCurrent().Name}.\n\n" +
                           $"{responseString}\n");
                }
                catch (Exception ex)
                {
                    result += ex.ToString();
                }
            });

and i'm still getting the same error. the workaround still works.

@davidsh
Copy link
Contributor

davidsh commented Apr 25, 2019

i'm still getting the same error.

@Tratcher Any thoughts? I thought impersonation worked ok as long as it was synchronous code?

@Tratcher
Copy link
Member

More likely we've ruled out the async impersonation as the cause of the error. I just wanted to make sure before we looked deeper.

This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server
That sounds suspiciously like a DNS error, but not one I've ever seen before.
Here's an example of that error coming from Dns.GetHostAddresses, which is likely the same API ConnectAsync is using:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/de8bdd3f-2535-4a57-9574-98730914392d/host-name-resolution?forum=netfxbcl
WinHttpHandler would not be (directly) using the same DNS API.

Do you still get this error if you completely remove RunImpersonated?

@LeroyPakade
Copy link

The error on my side is only with RunImpersonated, but its only effecting some people in my domain

@Cynthia-Jiang
Copy link
Author

this is the callstack i got while debugging:

Interop+Winsock.GetAddrInfoExW(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
Interop+Winsock.GetAddrInfoExW(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
DomainBoundILStubClass.IL_STUB_PInvoke(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)
System.Net.NameResolutionPal.GetAddrInfoAsync(System.Net.DnsResolveAsyncResult) 
System.Net.Dns.HostResolutionBeginHelper(System.String, Boolean, Boolean, System.AsyncCallback, System.Object) 
System.Net.Dns.BeginGetHostAddresses(System.String, System.AsyncCallback, System.Object) 
System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(System.Net.Sockets.SocketAsyncEventArgs, System.Net.DnsEndPoint) 
000000e588c7c390 00007ffc7d598f23 System.Net.Sockets.Socket.ConnectAsync
........

and i found inner exception is access deny error when opening thread token:


0:001> k
 # Child-SP          RetAddr           Call Site
00 000000e5`88c7bbe8 00007ffc`a5a5ef5d KERNELBASE!OpenThreadToken  
01 000000e5`88c7bbf0 00007ffc`a5a5ec39 ws2_32!WinsockThreadpool_CreateWorkContext+0x91  
02 000000e5`88c7bc30 00007ffc`a5a5722c ws2_32!WinsockThreadpool_SubmitWork+0x45  
03 000000e5`88c7bc80 00007ffc`a5a56814 ws2_32!NSSUBJOB::TrySubmit+0x13c  
04 000000e5`88c7bce0 00007ffc`a5a55f06 ws2_32!NSJOB::SubmitChild+0x44  
05 000000e5`88c7bd30 00007ffc`a5a56329 ws2_32!NSJOB::ForEachChild+0xd6  
06 (Inline Function) --------`-------- ws2_32!NSJOB::ForEachChild+0x12  
07 000000e5`88c7bd80 00007ffc`a5a55913 ws2_32!NSJOB::TrySubmit+0x89  
08 000000e5`88c7bde0 00007ffc`a5a6145c ws2_32!NSMASTERJOB::TrySubmit+0x53  
09 000000e5`88c7be20 00007ffc`210753c4 ws2_32!GetAddrInfoExW+0x12fc  
0a 000000e5`88c7c060 00007ffc`8037128f System_Runtime_Extensions!DomainBoundILStubClass.IL_STUB_PInvoke(System.String, System.String, Int32, IntPtr, System.Net.Sockets.AddressInfoEx ByRef, System.Net.Sockets.AddressInfoEx* ByRef, IntPtr, System.Threading.NativeOverlapped ByRef, LPLOOKUPSERVICE_COMPLETION_ROUTINE, IntPtr ByRef)+0x154
0b 000000e5`88c7c068 00000000`00000000 coreclr!MarshalNative::GCHandleInternalFree+0x3cf

0:001> !gle
LastErrorValue: (Win32) 0x5 (5) - Access is denied.
LastStatusValue: (NTSTATUS) 0xc0000022 - {Access Denied}  A process has requested access to an object, but has not been granted those access rights.

and, if you need a lab to do debugging, i can share mine to you through Microsoft corporate network.

@LeroyPakade
Copy link

LeroyPakade commented Apr 25, 2019

I still get the DNS error for some users , environmentVariable name="DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER" value="0" doesnt work for me . WinHttpHandler works for me but than i need the password for the NetworkCredential to be able to do an authenticated call , i previuous used UseDefaultCredentials with HttpClientHandler

@Cynthia-Jiang
Copy link
Author

I still get the DNS error for some users , environmentVariable name="DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER" value="0" doesnt work for me . WinHttpHandler works for me but than i need the password for the NetworkCredential to be able to do an authenticated call , i previuous used UseDefaultCredentials with HttpClientHandler

i don't understand. if you set the environment variable, then you should be using WinHttpHandler, like davidsh explained above.

@LeroyPakade
Copy link

when i set that in the config it still has the error only when i change HttpClient to use WinHttpHandler than it works but than i have a problem with credentials

@LeroyPakade
Copy link

Its working now with WinHttpHandler

@satyendrasinghpanwar
Copy link

I am also getting same error. I am getting this error in both version Net Core 2.1 and 2.2 . SocketHttpHandler is unable to resolve host if we use it with RunImpersonated. Strangely, it is working perfectly in Win 7 machine but failing in Win 10 machine

@Tratcher
Copy link
Member

Can you try a scoped repro that calls Dns rather than HttpClient inside RunImpersonated? I think SocketHttpHandler is using Dns.BeginGetHostAddresses. You could also try GetHostAddresses.

@davidsh
Copy link
Contributor

davidsh commented Apr 26, 2019

"This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server"
That sounds suspiciously like a DNS error, but not one I've ever seen before.

I have seen this error before. It usually happens when trying to forward resolve a DNS name but then the reverse DNS resolution is broken. We've had issues before with the how the System.Net Dns class implements GetHostAddresses() because there are cases when it does forward and reverse lookups before returning the results. I'm not sure if this is happening in this repro. And I do think that this DNS problem should be reduced down to try and isolate it.

@satyendrasinghpanwar
Copy link

@Tratcher , I tried running Dns.GetHostAddress inside RunImpersonated. It is returning correct IP address. But with HttpClient I am getting error.

@Tratcher
Copy link
Member

Interesting, I wonder if BeginGetHostAddresses is any different.

@satyendrasinghpanwar
Copy link

BeginGetHostAddresses doesn't work with impersonation.
GetHostAddresses internally call NameResolutionPal.TryGetAddrInfo, while BeginGetHostAddresses calls NameResolutionPal.GetAddrInfoAsync.
Both method have different way for resolving host.
TryGetAddrInfo uses Interop.Winsock.GetAddrInfoW for resolving host
GetAddrInfoAsync uses Interop.Winsock.GetAddrInfoExW for resolving host which always returns "WSATRY_AGAIN" as error code.
Surely, their is some problem with Dns.BeginGetHostAddress

@davidsh
Copy link
Contributor

davidsh commented Apr 30, 2019

I agree this difference needs to be investigated. We should be using the same underlying Windows Winsock APIs for of the different overloads of the *GetHostAddresses* methods.

@wfurt

@davidsh
Copy link
Contributor

davidsh commented May 1, 2019

Is it possible this PR dotnet/corefx#37327 will fix this?

@stephentoub

@michaelschmit
Copy link

I'm still running into this issue on 3.0 preview 5. I notice the linked PR is not going to be merged due to failed checks. Any chance this is going to be fixed?

@davidsh
Copy link
Contributor

davidsh commented May 29, 2019

I notice the linked PR is not going to be merged due to failed checks.

Is it possible this PR dotnet/corefx#37327 will fix this?

Sorry. It appears I used the wrong PR number in the link above. That link isn't related to this issue at all.

And this issue has not been resolved in .NET Core 3.0 branches yet. It still needs more investigation. But https://github.com/dotnet/corefx/issues/37120#issuecomment-487868688 points to a possible area that might be causing the problem.

@MarcoRossignoli
Copy link
Member

I did some more tests and seem related to usage with impersonation of different user(also on same machine, no domain)

.NET Core SDK (reflecting any global.json):
Version: 3.0.100-preview5-011568
Commit: b487ff10aa

class Program
    {
        [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
        public static extern bool LogonUser(String lpszUsername, String lpszDomain, String lpszPassword, int dwLogonType, int dwLogonProvider, out SafeAccessTokenHandle phToken);

        const int LOGON32_PROVIDER_DEFAULT = 0;
        const int LOGON32_LOGON_INTERACTIVE = 2;


        static void Main(string[] args)
        {
            SafeAccessTokenHandle tokenin;
            bool returnValue = false;

            Console.WriteLine("Current User");
            returnValue = LogonUser("marco", "xxxxx", "xxxxx", LOGON32_LOGON_INTERACTIVE, LOGON32_PROVIDER_DEFAULT, out tokenin);
            Debug.Assert(returnValue);
            Run(tokenin);
            tokenin.Dispose();
            Console.WriteLine("-------");

            Console.WriteLine("Another User");
            returnValue = LogonUser("anotheruser", "xxxx", "xxxxx", LOGON32_LOGON_INTERACTIVE, LOGON32_PROVIDER_DEFAULT, out tokenin);
            Debug.Assert(returnValue);
            Run(tokenin);
            tokenin.Dispose();
            Console.WriteLine("-------");

            return;

            static void Run(SafeAccessTokenHandle token)
            {
                string host = "www.google.it";
                WindowsIdentity.RunImpersonated(token, () =>
                {
                    Console.WriteLine($"{WindowsIdentity.GetCurrent().Name} {WindowsIdentity.GetCurrent().ImpersonationLevel}");
                    Console.WriteLine("Dns.GetHostAddresses(host) " + Dns.GetHostAddresses(host)[0].ToString());
                    Console.WriteLine("Dns.GetHostAddressesAsync(host) " + Dns.GetHostAddressesAsync(host).Result[0].ToString());
                });
            }

        }
    }
...
Current User
xxx\Marco Impersonation
Dns.GetHostAddresses(host) 216.58.198.3
Dns.GetHostAddressesAsync(host) 216.58.198.3
-------
Another User
xxx\AnotherUser Impersonation
Dns.GetHostAddresses(host) 216.58.198.3

Unhandled Exception: System.AggregateException: One or more errors occurred. (This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.) ---> System.Net.Sockets.SocketException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
   at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
   at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean w
0:008> k
 # Child-SP          RetAddr           Call Site
00 00000003`311feba0 00007ffd`b5bb6ad8 KERNELBASE!RaiseException+0x69
01 00000003`311fec80 00007ffd`b5bb63c3 coreclr!RaiseTheExceptionInternalOnly+0x28c [f:\workspace\_work\1\s\src\vm\excep.cpp @ 2974] 
02 00000003`311feda0 00007ffd`b28e4251 coreclr!IL_Throw+0x113 [f:\workspace\_work\1\s\src\vm\jithelpers.cpp @ 4932] 
03 00000003`311fef40 00007ffd`b28e4276 System_Private_CoreLib!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()$##6003017+0x21
04 00000003`311fef70 00007ffe`0e24bc1e System_Private_CoreLib!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(System.Exception)$##6003018+0x16
05 00000003`311fefa0 00007ffe`0e24bd13 System_Net_NameResolution!System.Net.Dns.HostResolutionEndHelper(System.IAsyncResult)$##6000028+0x17e
06 00000003`311fefe0 00007ffe`0e24e92e System_Net_NameResolution!System.Net.Dns.EndGetHostAddresses(System.IAsyncResult)$##6000032+0x33
07 00000003`311ff020 00007ffd`b28a5519 System_Net_NameResolution!System.Net.Dns+<>c.<GetHostAddressesAsync>b__25_1(System.IAsyncResult)$##600003D+0xe
08 00000003`311ff050 00007ffd`b28a7ce5 System_Private_CoreLib!System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+<>c.<ProcessAsyncIfNecessary>b__40_1(System.Object)$##6002061+0x1f89
09 00000003`311ff0a0 00007ffe`0e24c598 System_Private_CoreLib!System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+<>c.<ProcessAsyncIfNecessary>b__40_1(System.Object)$##6002061+0x4755
0a 00000003`311ff0f0 00007ffe`0e24d010 System_Net_NameResolution!System.Net.LazyAsyncResult.Complete(IntPtr)$##600007B+0x78
0b 00000003`311ff160 00007ffe`0e24d299 System_Net_NameResolution!System.Net.ContextAwareResult.CompleteCallback()$##6000067+0x30
0c 00000003`311ff190 00007ffd`b28959d0 System_Net_NameResolution!System.Net.ContextAwareResult+<>c.<Complete>b__15_0(System.Object)$##600006C+0x19
0d 00000003`311ff1c0 00007ffd`b2895940 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6001E7C+0x80
0e 00000003`311ff230 00007ffe`0e24cf24 System_Private_CoreLib!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6001E7B+0x10
0f 00000003`311ff260 00007ffe`0e24c47b System_Net_NameResolution!System.Net.ContextAwareResult.Complete(IntPtr)$##6000066+0xa4
10 00000003`311ff2b0 00007ffe`0e24d272 System_Net_NameResolution!System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)$##6000079+0xbb
11 00000003`311ff300 00007ffd`b28ab37f System_Net_NameResolution!System.Net.NameResolutionPal+GetAddrInfoExState+<>c.<CompleteAsyncResult>b__5_0(System.Object)$##600009B+0x22
12 00000003`311ff330 00007ffd`b28b0af2 System_Private_CoreLib!System.Threading.Tasks.Task.InnerInvoke()$##6002171+0x3f
13 00000003`311ff360 00007ffd`b2895b92 System_Private_CoreLib!System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)$##6002207+0x12
14 00000003`311ff390 00007ffd`b28ab1b2 System_Private_CoreLib!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6001E7E+0x42
15 00000003`311ff3e0 00007ffd`b28aaf93 System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)$##6002170+0x1c2
16 00000003`311ff480 00007ffd`b28aaf2b System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)$##600216E+0x53
17 00000003`311ff4c0 00007ffd`b289ee1c System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)$##600216D+0xb
18 00000003`311ff4f0 00007ffd`b288dddb System_Private_CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch()$##6001FBE+0x1bc
19 00000003`311ff5a0 00007ffd`b5c09953 System_Private_CoreLib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()$##6001D41+0xb
1a 00000003`311ff5d0 00007ffd`b5b1b2bb coreclr!CallDescrWorkerInternal+0x83 [F:\workspace\_work\1\s\src\vm\amd64\CallDescrWorkerAMD64.asm @ 101] 
1b (Inline Function) --------`-------- coreclr!CallDescrWorkerWithHandler+0x52 [f:\workspace\_work\1\s\src\vm\callhelpers.cpp @ 70] 
1c 00000003`311ff610 00007ffd`b5b1c558 coreclr!MethodDescCallSite::CallTargetWorker+0x1a7 [f:\workspace\_work\1\s\src\vm\callhelpers.cpp @ 612] 
1d (Inline Function) --------`-------- coreclr!MethodDescCallSite::Call_RetBool+0x1a [f:\workspace\_work\1\s\src\vm\callhelpers.h @ 477] 
1e 00000003`311ff740 00007ffd`b5ae21c1 coreclr!QueueUserWorkItemManagedCallback+0x38 [f:\workspace\_work\1\s\src\vm\comthreadpool.cpp @ 470] 
1f (Inline Function) --------`-------- coreclr!ManagedThreadBase_DispatchInner+0xd [f:\workspace\_work\1\s\src\vm\threads.cpp @ 7404] 
20 00000003`311ff830 00007ffd`b5ae20ab coreclr!ManagedThreadBase_DispatchMiddle+0x81 [f:\workspace\_work\1\s\src\vm\threads.cpp @ 7448] 
21 00000003`311ff940 00007ffd`b5ae1f87 coreclr!ManagedThreadBase_DispatchOuter+0xaf [f:\workspace\_work\1\s\src\vm\threads.cpp @ 7613] 
22 (Inline Function) --------`-------- coreclr!ManagedThreadBase_FullTransition+0x1f [f:\workspace\_work\1\s\src\vm\threads.cpp @ 7660] 
23 (Inline Function) --------`-------- coreclr!ManagedThreadBase::ThreadPool+0x1f [f:\workspace\_work\1\s\src\vm\threads.cpp @ 7702] 
24 00000003`311ff9e0 00007ffd`b5b0c8fa coreclr!ManagedPerAppDomainTPCount::DispatchWorkItem+0x77 [f:\workspace\_work\1\s\src\vm\threadpoolrequest.cpp @ 643] 
25 (Inline Function) --------`-------- coreclr!ThreadpoolMgr::ExecuteWorkRequest+0x3f [f:\workspace\_work\1\s\src\vm\win32threadpool.cpp @ 1556] 
26 00000003`311ffa40 00007ffd`b5bf58ba coreclr!ThreadpoolMgr::WorkerThreadStart+0x18a [f:\workspace\_work\1\s\src\vm\win32threadpool.cpp @ 2012] 
27 00000003`311ffc30 00007ffe`26937974 coreclr!Thread::intermediateThreadProc+0x8a [f:\workspace\_work\1\s\src\vm\threads.cpp @ 2103] 
28 00000003`311ffd70 00007ffe`2815a271 kernel32!BaseThreadInitThunk+0x14
29 00000003`311ffda0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:008> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc000007c - An attempt was made to reference a token that doesn't exist.  This is typically done by referencing the token associated with a thread when the thread is not impersonating a client.

.... We should be using the same underlying Windows Winsock APIs for of the different overloads of the GetHostAddresses methods.

For async scenario GetAddrInfoExW is used because supports async operations GetAddrInfoW doesn't.

@davidsh
Copy link
Contributor

davidsh commented Jun 18, 2019

@MarcoRossignoli Thanks for the additional repro case!

I have done further analysis and opened up a new issue, dotnet/corefx#38646 that is the root cause of this issue.

@4imble
Copy link

4imble commented Nov 17, 2020

Just to add further information to this, we are were having the same problem for some of our users. We narrowed it down to users that are local admins on the web server are fine but other users will fail unless we turn off the SocketsHttpHandler as was suggested by OP.

@ghost ghost locked as resolved and limited conversation to collaborators Dec 17, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
blocked Issue/PR is blocked on something - see comments bug
Projects
None yet
Development

No branches or pull requests

10 participants