Azure DevOps: 690 test failures. Many Identity + SQL Server tests fail to connect to LocalDB

[natemcmaster]

Something about the way the EF Core tests are configured does not work in the Hosted VS2017 agent pool. It seems most of the EF Core + SQL server tests are broken.

https://dotnet.visualstudio.com/public/_build/results?buildId=8559&view=ms.vss-test-web.test-result-details

System.Data.SqlClient.SqlException : Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=44133; handshake=39; 
---- System.ComponentModel.Win32Exception : The wait operation timed out.
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.DatabaseExists(String name) in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 110
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.GetScratchDbName() in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 44
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.CreateScratch(Boolean createDatabase) in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 27
   at Microsoft.EntityFrameworkCore.Internal.LazyRef`1.get_Value()
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.ScratchDatabaseFixture.get_ConnectionString() in /_/test/EF.Test/Utilities/ScratchDatabaseFixture.cs:line 19
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.CustomPocoTest.CanCreateUserInt() in /_/test/EF.Test/CustomPocoTest.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----

cc @HaoK

[Eilon]

Tagging @ajcvickers here for FYI.

[HaoK]

@ajcvickers so the identity queries typically all take the form of something like this:

Users.FirstOrDefaultAsync(u => u.NormalizedUserName == normalizedUserName, cancellationToken);

https://github.com/aspnet/Identity/blob/master/src/EF/UserStore.cs#L254

[natemcmaster]

FYI I am disabling Identity PR validation for this repo since this issue is still unresolved.

[HaoK]

@ajcvickers is there any update on the EF side for why async queries are failing on VSTS?

[ajcvickers]

@HaoK There are some deadlock issues with the current query pipeline. The issues are pretty deep and won't realistically be fixed until 3.0. We still don't know 100% if this is the root cause of the test deadlocks, but we're working on the assumption that it is.