Azure DevOps: 690 test failures. Many Identity + SQL Server tests fail to connect to LocalDB

[natemcmaster]

Something about the way the EF Core tests are configured does not work in the Hosted VS2017 agent pool. It seems most of the EF Core + SQL server tests are broken.

https://dotnet.visualstudio.com/public/_build/results?buildId=8559&view=ms.vss-test-web.test-result-details

System.Data.SqlClient.SqlException : Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=44133; handshake=39; 
---- System.ComponentModel.Win32Exception : The wait operation timed out.
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.DatabaseExists(String name) in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 110
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.GetScratchDbName() in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 44
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.Utilities.SqlServerTestStore.CreateScratch(Boolean createDatabase) in /_/test/EF.Test/Utilities/SqlServerTestStore.cs:line 27
   at Microsoft.EntityFrameworkCore.Internal.LazyRef`1.get_Value()
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.ScratchDatabaseFixture.get_ConnectionString() in /_/test/EF.Test/Utilities/ScratchDatabaseFixture.cs:line 19
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.CustomPocoTest.CanCreateUserInt() in /_/test/EF.Test/CustomPocoTest.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----

cc @HaoK

[Eilon]

Tagging @ajcvickers here for FYI.

[HaoK]

@ajcvickers so the identity queries typically all take the form of something like this:

Users.FirstOrDefaultAsync(u => u.NormalizedUserName == normalizedUserName, cancellationToken);

https://github.com/aspnet/Identity/blob/master/src/EF/UserStore.cs#L254

[natemcmaster]

FYI I am disabling Identity PR validation for this repo since this issue is still unresolved.

[HaoK]

@ajcvickers is there any update on the EF side for why async queries are failing on VSTS?

[ajcvickers]

@HaoK There are some deadlock issues with the current query pipeline. The issues are pretty deep and won't realistically be fixed until 3.0. We still don't know 100% if this is the root cause of the test deadlocks, but we're working on the assumption that it is.

[natemcmaster]

Identity tests are still consistently failing in PR checks on this repo. In #6350, I'm going to completely disable Identity functional test projects while we sort this out.

[HaoK]

@ajcvickers given that these tests are working fine on helix, do we care about the fact that these are disabled on AzDo anymore?

[natemcmaster]

Don't forget that the tests also fail in 2.1 and 2.2 branches.

[ajcvickers]

@HaoK I think we care until Helix is the thing we use everywhere.

[HaoK]

Ok when is EF going to fix it? :)

[ajcvickers]

That's not the question. It's when SQL Server is going to fix it. Which is never...so.

[HaoK]

Which brings me back to my original question, are we planning on doing anything other than waiting for helix? I only ask because I was just looking over bugs assigned to me, and I wasn't sure there was anything actionable

[ajcvickers]

There are some patterns that might help:

• Spin up the connection to the database early so that LocalDb has a chance to start and get going before it is hammered
• Make sure that connection resiliency is being used.
• Potentially add additional test-level retries for when the database is not responding.

Or try running SQL Server on a different machine--this is what we plan to try with EF tests.

[natemcmaster]

@ajcvickers the tests are still disabled in PRs. I think they still run on Helix, though.

https://github.com/aspnet/AspNetCore/blob/5af8e170bc11aa189f7be0aab777b1bdfb76a431/src/Identity/EntityFrameworkCore/test/EF.Test/Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.csproj#L6-L7