System.InvalidOperationException: Internal connection fatal error. Error state: 15, Token : 143

[chaybcr]

we have been receiving this error frequently in application when an API call happens between Application server and database server to fetch data.

while troubleshooting the issue restarted both Application and DB servers (using my account to logged into server) and there is no luck. Later we found that when logged into server using Service Account which is used as Application Pool Identity for the site resolved the issue.

Note: this issue is not happening for all API calls. only for the calls which involves fetching Encrypted data using SQL Always Encrypted method.

So, Alternative days or whenever we noticed the issue we are logging into Application server with Service account to resolve this issue.

we are using windows server 2016, IIS10.

can anyone help us why this peculiar behavior is happening?

Thanking you in Advance!!!

 at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
 at System.Data.SqlClient.SqlDataReader.TryCloseInternal(Boolean closeReader)
 at System.Data.SqlClient.SqlDataReader.Close()
 at System.Data.Common.DbDataReader.Dispose(Boolean disposing)
 at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.Finally()
 at System.Data.Entity.Internal.LazyEnumerator`1.Dispose()
 at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
 at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

[JRahnama]

@chaybcr thanks for bringing this up to our attention. issue #672 is looking like a similar issue with the very same stack. Can you provide us more detail such as what version of Microsoft.Data.SqlClient you are using and are you using Net Core or .NETFramework application plus can you please provide us a sample repro?

[chaybcr]

@JRahnama

we are using EntityFramework 6.0.0
.Net Framework - 4.5.2
SQL Server 2016
connection string:

add name="AppDBConnection" providerName="System.Data.SqlClient" connectionString="Data Source=ServerName;Initial Catalog=DBname;User Id=userID; Password=password;Column Encryption Setting=enabled"

we are using the similar Queries for all the methods...but this error occurs only when fetching the data from the tables which were encrypted using SQL Server Always Encrypted method with Windows certificate store- Current user . the certificate was imported in the Application server which uses its private key to decrypt the data

when this issue occurred we try to close all the Active connections in SQl server and even restarted the database server....when we logged into the Application server with Service Account, we are able to get the response from those API calls which fetches Encrypted data.

[cheenamalhotra]

@chaybcr

Could you wrap up a repro app that consistently reproduces this error?
That way we can reproduce, investigate and fix the issue faster.

[jsheetzati]

We are also seeing a somewhat similar error with a slightly different message.

Environment

• .NET Core 3.1
• EFCore 3.1.10

Stack trace

An exception occurred while iterating over the results of a query for context type 'Application.Services.App.Context'.
System.InvalidOperationException: Internal connection fatal error.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__164_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

[moxixuan]

We are also seeing a somewhat similar error with a slightly different message.

it will happend sometime but not always

i have multi-threaded misuse of connection

Environment

.NET Core 3.1
Dapper 2.0.0.0

Stack trace

at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreResults(Boolean& moreResults)
at Microsoft.Data.SqlClient.SqlDataReader.TryNextResult(Boolean& more)
at Microsoft.Data.SqlClient.SqlDataReader.NextResultAsyncExecute(Task task, Object state)
at Microsoft.Data.SqlClient.SqlDataReader.InvokeAsyncCall[T](AAsyncCallContext`1 context)
--- End of stack trace from previous location where exception was thrown ---
at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 441

Code

                await using var db = (Microsoft.Data.SqlClient.SqlConnection)DbContextFactory.GetDataSource(CompanyCode).ConnectionFacotry();
                var sql = $@"SELECT Goods, Type FROM [sync].[tbTaskUploadGoodsELMv2] (NOLOCK) WHERE TaskId = @TaskId";
                var eLMv2s = await db.QueryAsync<TaskUploadGoodsELMv2>(sql, new { TaskId }, commandTimeout: 120);

SQL Server 2016
connection string:
Server=192.168.1.12;Initial Catalog=fef;Persist Security Info=False;User ID=sa;Password=222;MultipleActiveResultSets=True;Encrypt=False;TrustServerCertificate=False;pooling=true;min pool size=5;max pool size=512;Connection Timeout=30;

[mpishchaev]

Just in case check your *.csproj for InvariantGlobalization attribute.
When I removed this attribute the error fade.