Execution Timeout Expired Error (258, ReadSniSyncOverAsync) #647
Comments
|
Have you tried increasing the command timeout? |
|
Thanks, @ericstj, No, I didn't try increasing command timeout. Currently, the command timeout is 3s. If a query executes timeout, I got stack like below, seems it is different than the above stack, do you mean the above stack is also a SQL execution timeout? By the way, from New relic, the select query's duration is 83.4ms, I'm not sure if it is the execution time. |
|
@frankyuan, thanks for bringing this up to our attention, can you please let me know if:
Thank you |
cheenamalhotra
moved this from Needs triage
to Waiting for customer
in SqlClient Triage Board
|
Sorry for reply delay, here is more detail information(by the way, not every cluster have above error, just in one cluster):
|
|
@frankyuan thanks for the response. Can you provide sample repro code? If sample repro is not possible (without sample app it is really difficult to understand what is happening), can you share the connection string properties please? |
|
SqlConnectionString like this: Data Source=xxx;Initial Catalog=xx;User ID=xx;Password=xx;Min Pool Size=5;Max Pool Size=100;Connect Timeout=8. CommandTimeout=3. As I can't reproduce in my local environment, I'm afraid can't provide sample repro for now. But I'm trying to reproduce, if find a method, will update here. |
|
Seems only the SQL mentioned in description will throw the exception. And sometimes there is another similar exception like below when executing the same SQL. |
|
@frankyuan any update on the issue on your side? Were you able to repro the issue? If not, can you kindly share your code or a similar code with me? I need the query you run and setting you have on your machine please. On the other note, can you test it with increasing Connection and command time out as well? |
|
I'm experiencing a similar issue
.NET target: Core 3.1 |
|
We are also having similar issue like @Svisstack . Microsoft.Data.SqlClient 2.0.1, .NET Core 3.1, running in Windows container and connecting to SQL Azure Hyperscale DB. We have a multi-threaded service, lets say about 30 parallel threads that are writing data to the same database. There is a transaction per thread. The service is running fine for most of the time, but from time to time, some threads start to get the exceptions below and they are stuck on it. The command that is failing on this timeout is the same for all threads, but in each occurrence it's usually a different command. Quite often it's a command that is sending some table valued parameters, but we saw it to fail on a very basic SELECT with a single parameter as well. There is no activity on the DB side, connections are in awaiting command state. From our observations it seems that when this exception is thrown, all threads are blocked on it. They never recovered from it. When we noticed this exception, it seems that only some threads were affected and after approximately 30 minutes the issue disappeared. I would like to provide a repro, but it's quite hard to reproduce that nondeterministic behavior. Interesting is that we have 6 instances of this service, where each instance is connecting to different DB and has different number of threads, but we noticed the issue only on 2 instances. One of them is the above mentioned one with 30-35 threads, another one is way smaller, only 3-4 threads. |
Sorry for getting back to you late, I couldn't reproduce this issue currently, and the only clue is that there are more than 3000 records in the result, after I changed the logic to get fewer records, the exception is gone. |
|
@frankyuan thanks for the update. @Svisstack, @lukas-navratil can any of you guys provide a sample repro application please? |
|
I dont have this possibility unfortunately.
…Sent from my iPhone
On 25 Sep 2020, at 19:27, Javad ***@***.***> wrote:
@frankyuan thanks for the update.
@Svisstack, @lukas-navratil can any of you guys provide a sample repro application please?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
We are having the same issue, it happens randomly on our Azure SQL Server instance. Usually the query is very fast and sometimes it raises the error: |
|
@JRahnama we are seeing the same issues, we will keep an eye on it an see if we can replicate it at will. At the momement is very random:
|
|
@twurm can you give us a repo application? |
|
@JRahnama we are working on a repo app. The event doesn't happen all the time and happens in different areas of the application so we are trying to figure that out. |
|
same experiences
…On Mon, Sep 28, 2020 at 11:14 PM Tom Wurm ***@***.***> wrote:
@JRahnama <https://github.com/JRahnama> we are working on a repo app. The
event doesn't happen all the time and happens in different areas of the
application so we are trying to figure that out.
|
|
@frankyuan, @twurm, @mischaherbrand, @Svisstack, @lukas-navratil while I was trying to make a scenario to reproduce the issue Thanks everybody |
|
Cant confirm that currently, too much code on my side, but will have this in mind
…Sent from my iPhone
On 16 Oct 2020, at 23:56, Javad ***@***.***> wrote:
@frankyuan, @twurm, @mischaherbrand, @Svisstack, @lukas-navratil while I was trying to make a scenario to reproduce the issue
, I came up with a sample code the each first try to read from the pool was throwing the very same error message while the SqlCommand was not wrapped in a using block, but the rest of the recycled connection worked fine until the pool was renewed by reaching its max poolsize or a fatal error happened and pool got cleared and then the first connection out of the pool had the very same issue again. As a workaround can you make sure all your SqlConnection, SqlCommands and SqlDataReaders are wrapped in a using block properly and see if the issue comes up? That will also help me to understand if I am on the right track with the repro.
Thanks everybody
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@JRahnama , If I didn't miss something, all |
|
@JRahnama , I found one command that wasn't wrapped in using block. I will deploy the fix and try to monitor if it helped or not. |
|
@JRahnama , so it seems that since we deployed 8 days ago the version with added using around SqlCommand, the Win32Exception 258 hasn't been thrown. However exception 121 was thrown several times. I can't tell whether it would recover from that as we implemented automatic restart of service when it's stuck on the same SqlException for too long. I'm not sure either that it's related to this issue, but it's just a heads up that this error still happens in our environment as I reported it above. |
|
@JRahnama, unfortunately, our app got stuck today on this issue again: |
|
I am getting this as well. In my case its from within EFC DbContext. ASP.NET Core 3.1 (and its bundled EFC and SqlClient). I also use the EFC's DB Context Pool: One of my colleagues insisted this is comming from SQL login timeouts (you can use that as a hint but I would not rely on that) My connection string: |
|
@lcheunglci |
|
When will this major years old linux issue be fixed? Using MS-SQL Docker on Linux fells like MySQL in the nineties. |
|
My platform After optimizing my API and implementing 'async' and 'await' with 'Task', all the errors have disappeared. |
|
Could you clarify what do you mean here: "implementing 'asyn'c and 'await' with 'Task'" ? |
just i make my api asyncronized. Yes it was synchronized. |
|
I had a very similar problem recently: In my case I have a dockerized webapi where I start a new Not creating a The exception was the same @frankyuan's error and appeared intermittently in logs, due to application usage and behaviour: ....which was not a super helpful error message - while trying to figure this out 👎 |
Were you able to resolve this? |
|
Minimizing Task.Run(...) calls did not solve the problem in my case. Looks like this is a regular threading bug. Read the other day of a SNI threading bugfix in a ef core preview version (can't find it right now). Hope this will solve this leaking problem soon. |
|
Still receiving this error quite often on .NET 7 running in a Linux (Debian) container. Any update on what might be the cause? |
|
@Malgefor Convert all asynchronous SQL calls into synchronous ones, and everything will be sorted :) |
|
We are also experiencing this issue with Linux App Services against Azure SQL. We only began seeing this after migrating our App Services from Windows to Linux. Also on 5.1.1 of SqlClient. |
|
I spent a few hours fighting this error. In my case, it was really just a slow transaction that timed out (I use the default CommandTimeout). My app runs in a dotnet 6.0-bullseye-slim Docker container. I use NServiceBus to handle incoming messages, up to 25 concurrently. When a message arrives, I use EF Core 7.0 to fetch a record from the database using The stacktrace doesn't always look the same. Here is another example without A side-effect of this Unknown Error seems to be that the connection is NOT (always?) returned to the connection pool. I think this has also been discussed previously as a side-effect of using sync-over-async, but I don't understand the machinery that leads to the connection remaining open but unavailable for reuse. Connections not being returned to the pool causes my application to come to almost a complete stop, with 100 open connections left unused. I have confirmed on SQL Server that my application has 100 active sessions, yet my message handlers are all crashing with a timeout due to connection pool starvation. In the end, I just ended up rewriting my queries and adding indexes, so the timeouts disappeared. But it still seems to me that there is something not really working right in the SqlClient on Linux. |
|
This issue has been open for over 3 years and I think that there are probably a lot of people who would be really appreciative of an update on this, even if it is just to say this isn't getting fixed any time soon. We experienced this issue about a year ago. We ended up shifting our hosting to Windows, at additional cost, to be sure that we wouldn't encounter it again. I came back here a year later expecting (well probably more like hoping) that it would have been resolved, or at least for some sort of update. IMO, this is a massive issue for a huge number of teams wanting to host on Linux. I'd probably go as far as saying that you shouldn't really be hosting on Linux in a production environment if you are using SqlClient. That's a lot of people and a really big deal. As far as I can tell, any resolutions anyone has above are just workarounds. They are not really fixing the issue. If I'm wrong, please let me know. Of the above workarounds, is there one that will totally prevent the issue from occurring, please? Thanks in advance for any feedback. |
|
I would like to share my experience on this. I used to get exactly the same exception/stack trace when running a query. It was odd because running the execution plan it always showed super optimized, it seemed not a performance problem, but often the exception occurred. We had no idea what to do. But since a time ago, we are rebuilding indexes every week, and exception was not raised anymore. My conclusion is, sometimes SQL server was doing something, or some overload in the SQL server but for any reason refreshing indexes reduces these overloads. |
|
For us, increasing the threadpool threads eliminated this issue. We're still on linux and are not seeing this (anymore) |
|
We also started seeing unknown error 258 after migrating from Windows to Linux App Services, but strangely only on one of our environments. @MichelZ Out of curiosity, what did you increase the minimum threads to? |
We went aggressive/overkill, but we haven't seen negative side effects currently. |
Thought I'd add that when we investigated this a year ago, we felt that analysis from @bennil, above, married up with what we were seeing. Not sure I'm comfortable with your solution @MichelZ even though you haven't suffered any negative consequences. Great it is working for you though. |
|
Thanks @co7e for bringing this matter up again. My insight after watching/analyzing this for years are the following: SqlClient is not thread safe on Linux (saw it on Intel and AMD). Reducing parallel work or adjust thread execution might help but not solve the problem. Sooner or later the exception will hit you. Surprisingly Microsoft doesn't care for years even though a huge number of enterprise software is running on this db access lottery. The bug is hard to reproduce as all threading issues are. I have some ideas how to construct testcode for further investigations, but this is not a charity project... |
Your stack trace doesn't contain |
|
In our case the problem has now as mysteriously disappeared as it has mysteriously appeared. The only change we made was migrate to a different virtual machine than the previous SQL Server was running on. By taking some packet captures by logging in with ssh on the Linux Web App, and typing A packet containing a SQL query that in the tcpdump was registered with size 2130 bytes and data length 2064 was being "partially acked". After sending the large packet out, the ACK came back only for the first 1398 bytes, causing a retransmission of the last 666 bytes. However, in response to this retransmission only came another ACK for only the first 1398 bytes, causing another retransmission of the last 666 bytes, and so on and so forth, until eventually the connection timed out. Rather than a threading issue this potentially seems like some kind of network layer problem. However, we can no longer reproduce it, so it's difficult to investigate more deeply. As @dazinator already suggested our particular problem was probably something entirely different than what this ticket is about. The 258 timeout error is fairly general - it just means that some kind of wait operation timed out and it can have multiple different underlying causes. Sorry for the noise. |
We have also experienced the same issue. The solution, for us, was to limit the packet size in the Connection string by appending See https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.packetsize?view=dotnet-plat-ext-8.0 for more info on SqlConnection.PacketSize property. |
|
https://www.codeproject.com/Tips/5378079/Challenges-in-Migrating-ASP-NET-Apps-to-Containers Maybe after all it relates to MARS? (DB connection string: MultipleActiveResultSets=True) |
Not in mine or many other cases. The MARS issue was seperate as far as I know. I use Note: After moving the work to a seperate dedicated process (and therefore removing contention of the application thread pool in the process) I no longer see this issue. The new process starts - on the same machine, runs the same queries, and terminates as expected. Also, not only are we running this logic in a dedicated process, we have 10 jobs to run, rather than them all being run in application at same time as before, they are now 10 seperate dedicated processes that start then terminate. So this has resulted in 11 seperate processes, each with own thread pool and using own connection. So it seems to be to do with:-
|
|
I also wondered if Azure SQL has some sort of connection "rejection" mechanism, such that when it scales up or CPU usage is high, it can reject queries with a 258 "immediately" and that perhaps when a connection is in this state, it doesn't clear. A bit like a rate limiting feature. However this is pure speculation, I don't have the networking skills to trace what packets is being sent when this occurs, as our apps run in docker, on linux azure VM's and things get pretty complicated pretty fast. |
and others
Describe the bug
When executing SQL such as
SELECT FieldA, FieldB FROM A INNER JOIN C ON A.FieldId = C.FieldId UNION SELECT FieldA, FieldD FROM A INNER JOIN D ON A.FieldId = D.FieldId, throw the error like below, not every time, just a little part of queries have this issue.To reproduce
Sorry, currently can't reproduce in the local environment, so can't provide more detail to reproduce.
Expected behavior
SQL should execute successfully every time.
Further technical details
Microsoft.Data.SqlClient version: 1.1.3
.NET target: Core 3.1
Operating system: Docker container
What I found/tried
https://stackoverflow.com/questions/57270245/sql-server-dbcommand-timeout-with-net-core-container-under-load
DapperLib/Dapper#1435
The text was updated successfully, but these errors were encountered: