Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EndOfStreamException in MySqlSession.ConnectAsync #388

Closed
bgrainger opened this issue Nov 3, 2017 · 11 comments
Closed

EndOfStreamException in MySqlSession.ConnectAsync #388

bgrainger opened this issue Nov 3, 2017 · 11 comments

Comments

@bgrainger
Copy link
Member

@bgrainger bgrainger commented Nov 3, 2017

@GomesNayagam writes:
can you reopen this issue @bgrainger
i am getting the same exception "Attempted to read past the end of the stream" now with 0.28.0 below are the call stack from dot net core library

"StackTrace":"   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.Protocol.Serialization.ProtocolUtility.DoReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   at MySql.Data.Protocol.Serialization.StandardPayloadHandler.ReadPayloadAsync(ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   at MySql.Data.Serialization.MySqlSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.Serialization.MySqlSession.TryAsyncContinuation(Task`1 task)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at MySql.Data.Serialization.MySqlSession.<ConnectAsync>d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.ConnectionPool.<GetSessionAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__61.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at Dapper.SqlMapper.<QueryImpl>d__124`1.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType)
   at Dapper.FastCrud.SqlStatements.GenericSqlStatements`1.BatchSelect(IDbConnection connection, AggregatedSqlStatementOptions`1 statementOptions)
   at Warp.Foundation.Application.Explorer.Dapper.Explorer`1.Query(Action`1 Query)
   at 

(split from #124)

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 3, 2017

What MySQL Server software and version are you connecting to? On what OS?
What is your connection string (with password redacted)?

@GomesNayagam
Copy link

@GomesNayagam GomesNayagam commented Nov 4, 2017

Environment: Ubuntu 16.04
DotNET core: 1.1.1
Mysql connector : MySqlConnector.0.28.0 (https://github.com/mysql-net/MySqlConnector)
Mysql version: 5.7

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 4, 2017

Judging from the exception (specifically, ReceiveReplyAsync), an incomplete packet is being received after sending the handshake response. If this is reproducible, are you able to get a packet capture with tcpdump, Wireshark, or similar? You can email that to the address on my profile.

@GomesNayagam
Copy link

@GomesNayagam GomesNayagam commented Nov 4, 2017

I saw that too, header packet(4 bytes) is corrupted.

This issue happening on production since the load is high, so taking packet data is non trivial task for me. i will try to do some load test in local environment and see if i can reproduce and setup wireshark, mean time if you get any hint let me know.

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 4, 2017

Do you have any kind of proxy (ProxySQL, HAProxy, etc.) between your .NET Core application and MySQL Server? Are you running MySQL Server 5.7.20 or an earlier release?

Are you using SslMode=Preferred (the default) or higher? If so, does changing to SslMode=None to force a plaintext connection avoid the problem?

@GomesNayagam
Copy link

@GomesNayagam GomesNayagam commented Nov 4, 2017

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 6, 2017

Suggested troubleshooting steps:

  • disable TLS (SslMode=None) if possible
  • connect directly to MySQL Server instead of HAProxy (may require #391)

@GomesNayagam
Copy link

@GomesNayagam GomesNayagam commented Nov 9, 2017

sslmode=none completely gave weird response for me, so i am not using it. production is not accepting without haproxy.

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 10, 2017

From all the information you've provided, it seems like this is a server issue (responding with invalid data).

Would the most helpful thing be for MySqlConnector to wrap the EndOfStreamException in an MySqlException so you can catch it and handle it like any other connection failure (timeout, etc.)?

@GomesNayagam
Copy link

@GomesNayagam GomesNayagam commented Nov 11, 2017

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Nov 14, 2017

Improved in 0.31.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants