Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix unhandled exception on k8s watcher background task #7168

Merged
merged 3 commits into from
Aug 5, 2021

Conversation

hami89
Copy link
Contributor

@hami89 hami89 commented Jul 23, 2021

Fix unawaited background task to suppress any exception from the ChannelReader's Completion task.

The channel reader's Completion task would re-raise the exception set on the channel writer and cause an unhandled exception in the unawaited task and crash silo.

We have witnessed multiple cases when the HTTPS connection between the k8s watcher and the k8s API was lost, and it caused a silo crash. The investigation led to this unawaited background task. (See attached exception.)

[12:04:14 FTL] An unhandled exception occurred during the execution of ***-silo on ******.svc.cluster.local <s:******.Platform.Hosting.UnhandledExceptionHandler>
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Unable to read data from the transport connection: Connection reset by peer.)
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   at k8s.WatcherDelegatingHandler.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal()
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
   at Orleans.Hosting.Kubernetes.KubernetesClientExtensions.<>c__DisplayClass0_0`2.<<WatchAsync>b__4>d.MoveNext()
   --- End of inner exception stack trace ---

…ted Completion task.

The channel reader's Completion task would re-raise the exception set on the channel writer and cause an unhandled exception and crash silo.
@dnfadmin
Copy link

dnfadmin commented Jul 23, 2021

CLA assistant check
All CLA requirements met.

@ReubenBond ReubenBond merged commit 11b5f88 into dotnet:main Aug 5, 2021
@ReubenBond
Copy link
Member

Apologies for the delay @hami89 and thank you for the contribution!

ReubenBond pushed a commit to ReubenBond/orleans that referenced this pull request Aug 5, 2021
ReubenBond added a commit that referenced this pull request Aug 5, 2021
Co-authored-by: Gergely Hámos <hami89@gmail.com>
Co-authored-by: ghamos <ghamos@itron.com>
ReubenBond added a commit that referenced this pull request Sep 22, 2021
Co-authored-by: Gergely Hámos <hami89@gmail.com>
Co-authored-by: ghamos <ghamos@itron.com>
@github-actions github-actions bot locked and limited conversation to collaborators Dec 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants