How to debug task hang/deadlock? #3802
Description
Hi, I have a .net 6 console application which also provide REST API with asp.net core. There is a background job in this application to periodically update remote files via HTTP. The console application is long running in a docker image on Linux, however the background job may stop execution although other tasks/APIs could run as usual.
After a dump on the running process with dotnet-dump, I do find the task of the background job(App.Store+<Loop>) is in WaitingForActivation state forever.
>dumpasync --address 0x00007f827c657cc8 --fields
STACK 1
00007f827c657cc8 00007f82b6b1cf18 ( ) System.Threading.Tasks.Task<Test.Storage.PutStateResult>
Address MT Type Value Name
0000000000000000 00007f82b10b7160 System.Delegate null m_action
0000000000000000 00007f82b1005290 System.Object null m_stateObject
0000000000000000 00007f82b21345e8 ...Threading.Tasks.TaskScheduler null m_taskScheduler
00007f827c67acf8 00007f82b1005290 System.Object 00007f827c67acf8 m_continuationObject
0000000000000000 00007f82b23611e0 ...sks.Task+ContingentProperties null m_contingentProperties
00007f827c657cf8 00007f82b10a9018 System.Int32 0 m_taskId
00007f827c657cfc 00007f82b10a9018 System.Int32 33555456 m_stateFlags
0000000000000000 00007f82b1005308 System.__Canon null m_result
00007f827c67acf8 00007f82b6b1f7d0 (0) App.Provider.StorageRemoteStore+<SendAsync>d__33<Test.Storage.PutStateResult>
Address MT Type Value Name
00007f827c657a78 00007f82b10ddb68 System.Func<T1> 00007f827c657a78 taskCreator
00007f827c318d18 00007f82b1245cc8 ...pp.Provider.StorageRemoteStore 00007f827c318d18 <>4__this
00007f827c67ad58 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c67ad60 00007f82b20ca868 ...MethodBuilder<System.__Canon> 00007f827c67ad68 <>t__builder
00007f827c67ad68 00007f82b20caf58 ..._Canon>+ConfiguredTaskAwaiter 00007f827c67ad70 <>u__1
00007f827c67ad80 00007f82b6b70068 (0) App.Provider.StorageRemoteStore+<UpdateStateSnapshot>d__19
Address MT Type Value Name
00007f827c318d18 00007f82b1245cc8 ...ogger.Provider.StorageRemoteStore 00007f827c318d18 <>4__this
00007f827c63fa50 00007f82b10bd2e0 System.String ...:1680173361}" content
00007f827c3230e8 00007f82b47b5510 App.Models.AccountInfo 00007f827c3230e8 account
0000000000000000 00007f82b1f63cc0 ...System.String, System.String> null metadataDict
00007f827c67adf0 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c67adf8 00007f82b31699c8 ...kMethodBuilder<System.String> 00007f827c67ae00 <>t__builder
00007f827c67ae00 00007f82b4bd7cb0 ...Result>+ConfiguredTaskAwaiter 00007f827c67ae08 <>u__1
00007f827c67ae18 00007f82b6b70908 (0) App.Provider.RemoteStoreManager+<UpdateStateSnapshot>d__36
Address MT Type Value Name
00007f827c318900 00007f82b47b5c00 ...Logger.Models.StateSnapShot 00007f827c318900 state
00007f827c31af38 00007f82b1246780 ...rovider.RemoteStoreManager 00007f827c31af38 <>4__this
00007f827c3230e8 00007f82b47b5510 App.Models.AccountInfo 00007f827c3230e8 account
00007f827c67ae80 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c67ae88 00007f82b31699c8 ...kMethodBuilder<System.String> 00007f827c67ae90 <>t__builder
00007f827c67ae90 00007f82b473eda8 ...es.TaskAwaiter<System.String> 00007f827c67ae98 <>u__1
00007f827c67aea0 00007f82b6b70d50 (0) App.Store+<UpdateStateIfNeeded>d__216
Address MT Type Value Name
00007f827c318428 00007f82b473a248 App.Store 00007f827c318428 <>4__this
00007f827c67aef8 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c67af00 00007f82b111a088 ...rvices.AsyncTaskMethodBuilder 00007f827c67af08 <>t__builder
00007f827c67af08 00007f82b473eda8 ...es.TaskAwaiter<System.String> 00007f827c67af10 <>u__1
00007f827c33d518 00007f82b4c01540 (11) App.Store+<Loop>d__214
Address MT Type Value Name
00007f827c318428 00007f82b473a248 App.Store 00007f827c318428 <>4__this
0000000000000000 00007f82b1005290 System.Object null <>7__wrap1
00007f827c33d578 00007f82b10a9018 System.Int32 11 <>1__state
00007f827c33d57c 00007f82b10a9018 System.Int32 0 <>7__wrap2
00007f827c33d580 00007f82b111a088 ...rvices.AsyncTaskMethodBuilder 00007f827c33d588 <>t__builder
00007f827c33d588 00007f82b1118308 ....CompilerServices.TaskAwaiter 00007f827c33d590 <>u__1
00007f827c33d590 00007f82b48739b8 ...kAwaiter<App.Address> 00007f827c33d598 <>u__2
> taskstate 00007f827c657cc8
WaitingForActivation
There is another Awaiting task which may be related since the request uri of 00007f827c687f88 request is the exact uri for above task. The response seems be null, but the remote file was probably updated based on file last modified time and content. The request was made a few days before, the awaiting state doesn't make sense.
> dumpasync --address 0x00007f827c35d338 --fields
STACK 1
<< Awaiting: 00007f827c35d3a0 00007f82b22bc928 System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<System.Int32>+ConfiguredValueTaskAwaiter >>
00007f827c35d338 00007f82b23546e0 (0) System.Net.Http.HttpConnection+<InitialFillAsync>d__101
Address MT Type Value Name
00007f827c17d628 00007f82b21231a0 System.Net.Http.HttpConnection 00007f827c17d628 <>4__this
00007f827c35d390 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c35d394 00007f82b100baa8 System.Boolean true async
00007f827c35d398 00007f82b2293c40 ...s.AsyncValueTaskMethodBuilder 00007f827c35d3a0 <>t__builder
00007f827c35d3a0 00007f82b22bc928 ...2>+ConfiguredValueTaskAwaiter 00007f827c35d3a8 <>u__1
00007f827c35d3b8 00007f82b2355b78 (21) System.Net.Http.HttpConnection+<SendAsyncCore>d__64
Address MT Type Value Name
00007f827c17d628 00007f82b21231a0 System.Net.Http.HttpConnection 00007f827c17d628 <>4__this
00007f827c687f88 00007f82b20c87b8 ...m.Net.Http.HttpRequestMessage 00007f827c687f88 request
0000000000000000 00007f82b22beee0 ...pletionSource<System.Boolean> null <allowExpect100ToContinue>5__2
0000000000000000 00007f82b1119bb0 System.Threading.Tasks.Task null <sendRequestContentTask>5__3
00007f827c156288 00007f82b20c8380 System.Net.Http.HttpMethod 00007f827c156288 <normalizedMethod>5__4
0000000000000000 00007f82b1005290 System.Object null <>7__wrap5
0000000000000000 00007f82b10bd2e0 System.String "" <cookiesFromContainer>5__7
0000000000000000 00007f82b1b9af30 ....Net.Http.HttpResponseMessage null <response>5__8
0000000000000000 00007f82b10bec90 System.Exception null <error>5__9
00007f827c35d450 00007f82b10a9018 System.Int32 21 <>1__state
00007f827c35d454 00007f82b100baa8 System.Boolean true async
00007f827c35d458 00007f82b20caa08 ...Net.Http.HttpResponseMessage> 00007f827c35d460 <>t__builder
00007f827c35d460 00007f82b197e780 ...m.Threading.CancellationToken 00007f827c35d468 cancellationToken
00007f827c35d468 00007f82b20c9a28 ...CancellationTokenRegistration 00007f827c35d470 <cancellationRegistration>5__5
00007f827c35d478 00007f82b20cb168 ...aitable+ConfiguredTaskAwaiter 00007f827c35d480 <>u__1
00007f827c35d488 00007f82b220b6a0 ...le+ConfiguredValueTaskAwaiter 00007f827c35d490 <>u__2
00007f827c35d498 00007f82b22bc928 ...2>+ConfiguredValueTaskAwaiter 00007f827c35d4a0 <>u__3
00007f827c35d4a8 00007f82b22bceb8 ...>>+ConfiguredValueTaskAwaiter 00007f827c35d4b0 <>u__4
00007f827c3494c8 00007f82b220eb20 (4) System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__83
Address MT Type Value Name
00007f827c349108 00007f82b2121f18 ...m.Net.Http.HttpConnectionPool 00007f827c349108 <>4__this
00007f827c687f88 00007f82b20c87b8 ...m.Net.Http.HttpRequestMessage 00007f827c687f88 request
0000000000000000 00007f82b1b9af30 ....Net.Http.HttpResponseMessage null <response>5__3
00007f827c17d628 00007f82b21231a0 System.Net.Http.HttpConnection 00007f827c17d628 <connection>5__4
00007f827c349538 00007f82b10a9018 System.Int32 4 <>1__state
00007f827c34953c 00007f82b10a9018 System.Int32 0 <retryCount>5__2
00007f827c349540 00007f82b100baa8 System.Boolean true async
00007f827c349541 00007f82b100baa8 System.Boolean false doRequestAuth
00007f827c349548 00007f82b212aca0 ...Net.Http.HttpResponseMessage> 00007f827c349550 <>t__builder
00007f827c349558 00007f82b197e780 ...m.Threading.CancellationToken 00007f827c349560 cancellationToken
00007f827c349560 00007f82b212b7b0 ...e>+ConfiguredValueTaskAwaiter 00007f827c349568 <>u__1
00007f827c349578 00007f82b212eb70 ...n>+ConfiguredValueTaskAwaiter 00007f827c349580 <>u__2
00007f827c349590 00007f82b20cb318 ...essage>+ConfiguredTaskAwaiter 00007f827c349598 <>u__3
00007f827c3495a0 00007f82b212ec40 ...n>+ConfiguredValueTaskAwaiter 00007f827c3495a8 <>u__4
00007f827c3495c0 00007f82b2220f20 (0) System.Net.Http.RedirectHandler+<SendAsync>d__4
Address MT Type Value Name
00007f827c3490e0 00007f82b20cd718 System.Net.Http.RedirectHandler 00007f827c3490e0 <>4__this
00007f827c687f88 00007f82b20c87b8 ...m.Net.Http.HttpRequestMessage 00007f827c687f88 request
00007f827c349620 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c349624 00007f82b10aa1d8 System.UInt32 0 <redirectCount>5__2
00007f827c349628 00007f82b100baa8 System.Boolean true async
00007f827c349630 00007f82b212aca0 ...Net.Http.HttpResponseMessage> 00007f827c349638 <>t__builder
00007f827c349640 00007f82b197e780 ...m.Threading.CancellationToken 00007f827c349648 cancellationToken
00007f827c349648 00007f82b212b7b0 ...e>+ConfiguredValueTaskAwaiter 00007f827c349650 <>u__1
00007f827c349668 00007f82b22218d8 (0) System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d
Address MT Type Value Name
00007f827c687f88 00007f82b20c87b8 ...m.Net.Http.HttpRequestMessage 00007f827c687f88 request
00007f827c688178 00007f82b1591c00 System.Net.Http.HttpClient 00007f827c688178 <>4__this
00007f827c348aa0 00007f82b1646310 ...ading.CancellationTokenSource 00007f827c348aa0 cts
00007f827c6881c8 00007f82b1646310 ...ading.CancellationTokenSource 00007f827c6881c8 pendingRequestsCts
0000000000000000 00007f82b1b9af30 ....Net.Http.HttpResponseMessage null <response>5__4
00007f827c3496e0 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c3496e4 00007f82b20c8868 ...Net.Http.HttpCompletionOption 1 completionOption
00007f827c3496e8 00007f82b100baa8 System.Boolean true disposeCts
00007f827c3496e9 00007f82b100baa8 System.Boolean false <telemetryStarted>5__2
00007f827c3496ea 00007f82b100baa8 System.Boolean false <responseContentTelemetryStarted>5__3
00007f827c3496f0 00007f82b20caa08 ...Net.Http.HttpResponseMessage> 00007f827c3496f8 <>t__builder
00007f827c3496f8 00007f82b197e780 ...m.Threading.CancellationToken 00007f827c349700 originalCancellationToken
00007f827c349700 00007f82b20cb318 ...essage>+ConfiguredTaskAwaiter 00007f827c349708 <>u__1
00007f827c349710 00007f82b20cb168 ...aitable+ConfiguredTaskAwaiter 00007f827c349718 <>u__2
00007f827c349728 00007f82b6b15fb8 (0) System.Net.HttpWebRequest+<SendRequest>d__195
Address MT Type Value Name
00007f827c664ea8 00007f82b6abc918 System.Net.HttpWebRequest 00007f827c664ea8 <>4__this
00007f827c688178 00007f82b1591c00 System.Net.Http.HttpClient 00007f827c688178 <client>5__3
00007f827c349788 00007f82b10a9018 System.Int32 0 <>1__state
00007f827c34978c 00007f82b100baa8 System.Boolean true async
00007f827c34978d 00007f82b100baa8 System.Boolean true <disposeRequired>5__2
00007f827c349790 00007f82b6b13570 ...ilder<System.Net.WebResponse> 00007f827c349798 <>t__builder
00007f827c349798 00007f82b20cb318 ...essage>+ConfiguredTaskAwaiter 00007f827c3497a0 <>u__1
00007f827c349870 00007f82b6b16768 ( ) System.Threading.Tasks.ContinuationTaskFromResultTask<System.Net.WebResponse> {System.Net.TaskExtensions+<>c__DisplayClass0_0`1[[System.__Canon, System.Private.CoreLib]].<ToApm>b__0(System.Threading.Tasks.Task`1<System.__Canon>)}
Address MT Type Value Name
00007f827c349830 00007f82b10b7160 System.Delegate 00007f827c349830 m_action
0000000000000000 00007f82b1005290 System.Object null m_stateObject
0000000000000000 00007f82b21345e8 ...Threading.Tasks.TaskScheduler null m_taskScheduler
0000000000000000 00007f82b1005290 System.Object null m_continuationObject
00007f827c3498b8 00007f82b23611e0 ...sks.Task+ContingentProperties 00007f827c3498b8 m_contingentProperties
00007f827c3498a0 00007f82b10a9018 System.Int32 0 m_taskId
00007f827c3498a4 00007f82b10a9018 System.Int32 33554944 m_stateFlags
00007f827c349728 00007f82b1b81b90 ...ng.Tasks.Task<System.__Canon> 00007f827c349728 m_antecedent
How should I debug further to figure out why these tasks are in such state for days?