Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More threadpool exhaustion fixes #1080

Merged
merged 1 commit into from Mar 19, 2023

Conversation

TheAngryByrd
Copy link
Member

Back with more threadpool exhaustion fixes!

  • One of the main offenders is hitting Adaptive structures more often than needed. Since Adaptive uses lots of locks to ensure consistency, we end up getting threads blocked.
Stackframes for FSAC showing where threadpool thread is getting blocked by Adaptive

Thread (0x111A4):
[Native Frames]
FSharp.Data.Adaptive!FSharp.Data.Traceable.History2[FSharp.Data.Adaptive.FSharpHashMap2[System.__Canon,System.__Canon],FSharp.Data.Adaptive.FSharpHashMapDelta2[System.__Canon,System.__Canon]].GetValue(value class FSharp.Data.Adaptive.AdaptiveToken) FSharp.Data.Adaptive!FSharp.Data.Adaptive.AValModule+MapVal2[FSharp.Data.Adaptive.FSharpHashMap2[System.__Canon,System.__Canon],System.__Canon].Compute(value class FSharp.Data.Adaptive.AdaptiveToken) FSharp.Data.Adaptive!FSharp.Data.Adaptive.AValModule+AbstractVal1[System.__Canon].GetValue(value class FSharp.Data.Adaptive.AdaptiveToken)
FSharp.Data.Adaptive!FSharp.Data.Adaptive.AValModule+BindVal2[System.__Canon,System.__Canon].Compute(value class FSharp.Data.Adaptive.AdaptiveToken) FSharp.Data.Adaptive!FSharp.Data.Adaptive.AValModule+AbstractVal1[System.__Canon].GetValue(value class FSharp.Data.Adaptive.AdaptiveToken)
fsautocomplete!FsAutoComplete.Lsp.AdaptiveFSharpLspServer.forceFindOpenFile(class System.String)
fsautocomplete!<StartupCode$fsautocomplete>.$AdaptiveFSharpLspServer+-ctor@1046-82.Invoke(class System.String)
FsAutoComplete.Core!FsAutoComplete.FileSystem.FSharp.Compiler.IO.IFileSystem.GetLastWriteTimeShim(class System.String)
FSharp.Compiler.Service!FSharp.Compiler.CompilerConfig+TimeStampCache.GetFileTimeStamp(class System.String)
FSharp.Compiler.Service!FSharp.Compiler.CodeAnalysis.IncrementalBuilderStateHelpers+computeStampedReferencedAssemblies@1116.Invoke(int32,class System.Tuple2<class Microsoft.FSharp.Core.FSharpChoice2<class System.String,class IProjectReference>,class Microsoft.FSharp.Core.FSharpFunc2<class TimeStampCache,value class System.DateTime>>) FSharp.Compiler.Service!FSharp.Compiler.CodeAnalysis.IncrementalBuilderStateHelpers.computeStampedReferencedAssemblies(class FSharp.Compiler.CodeAnalysis.IncrementalBuilderInitialState,class FSharp.Compiler.CodeAnalysis.IncrementalBuilderState,bool,class TimeStampCache) FSharp.Compiler.Service!FSharp.Compiler.CodeAnalysis.IncrementalBuilder.get_IsReferencesInvalidated() FSharp.Compiler.Service!<StartupCode$FSharp-Compiler-Service>.$Service+clo@419-350.Invoke(class System.Tuple2<class Microsoft.FSharp.Core.FSharpOption1<class FSharp.Compiler.CodeAnalysis.IncrementalBuilder>,class FSharp.Compiler.Diagnostics.FSharpDiagnostic[]>) FSharp.Compiler.Service!FSharp.Compiler.BuildGraph+Bind@61-1[System.__Canon,System.__Canon].Invoke(!0) FSharp.Core!Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck(value class Microsoft.FSharp.Control.AsyncActivation1<!!0>,!!1,class Microsoft.FSharp.Core.FSharpFunc2<!!1,class Microsoft.FSharp.Control.FSharpAsync1<!!0>>)
FSharp.Core!Microsoft.FSharp.Control.Trampoline.Execute(class Microsoft.FSharp.Core.FSharpFunc2<class Microsoft.FSharp.Core.Unit,class Microsoft.FSharp.Control.AsyncReturn>) FSharp.Core!Microsoft.FSharp.Control.AsyncPrimitives+AttachContinuationToUnitTask@1263.Invoke(class System.Threading.Tasks.Task) System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object) System.Private.CoreLib.il!System.Threading.Tasks.Task.ExecuteWithThreadLocal(class System.Threading.Tasks.Task&,class System.Threading.Thread) System.Private.CoreLib.il!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.TaskScheduler.TryRunInline(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.ContinueWithTaskContinuation.Run(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object) System.Private.CoreLib.il!System.Threading.Tasks.Task.FinishSlow(bool) System.Private.CoreLib.il!System.Threading.Tasks.Task.ExecuteWithThreadLocal(class System.Threading.Tasks.Task&,class System.Threading.Thread) System.Private.CoreLib.il!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.TaskScheduler.TryRunInline(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.ContinueWithTaskContinuation.Run(class System.Threading.Tasks.Task,bool) System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object) System.Private.CoreLib.il!System.Threading.SemaphoreSlim+<WaitUntilCountOrTimeoutAsync>d__31.MoveNext() System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object) System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[System.Boolean,System.Threading.SemaphoreSlim+<WaitUntilCountOrTimeoutAsync>d__31].MoveNext(class System.Threading.Thread) System.Private.CoreLib.il!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(class System.Action,bool) System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object) System.Private.CoreLib.il!System.Threading.Tasks.Task1[System.Boolean].TrySetResult(!0)
System.Private.CoreLib.il!System.Threading.Tasks.Task+CancellationPromise`1[System.Boolean].System.Threading.Tasks.ITaskCompletionAction.Invoke(class System.Threading.Tasks.Task)
System.Private.CoreLib.il!System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()


One of the threadpool exhaustion issues I can't fix yet is in the F# compiler itself.

setCurrentState uses a lock in node CE code (which is effectively async) so this should probably use SemaphoreSlim.WaitAsync. We probably won't see this fix for a bit but just pointing it out for awareness.

Thread (0x1A138):
  [Native Frames]
  FSharp.Compiler.Service!FSharp.Compiler.CodeAnalysis.IncrementalBuilder.setCurrentState(class FSharp.Compiler.CodeAnalysis.IncrementalBuilderState,class TimeStampCache,value class System.Threading.CancellationToken)
  FSharp.Compiler.Service!.$IncrementalBuild+clo@1241-278.Invoke(value class System.Threading.CancellationToken)
  FSharp.Compiler.Service!FSharp.Compiler.BuildGraph+Bind@61-1[System.Threading.CancellationToken,System.__Canon].Invoke(!0)
  FSharp.Core!Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck(value class Microsoft.FSharp.Control.AsyncActivation`1,!!1,class Microsoft.FSharp.Core.FSharpFunc`2>)
  FSharp.Core!Microsoft.FSharp.Control.Trampoline.Execute(class Microsoft.FSharp.Core.FSharpFunc`2)
  FSharp.Core!Microsoft.FSharp.Control.AsyncPrimitives+AttachContinuationToUnitTask@1263.Invoke(class System.Threading.Tasks.Task)
  System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.ExecuteWithThreadLocal(class System.Threading.Tasks.Task&,class System.Threading.Thread)
  System.Private.CoreLib.il!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.TaskScheduler.TryRunInline(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.ContinueWithTaskContinuation.Run(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.FinishSlow(bool)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.ExecuteWithThreadLocal(class System.Threading.Tasks.Task&,class System.Threading.Thread)
  System.Private.CoreLib.il!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.TaskScheduler.TryRunInline(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.ContinueWithTaskContinuation.Run(class System.Threading.Tasks.Task,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object)
  System.Private.CoreLib.il!System.Threading.SemaphoreSlim+d__31.MoveNext()
  System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)
  System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Boolean,System.Threading.SemaphoreSlim+d__31].MoveNext(class System.Threading.Thread)
  System.Private.CoreLib.il!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(class System.Action,bool)
  System.Private.CoreLib.il!System.Threading.Tasks.Task.RunContinuations(class System.Object)
  System.Private.CoreLib.il!System.Threading.Tasks.Task`1[System.Boolean].TrySetResult(!0)
  System.Private.CoreLib.il!System.Threading.Tasks.Task+CancellationPromise`1[System.Boolean].System.Threading.Tasks.ITaskCompletionAction.Invoke(class System.Threading.Tasks.Task)
  System.Private.CoreLib.il!System.Threading.ThreadPoolWorkQueue.Dispatch()
  System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
  

@baronfel baronfel merged commit 39427d4 into ionide:main Mar 19, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants