Stack overflow analyzing script snippets

[martincostello]

This afternoon I was writing a inline PowerShell script in a GitHub Actions workflow whose intention is to lint the usage of PowerShell in GitHub Actions workflows (♾), and while testing it locally with a slightly modified version I found I hit a stack overflow that terminated my pwsh session in Windows Terminal.

I can't reproduce it in GitHub Actions itself for some reason, but I can on my own laptop.

Steps to reproduce

1. Clone martincostello/github-automation@5563b3c
2. Run lint.ps1

Expected behavior

The script runs successfully.

Actual behavior

Linting actions/dotnet-release/action.yml...
Linting actions/dotnet-upgrade-report/action.yml...
Linting actions/dotnet-version-report/action.yml...
Linting actions/get-dotnet-repos/action.yml...
Linting actions/get-rebase-repos/action.yml...
Linting actions/get-sdk-repos/action.yml...
Linting actions/is-dotnet-change-available/action.yml...
Linting actions/lint-workflow-powershell/action.yml...
Linting .github/workflows/acr-housekeeping.yml...
Stack overflow.
Repeated 252 times:
--------------------------------
   at System.Reflection.RuntimeAssembly.<InternalLoad>g____PInvoke|49_0(System.Reflection.NativeAssemblyNameParts*, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.StackCrawlMarkHandle, Int32, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.ObjectHandleOnStack)
   at System.Reflection.RuntimeAssembly.InternalLoad(System.Reflection.AssemblyName, System.Threading.StackCrawlMark ByRef, System.Runtime.Loader.AssemblyLoadContext, System.Reflection.RuntimeAssembly, Boolean)
   at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(System.Reflection.Assembly, System.Globalization.CultureInfo, System.Version, Boolean)
   at System.Resources.ManifestBasedResourceGroveler.InternalGetSatelliteAssembly(System.Reflection.Assembly, System.Globalization.CultureInfo, System.Version)
   at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(System.Globalization.CultureInfo)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(System.Globalization.CultureInfo, System.Collections.Generic.Dictionary`2<System.String,System.Resources.ResourceSet>, Boolean, Boolean)
   at System.Resources.ResourceManager.InternalGetResourceSet(System.Globalization.CultureInfo, Boolean, Boolean)
   at System.Resources.ResourceManager.GetString(System.String, System.Globalization.CultureInfo)
   at System.Management.Automation.ErrorCategoryInfo.Ellipsize(System.Globalization.CultureInfo, System.String)
   at System.Management.Automation.ScriptBlock.GetContextFromTLS()
   at System.Management.Automation.ScriptBlock.InvokeAsDelegateHelper(System.Object, System.Object, System.Object[])
   at DynamicClass.lambda_method523(System.Runtime.CompilerServices.Closure, System.Object, System.ResolveEventArgs)
   at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(System.ResolveEventHandler, System.Reflection.RuntimeAssembly, System.String)
   at System.Reflection.RuntimeAssembly.<InternalLoad>g____PInvoke|49_0(System.Reflection.NativeAssemblyNameParts*, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.StackCrawlMarkHandle, Int32, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.ObjectHandleOnStack)
--------------------------------
   at System.Reflection.RuntimeAssembly.<InternalLoad>g____PInvoke|49_0(System.Reflection.NativeAssemblyNameParts*, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.StackCrawlMarkHandle, Int32, System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.ObjectHandleOnStack)
   at System.Reflection.RuntimeAssembly.InternalLoad(System.Reflection.AssemblyName, System.Threading.StackCrawlMark ByRef, System.Runtime.Loader.AssemblyLoadContext, System.Reflection.RuntimeAssembly, Boolean)
   at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(System.Reflection.Assembly, System.Globalization.CultureInfo, System.Version, Boolean)
   at System.Resources.ManifestBasedResourceGroveler.InternalGetSatelliteAssembly(System.Reflection.Assembly, System.Globalization.CultureInfo, System.Version)
   at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(System.Globalization.CultureInfo)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(System.Globalization.CultureInfo, System.Collections.Generic.Dictionary`2<System.String,System.Resources.ResourceSet>, Boolean, Boolean)
   at System.Resources.ResourceManager.InternalGetResourceSet(System.Globalization.CultureInfo, Boolean, Boolean)
   at System.Resources.ResourceManager.GetString(System.String, System.Globalization.CultureInfo)
   at Microsoft.PowerShell.Commands.CertificateProvider.InitializeDefaultDrives()
   at System.Management.Automation.SessionStateInternal.InitializeProvider(System.Management.Automation.Provider.CmdletProvider, System.Management.Automation.ProviderInfo, System.Management.Automation.CmdletProviderContext)
   at System.Management.Automation.SessionStateInternal.NewProvider(System.Management.Automation.ProviderInfo)
   at System.Management.Automation.SessionStateInternal.AddProvider(System.Type, System.String, System.String, System.Management.Automation.PSSnapInInfo, System.Management.Automation.PSModuleInfo)
   at System.Management.Automation.SessionStateInternal.AddSessionStateEntry(System.Management.Automation.Runspaces.SessionStateProviderEntry)
   at System.Management.Automation.Runspaces.InitialSessionState.Bind_LoadProviders(System.Management.Automation.SessionStateInternal)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.Task.WrappedTryRunInline()
   at System.Threading.Tasks.Task.WaitAllCore(System.ReadOnlySpan`1<System.Threading.Tasks.Task>, Int32, System.Threading.CancellationToken)
   at System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[])
   at System.Threading.Tasks.Parallel.Invoke(System.Threading.Tasks.ParallelOptions, System.Action[])
   at System.Management.Automation.Runspaces.InitialSessionState.Bind(System.Management.Automation.ExecutionContext, Boolean, System.Management.Automation.PSModuleInfo, Boolean, Boolean, Boolean)
   at System.Management.Automation.AutomationEngine..ctor(System.Management.Automation.Host.PSHost, System.Management.Automation.Runspaces.InitialSessionState)
   at System.Management.Automation.Runspaces.LocalRunspace.DoOpenHelper()
   at System.Management.Automation.Runspaces.RunspaceBase.CoreOpen(Boolean)
   at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.CreateRunspace()
   at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.OpenHelper()
   at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.CoreOpen(Boolean, System.AsyncCallback, System.Object)
   at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.Open()
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.CommandInfoCache..ctor()
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.Helper+<>c.<.ctor>b__24_0()
   at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ViaFactory(System.Threading.LazyThreadSafetyMode)
   at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExecutionAndPublication(System.LazyHelper, Boolean)
   at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CreateValue()
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.Helper.get_CommandInfoCache()
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.Helper.GetCommandInfo(System.String, System.Nullable`1<System.Management.Automation.CommandTypes>, Boolean)
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.AvoidAlias+<AnalyzeScript>d__9.MoveNext()
   at System.Collections.Generic.List`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)
   at System.Linq.Enumerable.ToList[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.ScriptAnalyzer+<>c__DisplayClass84_1.<AnalyzeSyntaxTree>b__2()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

[process exited with code 3221225725 (0xc00000fd)]

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.5.0
PSEdition                      Core
GitCommitId                    7.5.0
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

> (Get-Module -ListAvailable PSScriptAnalyzer).Version | ForEach-Object { $_.ToString() }
1.23.0

[bergmeister]

Can you please try to narrow it down to a simpler repro? Multithreading issues sometimes only occur on certain CPU speeds which is maybe why it does not happen in GitHub actions

[martincostello]

To be honest, I don't have any good idea on where to start to narrow it down.

I did try, for example I deleted the script it choked on to see if it then continued, but it just choked on the next one. It also seemed to be killing the pwsh terminal embedded within VS Code when I found it too.

[bergmeister]

Divide and conquer is an effective method to minimise it. Remove half the code, if it still happens, half it again, if not, undo and remove other half, rinse and repeat

[martincostello]

I did do that to a degree, but there's only so much time in the day to dig into a technology I don't know the internals of. I spent about 10-15 minutes try to get to an "aha" moment, but this was impacting me while trying to do other work and I didn't want to go off into a full Hal moment. Once I finished my task, I opened this information with as far as I got.

There's an issue somewhere because I don't reasonably see how this could be user error, but if you think the code provided to repro it is too much (you can ignore most of it, it's just the codebase the error was found in, the amount of PowerShell in there is minimal) and don't want to use that as a starting point then feel free to close this issue.

[bergmeister]

Issue is still good to have in case someone else sees something similar and then the combination of traces sometimes brings something to light. But equally, we don't have that much time to investigate either. I see it's due to CommandInfo cache and in the past already spent a lot of time on exceptions by it. Conclusion was that this PowerShell internal object itself is not thread safe so suspect it's related to that. See PowerShell/PowerShell#12865