Unexpected coupling between ordering of parsing .ps1 files with classes/enums and executing them

[jazzdelightsme]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Save the following script as repro.ps1, and run with pwsh -NoProfile -Command .\repro.ps1. Description of the problem in the comments:

<#
.SYNOPSIS
    This script demonstrates an unexpected coupling between the order in which you parse a
    script with custom types (enums or classes) and execute the resulting Ast's
    ScriptBlock: if you parse some other script before executing the type-declaring
    script's code, things appear to succeed (no errors generated when executing the parsed
    scripts), but attempting to use the custom types fails.

    Run with: pwsh.exe -NoProfile -Command ".\repro.ps1"

    (Or, for the "No Repro" case: pwsh.exe -NoProfile -Command ".\repro.ps1 -InOrder")

    This script generates three files:

        Types.ps1:        Defines an enum ("MyEnum").
        Code.ps1:         Defines a function ("Foo") that uses the enum.
        ReproModule.psm1: Parses and executes the .ps1 files.

    In the repro case, when the module is loaded, the sequence of events is:

        1. parse Types.ps1
        2. parse Code.ps1
        3. execute Types.ps1's Ast.GetScriptBlock()
        4. execute Code.ps1's Ast.GetScriptBlock()

    Finally, once the ReproModule is loaded, this script attempts to run "Foo -?".

    The Expected Result is that we should see a nice help message about how to call Foo.

    The actual result is an error, complaining that it can't find the MyEnum type:

        InvalidOperation: C:\Users\danthom\OutOfOrderParseProblem\Code.ps1:1
        Line |
           1 |  function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }
             |                                          ~~~~~~~~
             | Unable to find type [MyEnum].

    The -InOrder parameter changes the order of the steps performed by the
    ReproModule.psm1:

        1. parse Types.ps1
        2. execute Types.ps1's Ast.GetScriptBlock()
        3. parse Code.ps1
        4. execute Code.ps1's Ast.GetScriptBlock()

    When performed in this order, "Foo -?" behaves as expected.

    Q: Why is this interesting / important?

    Parsing a script into an AST should have *no* dependency on any kind of context.
    (This Issue demonstrates that unfortunately, there is some kind of hidden dependency.)
    If this hidden dependency did not exist, it would allow multi-threaded processing of
    many scripts, for significant perf gains (both for first-time loading, as well as
    being able to save the ASTs, to be used in other runspaces).

    Example C#:

        public static void ParseScripts( IList<ParsePsWorkItem> parseWorkItems )
        {
            Parallel.ForEach(parseWorkItems, parseWorkItem =>
            {
                parseWorkItem.ParseTask = File.ReadAllTextAsync(parseWorkItem.Path).ContinueWith( t =>
                {
                    Token[] tokens ;
                    ParseError[] errors;

                    var ast = Parser.ParseInput( t.Result, parseWorkItem.Path, out tokens, out errors );
                    return new PsParseResult( errors, ast );
                } );
            });
        }

    And corresponding code in a .psm1:

        foreach ($pwi in $parseWorkItems)
        {
            $parseResult = $pwi.ParseTask.GetAwaiter().GetResult()
            if( $parseResult.Errors -and $parseResult.Errors.Count )
            {
                throw "Parse error for $($pwi.Path): $($parseResult.Errors[ 0 ])"
            }

            $ExecutionContext.InvokeCommand.InvokeScript( $false,    # useLocalScope,
                                                          $parseResult.Ast.GetScriptBlock(),
                                                          @(),      # input
                                                          @() )     # args
        }

    This code will *appear* to work (no errors when loading up)... but things will fail at
    runtime, if your scripts use enums/classes. :'(
#>
[CmdletBinding()]
param(
    [switch] $InOrder
)

'enum MyEnum { None = 0; One = 1 }' > Types.ps1

'function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }' > Code.ps1

# The main .psm1:
@'
$parseWorkItems = @()

foreach( $file in 'Types.ps1', 'Code.ps1' )
{
    $parseWorkItems += [PSCustomObject] @{
        Path = (Resolve-Path $file).ProviderPath
        Errors = $null
        Ast = $null
    }
}

if( $env:_ParseAndExecInOrder -ne 0 )
{
    # THIS IS THE "NO-REPRO" CASE; LOOK AT THE 'ELSE' BLOCK FIRST

    Write-Host "We are going to parse and then immediately load each script, in order."

    foreach( $pwi in $parseWorkItems )
    {
        $tokens = @()
        $errors = @()

        Write-Host "Parsing: $($pwi.Path)"
        $ast = [System.Management.Automation.Language.Parser]::ParseInput( (gc -Raw $pwi.Path), $pwi.Path, ([ref] $tokens), ([ref] $errors) )

        if( $errors -and $errors.Count )
        {
            throw "Parse error in $($pwi.Path): $($errors[ 0 ])"
        }

        Write-Host "Loading: $($pwi.Path)"
        $ExecutionContext.InvokeCommand.InvokeScript( $false,    # useLocalScope,
                                                      $ast.GetScriptBlock(),
                                                      @(),      # input
                                                      @() )     # args
    }
}
else
{
    Write-Host "We are going to parse all scripts first, then go back and execute the results (`"out of order`")."

    foreach( $pwi in $parseWorkItems )
    {
        $tokens = @()
        $errors = @()

        Write-Host "Parsing: $($pwi.Path)"
        $ast = [System.Management.Automation.Language.Parser]::ParseInput( (gc -Raw $pwi.Path), $pwi.Path, ([ref] $tokens), ([ref] $errors) )

        $pwi.Errors = $errors
        $pwi.Ast = $ast
    }

    foreach( $pwi in $parseWorkItems )
    {
        Write-Host "Loading: $($pwi.Path)"

        if( $pwi.Errors -and $pwi.Errors.Count )
        {
            throw "Parse error in $($pwi.Path): $($pwi.Errors[ 0 ])"
        }

        $ExecutionContext.InvokeCommand.InvokeScript( $false,    # useLocalScope,
                                                      $pwi.Ast.GetScriptBlock(),
                                                      @(),      # input
                                                      @() )     # args
    }
}

'@ > ReproModule.psm1

if( $InOrder )
{
    $env:_ParseAndExecInOrder = 1
}
else
{
    $env:_ParseAndExecInOrder = 0
}

# Now we load the module and attempt to get the help for the Foo command.
ipmo .\ReproModule.psm1

Foo -?
# Expected result: We should see a nice help message about how to call Foo.
#
# Actual result (when !$InOrder):
#
#   Parsing: C:\Users\danthom\OutOfOrderParseProblem\Types.ps1
#   Parsing: C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
#   Loading: C:\Users\danthom\OutOfOrderParseProblem\Types.ps1
#   Loading: C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
#   InvalidOperation: C:\Users\danthom\OutOfOrderParseProblem\Code.ps1:1
#   Line |
#      1 |  function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }
#        |                                          ~~~~~~~~
#        | Unable to find type [MyEnum].

Expected behavior

NAME
    Foo

SYNTAX
    Foo [[-ME] {None | One}] [<CommonParameters>]


ALIASES
    None


REMARKS
    None

Actual behavior

InvalidOperation: C:\Users\danthom\OutOfOrderParseProblem\Code.ps1:1
Line |
   1 |  function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }
     |                                          ~~~~~~~~
     | Unable to find type [MyEnum].

Error details

Exception             :
    Type        : System.Management.Automation.RuntimeException
    ErrorRecord :
        Exception             :
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Unable to find type [MyEnum].
            HResult : -2146233087
        TargetObject          : MyEnum
        CategoryInfo          : InvalidOperation: (MyEnum:TypeName) [], ParentContainsErrorRecordException
        FullyQualifiedErrorId : TypeNotFound
        InvocationInfo        :
            ScriptLineNumber : 1
            OffsetInLine     : 41
            HistoryId        : 1
            ScriptName       : C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
            Line             : function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }

            Statement        : [MyEnum]
            PositionMessage  : At C:\Users\danthom\OutOfOrderParseProblem\Code.ps1:1 char:41
                               + function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }
                               +                                         ~~~~~~~~
            PSScriptRoot     : C:\Users\danthom\OutOfOrderParseProblem
            PSCommandPath    : C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
            CommandOrigin    : Internal
        ScriptStackTrace      : at <ScriptBlock>, C:\Users\danthom\OutOfOrderParseProblem\repro.ps1: line 184
                                at <ScriptBlock>, <No file>: line 1
    TargetSite  :
        Name          : ResolveTypeName
        DeclaringType : [System.Management.Automation.TypeOps]
        MemberType    : Method
        Module        : System.Management.Automation.dll
    Message     : Unable to find type [MyEnum].
    Data        : System.Collections.ListDictionaryInternal
    Source      : System.Management.Automation
    HResult     : -2146233087
    StackTrace  :
   at System.Management.Automation.TypeOps.ResolveTypeName(ITypeName typeName, IScriptExtent errorPos)
   at System.Management.Automation.Language.Compiler.GetAttribute(TypeConstraintAst typeConstraintAst)
   at System.Management.Automation.Language.Compiler.GetRuntimeDefinedParameter(ParameterAst parameterAst, Boolean& customParameterSet, Boolean&
usesCmdletBinding)
   at System.Management.Automation.Language.Compiler.GetParameterMetaData(ReadOnlyCollection`1 parameters, Boolean automaticPositions, Boolean&
usesCmdletBinding)
   at System.Management.Automation.CompiledScriptBlockData.InitializeMetadata()
   at System.Management.Automation.CompiledScriptBlockData.Compile(Boolean optimized)
   at System.Management.Automation.PSScriptCmdlet..ctor(ScriptBlock scriptBlock, Boolean useNewScope, Boolean fromScriptFile, ExecutionContext context)
   at System.Management.Automation.CommandProcessor.Init(IScriptCommandInfo scriptCommandInfo)
   at System.Management.Automation.CommandDiscovery.GetScriptAsCmdletProcessor(IScriptCommandInfo scriptCommandInfo, ExecutionContext context, Boolean
useNewScope, Boolean fromScriptFile, SessionStateInternal sessionState)
   at System.Management.Automation.CommandDiscovery.CreateCommandProcessorForScript(FunctionInfo functionInfo, ExecutionContext context, Boolean
useNewScope, SessionStateInternal sessionState)
   at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(CommandInfo commandInfo, CommandOrigin commandOrigin, Nullable`1 useLocalScope,
SessionStateInternal sessionState)
   at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
   at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
   at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst
commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
   at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements,
CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
   at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
TargetObject          : MyEnum
CategoryInfo          : InvalidOperation: (MyEnum:TypeName) [], RuntimeException
FullyQualifiedErrorId : TypeNotFound
InvocationInfo        :
    ScriptLineNumber : 1
    OffsetInLine     : 41
    HistoryId        : 1
    ScriptName       : C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
    Line             : function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }

    Statement        : [MyEnum]
    PositionMessage  : At C:\Users\danthom\OutOfOrderParseProblem\Code.ps1:1 char:41
                       + function Foo { [CmdletBinding()] param( [MyEnum] $ME ) $ME }
                       +                                         ~~~~~~~~
    PSScriptRoot     : C:\Users\danthom\OutOfOrderParseProblem
    PSCommandPath    : C:\Users\danthom\OutOfOrderParseProblem\Code.ps1
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, C:\Users\danthom\OutOfOrderParseProblem\repro.ps1: line 184
                        at <ScriptBlock>, <No file>: line 1

Environment data

Name                           Value
----                           -----
PSVersion                      7.5.1
PSEdition                      Core
GitCommitId                    7.5.1
OS                             Microsoft Windows 10.0.26422
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[jazzdelightsme]

I am seriously considering banning PS enums and classes in our module, just to enable the perf gains of multi-threaded loading and parsing.

[daxian-dbw]

The Engine WG reviewed this issue today. The root cause of the issue is related to how module defined classes get exposed outside of the module scope.

Root Cause

When using a script module as is shown in the repro step above, pwsh -NoProfile -Command .\repro.ps1 stops working, while pwsh -NoProfile -Command .\repro.ps1 -InOrder works.

This is because:

1. If Type.ps1 has already executed when parsing Code.ps1 (the -InOrder case), the TypeName 'MyEnum' will be successfully resolved within the SymbolResolver and the type will be cached in the TypeName instance.
2. The enum MyEnum is defined in the module scope, so the type cannot be discovered outside of the module scope (unless explicitly expose the PowerShell classes defined in a module with using module).
3. However, since the resolved type is already cached during parsing, 'Foo -?' will work even if the type resolution triggered by it runs from outside of the module scope.

☝️ This is why pwsh -NoProfile -Command .\repro.ps1 -InOrder works.

1. If Type.ps1 has not executed when parsing Code.ps1, the TypeName 'MyEnum' cannot be resolved during the parsing and thus there is no cached type for the TypeName instance.
2. When running Foo -?, the type resolution of the TypeName 'MyEnum' gets triggered, but it cannot be resolved because MyEnum is discoverable only within the module scope.

☝️ This is why pwsh -NoProfile -Command .\repro.ps1 does not work.

Use ReproModule.ps1 instead of ReproModule.psm1

If we change the repro.ps1 a bit to exclude the module by

1. Rename ReproModule.psm1 to ReproModule.ps1
2. Change ipmo .\ReproModule.psm1 (import the module) to . .\ReproModule.ps1 (dot source the ps1 file)

Then pwsh -NoProfile -Command .\repro.ps1 works as expected. That's because MyEnum gets defined in the same scope, and thus the type resolution triggered by Foo -? can find the type.

Force Resolving MyEnum within ReproModule.psm1

Similarly, if we force resolving MyEnum within ReproModule.psm1, then the type will be cached in the TypeName instance, and the Foo -? outside of the module will work. To try it out, add gcm Foo | % parameters > $null to the end the script to be written to ReproModule.psm1 and then run pwsh -noprofile -command .\repro.ps1.

Conclusion

As is described above, the execution order does affect the parsing in PowerShell, which can cause confusion as demonstrated by this issue. However, breaking changes would be inevitable if we attempt to change the current behavior.

Also, PowerShell was not designed to do the parsing on multiple threads and then run the generated ASTs on the pipeline thread in sequence, e.g. ThreadStatic is used for DSC related parsing code, which is likely causing issues in some cases when parsing on different threads.

PowerShell class is good for internal uses within a module, but it's not recommended to expose PowerShell class outside of a module. Take the repro script as an example, if we define a PowerShell class and use it for a parameter of a function, the user won't be able to construct an instance of that PowerShell class when invoking the function, because that PowerShell class is only discoverable within the module scope.

Given that above, the Engine WG's decision for this issue is: Won't Fix.

[microsoft-github-policy-service]

This issue has been marked as won't fix and has not had any activity for 1 day. It has been closed for housekeeping purposes.

[microsoft-github-policy-service]

📣 Hey @@jazzdelightsme, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms