'-ErrorAction Stop' not respected by 'Invoke-Pester'

[DarkLite1]

General summary of the issue

The code Invoke-Pester -ErrorAction Stop placed in a Try clause does not trigger the Catch clause.

Describe your environment

Pester version : 5.1.1 C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1
PowerShell version : 5.1.14393.3866
OS version : Microsoft Windows NT 10.0.14393.0

Steps to reproduce

The code below clearly is a failing test that was working with Pester 4 but no longer works with Pester 5. For that reason we would expect Invoke-Pester -ErrorAction Stop to trigger the Catch clause. But whatever we try we can't get it to do so.

$config = [PesterConfiguration]::Default
$config.Run.PassThru = $true
$config.Run.Exit = $true
$config.Run.ScriptBlock = {
    In $TestDrive {
        It 'should be green' {
            1 | Should be 1
        }
    }
}

Try {
    Write-Verbose 'Execute Pester tests'
    
    $invokePesterParams = @{
        Configuration = $config
        ErrorAction   = 'Stop'
    }
    $invokePesterResult = Invoke-Pester @invokePesterParams
}
Catch {
    throw "Failed to execute the Pester tests: $_ "
}

Expected Behavior

Throw the error message "Failed to execute the Pester tests: The 'In' command was found in the module 'Pester'..."

Current Behavior

Catch clause not reached, so error handling is very difficult. Although a big red error is visible in the console.

Possible Solution? (optional)

Respect the -ErrorAction request.

As a workaround to start with Pester 5 reporting we tried to exclude all paths that still contained old Pester 4 tests that would break 'Invoke-Pester'. But when we do that we see that Pester still fails and reports back the same error. When we use ExcludePath we would expect Pester to not look at these broken test files at all.

[PesterConfiguration]$config = @{
    Run = @{
        Path        = 'T:/Prod'
        ExcludePath = @(
            'T:\Prod\old script 1',
            'T:\Prod\old script 2',
            'T:\Prod\old script 3'
        )
        PassThru    = $true
        Exit        = $true
        ScriptBlock = {
            Describe 'test' {
                It 'should be green' {
                    1 | Should -Be 1
                }
            }
        }
    }
}

Try {
    Write-Verbose 'Execute Pester tests'
    
    $invokePesterParams = @{
        Configuration = $config
        ErrorAction   = 'Stop'
    }
    $invokePesterResult = Invoke-Pester @invokePesterParams
}
Catch {
    throw "Failed to execute the Pester tests: $_ "
}

Error:

Starting discovery in 37 files.
System.Management.Automation.CommandNotFoundException: The 'In' command was found in the module 'Pester', but the module could not be loaded. For more information, run 'Import-Module Pester'. ---> System.Management.Automation.CmdletInvocationException: Assertion operator name 'Be' has been added multiple times. ---> System.Management.Automation.RuntimeException: Assertion operator name 'Be' has been added multiple times.
at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)
--- End of inner exception stack trace ---
at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection1 input, PSDataCollection1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection1 input, PSDataCollection1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.InvokeT
at System.Management.Automation.CommandDiscovery.AutoloadSpecifiedModule(String moduleName, ExecutionContext context, SessionStateEntryVisibility visibility, Exception& exception)
--- End of inner exception stack trace ---
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at New-Block(Closure , FunctionContext )
at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess)
at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
at System.Management.Automation.CommandProcessorBase.Complete()
at , T:\Prod\GUI application\Show-SplashScreen\Show-SplashScreen.Tests.ps1: line 67
at New-Block, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 766
at Describe, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 8578
at , T:\Prod\GUI application\Show-SplashScreen\Show-SplashScreen.Tests.ps1: line 59
at , C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 2903
at Invoke-File, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 2912
at Invoke-BlockContainer, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 2837
at Discover-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 1411
at Invoke-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 2356
at Invoke-Pester, C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1: line 4839
at , T:\Test\Brecht\PowerShell\Testie2.ps1: line 46
T:\Test\Brecht\PowerShell>

[nohwnd]

Two things here. You are setting .exit = $true, which will exit on failure. If you are trying this in VSCode then seemingly nothing happens, because exit is ignored there. So you won't reach "catch" and you won't reach code after catch.

If you remove .exit it will still not work though. Your $invokePesterResult will be $null. The way that errors are reported in Discovery is still quite broken. It should either return an object with results or properly fail, but not none of those things.

[nohwnd]

I made it much better in the linked PR, it now reports this for your example:

Containers            : {[-] <ScriptBlock>:C:\Users\jajares\Desktop\blbl.ps1:4}
Result                : Failed
FailedCount           : 0
FailedBlocksCount     : 0
FailedContainersCount : 1
PassedCount           : 0
SkippedCount          : 0
NotRunCount           : 0
TotalCount            : 0
Duration              : 00:00:00.0140865
Executed              : True
ExecutedAt            : 08/04/2021 00:04:31
Version               : 5.2.0-alpha3
PSVersion             : 5.1.19041.610
PSBoundParameters     : {[Configuration, PesterConfiguration], [ErrorAction, Stop]}
Plugins               :
PluginConfiguration   : {}
PluginData            : {}
Configuration         : PesterConfiguration
DiscoveryDuration     : 00:00:00.0140865
UserDuration          : 00:00:00
FrameworkDuration     : 00:00:00
Failed                : {}
FailedBlocks          : {}
FailedContainers      : {[-] <ScriptBlock>:C:\Users\jajares\Desktop\blbl.ps1:4}
Passed                : {}
Skipped               : {}
NotRun                : {}
Tests                 : {}
CodeCoverage          :

You still have to throw the exception yourself though.

[DarkLite1]

Thank you Jakub for fixing this. So if I understand it correctly the property FailedContainers can contain multiple FullName s of files. This will hopefully allow Pester to continue with the .Tests.ps1 files that don't have old Pester 4 broken code in them, i.e. code that isn't compatible with Pester 5.

[nohwnd]

Yes pretty much. FailedContainers has the whole Container object in it (a container represents a file / scriptblock). This is meant to give you feedback when there is code that prevents the whole container from running (in the Run phase), e.g. code outside of a test that throws, or a bug in pester code. I am re-using that here for discovery as well because it makes sense, I just did not do it that way originally for some reason.

It will continue to next file and it will continue to test run, but the whole run will be marked as failed (see Result in the object above).

I will be adding an option to throw on failed tests (instead of exiting).

(Note to self: Not sure if I can exit in finally after throwing an exception, when both exit and throw is enabled, but if I can't I should prefer throwing over exit I think, because exit does not work correctly in VSCode, but throwing exception should fail both CI run as well as VSCode run).

[DarkLite1]

One small tip I would like to suggest with regards to naming properties and what they stand for. It might be a good idea to use the name Success combined with a boolean true or false instead of Result = Passed. I think you probably use Passed to indicate Passed, Skipped, Failed. But it might be better to use:

@{
     Success: $true # or $false or NULL if skipped
     Skipped: $true # or $false
}

I see there's already an Executed and ExecutedAt property. They can probably be combined into on. If there is no date then it is not executed, if there is a date it is executed right? Then you are just left with ExecutedAt which holds a date. A test that is skipped will nog have ExecutedAt set right? So that will simply become:

@{
     Success: $true # or $false or NULL if skipped
     ExecutedAt: 08/04/2021 00:04:31
}

Just an idea, I might not understand all the implications involved. It would just be a lot simple to have less duplication in property fields and names. Smaller objects are also more easy to understand and to work with. I'm already happy that Pester will not break anymore on a broken test in a single file :)

[nohwnd]

I tried to avoid all the dependencies in between fields because then changing one (because of fixing a bug) will change behavior in many other places. For that reason I am trying to keep the data as raw as possible. I can also use that to debug when I get a repro and something is wrong. Passed means that the test passed, but it will be false when the test is skipped or not run for example. This does not mean the test failed though.

If you want to know what exactly happened look at the Result field. That one indicates the actual result.