Pester CI Pipeline Should Halt on 'BeforeDiscovery' Failure

[Nadiahansen15]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

When running Pester tests in a CI/CD pipeline stage, if the BeforeDiscovery script fails or throws an error for any reason, the Pester test suite should immediately stop and cause the stage to fail. Currently, this behavior is not consistent, as Pester continues execution even when the BeforeDiscovery script fails. This can lead to a false indication that the stage has passed successfully, even though it has encountered a failure.

Expected Behavior

When the BeforeDiscovery script fails during Pester test execution in a CI/CD pipeline, Pester should immediately halt further tests and return a non-zero exit code, indicating that the stage has failed

Steps To Reproduce

• Set up a CI/CD pipeline that includes Pester tests.
• Introduce an error or failure condition in the BeforeDiscovery script.
• Run the pipeline.

Describe your environment

pesterversion = '5.3.1'
CI/CD System = 'azure pipelines'

Possible Solution?

Implement a mechanism within Pester that checks the exit code of the BeforeDiscovery script and ensures that if it fails, the entire Pester test suite is aborted and the pipeline stage is marked as failed.

[fflaten]

Thanks for the feature request.

This can lead to a false indication that the stage has passed successfully, even though it has encountered a failure.

$LASTEXITCODE will always be non-zero when a container has failed, but you might have to check it in your step? Try the Run.Exit option (or -CI switch if you don't use advanced configuration) which will exit the powershell-prosess with the exit code.

It won't exit though, but it will skip the file where BeforeDiscovery failed.

Implement a mechanism within Pester that checks the exit code of the BeforeDiscovery script and ensures that if it fails, the entire Pester test suite is aborted and the pipeline stage is marked as failed.

Not sure if you meant it this way, but checking exit codes from console applications etc. are unlikely to be supported. Atm. we only consider PowerShell exceptions and terminating errors to be a failure, which will stop the current file only.

[Nadiahansen15]

When I execute the tests and encounter a failure in the "beforeDiscovery" phase, the system reports a failed container with the following message:

Container failed: 1
/azp/_work/1/s/gov/tests/there.Tests.ps1

Interestingly, even though the tests within that container are not executed, the pipeline does not mark it as a failure. As a result, we won't be alerted to any potential issues with the tests unless we manually inspect them after each run.

However, I've observed that if a failure occurs in the "beforeAll" phase, it is correctly flagged as a test failure.

Is it possible to make "beforeDiscovery" cause the test to fail or take an action that flags it in the CI pipeline stage, so that the stage doesn't return as "all good"?

[fflaten]

IIRC there's no difference between BeforeAll and BeforeDisovery regarding this behavior. So I'm not sure why one would work differently. Hard to troubleshoot without seeing the specific pipeline.

As mentioned the exit code will be positive (non-zero) when either block types fail. Ex.

# demoCI.ps1
param([switch]$FailFirst = $false)

$sbFailed = {
    BeforeDiscovery {
        throw 'oh nooo'
    }
    Describe 'Fail in BeforeDiscovery' {
        It 'never runs' { 1 | Should -Be 1 }
    }
}

$sbWorks = {
    Describe 'Working file' {
        It 'works' { 1 | Should -Be 1 }
    }
}

$conf = New-PesterConfiguration
$conf.Run.ScriptBlock = @(if ($FailFirst) { $sbFailed }) + @($sbWorks)
# This exits the script/powershell with the exit code.
# Without it you'd usually have to check $LASTEXITCODE after Invoke-Pester and throw/exit if non-zero.
$conf.Run.Exit = $true 
$conf.Output.Verbosity = 'None' # To shorten output

Invoke-Pester -Configuration $conf

Demo:

# Run with only successful container ($sbWorks)
> $proc = Start-Process pwsh -ArgumentList "-NoProfile -File ./demoCI.ps1" -PassThru -Wait
> $proc.ExitCode
0

# Run with failing container first ($sbFailed, $sbWorks)
> $proc = Start-Process pwsh -ArgumentList "-NoProfile -File ./demoCI.ps1 -FailFirst" -PassThru -Wait
> $proc.ExitCode
1 # This would usually fail a pipeline step unless configured to continue on error. YMMW depending on CI