Invoke-Pester reports negative exit code, despite: Exit = $false

[xenadmin]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

I use Pester in my Packer build process, to check if all programs I wanted to install, are existing in the master image. Even if tests fail, I need the packer build to continue. As of now, as soon as one test fails, packer reports the number of failed tests as exit code. This is interpreted as an error by Packer and the build process breaks.
As to my understanding, the Pester Configuration for Exit and Throw should handle that:

Exit: Exit with non-zero exit code when the test run fails.

I've set that to $false, see code below. But I've also tested true, and it always reboots the number of failed tests. May I do something wrong, but it doesn't seem to work as I would expect.
Now I help myself, as the last line in my Pester script is just exit 0, which works around that issue.

(I interpret the errors itself later in the pipeline, I want it to break after all, but not at this step of the build pipeline.)

Expected Behavior

I would expect Pester configuration to include a switch, to turn on or off, if the number of failed tests is reported as an exit code or not. Like the documentation already suggests -> https://pester.dev/docs/commands/New-PesterConfiguration

Steps To Reproduce

Write-Verbose "Using Pester version $((Get-Module -Name Pester -ListAvailable | Select-Object -First 1).Version)" -Verbose
Get-ChildItem "$env:downloadcache" -Recurse

$pesterConfiguration = @{
    Run        = @{
        Path  = @("$env:downloadcache\")
        Exit  = $false
        Throw = $false
    }
    Should     = @{
        ErrorAction = 'Continue'
    }
    TestResult = @{
        Enabled      = $true
        OutputFormat = 'NUnitXml'
        OutputPath   = "$env:downloadcache\testResults.xml"
    }
}

#Invoke pester with the configuration hashtable
$config = New-PesterConfiguration -Hashtable $pesterConfiguration
Invoke-Pester -Configuration $config

Describe your environment

PS C:\Users\xenadmin> (Invoke-WebRequest -Uri "https://git.io/JTinj" -UseBasicParsing).Content | Invoke-Expression
Pester version     : 3.4.0 C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1
PowerShell version : 7.3.3
OS version         : Microsoft Windows NT 10.0.19045.0

Windows 10 22H2 Enterprise Multi User on Azure
PowerShell 7.3.x installed
Latest Pester 5.x from PowerShell Gallery

During my build pipeline, it's logged as:

Write-Verbose "Using Pester version $((Get-Module -Name Pester -ListAvailable | Select-Object -First 1).Version)" -Verbose
VERBOSE: Using Pester version 5.4.1

Possible Solution?

[fflaten]

Hi. Do you get an error message? -1 is for runtime errors, so something went wrong, not just a failed test.

Failed tests would set a positive exitcode. Run.Exit will also exit the script, but when false Pester will still set the exit code to number of failed tests to avoid inheriting a bad code from a previous command.

[xenadmin]

I don't get errors. I just want Pester to honor the Exit = $false switch (giving the fact I do in fact understand, correctly, what it's supposed to do). To give this some more input, I did more tests and screenshots. Four tests:

Code which has been tested:

$pesterConfiguration = @{
    Run        = @{
        Path  = @("$env:downloadcache\")
        Exit  = $false/$true
        Throw = $false
    }
    Should     = @{
        ErrorAction = 'Continue'
    }
    TestResult = @{
        Enabled      = $true
        OutputFormat = 'NUnitXml'
        OutputPath   = "$env:downloadcache\testResults.xml"
    }
}

#Invoke pester with the configuration hashtable
$config = New-PesterConfiguration -Hashtable $pesterConfiguration
Invoke-Pester -Configuration $config

Write-Host $LASTEXITCODE

1. Exit = $false
   No errors during the tests
   EXITCODE = 0
   
2. Exit = $false
   Errors during the test
   EXITCODE = Number of failed tests
   
3. Exit = $true
   No errors during the tests
   EXITCODE = 0
   
4. Exit = $true
   Errors during the test
   EXITCODE = null
   

So after my test, it might be the case, that I still don't fully understand the docs. Or the docs are wrong, but not the code.
Test 4 seems to be the variant, of what I want for my use-case. There are errors, but it doesn't throw any exit code at all. Which is not what I expected, but also what should serve my purpose.
Which leaves the question, if the description of the switch is wrong?
I will try a new Packer build with Exit = $true in the Pester configuration, and I will skip my workaround of exit 0 as the last line of my script.

[fflaten]

Test 4 actually sets the exit code, you're just not seeing it because you invoked Pester through a script.

Run.Exit = $true will run exit $numberOfFailedTestsEtc. If you're inside a script exit kills the script, if not it kills the PowerShell-process itself. So in Test 4 the script is stopped before your Write-Host $LASTEXITCODE line.

If you write $LASTEXITCODE in the console after test 4 you'd still see the same value as in test 2.

I don't get errors. I just want Pester to honor the Exit = $false switch (giving the fact I do in fact understand, correctly, what it's supposed to do).

Which leaves the question, if the description of the switch is wrong?

It does honor the option. Docs says Run.Exit will exit when the test run fails = skip any remaining code in the script in your scenario. So it's not wrong, but I'm sure it could be improved somehow.

Summary:

• $LASTEXITCODE is always set, either to 0 (all good) or to number of failing tests. As mentioned above, this is to avoid leaving it at the value set by a command prior to Pester.
• Run.Exit only affects whether a script or process continues after a failed Pester run or if it will be stopped. This is useful to abort a CI-pipeline or other post-processing, publish/release etc. when the code is not in a good state.

[nohwnd]

Adding to fflaten answer, the exit code is figured out here:

https://github.com/pester/Pester/blob/main/src/Main.ps1#L1294-L1316

Setting the exit code to non-zero when tests fail even when we won't exit explicitly is by design. You don't always want to exit right away, for example if you run Invoke-Pester as part of a test.ps1 script that has some steps after running tests. Unless overwritten the $LASTEXITCODE will flow outside that script and stop your CI (at least AzDO stops at non-0 exit code by default).

To work around your issue you can ignore Pester exit code by doing the same thing we do to ignore any previous exit codes, you set:

$global:LASTEXITCODE = 0