Invoke-Pester doesn't fail in Azure Devops Build if Invoke-Command shortcut used with invalid path

[asears]

1. General summary of the issue

Using Pester 5.0.2 and Azure Devops Azure Powershell Code cli.

scratch.Tests.ps1

$cmd = "${PSScriptRoot}\.\..\test.ps1"
. $cmd

Script displays an error, however error does not fail pipeline.

2. Describe Your Environment

Running Invoke-Pester with Powershell 7 locally and in Azure Devops.
Azure Powershell 5.0 task
Fail on standard error set
Powershell core
Erroraction Stop set

3. Expected Behavior

Build fails or warns, build breaker on PR

4.Current Behavior

Devops Step succeeds with warning messages shown.

5. Possible Solution

This may not be a Pester issue, could be Devops is masking the error. Is there a way to identify if there was an error returned as part of this script invoke-pester call and we can add a workaround throw?

6. Context

[asears]

#1643 might relate.

[vcalatayud]

Have you tried with Invoke-Pester -CI ?

[fflaten]

@asears Have you tried the suggestion above? The -CI switch enables the Run.Exit configuration option which makes Pester exit with a non-zero exit code when one or more tests failed. The exit code is equal to the sum of failed tests, blocks and containers.

Pester/src/Pester.ps1

Lines 1135 to 1137 in 8d55526

	if ($PesterPreference.Run.Exit.Value -and 'Failed' -eq $run.Result) {
	exit ($run.FailedCount + $run.FailedBlocksCount + $run.FailedContainersCount)
	}

Non-zero exit code will fail the powershell step in a DevOps pipeline.

[nohwnd]

@asears gentle ping on this.

[nohwnd]

Closing for inactivity.

[asears]

Sorry for the delay. No issues closing here! It's probably on the Devops side with possible improvements to how Pester throws errors vs just warns on them in the Devops pipeline, or how CI behavior may be different / support less than non-CI.

I believe I tried CI and at the time encountered some differences / limitations that I can't recall right now.

Perhaps the issue was with code coverage and having multiple paths to cover?

In one of our projects, scripts are structured as:

Project/ScriptCategory1/scriptname1.ps1
Project/ScriptCategory2/scriptname2.ps1

Tests are currently structured as
TestProject/ScriptCategory1/scriptname1.tests.ps1
TestProject/ScriptCategory2/scriptname2.tests.ps1

There's about 40+ categories of scripts.

I was running blocks of tests separately due to discovery phase being slow, priority of tests, and was passing each set of categories in as a block.

There was some challenges getting test results and code coverage consolidated, outputted and published, and merged in Devops with this pattern.

I'll see what's currently used in the pipeline.

[nohwnd]

Not sure how to help here, we need a proper repro.