Idea: Ability to make failed tests show up in the run summary as warnings instead of errors in Azure DevOps run summary

[erikgraa]

Checklist

• Feature request has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

Summary of the feature request

If Pester >= 5.3 is run in Azure DevOps, the run summary gets errors, i.e.

Would it be possible to introduce a parameter that lets you choose whether these are shown as errors or warnings, i.e. ##vso[task.LogIssue type=warning;]?

How should it work?

$conf = New-PesterConfiguration

$conf.Output

Verbosity : [..]
StackTraceVerbosity : [..]
CIFormat : [..]
CISummaryMode : Whether to show CI output as errors or warnings, options are Error and Warning (Error, default: Error)
RenderMode : [..] 

[fflaten]

Thanks for the suggestion! Definitely possible and compatible with both CIs.

This is just me being curious: In what scenario would you use this?

[erikgraa]

Hi!

My idea was to use "warning" as guidelines or pointers for less dramatic "failures", e.g. when you are linting code using PSScriptAnalyzer and get suggestions to not use e.g Write-Host.

At least in my mind it seems counterintuitive to have a bunch of red errors in the run summary that won't stop the build because they are less serious "errors".

[fflaten]

Thanks, good point. This would be a run-specific setting, so then you'd likely want to run Pester in two steps:

• Once for linting/style-tests where it's configured for warnings. Step would also need continueOnError/ignoreLASTEXITCODE because Pester would still give non-zero exit code on failing tests.
• Once for normal tests where I assume you'd want errors due to broken functionality or quality-issue like missing help.

For PSSA specifically I'd generally recommend a separate step in CI for that. There's also prebuilt actions in marketplace for it. We use a dedicated workflow/pipeline for it here.

[erikgraa]

Thanks, good point. This would be a run-specific setting, so then you'd likely want to run Pester in two steps:

• Once for linting/style-tests where it's configured for warnings. Step would also need continueOnError/ignoreLASTEXITCODE because Pester would still give non-zero exit code on failing tests.
• Once for normal tests where I assume you'd want errors due to broken functionality or quality-issue like missing help.

Exactly!