Output Azure DevOps Logging Commands in Passthru to help find failed tests and exceptions

[o-o00o-o]

1. General summary of the issue

We are running Pester as part of a Deploy Pipeline on Azure DevOps, in hundreds of tests if an issue happens in one, it is very difficult to find in full log output.

2. Describe Your Environment

Pester version : 4.8.1 C:\Users\myusername\OneDrive - Value Retail Plc\Documents\WindowsPowerShell\Modules\Pester\4.8.1\Pester.psd1
PowerShell version : 5.1.17763.592
OS version : Microsoft Windows NT 10.0.17763.0

3. Expected Behavior

Azure DevOps provides some help for this. https://docs.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash

If you write to the output with a tag

##vso[task.logissue]error/warning message

It will highlight this in red and make it easy to find.

4.Current Behavior

example log output I'm seeing today

Executing script C:\vstsagent\A1_work\r1\a_VegaDW-CI\Deployment Package\test\suites\character-test\v2\GL Forecast\FactGLBudgetBalance.Tests.ps1
2019-09-17T16:17:47.3761564Z
2019-09-17T16:17:47.3775598Z Describing GL Forecast
2019-09-17T16:17:47.4150045Z [+] should have the correct sql count value in dbo.FactGLBudgetBalance 69ms
2019-09-17T16:17:47.4931722Z [+] should have the correct sql values in dbo.FactGLBudgetBalance 78ms
2019-09-17T16:17:47.4996016Z VERBOSE: Connecting to server:
2019-09-17T16:17:47.5006349Z "Provider=MSOLAP;DataSource=asazure://westeurope.asazure.windows.net/weu0assrv0dev0vegadw01;Integrated
2019-09-17T16:17:47.5017659Z Security=ClaimsToken; Database="VR_BI_1864"; Connect Timeout=30; Connect Timeout=5"
2019-09-17T16:17:47.5340229Z [-] Should have the correct cube values - GL Forecast by ForecastType 36ms
2019-09-17T16:17:47.6117988Z ArgumentException: Authentication failed: User ID and Password are required when user interface is not available.
2019-09-17T16:17:47.6130509Z at , C:\vstsagent\A1_work\r1\a_VegaDW-CI\Deployment Package\test\suites\character-test\v2\GL Forecast\FactGLBudgetBalance.Tests.ps1: line 71

No colouring or anything in the output to distinguish errors from normal log output that the ##vso pattern would provide

5. Possible Solution

using ##vso tags as indicated above. Perhaps with a -ADO flag to ask pester inject the appropriate text at the beginning of the exception/failure stdout output?

Any other ideas on how I could make this happen as is or will it need a code change?

6. Context

It is often useful to see the details in the log rather than using the Log output.

[nohwnd]

Personally I just search the output for [-] but I understand that seeing a red message would be nice. 🙂 I think that rather than adding another switch we should make this automatic via environment variable detection. I am pretty sure the build server defines some kind of variable that says where the test is running, and we could grab that on the module import.

(hopefully because then it is the least expensive, because we just do it once, rather than on every output. because you need to check if the variable exists and if it has a value, and that is not as cheap as just asking for a value of a variable. This is a safe, but I think a bit expensive way of doing this. But in this case it does not matter that much because it happens "only" when test fails which is not that often.

Pester/Functions/Output.ps1

Line 541 in f4bab35

if ($ExecutionContext.SessionState.PSVariable.GetValue("PesterDebugPreference_ShowFullErrors")) {

)

Anyways if you'd like to make a PR for this I would accept it.

[fflaten]

Possibly tracked in new issue #1953 if ANSI coloring is acceptable as a generic solution. If not, then this is still open for AzDO-specific output

[nohwnd]

Marked this as good first issue. Implementer should check env variable on start to detect we are running in CI, add an option to enable the AzDO autodetection to the configuration object for Output, probably enable this when -CI parameter is used, and document this change.

[ArmaanMcleod]

Marked this as good first issue. Implementer should check env variable on start to detect we are running in CI, add an option to enable the AzDO autodetection to the configuration object for Output, probably enable this when -CI parameter is used, and document this change.

I'd be happy to look at this.

I currently face the same issue when running Pester tests inside AzDO pipelines, and seeing red errors would make debugging a lot easier when using CI.

When the option is enabled, would need to use Write-Host "##vso[task.logissue type=error]<error message> or Write-Host "##[error] <error message>" to log errors in red, as shown here https://docs.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=powershell#example-log-an-error.

[nohwnd]

@ArmaanMcleod it's all yours.