Data driven tests incorrectly generate the $null case

[DominikJaniec]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

There is a great feature, where one can generate many tests based on an array of examples using -ForEach or -TestCases parameters. However, for the $null case, the test does not get that $null under $_ variable.

Expected Behavior

Generated parametrized test when executing should obtain given value under $_ - i.e. $true when test cases array contains $true, a 42 if there is that number, and the $null when list contains that.

Moreover, templated name of it, should present $null (or something "meaningful") and not System.Collections.Hashtable as currently. I'm not sure if the $null is the-correct-name, as currently -ForEach renders simple values like:

• @() -> ​ - a little bit "unmeaning"
• @{} -> System.Collections.Hashtable
• $null -> System.Collections.Hashtable
• $true / $false -> True / False
  • which is how PowerShell prints them, but it's kind of inconsistent with how (e.g.) Should -BeTrue renders them as: Expected $true, but got $false.

Steps To Reproduce

Have a test like this one:

Describe "Pester 'ForEach' support" {
    It "provides `$_ with '<_>'" -ForEach @($null) {
        $null -eq $_ | Should -BeTrue
    }
}

Then, when executed:

Running tests from 'C:\Users\domin\Test.Tests.ps1'
Describing Pester 'ForEach' support
  [-] provides $_ with 'System.Collections.Hashtable' 15ms (6ms|10ms)
   Expected $true, but got $false.
   at $null -eq $_ | Should -BeTrue, C:\Users\domin\Test.Tests.ps1:3
   at <ScriptBlock>, C:\Users\domin\Test.Tests.ps1:3
Tests completed in 141ms
Tests Passed: 0, Failed: 1, Skipped: 0 NotRun: 0

Describe your environment

Pester version     : 5.4.0 C:\Users\domin\Documents\PowerShell\Modules\Pester\5.4.0\Pester.psm1
PowerShell version : 7.3.3
OS version         : Microsoft Windows NT 10.0.19044.0

Possible Solution?

No response

[DominikJaniec]

I've had found that behavior of -ForEach @($null) is even more unexpected, when it happens within InModuleScope 😕, as it looks like the $_ is not provided at all.

---

Let's have a magic-module.psm1 nearby with content like:

param($makeStrict)

if ($makeStrict) {
    Set-StrictMode -Version Latest
    Write-Host "latest StrictMode!"
}

Write-Host "magic module imported"

Then, with a test file like:

BeforeDiscovery {
    $makeModuleStrict = $false
    # $makeModuleStrict = $true
    Import-Module "$PSScriptRoot\magic-module.psm1" `
        -Force -ArgumentList $makeModuleStrict
}

Describe "Pester 'ForEach' support" {
    InModuleScope "magic-module" {
        It "provides `$_ with: '<_>'" -ForEach @($null) {
            Test-Path variable:_ | Should -BeTrue
        }
    }
}

Then, output will show that $_ is not available:

Describing Pester 'ForEach' support
  [-] provides $_ with: '' 16ms (10ms|6ms)
   Expected $true, but got $false.
   at Test-Path variable:_ | Should -BeTrue, C:\Users\domin\Test.Tests.ps1:11
   at <ScriptBlock>, C:\Users\domin\Repos\Test.Tests.ps1:11

And, when module under investigation sets strict-mode (i.e. $makeModuleStrict = $true in our case), the $null test case is not correctly generated and fails like:

Describing Pester 'ForEach' support
  [-] provides $_ with: '<_>' 6ms (2ms|4ms)
   RuntimeException: The variable '$_' cannot be retrieved because it has not been set.
   at <ScriptBlock>, <No file>:1

[fflaten]

Thanks for the detailed issue and repro!

$null -> System.Collections.Hashtable

This hashtable is actually inherited. In your example it's the default empty hashtable Data for the container/testfile. If your testfile began with param($foo = 'bar') or you used -ForEach on your Describe-block it would be more obvious.

$null values provided to -ForEach are currently ignored as it usually means -ForEach/TestCases was not used.

We should probably differentiate between no data or a null-value inside an array. 🙂

[fflaten]

I've had found that behavior of -ForEach @($null) is even more unexpected, when it happens within InModuleScope 😕, as it looks like the $_ is not provided at all.

When you use InModuleScope you go into a different session state (the module's state) which has it's own scopes. So you won't inherit variables like $_ from parent blocks or container outside it. This is by design.

[DominikJaniec]

Thanks for very fast response 🙂

I'm not sure if I understood you correctly, in regard to InModuleScope case, but other values from -ForEach are provided correctly. As in my previous example with a "magic-module", let's expand the data set a little bit more, and assert "correctness" of value under $_:

Describe "Pester 'ForEach' support" {
    InModuleScope "magic-module" {
        It "provides `$_ with: '<_>'" -ForEach `
            @("stuff", $null, 42, @(), $false) `
        {
            Test-Path variable:_ | Should -BeTrue
            @("stuff", $null, 42, @(), $false) `
            | Should -Contain $_
        }
    }
}

Then, the result for $null will be as I've shown before - depending on strictness mode. However, other values are passed without issues, and theirs tests will pass:

Describing Pester 'ForEach' support
  [+] provides $_ with: 'stuff' 11ms (2ms|9ms)
  [-] provides $_ with: '' 3ms (3ms|1ms)
   Expected $true, but got $false.
   at Test-Path variable:_ | Should -BeTrue, C:\Users\domin\Test.Tests.ps1:13
   at <ScriptBlock>, C:\Users\domin\Test.Tests.ps1:13
  [+] provides $_ with: '42' 5ms (4ms|1ms)
  [+] provides $_ with: '' 15ms (15ms|1ms)
  [+] provides $_ with: 'False' 3ms (2ms|1ms)

Moreover, I'm not sure what should change there with -ForEach on Describe as results are basically the same. On the other side, I see what you mean with param($foo = 'bar'), but only once I've printed it in tests and saw that $_ in the $null case became: [foo, bar]. As I understand, this is related to "providing external data to tests", but I can't grasp why this is substituted under $_ for the $null case 🙁 Does that mean the $_ is not "created" for that test case, and it stays the automatic $_ variable?

---

$null values provided to -ForEach are currently ignored as it usually means -ForEach/TestCases was not used.

Well to be clear, as in my use-case, I have a list of different values, where the $null is one of them. Whereas I believe, ignoring and skipping whole "tests-block", is caused by -ForEach $null or ForEach @(), which is going to throw in v6 according to: #2151 (comment)

[DominikJaniec]

PS / Off Topic: I've learned one strange thing with PowerShell itself today 😕, as I was writing the comment above, and wanted to have $true as one of the test cases.

The thing is: a list with $true in it will -contains almost anything when it visually does not contain that, with exception to $null, $false, 0, "", or an empty list @() - i.e. anything which when cast to [bool] will produce $false. If one is curious, please check this test example below, which currently only does not fail as expected for the $null case:

Describe "This is PowerShell" {
    It "makes a list with `$true in it, to contains almost anything, like: '<_>'" -ForEach `
    @($false, "other", "", 24, @(), 3.14, [DateTime]"1989-05-15", @{ x = "y" }, 0, $null) {
        $theList = @("someting", 42, $(Get-Date), @(1, 2, 3))
        $theList -contains $_ | Should -BeFalse
        $theList += $true
        $theList -contains $_ | Should -BeTrue
    }
}

[fflaten]

I'm not sure if I understood you correctly, in regard to InModuleScope case, but other values from -ForEach are provided correctly.

The other values are not $null, which is specifically ignored atm = we never define $_ = $null. So the other non-null values are unaffected by the bug (current design).

PowerShell use scopes to isolate local changes to variables. Every function/scriptblock runs in a new child scope by default. The new scope inherits variables from parent scopes, but any changes are lost when the scope is removed (function/scriptblock is done). Pester follows the same pattern. When It doesn't set a new $_ value (because we ignore $null), it contains the value from a parent block. In this case it was the empty default hashtable for the container/file itself.

All scopes are associated with a session state. A session state is a isolated "world" with it's own set of scopes. They can't see each other (except global-scoped variables). Each module has it's own state (and scopes). When you invoke a function from a module or use InModuleScope, it runs in the module state. That code is unable to inherit anything from scopes outside it (like the default hashtable). Demo-time:

Get-Module Demo | Remove-Module
New-Module Demo { } | Import-Module

# Invoking scriptblock in new scope to avoid defining a global variable (that's the exception to the rule)
& {
    $myvar = 'outer scope value'
    Write-Host "Running in session state '$($ExecutionContext.SessionState.Module)' - myvar is: '$myvar'"

    & {
        Write-Host "Running in session state '$($ExecutionContext.SessionState.Module)' - myvar is inherited: '$myvar'"
        $myvar = 'inner scope value'
        Write-Host "Running in session state '$($ExecutionContext.SessionState.Module)' - myvar is changed in current scope: '$myvar'"

        InModuleScope Demo {
            Write-Host "Running in session state '$($ExecutionContext.SessionState.Module)' - myvar is not visible: '$myvar'"
        }
    }
}

# output -  state '' below means "script state" (default console session)
Running in session state '' - myvar is: 'outer scope value'
Running in session state '' - myvar is inherited: 'outer scope value'
Running in session state '' - myvar is changed in current scope: 'inner scope value'
Running in session state 'Demo' - myvar is not visible: ''

Well to be clear, as in my use-case, I have a list of different values, where the $null is one of them. Whereas I believe, ignoring and skipping whole "tests-block", is caused by -ForEach $null or ForEach @(), which is going to throw in v6 according to: #2151 (comment)

Yes, they are not the same scenario. Your testcase isn't skipped, but the value is just not mapped to $_ atm. As mentioned, that's because Pester can't tell the difference between value not provided (-ForEach <something> wasn't used) and "current item is $null". That's the bug.

As a workaround for now, you can use a hashtable per testcase. Then you can access the value with $KeyName or $_.KeyName. Ex:

It "provides `$_ with: '<ValueToTest>'" -ForEach @(
    @{ ValueToTest = 'stuff' },
    @{ ValueToTest = $null },
    @{ ValueToTest = 42 },
    @{ ValueToTest = @() },
    @{ ValueToTest = $false }) {
    Test-Path variable:ValueToTest | Should -BeTrue
        @('stuff', $null, 42, @(), $false) | Should -Contain $ValueToTest
}

[DominikJaniec]

Thanks for your detailed explanation 🙂

Yes, I've made that workaround too, and I think that using items like:

@{ Tested = $whatever; What = "useful and human-readable" }

is often even better than just pure values, as one has a "full-control" over test cases name.