Community talk about Pester 5

[nohwnd]

TLDR;

Personally I would love to do a lot of breaking changes in the release of v5, because the advantages of the new features are far outweighting the downsides. But I understand that people have a sizeable test suites that would be very difficult to migrate to v5, so a side-by-side support for v4 would be needed.

Which route to choose:

• Better compatibility with Pester v4, at the price of less features now, and in the future, but simpler migration, and no future support for Pester v4.
• Less compatibility with Pester v4, but a lot of new functionality and possibilities, but more difficult migration. And x years of patching v4 but no new features.
• Some other option?

New features

Dropping PowerShell v2 support

The supported versions of PowerShell will change from PowerShell 2+ to PowerShell 3+ (including 3) in Pester v5. Dropping support for v2 is long overdue, and it does not make sense to keep dragging this burden into the next version of Pester. Pester v4 still supports v2, but I plan to cap it at version 4.9.x, and make future updates to Pester v4 only support PowerShell 3+.

Test discovery

Discovering which tests are in a given *.Tests.ps1 file was something that I wanted Pester to do since I joined the project some 5 years ago. And now it can finally do it. The price to pay for it is resonably small, you need to put all your code into Pester controlled blocks. On the other hand you got a lot of benefits:

• Better filtering that has potential to work on any aspect of the test or block, and already works based on Tags on Describe, Context and It.
• Running only setups and teardowns that need to run. When there are no tests in the block, the setup for it will be skipped, and the same for a file, when no tests in that file passed the test, then the whole file is simply skipped.

Better result object

The whole internal architecture is now centered around a hierarachical result object, that looks something like this, but with way more information, and more nested:

@{
    Name = "d1"
    ScriptBlock = ...
    Blocks = @( ... blocks within this block )
    Tests = @(
        @{
            Name = "t1"
            ScriptBlock = { throw “Fail!...
        }
    )
}

Such structure reflects the actual layout of your test file much better, because there is one level per describe, and allows for much easier and more flexible post processing. For example a file with:

Describe "d1" { 
    Describe "d2" {
        It "i1" {
            $true | Should -BeFalse
        }
    }
}

Would produce a result object that contains three levels, 1st level describing the file itsels, 2nd level describing Describe d1, and 3rd level describing Describe d2.

So using $result.Blocks[0].Blocks[0].Tests shows metadata of test i1, including if it passed, the error record associated with it, the time it took to run, and so on.

Operational state is stored in the result object

This tree structure is also used by Pester to store it's internal state, so the state that used to be runtime-only is stored in here as well. After execution you can inspect the object and see for example which mocks were defined in this test, how many calls were made to that mock or other. This brings a much better post-mortem capabilities because storing this result object will allow you to inspect the run without debugging it live.

Corrected scoping of Describe

Describe "d1" {
    BeforeAll {
        $a = 10 
    }
    It "i1" {
        $true
    } 
}

Describe "d2" {
    It "i2" {
        $a | Should -Be 10
    }
}

Test such as this will pass in Pester v4, because BeforeAll is invoked before the Describe in which it is defined. This means that the variable $a will be available for the whole subsequent script, leading to unexpected results, because the script scope is polluted by the variables.

Describing d1
  [+] i1 1ms

Describing d2
  [+] i2 5ms

Schematically it looks like this, . means import into the current scope and & run in new scope:

. { <# BeforeAll d1 #> }
& { <# Describe d1 #> } 
& { <# Describe d2 #> }

In Pester v5, the scoping is corrected. One more scope is added above Describe so d1 BeforeAll is isolated and does not leak variables into the script, and subsequest Describe.

Describing d1
    [+] i1 47ms (10ms|37ms)

Describing d2
    [-] i2 112ms (104ms|8ms)
      Expected 10, but got $null.
      12:         $a | Should -Be 10
Tests completed in 611ms
Tests Passed: 1, Failed: 1, Skipped: 0, Pending: 0, Inconclusive: 0 

Schematically it looks like this, notice that each describe is contained in another scope:

&{
    . { <# BeforeAll d1 #>}
    & { <# Describe d1 #> } 
}

& { 
    & { <# Describe d2 #> }
}

Corrected scoping of It

Describe "d1" {
    BeforeEach { $be = 1 }
    It "i1" {  
        $be = 2
    }
    AfterEach { Write-Host "AfterEach: $be" }
}

In Pester v4 the BeforeEach is executed in a scope above It, so variables defined in BeforeEach are not writable from It and not changed in AfterEach, this makes sharing data between the test and the teardown difficult. For example when you created a file in the test and want to remove it, or close a connection to a database etc.

Notice that the value of variable $be is not changed.

Describing d1
AfterEach: 1 1
  [+] i1 2ms
AfterAll: 1

Schematically it works like this, the one extra scope around It is causing the problem:

&{
    . { <# BeforeEach #>}
    & { <# It i1 #> }
    . { AfterEach }
}

In Pester v5 BeforeEach It and AfterEach all run in the same scope, notice that $be changed to 2:

Describing d1
AfterEach: 2
    [+] i1 33ms (3ms|30ms)
Tests completed in 90ms

Schematically:

&{
    . { <# BeforeEach #>}
    . { <# It i1 #> }
    . { AfterEach }
}

Overall the idea here is that, the two following should be equivalent, so if multiple tests have the same setup or teardown you can extract the setup or teardown to BeforeEach / AfterEach without changing the behavior:

It 'i1' {
    try {
        $resource = Get-Resource
    }
    finally {
        if ($null -ne $resource)  {
            Remove-Resource $resource
        }
    }
}

Should be equivalent to:

It 'i1' {
    $resource = Get-Resource
}
AfterEach {
    if ($null -ne $resource)  {
        Remove-Resource $resource
    }
}

Fixed scoping of Mock

function f () { "real f" }
function g () { "real g"}
Describe "d1" {
    BeforeAll { 
        Mock g { "mock g" }
    }
    It "i1" {
        Mock f { "mock f" } 
        Write-Host "in i1 $(f), $(g)"
    }
    It "i1" {
        Write-Host "in i2 $(f), $(g)"
    }
}

In Pester v4 the mock defined in the It block is defined on the containing Describe, so it leaks to the subsequest It blocks. This is a hurdle that everyone learning Pester faces. It also forces the test author to add unnecessary Context blocks when the mock should be "isolated" to one It block.

Describing d1
in i1 mock f, mock g
  [+] i1 30ms
in i2 mock f, mock g
  [+] i1 19ms

In Pester v5 the Mock is defined on the block that contains it. So putting it within It will define this mock for that It and putting it into BeforeAll will define it for the Describe/Context that contains it. Leading to a more obvious behavior.

Notice that the function f is mocked only in the first It and the function g is mocked in both because the mock is defined on the Describe level.

Describing d1
in i1 mock f, mock g
    [+] i1 111ms (105ms|6ms)
in i2 real f, mock g
    [+] i1 24ms (19ms|6ms)

Mock assertion name change

Should is the way to do assertions, yet in Pester v4 there are two notably different assertions, Assert-MockCalled and Assert-VerifiableMock, in Pester v5 these commands are deprecated and Should -Invoke and Should -InvokeVerifiable are newly available, and recommended to use.

Describe "d1" {
    It "i1" {
        Mock Get-Command { }
        Get-Command -CommandName abc
        Should -Invoke Get-Command -Times 1 -Exactly
    }
}

Proposal: Compound assertions as default & default PassThru on assertions

Writing tests with multiple assertions is not-recommended because it gives you just partial information on failure. The way to do this correctly is to write multiple It blocks, each having just a single assertion. Or we could allow assertions to tell the framework that there was an error, but not actually fail. This combined with outputting the asserted object, and maybe some extra assertions, would allow for chained assertions that only fail when the test is done, not at the first failed assertion:

Describe "d1" { 
    It "i1" {
        $user = Get-User

        $user | Should -NotBeNullOrEmpty -ErrorAction Stop

        $user | 
            Should -HaveProperty Name -Value "Jakub" |
            Should -HaveProperty Age  -Value 30 
    }
}

So here the first assertion would behave like the normal Pester assertion, it would fail the whole test, because it uses -ErrorAction Stop. The next two assertions would inform the framework that there is a new error and the test would fail in the end, but both of the assertions would run, not just the first one to fail.

I think this leads to a very readable tests, and would be a huge improvement over the current approach.

🙋‍ Add other stuff, suggest your own.

[nohwnd]

We had a discussion, and the consensus was that everybody prefers to go with the new features, even though the migration will be more difficult. And that Pester v4 will only receive patches.

There were multiple questions:

What features would be impossible without breaking the compatibility?

That would be everything connected to the discovery, better filtering, the new result object, and audit etc.

Why drop PowerShell v2 support

PowerShell 2 support in Pester is slowing the development down and making contributions difficult because not everyone has experience with it. So it's about time.

Are compound assertions just an option or something that will be there?

It was discussed and I think it will be a killer feature to have, especially for environment testing. So it will be in v5.

_ I might have missed some of the questions, I can't remember more than those. _

Have another opinion? Let me know.

[Jawz84]

There was also the question about speed. Would keeping Pester 4 compatibility make Pester 5 faster or slower. The answer was that at this point in time, Pester 5 is a little slower, especially around Mocks, but it's early implementation. There is still room for improvement in terms of speed. Converting the core functionality into a binary module to speed things up is also a possibility under consideration.

[fourpastmidnight]

I'm concerned about Gherkin support 😉 . I'm not saying that I don't want to move forward with v5 and the breaking changes that entails. I'm in favor of the breaking changes. But, a lot of these new features are being developed without any regard to Gherkin (as I perceive things, and, if my perception is correct, understandably so).

It begs the question: what is the future of Gherkin within the Pester framework? Through my usage of the Gherkin runner, I have found that, while the concepts between Gherkin-style and RSpec-style tests are similar, they are not similar enough to share the same code. In v4, this has manifested when trying to use mocks, and they just don't work the same as they do for Pester proper. I've had to write a lot of custom code (which works well, but it was a lot of work) to work around the issue. It's not a fix, it's merely a workaround. Also, the way in which Gherkin-style tests are executed is inefficient compared to Pester. (That may be helped by some v5 features--but I also have a branch/PR that is attempting to resolve this issue specifically for Gherkin).

But, will the new features being developed for Pester-proper work for Gherkin-style tests? Or are these two testing styles different enough (as I suspect they are) that, realistically, Gherkin should be maintained almost as a separate "sub-framework" within Pester? (If I had a "Magic Button", the Pester proper would be a PowerShell test runner/executor, and there would be an RSpec and Gherkin "plug-ins" that the runner would use to execute the tests.)

[nohwnd]

@fourpastmidnight I don't know much about Gherkin. But right now there are the internals and the rspec layer over it, the mocking has separate implementation as well where it is separated into two parts so I am thinking about Gherkin but I am not doing any implementations towards it right now as I want to get the rspec version shipped first. As I said few times, once v5 is out let's talk about this, then I will have a good idea of what is possible and how to use it for gherkin 🙂

[TigerBoom]

I understand the desire to drop PowerShell v2 support in Pester v5 to aid future development and agree with that approach. However, I would suggest not capping PowerShell v2 support at any point in Pester v4. It is so great to have testing available for PowerShell v2, as I still come across old dusty customer servers running v2.

As far as I understand it, you don't plan new features for Pester v4, so I would hope that the pain of supporting PowerShell v2 should be minimal and restricted to bug fixes. Apologies if I'm off the mark.

[nohwnd]

@TigerBoom that sounds reasonable to me. :)

[tmack8080]

What about PowerShell v7 support in Pester v5?

[nohwnd]

@tmack8080 did not try it yet, need to research if we can already build it somewhere on one of the build servers or if I need to provision a vm for it. I guess it helps that PowerShell itself is using Pester to test it, so if it won't work we hopefully will hear about it from the PowerShell team.

[nohwnd]

@tmack8080 asked it on twitter and people successfully used Pester v4 on PowerShell 7. The goal is to make Pester available on any version and any platform so its users can test all their code no matter where it runs.

[iSazonov]

Is there "Whats new in Pester 5.0" docs? Or all news present in the issue?

[nohwnd]

@iSazonov see this readme: https://github.com/pester/Pester/blob/v5.0/README.md

The content will move to release notes after release and the old readme will come back. Also there is more info in the releases already: https://github.com/pester/Pester/releases