RFC - Pester v. 3.x - an maintenance plan

[it-praktyk]

I would like to start a discussion about a future of the Pester 3.x branch.

IMHO - because Pester 3.4.0 is a part of Windows 10 and Windows Server 2016 - we should still maintain it for some time.

Maintain means - for me

• resolve major bugs what are know or will be discovered
• backport features what doesn't require major refactor to Pester v 3.x code
• update documentation
• analyze and accept code from external contributors if an amount of work for that will be reasonable

How long? I propose clearly communicate deprecation period - at least 12 months.

[alx9r]

This raises a number of questions in my mind as follows:

1. Are updates to Pester deployed via Windows Updates and/or WSUS? If so, how is that decision made?
2. At what point will Windows Update and/or WSUS deploy Pester 4.x? At that point, what will happen to instances of Pester 3.x that were deployed as part of Windows 10 and Server 2016?

[it-praktyk]

@SteveL-MSFT, would you like to join the discussion too?

[it-praktyk]

@alx9r, I can't answer your questions. Those decisions are outside of the Pester team.

[nohwnd]

@it-praktyk Pester being shipped with Windows is cool, but I haven't heard about anyone who actually uses the version shipped with Windows, and if they do, we can easily migrate their test base. ( MSFT has it's own fork of Pester that works on PSCore, so the they could migrate only when Pester is compatible with PSCore, which is not gonna happen for Pester 3. ) So I would vote for Pester 3 being marked as obsolete, without maintenance support. I can make a poll on Twitter to ask if people still use 3 and are unwilling to migrate, but I hope there will be 1% of such people with 200 tests.

@alx9r Dunno, but maybe @JamesWTruher could shed some light on this?:)

[SteveL-MSFT]

@it-praktyk yes, please keep me in the loop.

We put Pester in Windows to help bootstrap users and ensure they can leverage Pester since it's available, but our expectation is that most users would want to get the latest version from PowerShellGallery. I would be fine if it is declared that Pester 3 is obsolete, but it will take some time to replace it in Windows.

[it-praktyk]

Thank you all for your comments.

I propose to release Pester 3.4.7 and include in it a warning (displayed under importing it) that the module is obsoleted.

Update README.md and include information about an obsoleting, a lack of support (?), link to migration.

Do you agree?

[it-praktyk]

I would like to return to a discussion. What do you think about my previous proposal?

[SteveL-MSFT]

@it-praktyk proposal seems fine to me, no work in immediate future to replace what we have in Windows. What I'd like to do in the future is to have a stub implementation of Pester in Windows, so if it's invoked, it tells the user to install it from PSGallery. I don't think we can just call Install-Module on the user's behalf since 4.x breaks 3.x tests.

[alx9r]

@it-praktyk Your proposal seems fine to me. I haven't been able to think of a compelling reason to do much maintenance on 3.x. Do you have an example we should expect? It seems like it would be a rather extraordinary situation where the cost of moving a project to 4.x is higher than fixing 3.x.

[nohwnd]

Tweeted it out so we will see. I doubt there will be many problems. The migration script worked well for us, and we test a lot of weird stuff and edge cases, so I hope it works for others.

[it-praktyk]

In the next few days, I'll update

• the migration script provided by @indented-automation; I'll add a detection/correction an used charset
• Pester wiki with information about that script

[MathieuBuisson]

Unfortunately, I had to revert to Pester 3.4.6 at work.
This is because the test result XML file generated by Pester 4.x seems to mess up ReportUnit's reports.
Each test is duplicated multiple times.
I think it has something to do with Describe or Context blocks being seen as separate test suites.

They have an issue about it :
reportunit/reportunit#110

But I think it is more likely due to the changes made Pester's NUnit XML format in version 4.x.
Perhaps related to this :
#571

[nohwnd]

@MathieuBuisson could you provide a sample project so I can try it? I worked with report unit few times so maybe I can get it fixed. Or at least see what the problem is.

[MathieuBuisson]

Hi @nohwnd ,

I created a new repository to reproduce and test the problem.
I used the script repro.ps1 to generate sample output with different ReportUnit versions and test it with Pester version 3.4.6 and 4.08.
You can use the same script to test things out.

In the directory SampleOutPut, you can find the Pester test result files and the HTML files ReportUnit generated based on these XML files.
To make it easy to track how the files were generated, the file naming structure is :
${DateTime}_${ReportUnitVersion}_${PesterVersion}.

We can see that when the HTML reports based on test result files generated by Pester 3.4.6 show :

• 12 tests (which is consistent with the number of It blocks in the test script)
• 3 test suites (which is consistent with the number of Describe blocks in the test script)

We can see that when the HTML reports based on test result files generated by Pester 4.0.8 show :

• 12 tests for ReportUnit version 1.2.0
• 48 tests for ReportUnit version 1.5.0-beta (lots of duplicates)
• 11 test suites (which is not consistent with the number of Describe blocks in the test script)

Let me know if you have any question about this repro.
Hopefully, this will help figure out what is going on.

[dlwyatt]

Looks to me like ReportUnit doesn't handle nested TestFixtures properly. In Pester v4, we make use of that structure to allow for any number of nested Describe or Context blocks (plus we log a fixture per script file, and one root fixture for the entire run of Invoke-Pester.)

[it-praktyk]

The pull request #947 created.

[nohwnd]

@it-praktyk will release the version 3 one last time if I figure out how to pack it and release it and then I close this.

[nohwnd]

Aaaand I never did, and no-one else ever complained.