Start Pester clean

[DarkLite1]

General summary of the issue

We're having a bit of a strange issue that's hard to replicate but happens once in a while. When we start the terminal fresh, all tests pass. However, once we run the same tests again, they might fail. I believe it has something to do with setting the stage and not cleaning up stale mocks from a past run.

After testing further I've noticed that the issue only occurs when Invoke-Pester is called within a file.

# T:\RunPesterTest.ps1
$path = 'T:\myScript.Tests.ps1'
Invoke-Pester $path -Output Detailed

When copy pasting the code above in the terminal, all works fine. However, when invoking the file (& 'T:\RunPesterTest.ps1' ) that contains the same code from the terminal , all or most Pester tests fail on the second run. This happens in vscode, PowerShell ISE, ....

Describe your environment

Pester version     : 5.1.1 C:\Program Files\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1
PowerShell version : 5.1.14393.3866
OS version         : Microsoft Windows NT 10.0.14393.0

Steps to reproduce

Reproduction is really hard as it's not constant. But sometimes we even interrupt the Pester process as we see notices we 'forgot' something in the code and run it again. At times like these the chance of this happening is bigger.

Expected Behavior

Make sure the stage is clean before attempting to run any tests.

Current Behavior

On a clean run, when nothing from Pester ran before, all is green. On subsequent runs it appears that some code fails that was passing the previous tests.

Possible Solution? (optional)

Clean the stage, Pester environment, mocks. .. before every invocation of Invoke-Pester.

[nohwnd]

I think there was an issue that Mock only cleans up the mocks in current session state on startup, but not in all loaded modules, so when you use mock with -ModuleName and cancel the run it will keep the mock in the module, causing issues for the future. Can't really think of an easy way to handle this, without some startup overhead by inspecting all loaded modules.

Maybe add an Id on Pester state and use that in mock definition, then when Pester sees that it resolved to a previously defined mock it can check if it was the current pester that created the mock and re-define it. But that would not solve the original issue of having a mock in scope while a real command should have been used instead.

[DarkLite1]

I don't know the internals of Pester at all, but something related happened in the Pester 4. Maybe this approach can be considered again? Or maybe it has nothing to do with this issue at all. Just wanted to help you.

Is it an idea to maybe start a new PowerShell session every time Invoke-Pester is called? This way the stage will always be clean and there's no before and after cleanup to be done.. Something like this:

Function Invoke-Pester {
    $scriptBlock = {
        'Pester test logic'
    }
    Invoke-Command -ScriptBlock $scriptBlock
}

Invoke-Pester

[nohwnd]

That clean up is happening, problem is that it does not look into modules, so if you use -ModuleName and cancel in the middle it might leave Mock bootstrap function ins the module. At least that is the theory.

Pester/src/functions/Mock.ps1

Line 1485 in da5f776

function Remove-MockFunctionsAndAliases {

Maybe breaking the rule and adding a global variable (or maybe static property on a static type, because types are unexportable, so it is as good as global var), where we write all the mocks that were setup to persist the info across runs and across Pester module import an export. That way future Pester invocation can look in that table and tear all of them down.

[DarkLite1]

A global variable for this purpose seems to be ok to me. I always try to avoid them but I can understand that you need some kine of 'state' between different Pester invocations. I'm just trying to understand why 'state' is needed for Pester.

In normal situations one invokes Invoke-Pester on one file and gets the result of the run. When using Invoke-Pester on a directory with multiple .Tests.ps1 files, Pester also returns a summary of the results. Maybe I'm not understanding it completely but would something like this not work:

Function Invoke-Pester {
    Param (
        [String[]]$Path
    )

    Function Start-PesterTest {
        Param (
            [String]$Path
        )
        Invoke-Command -ScriptBlock {
            Write-Verbose "Start pester test on '$Path' and return result"
            [PSCustomObject]@{
                Path   = $Path
                Result = @{
                    Passed = $true
                    Tests  = 5
                }
            }
        }
    }
    
    $results = foreach ($P in $Path) {
        Start-PesterTest -Path $P
    }

    $results
}

Invoke-Pester

Just collect the results in an array? Don't know, I might just be totally besides the point. But when all results are collected in an array there's no need for a global variable I think. Can you not use an array for the mocks that is only know inside the function Invoke-Pester? The module part might indeed be an issue, don't have a clue about hos to something like that. Just wanted to share maybe some ideas that could help.