Error when running multiple concurrent pester test suites on same machine

[sundeep-ssi]

If 2 separate powershell sessions are concurrently running a pester test suite on the same machine with the same account, pester will throw this error when running a test:

Task should be Sucessful(The term temp is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.)

It seems that the 2 sessions are both trying to put dynamically compiled code into the same Temp directory, and so are clobbering each other. This came up b/c we're running pester tests from Jenkins and often doing multiple simultaneous runs of different test suites.

[anweiss]

New to code contributions myself but maybe an idea to resolve this would be to add an additional function that checks whether or not files in $TestDrive are in use by other tests that have been previously invoked.

Maybe something like this:

function Check-FileLock([IO.FileInfo]$file)
{
    [IO.FileStream]$stream
    try
    {
        $stream = $file.Open([IO.FileMode]::Open(), [IO.FileAccess]::ReadWrite(), [IO.FileShare]::None())
    }
    catch [IOException]
    {
        return true;
    }
    finally
    {
        if ($stream) { $stream.Close() }
    }
    return false;
}

This could then be called against files in the PSDrive in the Setup function in Setup.ps1 after Initialize-Setup is called. I don't see anywhere else in the code base where there is any sort of asynchronous tracking of actively running tests. I guess that is all handled by whatever build system is invoking the tests?

I just forked the repo but haven't created any additional branches or submitted a Pull Request yet for this. Just adding thoughts first.

[scottmuc]

I'm not sure what to think about this. Adding the ability for parallelization is usually a pathway to complexity. Then again, I think running a test runner in multiple processes is a reasonable expectation. I'm going to reference this from the mailing list to get more feedback.

This is something I would like to have a significant integration test suite due to the shape of problems that can result from developing this.

[nohwnd]

If Pester is invoked in separate processes and all the test files are placed to TestDrive, then changing the TestDrive location to random temp folder should do the trick, doesn't it?

[scottmuc]

@nohwnd I believe so, just trying not to jump to any solutions right away.

[anweiss]

@scottmuc @nohwnd in the case of invoking Pester tests, what do you mean by separate processes? are you referring to the use of jobs, separate runspaces, etc? or simply an invocation of multiple commands within the same runspace?

[nohwnd]

@anweiss I meant separate PowerShell processes, so it can mean using two poweshell consoles for example or jobs. Not sure about runspaces, I know runspaces run in separate threads but I think the global scope is shared - and the test-drive is set to be global. How do you invoke parallel commands in the same runspace?

[anweiss]

@nohwnd I meant the use of Runspace Pools to allow for "pseudo-multithreading" functionality...albeit way more involved and complicated than something that might work better for Pester

[scottmuc]

@anweiss I was thinking of the example of a Jenkins agent that has multiple workers, which could potentially run pester in 2 processes. Adding multiple runspaces sort of scares me. I like that Pester is simple (perhaps overly simple) but that's ok with me.

[anweiss]

@scottmuc couldn't agree more...the fact that Pester is driven without the use of compiled code is a win. The speed at which the majority of Pester tests are conducted is fast enough where complex multithreading would have no added benefit

[rismoney]

I am not sure how complex it would have to be.

i have thought of using workflows since there is a parallel nature (including foreach -parallel) built into them. I just haven't had a use case to warrant it. Plus its still a hype feature of powershell vs mainstream imho.

[anweiss]

@rismoney workflows are actually becoming more prevalent and is definitely another option to throw out there. There are certainly caveats though; even with v4, for example, I wouldn't be surprised if a significant number of InclineScript blocks had to be used to compensate for some of the inadequacies of WWF. As far as I know, for every invocation of an InlineScript block, a new PSRemoting session is opened, even if it's against localhost.

[scottmuc]

Trying to tie everything together in this thread. The original issue is related to the fact that the $TestDrive is not unique per Pester invocation, correct?

The curious bit is around compiled code. What compiled code is involved here? Essentially I would like to focus on fixing your problem first, and then figure out how the Pester architecture can be improved.

[anweiss]

I don't think any compiled code is involved in this case. Per the discussion on the Google group thread, it's simply a matter of identifying uses of $TestDrive that would cause issues when invoking Pester in separate processes

[sundeep-ssi]

Pester dumps the contents of each IT block in a file called $TestDrive/temp.ps1, then executes that temp.ps1 script (see below). This is what I meant by dynamically executed code - and the temp.ps1 files getting clobbered by 2 running pester jobs is what caused my issue. I think

you're right that randomizing $TestDrive would take care of this problem and any other conflicts.

Also my 2 cents is I'd like to have pester stay simple because we already manage the complexity of parallel jobs in Jenkins, where it's easy to administer since we have other tests that aren't pester-based.

function It {
...
param(
$name,
[ScriptBlock] $test
)
$pester.results = Get-GlobalTestResults
$pester.results.TestCount += 1

Setup-TestFunction
. $TestDrive\temp.ps1