Parallel test execution is incompatible with Micorsoft Fakes' shims

[kkm000]

I have 2 test classes using Microsoft Fakes with quite a similar setup. The following 3 lines shim a file load function and force loading of the set of fake files provided.

    var files = new Dictionary<string, string>() {  // Maps file name to contents
        { "1", "[142]\nv=0\n[143]\nv=0"},
. . . .
    using (ShimsContext.Create()) {
        ShimFile.LoadString = s => files[s];
        var ef = ExperimentFactory.Load(files.Keys, .......);
. . . .

Another test class uses a very similar setup, shimming the same File.Load() function. When both tests are run, files from one shim set up end up loaded in another test (I can clearly tell that from faked file names, which differ in the tests).

Is there a way to force sequential test execution?

EDIT: I tried adding

[assembly:CollectionBehavior(DisableTestParallelization = true)]

to the test assembly, but that did not change anything.

EDIT2: But

[assembly:CollectionBehavior(MaxParallelThreads = 1)]

did it! Does DisableTestParallelization supposed to do what I think it supposed to do?

[bradwilson]

DisableTestParallelization turns off any attempt to run tests in parallel in the same assembly. This forces all tests in the given assembly to run sequentially.

MaxParallelThreads of 1 does not disable parallelism, but it does limit Tasks to a single thread (all parallelization in xUnit.net is done with the Task Parallel Library). This would have the effect of attempting to run all tests in parallel, but with only one active thread of execution. In point of fact, this could cause tests to run interleaved with one another, but never at the same time (though that would require your tests to be async, since using await would be the only way to "switch" the test that is running).

To sum up: I would have expected to see the exact opposite effects that you're saying you're seeing.

Where is the shimming code, exactly? Is it inside the test method, or is it somewhere else? If the Fakes library uses thread locals to perform its work, and its done so in a non-TPL-compatible way, then I could see how setting MaxParallelThreads to 1 might solve your problem, because it would guarantee that all test code ran on the exact same thread.

Do you have a repro you can provide?

[wholroyd]

I can reproduce this issue without using Fakes. I have just short of 2000 tests and this causes about ~560 to fail, when switching to a single thread only leaves about 50 failing (which is what I should expect). Give me to mid next week to see if I can't create a sample project.

[wholroyd]

I figured it out.

In my scenario through the use of AutoMapper and @kkm000's use of the Fakes framework, there are configuration settings held within a static variable in each of those frameworks. Tests within a single class aren't run parallelized to each other, but multiple classes are run asynchronously to each other (assuming I understand the xunit behavior correctly), which is where that static configuration gets out of sync between the multiple running classes' tests.

I'll upload a sample project sometime this weekend.

[kkm000]

@wholroyd, I would appreciate if you do! I am swamped in other things, and I do not know when I could get back to building a minimum sample for @bradwilson.

[wholroyd]

@kkm000, no worries. The project is at https://github.com/wholroyd/xunit.parallelization-repro and you can see here the current state of the CollectionBehaviorAttribute with DisableTestParallelization enabled with multiple tests in current execution in the test explorer...

[wholroyd]

@kkm000 - Xunit has a few different ways of performing parallelized unit tests. Parallelization boundaries are defined by collection attributes, where collections are performed in parallel to each other. However, tests within a collection are always guaranteed to occur synchronously.

If you don't utilize the collection attributes, a 'catch all' collection is used under the covers. By default, this collection is generated on a per-class basis. You can change that behavior by generating the collection at the assembly level through using this attribute instead...

[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]

This change is another option that you have if you don't want to completely de-parallelize everything and not bomb your tests. Every assembly in your solution will execute using the de facto standard of per-class, but then switch to per-assembly level collections when using this attribute on selected projects. Assemblies by default do not run in parallel to each other, but can be tweaked via Xunit config files on a per-assembly basis. I didn't see a way via attributes in code to allow this.

[bradwilson]

There's no bug here.

You're mutating shared state. That's never going to work with test parallelization. This isn't a bug with test parallelization; it's a bug with the fact that they're expecting it to work with mutable shared state. Unless you protect that mutable shared state (or get rid of it), you will be unable to parallelize tests.

I'm not sure whether this truly illustrates the problem with the Fakes support or not, since I don't have source for that. I will say that, given xUnit.net's heavy reliance on Task-based parallelization, that any code which mutates shared state, or attempts to use thread local storage, it bound to fail.

[wholroyd]

@kkm000 - Does this issue still occur after upgrading to RC2 or later? The repro I had posted above now works as expected in the change from #276 which shipped in RC2. So the weirdness I was seeing without even using Fakes is now gone.

I completely understand and agree with @bradwilson's stance on not supporting Fakes, but I'm personally just curious to see if your problem is now gone.