Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel test execution is incompatible with Micorsoft Fakes' shims #244

Closed
kkm000 opened this issue Dec 22, 2014 · 8 comments

Comments

@kkm000
Copy link

commented Dec 22, 2014

I have 2 test classes using Microsoft Fakes with quite a similar setup. The following 3 lines shim a file load function and force loading of the set of fake files provided.

    var files = new Dictionary<string, string>() {  // Maps file name to contents
        { "1", "[142]\nv=0\n[143]\nv=0"},
. . . .
    using (ShimsContext.Create()) {
        ShimFile.LoadString = s => files[s];
        var ef = ExperimentFactory.Load(files.Keys, .......);
. . . .

Another test class uses a very similar setup, shimming the same File.Load() function. When both tests are run, files from one shim set up end up loaded in another test (I can clearly tell that from faked file names, which differ in the tests).

Is there a way to force sequential test execution?

EDIT: I tried adding

[assembly:CollectionBehavior(DisableTestParallelization = true)]

to the test assembly, but that did not change anything.

EDIT2: But

[assembly:CollectionBehavior(MaxParallelThreads = 1)]

did it! Does DisableTestParallelization supposed to do what I think it supposed to do?

@bradwilson

This comment has been minimized.

Copy link
Member

commented Dec 30, 2014

DisableTestParallelization turns off any attempt to run tests in parallel in the same assembly. This forces all tests in the given assembly to run sequentially.

MaxParallelThreads of 1 does not disable parallelism, but it does limit Tasks to a single thread (all parallelization in xUnit.net is done with the Task Parallel Library). This would have the effect of attempting to run all tests in parallel, but with only one active thread of execution. In point of fact, this could cause tests to run interleaved with one another, but never at the same time (though that would require your tests to be async, since using await would be the only way to "switch" the test that is running).

To sum up: I would have expected to see the exact opposite effects that you're saying you're seeing.

Where is the shimming code, exactly? Is it inside the test method, or is it somewhere else? If the Fakes library uses thread locals to perform its work, and its done so in a non-TPL-compatible way, then I could see how setting MaxParallelThreads to 1 might solve your problem, because it would guarantee that all test code ran on the exact same thread.

Do you have a repro you can provide?

@wholroyd

This comment has been minimized.

Copy link

commented Jan 17, 2015

I can reproduce this issue without using Fakes. I have just short of 2000 tests and this causes about ~560 to fail, when switching to a single thread only leaves about 50 failing (which is what I should expect). Give me to mid next week to see if I can't create a sample project.

@wholroyd

This comment has been minimized.

Copy link

commented Jan 17, 2015

I figured it out.

In my scenario through the use of AutoMapper and @kkm000's use of the Fakes framework, there are configuration settings held within a static variable in each of those frameworks. Tests within a single class aren't run parallelized to each other, but multiple classes are run asynchronously to each other (assuming I understand the xunit behavior correctly), which is where that static configuration gets out of sync between the multiple running classes' tests.

I'll upload a sample project sometime this weekend.

@kkm000

This comment has been minimized.

Copy link
Author

commented Jan 17, 2015

@wholroyd, I would appreciate if you do! I am swamped in other things, and I do not know when I could get back to building a minimum sample for @bradwilson.

@wholroyd

This comment has been minimized.

Copy link

commented Jan 18, 2015

@kkm000, no worries. The project is at https://github.com/wholroyd/xunit.parallelization-repro and you can see here the current state of the CollectionBehaviorAttribute with DisableTestParallelization enabled with multiple tests in current execution in the test explorer...

xunit

@wholroyd

This comment has been minimized.

Copy link

commented Jan 20, 2015

@kkm000 - Xunit has a few different ways of performing parallelized unit tests. Parallelization boundaries are defined by collection attributes, where collections are performed in parallel to each other. However, tests within a collection are always guaranteed to occur synchronously.

If you don't utilize the collection attributes, a 'catch all' collection is used under the covers. By default, this collection is generated on a per-class basis. You can change that behavior by generating the collection at the assembly level through using this attribute instead...

[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]

This change is another option that you have if you don't want to completely de-parallelize everything and not bomb your tests. Every assembly in your solution will execute using the de facto standard of per-class, but then switch to per-assembly level collections when using this attribute on selected projects. Assemblies by default do not run in parallel to each other, but can be tweaked via Xunit config files on a per-assembly basis. I didn't see a way via attributes in code to allow this.

@bradwilson

This comment has been minimized.

Copy link
Member

commented Feb 12, 2015

There's no bug here.

You're mutating shared state. That's never going to work with test parallelization. This isn't a bug with test parallelization; it's a bug with the fact that they're expecting it to work with mutable shared state. Unless you protect that mutable shared state (or get rid of it), you will be unable to parallelize tests.

I'm not sure whether this truly illustrates the problem with the Fakes support or not, since I don't have source for that. I will say that, given xUnit.net's heavy reliance on Task-based parallelization, that any code which mutates shared state, or attempts to use thread local storage, it bound to fail.

@wholroyd

This comment has been minimized.

Copy link

commented Mar 9, 2015

@kkm000 - Does this issue still occur after upgrading to RC2 or later? The repro I had posted above now works as expected in the change from #276 which shipped in RC2. So the weirdness I was seeing without even using Fakes is now gone.

I completely understand and agree with @bradwilson's stance on not supporting Fakes, but I'm personally just curious to see if your problem is now gone.

wieslawsoltes added a commit to wieslawsoltes/Avalonia that referenced this issue Jun 8, 2017

wieslawsoltes added a commit to wieslawsoltes/Avalonia that referenced this issue Jun 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.