DisableParallelization ignored when running tests in VisualStudio

[SirDudeMcAwesome]

Most of our tests we want to run in parallel but we are trying to use [CollectionDefinition(DisableParallelization = true)] to prevent parallel execution of some of our tests however even with this attribute in there the tests are still running in parallel. If it helps we are using VS2019 and xunit 2.4.1. Thanks!

[SirDudeMcAwesome]

To clarify what I am trying to do is get it so the tests in one of our classes has its test run at a time when no other tests in the project are running. From the documentation it sounds like that is what [CollectionDefinition(DisableParallelization = true)] is supposed to do.

[bradwilson]

Can you provide a simple repro? Does it only fail inside of Visual Studio?

[SirDudeMcAwesome]

I haven't tried to repro it outside of Visual Studio but it repro's consistently in VS. Here is a simple repro. Run the tests below individually and they pass. Put both in a single test run and they both fail.

[CollectionDefinition("DisableParallelizationTests", DisableParallelization = true)]
public class DisableParallelizationTests
{
    public const int WaitSeconds = 2;
    public static bool TestFailed = false;

    [Fact]
    public async Task ParallelismTest()
    {
        var sw = Stopwatch.StartNew();
        while (sw.Elapsed.TotalSeconds < WaitSeconds)
        {
            if (DisableParallelizationTests2.AnotherTestIsRunning)
            {
                DisableParallelizationTests.TestFailed = true;
                Assert.True(false, "DisableParallelization is not working!!!!!!!!!!!!!!!!");
            }
            await Task.Delay(100);
        }
    }
}

public class DisableParallelizationTests2
{
    public static bool AnotherTestIsRunning = false;

    [Fact]
    public async Task ParallelismTest()
    {
        DisableParallelizationTests2.AnotherTestIsRunning = true;
        var sw = Stopwatch.StartNew();
        while (sw.Elapsed.TotalSeconds < DisableParallelizationTests.WaitSeconds)
        {
            Assert.False(DisableParallelizationTests.TestFailed, "DisableParallelization is not working!!!!!!!!!!!!!!!!");
            await Task.Delay(100);
        }
        DisableParallelizationTests2.AnotherTestIsRunning = false;
    }
}

[bradwilson]

Ah, I see the problem. You've put [CollectionDefinition] on your test class, but it should be on the test collection class (basically a dummy class used to define the collection).

This code works as expected:

using System.Diagnostics;
using System.Threading.Tasks;
using Xunit;

[CollectionDefinition("DisableParallelizationTests", DisableParallelization = true)]
public class DisableParallelizationTestsCollection {}

[Collection("DisableParallelizationTests")]
public class DisableParallelizationTests
{
    public const int WaitSeconds = 2;
    public static bool TestFailed = false;

    [Fact]
    public async Task ParallelismTest()
    {
        var sw = Stopwatch.StartNew();
        while (sw.Elapsed.TotalSeconds < WaitSeconds)
        {
            if (DisableParallelizationTests2.AnotherTestIsRunning)
            {
                DisableParallelizationTests.TestFailed = true;
                Assert.True(false, "DisableParallelization is not working!!!!!!!!!!!!!!!!");
            }
            await Task.Delay(100);
        }
    }
}

public class DisableParallelizationTests2
{
    public static bool AnotherTestIsRunning = false;

    [Fact]
    public async Task ParallelismTest()
    {
        DisableParallelizationTests2.AnotherTestIsRunning = true;
        var sw = Stopwatch.StartNew();
        while (sw.Elapsed.TotalSeconds < DisableParallelizationTests.WaitSeconds)
        {
            Assert.False(DisableParallelizationTests.TestFailed, "DisableParallelization is not working!!!!!!!!!!!!!!!!");
            await Task.Delay(100);
        }
        DisableParallelizationTests2.AnotherTestIsRunning = false;
    }
}

[SirDudeMcAwesome]

Ah of course user error on my part. Now its working. Thanks!