Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In-assembly parallel execution #142

Closed
pvlakshm opened this issue Apr 7, 2017 · 75 comments

Comments

@pvlakshm
Copy link
Contributor

commented Apr 7, 2017

Description

Enable tests runs to complete faster by allowing tests within a single assembly to execute in parallel.

Please see RFC here: 004-In-Assembly-Parallel-Execution

@pvlakshm pvlakshm referenced this issue Apr 7, 2017

Open

MSTest V2 backlog #144

9 of 16 tasks complete

@AbhitejJohn AbhitejJohn modified the milestone: S117 Apr 14, 2017

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

Here is the RFC for this. Feedback is welcome.

@andersforsgren

This comment has been minimized.

Copy link

commented Apr 26, 2017

Should parallel test execution not (at least optionally) be possible for non parallel-ready code? A container with 1000 tests that can't run in parallel can be divided in chunks and run sequentially in N appdomains/processes.

@rajbos

This comment has been minimized.

Copy link

commented Apr 26, 2017

We have been using ‪ https://github.com/sscobici/ParallelTestRunner‬ for this for a while now. Can split on Class level, Method level and group tests via annotations. Would be very nice to have this build in, so there is no longer a need to run an executable to split the tests, run them and then summarize the testresults.

@AdamDotNet

This comment has been minimized.

Copy link

commented May 1, 2017

I'm all for letting tests run in parallel, especially since it doesn't seem to work right now in MSTestV2, at least in Visual Studio 2015. The RFC linked mentions VSTest is capable of allowing all test providers to do test parallelization by spawning multiple processes.

I know this feature is for in assembly parallelization, but for my sake, can someone confirm to me that regular old VSTest multiprocess parallelization isn't meant to work right now in MSTestV2? I would appreciate the clarity. Thank you!

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 2, 2017

@AdamDotNet : The container level parallelization for vstest works irrespective of the Framework(mstestv2/mstest/xunit/nunit). If your solution has multiple test projects, it should spawn a host process for each test container and run them in parallel when enabled just like this blog details. This only works if there is more than one test container. You aren't seeing that happen in VS 2015?

@AdamDotNet

This comment has been minimized.

Copy link

commented May 2, 2017

@AbhitejJohn Thanks for the information. I never realized that multiprocess parallelism was dependent on having multiple projects. I just observed it working in a sample project with two test projects.

Back to this feature of in assembly parallelism, if I'm reading the RFC right, you can mark an assembly with how you want it to handle parallelism in that assembly. I'm imagining in my solution, it would be generally acceptable to mark the assembly as parallel by class by default, but it would be nice if certain classes could run faster by running the test methods in a particular class in parallel.

So, will it be possible to default an assembly to class level parallelization, but you can override that behavior by placing attributes on a specific class to enable method parallelization for that test class? Will this work with only one test project?

To illustrate, I have one test project with the desire for the following behavior:

  • Classes that contain tests that are totally in memory using Moq: run as class parallelization because each test takes only a handful of milliseconds to run.
  • Classes that contain a number of tests that integrate with other pieces: run as method parallelization because each test takes about 5 seconds to run.
@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 4, 2017

@AdamDotNet : That's a great point. This is essentially a mix and match of multiple modes in the same assembly. Our thought process was that integration/End-to-End tests would actually be split into a different assembly from unit tests, whose test mode can then be configured accordingly.

That said, for the scenario described above, as long as the fast running tests within a test class do not interfere with each other, the mode for the assembly can be made method level as a whole. That ensures that all methods irrespective of what class they are under are run in parallel.

However this does look like a valid ask and could surface more when we add in custom modes. The only choice that the RFC provides today is to opt-out of the default mode for an assembly. We hope that these scenarios however are relatively fewer in number and come under a more advanced set of capabilities that we can add in the future. Do let us know otherwise.

@JoshCollinsMSFT

This comment has been minimized.

Copy link

commented May 4, 2017

This is something that is incredibly frustrating to my team. This seems like a regression in behavior from the parallelization control we got from the testsettings file. In fact, my team is still currently are using a testsettings file to get that level of parallelization of our tests because one of our test categories takes around 3 hours to complete without it, and only 20 minutes to complete with it (with a setting of parallelTestCount="18").

Unfortunately we're banging our heads against this problem right now because without using a runsettings file we can't easily use test parameters, and using a runsettings file we can't get parallel execution.

The sooner we can get support for this in MSTest while using a runsettings file the better.

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 5, 2017

Thanks for letting us know @JoshCollinsMSFT . We are in the phase of collecting feedback on the RFC that Pratap posted above. Do let us know if that meets your requirements. We plan to start off on this in the weeks to come.

@JoshCollinsMSFT

This comment has been minimized.

Copy link

commented May 8, 2017

@AbhitejJohn How do we provide feedback to the RFC?

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 9, 2017

@JoshCollinsMSFT : you can do so here :)

@JoshCollinsMSFT

This comment has been minimized.

Copy link

commented May 11, 2017

The RFC looks good. The only comments I have are that I hope the runsettings file will allow you to set both TestParallelization as well as TestParallelizationLevel.

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 12, 2017

@JoshCollinsMSFT : We deliberately left that out. In our thought process TestParallelizationLevel is environment dependent and not-really tied to a test container. Users would want to change this globally and want that to take precedence over what is specified in an assembly based on what environment they are run on.
TestParallelization on the other hand is specific to a test container. Users would explicitly add that into a container, as opposed to letting the default kick in, because that specific container functions best in that mode. One would not also want a global setting to override the assembly specification in this case. Adding in a global default which unlike TestParallelizationLevel does not take precedence over the assembly specification would cause more confusion. Hence we left that out.

@JoshCollinsMSFT

This comment has been minimized.

Copy link

commented May 12, 2017

We were hoping for it to be here to reduce the amount of refactoring necessary. However, I can buy the argument that it would be too confusing or allow someone to easily break their test runs.

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 12, 2017

@JoshCollinsMSFT : I see. Then I think the question would be, "what is the right default?". Is parallelization at the TestClass level where all TestClasses are running in parallel and methods in them are running serially the default for most scenarios? In your case, what would work best?

@ChristopherHaws

This comment has been minimized.

Copy link

commented May 12, 2017

@AbhitejJohn Maybe the default should be assembly level so that there is no change of users? Then users could opt in to parallelization. This would prevent this feature from being a breaking change.

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 12, 2017

@ChristopherHaws : Oh yes, in-assembly parallelization is off by default. so this wouldn't be a breaking change. From the RFC this would primarily be turned on via the TestParallelizationLevel setting in either the runsettings or at an assembly level. When we have this turned on, the question was whether running all TestClasses in parallel was the right default as opposed to (say) running all test methods, irrespective of what TestClass they belong to, in parallel. The former seemed the most convenient pivot to parallelize in terms of minimal inter-dependency of tests and minimal refactoring required to enable this.

@ChristopherHaws

This comment has been minimized.

Copy link

commented May 16, 2017

@AbhitejJohn As a user, I would have assumed the default was the most parallel (TestMethod) if TestParallelization is not explicitly set. I think that doing it any other way would simply be confusing to a new user.

I also think it's good idea to start out with the highest level of parallelism so that new test projects get authored in the most parallel manor (aka discourage the use of statics in test classes). I feel like the TestParallelization setting should be used as a mechanism for decreasing the level of parallelization due to legacy code.

Also, is there talk about having a way to specify that multiple tests must not run in parallel together, but can still run in parallel with other tests? In the SpecRun test adapter you can assign trait to tests and you can configure your test threads to only run certain categories.

In MSTest, it would be nice to be able to specify that certain traits must run sequentially together, but can run in parallel with other tests. I could see it looking something like:

// Run tests in CategoryA sequentially in one thread.
[assembly:TestParallelizationCategoryAffinity("CategoryA");

// Run test in CategoryB sequentially in one thread.
[assembly:TestParallelizationCategoryAffinity("CategoryB");

// Run tests in CategoryC and CategoryD sequentially in one thread.
[assembly:TestParallelizationCategoryAffinity("CategoryC", "CategoryD", ...);

With something like this, you could accomplish test runs like this:

Thread 0: Test1WithNoCategory -> Test2WithNoCategory -> Test3WithNoCategory
Thread 1: Test1WithCategoryA  -> Test2WithCategoryA  -> Test1WithCategoryB  -> Test2WithCategoryB
Thread 2: Test1WithCategoryC  -> Test1WithCategoryD  -> Test2WithCategoryC  -> Test2WithCategoryD
@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented May 16, 2017

@ChristopherHaws : Thanks for that feedback. I see your point on maximum parallelization and I think it does make sense. That would probably be a better default.

On the grouping of parallelizable tests, we did give this some thought and this was falling more into an advanced phase of this support. We plan to wait for more feedback before heading that direction.

/cc @pvlakshm

@ivonin

This comment has been minimized.

Copy link

commented Sep 1, 2017

Any news on this?

Am I correct in my understanding that there is currently no way to use MSTest v2 and run tests from the single assembly in parallel?

As I see it, MS Test v2 doesn't support .testsettings config which has a parallelTestCount option and
.runsettings configuration doesn't allow that.

Should we avoid migrating to MS Test v2 as long as this is the case if parallel test execution is required?

@Steinblock

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2017

@ivonin

  • Create a new Solution
  • Add a class library "Class Library1"
  • Add MSTest.TestAdapter and MSTest.TestFramework via nuget
  • Add a file Class1.cs with the following content
    [TestClass]
    public class TestClass1
    {
        [TestMethod]
        public void TestMethod1()
        {
            Thread.Sleep(10000);
        }
        [TestMethod]
        public void TestMethod2()
        {
            Thread.Sleep(10000);
        }
        [TestMethod]
        public void TestMethod3()
        {
            Thread.Sleep(10000);
        }
    }
  • Run your tests. Look at the result:
 ========== Run test finished: 3 run (0:00:31,2914136) ==========

That's a poor result.

  • Now enable parallel test execution
    image
  • Run your tests:
========== Run test finished: 3 run (0:00:30,308171) ==========

Not what you would expect at first. Why: Because the setting only affects different test projects that are run simultaniously in their own appdomains. To verify that:

  • disable parallel test execution
  • add another class library with the mstest packages and file Class1 and build your solution
  • run your tests again
========== Run test finished: 6 run (0:01:00,557928) ==========
  • reenable paralle test execution
  • run your tests again
========== Run test finished: 6 run (0:00:31,7412897) ==========

Now with MSTestV1 you could create a name.runSettings file with the content

<?xml version="1.0" encoding="utf-8"?>  
<RunSettings>  
  <!-- Configurations that affect the Test Framework -->  
  <RunConfiguration>  
    <MaxCpuCount>0</MaxCpuCount>  
  </RunConfiguration>  
</RunSettings>  

MaxCpuCount: 0 = All cpus or 4 to limit to 4 concurrent threads.

This value is ignored by MSTest V2. You can verify this by choosing the file and running your tests again. Still 30 seconds.

I believe just making mstest v2 running tests in parallel would not be the best choice. Currently I am using mstest v1 with multiple test projects in one solution. A normal test run takes 90 seconds. With "enable parallel test execution" I could reduce the time to 30 seconds (the biggest test project takes 30 seconds) which is great.

But if I enable mstest parallel execution with a runSettings file improves execution even more. But the main difference is that this does not isolate the tests. So my test projects use another class library which is not thread save and half my tests fail.

The best solution would be that vstest engine would run every test in it's own isolated test domain if I choose "Run tests in parallel"

And another thing: If I write

        [TestMethod]
        public async Task TestMethod1()
        {
            await Task.Delay(10000);
        }
        [TestMethod]
        public async Task TestMethod2()
        {
            await Task.Delay(10000);
        }
        [TestMethod]
        public async Task TestMethod3()
        {
            await Task.Delay(10000);
        }

I would expect to get a better performance since the scheduler could switch to the next test, but that's not the case. This could be something that the dev team could implement so everybody could write async tests.

So what are your choices?

  • Split your tests into multiple test projects and use "Run tests in parallel"
  • Switch back to mstest v1 (if your test will be thread save)
  • Use a different runner. I don't want to advertise but ncrunch (commercial, I am not related) can isolate your test, covers your code, can be configured to only run impacted test and even run the tests on one or more different machines.
@ivonin

This comment has been minimized.

Copy link

commented Sep 1, 2017

Thanks for your analysis. Yes, that's what I see as my options at the moment.

MaxCpuCount in .runsettings is not the same as parallelTestCount in .testsettings. There is no alternative to the latter in MS Test v2 as far as I can see.

What I want is to hear from the project maintainers whether it's a final design decision or if it's going to change in the future.

@AbhitejJohn

This comment has been minimized.

Copy link
Member

commented Sep 2, 2017

@ivonin: would the workaround from this comment work while we get in-assembly parallelization up?

@ivonin

This comment has been minimized.

Copy link

commented Sep 3, 2017

@AbhitejJohn: it feels very odd to include somethig like this in our testing process, both on developers machines and on a build server. I'll try it in my spare time but it feels like we'll have to stick with MSTest v1 for the time being (which is unfortunate, by the way, because it's one of the things that keeps us from migrating to .NET Core).

@KthProg

This comment has been minimized.

Copy link

commented Dec 22, 2017

My reply to this is: too bad? That functionality already existed. Parallelism between multiple cores is obviously much different than between multiple processes and in terms of speedup only the first provides a real genuine speedup (in fact splitting between processes is much slower, because of the time to switch between processes on the same core). You can easily call the test runner with specific groups of tests from the CLI to run them under multiple processes.

It only took me about 1 day to update my code to be parallelizable and I think it's an extremely useful feature and changing code in that way is not that challenging.

If what you want is to be able to run non-thread safe code in parallel then I think you'd find there is no way for them to do that in a way that handles all cases. They can't just spin up separate app domains to handle my selenium tests. They all work with the same web application and will interfere with each other, so making the code thread-safe is necessary anyways. TLDR; there's no generic way for them to do that that works for everyone's code, it's much better to put the onus on the dev to make it work with parallel execution.

@andersforsgren

This comment has been minimized.

Copy link

commented Dec 22, 2017

splitting between processes is much slower, because of the time to switch between processes on the same core

The idea is running N processes when you have N cores. Obviously the OS does other things at the same time, but the idea is that 1 process runs on each core, and the core stays as busy as possible.

It only took me about 1 day to update my code to be parallelizable and I think it's an extremely useful feature and changing code in that way is not that challenging.

It would take tens of years for me to do across our code base (which is over 200 man years). The test suite of 25000 tests can take 20 minutes to run using a custom test runner that saturates all cores, and 1.5 hours to run in vstest. We are in different situations there, obviously.

it's an extremely useful feature and changing code in that way is not that challenging.

In general it's neither desireable nor possible to make code that won't be executed in parallel (in production) thread safe. For example, changing a Dictionary<K, V> to a ConcurrentDictionary<K,V> (or adding the equivalent locks) introduces a nontrivial performance overhead. And even if you did this, you probably failed in a few places (because this is hard). So you end up having test failures which show up as flaky flip-flopping tests due to race conditions, and that are also false positives because in production you run on a single thread. Not a good situation to be in.

I think you'd find there is no way for them to do that in a way that handles all cases.

I completely understand that. My point is merely that "1 process per core" is conceptually very simple - existing test runners that do this are very little code, and it offers a benefit over all the other threading modes, namely that it runs the production code on multiple without modification - so long as it's independent of external factors (such as web browsers, databases). A typical suite of unit tests will fit this description.

The mode I'm describing is almost exactly the "process parallel" mode that is already described (and has existed for a long time). The suggestion is merely a small and easily implemented improvement - that the unit of parallelism isn't one assembly. All the test runner has to do is split the work in smaller chunks ("virtual test assemblies" if you want) - in order to keep the CPU cores busy.

@KthProg

This comment has been minimized.

Copy link

commented Dec 22, 2017

Honestly I already disagree with my own post lol. Our test suite is not that large and fairly simple. So yes in those cases I could see it being a huge issue. And splitting each process onto a separate thread is certainly possible. I didn't think about things like collections. I'd think that both modes should be offered, and I take your point that conceptually it's not hard to implement for the benefit.

@PureKrome

This comment has been minimized.

Copy link

commented Jan 8, 2018

@aanurca - this might be starting to get a bit offtopic, but .. can you switch from Log4Net to something more modern like Serilog or the built in Logger in .NET Core? Might find things working better, now...

@KthProg

This comment has been minimized.

Copy link

commented Jan 10, 2018

Hey @aanurca . If you are accessing a shared resource like logging to a file, then you may want to add an extension method to your logger class that wraps the log call in a lock like this:

// ... static extension calss def here
static object _fileLock = new object();

lock(_fileLock){
  // ... write to log here
}

There are all kinds of semaphore and locking classes to lock across processes and threads as well, if you look into it you'll find a lot of examples.

@pvlakshm

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2018

This is now available as a Beta on NuGet.
Framework: https://www.nuget.org/packages/MSTest.TestFramework/1.3.0-beta2
Adapter: https://www.nuget.org/packages/MSTest.TestAdapter/1.3.0-beta2

Will blog about it soon.

@AkankshaCreates

This comment has been minimized.

Copy link

commented Jan 22, 2018

I hit the parallelism issue while writing an application and used this solution by taking the aforementioned, prerelease nuget pkgs. The parallelism works as explained (with the appropriate scope in run settings file). I am, however running into a problem where using ClassInitialize attribute is throwing this particular error while running the tests -

Result Message: Method E2ECustomerSimTests.DSATests.Init has wrong signature. Parameter 1 should be of type Microsoft.VisualStudio.TestTools.UnitTesting.TestContext.

Note that I cannot control the version of Microsoft.VisualStudio.TestTools.UnitTesting class because that comes from the aforementioned TestFramework pkg.

My class init method looks like the following (and, as you can see it has the correct syntax) -

    [ClassInitialize()]
    public static async void ClassInit(TestContext tc)
    {
        // random code

        // Wait for DE to receive the entities created
        bool proceedToVerifyInDE = await WaitAndPollDE(sdkHelper);
        if (!proceedToVerifyInDE)
            throw new Exception("Could not find the campaign creatred in WaitAndPollDE within appropriate time");
    }

Nuget version used for TestAdapter and TestFramework - 1.3.0-build-20180116-01
Nuget repository used - https://dotnet.myget.org/F/mstestv2/api/v3/index.json

Any help on fixing this issue? (Happy to provide any other info that you need.)

@jayaranigarg

This comment has been minimized.

Copy link
Member

commented Jan 30, 2018

Talked offline with Akanksha and apparently changing method return type from 'void' to 'Task' did the trick.

@Nytrodyne

This comment has been minimized.

Copy link

commented Feb 2, 2018

Wow! It's like the SAS of testing "Sweet And Simple". Great job guys.

@Steinblock

This comment has been minimized.

Copy link
Contributor

commented Feb 2, 2018

Just wanne share some results.

Solution with 1172 tests (all passing)

  • Mstest V1 test run: 0:12:33 / all tests passing

  • Mstest V1 test run with "Run tests in parallel" test explorer setting: 08:17 / all tests passing

    I already split the biggest test project in 3 smaller tests projects because the parallel execution in on assembly level only.

  • Mstest V2 1.30-beta2 (without runsettings): 09:38 / all tests passing

  • Mstest V2 1.30-beta2 (with 4 workers): 0:02:27 / ~ 100 tests failing.

    Wow. That's amazing.

  • Mstest V2 1.30-beta2 (with 56 workers): 0:02:03 / ~ 100 tests failing.

    • Not a big improvement to 4 workers. Most tests are finished after a couple of seconds. Some tests took very long (longer as usual, propably because of locking). It would be need to provide a per test timeout as proposed here: #26
      But you can see that this really uses every resource it can get.

#26
image

Awsome work.

@KthProg

This comment has been minimized.

Copy link

commented Mar 5, 2018

Awsome work.

It's only truly awesome when you leave out the "e". It's the same principle behind thicc and succ really.

@ravarnamsft

This comment has been minimized.

Copy link

commented Mar 21, 2018

Is there a way to opt in only certain test classes within an assembly and not the others? We have a few long running functional tests that we want to run in parallel. However, the classes in the rest of the assembly are not yet ready for parallelism. Right now, I will have to configure a separate test task in VSTS for the functional ones and specify a different runsettings file from the rest of the tests. Is there another way? Currently, we find tests in all assemblies by using regex matching *test.dll for example in the VSTS test task.

@abatishchev

This comment has been minimized.

Copy link

commented Mar 21, 2018

You can decorate some tests with [TestCategory("Functional")] and use test filter in test task to run these or those. But still will have to have 2 tasks, one to include and another to eluded target tests.

@ravarnamsft

This comment has been minimized.

Copy link

commented Mar 21, 2018

@abatishchev Thanks for the suggestion. All of these are actually marked Functional. Perhaps, the long running ones can also be marked as RunInParallel and then we can target them specifically. I was hoping there was a ParallelizationStrategy: Optin/OptOut in the runsettings and we could have used OptIn. But setting up a new VSTS task shouldn't be that hard. Thanks again.

@pjcalvo

This comment has been minimized.

Copy link

commented Mar 21, 2018

Hi @pvlakshm and team, this is so cool. I have been waiting on this feature all my life. I had problems at the beginning migrating to MSTest v2, but eventually i figured out that I needed to remove every reference to QualityTools and now it is working fine.

Works smoothly on TFS. Sorry the editing on this post.

@pjcalvo

This comment has been minimized.

Copy link

commented Mar 21, 2018

Hi, I got some feedback.

I want to be able to mark some tests as dependent on each other. And what that means is that the dependent tests will not run in parallel, but they can run in parallel with the rest of the tests.

A better explanation, given the next tests:

  • TestA (Depend on testC)
  • TestB
  • Test C
  • Test ...
  • Test Z

I want all the tests to run in parallel, but if testC is running then testA will not run until testC is complete.

Does it makes sense?

@abatishchev

This comment has been minimized.

Copy link

commented Mar 21, 2018

Why your tests depends on each other? Make them independent and isolated. If needed, call dependencies explicitly (make them private methods, not public test methods).

@pjcalvo

This comment has been minimized.

Copy link

commented Mar 21, 2018

@abatishchev They are dependent because they both are UI tests that send an email to a specific test account, and I need to verify each email, but if they run at the same time, and 2 emails are sent, it is really messy to identify which email belong to each test. So I want to run one first, then the other one.

The example above is just an example, they are more complicated scenarios of tests using the same test data and I think someone else might face a related issue.

@abatishchev

This comment has been minimized.

Copy link

commented Mar 21, 2018

Ok, then see the discussion/workaround above: how to decorate tests the way that some run in parallel, some not. P.S. I'm just a user with very similar requirements/issues, not a dev on the MsTest team.

@pjcalvo

This comment has been minimized.

Copy link

commented Mar 21, 2018

@abatishchev The workaround works fine, but all the tests decorated to not run in parallel will be executed one at the time and that add extra time to the execution. I want them to be in parallel, just some tests to run after others.

@abatishchev

This comment has been minimized.

Copy link

commented Mar 21, 2018

You can have different categories, like Parallel 1,2,3, and run a set of tests in parallel, then another, then another, then everything else. Again yet another workaround but it works for us.

@gnaneswarreddy

This comment has been minimized.

Copy link

commented Apr 6, 2018

I have recently moved our framework from mstest to mstestv2.0 using dotnetcore and observing few issues when running tests in parallel. If i execute single test i am able to connect to DB for my test data and get the data but when i run two tests in parallel i am not getting any data. Could anyone let me know what could be the issue. I am using sqldataadapter. I have been using datarow in mstest but mstest2.0 doesnot have any data row. Could this be an issue or is this related to any Sql implementation?

@abatishchev

This comment has been minimized.

Copy link

commented Apr 6, 2018

Most likely it's your code's issues. We're making numerous async HTTP calls from tests and all works just fine.
Post a question on Stack Overflow and provide a link here.

@gnaneswarreddy

This comment has been minimized.

Copy link

commented Apr 10, 2018

Thanks for the update. I was able to fix it. I am new to .netcore and when i run the tessts in parallel using .netcore it is not taking two tests Any idea how we can identify the number of test cases i trigerred as i have some predefined things which we needs to run in initialize method. I have ran two test cases but it is only taking one tests in to initialize method

@rummelsworth

This comment has been minimized.

Copy link

commented Apr 26, 2018

From the RFC it isn't clear whether a single test method with multiple DataRow attributes will be parallelized. My own testing right now seems to indicate that they are not parallelized.

  • What are the considerations for allowing DataRow-annotated test methods to be parallelized?
  • Do the semantics of the data source concept in MSTest not work well with parallelization?

Being able to parallelize over DataRow attributes would help me a great deal with making some of my integration tests more convenient to run.

I realize I can refactor so that the test method itself uses Parallel.ForEach (or similar) to explicitly parallelize over the test cases I have, but then I'd lose the separate-outcome-per-case display in Test Explorer. (I'd certainly consider doing this anyway if my integration test suite grows much bigger.)

@pvlakshm

This comment has been minimized.

Copy link
Contributor Author

commented Jun 4, 2018

I have clarified this in the RFC - indeed, parallelizing over DataRow attributes is not supported.
Thank you for bringing it up.

@pvlakshm

This comment has been minimized.

Copy link
Contributor Author

commented Jun 5, 2018

This has been shipping since v1.1.3.0 Beta, and went RTM in v1.3.0.
Accordingly, closing this issue.

@pvlakshm pvlakshm closed this Jun 5, 2018

@KthProg

This comment has been minimized.

Copy link

commented Jun 5, 2018

Cooooool

@pvlakshm

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2018

I recommend moving to v1.3.1. It has the fix for AppDomain creation should honor runsettings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.