Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Parallelization for Data-Driven Tests #405

Closed
sroei opened this issue Apr 19, 2018 · 5 comments

Comments

@sroei
Copy link

commented Apr 19, 2018

Description

following your documentation here:
https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md

did not execute my tests on parallel. furthermore, the Parallelize attribute cannot be found (you do not mention where this attribute is coming from)

Steps to reproduce

complete code:

using System;
using System.Collections.Generic;
using System.Threading;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [DataTestMethod]
        [DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
        public void TestMethod1(int a, int b, int expected)
        {
            Thread.Sleep(10000);
            var actual = a + b;
            Assert.AreEqual(expected, actual);
        }

        [DataTestMethod]
        [DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
        public void TestMethod2(int a, int b, int expected)
        {
            Thread.Sleep(10000);
            var actual = a + b;
            Assert.AreEqual(expected, actual);
        }

        public static IEnumerable<object[]> GetData()
        {
            yield return new object[] { 1, 1, 2 };
            yield return new object[] { 14, 1, 15 };
            yield return new object[] { 12, 30, 42 };
        }
    }
}

Run Settings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <MSTest>
    <Parallelize>
      <Workers>8</Workers>
      <Scope>MethodLevel</Scope>
    </Parallelize>
  </MSTest>
</RunSettings>

Expected behavior

according to the tutorials, total execution of each test will be around 10 seconds (all iterations in parallel)

Actual behavior

nothing runs in parallel

Environment

windows 10.
NuGet Packages: MSTest.TestAdapter (v.1.2.1), MSTest.TestFramework (v1.2.1)
Latest vstest.console (latest vs2017)

@cltshivash cltshivash added doc bug labels May 9, 2018

@nigurr

This comment has been minimized.

Copy link

commented May 11, 2018

@sroei it's available with https://github.com/Microsoft/testfx/releases/tag/v1.3.0-beta2.
Please upgrade your package and try the scanrio.

@GoFightNguyen

This comment has been minimized.

Copy link
Contributor

commented May 16, 2018

@sroei As an update to @nigurr's response, the release version of 1.3.0 is here.

Wrong Expectation

TestMethod1 and TestMethod2 will run in parallel to each other, but not at the level you are expecting. For example, although TestMethod1 has 3 scenarios, those scenarios will run sequentially.

Correct Expectation

Each test method will take 30 seconds to execute, but the total execution time should be around 30 seconds since TestMethod1 and TestMethod2 will run in parallel to each other.

@singhsarab

This comment has been minimized.

Copy link
Member

commented May 23, 2018

@sroei Can you please try and confirm, so we can close this issue ? Thanks.

@sroei

This comment has been minimized.

Copy link
Author

commented May 23, 2018

Sorry for taking me a while. If this is not supported, please close the issue

This is the expected I was aiming for and the initial problem was that I could not run parallel on the iteration level. Running parallel on the test level is relatively easy and I use multiple test agents to achieve that (I have like 100 + Selenium Grid).

An example of the desired expected you can see on TestNG while set it to parallel data-source.

Use Case

  1. Data-Driven Test with 250 Iterations
  2. Each Iteration Runs for ~30sec

Problem: cannot be distributed around multiple agents nor run in parallel on a single machine.

I will try to open a feature request and hopefully this will be supported in the future - the current workaround (without splitting the data-source) is very ugly

@nigurr nigurr added enhancement tracking and removed bug labels May 29, 2018

@jayaranigarg jayaranigarg changed the title Tutorials is missing a crucial information Enable Parallelization for Data-Driven Tests Jun 7, 2018

@jayaranigarg

This comment has been minimized.

Copy link
Member

commented Jun 27, 2018

Closing this thread and using this issue to track the feature request..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.