@BeforeClass not executed in parallel when parallel="instances"

[adamjson]

I personally came across the source of my problem when I found this stack overflow thread which helped me make sense of some strange behavior I've been encountering with a Factory + DataProvider + Parallel Instances

http://stackoverflow.com/questions/30027872/beforeclass-in-testng-is-not-executed-in-parallel-with-parallel-instances

It doesn't look like a bug was ever filed, but I am getting burned with my selenium tests queueing against the grid, and waiting for a browser in the @BeforeClass annotation which sets up my test in a parent class. The child class has a @Factory to create all the instances up front, and the @Test method to actually carry out the test. The problem is that once a browser hits the grid queue, no other tests will start up, even though there are plenty of instances available (we test against various browsers, so having 1 browser flavor queue up doesn't mean that the grid is completely busy).

The example posted in the stack overflow thread is:

@BeforeClass
public void beforeClass() throws InterruptedException{
    System.out.println("In Before Class ::: " + Thread.currentThread().getId());
    Thread.sleep(2000);
}


@Test
public void test(){
    System.out.println("In Test ::: " + Thread.currentThread().getId());
}

@AfterClass
public void afterClass(){
    System.out.println("In After Class ::: " + Thread.currentThread().getId());
}

I am running 6.9.9.

[adamjson]

To clarify, I would expect that when a Factory is used and thousands of instances are created with a DataProvider, the BeforeClass annotation should run in parallel, instead of running each sequentially (parallel=instances and the thread count is high).

[juherr]

@adamjson Thank for the issue. Are you make some analyze? Or are you open to try to understand it?

[rouke-broersma]

@adamjson I have pretty much the same setup, I think dataproviderthreadcount needs to be set as well for the dataprovider to run in parallel.

I do have one issue, testng creates a webdriver instance for every single test in my suite before even starting one test due to the factory, even when I don't include some tests groups to run. Opening 40 chromebrowsers before even starting one test is extremely wasteful and resource intensive on my server, and it also doesn't close the browsers for the tests that were skipped. I just ran another test and I selected 23 of 40 tests to be run (excluded based on group) and 40 browser windows were opened, so I was left with 17 after the tests were done.. Does your setup do the same or if not how did you get around that?

edit:
This was actually due to creating the browser in my test constructor, should have thought of that, my bad.

[adamjson]

Setting the dataproviderthreadcount did not help when I tried. The problem is not running a data provider in parallel, but the configurations that the data provider generates need to be run in parallel. Maybe I am doing something wrong?

I do not have a problem with all WebDriver instances spinning up before the test starts. Where are you initializing the WebDriver? If you have the WebDriver init in the constructor of a @factory test, that would explain that behavior. All instances are constructed at the start of the test suite.

I've attempted two workarounds to this problem. The first, was to set "group-by-instances" to true, which means that the test / config methods in different instances will be run independently of each other, and in theory allow me to run all my Factory + DataProvider tests in parallel. However, another bug is preventing @factory tests from running in parallel with group-by-instances=true (#326).

So, the very hacky solution I've implemented for now is to perform all setup of the test, including the init of the WebDriver, inside of a single @test annotation. Luckily, this particular test has few methods inside of our single @test method because we delegate the heavy lifting to a different internal framework (in this instance, this is not always the case for us).

[rouke-broersma]

Mine run independently from each other as far as I can tell. I just tried a short test with my login test and two chrome browsers and they both started and finished almost exactly the same time:

17:19:14.375 [pool-2-thread-1] INFO nl.adaption.it.selenium_test.tests.login_logout.ITlogin - ### STARTING TEST: whenLoginWithValidUser_HomePageShouldBeVisible[CHROME] ###
17:19:14.375 [pool-2-thread-2] INFO nl.adaption.it.selenium_test.tests.login_logout.ITlogin - ### STARTING TEST: whenLoginWithValidUser_HomePageShouldBeVisible[CHROME] ###
17:19:28.374 [pool-2-thread-1] INFO nl.adaption.it.selenium_test.tests.login_logout.ITlogin - ### STOPPING TEST: whenLoginWithValidUser_HomePageShouldBeVisible[CHROME] ###
17:19:28.770 [pool-2-thread-2] INFO nl.adaption.it.selenium_test.tests.login_logout.ITlogin - ### STOPPING TEST: whenLoginWithValidUser_HomePageShouldBeVisible[CHROME] ###

[adamjson]

Also, I'm not sure that I fully answered your question earlier. Our tests will naturally queue up when the Selenium Grid is out of empty slots, so we haven't had to worry about throttling.

That's interesting. What version of TestNG are you using? How are you executing your suite? Do you have parallel=instances?

[rouke-broersma]

You did fully answer my question, thank you, It was my silly mistake.

As for my testsuite, I'm using TestNG 6.9.11 and run my tests with maven failsafe. I tag my groups (All, DataCreatingTest, etc..) and then include/exclude tests based on the tag in maven failsafe configuration. I do not have parallel=instances because I'm not even sure where to enable this on my current setup. I'm guessing I would have to do that on my factory annotation? Haven't needed it so haven't tried it.

[rouke-broersma]

I only just now noticed you're using a @BeforeClass, which I am not. I use @BeforeMethod to set up my tests.

[juherr]

From the doc:

parallel="instances": TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads.

As I understand it, the observed behavior could be the expected one.
Could you try with: parallel="classes"

parallel="classes": TestNG will run all the methods in the same class in the same thread, but each class will be run in a separate thread.

[rouke-broersma]

@juherr He is running multiple instances of the same class (factory with dataProvider) parallel per instance of the class. If he would use parallel per classes would that not mean that he can never have the same test running at the same time with a different dataProvider parameter?

I do not think the observed behaviour is expected as every instance of the class should have it's own @BeforeMethod executed, so those should be running in parallel when choosing parallel per instance.

As I understand it parallel per classes would mean class A and class B can run parallel, but what OP wants is var 1 = new Class A and var 2 = new Class A and var 3 = new Class A to run at the same time, which would be parallel per instance of class A.

[juherr]

If he would use parallel per classes would that not mean that he can never have the same test running at the same time with a different dataProvider parameter?

Yes, I suppose you're right. And your workarround looks good.

@cbeust What do you think? What is the expected parallel behavior on @BeforeClass with parallel="instances"

[adamjson]

@Mobrockers Thanks for clarifying. Also, @juherr am I correct in thinking that setting group-by-instances=true would have also solved my problem if not for #326 ?

[rouke-broersma]

@adamjson perhaps you could simply use an @BeforeMethod instead of an @BeforeClass to set up your webdriver instance? I think it is recommended to use a fresh webdriver instance per every test anyways, and as far as I can tell it does run parallel per method (if this is because I run my tests parallel per method through maven I do not know, but worth a try anyway?) for me.

[adamjson]

@Mobrockers Given that I have collapsed all my test logic into a single @test method, that would help me break the setup out of the test method in this case. However, if I were to have > 1 @test method in my class, and wanted each of those to run on the same WebDriver instance sequentially, I wouldn't want to use @BeforeMethod.

[rouke-broersma]

That is true. I have not set up my test base in a way that is likely to
work sequentially on the same webdriver, on purpose, because I don't think
having to deal with such headaches is worth my time, so I can always be
sure the beforemethod works in my case.

[max-demidov]

Hi colleagues. I have faced the same problem as described here.

public class Test {

    private Map<String, String> params;
    private PageToBeTested page;

    @Factory(dataProvider = "data")
    public Test(Map<String, String> params) {
        this.params = params;
    }

    @DataProvider
    public static Iterator<Object[]> data() {
        List<Map<String, String>> data = ...;
        return data.stream().map(i -> new Object[]{i}).iterator();
    }

    @BeforeClass
    public void startBrowser() {...}

    @BeforeClass(dependsOnMethods = "startBrowser")
    public void openProductPage() {...}

    @BeforeClass(dependsOnMethods = "openProductPage")
    public void doLogin() {...}

    @BeforeClass(dependsOnMethods = "doLogin")
    public void proceedToPageToBeTested() {
        fillPageInputsWith(params);
        ...;
        page = new PageToBeTested();
    }

    @AfterClass
    public void quitBrowser() {...}

    @Test
    public void checkFirstThing() {
        String expVal = param.get(...);
        String actVal = page.getFirstThing();
        Assert.assertEquals(actVal, expVal);
    }

    @Test
    public void checkSecondThing() {
        String expVal = param.get(...);
        String actVal = page.getSecondThing();
        Assert.assertEquals(actVal, expVal);
    }
...
    @Test
    public void checkTwentiethThing() {
        String expVal = param.get(...);
        String actVal = page.getTwentiethThing();
        Assert.assertEquals(actVal, expVal);
    }

}

I try to run it in parallel with
<suite name="Pain with parallel @BeforeClass" parallel="instances" configfailurepolicy="continue">
but see that they work sequentially - instance by instance.

Is there any update with this issue? Is it even planned to be fixed someday?

@cbeust could you advise anything with my case?

@adamjson , @Mobrockers have you guys eventually found any workaround for this?

[ghost]

Has anyone found a workaround?

[krmahadevan]

The fix should be available either in the next beta version or the released version of TestNG.