New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel=instances run tests of same instance on different threads #751

Open
nmanandhar opened this Issue Jul 10, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@nmanandhar

nmanandhar commented Jul 10, 2015

The documentation says that
• parallel="instances": TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads.

However test methods for the same instances are running on different threads. Please see https://github.com/nmanandhar/TestNGParallelBehaviorTest.git

public class TestClassA {
    private static AtomicInteger instanceCount = new AtomicInteger(0);

    public TestClassA() {
        instanceCount.incrementAndGet();
        log("A new instance of TestClassA created");
    }

    @AfterSuite
    public void printNumberOfInstances() {
        log("Number of instances of TestClassA = " + instanceCount.get());
    }

    @Test
    public void testA1() throws Exception {
        log("testA1 of TestInstance " + this);
    }

    @Test(dependsOnMethods = "testA1")
    public void testB1() {
        log("testB1 of TestInstance " + this);
    }

    @Test(dataProvider = "integerProvider")
    public void testDataProvider(int number) throws Exception {
        log("testDataProvider : number = " + number + " testInstance = " + this);
    }

    @DataProvider
    public Object[][] integerProvider() {
        return new Object[][]{
                {1},
                {2}
        };
    }

    private void log(String message) {
        String threadName = Thread.currentThread().getName();
        synchronized (System.out) {
            System.out.println(String.format("Thread %s : %s ", threadName, message));
        }
    }
}

Suite File

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="InstanceTestSuite" parallel="instances" thread-count="5">
    <test name="InstanceTest">
        <classes>
            <class name="com.nm.TestClassA"/>
        </classes>
    </test>
</suite>

Output of mvn test

Running TestSuite
Thread main : A new instance of TestClassA created
Thread pool-1-thread-1 : testA1 of TestInstance com.nm.TestClassA@58ee2549
Thread pool-1-thread-1 : testDataProvider : number = 1 testInstance = com.nm.TestClassA@58ee2549
Thread pool-1-thread-1 : testDataProvider : number = 2 testInstance = com.nm.TestClassA@58ee2549
Thread pool-1-thread-2 : testB1 of TestInstance com.nm.TestClassA@58ee2549
Thread main : Number of instance = 1
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.835 sec - in TestSuite
@nmanandhar

This comment has been minimized.

Show comment
Hide comment
@nmanandhar

nmanandhar Jul 23, 2015

Motivation For Need for Running Tests in the Same Thread

  • Ensuring that all test of a single instance run in the same thread will allow us confine objects used by test to a single thread and to avoid thinking about threading issues(visiblity,race conditions etc).
  • We are trying to run Selenium tests in parallel. Each test instance will open a browser instance and then run tests against this instance. When thread count is set to 5, I expected only 5 different instance of browser to be opened, more than 10 different browsers were opening. My first instinct was that TestNG somehow got the thread-count wrong.But we found that for a test instance, a thread picked up some of the test, then a long time later another thread picked up some of the
    remaining tests.
TestA started, browser opened and two test ran on this thread
|-------------------|
Thread 1

                           ..A long time later

                                                            |--------------------|
                                                            Thread 2
                                                            Continuing with the remaining tests on TestA instance

Because of this a browser could open run some test , then remain idle for a long time, then resume test after an hour later when some other thread is ready to pick up the remaining tests.
Thus even with a thread count of 5, 10 or 20 instance of test were running concurrently, and some browsers remained idle for a long time. With the browser remaining idle
for some time ,the selenium hub expires session on the particular browser instance failing the test. Changing the session timeout fixes the issue (but is unsatisfactory) however
we can never be sure how many browser instance will be opened. Also with tests being run in multiple threads, we cannot be sure of threading issues that will arise.

nmanandhar commented Jul 23, 2015

Motivation For Need for Running Tests in the Same Thread

  • Ensuring that all test of a single instance run in the same thread will allow us confine objects used by test to a single thread and to avoid thinking about threading issues(visiblity,race conditions etc).
  • We are trying to run Selenium tests in parallel. Each test instance will open a browser instance and then run tests against this instance. When thread count is set to 5, I expected only 5 different instance of browser to be opened, more than 10 different browsers were opening. My first instinct was that TestNG somehow got the thread-count wrong.But we found that for a test instance, a thread picked up some of the test, then a long time later another thread picked up some of the
    remaining tests.
TestA started, browser opened and two test ran on this thread
|-------------------|
Thread 1

                           ..A long time later

                                                            |--------------------|
                                                            Thread 2
                                                            Continuing with the remaining tests on TestA instance

Because of this a browser could open run some test , then remain idle for a long time, then resume test after an hour later when some other thread is ready to pick up the remaining tests.
Thus even with a thread count of 5, 10 or 20 instance of test were running concurrently, and some browsers remained idle for a long time. With the browser remaining idle
for some time ,the selenium hub expires session on the particular browser instance failing the test. Changing the session timeout fixes the issue (but is unsatisfactory) however
we can never be sure how many browser instance will be opened. Also with tests being run in multiple threads, we cannot be sure of threading issues that will arise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment