Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expected behavior of annotated methods on an @Test(groups="mygroup") annotated class #2366

Closed
2 of 7 tasks
cardonator opened this issue Sep 13, 2020 · 16 comments
Closed
2 of 7 tasks

Expected behavior of annotated methods on an @Test(groups="mygroup") annotated class #2366

cardonator opened this issue Sep 13, 2020 · 16 comments
Labels
Feature: before/after Feature: groups

Comments

@cardonator
Copy link

cardonator commented Sep 13, 2020 •
edited

I have experienced a strange issue in TestNG 7+ that used to work in TestNG 6. Basically I have a suite defined that runs a group of mygroup and I have a few test classes annotated with @Test(groups="mygroup"). These classes also have a defined @BeforeClass and @BeforeMethod and @AfterMethod which are public void.

When I run my suite, all of the public functions inside the class are picked up and run as if they had an @Test annotation. That includes the public functions that have the above three listed annotations, which all fail with NPEs.

So, my question is, what is the expected behavior in this scenario and what should my annotations look like? Here's a very rudimentary code sample of what I have for one of the tests:

package com.company.app.integration.suite.app;

@Test(groups = "mygroup")
public class AppRestTest {
    @BeforeClass
    public void setupRestTest() {
        // A bunch of setup that is run as a Test
    }

    @BeforeMethod(alwaysRun = true)
    public void beforeMethod(Method method) {
        System.out.println("Begin " + method.getName() + "..."); // NPE
    }

    @AfterMethod(alwaysRun = true)
    public void afterMethod(ITestResult result) {
        System.out.println("End " + result.getMethod().getMethodName()); // NPE
    }

    @Test
    public void testRestPostGetAll() {
        // works fine and passes as expected
    }

    @Test(groups="othergroup")
    public void testRestPostGetAllNoFilter() {
        // works fine but runs with both group names
    }
    
}

TestNG Version

7.0.0, 7.1.0, 7.3.0

Expected behavior

The previous behavior was that the annotated methods would be skipped and any annotated method with a group that shouldn't be run was excluded. (It's worth noting that I am working on an upgrade from TestNG 6.8.8 to 7.x.x and so if the expected behavior has changed that's fine, I'm looking for clarification as this is not expressed anywhere in the docs I could find).

Actual behavior

Any public function, even TestNG annotated, in the Test class is run as a test when the Class itself has an included groups tag.

Is the issue reproductible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

Test case provided above.
@juherr
Copy link
Member

juherr commented Sep 15, 2020

@BeforeClass, @BeforeMethod and @AfterMethod are not supposed to be run as test methods.
Could you share a full runnable sample in order to reproduce the behavior easily?

@cardonator
Copy link
Author

Sure, I have created a SB starter app configured in a similar way to my application here: https://github.com/cardonator/demo

When you right click and run the integration-testng.xml file in demo-integration/src/test/resources/jenkins, it tries to run the annotated methods as tests in IntelliJ 2020.2 (I have included testng 7.1.0 in the project because 7.3.0 is incompatible with IntelliJ).

You can also observe the same behavior if you run mvn test -Pintegration -pl demo-integration from the CLI. (If you use testng 7.3.0 it also exhibits the same behavior using this command).

Let me know if you need additional information.

@krmahadevan
Copy link
Member

Here's a much more simpler example that can be used to reproduce the problem.

import java.lang.reflect.Method;
import org.testng.ITestResult;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

@Test(groups = "acceptance")
public class SampleTestCase {

  @BeforeClass
  public void setupOrgMatching() {
    System.err.println("Running @BeforeClass");
  }

  @BeforeMethod(alwaysRun = true)
  public void beforeMethod(Method method) {
    System.err.println("Running @BeforeMethod for " + method.getName());
  }

  @AfterMethod(alwaysRun = true)
  public void afterMethod(ITestResult result) {
    System.err.println("Running @AfterMethod for " + result.getMethod().getQualifiedName());
  }

  @AfterClass(alwaysRun = true)
  public void cleanup() {
    System.err.println("Running @AfterClass");
  }

  @Test
  public void testGetHeartbeat() {
    System.err.println("Running test method testGetHeartbeat()");
  }

  @Test(groups = "fragile")
  public void testGetHeartbeatSkipped() {
    System.err.println("Running test method testGetHeartbeatSkipped()");
  }
}

Suite xml file

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="2366_suite" verbose="2">
  <test name="2366_test">
    <groups>
      <run>
        <include name="acceptance"/>
      </run>
    </groups>
    <classes>
      <class name="com.rationaleemotions.github.issue2366.SampleTestCase"/>
    </classes>
  </test>
</suite>

Output

...
... TestNG 7.3.0 by Cédric Beust (cedric@beust.com)
...


Running @BeforeClass
Running @BeforeMethod for testGetHeartbeat
Running test method testGetHeartbeat()
Running @AfterMethod for com.rationaleemotions.github.issue2366.SampleTestCase.testGetHeartbeat
Running @BeforeMethod for testGetHeartbeatSkipped
Running test method testGetHeartbeatSkipped()
Running @AfterMethod for com.rationaleemotions.github.issue2366.SampleTestCase.testGetHeartbeatSkipped


Running @AfterClass

PASSED: testGetHeartbeat
PASSED: testGetHeartbeatSkipped

===============================================
    2366_test
    Tests run: 2, Failures: 0, Skips: 0
===============================================


===============================================
2366_suite
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

Analysis

The configuration method is ending up inheriting the groups defined at the class level defined @Test annotation because inheritGroups attribute in @BeforeClass is by default true.

Quoting the java docs

  /**
   * If true, this Configuration method will belong to groups specified in the @Test
   * annotation on the class (if any).
   *
   * @return the value (default true)
   */

I think TestNG is working as designed here in terms of the published behavior.

@cardonator - In your case, you might want to explicitly set @BeforeClass(inheritGroups=false) and try again.

Let me know if that explains the problem and if this can be closed out.

@cardonator
Copy link
Author

Thanks @krmahadevan for attempting to research into this issue!

I actually don't think your simpler test is exhibiting the same problem as my demo repo does. The reason I say that is because you aren't getting any NPE in your output. When I run the maven command from my post above on the demo repo I get the following output:

[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< com.example.demo:demo-integration >------------------
[INFO] Building demo-integration 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /code/demo-integration/src/main/resources
[INFO] skip non existing resourceDirectory /code/demo-integration/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ demo-integration ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ demo-integration ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ demo-integration ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
Starting EndpointsTest...
Finished EndpointsTest in 248 ms
Begin testGetHeartbeat...
End testGetHeartbeat
Finished EndpointsTest in 264 ms
Finished EndpointsTest in 269 ms
[ERROR] Tests run: 7, Failures: 4, Errors: 0, Skipped: 3, Time elapsed: 1.36 s <<< FAILURE! - in TestSuite
[ERROR] afterMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.827 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.afterMethod(EndpointsTest.java:28)

[ERROR] beforeMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.828 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.beforeMethod(EndpointsTest.java:23)

[ERROR] beforeMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.007 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.beforeMethod(EndpointsTest.java:23)

[ERROR] afterMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.011 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.afterMethod(EndpointsTest.java:28)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR] com.example.demo.integration.suite.EndpointsTest.afterMethod(com.example.demo.integration.suite.EndpointsTest)
[ERROR]   Run 1: EndpointsTest.afterMethod:28 NullPointer
[ERROR]   Run 2: EndpointsTest.afterMethod:28 NullPointer
[INFO] 
[ERROR] com.example.demo.integration.suite.EndpointsTest.beforeMethod(com.example.demo.integration.suite.EndpointsTest)
[ERROR]   Run 1: EndpointsTest.beforeMethod:23 NullPointer
[ERROR]   Run 2: EndpointsTest.beforeMethod:23 NullPointer
[INFO] 
[INFO] 
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 2
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.095 s
[INFO] Finished at: 2020-09-16T06:50:47Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project demo-integration: There are test failures.
[ERROR] 
[ERROR] Please refer to /code/demo-integration/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

As you can see from the output, the issue isn't that the annotated functions are inheriting the group and thus running as part of the test (they should be run as part of the group because they are necessary to successfully run the tests in the rest of the class), the issue is that the functions annotated with @BeforeClass, @BeforeMethod, @AfterMethod, and @AfterClass seem to be automatically annotated with @Test AND inheriting the group from the class, and the methods are then being run as their own Test cases and not as part of other test cases.

I did try your suggestion and observed the behavior I expected to see: @BeforeClass wasn't run at all when running the suite with the group included.

@krmahadevan
Copy link
Member

@cardonator - I noticed that your pom says need JDK11. TestNG is not officially yet certified to be compliant with JDK11.

I have JDK8. So I altered the pom file to use JDK8 and also to use TestNG 7.3.0 and here's the output when I ran from command line

So can you try again with JDK8 and post back your results ?

Also if you are running with JDK11, please add verbose=2 in your suite file and see if you can find additional log messages that can hint at the problem.

$ mvn test -Pintegration -pl demo-integration
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< com.example.demo:demo-integration >------------------
[INFO] Building demo-integration 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/user/temp/demo/demo-integration/src/main/resources
[INFO] skip non existing resourceDirectory /Users/user/temp/demo/demo-integration/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ demo-integration ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ demo-integration ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ demo-integration ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
...
... TestNG 7.3.0 by Cédric Beust (cedric@beust.com)
...

Starting EndpointsTest...
Begin testGetHeartbeat...
End testGetHeartbeat
Finished EndpointsTest in 560 ms
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.886 s <<< FAILURE! - in TestSuite
[ERROR] testGetHeartbeat(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.474 s  <<< FAILURE!
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
	at com.example.demo.integration.suite.EndpointsTest.testGetHeartbeat(EndpointsTest.java:38)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at com.example.demo.integration.suite.EndpointsTest.testGetHeartbeat(EndpointsTest.java:38)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   EndpointsTest.testGetHeartbeat:38 » Processing java.net.ConnectException: Conn...
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.056 s
[INFO] Finished at: 2020-09-16T13:48:10+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project demo-integration: There are test failures.
[ERROR]

@cardonator
Copy link
Author

Hmm, okay. Thanks for letting me know. I assumed that TestNG was expected to work in 11 since the docs say it requires JDK 8+. Since 6.14.3 works in JDK 11, I guess I'll just stick with that until 11 support is improved in the 7.x branch.

I tried increasing verbosity to collect more information but I didn't seem to get any additional output.

@krmahadevan
Copy link
Member

@cardonator - Before concluding on the role of JDK11, can you please try once at your side with JDK8 and post back your results ? When I get some time over the weekend, I will try to simulate this issue with JDK11 to see what may be causing this.

@krmahadevan
Copy link
Member

@cardonator - I have tried this on JDK11 (with TestNG 7.3.0) and I still am not able to reproduce the NPE that you are experiencing.

krmahadevan@hades:~/githome/demo(master)$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.4, vendor: Oracle Corporation, runtime: /home/krmahadevan/jdk-11.0.4
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-47-generic", arch: "amd64", family: "unix"
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ demo-integration ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
Starting EndpointsTest...
Begin testGetHeartbeat...
End testGetHeartbeat
Finished EndpointsTest in 868 ms
FAILED: testGetHeartbeat
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:261)
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:296)
	at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:623)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:621)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:404)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:300)
	at com.example.demo.integration.util.DemoClient.get(DemoClient.java:53)
	at com.example.demo.integration.util.DemoClient.heartbeat(DemoClient.java:73)
	at com.example.demo.integration.suite.EndpointsTest.testGetHeartbeat(EndpointsTest.java:38)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:591)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1248)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:367)
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:259)
	... 27 more


===============================================
    Demo Integration Tests
    Tests run: 1, Failures: 1, Skips: 0
===============================================

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.424 s <<< FAILURE! - in TestSuite
[ERROR] testGetHeartbeat(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.763 s  <<< FAILURE!
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
	at com.example.demo.integration.suite.EndpointsTest.testGetHeartbeat(EndpointsTest.java:38)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at com.example.demo.integration.suite.EndpointsTest.testGetHeartbeat(EndpointsTest.java:38)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   EndpointsTest.testGetHeartbeat:38 » Processing java.net.ConnectException: Conn...
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.393 s
[INFO] Finished at: 2020-09-19T20:14:17+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project demo-integration: There are test failures.
[ERROR]
[ERROR] Please refer to /home/krmahadevan/githome/demo/demo-integration/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
krmahadevan@hades:~/githome/demo(master)$

@cardonator
Copy link
Author

cardonator commented Sep 20, 2020 •
edited

Hi @krmahadevan, I'm not sure how to explain the difference in what is happening. I can reproduce the issue consistently across multiple environment and machines.

$   mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/share/maven
Java version: 11.0.4, vendor: Eclipse OpenJ9, runtime: /opt/java/openjdk
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-47-generic", arch: "amd64", family: "unix"

I can reproduce this even in IntelliJ (Community, 2020.2). Only when running the Suite XML.

I cloned to a new machine and ran:

mvn clean install
mvn test -Pintegration -pl demo-integration

and this is the output I get:

$  mvn test -Pintegration -pl demo-integration
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< com.example.demo:demo-integration >------------------
[INFO] Building demo-integration 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /code/demo-integration/src/main/resources
[INFO] skip non existing resourceDirectory /code/demo-integration/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ demo-integration ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ demo-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ demo-integration ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ demo-integration ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
Starting EndpointsTest...
Finished EndpointsTest in 109 ms
Begin testGetHeartbeat...
End testGetHeartbeat
Finished EndpointsTest in 123 ms
Finished EndpointsTest in 126 ms
[ERROR] Tests run: 7, Failures: 4, Errors: 0, Skipped: 3, Time elapsed: 0.672 s <<< FAILURE! - in TestSuite
[ERROR] afterMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.382 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.afterMethod(EndpointsTest.java:28)

[ERROR] beforeMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.384 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.beforeMethod(EndpointsTest.java:23)

[ERROR] beforeMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.006 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.beforeMethod(EndpointsTest.java:23)

[ERROR] afterMethod(com.example.demo.integration.suite.EndpointsTest)  Time elapsed: 0.008 s  <<< FAILURE!
java.lang.NullPointerException
        at com.example.demo.integration.suite.EndpointsTest.afterMethod(EndpointsTest.java:28)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR] com.example.demo.integration.suite.EndpointsTest.afterMethod(com.example.demo.integration.suite.EndpointsTest)
[ERROR]   Run 1: EndpointsTest.afterMethod:28 NullPointer
[ERROR]   Run 2: EndpointsTest.afterMethod:28 NullPointer
[INFO] 
[ERROR] com.example.demo.integration.suite.EndpointsTest.beforeMethod(com.example.demo.integration.suite.EndpointsTest)
[ERROR]   Run 1: EndpointsTest.beforeMethod:23 NullPointer
[ERROR]   Run 2: EndpointsTest.beforeMethod:23 NullPointer
[INFO] 
[INFO] 
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 2
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.287 s
[INFO] Finished at: 2020-09-20T23:45:59Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project demo-integration: There are test failures.
[ERROR] 
[ERROR] Please refer to /code/demo-integration/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Again, thanks so much for looking into this issue. I know sometimes issues like this can be the most frustrating.

@juherr
Copy link
Member

juherr commented Sep 22, 2020

I see a difference between you:

@cardonator

Java version: 11.0.4, vendor: Eclipse OpenJ9

@krmahadevan

Java version: 11.0.4, vendor: Oracle Corporation

@cardonator
Copy link
Author

I am running the OpenJ9 compiler on one of my machines but have also seen the same behavior with the hotspot compiler as well.

@wenijinew
Copy link
Contributor

wenijinew commented Sep 26, 2020 •
edited

@cardonator

I managed to reproduce the problem. See wiki page for this issue for details.
The problem is your @BeforeMehod and @AfterMethod design:

TestNG won't pass value to your beforeMethod and afterMethod for arguments method and result.
How can they be non-null? (Update: this is really different in the version 6.8.8 - TestNG passed value to those arguments)

    @BeforeMethod(alwaysRun = true)
    public void beforeMethod(Method method) {
        System.out.println("Begin " + method.getName() + "..."); // NPE
    }

    @AfterMethod(alwaysRun = true)
    public void afterMethod(ITestResult result) {
        System.out.println("End " + result.getMethod().getMethodName()); // NPE
    }

One more difference is:
In 7.1.0, all configuration methods are executed 3 times, and in 6.8.8 only once.

It seems something wrong in 7.1.0. I haven't try other versions.

@cardonator
Copy link
Author

@WengM so first, these function definitions work on 6.x so I think they are right but open to suggestions if they are wrong in 7.x.

Second, isn't that the purpose of the annotations to indicate the behavior of those methods when testng parses them before and after each @Test method? What should those functions look like?

FYI, the behavior on 6 is that the test class is picked up in the group properly and any testng annotated functions are not run as tests.

@wenijinew
Copy link
Contributor

wenijinew commented Sep 26, 2020 •
edited

@cardonator
I just tested 7.3.0 and got the same result with 6.8.8. Please check on wiki page for this issue for details.

And 7.0.0 has the same test result with 7.1.0. I will not update on the wiki page.

So, finally, please double check 7.3.0 on your side.

I used below JDK version for testing. Didn't run with mvn test command but java command to run TestNG.

openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment 18.9 (build 11.0.8+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10, mixed mode, sharing)

@cardonator
Copy link
Author

cardonator commented Sep 27, 2020 •
edited

Ahh, thanks for continuing to look into this. So I tried again on 7.3.0 after basically clearing out my maven cache on the demo project and that does indeed appear to fix the issue! When I was testing this before I was having issues with 7.3.0 testing due to conflicts with the IDE. It's possible that the times I was "testing" with 7.3.0 I was actually somehow still using compiled classes with an earlier version of TestNG included.

I really appreciate the help researching and testing. Sorry it turned out to be already resolved!

@krmahadevan
Copy link
Member

@cardonator - I am going ahead and closing off this issue since it turns out to be resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature: before/after Feature: groups
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@juherr @krmahadevan @wenijinew @cardonator