Test methods references not passed correctly with @Test class

[EugeneAbramchuk]

Hi guys,

I have a class annotated with @test with several public methods meant to be tests. I also have a custom annotation on each public method to be used internally to deduce if this method needs to be skipped.

My problem is, when I execute the whole class,
org.testng.internal.annotations.JDK15AnnotationFinder#transform passes to my annotation implementation a null test method object for each public method without a @test annotation. If I annotate several methods with @test, they will be detected correctly.

A snippet to illustrate a concept:

@test
public class FooBar {

@test
@MyAnnotation(params)
public void foo() {

// org.testng.internal.annotations.JDK15AnnotationFinder#transform
// private void transform(IAnnotation a, Class testClass, Constructor testConstructor, Method testMethod)
// will have foo as a testMethod

}

@MyAnnotation(params)
public void bar() {

// org.testng.internal.annotations.JDK15AnnotationFinder#transform
// private void transform(IAnnotation a, Class testClass, Constructor testConstructor, Method testMethod)
// will have null as a testMethod

}
}

If I remove @test from the class name and just annotate each method, every method reference is passed as expected. But maybe there is a more inventive way?

My testNG version is 6.8.7

[missedone]

@EugeneAbramchuk
could you share a small project for reproducing this issue, thanks

[EugeneAbramchuk]

Hi @missedone
I uploaded the small project to https://github.com/EugeneAbramchuk/TestNG-issue. Please add DefaultListener as a listener to the TestNG run configuration and execute the SampleTests. One of them will fail and I would expect them both to be skipped.

[missedone]

@cbeust and @juherr

do you think it's an issue of testng?

here is the testng.xml I created for running the test @EugeneAbramchuk provided:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default suite">
    <listeners>
        <listener class-name="com.testngissue.util.DefaultListener" />
    </listeners>

    <test verbose="10" name="Default test">
        <classes>
            <class name="com.testngissue.tests.SampleTests" />
        </classes>
    </test>
</suite>

and here is the output:

[TestRunner] Running the tests in 'Default test' with parallel mode:false
[RunInfo] Adding method selector: org.testng.internal.XmlMethodSelector@64cee07 priority: 10
[TestClass] Creating TestClass for [ClassImpl class=com.testngissue.tests.SampleTests]
[TestClass] Adding method SampleTests.notAnnotatedTest()[pri:0, instance:null] on TestClass class com.testngissue.tests.SampleTests
[TestClass] Adding method SampleTests.annotatedTest()[pri:0, instance:null] on TestClass class com.testngissue.tests.SampleTests
[XmlMethodSelector] Including method com.testngissue.tests.notAnnotatedTest()
[XmlMethodSelector] Including method com.testngissue.tests.annotatedTest()
[TestNG] Running:
  /Users/nick/workspace/myproject/TestNG-issue/testng.xml

[SuiteRunner] Created 1 TestRunners
[TestRunner] Running test Default test on 1  classes,  included groups:[] excluded groups:[]
===== Test class
com.testngissue.tests.SampleTests
    @Test SampleTests.notAnnotatedTest()[pri:0, instance:com.testngissue.tests.SampleTests@2fc14f68]
    @Test SampleTests.annotatedTest()[pri:0, instance:com.testngissue.tests.SampleTests@2fc14f68]
======
[Invoker 1360767589] Invoking com.testngissue.tests.SampleTests.notAnnotatedTest
===== Invoked methods
    SampleTests.notAnnotatedTest()[pri:0, instance:com.testngissue.tests.SampleTests@2fc14f68] 801197928
=====
Creating /Users/nick/workspace/myproject/TestNG-issue/test-output/Default suite/Default test.html
Creating /Users/nick/workspace/myproject/TestNG-issue/test-output/Default suite/Default test.xml
FAILED: notAnnotatedTest
java.lang.AssertionError: Test supposed to be ignored
    at org.testng.Assert.fail(Assert.java:94)
    at com.testngissue.tests.SampleTests.notAnnotatedTest(SampleTests.java:19)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:821)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    at org.testng.TestRunner.privateRun(TestRunner.java:773)
    at org.testng.TestRunner.run(TestRunner.java:623)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
    at org.testng.SuiteRunner.run(SuiteRunner.java:259)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
    at org.testng.TestNG.run(TestNG.java:1018)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)


===============================================
    Default test
    Tests run: 1, Failures: 1, Skips: 0
===============================================

the DefaultListener implements IAnnotationTransformer which skips the test methods annotated with custom annotation '@Team', i saw it calls annotation.setEnabled(false); for each test method.
but as you can see from the output, notAnnotatedTest is still executed.

[juherr]

I didn't check yet but I suppose IAnnotationTransformer only transforms annotations and doesn't call back when there is no annotation.
I don't know if TestNG should or not simulate annotation when missing.
If the only need is enabling or enabling a test, IHookable should work.

[missedone]

@EugeneAbramchuk,
could try @juherr 's suggestion and let us know?
thanks

[EugeneAbramchuk]

@missedone @juherr Hi guys, In my framework I have more complex annotations and I am afraid I need more control over the test, so IHookable might not work for me. But anyway, according to the testNG's documentation (http://testng.org/doc/documentation-main.html#class-level):

The effect of a class level @test annotation is to make all the public methods of this class to become test methods even if they are not annotated.

I have done just that, but in reality it does not behave this way, therefore I'd suggest an issue with testNG or documentation mistake. I have already looked briefly into the source but could not pinpoint issue so far.

[juherr]

public methods of this class to become test methods even if they are not annotated.

doesn't means "public methods will have virtual @test annotation".
It means non-annotated methods will be found by TestNG as test methods.

I understand your usecase but it won't be easy to add.

I propose to close this issue and open an new one on the TestNG project: https://github.com/cbeust/testng/issues

[EugeneAbramchuk]

Point accepted, I will try to think of a solution.