ISPN-7116 Test methods that use a TestNg dataProvider are duplicated in

[alanfx]

JUnit reports

https://issues.jboss.org/browse/ISPN-7116

I was unable to find the class that writes the current
"TEST-.xml" file, so I created a new reporter. I know it isn't
ideal to have two files generated, but I need this fix to proceed with
Polarion onboarding. The JIRA can be left open, if this solution isn't
satisfactory.
Also updated TestNGTestListener to output the parameters to the console

[alanfx]

@danberindei can you take a look? Thanks

[alanfx]

I need to work on the file naming code a little more. Picking the package of the first method doesn't seem to always be a good choice. :-(

[alanfx]

Ok, this is ready for review

[danberindei]

@alanfx I was able to debug by setting <forkCount>0</forkCount> and see that surefire uses its own listener for writing the report, org.apache.maven.surefire.testng.ConfigurationAwareTestNGReporter. I'm not sure if you can disable it from the configuration...

[danberindei]

I'm getting a NullPointerException here if I run the build from IntelliJ.

TBH I don't see a reason to treat runs from the project directory and from the module directory differently, as the output directory is already different for each module.

[alanfx]

Yeah, this environment variable won't be defined unless you are running it from Maven. So here is what I am trying to handle. If you run the command from the top level of the project and use --project to run tests in a sub-module, everything is ok. MAVEN_PROJECTBASEDIR includes the directory name for the top level of the project, so I can strip that out to name the report file which may include multiple directories above the target directory. However, if you run the maven command from a subdirectory, then MAVEN_PROJECTBASEDIR is set to the sub-module name, so then the code picks the first directory name above target on the output directory. The issue is that multiple modules use a name like remote or embedded that could use the extra context in the results file name. I hope that makes sense.

I could see the org.apache.maven.surefire.testng.ConfigurationAwareTestNGReporter in the list of listeners, but I could not get the debugger to break into it. Thanks for the forkCount tip. I'll try that to see if I can either reuse that code, or at least disable the creation of the TEST-suite.xml files.

[alanfx]

@danberindei I pushed a small change to reverse the order of the comparison on line 318 that I think will fix the NullPointerException in IntelliJ. So far, I haven't been able to figure out how to disable ConfigurationAwareTestNGReporter.

[danberindei]

@alanfx I now get a NPE on line 320 when I run it from IntelliJ.

My point was that it should be enough to store each module's report in a separate basedir, you don't really need a different file name. E.g. surefire generates core/target/surefire-reports/TEST-TestSuite.xml, not core/target/surefire-reports/TEST-core.xml. If you need to move all the reports in a single directory later, you could rename the files to include the module name at that stage.

Regarding debugging, I was able to sort-of debug ConfigurationAwareTestNGReporter by adding this dependency to core:

  <dependency>
     <groupId>org.apache.maven.surefire</groupId>
     <artifactId>surefire-testng</artifactId>
     <version>2.18.1</version>
     <exclusions>
        <exclusion>
           <groupId>org.testng</groupId>
           <artifactId>testng</artifactId>
        </exclusion>
     </exclusions>
  </dependency>

It didn't really work for me, as the downloaded sources didn't match the loaded classes (probably because I tried with 2.19.1 and there were other indirect dependencies on 2.18.1), but at least you'll be able to see some code :)

[alanfx]

Hey @danberindei, yes I forgot to add a null check for System.getenv("MAVEN_PROJECTBASEDIR"). I can push that change as well. QE will be taking these files and importing them into Polarion for each build, so I'd like to avoid the extra step of renaming the files. I was able to step into ConfigurationAwareTestNGReporter with the forkCount change, but in the end I'd still have to create a custom reporter. The test case name in ConfigurationAwareTestNGReporter is generated in a private method, and I have not been able to figure out a way to disable that listener. The POM file already sets usedefaultlisteners=false.

[alanfx]

@danberindei suggested using the infinispan.module-suffix property to name the results file which works, except for the Integration tests. I'm trying to figure that out now.

[slaskawi]

Hey @alanfx !

Just thinking out laud... wouldn't it be better to create a small script that would scan all directories after the build and push those results into Polarion? Surefire already spits out an xml and a text file (the latter is probably easier to parse).

This way we wouldn't interfere with the build itself and adding yet another step in Jenkins or TeamCity is really easy.

[alanfx]

Hey @slaskawi, eventually these steps will need to be automated, but initially this will be a more manual process. It usually takes a few runs of the tests to get a set of results that is worth reporting!

@danberindei I think this is ready to merge now, and I hope you'll be pleased! I got tripped up yesterday, because some of the integration tests do not use TestNg.

[danberindei]

Thanks Alan, integrated!

@slaskawi Even if the file name didn't matter, we'd still have the problem of test methods with dataProviders being duplicated...