how to enable generateTestResultAttributes using Maven #1673
Comments
|
@bitcoder - Its not clear what exactly is the problem. Can you please help elaborate a bit more on what is this issue all about ? |
|
Sure. For this I built an annotation listener which extends ITestListener and where I add an attribute after the test is run: Anyway, I need that these testResult attributes be saved in the generated XML file. Anyway, I'm using maven and I was unable to succesffuly make this attribute happen as expected. Here is my sample project: |
|
@bitcoder - I think the documentation is a bit outdated. I don't find any way in the codebase to basically enable what you are looking for. That said and done, here's a way in which you can still get this done.
Here's a sample that shows all of this in action. import org.testng.ITestNGListener;
import org.testng.ITestNGListenerFactory;
import org.testng.Reporter;
import org.testng.TestNG;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
import org.testng.internal.ClassHelper;
import org.testng.reporters.XMLReporter;
import java.io.File;
@Listeners({Testclass.MyListenerFactory.class, Testclass.MyReporter.class})
public class Testclass {
@Test
public void testMethod() {
System.err.println("Hello World");
Reporter.getCurrentTestResult().setAttribute("Krishnan", "Mahadevan");
}
public static class MyListenerFactory implements ITestNGListenerFactory, ITestNGListener {
@Override
public ITestNGListener createListener(Class<? extends ITestNGListener> listenerClass) {
System.err.println("Printing " + listenerClass.getName());
ITestNGListener instance = ClassHelper.newInstance(listenerClass);
if (instance instanceof XMLReporter) {
String current = TestNG.getDefault().getOutputDirectory() + File.separator + "new" + File.separator;
((XMLReporter) instance).getConfig().setOutputDirectory(current);
((XMLReporter) instance).getConfig().setGenerateTestResultAttributes(true);
}
return instance;
}
}
public static class MyReporter extends XMLReporter {
}
}The resultant xml would look like below <?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="0" failed="0" ignored="0" total="1" passed="1">
<reporter-output>
</reporter-output>
<suite name="Default Suite" duration-ms="32" started-at="2018-01-26T17:52:33Z" finished-at="2018-01-26T17:52:33Z">
<groups>
</groups>
<test name="testbed" duration-ms="32" started-at="2018-01-26T17:52:33Z" finished-at="2018-01-26T17:52:33Z">
<class name="com.rationaleemotions.github.beanshellproblem.Testclass">
<test-method status="PASS" signature="testMethod()[pri:0, instance:com.rationaleemotions.github.beanshellproblem.Testclass@262b2c86]" name="testMethod" duration-ms="8" started-at="2018-01-26T17:52:33Z" finished-at="2018-01-26T17:52:33Z">
<reporter-output>
</reporter-output>
<attributes>
<attribute name="Krishnan">
<![CDATA[Mahadevan]]>
</attribute> <!-- Krishnan -->
</attributes>
</test-method> <!-- testMethod -->
</class> <!-- com.rationaleemotions.github.beanshellproblem.Testclass -->
</test> <!-- testbed -->
</suite> <!-- Default Suite -->
</testng-results>
|
|
Thanks @krmahadevan :) |
|
@bitcoder - Have you tried following the section "Using Custom Listeners and Reporters" in that link ? That basically tells you how to wire in your listeners ( This reporter is also a listener) If not, then you can try going through my blog which talks about setting up TestNG listeners in general via various different mechanisms Blog link : https://rationaleemotions.wordpress.com/2012/01/27/listen-to-what-i-have-to-say-about-testng-listeners/ |
|
@bitcoder |
|
@krmahadevan Example, Sometimes it is more better to enable some functionality just by setting parameter but not writing listeners |
|
@Kanaduchi - I hear you. But no that's not possible as of today. I dont see any mechanisms in the codebase that would facilitate this. |
|
@krmahadevan Am I right? I can try to modify code and test my suggestion |
|
you can pass system properties in surefire plugin if the listener/reporter supports read parameters from system properties. |
|
@krmahadevan , your comment is really a good addition for the doc: [1], [2] and the snippet of surefire configuration is quite good example as well, I think it's better to revise the documentation with these. |
|
It could be an improvement for the surefire documentation too. I'm sure @Tibor17 will love it too. |
|
@juherr |
|
@Tibor17 The IDEA is just to improve the surefire documentation of reporters with a sample. |
|
@juherr |
|
Here's the content that needs to be added Here's how it can be done (I have found that it can be done ONLY via the Maven surefire plugin configuration i.e., by passing via surefire properties. ) Surefire plugin configuration would look like below <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
<properties>
<property>
<name>reporter</name>
<value>org.testng.reporters.XMLReporter:generateTestResultAttributes=true,generateGroupsAttribute=true</value>
</property>
</properties>
</configuration>
</plugin>The value for the property name
Here
name value pair should be separated by Can we add a new page under http://maven.apache.org/surefire/maven-surefire-plugin/examples/ with its title as *Customizing TestNG reporters via Properties" |
|
@missedone for [1] there's already documentation available. Here's the contents
for [2] I have raised a new PR testng-team/testng-team.github.io#23 |
|
thanks @krmahadevan and everyone here involved! |
|
I am using this configuration <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version> 2.20.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
<properties>
<property>
<name>usedefaultlisteners</name>
<value>false</value>
<!-- disabling default listeners is optional -->
</property>
<property>
<name>reporter</name>
<value>org.testng.reporters.XMLReporter:generateTestResultAttributes=true,generateGroupsAttribute=true</value>
</property>
</properties>
</configuration>
</plugin>but still the attributes are not generated. what could be the issue? |
|
@vivekthangathurai - You seem to have disabled the default reports via |
|
@krmahadevan tried but same result, no attributes generated. using Oxygen eclipse EE java. |
|
Surefire maps the properties in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-testng/src/main/java/org/apache/maven/surefire/testng/conf/TestNGMapConfigurator.java
…On Mon, May 14, 2018 at 8:40 PM, Vivek Thangathurai < ***@***.***> wrote:
@krmahadevan <https://github.com/krmahadevan> tried but same result, no
attributes generated. using Oxygen eclipse EE java.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1673 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AA_yR-8xZGMDHbUJ759LtdGUThSkE29dks5tyc-3gaJpZM4RuUUM>
.
--
Cheers
Tibor
|
|
@krmahadevan I know issue is closed but i need help on this and I am working on XRAY and facing the same issue. I tried the above solution but nothing happens.no attributes generated ..................... package CalcTest;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.Assert;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.Reporter;
import org.testng.reporters.XMLReporter;
import org.testng.ITestResult;
import Xray_Automation.Demo.*;
import Xray_Automation.Demo_annotations.Xray;
import Xray_Automation.Demo_annotations.XrayListener;
@Test
//@Listeners({XrayListener.class})
public class TMTest2 {
@BeforeMethod
@BeforeSuite
public void setUp() throws Exception {
}
@AfterMethod
@AfterSuite
public void tearDown() throws Exception {
}
@DataProvider
public Object[][] ValidDataProvider() {
return new Object[][]{
{ 1, 2, 3 },
{ 2, 3, 4 }, // error or the data itself :)
{ -1, 1, 0 }
};
}
@Test(dataProvider = "ValidDataProvider")
@Xray(requirement = "CALC-1234", test = "CALC-1")
public void CanAddNumbersFromGivenData(final int a, final int b, final int c)
{
AssertJUnit.assertEquals(Calculator.Add(a, b), c);
// System.out.println(requirement);
}
@Test
@Xray(requirement = "CALC-1234", test = "CALC-2", labels = "core addition")
public void CanAddNumbers()
{
AssertJUnit.assertEquals(Calculator.Add(1, 1),2);
AssertJUnit.assertEquals(Calculator.Add(-1, 1),0);
}
@Test
@Xray(requirement = "CALC-1235", labels = "core")
public void CanSubtract()
{
AssertJUnit.assertEquals(Calculator.Subtract(1, 1), 0);
AssertJUnit.assertEquals(Calculator.Subtract(-1, -1), 0);
AssertJUnit.assertEquals(Calculator.Subtract(100, 5), 95);
}
@Test
@Xray(requirement = "CALC-1236")
public void CanMultiplyX()
{
AssertJUnit.assertEquals(Calculator.Multiply(1, 1), 1);
AssertJUnit.assertEquals(Calculator.Multiply(-1, -1), 1);
AssertJUnit.assertEquals(Calculator.Multiply(100, 5), 500);
}
@Test
@Xray(requirement = "CALC-1237")
public void CanDivide()
{
AssertJUnit.assertEquals(Calculator.Divide(1, 1), 1);
AssertJUnit.assertEquals(Calculator.Divide(-1, -1), 1);
AssertJUnit.assertEquals(Calculator.Divide(100, 5), 20);
}
@Test
@Xray(requirement = "CALC-1237")
public void CanDoStuff()
{
Assert.assertNotEquals(true, true);
}
}.................................................................................................................................... package Xray_Automation.Demo_annotations;
import java.lang.reflect.Method;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import Rest_Request.ImportTesngResults;
import static java.lang.System.out;
import java.io.IOException;
import static java.lang.System.err;
/**
* The listener interface for receiving Xray events.
* The Listener can be automatically invoked when TestNG tests are run by using ServiceLoader mechanism.
* You can also add this listener to a TestNG Test class by adding
* <code>@Listeners({com.xpand.java.XrayAnnotationListener.class})</code>
* before the test class
*
* @see Xray
*/
public class XrayListener implements IInvokedMethodListener, ITestListener {
String requirement=null;
String test =null;
String labels=null;
boolean testSuccess = true;
/* (non-Javadoc)
* @see org.testng.IInvokedMethodListener#beforeInvocation(org.testng.IInvokedMethod, org.testng.ITestResult)
*/
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
if(method.isTestMethod() && annotationPresent(method, Xray.class) ) {
testResult.setAttribute("requirement", method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Xray.class).requirement());
testResult.setAttribute("test", method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Xray.class).test());
testResult.setAttribute("labels", method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Xray.class).labels());
}
}
private boolean annotationPresent(IInvokedMethod method, Class clazz) {
boolean retVal = method.getTestMethod().getConstructorOrMethod().getMethod().isAnnotationPresent(clazz) ? true : false;
return retVal;
}
/* (non-Javadoc)
* @see org.testng.IInvokedMethodListener#afterInvocation(org.testng.IInvokedMethod, org.testng.ITestResult)
*/
public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
if(method.isTestMethod()) {
if( !testSuccess ) {
testResult.setStatus(ITestResult.FAILURE);
}
}
}
public void onTestStart(ITestResult result) {
// TODO Auto-generated method stub
}
public void onTestSuccess(ITestResult result) {
// TODO Auto-generated method stub
}
public void onTestFailure(ITestResult result) {
// TODO Auto-generated method stub
}
public void onTestSkipped(ITestResult result) {
// TODO Auto-generated method stub
}
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
// TODO Auto-generated method stub
}
public void onStart(ITestContext context) {
for(ITestNGMethod m1 : context.getAllTestMethods()) {
if(m1.getConstructorOrMethod().getMethod().isAnnotationPresent(Xray.class)) {
//capture metadata information.
requirement= m1.getConstructorOrMethod().getMethod().getAnnotation(Xray.class).requirement();
test= m1.getConstructorOrMethod().getMethod().getAnnotation(Xray.class).test();
labels= m1.getConstructorOrMethod().getMethod().getAnnotation(Xray.class).labels();
}
}
}
public void onFinish(ITestContext context) {
// TODO Auto-generated method stub
ImportTesngResults callrest= new ImportTesngResults();
try {
callrest.createtestexec();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}......................................................................................................................................................................... <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Xray_Automation</groupId>
<artifactId>Demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<!--XRay Properties -->
<!--IN PROFILE ~.m2/settings.xml-->
<!--<xray.jiraURL></xray.jiraURL>
<xray.resultsFormat>JUNIT</xray.resultsFormat>
<xray.username>syed</xray.username>
<xray.password>123qwe</xray.password>-->
<xray.projectKey>TM</xray.projectKey>
<!--
<xray.testExecKey></xray.testExecKey>
<xray.testPlanKey></xray.testPlanKey>
<xray.testEnvironments></xray.testEnvironments>
<xray.revision></xray.revision>
-->
<xray.surefire.location>${basedir}/target/surefire-reports</xray.surefire.location>
<!--End Xray Properties -->
</properties>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<debug>true</debug>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
<properties>
<property>
<name>reporter</name>
<value>org.testng.reporters.XMLReporter:generateTestResultAttributes=true,generateGroupsAttribute=true</value>
</property>
</properties>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.11</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.mashape.unirest/unirest-java -->
<dependency>
<groupId>com.mashape.unirest</groupId>
<artifactId>unirest-java</artifactId>
<version>1.4.9</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpasyncclient-osgi -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpasyncclient-osgi</artifactId>
<version>4.1.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.json/json -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20190722</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.12</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpmime -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-surefire-report-plugin -->
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>3.0.0-M3</version>
</dependency>
</dependencies>
<reporting>
<plugins>
<plugin>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>3.0.0-M3</version>
</plugin>
<plugin>
<groupId>CalcTest</groupId>
<artifactId>xray-maven-plugin</artifactId>
<version>1.0.0</version>
</plugin>
</plugins>
</reporting>
</project>.......................................................................... <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test name="Test">
<classes>
<!-- class name="CalcTest.CalcTest"/-->
<class name="CalcTest.TMTest2"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->.............................................................. No attribute is showing |
|
@ssoban Please, could you open a new issue with a full runnable project? |
|
Hi @ssoban , |
|
@krmahadevan |
|
All of the responses are talking of the class name as "org.testng.reporters.XMLReporter". But I'm using the "org.testng.Reporter" class for generating the testng-results.xml. I'ven't explicitly used XMLReporter in my code. Even then, should I add below line in POM under SureFire Plugin Configuration. @krmahadevan <property>
<name>reporter</name
<value>org.testng.reporters.XMLReporter:generateTestResultAttributes=true,generateGroupsAttribute=true</value>
</property> |
|
@ChathaKiran - The above configuration will basically enable dumping of attributes embedded in test results in the xml report (This is a default report that TestNG automatically wires in).
|
|
Hi guys, I'm working on an integration between TestNG + Xray Jenkins plugin + Jira Xray in order to report results. I have managed to get it working (it creates test executions, summary in Jira=test method name and passes the correct test status to them) using the TestNG XML multipart option but I would also like to push the @test method description to Jira Xray which is present inside the
I assume this needs to be configured in the Test Fields JSON inside the Jenkins "Xray Results import task" with some sort of variable that reads the description from the testng-results.xml and passes it to the Summary or Description fields in Jira:
Any ideas on how to do this? |
|
Hi @vlad230 , |
|
Hi @bitcoder, Thanks a lot for the fast response :) But maybe I need to somehow enable the listeners to generate/see the extra information in the testng-results.xml? I have used it like this: I have tried to use the one mentioned in your link but Maven cannot resolve it and I cannot find it on maven central: Was it removed from the public repository? I only see xray-maven-plugin & xray-junit-extensions from this groupId. Also, is there a way to use the description defined in the @test annotation from TestNG as Jira summary/description? I need it to also to show up in Allure Reports and it uses it from here (of course I could duplicate the description in both annotations but was looking for a more elegant way). |
|
Ok, so first of all, please discard these old dependencies: <dependency>
<groupId>com.xpandit.xray</groupId>
<artifactId>xray-testng-extensions</artifactId>
<version>1.1.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.xpandit.xray</groupId>
<artifactId>xray-maven-plugin</artifactId>
<version>1.1.0</version>
<scope>test</scope>
</dependency>Use this one instead: <dependency>
<groupId>app.getxray</groupId>
<artifactId>xray-testng-extensions</artifactId>
<version>0.1.0</version>
<scope>test</scope>
</dependency> It was not published on Maven Central (I was pending an internal ok); I just published it. Please note that the annotations changed with the new xray-testng-extensions package, as detailed in the docs. Concerning your last question, now you'll have something like this: @Test
@XrayTest(summary = "addition of two numbers", description = "tests the sum of two positive integers")
public void CanAddNumbers()The problem is that allure uses the |
|
Hi @bitcoder, I've created the issue on your original project since issues are disabled on the app.getxray one. Thanks a lot for publishing the library, looking forward to seeing it available on maven central :) I have read the docs on the new plugin and it should be OK for me to use for now :) |
|
Oh, I fixed that Issues "issue" :) |
|
@bitcoder added the issue in the new project here: Xray-App/xray-testng-extensions#1 Any ideas how long does it usually take for the published library to be available? https://search.maven.org/search?q=g:app.getxray |
|
it can be a few hours @vlad230 per Maven Central documentation |
TestNG Version
Expected behavior
testresult attributes should be present in the XML report
Actual behavior
there are no testresult attributes present in the XML report
Is the issue reproductible on runner?
Test case sample
How can I setup TestNG, using Maven, for setting up the generateTestResultAttributes flag to true?
I tried several ways but I was unable to set it up; i'm unsure if there is a bug or not.
In my pom.xml I added this kind of config:
The text was updated successfully, but these errors were encountered: