Define A TestSuite In A Class And Not XML

[opticyclic]

TestNG Version

7.1.0

Expected behavior

TestNG should be able to specify the test classes with annotations just like JUnit can:
https://github.com/junit-team/junit4/wiki/aggregating-tests-in-suites

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
  TestFeatureLogin.class,
  TestFeatureLogout.class,
  TestFeatureNavigate.class,
  TestFeatureUpdate.class
})

public class FeatureTestSuite {
  // the class remains empty,
  // used only as a holder for the above annotations
}

Actual behavior

Currently you have to define your suite in XML and can't use annotations in a class.

[krmahadevan]

@opticyclic - This is an interesting proposition :)
But can you please help explain as to why you wouldn't want to rely on a suite xml created and run tests from that? JUnit I believe didn't have the notion of xml suites and so they resorted to defining a suite via a Java class. TestNG on the other hand is aware of a suite xml file and so can work with it.

Essentially both these are accomplishing the same ask (of course with testng doing more by letting you define groups or other method selectors via a bean shell, letting your define listeners, letting you control your parallelism strategy etc.,)

@juherr @cbeust WDYT ?

[opticyclic]

Basically the same reasons why Spring added the ability to define Beans in a class instead of only in xml.

Their competitor (Guice) had it and people don't like editing xml as much.

Just to be clear, I'm not asking to get rid of the xml.
I'm asking for the option to have a choice of whether I want to use xml or annotations to define my suite.

[krmahadevan]

@opticyclic - While we wait to hear back from @cbeust and @juherr can you please check if something like this would work for you ?

1. Create a custom annotation that looks like below

import static java.lang.annotation.ElementType.TYPE;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({TYPE})
public @interface SuiteClass {

  String name();

  Class<?>[] classes();
}

2. Here's how a custom suite altering listener which is responsible for parsing this above cited custom annotation (You can wire this listener as a mandatory listener by using the Service Loader approach)

import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
import org.testng.IAlterSuiteListener;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class LocalSuiteAlterer implements IAlterSuiteListener {

  @Override
  public void alter(List<XmlSuite> suites) {
    Optional<? extends Class<?>> found = suites.stream()
        .flatMap(xmlSuite -> xmlSuite.getTests().stream())
        .flatMap(xmlTest -> xmlTest.getXmlClasses().stream())
        .map(XmlClass::getSupportClass)
        .filter(cls -> cls.getAnnotation(SuiteClass.class) != null)
        .findAny();
    if (found.isPresent()) {
      SuiteClass suiteClass = found.get().getAnnotation(SuiteClass.class);
      Class<?>[] classes = suiteClass.classes();
      XmlSuite xmlSuite = new XmlSuite();
      xmlSuite.setName(suiteClass.name() + "_suite");
      XmlTest xmlTest = new XmlTest(xmlSuite);
      xmlTest.setName(suiteClass.name() + "_test");
      List<XmlClass> xmlClasses = Arrays.stream(classes).map(XmlClass::new).collect(Collectors.toList());
      xmlTest.setXmlClasses(xmlClasses);
      suites.clear();
      suites.add(xmlSuite);
    }
  }
}

3. Here's how the test class would look like

@SuiteClass(name = "CodingSuite", classes = {One.class, Two.class, Three.class})
public class RunnerTest {

}

All the sample test classes (One, Two and Three would look like below)

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class One {

  @Test
  public void method() {
    ITestResult result = Reporter.getCurrentTestResult();
    System.err.println("Executing " + result.getMethod().getQualifiedName() + "()");
  }
}

Caveats:
IDE may not be able to run this test (RunnerTest) out of the box from IDE because that class doesn't contain any TestNG annotation. So you would have to create a custom run configuration to run it from the IDE. I tested this on IntelliJ and it works fine.

[krmahadevan]

[opticyclic]

That probably would work in the interim and is likely to be useful in projects where I can't update to a newer version. 👍

Hopefully, this means that it shouldn't be too hard to add it to core TestNG.

After all @cbeust we don't want a comparison table with JUnit having a tick in a box and TestNG having a cross! :)

[juherr]

The idea is good but I don't know if the 3rd party support will follow quickly.
An additional way to configure could be a DSL instead of annotations too (a move some libraries/frameworks are doing).

IMO, this feature is a very good excuse to rework the model around XmlXxx where XmlXxx should be the way to transform XML into the model and the new feature a way to transform annotations into the model.

I have just some doubts about the naming because @Test from test class won't be the same as @Test from the description. We will have to be good.

[krmahadevan]

@opticyclic - I spent some time and built a full fledged library that now supports this outside of TestNG.

Take a look at https://github.com/RationaleEmotions/sangrahah

[krmahadevan]

@juherr - Do you think we can close this issue given the fact that there are alternatives and now a full fledged library that I built which can solve this issue ?

[juherr]

Feature implemented in a 3rd party