TestNG or Standalone support

[andrewtytula]

Trying to use spring-restdocs but it is tightly coupled to JUnit. The main issue is in RestDocumentation which implements TestRule (JUnit specific). The apply method returns a wrapper to the provided statement that when evaluated sets up the context, runs the test and then sets the context back to null. When running the test in TestNG, I get a NullPointerException in the beforeOperation method since the context is never created.

I used reflection to make the context accessible and assigned it a value myself and the test runs properly but it is a hack solution. Simple solution would be to provide a way for the test to manually tell the RestDocumentation that it has started or stopped (and maybe provide a TestNG listener that uses it) or make context accessible. It would be better if the JUnit code lived one layer deeper so if JUnit isn't on the classpath so it can detect TestNG and utilize it's listeners instead.

[wilkinsona]

I certainly have no objections to supporting TestNG and I'm happy, within reason, to make the necessary API changes to make that happen.

I'm not very experienced with TestNG, but from what I've seen a TestNG listener isn't a replacement for a JUnit rule. The biggest difference seems to be that you can't create a listener with any arguments. That means that, as far as I can tell, there's no TestNG equivalent of:

@Rule
public final RestDocumentation restDocumentation = new RestDocumentation("target/generated-snippets");

Have I missed something in TestNG's capabilities?

[andrewtytula]

No, this seems to be a limitation to TestNG. Since posting this issue, I have been trying to create my own workaround solution. The only problem I am having using listeners is getting a handle on RestDocumentation instance since listeners are created by TestNG.

I think a solution might be to provide a startTest(...) and finishTest(...) method that can be called manually by the test when not using JUnit. It will make the TestNG tests a little more verbose but at least they will work.

I have found threads where people are asking TestNG to implement something like @rule so maybe in the future.

[wilkinsona]

Thanks, @andrewtytula

[vodrazka]

It would be great if spring-restdocs worked with testng (out of box) :)

[sg-ad]

Just wondering if anybody made any progress in addressing this? On my side, I also started looking at TestNG listeners. I find the cleanest type to use is IInvokedMethodListener, but there are difficulties. Here are some main differences that pose a challenge:

1. TestNG itself creates an instance of the listener, so no parameters can be provided by constructor. It would be possible to pass attributes via the TestContext, but doing so means intrusive setup in a test since you would now have add a configuration method to setup these attributes.
2. Since TestNG creates the instance and the test has no handle on it, there's no way to explicitly pass it to the configurer.
3. AFAIK, TestNG listeners are not guaranteed to be thread-safe.

For the first item, I found what I think is a decent solution and created an annotation that can be placed either on the test class or test method to pass some parameters that the listener can retrieve.

For items 2 and 3, one possibility would be to provide a TestNG listener factory, but that could possibly interfere with existing tests (TestNG only allows a single factory to be defined, so if a user already has a factory defined they will clash). The other possibility I see is to manage the listener instance using thread local so that it can be retrieved easily. This could offer a way to address the thread-safety issue.

What do you think? I'll continue exploring it a bit and could present something a bit more concrete, but if you don't see it as a viable option I might drop it because using a customized version of Spring-Restdocs is not really where I want to go. :)

[wilkinsona]

From Spring REST Docs' perspective, I think the best thing to do is to ignore TestNG listeners. As @andrewtytula suggested above, having a pair of methods that can be called before and after a test runs feels like the best way to solve this. These methods can then be called in @BeforeMethod and @AfterMethod-annotated methods in your TestNG-based tests.

The remaining tricky part of this is introducing these new methods in an elegant way and without breaking backwards compatibility. They could just be added to RestDocumentation but then people would be required to have both JUnit and TestNG on the classpath.

[sg-ad]

Yes, that would be simpler. What about extracting an abstract superclass for RestDocumentation, keeping the existing name as the JUnit flavour, and introducing a TestNG flavour like TestngRestDocumentation? That would avoid the need to have both frameworks on the classpath. Granted, the naming isn't ideal for the JUnit flavour, but at least it's backward compatible.

[wilkinsona]

Thanks for the suggestion. I've considered something along those lines, I just haven't managed to convince myself that it's a sufficiently elegant solution. Ideally, the superclass would be called RestDocumentation as it's not specific to TestNG or any other test framework.There'd then be a JUnitRestDocumentation subclass that implements TestRule. I can't do that without breaking backwards compatibility though. I'll keep thinking. Perhaps I can convince myself that StandaloneRestDocumentation isn't too bad. RestDocumentation could then be a subclass of it. It could even be deprecated with a JUnitRestDocumentation being provided as the preferred alternative.

[sg-ad]

As goes the saying: naming things is one of the great problems of computer science! :)