When using REST Assured, investigate if it's possible to defer documentation generation until after other conditions have been evaluated

[wilkinsona]

Feedback from @jlstrater:

When using the rest assured version, the documentation tests fail before other conditions like the status code. I'm not sure if this is by design but it was slightly confusing

This is due to the use of a Filter and the fact that it's called before the assertions in the then phase.
I've just looked at the possibility of the Filter returning a Response wrapper that can then hook into then() being called, but I don't think that would be enough to allow the documentation to be generated after all of the other conditions had been evaluated.

Another possibility (that would also read better) would be something like then().do(document(…)). This would also be a closer match for the approach with Spring MVC Test. Unfortunately, there's no such API in REST Assured that I'm aware of and then deals purely with responses whereas REST Docs needs access to both the request and the response.

@johanhaleby If you have a moment, I'd welcome your thoughts on this one please. The current approach looks like this:

given(this.documentationSpec)
        .accept("text/plain")
        .filter(document("sample", 
                preprocessRequest(modifyUris()
                        .scheme("https")
                        .host("api.example.com")
                        .removePort())))
.when()
        .port(this.port)
        .get("/")
.then()
        .assertThat().statusCode(is(200));

The Filter that's returned from document can throw an exception if the documentation doesn't match the service. Ideally, the failure would occur after checking that the response's status code was 200.

[johanhaleby]

There's currently no such feature in REST Assured. Filters, as you indicate, are executed before assertions take place. But why is this a problem? Can assertions included in the documentation somehow?!

[wilkinsona]

Can assertions included in the documentation somehow?!

They can indeed, that's one of Spring REST Docs' main features.

REST Docs will verify that what you have documented actually matches the request and response. The example above was probably poorly chosen as it doesn't show this in action. A better example is:

given(this.documentationSpec)
        .accept("application/json")
        .filter(document("hypermedia-sample", 
                links(
                        linkWithRel("self").description("Canonical location of this resource"),
                        linkWithRel("foo").description("Blah"))))
.when()
        .port(this.port)
        .get("/")
.then()
        .assertThat().statusCode(is(200));

In this case, a failure will occur if the response doesn't contain self and foo links (in HAL format). The failure is reported to the user by an exception being thrown from the Filter that's returned from the document method.

[wilkinsona]

@johanhaleby Sorry, belated thanks for responding so quickly.

[johanhaleby]

So is the problem that the user is presented with the error thrown from the filter and not errors defined in the "then-assertions" (then().assertThat()...) and this is confusing? (which I agree with btw, I just want to understand :))

[jlstrater]

Correct.

[johanhaleby]

Ok I see. Hmmm, I wonder if it's always the case that you want to defer the exceptions thrown by filters? If so maybe I could implement support for this. But it might be difficult since RA doesn't require assertions (then()..). It will be more tricky if this should be configurable somehow.. But the way I see it this is not an issue with restdocs but rather with rest-assured itself?

[wilkinsona]

@johanhaleby From a readability perspective, I'd really like Spring REST Docs' users to be able to do something like this:

given(this.documentationSpec)
        .accept("application/json")
.when()
        .port(this.port)
        .get("/")
.then()
        .assertThat().statusCode(is(200))
        .do(document("hypermedia-sample", 
                links(
                        linkWithRel("self").description("Canonical location of this resource"),
                        linkWithRel("foo").description("Blah"))));

I'm probably asking a lot, though. The new do method would need to take a type that has access to both the request and the response.

But the way I see it this is not an issue with restdocs but rather with rest-assured itself?

It's looking that way. I raised the issue here as I wasn't sure if I'd missing something, for example wrapping the Response and trying to hook into then somehow, and also as, if there's a change to REST Assured, then REST Docs may also need to change to take advantage of some new goodness in REST Assured.

[johanhaleby]

@wilkinsona I understand what you mean.

But what happens when you do links(linksWithRel("self"), ...? If I remember it correctly rest docs uses json-path to validate if the rel is correct and I assume that rest docs throws exception if it doesn't match?

From RA's perspective it would have been better if this exception was not thrown and instead linksWithRel would somehow add an assertion to rest assured instead using rest assured's json path (which confusingly is not the same). For example if the response looked like this:

{
    "_links": {
        "foo": { "href": "/orders", "description" : "Blah"},
    }, ...

Rest Assured's jsonpath could be used to extract the value of foo's description like and just append it to the assertion that will be checked later by (from the filter):

given().filter((requestSpec, responseSpec, ctx) -> {
             requestSpec.body("_links.foo.description", equalTo("<expected value>"));
         }). ...

I suspect that this will require a lot of work from your side though and perhaps it's not even possible without having to supply the requestSpec instance to the links or linksWithRel method?

A work-around that might work is if I allow passing a lambda expression to body in the requestSpec? If so you might not have to eagerlt throw an exception when using linkWithRel but rather just do something like this:

requestSpec.body((response) -> <do something that might throw exception>));

Could this help? (sorry if this message is incoherent, I've written it in a hurry)

[wilkinsona]

From RA's perspective it would have been better if this exception was not thrown and instead linksWithRel would somehow add an assertion to rest assured instead using rest assured's json path

The links support is only one example that happens to map reasonably nicely onto a existing REST Assured API. There are others that don't have such a mapping. For example, Spring REST Docs will throw an exception if you document a query parameter that wasn't actually part of the request (unless you mark it as optional).

I suspect that this will require a lot of work from your side though and perhaps it's not even possible without having to supply the requestSpec instance to the links or linksWithRel method?

It would indeed be a lot of work, and, as you suggest, would almost certainly require core types in Spring REST Docs to become aware of REST Assured with I don't want to do.

Rest Assured's jsonpath could be used to extract the value of foo's description

The description isn't extracted from the response. It's a description of the link that should be included in the generated documentation. REST Docs just looks for a link with the foo relation to ensure that you're not documenting something that doesn't actually exist.

A work-around that might work is if I allow passing a lambda expression to body in the requestSpec?

REST Docs verifies things based on both the request and the response so it might be a bit confusing for this to be done on either the request spec or the response spec. That said, it would be internal to REST Docs so I could certainly live with it.

Could this help?

I just tried something that I think is equivalent to this using responseSpec.body(Matcher) in REST Docs' Filter. The matcher was still called before the status code expectation was checked so it didn't produce the desired end result.

[johanhaleby]

Thanks for the clarification. But if linksWithRel throws exception on every failure how can you present all failures at once when using MockMvc?

The description isn't extracted from the response. It's a description of the link that should be included in the generated documentation. REST Docs just looks for a link with the foo relation to ensure that you're not documenting something that doesn't actually exist.

Of course, I do actually know this :) (was in hurry when I answered)

A work-around that might work is if I allow passing a lambda expression to body in the requestSpec?

Here I actually meant responseSpec (if it matters).

I agree that from rest docs perspective this would be better:

given(this.documentationSpec)
        .accept("application/json")
.when()
        .port(this.port)
        .get("/")
.then()
        .assertThat().statusCode(is(200))
        .do(document("hypermedia-sample", 
                links(
                        linkWithRel("self").description("Canonical location of this resource"),
                        linkWithRel("foo").description("Blah"))));

But it will require a lot of time to implement this (I think) and currently I cannot see any other use cases for it (which make me a bit reluctant to add support for this). This is what you would refer to as ResultHandlers in MockMvc right? REST Assured has historically been able to use filters to handle this.

Perhaps the conclusion is that we'll have to live with this for now?

[wilkinsona]

Thanks for the clarification. But if linksWithRel throws exception on every failure how can you present all failures at once when using MockMvc?

It throws an exception describing all of the problems with the documentation of the links. The key thing is that this exception is thrown before any other expectations are checked. That's the opposite of what happens with MockMvc and is what left @jlstrater slightly confused.

But it will require a lot of time to implement this (I think) and currently I cannot see any other use cases for it (which make me a bit reluctant to add support for this).

Understood. I thought I might be asking a lot.

Perhaps the conclusion is that we'll have to live with this for now?

Yes, I think so. Thanks for taking the time to help us to explore a few possibilities.

[johanhaleby]

I'll keep this in the back of my head to see if something pops up in the future.