Support for JSON Schema

[francescobbo]

Hi,

at the company I work for we're evaluating a few different tools for the complete workflow of our API management and this project really seems brilliant. Kudos to the maintainer(s)!

I was very interested in the testing functionality but we don't want to use SoapUI or Postman to define our tests. Long story short we decided to focus our efforts on a single official Swagger / OpenAPI documentation, both for the public API and for each service API, as we were looking for some more widely accepted/open/interoperable formats. (Also, adding JS code to an API doc just feels wrong, but that's my personal opinion.)

The thing is that the only test runner that is left would be the HTTP one which, as described in the docs, is just a smock test!

Is there any plan or even desire to support a JSON Schema test runner? It would mean supporting a vendor agnostic, standard (to be), open source (arguably less powerful, compared to JS) format for describing the response body which seems like a good middle ground between the simplicity of the HTTP runner and the power of the POSTMAN runner.

Thanks

[nmasse-itix]

Hi Francesco !

Thanks for your contribution. Are you looking at Microcks for the mocking part, the testing part or both ?

We are using the Postman/SOAPUI collections to define examples and tests. The examples are used for the mocking part whereas the tests are used, well, for the test part.

We could use the JSON Schema from the swagger to run conformance tests on the existing implementations. I think the expected behavior is well understood.

We could also use the example/examples fields when included in the swagger file to generate the mocks but the expected behavior is less commonly accepted.

For instance, you can define multiple examples for input payload, multiple examples for input parameters, multiple examples for output payload. But, there is no existing concepts to relate them together.

For instance, if an operation is defined in a swagger file for POST /owner/{owner}/car/{car}/passenger (to add a passenger to a car owned by a person), you could define :

• two examples for the id parameters:

- car: redcar
  owner: john
- car: bluecar
  owner: jane

• two examples for the input body:

- firstName: Nicolas
  lastName: Massé
- firstName: Laurent
  lastName: Broudoux

• and two examples for the output body:

- id: 123
  firstName: Nicolas
  lastName: Massé
  in: 
    - redcar
- id: 456
  firstName: Laurent
  lastName: Broudoux
  in: 
    - bluecar

But there is no concept in a swagger file to relate them together as a story.

Plus, to generate a mock from the examples, we need to bind an input parameters and an input body to an output body.

We could of course adopt a convention saying that the first parameter example goes with the first input body example and with the first output body example. But this puts an external constraint on the Swagger file.

What your you view on this ? And again, thanks for your feedback ! :)

[francescobbo]

Hi @nmasse-itix and thanks for the quick reply.

We were evaluating Microcks for both concerns actually.

We could use the JSON Schema from the swagger to run conformance tests on the existing implementations. I think the expected behavior is well understood.

I think this is a reasonable and very desirable behaviour, and a bonus point for open standards like Swagger! The only thing that slightly bothers me is that the JSON Schema in a swagger file is not actually JSON Schema but an extension built on top of it. I don't know the exact reasoning but for some reason the Swagger team made it incompatible which makes interoperability slightly harder.

Personally I don't believe that an API documentation file like Swagger, Postman and SoapUI should contain tests, but on the other hand the tools they built work and are used, which means that they made something easy to use, but not necessarily correct. What would be an alternative? Maybe Microcks may accept a Swagger definition and allow to create tests (as in coded tests, not conformance to a schema) in its UI without adding them to the documentation file.

But in the end who decided that doc, examples and test should live all in the same place? Maybe an alternative would be taking a zip file with a tree like:

/swagger.yml
/examples/endpoint1/request1.json
/examples/endpoint2/request1.json
/examples/endpoint2/request2.json
/tests/endpoint1/test.js

If it all sounds random and nonsense, keep in mind I just finished my long working day and my brain didn't receive enough caffeine :)

[andrewdavidmackenzie]

I tend to agree that "an API documentation file like Swagger, Postman and SoapUI should NOT contain tests" and that extensive use of examples is a workaround at best.... but difficult to capture all you would want to do in tests also.

It seems like there is scope for an Open file format for describing API tests, that leverages what's defined in the OAI spec (and probably references it), that multiple test tools could use to test the API, functional tests but also generating a set of valid requests for load testing (which is a separate subject).

[lbroudoux]

Hi Fabio, Andrew, Nicolas,

I agree for the separation of concerns for tests regarding Swagger/OpenAPI files.

However, I would say that Postman collections or SoapUI projects are different beasts as they don't carry any syntactical contractual informations. They can be purely dedicated to samples definitions and extra expectations and assertions (call them tests for short) regarding API usage. They can be used to define some kind behavioral contractual informations. Don't want to start a flamewar but Swagger/OpenAPI is much more like WSDL to my mind ;-)

That said, I tend to think that practically speaking, samples and extra expectations are closely tied together. Samples definition is not only a matter of validating syntactic contract but also finding the good examples that will illustrate correctly the behavioral expectations. In my opinion, Postman and SoapUI have succeed here because they are simple and may be used by functional people for this steps - even if they are not fully correct and may embed some other concerns (documentation, load tests, ...)

What would be the alternatives ? I can think of 2 options for now:

• We may add direct Swagger/OpenAPI spec to Microcks. As Nicolas highlighted it above, there's still issues regarding how to bind examples together to create whole story for mocking but I'm sure we can rely on naming conventions or schema extensions for that.

=> As a result we can have mocks directly from examples representing basic samples,
=> We can also have syntactic tests by simply checking JSON schema compliance.

• We may also work at some other tools / standards to allow easy definitions of samples and expectations regarding an API defined using Swagger/OpenAPI. It may allow editing samples in the respect of syntactic contract. It may allow definition of expectations regarding different use-cases of API. Ideally, it may be usable by functional people to allow capture of business requirements easily. Finally, it may take the form suggested by Fabio with samples and expectations defined alongside the openapi.yaml file.

=> As a result, we can have mock representing advanced samples,
=> We can also have behavior tests that validates conformity to business expectations.

Of course, the 2 options are not exclusive.

[lbroudoux]

Hi @aomega08 ,

did you have the chance to check ongoing related issues ? I think things are getting quite complete. I'll be good having your thoughts on this.

Cheers,

[lbroudoux]

No activity for more than 30 days. Closing it.