Abstract Test Suite

[cportele]

The Abstract Test Suite (Annex A) is work-in-progress. Currently, only a draft for the Core conformance class is available, the other five conformance classes (HTML, GeoJSON, GML-SF0, GML-SF2, OpenAPI 3.0) are not yet covered.

OGC Testbed 14 is expected to contribute, too.

[cmheazel]

The ATS does provide testing for HTML, GeoJSON, GML-SF0, GML-SF2, and OpenAPI 3.0. However, these conformance classes do not come into play until you are way down in the details of individual tests. An additional section in the introduction which explains how the conformance classes are addressed may help.

[cmheazel]

I'm OK with your additions. We need to give some thought as to how far to go in content validation:

1. Should we follow every link to an alternate encoding and validate that document as well?
2. How rigorous should we be in validating that the Path Item and Operation objects are complete and accurate representations?
   This gets very complex very quickly.

[dstenger]

I just reviewed the current status of the ATS.

These are a few general thoughts:

• Shouldn't we remove the term 'micro-service' as an WFS 3.0 interface is not restricted to micro-services?
• Structure of test suite: This is just a very general question. Why is the structure of the tests different to the chapters of the specification and the corresponding requirements? Wouldn't it be easier to use the same structure as introduced by chapters and to create one test for each requirement?
• Quite a few tests name OpenAPI as precondition or validate against OpenAPI 3.0 schema. However, chapter 5.6.2. Role of OpenAPI states that OpenAPI is not mandatory and is covered by a separate conformance class. Thus, shouldn't dependencies to OpenAPI be removed from tests for Core conformance class?

These are my comments targeting single tests (out of the perspective of the person implementing the tests):

• A.4.1.1. HTTP 1.1
  • Build all requests using the HTTP 1.1 protocol. is very abstract. In practice, a test suite cannot build all possible requests. This could be reduced to a couple of requests
• A.4.1.2. Coordinate Reference Systems
  • If there is no default encoding for data, how can data be validated? Currently, the specification states that not even one of the four proposed encodings (HTML, GeoJSON, GML-SF0, GML-SF2) is mandatory. Thus, it is impossible to implement a test working with all implementations.
• A.4.4.10. Validate the Get Features Operation Response
  • Test is just possible if one of the four proposed, optional encodings (HTML, GeoJSON, GML-SF0, GML-SF2) is used.
• A.4.4.15. Validate the Get Feature Operation Response
  • Same as for A.4.4.10.

I hope my thoughts help you to further improve the ATS.

[cmheazel]

Re: Microservices. While it is true that WFS 3.0 is not restricted to microservices, it is also true that this is not a traditional OWS service. The REST/API design pattern essentially does away with the idea of a service in favor of a collection of web accessible resources. This has surprising impacts on how the "service" is designed and used. How do we foot-stomp that this is not Web Services as you know it? Using the term microservice vs. Web Service is one approach.

[cmheazel]

RE: Test suite structure. The first version of the ATS followed the traditional requirement by requirement approach. It rapidly became unmanageable. This version uses a client-centric approach. It mimics the steps a client (or test engine) would have to execute to discover testable end-points and then execute them. Keep in mind that:

1. WFS 3.0 does not prohibit end-points which are not part of the standard,
2. The REST/API pattern does not require that all end-points reside on the same server,
3. Most everything is optional. You have to process the OpenAPI doc to discover the requirements you must test against.

[cmheazel]

RE: OpenAPI. See the API section of the ATS. Development of a compliance test is almost impossible without some assumptions about the API description. This ATS assumes OpenAPI 3.0. Most of the OpenAPI 3.0 test suite will be reusable if we decide to test against Swagger or future versions of OpenAPI.

[cmheazel]

RE: A.4.1.1. HTTP 1.1. Agree. I'm open to suggestions.

[cmheazel]

RE: A.4.1.2, 4.4.11, 4.4.15. At a minimum the compliance test should support validation of the two encodings recommended for the core (HTML and GeoJSON). The two GML encodings should also be supported, but at a lower priority. Each encoding has a corresponding conformance class. You should be able to determine which encodings are supported through the /conformance path. We are exploring extensions to the OpenAPI Media Type Object which will allow us to associate a specific schema (ex. GML 3.2) with a response. This should provide a finer level of control over the encodings supported by each resource path.

[dstenger]

@cmheazel Thank you for the detailed answers.

Re: Microservices
I get your point. However, I do not like the term micro-service here. My proposal is to just use RESTful service.

RE: A.4.1.1.
As implementation of HTTP is usually done by used web server (e.g. Apache HTTP server, nginx), I don't think that we need extensive testing here. If just a basic request complies to HTTP 1.1, we can assume that the web server uses this protocol for all requests.
So, my proposal is to just request the landing page (path /).

RE: OpenAPI
RE: A.4.1.2, 4.4.11, 4.4.15.
So, we can conclude that the Core conformance class is so generic that it is not possible to develop an generic abstract test suite which can be used by all implementations.
I conclude that there are currently following assumptions:

• OpenAPI 3.0 is used.
• One of the following encodings is supported: HTML, GeoJSON, GML-SF0 or GML-SF2.

This should be marked clearly in ATS preconditions. For OpenAPI 3.0 this is already done but not for encodings. Maybe, the assumption can even be placed in a general preconditions or assumptions chapter?
Also, we can think of making the corresponding conformance classes mandatory for ATS: Tested implementations must also implement OpenAPI and one of HTML, GeoJSON, GML-SF0 or GML-SF2 conformance classes.
These assumptions prevent testing Core conformance class individually.
Maybe, we should also add thoughts about further ATS to specification which can be used to test implementations not using OpenAPI or one of the four encodings.

@ALL Especially the last section should be reviewed by other persons as well.

[cmheazel]

RE: Microservices - RESTful service works for me. As long as we capture the concept that:

1. We are not talking about monolithic services
2. We recognize that the "service" may be implemented across multiple servers. So there may not be a common root to the paths.

[cmheazel]

RE: A.4.1.1 - HTTP compliance
Concur. It's a requirement so we need to test it but not exhaustively.

[cmheazel]

RE: OpenAPI
I think we are in agreement that OpenAPI 3.0 and that at least one of the four encodings must be supported. Failure to meet these assumptions does not mean that your service is not compliant, just that it is not testable.

I tried to capture this concept in the ATS but clearly did not do a good enough job. Suggestions and contributions are welcome.