Abstract Test Suite

[cportele]

The Abstract Test Suite (Annex A) is work-in-progress. Currently, only a draft for the Core conformance class is available, the other five conformance classes (HTML, GeoJSON, GML-SF0, GML-SF2, OpenAPI 3.0) are not yet covered.

OGC Testbed 14 is expected to contribute, too.

[cmheazel]

The ATS does provide testing for HTML, GeoJSON, GML-SF0, GML-SF2, and OpenAPI 3.0. However, these conformance classes do not come into play until you are way down in the details of individual tests. An additional section in the introduction which explains how the conformance classes are addressed may help.

[cmheazel]

I'm OK with your additions. We need to give some thought as to how far to go in content validation:

1. Should we follow every link to an alternate encoding and validate that document as well?
2. How rigorous should we be in validating that the Path Item and Operation objects are complete and accurate representations?
   This gets very complex very quickly.

[dstenger]

I just reviewed the current status of the ATS.

These are a few general thoughts:

• Shouldn't we remove the term 'micro-service' as an WFS 3.0 interface is not restricted to micro-services?
• Structure of test suite: This is just a very general question. Why is the structure of the tests different to the chapters of the specification and the corresponding requirements? Wouldn't it be easier to use the same structure as introduced by chapters and to create one test for each requirement?
• Quite a few tests name OpenAPI as precondition or validate against OpenAPI 3.0 schema. However, chapter 5.6.2. Role of OpenAPI states that OpenAPI is not mandatory and is covered by a separate conformance class. Thus, shouldn't dependencies to OpenAPI be removed from tests for Core conformance class?

These are my comments targeting single tests (out of the perspective of the person implementing the tests):

• A.4.1.1. HTTP 1.1
  • Build all requests using the HTTP 1.1 protocol. is very abstract. In practice, a test suite cannot build all possible requests. This could be reduced to a couple of requests
• A.4.1.2. Coordinate Reference Systems
  • If there is no default encoding for data, how can data be validated? Currently, the specification states that not even one of the four proposed encodings (HTML, GeoJSON, GML-SF0, GML-SF2) is mandatory. Thus, it is impossible to implement a test working with all implementations.
• A.4.4.10. Validate the Get Features Operation Response
  • Test is just possible if one of the four proposed, optional encodings (HTML, GeoJSON, GML-SF0, GML-SF2) is used.
• A.4.4.15. Validate the Get Feature Operation Response
  • Same as for A.4.4.10.

I hope my thoughts help you to further improve the ATS.

[cmheazel]

Re: Microservices. While it is true that WFS 3.0 is not restricted to microservices, it is also true that this is not a traditional OWS service. The REST/API design pattern essentially does away with the idea of a service in favor of a collection of web accessible resources. This has surprising impacts on how the "service" is designed and used. How do we foot-stomp that this is not Web Services as you know it? Using the term microservice vs. Web Service is one approach.

[cmheazel]

RE: Test suite structure. The first version of the ATS followed the traditional requirement by requirement approach. It rapidly became unmanageable. This version uses a client-centric approach. It mimics the steps a client (or test engine) would have to execute to discover testable end-points and then execute them. Keep in mind that:

1. WFS 3.0 does not prohibit end-points which are not part of the standard,
2. The REST/API pattern does not require that all end-points reside on the same server,
3. Most everything is optional. You have to process the OpenAPI doc to discover the requirements you must test against.

[cmheazel]

RE: OpenAPI. See the API section of the ATS. Development of a compliance test is almost impossible without some assumptions about the API description. This ATS assumes OpenAPI 3.0. Most of the OpenAPI 3.0 test suite will be reusable if we decide to test against Swagger or future versions of OpenAPI.

[cmheazel]

RE: A.4.1.1. HTTP 1.1. Agree. I'm open to suggestions.

[cmheazel]

RE: A.4.1.2, 4.4.11, 4.4.15. At a minimum the compliance test should support validation of the two encodings recommended for the core (HTML and GeoJSON). The two GML encodings should also be supported, but at a lower priority. Each encoding has a corresponding conformance class. You should be able to determine which encodings are supported through the /conformance path. We are exploring extensions to the OpenAPI Media Type Object which will allow us to associate a specific schema (ex. GML 3.2) with a response. This should provide a finer level of control over the encodings supported by each resource path.

[dstenger]

@cmheazel Thank you for the detailed answers.

Re: Microservices
I get your point. However, I do not like the term micro-service here. My proposal is to just use RESTful service.

RE: A.4.1.1.
As implementation of HTTP is usually done by used web server (e.g. Apache HTTP server, nginx), I don't think that we need extensive testing here. If just a basic request complies to HTTP 1.1, we can assume that the web server uses this protocol for all requests.
So, my proposal is to just request the landing page (path /).

RE: OpenAPI
RE: A.4.1.2, 4.4.11, 4.4.15.
So, we can conclude that the Core conformance class is so generic that it is not possible to develop an generic abstract test suite which can be used by all implementations.
I conclude that there are currently following assumptions:

• OpenAPI 3.0 is used.
• One of the following encodings is supported: HTML, GeoJSON, GML-SF0 or GML-SF2.

This should be marked clearly in ATS preconditions. For OpenAPI 3.0 this is already done but not for encodings. Maybe, the assumption can even be placed in a general preconditions or assumptions chapter?
Also, we can think of making the corresponding conformance classes mandatory for ATS: Tested implementations must also implement OpenAPI and one of HTML, GeoJSON, GML-SF0 or GML-SF2 conformance classes.
These assumptions prevent testing Core conformance class individually.
Maybe, we should also add thoughts about further ATS to specification which can be used to test implementations not using OpenAPI or one of the four encodings.

@ALL Especially the last section should be reviewed by other persons as well.

[cmheazel]

RE: Microservices - RESTful service works for me. As long as we capture the concept that:

1. We are not talking about monolithic services
2. We recognize that the "service" may be implemented across multiple servers. So there may not be a common root to the paths.

[cmheazel]

RE: A.4.1.1 - HTTP compliance
Concur. It's a requirement so we need to test it but not exhaustively.

[cmheazel]

RE: OpenAPI
I think we are in agreement that OpenAPI 3.0 and that at least one of the four encodings must be supported. Failure to meet these assumptions does not mean that your service is not compliant, just that it is not testable.

I tried to capture this concept in the ATS but clearly did not do a good enough job. Suggestions and contributions are welcome.

[cmheazel]

In a RESTful world I think we have to treat each path as a discrete service. The basic approach is to crawl the OpenAPI document exploring each path. If the path is a WFS 3.0 defined path, then you evaluate it against the requirements for that path. If it is not a WFS 3.0 defined path then you ignore it.
So testing is based on paths and the associated requirements. Compliance classes assist in expressing the requirements, but they are not themselves testable.

[dstenger]

Currently, the abstract test suite just covers Core conformance class (Requirement 1 - 32).

As already mentioned in the first comment, Requirement 33+ are not covered yet.

Are there already any thoughts how to address the other conformance classes and how to integrate them in the abstract test suite? For implementation of test suite (ETS) it is important that the conformance classes are covered by different tests.

[cmheazel]

@dstenger : In a number of places in the ATS you will find text like the following
"Validate the structure of the response as follows:
For HTML use TBD
For GeoJSON use featureCollectionGeoJSON.yaml
For GML use featureCollectionGML.yaml"
This is where the ATS addresses requirements 33+. WFS 3.0 requires that a service return links to all supported encodings of a response. Therefore, the ATS resolves and validates all returned links, even if the media type is not identified by a conformance class. The only addition I can think of would be to validate that the returned links agree (no more and no less) with the encoding conformance classes.

[dstenger]

Thank you, yes, now I am seeing the connection.

However, this leads to a general problem linking the Abstract Test Suite (ATS) to the Executable Test Suite (ETS).
All executions of ETS produce a report which groups the results of tests into conformance classes. By this, users can easily see if all tests for a single conformance class pass and thus if the tested instance is compliant to this conformance class.
In this case, the ATS defines tests which cover more than one conformance class. As the ETS represents an implementation of the ATS (as close as possible), the ATS breaks the requirement that a single test can be assigned to a single conformance class.
Thus, I propose to separate tests which cover different conformance classes (from view of practical testing).
@bermud We should discuss this (maybe also during Testbed 14 meeting).

[cmheazel]

@dstenger - I pondered this issue as well. It isn't clear to me how you can structure the tests so that different conformance classes are separate ETS unless you re-run the entire test for each encoding. That doesn't seem to be an efficient approach to me. We could consider an alternate approach where you select the conformance classes first, then the test exercises all of the conformance classes selected. Or we could maintain that these are configuration parameters and not conformance classes at all. Are they really in the same class as - say - the Transaction extension?

[dstenger]

A.4.3.3. Process Server Object seems to be incomplete. The case that an URI template variable does not have have an enumerated set of valid values and no default value is not considered.

Also see opengeospatial/ets-ogcapi-features10#5 (comment)

[dstenger]

In A.4.4.1. Validate /api path [1] a test method is not described.

Also see opengeospatial/ets-ogcapi-features10#6 (comment)

[1] https://rawgit.com/opengeospatial/WFS_FES/master/docs/17-069.html#_validate_api_path

[dstenger]

The test A.4.4.6. seems to be a copy from A.4.4.5., test step 2. does not describe Requirement 13:

Test method:

2. Validate that the collection metadata document includes links for:
  * Itself
  * Alternate encodings of this document in every other media type as identified by the 
     compliance classes for this server.

Requirement 13:

/req/core/fc-md-items-links

For each feature collection in this distribution of the dataset, the links property of the 
collection SHALL include an item for each supported encoding with a link to the 
collection resource (relation: item).

All links SHALL include the rel and type properties.

Expected test method:

2. Validate that the collection metadata document includes links to the collection for each 
    supported media type as identified by the compliance classes for this server.

Also see opengeospatial/ets-ogcapi-features10#7 (comment) and opengeospatial/ets-ogcapi-features10#36 (comment)

[dstenger]

A.4.4.15. Validate the Get Feature Operation Response: Missing hint to assertion "All links SHALL include the rel and type link parameters." (Requirement 32).

Also see opengeospatial/ets-ogcapi-features10#17 (comment)

[lgoltz]

A.4.4.10. Validate the Get Features Operation Response, test method 6:

If a property numberMatched is included in the response, iteratively follow the next links 
until no next link is included and count the aggregated number of features returned in all 
responses during the iteration. Validate that the value is identical to the numberReturned 
stated in the initial response.

Shouldn't numberReturned be numberMatched in the last sentence?