Feature Test Summary for OpenAPI 2.0

[scottcurtis2605]

Epic: #11020

1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature. The description should include the positive and negative testing done, whether all testing is automated, what manual tests exist (if any) and where the tests are stored (source control). Automated testing is expected for all features with manual testing considered an exception to the rule.

For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.

If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?

Test Summary for Open Liberty OpenAPI 2.0

For the introduction of the OpenAPI 2.0 feature, we aimed to maximise code coverage through automated FAT and TCK tests through various means. This strategy involved three steps:

1. Executing the Eclipse Microprofile OpenAPI TCKs specific to 2.0's new functionality. This is detailed in the New OpenAPI 2.0 Features section of the summary.

2. Updating the existing test bucket from OpenAPI 1.x to repeat 1.x tests against 2.0, and implementing new 1.x tests to maximise compatibility for the SmallRye implementation of OpenAPI 2.0. This is detailed in the OpenAPI 1.x Repeated FATs section of the summary.

3. Implementing a new FAT bucket specifically for 2.0, testing what we believed necessary. This is detailed in the OpenAPI 2.0 Specific FATs section of the summary.

New OpenAPI 2.0 Features

Below is the list of new features exclusive to OpenAPI 2.0 (taken from the 2.0 release notes), each followed by a sub-list of the PRs from the MicroProfile Open Api TCKs which test the feature. These TCKs are ran from the io.openliberty.microprofile.openapi.2.0.internal_fat_tck bucket.

API/SPI Changes

• The @SchemaProperty annotation has been added to allow the properties for a schema to be defined
      - This PR includes updates to an example application. The JAXRSApp and AirlinesAppTest classes now implement a single test to validate inline schema definitions with @SchemaProperty.

• The @RequestBodySchema annotation has been added to provide a shorthand mechanism to specify the schema for a request body.
      - This PR includes updates to an example application. The PetResource and PetStoreAppTest classes now implement a single test to validate positive @RequestBodySchema POST functionality from a client perspective.

• The @APIResponseSchema annotation has been added to provide a shorthand mechanism to specify the schema for a response body.
      - This PR includes updates to an example application. The PetResource and PetStoreAppTest classes now implement two tests to validate @APIResponseSchema positive POST and GET functionality from a client perspective.

• The mp.openapi.schema.* MicroProfile Config property has been added to allow the schema for a specific class to be specified. This property would typically be used in cases where the application developer does not have access to the source code of a class.
      - This PR includes updates to an example application. The PetStoreApp, OASConfigSchemaTest, and schema-microprofile-config.properties classes were updated to provide a test to validate positive class Schema Registration without access to a class source by using mp.openapi.schema.*.

Functional Changes

• Getter methods on model interfaces that return a list or map now return a copy of the list/map containing the same items. This list/map CAN be immutable.
      - This PR includes the ModelConstructionTest class to validate immutability holds on list/map collections containing openapi/models/ interfaces.

• Setter methods on model interfaces that take a list or a map as a parameter MUST not use the list/map instance directly.
      - This PR includes updates to the ModelConstructionTest class (as modified above) to to ensure list/map setters can accept immutable list/maps.

Other Changes

• JavaDoc updates to clarify the behaviour of getter methods on model interfaces that return a list or map.
      - This PR contains updates for this feature. Of course, no tests for JavaDocs.

• TCK updates to verify that getter methods on model interfaces return a list or map, return a copy of underlying collection.
      - Changes in this PR, mentioned in Functional Changes above, updates the ModelConstructionTest class to validate immutability holds on list/map collections containing openapi/models/ interfaces.

New and Updated FATs Beyond New OpenAPI 2.0 Features

The specification update between MicroProfile OpenAPI 1.1 and 2.0 has resulted in incompatible changes. Therefore, the following PR provides new FATs exclusive to 2.0, and updates to repeat those from 1.x against 2.0.

OpenAPI 1.x Updated FATs

The existing OpenAPI 1.x FAT bucket, com.ibm.ws.microprofile.openapi_fat, has been updated to provide coverage and compatibility for the SmallRye implementation of OpenAPI 2.0. This requires 1.x FATs to be repeated, and in some cases, new 2.0-compatable implementations of tests within this bucket.

The following are the 1.x tests which have been updated to be repeated against version 2.0:

• ApplicationProcessorTest
• OpenAPIValidationTestOne
• OpenAPIValidationTestTwo
• OpenAPIValidationTestThree
• OpenAPIValidationTestFour
• OpenAPIValidationTestFive
• OpenAPIValidationTestSix
• ProxySupportTest
• EndpointAvailabilityTest
• UICustomizationTest

In addition, the following 1.x classes were updated with 2.0-compatable tests to account for the SmallRye implementation differences between 1.x and 2.0. This ensures compatibility between the com.ibm.ws.microprofile.openapi_fat bucket and OpenAPI 2.0.

• OpenAPIValidationTestOne - testInfoValidation20 to be ran in place of testInfoValidation under OpenAPI 2.0.

• OpenAPIValidationTestThree - testBlankInfo20 to be ran in place of testBlankInfo under OpenAPI 2.0.

• ApplicationProcessorTest class has been duplicated accross to the 2.0 bucket, due to the necessity of underlying environment changes to execute testCompleteFlow against 2.0. The remaining tests within ApplicationProcessorTest are natively ran within the 1.x bucket, against 2.0.

OpenAPI 2.0 Specific FATs

The following FAT test classes present in the io.openliberty.microprofile.openapi.2.0.internal_fat bucket from the FATSuite.

• CacheTest, which tests scenarios involving the model cache; this was implemented to improve performance by avoiding rescans of classes upon server restart.
• DeploymentTest, which tests scenarios which were previously missing in previous buckets - these ensure OpenAPI works correctly when the app is deployed in different formats (ear, war, etc.).

As a team

2) Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

0 - No automated testing delivered

1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.

2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

As per @tevans78's comment, "We discussed at length and agreed, on average, a score of 4".

[tevans78]

We discussed at length and agreed, on average, a score of 4.

[ayoho]

Additional negative scenarios are being added to the TCK and being tracked by smallrye/smallrye-open-api#621. With those appearing to be in hand, I'll go ahead and close out this test summary.