Support MicroProfile Open API 2.0 (part of MP 4.0)

[Emily-Jiang]

Support MicroProfile Open API 2.0 release and also baseline Jakarta EE9.

• UFO: https://ibm.box.com/s/3fsseruy0s0rjc3opk9ajzs4t0gxj5o7
• FTS: Feature Test Summary for OpenAPI 2.0 #14186
• Beta blog: BETA BLOG - MicroProfile OpenAPI 2.0 Support #13970

---

List of Steps to complete or get approvals / sign-offs for Onboarding to the Liberty release (GM date)

Instructions:

• Do the actions below and mark them complete in the checklist when they are done.
• Make sure all feature readiness approvers put the appropriate tag on the epic to indicate their approval.

---

TARGET COMPLETION DATE Before Development Starts or 8 weeks before Onboarding

• POC Design / UFO Review Scheduled (David Chang) or N/A.
• POC Design / UFO Reviewed (Feature Owner) or N/A.
• Complete any follow-ons from the POC Review.
• Add "Design Approval Request" tag
• Design / UFO Approval (Alasdair Nottingham) or N/A.
• SVT Requirements identified. (Epic owner / Feature owner with SVT focal point)
• ID Requirements identified. (Epic owner / Feature owner with ID focal point)
• Create a child task of the epic entitled "FAT Approval Test Summary". Add and fill in the template as described here: https://github.ibm.com/was-liberty/WS-CD-Open/wiki/Feature-Review-(Feature-Test-Summary-Process . Mark "FAT complete" on the Epic when it's ready for review

1 week before beta GA

• Create, populate, and link to the Beta blog post issue

Legal

3 weeks before Onboarding

• Identify all open source libraries that are changing or are new. Work with Legal Release Services (Cass Tucker or Release PM) to get open source cleared and approved. Or N/A. (Epic Owner). New or changed open source impacts license and Certificate of Originality.

Translation

3 weeks before Onboarding

• All new or changed PII messages are checked into the integration branch, before the last translation shipment out. (Epic Owner)

Feature Complete

2 weeks before Onboarding

• Implementation complete. (Epic owner / Feature owner)
• All function tests complete. Ready for FAT Approval. (Epic owner / Feature owner)
• Review all known issues for Stop Ship. (Epic owner / Feature owner / PM)

Focal Point Approvals

2 to 1 week before Onboarding

You MUST have the Design Approved or No Design Approved label before requesting focal point approvals.

All features (both "Design Approved" and "No Design Approved")

• FAT - (Adam Yoho, Martin Holder, or Dave Waddling). SOE FATS are running successfully or N/A . Approver adds label focalApproved:fat to the Epic in Github.
• Demo - (Tom Evans or Chuck Bridgham). Demo is scheduled for an upcoming EOI. Approver adds label focalApproved:demo to the Epic in Github.
• Globalization (Sam Wong - Liberty / Simy Cheeran - tWAS). Translation is complete or N/A. TVT - complete or N/A. Approver adds label focalApproved:globalization to the Epic in Github.

"Design Approved" features

• Accessibility - (Steven Zvonek). Accessibility testing is complete or N/A. Approver adds label focalApproved:accessibility to the Epic in Github.
• ID - (Kareen Deen). Documentation work is complete or N/A . Approver adds label focalApproved:id to the Epic in Github.
• Performance - (Jared Anderson). Performance testing is complete with no high severity defects or N/A . Approver adds label focalApproved:performance to the Epic in Github.
• Serviceability - (Don Bourne). Serviceability has been addressed.
• STE - (Swati Kasundra). STE chart deck is complete or N/A . Approver adds label focalApproved:ste to the Epic in Github.
• SVT - (Brian Hanczaryk - APS). SVT is complete or N/A . Approver adds label focalApproved:svt to the Epic in Github.

Ready for GA

1 week before Onboarding

• No Stop Ship issues for the feature. (Epic owner / Feature owner / Release PM)
• Ship Readiness Review and Release Notes completed (Epic owner / Feature owner / Release PM)
• Github Epic and Epic's issues are closed / complete. All PRs are committed to the master branch. (Epic owner / Feature owner / Backlog Subtribe PM)

1 week before GA

• Create, populate, and link to the Blog post issue

Other deliverbles

• OL Guides - (Yee-Kang Chang). Assessment for OL Guides is complete or N/A.
• WDT - (Leonard Theivendra). WDT work complete or N/A.
• Blog - (Laura Cowen) Blog article writeup (Epic owner / Feature owner / Laura Cowen)

[msmiths]

The MicroProfile OpenAPI Specification provides a unified Java API for the OpenAPI v3 specification, that all application developers can use to expose their API documentation.

Support for version 1.0 of the MicroProfile OpenAPI Specification was delivered as part of the mpOpenAPI-1.0 feature in December 2017

Support for version 1.1 of the MicroProfile OpenAPI Specification was delivered as part of the mpOpenAPI-1.1 feature in February 2019

This issue captures the work required to add support for version 2.0 of the MicroProfile OpenAPI Specification, delivered as part of a new mpOpenAPI-2.0 feature.

Unlike previous versions, the mpOpenAPI-2.0 feature will be based on the SmallRye OpenAPI implementation.

[msmiths]

Review of the MicroProfile OpenAPI 2.0 UFO is scheduled for Monday 27th July at 10am EDT.

[jhanders34]

Actions from this mornings UFO socialization:

• Smallrye appears to fix all (so far) of the externally opened defects on mpOpenAPI 1.1. Either will close defects as fixed in 2.0 or work with mpOpenAPI 1.1 team to get them fixed in 1.1.

• Investigate SmallRye ui as a possible replacement for our UI?
  The various modifications that are made to the embedded Swagger UI in OL would need to be replicated in the SmallRye ui when repackaged as part of OL. This would require unpacking the JAR that is built by SmallRye, making the necessary modifications and then repackaging the JAR. Given that the Swagger UI in OL is used in more than one place... not just as part of the MP OpenAPI support, I believe that it adding a second copy does not provide any technical benefit.

• Problem statement should include information about the problem that the new annotations and mpConfig properties are solving, for example, 1. It is hard to configure some of the annotations. 2. Do not always have access to source to add annotations so want to be able to configure using config properties. 3. Scanning can hurt startup performance so want ways to tune the annotation scanning of libraries, certain annotations, etc.
  Done

• The mpConfig schema and small rye config functions are not included as user stories. They are a big part of your design. Relatively high level what the solutions to the problem statements that are above.
  Done

• Didn't see what APIs were being removed in the APIs section and it wasn't covered as a user story. Covered in the migration impact section at least. It was part of your user overview so it should be in the problem statement and user stories since you are removing function that does not have value and can be confusing.
  Done

• Smallrye also doing annotation scanning instead of using our own scanner is concerning for performance reasons. Need to follow up with Tom Bitonti more about making small rye more pluggable to be able to use the liberty annotation scanner which includes a caching between server starts if nothing changes. Performance team to do testing with POC build when available comparing mpOpenAPI 1.1 and the 2.0 POC with and without Jandex, etc. Mostly we are concerned of the impact to startup performance, but also throughput of mpOpenAPI endpoint calls.
  The Performance Team have analysed the performance of the new functionality and have found that there is no noticeable effect on startup time between mpOpenAPI-1.1 and mpOpenAPI-2.0. Making the annotation scanner that is use by the SmallRye implementation is possible but it will require several more weeks of development/test effort to implement and there is no guarantee that the resulting performance will be any better than it currently is. Also, concerns about the throughput of the /openapi endpoint are unfounded since this endpoint will receive a significantly fewer requests than the actual endpoints of the REST API being exposed (probably < 1%).

• For scan-dependencies.jars property is there any validation of the list? Detecting typos. Possibly a value add if it isn't in small rye already.
  Looking at the SmallRye code, the relevant getter method on their OpenApiConfigImpl class is not referenced, i.e., it is up to the SmallRye consumer to apply this configuration when building the IndexView that is passed to SmallRye. I already do this in the mpOpenAPI-2.0 implementation, but I do not check to ensure that every JAR file name that is specified in the list actually exists in the WEB-INF/lib directory of the application.
Of course, this does raise the question of whether we should perform similar validation on the lists of classes/packages defined by the following MP Config keys, which is something that we do not currently do:

- mp.openapi.scan.packages
- mp.openapi.scan.classes
- mp.openapi.scan.exclude.packages
- mp.openapi.scan.exclude.classes

• Follow up with Red Hat on why you would do schema references or not
  Done. The relevant MP Config property has not been removed from the SmallRye implementation.

• enable = true/false, disable = true/false. Not consistent. disable = false is double negative. Better to do do enable = true being default than disable=false. Follow up with Red Hat about being consistent. May have to make our own properties and just call down to the small rye ones if they can't change the small rye code.
  Actually, using disable=true/false is consistent with other MP OpenAPI related properties, i.e., mp.openapi.scan.disable which defaults to false. Considering this along with the context of the properties in question, I am satisfied that the property names are both suitable and consistent. I am still attempting to get agreement to drop the mp.openapi.extensions.smallrye.schema-references.enable property.

• Follow up with Red Hat about why ApplicationPath disable is in there at all.
  The rationale for the mp.openapi.extensions.smallrye.application-path.disable property is explained in the following issue: https://github.com/smallrye/smallrye-open-api/issues/247

[arthurdm]

Unfortunately I missed the review. Does the new design cover the new openapi/platform endpoint define in the spec?

[msmiths]

@arthurdm Good catch... no this was missed. However, reading over the Exposing platform OpenAPIs sections of the specification, it is not clear how this SPI works. Is the following pattern intended to show how to define a MP Config key for a specific platform API:

mp.openapi.spi.platform.<component>=<path_to_OAS3>

[arthurdm]

right - that MP Config defines a particular component that will be part of the platform endpoint.

[msmiths]

@arthurdm So, in OL, each MP component that wants to expose its own platform REST API will need to register an MP Config property of the form shown above... presumably using a custom config source? The MP OpenAPI feature will need to dynamically monitor these config properties so that it can respond to changes at runtime... and then server up the OpenAPI docs on the relevant URLs. Correct?

[arthurdm]

high level concept is correct @msmiths - details may vary, depending on how you want to implement it. =)

[scottcurtis2605]

Serviceability Approval Comment

1. WAD -- does the WAD identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

In the UFO, a few brief problems which developers faced with the 1.x implementation, which 2.0 resolves with API/SPI changes, are stated. These are addressed within the Externals Design section of the UFO.
For diagnostics, trace strings are also provided in the STE and UFO. The strings provided here give information for debugging the feature as a whole, as well as traces specifically for the SmallRye OpenAPI libraries which are consumed within mpOpenAPI-2.0.

However, aside from the new additions to mpOpenAPI-2.0 to comply with the MP OpenAPI-2.0 specification, the major serviceability concerns for this feature were regarding the changes to the underlying implementation between mpOpenAPI-1.x and mpOpenAPI-2.0. As 2.0 now consumes SmallRye OpenAPI Core and JAX-RS libraries, serviceability investigations involved ensuring that any issues encountered
during typical uses cases of OpenAPI were able to be resolved without external help from L2 and beyond. Typical use cases were defined as the following:

• Generating an OpenAPI document programmatically
• Filtering the OpenAPI tree elements
• Providing/extending an existing OpenAPI document

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. L2, test team, or another development team).

a) What problem paths were tested and demonstrated?

Common problem paths were identified through configuring Open Liberty with mpOpenAPI-2.0 and attempting the use cases described above. If issues were encountered while exercising the use cases, these became problem paths.

Serviceability of the feature was then defined through the user's ability to resolve issues encountered on the problem path using the warning/error messages produced by the feature. When an issue was encountered, stdout messages (without trace strings configured) were consoled.

Even if stdout was sufficient to overcome the issue, tracing was also enabled. This resulted in comparison of the more detailed trace against the regular output – if the trace was found to be more informative when retrospectively tackling an issue, this was noted.

Overall, if enabling tracing was perceivably beneficial to diagnose issues over regular output, this would justify raising a feature for greater emphasis on informing users of the trace strings provided with mpOpenAPI-2.0, making the feature more accessible. The following trace strings were utilised:

• 'mpopenapi=event=enabled' - for general feature logs to debug
• 'io.smallrye.openapi.*=all' - for logs generated specifically by the SmallRye Core and JAX-RS libraries.

In general, performing the actions of each of the use cases did not result in any issue; problem paths were then defined by making deliberate but plausible mistakes when attempting the use cases.
The following problem paths were derived:

1. Common Problem Path - Generating an OpenAPI document programmatically:

• Annotating a JAX-RS application class with malformed attributes
  Example: @APIResponseSchema annotation configured with an attribute which does not exist on the interface.
  Result: No OpenAPI-specific library errors as this is caught at compile time. Compile time errors are informative enough rectify the issue:

error: cannot find symbol, <offending-line>, symbol: method <non-existent attribute>(), location: @interface APIResponseSchema

• Annotating a JAX-RS application class with missing attributes which are required (as per the JavaDoc for the given annotation)
  Example: @APIResponseSchema annotation without defining the value attribute.
  Result: No OpenAPI-specific library errors as this is caught at compile time. Compile time errors are informative enough rectify the issue:

 error: annotation @APIResponseSchema is missing a default value for the element 'value'
 @APIResponseSchema(responseDescription = "host:properties pairs stored in the inventory.",

• Annotating a POJO with @Schema and @SchemaProperty annotations
  Example: Nesting @SchemaProperty within a @Schema annotation on a POJO to describe POJO fields.
  Result: Incorrect behaviour, with no output (with/without trace strings configured). @SchemaProperty attributes do not render on the resultant OpenAPI document, unless the value passed to the name attribute
  does not exist within the source for the POJO. The JavaDoc suggests that @SchemaProperty should override values of the POJO in the resultant OpenAPI document if the name attribute collides with an existing POJO field.

In response, I raised an issue for this, for which a PR has been raised which has since been tested, and resolves the @SchemaProperty bug - the behaviour is now consistent with the JavaDoc.

However, it should be noted that @SchemaProperty does work as intended through overriding/extending POJO field annotation when applied at the JAXRS application class level.

2. Common Problem Path - Filtering the OpenAPI tree elements:

• Failing to re-configure MP Config key, mp.openapi.filter, when the Filter class package/location has been modified
  Example: Refactoring the package structure of the filter class, but forgetting to update config properties.
  Result: No document is produced - OpenAPI Validation outputs a generic error message, which is not informative about what caused the error:

[ERROR   ] CWWKO1661E: An error occured when processing application {0} and an OpenAPI document was not produced.

The Knowledge Center entry for CWWKO1661E states to "Gather logs and contact IBM service". This was not sufficient to service this issue -
enabling trace strings was necessary to resolve this without external assistance.

When enabling the 'mpopenapi=event=enabled' trace string, the following output is more informative as to the cause of the error:

io.openliberty.microprofile.openapi20.ApplicationProcessor   1 Failed to process application: Failed to process application {0}: java.lang.ClassNotFoundException: <class path specified as the value of mp.openapi.filter>
io.openliberty.microprofile.openapi20.ApplicationProcessor   E CWWKO1661E: An error occured when processing application {0} and an OpenAPI document was not produced.

Enabling the 'io.smallrye.openapi.*=all' trace string led to diagnosis of the issue. This directs the user toward an FFDC, indicating that the problem is caused by the getFilter method:

Stack Dump = io.smallrye.openapi.runtime.OpenApiRuntimeException: java.lang.ClassNotFoundException: io.openliberty.guides.inventory.filter.NonExistentClass
at io.smallrye.openapi.runtime.OpenApiProcessor.getFilter(OpenApiProcessor.java:180)

3. Common Problem Path - Providing/extending an existing OpenAPI document:

• Passing an invalid/malformed OpenAPI document
  Example: Specifying a static openapi.yaml document, referencing components which were not defined
  Result: Validation component produces useful diagnostic information without requiring trace strings:

[INFO] [ERROR   ] CWWKO1650E: Validation of the OpenAPI document produced the following error(s):
[INFO]  - Message: The "#/components/schemas/InventoryList" reference value is not defined within the Components Object, Location: #/paths/~1inventory~1systems/get/responses/200/content/application~1json/schema
[INFO]  - Message: The "#/components/schemas/InventoryList" value is an invalid reference, Location: #/paths/~1inventory~1systems/get/responses/200/content/application~1json/schema

This gave co-ordinates to the section of the OpenAPI document which was missing, followed by the section of the document which refers to the missing content, making this issue easy to rectify.

• Specifying an invalid schema (mp.openapi.schema.*) for a class
  Example: Defining a schema with an array field, but failing to define the data type of the array
  Result: Validation component produces diagnostic information which enables the user to correct the issue:

[INFO] [ERROR   ] CWWKO1650E: Validation of the OpenAPI document produced the following error(s): 
[INFO]                                                                                                           
[INFO]  - Message: The Schema Object of "array" type must have "items" property defined, Location: #/components/schemas/AutomaticTransmission
[INFO] 

• Specifying invalid JSON (mp.openapi.schema.*) when defining class schema
  Example: Defining a schema with invalid JSON
  Result: SmallRye library outputs the following (with/without trace strings) and the schema is omitted from the resultant OpenAPI document. This was descriptive enough such that fixing the issue was straight-forward:

[WARNING ] SROAP04003: Configured schema for <FQCN which couldn't be defined> could not be parsed

In conclusion, the problems encountered with mpOpenAPI-2.0 were resolvable without needing to escalate to L2 when enabling trace strings. However, to devs new to OpenAPI, the available tracing/logging options are not well emphasised or documented. I personally found serviceability frustrating when facing an issue which only output the generic CWWKO1661E error message. This made the debugging process extremely difficult without prior knowledge of the trace settings available. It's important to distinguish that this is not a serviceability issue with the feature - this is an issue with the feature's documentation which impacts serviceability by proxy.

We should make a note of tracing/debugging materials within the GA release documentation/material, as
this could greatly improve the developer experience with mpOpenAPI-2.0.

b) Who did you demo to?

It is important to note that as I am new to the development of this feature, I was able to take the perspective of a user without comprehensive knowledge of the internals of mpOpenAPI-2.0. Therefore, my ability to resolve any issue which was encountered was dependent on the produced output; if I was able to resolve encountered issues, this would provide evidence to suggest that this feature is serviceable for common use cases.

The same use cases were also demonstrated to @Joseph-Cass, who develops mpConfig rather than mpOpenAPI. I believe that our combined ability to successfully resolve any encountered issue then implies the overall serviceability of mpOpenAPI-2.0 to those who are not involved in the development of this feature.

c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that L2 should be able to quickly address those problems without need to engage L3?

@Joseph-Cass Agreed:
If the issues mentioned (and those raised on gitub) are addressed, then the serviceability should be sufficient.

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
   a) Who conducted SVT tests for this feature?

@hanczaryk

b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that L2 should be able to quickly address those problems without need to engage L3?

Yes, see previous comment.

4. Which L2 / L3 queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective L2/L3 teams know they are supporting it. Ask Don Bourne if you need links or more info.

WL3-CDI

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

N/A

[chirp1]

ID issues are #3445, #340. Martin Smithson has move to WebSphere Automation. Tom Evans talked to Martin who agreed to be the SME, but the WebSphere Automation eGA takes priority. The blog post for the beta will be used for the blog post for the eGA of this epic. Customers will have the necessary information to use this epic until the docs are done. I'm approving this epic.

[donbourne]

@scottcurtis2605 , a few comments on serviceability:
I like your approach on investigating the serviceability on the new SR implementation. In particular...

• "Even if stdout was sufficient to overcome the issue, tracing was also enabled. This resulted in comparison of the more detailed trace against the regular output – if the trace was found to be more informative when retrospectively tackling an issue, this was noted."
• "making deliberate but plausible mistakes when attempting the use cases."

Your comments about CWWKO1661E error message suggest that something should be done to make the problem determination easier. Do you think it's possible for that to be simpler, or is there some reason that trace and FFDC are the only way to diagnose this? Documenting how to trace this could help, but that seems like a temporary solution.

[scottcurtis2605]

@donbourne
Thanks for the feedback!

I'll open an issue regarding this - while the documentation does give a temporary solution, I think for a permament solution going forward it would be good to have the CWWKO1661E message reading something like the following, giving information about the exception which has occurred, rather than just the application name:

Current:

[ERROR   ] CWWKO1661E: An error occured when processing application <application name> and an OpenAPI document was not produced.

Proposed:

[ERROR   ] CWWKO1661E: An error occured when processing application <application name> and an OpenAPI document was not produced. The error is <stack-trace>

[donbourne]

@scottcurtis2605 that sounds better (so that you don't have to look in FFDC to find exception info). Can you please open an issue for that and provide link here, so I can sign off?

[scottcurtis2605]

@donbourne Here is the issue and the associated PR.