Proposal: a consumer-optimized form of OpenAPI descriptions

[earth2marsh]

TL;DR Introduces the concept that publishing an OpenAPI document for consumer-use means resolving any remote references.

An OpenAPI description can be spread across multiple files using JSON Pointers. This is particularly useful for authoring, such as when different people are responsible for different parts of the description.

But in order for that description to be used in practice, all of those references need to be resolved. This is true whether it is being used to rendering documentation for humans, validating the well-formedness of requests at runtime, or generating an SDK.

Implications:

• If a circular reference cannot be resolved, then the description is not able to be made publishing-ready
• OpenAPI should consider offering an open source publishing tool and make it available as a service
• Any implementation-oriented information, like x-extensions that configure policy, should be stripped

[lornajane]

I'm unclear if this solves an existing problem. How can an OpenAPI description ever not meet the criteria of having resolvable references and being useful to consumers? Already it's common practice to validate, or even dereference, OpenAPI descriptions during an API workflow, could you say a bit more about where you see problems and how this helps? (sorry, I think I just don't understand)

[handrews]

@earth2marsh I agree (I think) that there's a need for a way to package up a complete OAD for easy transfer, deployment, and processing.

I also think that the combination of Darrel's imports syntax and the additional requirements I added around import resolution and loading changes the calculus here, particularly when you include ways to fence in JSON Schema's $ref usage.

In 3.x, there are a lot of complex steps to implement, and ambiguities to resolve, to support referencing. In Moonwalk, the imports process is just:

1. establish a base IRI (self if present, or externally provided if not)
2. resolve relative IRI-reference imports (which are all in one place) to full IRIs
3. map the full import IRIs to a location
4. perform any necessary I/O to retrieve the document from the location
5. index the Components Object for each namespace by semantic type + name
6. resolve references from the index as a simple (namespace, semantic type, name) hash lookup (no document tree walking)

Steps 1, 2, 5, and 6 are trivial. The imports proposal (#72) is structured to allow steps 3 and 4 to be handled outside of OAS implementations, with the results provided to implementations in an interoperable way.

Steps 3 and 4 can involve a lot of complexity, and I think you are right to try to eliminate it. A single-document package format would make steps 3 and 4 trivial for tools that understand that format.

There are a few things that I think we want to avoid here:

• ballooning the size of the packaged format, especially for large APIs with high levels of re-use (certain types of reference removal can lead to exponential growth in schema size)
• slowing down processing (it's slower to parse a much larger spec, especially in YAML, than just indexing the namespaces and components, which you have to parse either way)
• reducing the set of data structures that can be described (tree and other structures require cyclic references)
• making it too hard to provide meaningful file/url+line number error messages (too much processing and transformation is hard to track)
• stripping extensions (makes them kinda pointless and removes a way for tooling vendors to differentiate themselves)

---

A simple size-and-line-number-preserving packaging solution would be to take inspiration from JSON Text Sequences (application/json-seq) and define an OpenAPI Document Sequence format (YAML also has document streams, but if were trying to optimize performance, we'll want JSON). The rules to this would be very simple (mostly ignoring JSON Schema for now):

• all imports (and JSON Schema references if they are allowed in inline schemas) MUST be resolvable within the sequence, otherwise it is considered an error
• the first document in the sequence is the entry document
• all documents MUST specify a self IRI (could potentially relax this)
• all imports MUST use the target's self IRI (could potentially relax this)

Processing this and building the namespace and component index would be very fast. Step 3 becomes trivial because the only valid location is within the document sequence. Step 4 is trivial because you are already parsing the document stream and therefore something already retrieved it.

Documents that do not have a self IRI but can be retrieved from an import URL can be packaged by placing their retrieval URL into the self field. If we want to allow imports to mix self IRIs and retrieval URLs, we could add meta-data to the sequence that would include each document's retrieval URL.

---

Now let's take JSON Schema into account. We can include JSON Schema documents in the stream (either by putting then at the end or ensuring there's a way to distinguish document types while parsing the stream) with the following constraints:

• All references MUST be resolvable within the stream, otherwise it is considered an error
• All JSON Schema documents MUST have an $id in the document root
• All references MUST use the $id URIs (for the same reason as mandating import via self IRIs above)
• All non-standard metaschemas MUST be included in the sequence (for maximum interoperability, standard metaschemas MAY be included)

This could be futher condensed by requiring that all JSON Schema documents be packaged as a single bundled document.

This does not eliminate the complexity of handling JSON Schema references, but (combined with the fencing techniques suggested in #73) does reduce it to the existing problem handled by all compliant JSON Schema tools.

---

I think this sort of document sequence format would accomplish the key goals while adding a very small amount of format-specific work. Also, nothing prevents tools from doing whatever other processing they want with documents before putting them in a sequence. If it makes sense to inline reference targets, they can do that (and even have a sequence of one document for consistency). But inlining isn't always possible and is a trade-off rather than always being an optimization. it can consume a lot of space, and loses information in the form of persistent component identifiers. And with standards groups publishing component shapes for use across many APIs, use cases for preserving identifiers are becoming more significant.

[handrews]

An alternative to basing this on json-seq would be to define our own multipart media type, which could give us more flexibility because of per-part headers including Content-Disposition which can be extended with additional dispositions and parameters.

[handrews]

Note that I proposed an implementation of this for OAS 3.2 alongside the self field from the Imports Proposal (#72), but we decided to accept self and revisit the bundling aspect as people were not sold on the need for it:

• Proposal: Bundled streams w/ self-identification OpenAPI-Specification#3875

[handrews]

See also (closed in favor of continuing the discussion here):

• OpenAPI bundling proposal OpenAPI-Specification#2685