Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: x-oas-draft-alternativeSchemas #1532

Closed
darrelmiller opened this issue Apr 9, 2018 · 119 comments
Closed

Proposal: x-oas-draft-alternativeSchemas #1532

darrelmiller opened this issue Apr 9, 2018 · 119 comments
Labels
draft:pilot Go forth and try and implement, let us know what needs fixing draft-feature

Comments

@darrelmiller
Copy link
Member

A long standing request has been to add support for alternate schemas. See #764 #1443

There are many concerns about the effect this will have on the tooling ecosystem, however, there is also a strong desire from the community to support both more recent versions of JSON Schema and other formats such as XSD Schema, Protobuf Schema.

Instead of simply adding this feature to the specification and hoping that tooling vendors will implement it. I propose we use the draft feature (PR #1531 ) process to validate the value and feasibility of this feature.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              x-oas-draft-alternate-schema:
                type: json-schema
                externalValue: ./rootschema.json
            application/xml:
              x-oas-draft-alternate-schema:
                type: xml-schema
                externalValue: ./rootschema.xsd
  • The property is called alternate-schema because it can be used in instead of, or in addition to the OAS Schema object. If both are present then both schemas must be respected.
  • The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
  • It is recommended that tools provide users with warnings when they encounter an alternate schema type that they do not support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
  • The externalValue property is required and must be a relative or absolute URL.
  • The alternate-schema property can be used anywhere the schema property can be used.
@philsturgeon
Copy link
Contributor

philsturgeon commented Apr 9, 2018

Where did we land on alternate schemas various versions? Saying "json-schema" here might be fine, as JSON Schema files contain "$schema": "http://json-schema.org/draft-XX/schema#", and Protobuf has syntax = "proto3";, but other formats might not?

It would be up to tool vendors to sniff out the right version, which might cause trouble.

Adding the version in the file does add a lot more work for everyone. When users upgrade their JSON Schema files, they would have to update all the OpenAPI references too, which doesn't sound ideal. Tool vendors would also need to update their registries of "supported alternate schemas" just to support a newer JSON Schema draft, which the underlying tools might already support just fine.

I'm conflicted. Probably leave them out and assume that formats declare their versions?

@handrews
Copy link
Contributor

handrews commented Apr 9, 2018

Please, please, PLEASE do not make people specify the JSON Schema draft version. Right now that's fairly straightforward, but as we move towards full vocabulary support including all of the things that will facilitate code generation, this will become far too complex. The whole point is to allow schema authors and vendors more flexibility in using and detecting support for vocabularies. Please don't duplicate the entire effort inside of OpenAPI.

@darrelmiller
Copy link
Member Author

@philsturgeon Can we identify existing schema languages that are not clearly self-description as to version? I'm going to guess all the XML ones will use a namespace. JCR doesn't seem to have any way to version stuff, but then it is still draft.

Personally I'm a fan of formats that identify their own version. There is something icky about relying on external metadata. So, I wouldn't be opposed to just not offering a way to specify a version.

@philsturgeon
Copy link
Contributor

I am no longer conflicted. 👍🏼

@handrews
Copy link
Contributor

@darrelmiller do people ask about JCR in OpenAPI? It's on draft-09 but still lists only one complete implementation. No need for a big tangent here, just curious as to whether it comes up a lot or is just the first JSON-related example you thought of :-)

@darrelmiller
Copy link
Member Author

@handrews It's the only other schema language in JSON that I know of. And no, nobody has ever asked for support.

@MikeRalphson
Copy link
Member

MikeRalphson commented Apr 10, 2018

There is also schematype for Yaml which has come up before as a possible alternate schema-description language, though I'm not sure about requests for support.

@handrews
Copy link
Contributor

the only other schema language in JSON

Technically "for JSON", as its syntax is not JSON, merely "JSON-like", possessing the conciseness and utility that has made JSON popular. (oh wait, I'm being pedantic again, oops...)

@MikeRalphson
Copy link
Member

As the guidance for the json schema $schema keyword is

The "$schema" keyword SHOULD be used in a root schema. It MUST NOT appear in subschemas.

I don't think we need any additional recommendation that people declare their json schema version. Presumably the $schema keyword is 'in scope' / visible the same way $refs outside the fragment being $ref'd by the OpenAPI document are resolvable.

Unless the fragment being $ref'd from the OpenAPI document counts as a "root schema"?

@handrews
Copy link
Contributor

@MikeRalphson "root schema" is the root of a "schema document" which is a whole resource. If you are referencing a fragment other than "#" then you're referencing a subschema.

Certain concepts, most notably $schema and base URI, work based on the schema document scope, not how you got to a schema object.

@handrews
Copy link
Contributor

I think with the vocabulary stuff in draft-08 we'll be clarifying the schema document / file-scope stuff a lot, including how $schema and the closely related proposed $vocabulary work.

@MikeRalphson
Copy link
Member

Thanks @handrews

@darrelmiller
Copy link
Member Author

darrelmiller commented Apr 23, 2018

UPDATED Proposal based on feedback from TSC meeting of April 16th.

This proposal is to enable to use of schemas other than the native OAS Schema object for describing request and response payloads and complex parameter/header structures.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /cat:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              schema:
                type: object
              x-oas-draft-alternateSchemas:
                - type: json-schema
                  externalValue: ./cat-schema-draft-08.json
                - type: json-schema
                  externalValue: ./cat-schema-draft-05.json
                - type: json-content-rules
                  externalValue: ./cat-JCRschema.json
            application/xml:
              x-oas-draft-alternateSchemas:
                - type: xml-schema
                  externalValue: ./cat-schema.xsd
                - type: relax-ng
                  externalValue: ./cat-schema.ng
                - type: schematron
                  externalValue: ./cat-schema.st
  • The property is called alternate-schemas because they can be used in instead of, or in addition to the OAS Schema object. If both OAS Schema and alternate schemas are present then both the OAS schema and one of the alternate schemas SHOULD be respected.
  • The alternate-schemas property is a list of schemas. They SHOULD be processed in the order included in the list. Tooling is expected to walk the list until it finds an alternate schema that it knows how to process. Multiple alternate-schemas are supported to enable a single specification be used to drive tools with different schema support capababilities.
  • The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
  • It is possible for the same schema type to appear multiple times in the list of supported schemas. This is to provide support for different versions of the same schema type.
  • It is recommended that tools provide users with warnings when they are unable to find an alternate schema that they support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
  • The externalValue property is required and must be a relative or absolute URL.
  • The alternate-schema property can be used anywhere the schema property can be used.
  • The use of an array of alternate schemas provides the ability for x- extensions to be included in the alternate schema information.

@cmheazel
Copy link
Contributor

cmheazel commented Apr 23, 2018

In this example shouldn't the externalValue property reference the schema for the returned content? For example, if I want to return GeoJSON (an IETF standard) then the externalValue could resolve to https://github.com/fge/sample-json-schemas/blob/master/geojson/geojson.json. The client would then validate the returned content against the GeoJSON schema.

@darrelmiller
Copy link
Member Author

@cmheazel Yes you are correct. I see how my example file names are confusing. I'll fix that.

@cmheazel
Copy link
Contributor

@darrelmiller here are some candidates (in addition to GeoJSON)

Geography Markup Language (GML) - MIME type application/gml+xml
Version 3.2.1 = http://schemas.opengis.net/gml/3.2.1/gml.xsd
Version 3.1.1 = http://schemas.opengis.net/gml/3.1.1/base/gml.xsd
Version 3.0.0 = http://schemas.opengis.net/gml/3.0.0/base/gml.xsd

And an in-development JSON encoding of the GML data model:
http://example.com/GML-SF0-JSON.json

@cmheazel
Copy link
Contributor

Another question - XML validation can be a two step process; XML schema validation followed by XML content validation (Schematron). Would it be legal to have two externalValue properties? Validation would be the AND of the two.

@cmheazel
Copy link
Contributor

cmheazel commented May 7, 2018

I have some folks who want to use this feature in a RESTful Web Services standard. When can we lock this feature down and establish a timeline for adoption?

@darrelmiller darrelmiller changed the title Proposal: x-oas-draft-alternate-schema Proposal: x-oas-draft-alternateSchema May 13, 2018
@darrelmiller
Copy link
Member Author

darrelmiller commented May 14, 2018

This a proposal to add a new field called alternativeSchemas to the Media Type Object. We are considering the possibility of also allowing it to be added to the Parameter Object for simple parameter definitions. However, we will wait for implementer feedback on that issue.

Media Type Object

Fixed Fields
Field Name Type Description
alternativeSchemas [alternative Schema Object] List of alternative schemas. These schemas can be used in addition to, or as an alternative to any existing schema property. If both OAS Schema and alternative schemas are present then both the OAS schema and one of the alternative schemas SHOULD be respected. Alternative schemas MUST be processed in order. It is sufficient for tooling to process just the first alternative schema they are capable of processing in order to consider the content as valid. If tooling cannot process any alternative schemas, then they MAY issue a warning, but MUST not report the OpenAPI description as invalid.

Alternative Schema Object

This object makes it possible to reference an external file that contains a schema that does not follow the OAS specification.

Fixed Fields
Field Name Type Description
type string REQUIRED. The value MUST match one of the values identified in the alternative Schema Registry name of the tag.
externalValue url REQUIRED. This is a absolute or relative reference to an external file containing a schema of a known type.

This object MAY be extended with Specification Extensions.

Alternative Schema Registry

The Alternative Schema Registry is located at https://spec.openapis.org/registries/alternative-schema. Inital contents of the registry include:

Name Link Description
jsonSchema
xmlSchema

@aowss
Copy link

aowss commented Apr 3, 2019 via email

@cmheazel
Copy link
Contributor

cmheazel commented Apr 9, 2019

@aowss I work with systems that return XML. The response undergoes syntax validation through an XML Schema document and valid value validation through a set of Schematron rules. Due to the complexity of Schematron, these rules are packaged in multiple Schematron files. So the answer to question #3 is yes.

@Relequestual
Copy link
Contributor

Is this resolved by #1270 ?

@MikeRalphson
Copy link
Member

@Relequestual no, #1270 was for a schema instance for partially validating OAS 3.0.x documents. This issue concerns the tentative process (using x-oas-draft- extensions so it would not have to wait for OAS 3.1 for implementations) of support for non-OAS 3.0 schema languages in requests and responses.

@Relequestual
Copy link
Contributor

Ach, sorry. I should learn to read properly!

@BlueCoder77
Copy link

I'm confused as to why this was not implemented. I really need the ability to reference an XML Schema as the schema for my request and response. Nothing fancy, just an external reference to a single XML Schema document.

Can we please get this implemented? Pretty please?

@philsturgeon
Copy link
Contributor

@BlueCoder77 sorry. It's not really a matter of how nice you ask, it's that the proposal in its current shape was no good (#1943). We mostly decided that 3.1 wasn't the right time for alternative schemas, and maybe v4.0 would be instead. 3.1 is getting proper JSON Schema support (#1977), which is what the majority of folks are doing¡ with OpenAPI anyhow.

XML Schema support is important, but it's enough hard work trying to get this smaller thing done. 😅

Maybe you could check out #1943 and help get the feedback implemented? Might be doable for 3.2 with a bit of effort. Who knows.

@BlueCoder77
Copy link

@philsturgeon - Thanks for the help.

@darrelmiller
Copy link
Member Author

@BlueCoder77 @philsturgeon

Allow me to add a slightly different perspective on this story. Phil isn't wrong, he just has a perspective :-)

The Alternative Schema proposal is in a state that we considered incorporating into the specification. However, when doing a "pilot" proposal we require multiple actual implementations of the feature before we incorporate into the specification. This is a validation process to ensure we don't add big features to the spec that are problematic for tooling.

Phil and the Stoplight crew attempted to implement the current proposal (Yay!) and had a bunch of feedback that they felt the approach was over-complicated and suggested ways it could be simplified. This is how the process is supposed to work.

In theory, multiple other implementers could come along and implement the proposal as is and say THIS IS AWESOME, SHIP IT. And in theory we could allow that feedback to overrule the Stoplight opinion. Unlikely, but theoretically possible.

There wasn't really a decision to say 3.1 is not the right time for alternative schema. We are in a place now, were we either need to re-work the proposal to make Phil et al. happy, or wait for feedback from other implementers who have actually tried to implement it. We missed the 3.1 shipping window for this proposal.

I don't think we need to wait until 4.0 for this. We do need more participation from people who actually build tooling.

@BlueCoder77
Copy link

BlueCoder77 commented Dec 27, 2019

@darrelmiller - Okay, so what do y'all consider an implementer? I really need this feature available for an OpenAPI I am writing for a client. Can I implement this and give me feedback?

@darrelmiller
Copy link
Member Author

@BlueCoder77 An "implementer" is not someone who is creating OpenAPI descriptions. An "implementer" is someone who is writing tools to process an OpenAPI description. Processing could be for validation, editing, client code generation, mock servers, testing, or documentation generation.

If you are writing tools for your client to process OpenAPI descriptions that have included alternative schemas then your feedback would be most appreciated.

@BlueCoder77
Copy link

@darrelmiller - Okay, I understand. No, I am not an implementer. Are y'all aware of any current editor implementers that support importing an XML schema to a component model?

@philsturgeon
Copy link
Contributor

I am not.

@matjung
Copy link

matjung commented Apr 8, 2020

Regarding alternative schemas.
I was looking for a way to model CSV within OpenAPI and did not find out how to do it yet.
Is it possible to make references to another organizations schemata/specifications?
My thinking is to use DFDL (https://github.com/DFDLSchemas/CSV)
With DFDL one can use XML as model for CSV.
Or is there already an Open API method available for handling CSV in the response.

@Fak3
Copy link

Fak3 commented Apr 8, 2020

I would also love to see a way to link openapi schema with a csvw or frictionless data csv schemas

@dret
Copy link
Contributor

dret commented Dec 21, 2020

can somebody please summarize where this proposal stands in terms of OpenAPI 3.1? https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.1.0.md#schemaObject looks as if JSON schema is still hard-coded into OpenAPI? specifically, i am looking for a way how https://tools.ietf.org/html/rfc8927 could be used with OpenAPI. looking at this issue, it seems that this issue is the right place to look at, but looking at the current 3.1 RC it seems that this proposal didn't make it into 3.1?

@philsturgeon
Copy link
Contributor

That’s right. The tldr was that instead of getting multiple schemas supported we focused on supporting the one same single type of schema but removing all the problematic discrepancies.

I think the thinking was 3.1: JSON Schema Properly, 4.0: multiple schemas.

@dret
Copy link
Contributor

dret commented Dec 22, 2020 via email

@philsturgeon
Copy link
Contributor

OpenAPI is absolutely it's own thing, its an API Description Format that covers the Service Model: Headers, Paths, Parameters, everything that is API specific. v3.1 just means it now uses an existing standard for describing the "Data Model", which is a huge improvement for 3.1 as it means tooling can converge instead of people copying and pasting tools to add exceptions for nullable and other discrepancies.

I remember you being pretty excited about the discrepancy issue being resolved and wanted it to happen sooner. Well, we got it done for 3.1, and this all takes a lot more time than anyone would think! 😅

I would like to figure out how we can make OpenAPI support XML Schema, Protobuf, and GraphQL Types, as they're things we want at Stoplight anyway. If anyone would be interested in making a working group to go make a plan for this, lmk, and I can set up some calls and shuffle it along. In the past it was just lots of people saying "why isn't this done yet" then @darrelmiller trying to find time to satisfy those demands, which probably isn't how we should do things going forward.

@dret
Copy link
Contributor

dret commented Dec 23, 2020 via email

@darrelmiller
Copy link
Member Author

This has been a long open issue and does not seem to have gained traction with implementations. If someone wants to reboot this effort, let us know, but for the moment we are going to close it.

@no-identd
Copy link

Shouldn't this become more of a roadmap item with an as of yet unset implementation date and less something remaining as a left over, essentially un(der)discoverable, closed issue entry?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
draft:pilot Go forth and try and implement, let us know what needs fixing draft-feature
Projects
None yet
Development

No branches or pull requests