Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: x-oas-draft-alternativeSchemas #1532

Open
darrelmiller opened this issue Apr 9, 2018 · 117 comments
Open

Proposal: x-oas-draft-alternativeSchemas #1532

darrelmiller opened this issue Apr 9, 2018 · 117 comments

Comments

@darrelmiller
Copy link
Member

@darrelmiller darrelmiller commented Apr 9, 2018

A long standing request has been to add support for alternate schemas. See #764 #1443

There are many concerns about the effect this will have on the tooling ecosystem, however, there is also a strong desire from the community to support both more recent versions of JSON Schema and other formats such as XSD Schema, Protobuf Schema.

Instead of simply adding this feature to the specification and hoping that tooling vendors will implement it. I propose we use the draft feature (PR #1531 ) process to validate the value and feasibility of this feature.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              x-oas-draft-alternate-schema:
                type: json-schema
                externalValue: ./rootschema.json
            application/xml:
              x-oas-draft-alternate-schema:
                type: xml-schema
                externalValue: ./rootschema.xsd
  • The property is called alternate-schema because it can be used in instead of, or in addition to the OAS Schema object. If both are present then both schemas must be respected.
  • The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
  • It is recommended that tools provide users with warnings when they encounter an alternate schema type that they do not support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
  • The externalValue property is required and must be a relative or absolute URL.
  • The alternate-schema property can be used anywhere the schema property can be used.
@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Apr 9, 2018

Where did we land on alternate schemas various versions? Saying "json-schema" here might be fine, as JSON Schema files contain "$schema": "http://json-schema.org/draft-XX/schema#", and Protobuf has syntax = "proto3";, but other formats might not?

It would be up to tool vendors to sniff out the right version, which might cause trouble.

Adding the version in the file does add a lot more work for everyone. When users upgrade their JSON Schema files, they would have to update all the OpenAPI references too, which doesn't sound ideal. Tool vendors would also need to update their registries of "supported alternate schemas" just to support a newer JSON Schema draft, which the underlying tools might already support just fine.

I'm conflicted. Probably leave them out and assume that formats declare their versions?

@handrews
Copy link
Contributor

@handrews handrews commented Apr 9, 2018

Please, please, PLEASE do not make people specify the JSON Schema draft version. Right now that's fairly straightforward, but as we move towards full vocabulary support including all of the things that will facilitate code generation, this will become far too complex. The whole point is to allow schema authors and vendors more flexibility in using and detecting support for vocabularies. Please don't duplicate the entire effort inside of OpenAPI.

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Apr 9, 2018

@philsturgeon Can we identify existing schema languages that are not clearly self-description as to version? I'm going to guess all the XML ones will use a namespace. JCR doesn't seem to have any way to version stuff, but then it is still draft.

Personally I'm a fan of formats that identify their own version. There is something icky about relying on external metadata. So, I wouldn't be opposed to just not offering a way to specify a version.

@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Apr 9, 2018

I am no longer conflicted. 👍🏼

@handrews
Copy link
Contributor

@handrews handrews commented Apr 10, 2018

@darrelmiller do people ask about JCR in OpenAPI? It's on draft-09 but still lists only one complete implementation. No need for a big tangent here, just curious as to whether it comes up a lot or is just the first JSON-related example you thought of :-)

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Apr 10, 2018

@handrews It's the only other schema language in JSON that I know of. And no, nobody has ever asked for support.

@MikeRalphson
Copy link
Member

@MikeRalphson MikeRalphson commented Apr 10, 2018

There is also schematype for Yaml which has come up before as a possible alternate schema-description language, though I'm not sure about requests for support.

@handrews
Copy link
Contributor

@handrews handrews commented Apr 10, 2018

the only other schema language in JSON

Technically "for JSON", as its syntax is not JSON, merely "JSON-like", possessing the conciseness and utility that has made JSON popular. (oh wait, I'm being pedantic again, oops...)

@MikeRalphson
Copy link
Member

@MikeRalphson MikeRalphson commented Apr 10, 2018

As the guidance for the json schema $schema keyword is

The "$schema" keyword SHOULD be used in a root schema. It MUST NOT appear in subschemas.

I don't think we need any additional recommendation that people declare their json schema version. Presumably the $schema keyword is 'in scope' / visible the same way $refs outside the fragment being $ref'd by the OpenAPI document are resolvable.

Unless the fragment being $ref'd from the OpenAPI document counts as a "root schema"?

@handrews
Copy link
Contributor

@handrews handrews commented Apr 10, 2018

@MikeRalphson "root schema" is the root of a "schema document" which is a whole resource. If you are referencing a fragment other than "#" then you're referencing a subschema.

Certain concepts, most notably $schema and base URI, work based on the schema document scope, not how you got to a schema object.

@handrews
Copy link
Contributor

@handrews handrews commented Apr 10, 2018

I think with the vocabulary stuff in draft-08 we'll be clarifying the schema document / file-scope stuff a lot, including how $schema and the closely related proposed $vocabulary work.

@MikeRalphson
Copy link
Member

@MikeRalphson MikeRalphson commented Apr 10, 2018

Thanks @handrews

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Apr 23, 2018

UPDATED Proposal based on feedback from TSC meeting of April 16th.

This proposal is to enable to use of schemas other than the native OAS Schema object for describing request and response payloads and complex parameter/header structures.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /cat:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              schema:
                type: object
              x-oas-draft-alternateSchemas:
                - type: json-schema
                  externalValue: ./cat-schema-draft-08.json
                - type: json-schema
                  externalValue: ./cat-schema-draft-05.json
                - type: json-content-rules
                  externalValue: ./cat-JCRschema.json
            application/xml:
              x-oas-draft-alternateSchemas:
                - type: xml-schema
                  externalValue: ./cat-schema.xsd
                - type: relax-ng
                  externalValue: ./cat-schema.ng
                - type: schematron
                  externalValue: ./cat-schema.st
  • The property is called alternate-schemas because they can be used in instead of, or in addition to the OAS Schema object. If both OAS Schema and alternate schemas are present then both the OAS schema and one of the alternate schemas SHOULD be respected.
  • The alternate-schemas property is a list of schemas. They SHOULD be processed in the order included in the list. Tooling is expected to walk the list until it finds an alternate schema that it knows how to process. Multiple alternate-schemas are supported to enable a single specification be used to drive tools with different schema support capababilities.
  • The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
  • It is possible for the same schema type to appear multiple times in the list of supported schemas. This is to provide support for different versions of the same schema type.
  • It is recommended that tools provide users with warnings when they are unable to find an alternate schema that they support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
  • The externalValue property is required and must be a relative or absolute URL.
  • The alternate-schema property can be used anywhere the schema property can be used.
  • The use of an array of alternate schemas provides the ability for x- extensions to be included in the alternate schema information.
@cmheazel
Copy link
Contributor

@cmheazel cmheazel commented Apr 23, 2018

In this example shouldn't the externalValue property reference the schema for the returned content? For example, if I want to return GeoJSON (an IETF standard) then the externalValue could resolve to https://github.com/fge/sample-json-schemas/blob/master/geojson/geojson.json. The client would then validate the returned content against the GeoJSON schema.

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Apr 23, 2018

@cmheazel Yes you are correct. I see how my example file names are confusing. I'll fix that.

@cmheazel
Copy link
Contributor

@cmheazel cmheazel commented Apr 23, 2018

@darrelmiller here are some candidates (in addition to GeoJSON)

Geography Markup Language (GML) - MIME type application/gml+xml
Version 3.2.1 = http://schemas.opengis.net/gml/3.2.1/gml.xsd
Version 3.1.1 = http://schemas.opengis.net/gml/3.1.1/base/gml.xsd
Version 3.0.0 = http://schemas.opengis.net/gml/3.0.0/base/gml.xsd

And an in-development JSON encoding of the GML data model:
http://example.com/GML-SF0-JSON.json

@cmheazel
Copy link
Contributor

@cmheazel cmheazel commented Apr 23, 2018

Another question - XML validation can be a two step process; XML schema validation followed by XML content validation (Schematron). Would it be legal to have two externalValue properties? Validation would be the AND of the two.

@cmheazel
Copy link
Contributor

@cmheazel cmheazel commented May 7, 2018

I have some folks who want to use this feature in a RESTful Web Services standard. When can we lock this feature down and establish a timeline for adoption?

@darrelmiller darrelmiller changed the title Proposal: x-oas-draft-alternate-schema Proposal: x-oas-draft-alternateSchema May 13, 2018
@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented May 14, 2018

This a proposal to add a new field called alternativeSchemas to the Media Type Object. We are considering the possibility of also allowing it to be added to the Parameter Object for simple parameter definitions. However, we will wait for implementer feedback on that issue.

Media Type Object

Fixed Fields
Field Name Type Description
alternativeSchemas [alternative Schema Object] List of alternative schemas. These schemas can be used in addition to, or as an alternative to any existing schema property. If both OAS Schema and alternative schemas are present then both the OAS schema and one of the alternative schemas SHOULD be respected. Alternative schemas MUST be processed in order. It is sufficient for tooling to process just the first alternative schema they are capable of processing in order to consider the content as valid. If tooling cannot process any alternative schemas, then they MAY issue a warning, but MUST not report the OpenAPI description as invalid.

Alternative Schema Object

This object makes it possible to reference an external file that contains a schema that does not follow the OAS specification.

Fixed Fields
Field Name Type Description
type string REQUIRED. The value MUST match one of the values identified in the alternative Schema Registry name of the tag.
externalValue url REQUIRED. This is a absolute or relative reference to an external file containing a schema of a known type.

This object MAY be extended with Specification Extensions.

Alternative Schema Registry

The Alternative Schema Registry is located at https://spec.openapis.org/registries/alternative-schema. Inital contents of the registry include:

Name Link Description
jsonSchema
xmlSchema
@handrews
Copy link
Contributor

@handrews handrews commented Mar 28, 2019

@tedepstein the thing I'm focused on here is that part of this proposal has a solid consensus behind it, while the inline part clearly does not (never mind why or whether it is reasonable, there is clearly no consensus and it will take effort to work through this).

I am in favor of decisively including the part that we have consensus on and continuing to discuss inlining separately. If inlining makes it into 3.1, great!

But please let's move ahead with the part we agree on. It does not make inlining harder to add if we decide to do it in 3.1.

I have a number of concerns about inlining, but I would prefer to not continue to discuss the topic here. I will chime in if we split it off.

@tedepstein
Copy link
Contributor

@tedepstein tedepstein commented Mar 28, 2019

@handrews , I would like to give others a little time to respond.

IMO, this debate has gotten dysfunctional. For my part, I have tried to keep it clear and focused, but there is a lot of smoke here. I am not at all happy to have this issue decided, or deferred, because the discussion degenerated into pressure tactics, memes, and misinformation. For the record, this sucks.

I will feel so much better letting this go, or taking it into a new issue, if your concerns about inlining, or @darrelmiller's, or anyone else's, turn out to be significant issues that we had not considered. So whether we end up discussing it here or elsewhere, if you get a few minutes to summarize those concerns, I'd like to hear them.

@handrews
Copy link
Contributor

@handrews handrews commented Mar 28, 2019

@tedepstein I think I'll also wait on others before going further into the substance of this issue.

Regarding the process, I am sympathetic to your frustration. Part of my suggestion for splitting this out into a separate topic is that, having dealt with more than a few thread of this nature in the JSON Schema project, splitting has been one of the most effective ways to de-escalate arguments and make progress where we were able to do so.

In this particular case, I am very concerned that alternativeSchema will miss the 3.1 release because of a late-added feature request that is not at all essential to alternativeSchema's usefulness. It may be essential for your tooling, but there are many people who are waiting on this feature who do not need inline schemas.

If alternativeSchema gets punted out of 3.1 over the question of inline schemas, I'm going to be more than a little upset over that 😢

So I don't really want to get into the merits. I want to ensure that the core of the proposal moves forward, because it's extremely important to me that alternativeSchema be in 3.1. Obviously as someone driving the JSON Schema project, the single most important thing for me in OpenAPI is to be able to use actual JSON Schemas.

@aowss
Copy link

@aowss aowss commented Mar 28, 2019

I completely agree with @handrews' comment : inlining can be considered as a separate request.
Being able to use other schema languages is essential for some of us.

I am not a big fan of the syntax but that's a separate issue and I seem to be the only one with this opinion so feel free to ignore that.

@adjenks
Copy link

@adjenks adjenks commented Mar 29, 2019

Not a problem with me, fork the discussion. I'm not one to call the shots on what your deadlines are and what has the greatest priority. I'm just here to tell you that in the context of my project code it makes no sense to externalize the schemas, so I would like to be able to inline it eventually. In the meantime I can make workarounds and things. Feel free to prioritize however you deem fit, and fork the discussion. I don't want to stall things, just want to inform you of my needs and I thought this was the place to do that. I really like this project, so thank you for your hard work.

@aowss
Copy link

@aowss aowss commented Apr 3, 2019

@cmheazel
Copy link
Contributor

@cmheazel cmheazel commented Apr 9, 2019

@aowss I work with systems that return XML. The response undergoes syntax validation through an XML Schema document and valid value validation through a set of Schematron rules. Due to the complexity of Schematron, these rules are packaged in multiple Schematron files. So the answer to question #3 is yes.

@Relequestual
Copy link
Contributor

@Relequestual Relequestual commented Jun 13, 2019

Is this resolved by #1270 ?

@MikeRalphson
Copy link
Member

@MikeRalphson MikeRalphson commented Jun 13, 2019

@Relequestual no, #1270 was for a schema instance for partially validating OAS 3.0.x documents. This issue concerns the tentative process (using x-oas-draft- extensions so it would not have to wait for OAS 3.1 for implementations) of support for non-OAS 3.0 schema languages in requests and responses.

@Relequestual
Copy link
Contributor

@Relequestual Relequestual commented Jun 13, 2019

Ach, sorry. I should learn to read properly!

@BlueCoder77
Copy link

@BlueCoder77 BlueCoder77 commented Dec 27, 2019

I'm confused as to why this was not implemented. I really need the ability to reference an XML Schema as the schema for my request and response. Nothing fancy, just an external reference to a single XML Schema document.

Can we please get this implemented? Pretty please?

@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Dec 27, 2019

@BlueCoder77 sorry. It's not really a matter of how nice you ask, it's that the proposal in its current shape was no good (#1943). We mostly decided that 3.1 wasn't the right time for alternative schemas, and maybe v4.0 would be instead. 3.1 is getting proper JSON Schema support (#1977), which is what the majority of folks are doing¡ with OpenAPI anyhow.

XML Schema support is important, but it's enough hard work trying to get this smaller thing done. 😅

Maybe you could check out #1943 and help get the feedback implemented? Might be doable for 3.2 with a bit of effort. Who knows.

@BlueCoder77
Copy link

@BlueCoder77 BlueCoder77 commented Dec 27, 2019

@philsturgeon - Thanks for the help.

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Dec 27, 2019

@BlueCoder77 @philsturgeon

Allow me to add a slightly different perspective on this story. Phil isn't wrong, he just has a perspective :-)

The Alternative Schema proposal is in a state that we considered incorporating into the specification. However, when doing a "pilot" proposal we require multiple actual implementations of the feature before we incorporate into the specification. This is a validation process to ensure we don't add big features to the spec that are problematic for tooling.

Phil and the Stoplight crew attempted to implement the current proposal (Yay!) and had a bunch of feedback that they felt the approach was over-complicated and suggested ways it could be simplified. This is how the process is supposed to work.

In theory, multiple other implementers could come along and implement the proposal as is and say THIS IS AWESOME, SHIP IT. And in theory we could allow that feedback to overrule the Stoplight opinion. Unlikely, but theoretically possible.

There wasn't really a decision to say 3.1 is not the right time for alternative schema. We are in a place now, were we either need to re-work the proposal to make Phil et al. happy, or wait for feedback from other implementers who have actually tried to implement it. We missed the 3.1 shipping window for this proposal.

I don't think we need to wait until 4.0 for this. We do need more participation from people who actually build tooling.

@BlueCoder77
Copy link

@BlueCoder77 BlueCoder77 commented Dec 27, 2019

@darrelmiller - Okay, so what do y'all consider an implementer? I really need this feature available for an OpenAPI I am writing for a client. Can I implement this and give me feedback?

@darrelmiller
Copy link
Member Author

@darrelmiller darrelmiller commented Dec 27, 2019

@BlueCoder77 An "implementer" is not someone who is creating OpenAPI descriptions. An "implementer" is someone who is writing tools to process an OpenAPI description. Processing could be for validation, editing, client code generation, mock servers, testing, or documentation generation.

If you are writing tools for your client to process OpenAPI descriptions that have included alternative schemas then your feedback would be most appreciated.

@BlueCoder77
Copy link

@BlueCoder77 BlueCoder77 commented Dec 27, 2019

@darrelmiller - Okay, I understand. No, I am not an implementer. Are y'all aware of any current editor implementers that support importing an XML schema to a component model?

@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Dec 27, 2019

I am not.

@matjung
Copy link

@matjung matjung commented Apr 8, 2020

Regarding alternative schemas.
I was looking for a way to model CSV within OpenAPI and did not find out how to do it yet.
Is it possible to make references to another organizations schemata/specifications?
My thinking is to use DFDL (https://github.com/DFDLSchemas/CSV)
With DFDL one can use XML as model for CSV.
Or is there already an Open API method available for handling CSV in the response.

@Fak3
Copy link

@Fak3 Fak3 commented Apr 8, 2020

I would also love to see a way to link openapi schema with a csvw or frictionless data csv schemas

@dret
Copy link
Contributor

@dret dret commented Dec 21, 2020

can somebody please summarize where this proposal stands in terms of OpenAPI 3.1? https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.1.0.md#schemaObject looks as if JSON schema is still hard-coded into OpenAPI? specifically, i am looking for a way how https://tools.ietf.org/html/rfc8927 could be used with OpenAPI. looking at this issue, it seems that this issue is the right place to look at, but looking at the current 3.1 RC it seems that this proposal didn't make it into 3.1?

@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Dec 21, 2020

That’s right. The tldr was that instead of getting multiple schemas supported we focused on supporting the one same single type of schema but removing all the problematic discrepancies.

I think the thinking was 3.1: JSON Schema Properly, 4.0: multiple schemas.

@dret
Copy link
Contributor

@dret dret commented Dec 22, 2020

@philsturgeon
Copy link
Contributor

@philsturgeon philsturgeon commented Dec 23, 2020

OpenAPI is absolutely it's own thing, its an API Description Format that covers the Service Model: Headers, Paths, Parameters, everything that is API specific. v3.1 just means it now uses an existing standard for describing the "Data Model", which is a huge improvement for 3.1 as it means tooling can converge instead of people copying and pasting tools to add exceptions for nullable and other discrepancies.

I remember you being pretty excited about the discrepancy issue being resolved and wanted it to happen sooner. Well, we got it done for 3.1, and this all takes a lot more time than anyone would think! 😅

I would like to figure out how we can make OpenAPI support XML Schema, Protobuf, and GraphQL Types, as they're things we want at Stoplight anyway. If anyone would be interested in making a working group to go make a plan for this, lmk, and I can set up some calls and shuffle it along. In the past it was just lots of people saying "why isn't this done yet" then @darrelmiller trying to find time to satisfy those demands, which probably isn't how we should do things going forward.

@dret
Copy link
Contributor

@dret dret commented Dec 23, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.