Extend Avro and OpenAPI schema versions

[Pakisan]

I want to extend supported version here: https://www.asyncapi.com/docs/reference/specification/v3.0.0#multiFormatSchemaFormatTable

It's important for our users, who are working with latest versions of Avro and OpenAPI specifications

• OpenAPI:

  • 3.0.1
  • 3.0.2
  • 3.0.3
  • 3.1.0

• Avro:

  • 1.9.1
  • 1.9.2
  • 1.10.0
  • 1.10.1
  • 1.10.2
  • 1.11.0
  • 1.11.1

[dalelane]

From a pure-spec perspective, I agree that allowing use of other Avro versions would be useful

I'm not sure what knock-on impact this will have on tooling, however - have you thought about that? I think we'd at least need an update in avro-schema-parser but I'm not sure if anything else would need updating

[Pakisan]

From a pure-spec perspective, I agree that allowing use of other Avro versions would be useful

I'm not sure what knock-on impact this will have on tooling, however - have you thought about that? I think we'd at least need an update in avro-schema-parser but I'm not sure if anything else would need updating

I'm ready to update documentation and spec-json-schemas repo also as related tools

Anyway I'll add this versions in first RC release of JAsyncAPI.

It's more about synchronization, than do or not to do)

[dalelane]

That makes sense. In that case, yes - I think this would be a useful update to the Avro support, thanks! 👍

[derberg]

From the spec perspective, we only say what MUST be supported, and we not say what MUST NOT

So avro 1.9 is for example recommended, but nobody prohibits tooling providers from supporting other versions. For example current official avro parser supports 1.9 and 1.8.2.

What change do you suggest @Pakisan 🤔

[Pakisan]

From the spec perspective, we only say what MUST be supported, and we not say what MUST NOT

So avro 1.9 is for example recommended, but nobody prohibits tooling providers from supporting other versions. For example current official avro parser supports 1.9 and 1.8.2.

What change do you suggest @Pakisan 🤔

My point is short - libraries, schemas, tools under our umbrella

• MUST recognize latest versions of Schemas
• MUST render and handle correctly latest Schemas

to shift EOL, of our spec, as much as it possible

[fmvilas]

Please note that the spec says "recommended":

The following table contains a set of values that every implementation is RECOMMENDED to support.

I'm fine with adding more recommendations but it should not definitely change to a "MUST". Otherwise, implementations will become quite too complex to maintain.

[derberg]

latest versions of Schemas

which ones?

atm MUST is set only on default AsyncAPI Schema and also on JSON Schema that AsyncAPI Schema is a superset of

then you have many others, raml data types, openapi, avro, protobuf, xsd and I bet many other formats that are used in EDA. And you want them all as MUST?

also in main issue description you write latest and then mention all Avro versions above 1.9.0

What problems are you trying to solve with such change in the spec?

[Pakisan]

latest versions of Schemas

which ones?

atm MUST is set only on default AsyncAPI Schema and also on JSON Schema that AsyncAPI Schema is a superset of

then you have many others, raml data types, openapi, avro, protobuf, xsd and I bet many other formats that are used in EDA. And you want them all as MUST?

also in main issue description you write latest and then mention all Avro versions above 1.9.0

What problems are you trying to solve with such change in the spec?

I'm not only one who are using our specification with latest versions of OpenAPI and Avro. It's a big problem to use our toolchain out of the box

If I'll try to use 1.11.0 with our studio, for example, I'll receive next error

Error thrown during AsyncAPI document parsing. Name: Error, message: Unknown schema format, stack:

Same story for OpenAPI

I have simple point - our users must have a chance to use any version of supported schema, above defined minimum. Avro 1.9.0, for example

If we have decided to allow use external dependencies, it's in our interests to always be up to date with them and render/handle them properly

Entry point:

Otherwise it's really interesting situation, where we as a vendors, not interested in developer experience and are relating on external tools to support our features

Result:

External tools or my tools will generate specification not compliant with our definition

[derberg]

@Pakisan thanks but I do not question the idea that we should always support latest schemas and the best if we support all of them. I 💯 agree.

My question is, why you think that setting all schemas to MUST in spec will change anything? I do not recall anyone refusing to support some schema formats.

We are not vendor but a community driven project. If someone needs support for some specific schema format, best if they contribute asyncapi/avro-schema-parser#6. And if someone wants to have a new schema parser, for new format, like for example XSD, we welcome them with open hands to contribute and maintain one, like @GreenRover did with https://github.com/asyncapi/protobuf-schema-parser

Maybe enough is just to specify in spec that the version shown in table is just an example, not a version we recommend to support? So that it is clear that in case of other schema formats, other than AsyncAPI and JSON Schema, we recommend to support other schema formats, period, not any specific version of it.

[jonaslagoni]

Can the JSON Schema documents contain formats that is not defined within the spec itself? And to which length should the validation/autocompletion of those include sub-versions of (for example) Avro that extensively use features and patch version changes?

Or should that just reference the newest version, and should all available versions even be "auto-suggested"? 🤔

The schema parsers are easier to handle I think, but where is the boundary for the official JSON Schema documents?

[VisualBean]

The only problem I see with this, is supporting multiple version that have breaking changes between them.
At least as a parser maintainer, it's hard to keep up and support everything. Just supporting both openapi and avro at the same time is complex to begin with.

As a maintainer I really appreciate the recommendation , rather than the "must".

[Pakisan]

My question is, why you think that setting all schemas to MUST in spec will change anything? I do not recall anyone refusing to support some schema formats.

...

Maybe enough is just to specify in spec that the version shown in table is just an example, not a version we recommend to support? So that it is clear that in case of other schema formats, other than AsyncAPI and JSON Schema, we recommend to support other schema formats, period, not any specific version of it.

I want to clarify this moment

This requirement MUST be required for our tools. Side tools can interpret it as RECOMMENDED

fyi: @derberg

The only problem I see with this, is supporting multiple version that have breaking changes between them.
At least as a parser maintainer, it's hard to keep up and support everything. Just supporting both openapi and avro at the same time is complex to begin with.

As a maintainer I really appreciate the recommendation , rather than the "must".

I understand your concerns, and agree with them. But it's all about how we want to evolve our specification

I released new version of JAsyncAPI, few hours ago, with actual Avro schema versions - [1.9.0,1.11.1] and OpenAPI schema [3.0.0, 3.0.3].

From this moment users who are using 1.9.1+ and 3.0.1+ versions of Avro or Open API schemas exactly will receive errors in our studio, and maybe in other tools too

That's what I'm talking about:

We announced that users can use Avro, OpenAPI, Protobuf in MutliFormatSchema, but not decided how to add new version to our tools. It's 100% strange behavior

When I'm using our specification on daily basis, I'm using latest version of Avro and OpenAPI to describe my APIs.

And it's really annoying when I can't do it, because of our ecosystem, where actual version are not supported

I as API and EDA enthusiast, I'm ready to create MRs to propose changes and update our tools.

But those who are just want to use specification for APIs documenting, will pause adoption of new version of our specification because of lack of synchronization between us, as providers, and them, as users

They just want to describe their APIs with proposed construction. If they can't, they will look for alternative ways to achive what they want

fyi: @VisualBean

Can the JSON Schema documents contain formats that is not defined within the spec itself? And to which length should the validation/autocompletion of those include sub-versions of (for example) Avro that extensively use features and patch version changes?

Or should that just reference the newest version, and should all available versions even be "auto-suggested"? 🤔

The schema parsers are easier to handle I think, but where is the boundary for the official JSON Schema documents?

As I remember JsonSchema, out of the box, can handle only nested JsonSchemas. Off course it's possible to change this behavior by introducing custom properties, but it's not our case

We have MultiFormatSchema where we can add new versions of OpenAPI and Avro and bind them with their schemas, like we did for current versions

fyi: @jonaslagoni

[fmvilas]

@Pakisan I see your concern actually relates more to our support in the tools, which I also 100% agree is a pain. That said, this is the wrong place for this discussion. This is something that has to be discussed with maintainers of jasyncapi, cli, studio, glee, etc. And then contribute the necessary tools to support as much as we can. But again, it's a tooling problem and this is a spec. As Lukasz said, changing it to MUST will change exactly nothing because most maintainers will be overwhelmed and will decide not to even try it. So yeah, enforcing is not a great strategy. Instead, you should try to convince and organize people around these tools to understand the value of what you're proposing and contribute the necessary changes. @asyncapi/developer_experience_wg can actually be a good starting point for this initiative.