Update schema to match documentation and require schema_version

[joshbuker]

Per https://github.com/ossf/osv-schema/blob/main/docs/schema.md?plain=1#L185 and #122

schema_version is required, which should be reflected in the schema for versions 1.4.0 and above.

[oliverchang]

Hey, thanks for this change!

However, we can't do this, because the intention of the schema validator is to provide a single schema that can validate all OSV schema versions up to the given version (Because of our backwards compatibiltiy guarantees -- see #116 (comment)).

That's not to say we shouldn't validate this, and I think we should address this as part of a tool instead, which can be a bit smarter about knowing when schema_version needs to be applied: #90

[joshbuker]

Can this be distinguished with a "rolling" schema, and a versioned schema?

In v1.4.0 and above, schema_version is required. That should be reflected in the schema... If it's not, then it's not truly required.

[oliverchang]

the schema_version has always been required > 1.0.0, we just haven't had the chance to implement the proposed tool to check this.

We just made the text a bit clearer in the 1.4.0 version.

Having two schemas to maintain sounds like a fair bit of maintenance burden, when I think the correct approach to go with is having a smart tool that can handle all of this and more.

[joshbuker]

Also, it shouldn't cause any issues for those producing osv < 1.4.0, as those versions of the schema don't require it. Only those complying to future versions would need to include it.

[joshbuker]

Having two schemas to maintain sounds like a fair bit of maintenance burden, when I think the correct approach to go with is having a smart tool that can handle all of this and more.

Using proper semver and the built-in required properties field is low maintenance and solves the issue with existing tooling like python-jsonschema/jsonschema.

Those producing osv versions before this would be unaffected, as they would point to a version of the schema without said requirement...Unless they're pointing at main, which implies they're using the latest anyway.

Those who don't specify schema_version being implied to be v1.0.0 would mean they point at https://raw.githubusercontent.com/ossf/osv-schema/v1.0.0/validation/schema.json, which...404's. There should be a tag added for v1.0.0 so that resolves properly. Created #139 to track this.

But those pointing at https://raw.githubusercontent.com/ossf/osv-schema/v1.0.0/validation/schema.json would not require schema_version as it wasn't introduced as being required yet. This is a non-blocker for maintaining backwards compatibility for anyone using the appropriate schema and semver.

[oliverchang]

We've tried to avoid requiring tooling to have to understand multiple copies of the OSV schema, because that inevitably lead to a fair bit of complexity. A client consuming OSV shouldn't have to maintain several copies of the schema to validate/understand. They should just be able to rely on the latest and expect that to validate older versions as well.

Our backwards compatibility guarantees are (from the schema documentation for schema_version):

Clients can assume that new minor and patch versions of the schema only add new fields, without changing the meaning of old fields, so that a client that knows how to read version 1.2.0 can process data identifying as schema version 1.3.0 by ignoring any unexpected fields.

Thanks for catching #139 !

[joshbuker]

Would this not work by parsing >=1.0.0 && < 2.0.0?

The client parsing would be able to either pull dynamically via the canonical URL combined with the SEMVER provided in schema_version (or 1.0.0 if null), or simply use the latest schema they have compatibility with.

This is what we're doing with GSD for example, as @kurtseifried can elaborate.