Schema extension is vague in terms of what compatibility means

[amluto]

The schema extension chapter says "Compatibility is only ensured under these conditions:"

What does "compatibility" mean? Certainly, an old decoder can parse a message generated by a newer compatible schema. But what happens if a new decoder parses a message generated by an older schema? What are the semantics?

(It's worth noting that the C++ reference implementation is unlikely to work well in this case: aeron-io/simple-binary-encoding#300)

[donmendelson]

A receiver should compare the schema version in the received message header to the version that the decoder was built with.

If the received version is greater than or equal to the decoder's version, then all fields known to the decoder may be parsed.

If the received version is less than the decoder's version (that is, the producer encoder is older than the consumer decoder), then only the fields of the older version may be parsed. This information is available through metadata as "sinceVersion" attribute of a field. If sinceVersion is greater than received schema version, then the field is not available.

How the decoder signals a calling application that a field is unavailable is an implementation detail.

[donmendelson]

We can state that changes to metadata that do not change wire format are compatible, e.g. change of description or correction to spelling of a name.

[RFrenkel]

Hi Don,
Would you recommend schema version change when need to correct semantic type from String to Int for a field using Int32 binary type (SecurityID in this case)? We didn't find any rules describing rule regarding this type of correction. We don;t see how mismatch of Semantic type in this case may create an issue with decoding.

[donmendelson]

We discussed a closely related use case in the FIX Orchestra working group ("machine readable rules of engagement"): the FIX datatype (application layer) of SecurityID may be dependent on SecurityIDSource, string in one case or numeric in another. We are making a provision for that in the Orchestra schema.

In my view, change of application layer datatype is a significant change, even if the wire format doesn't change, and therefore, should result in a version bump. However, we should get others' views on this. Rosa, please enter this as a new issue to be considered for any future SBE versions.