Schema evolution in zserio

[PieterSmith-TomTom]

How does zserio support schema evolution?

In Google Protocol Buffers, the field numbers are an important component to supporting schema evolution, but I cannot find anything similar or documentation on schema evolution in the documentation.

[fklebert]

zserio does not have a so-called wire format, which means that it does not add stuff to the binary stream that is not written by the designer/developer. Protobuf does that.
It is one of the reasons why zserio outperforms protobuf in terms of data size since it is more bare-metal than proto.

So schema evolution is at the moment not directly supported by zserio. Basically you can only do some backward compatible stuff that includes:

• Add value to enum (do not change underlying datatype of enum)
• Add const
• add stuff to the end of a structure (which itself may not be contained in other structures)

I do not know if we have anything in the backlog on this, but there is a concept lingering around for some years to accomplish a similar feature like protobuf but basically decouple this from the original bytestream. Basically it is adding wireformat info into a separate blob that can then be read (or not). So basically the lightweightness of zserio is maintained but if somebody wants to add the additional data size then we can also accomplish schema evolution like protobuf or others.

[mikir]

I do not know if we have anything in the backlog on this, but there is a concept lingering around for some years to accomplish a similar feature like protobuf but basically decouple this from the original bytestream. Basically it is adding wireformat info into a separate blob that can then be read (or not). So basically the lightweightness of zserio is maintained but if somebody wants to add the additional data size then we can also accomplish schema evolution like protobuf or others.

There is already a dedicated issue for compatibility data: #43. We even did some quick investigation of this issue in the past to make sure that the suggested approach is feasible.

[PieterSmith-TomTom]

Hi @fklebert ,
Given your explanation, it sounds like schema evolution with zserio is possible, but not intuitive. Before diving into implementation change proposals and the compatibility data (#43) proposal, might it not be a better to just document a workable approach / recommendation to schema evolution based on existing functionality?

[mikir]

Given your explanation, it sounds like schema evolution with zserio is possible, but not intuitive. Before diving into implementation change proposals and the compatibility data (#43) proposal, might it not be a better to just document a workable approach / recommendation to schema evolution based on existing functionality?

OK, we will try to improve documentation for schema evolution. At least some notes what is currently possible and what is not possible would be better than nothing.

[mikir]

Documentation will be done within scope of issue #43.

[mikir]

New Zserio Schema Evolution Guide has been created.