Reconsider cross-version runtime guarantee

[annettejanewilson]

From the Cross-Version Runtime Guarantee:

Protobuf will not support mixing generated code and runtimes across major version boundaries. We will add “poison pills” where possible to detect these mismatches.

As it stands, it is not clear how to reasonably build and distribute packages of software that contain generated code without causing major headaches for their consumers. Library authors seem to be faced with a number of unpalatable options:

• Upgrade promptly to a newer major version* N+1. Many of their consumers will be unable to upgrade, because consumers must upgrade all the libraries they depend on simultaneously. If any protobuf-using library they depend on doesn't have a new version for protobuf N+1, they cannot upgrade at all.
• Publish packages of their library for both protobuf N and N+1 for a period of time. This is messy and complicated, but gives maximum flexibility to their consumers.
• Do not publish compiled packages, require users to build from scratch. This also forces any other libraries that depend on such a library to adopt the same policy. It requires users to have a more complex build system.
• Stick with version N of the protobuf compiler while upgrading to version N+1 of the runtime. This is not supported by the cross-version runtime guarantee. In practice it has sometimes worked in the past, but it is unclear territory and seems unlikely to work in the future, especially in the face of poison pills.

Ideally there would be some way to generate code that is guaranteed to be compatible with two adjacent protobuf runtime major versions. It is not necessary for every published version of the compiler to be able to do this, just that, when version N+1 of a runtime is released, there is some version of the compiler that produces code compatible with version N and version N+1 of the runtime.

Here's an example with concrete but made-up numbers. Suppose protobuf compiler version 35.7 builds code that runs with protobuf Java runtime 10.35.x, and compiler 36.0 is coming along that will require protobuf Java runtime 11.36.x. It may be unreasonable to mandate that all existing compiler versions 35.x build code compatible with runtime 11.36.x, but it would be immensely helpful if there was some guarantee that there's some version, e.g 35.8 or 36.0, that produces code that is compatible with both protobuf Java runtimes 10.35.x and 11.36.x, either all the time or with a particular invocation.

Without this, every application has to do a big-bang upgrade of every single library at once, and those producing libraries have an uncomfortable decision about when to upgrade to the new protobuf version - too soon and none of their users can use their newly published version, too late and they are holding back their users from upgrading any of their protobuf-using libraries.

At my organisation we have more than one broadly used, and unrelated, internal library using protobuf to generate code. We have applications that depend on both these such libraries and on third party libraries using protobuf. It requires a lot of coordination and synchronised upgrades between these libraries when a new major version of protobuf is released. If there is a divergence in the speed at which these libraries upgrade we end up with users of these libraries stuck.

What language does this apply to?

All of them (except perhaps C++, due to its strict version requirements), but my area of experience is Java and Python.

Describe the problem you are trying to solve.

It's very painful to build and package protobuf-using libraries in a way that doesn't place onerous requirements on their users.

Describe the solution you'd like

A broadened cross-version runtime guarantee and the existence of "stepping stone" protoc compilers that can generate code compatible with two adjacent major versions of the runtime.

Describe alternatives you've considered

See above for the options I'm aware of for users of protobuf building libraries. I'm open to hearing other ways to solve the described problem, or to clarification of the cross-version runtime guarantee if I have misinterpreted it.

* In the above, I use "major version N" and "major version N+1" when talking about the protoc compiler and runtime. This should be understood to mean a major version of some protobuf runtime and the corresponding versions of the compiler.

[zhangskz]

Thanks for the feedback and suggestions here -- we are indeed considering some changes here given feedback about the difficulty making atomic upgrades in the Java ecosystem for gencode + runtime breaks.

This is likely to look something like your proposal of having a release with gencode compatible with version N and N+1, possibly with a patch to 25.x accordingly.

[annettejanewilson]

Thank you, that's very welcome news! I'm just going to include some links for anyone following along.

• More specific elaboration of the plan to create a new patch release of protoc 25 that generates code compatible with Java runtime 4.x.x.
• The Cross-Version Runtime Guarantee has been updated.

[zhangskz]

Closing since I think this issue has been addressed -- please feel free to reopen or file another issue if you have further questions or clarifications!