KEP-19: Versioning for operator packages

[nfnt]

What this PR does / why we need it:
By adding a KUDO API version to package indexes, we will avoid that changes to the operator schema will affect existing KUDO versions and package definitions.

Closes #163

[alenkacz]

I think it's a good start, it just needs a bit more content because I have a lots of questions I don't know answer to

[zen-dog]

Before diving into the implementation details, I wanted to talk about the overall "versioning context". Having KUDO API version indexed, only solve a smaller part of the versioning chaos. Below a scenario that I would like to discuss:

tl;dr:

1. KUDO API version is part of the repo path e.g. storage.googleapis.com/kudo-repository/v1beta1/
2. Double semantic versioning: e.g. kafka_2.3.0_1.0.0 where 2.3.0 is Kafka version and 1.0.0 operator
3. Operator version alone is not unique

# 
# The beginning. KUDO API version is v1beta1, only one kafka_2.3.0_1.0.0 operator with kafka 2.3.0 inside:
# ----
/kudo-repository/v1beta1/kafka_2.3.0_1.0.0
                /v1beta1/kafka_2.3.0_1.0.1 // operator update: operator 1.0.0 -> 1.0.1
                /v1beta1/kafka_2.3.0_1.0.2 // yaop update: operator 1.0.1 -> 1.0.2
                /v1beta1/kafka_2.3.1_1.0.3 // kakfa bugfix: kafka 2.3.0 -> 2.3.1, operator 1.0.2 -> 1.0.3
                /v1beta1/kafka_2.4.0_1.1.0 // new minor kakfa version: 2.3.1 -> 2.4.0, operator 1.0.3 -> 1.1.0


# New KUDO API version v1. We need to support at least two last kafka version so they are released with:
# ----
/kudo-repository/v1/kafka_2.3.1_1.0.4 // bumped operator version: 1.0.3 -> 1.0.4 
                /v2/kafka_2.4.0_1.1.1 // bumped operator version: 1.1.0 -> 1.1.1


# Both, v1beta1 and v1 repos are suppored. Kafka releases bugfix versions for 2.3.x and 2.4.x. 
# We need to update both repos:
# ----
/kudo-repository/v1beta1/kafka_2.3.2_1.0.4 // kafka 2.3.1 -> 2.3.2, operator 1.0.3 -> 1.0.4
                /v1beta1/kafka_2.4.1_1.1.1 // kafka 2.3.1 -> 2.4.0, operator 1.1.0 -> 1.1.1

/kudo-repository/v1/kafka_2.3.2_1.0.5 // kafka 2.3.1 -> 2.3.2, operator 1.0.4 -> 1.0.5
                /v2/kafka_2.4.1_1.1.2 // kafka 2.3.1 -> 2.4.0, operator 1.1.1 -> 1.1.2


# Note: With the above schema we now have two IDENTICAL operator versions 1.0.4, but with DIFFERRENT Kafka versions:
/kudo-repository/v1beta1/kafka_2.3.2_1.0.4
/kudo-repository/v1/     kafka_2.3.1_1.0.4

/cc @mpereira @nfnt @gerred @kensipe

[zen-dog]

Here is the same scenario from @mpereira (this time for Cassandra) in form of the table which resets operator version Semver every time a new major Cassandra version is released.

[gerred]

We will need an alternative registry for KUDO once this gets implemented as to not break 0.8.0, yeah?

[nfnt]

This should not break 0.8.0, as it's only adding field to the index, which is backwards-compatible. However, once there are newer API versions, it becomes a bit more complicated: While operators versions that are compatible with 0.8.0 will still get list in the index, the user needs to explicitly select these versions (using the --version flag, or by directly installing from a tarball URL) to avoid using a newer version that's requiring a newer API.

[nfnt]

An alternative would be the have separate URLs for each API version, e.g. https://kudo-repository.storage.googleapis.com/v1 once KUDO is at v1. This way there wouldn't be any of the issues with 0.8.0 described above once KUDO's API is updated.

[gerred]

Ah, yes, I misread the kafka section where it was updated and thought the list changed to a map. That makes sense.

On the surface, I do like the idea of having separate registries for each API version - although this presents a challenge when a user wants to upgrade from one Operator API version to another. I would see that as a major version change inside the same repository, and the KUDO tool could be aware of what API versions it works with and ones it doesn't (which could be useful for CLI tooling).

[nfnt]

Been thinking about it for a while, actually I like having separate registries for each API versions a bit more. It would allow use to have backwards-incompatible changes to the repo index in the future. I'll update the KEP accordingly.

[kensipe]

We had a lot of discussion at eng week that are not capture here... for instance: it was proposed that we change the version to lead with underlying tech and have incrementing version for each tech version. I'm not sure I see that completely captured here. What is definitely missing is what to do when there is multiple underlying tech versions or none.

[kensipe]

I agree with this statement in general.. however :) . it isn't the whole story. A version of KUDO should have awareness of which apiVersions it supports. Then I expect that it could at some point deprecate an apiVersion. It would be better to state the problem this way IMO. kudo 2.1.1 supports apiVersions 2.0, 2.1, 2.2 AND support 1.9 in deprecated warning mode AND does NOT support <= 1.8. In this way kudo should have 3 apiVersions sets it manages...

1. active support
2. deprecate support
3. no longer supported

[nfnt]

Good point! I'll clarify this in the KEP.

[nfnt]

After thinking about it for 18 days 😆, IMO this shouldn't be part of this KEP. Let's focus this KEP on operator packages. How and which API versions are supported by KUDO and what a deprecation cycle for these APIs would look like should be part of a separate KEP. @kensipe wdyt?

[nfnt]

I've updated the KEP to present the "composite version" approach, as per our discussions. Many of the existing comments are outdated and I resolved them. If anyone feels that this is still not covered here, please unresolve these conversations.

[ANeumann82]

There are still some open questions:

• Is appVersion a required field now?
  • Do we provide a default of "0.0.0" or "" something?
  • Can we keep it optional, and if so how does a package file would look like?
• What do we do when a user does not provide an --appVersion but the repository shows us having multiple appVersions? Do we just require a specified --appVersion in this case? Simple lexicographical ordering does not work.

[nfnt]

@mpereira, this makes sense, yes. I'm fine with having a parameter --version that can use a single identifier, though this wouldn't be much different from having an additional --app-version parameter, IMO. We already surface all of these versions to users as part of the Operator, I'm not sure what the difference between this and surfacing a single identifier would be.

[nfnt]

@ANeumann82

Do we provide a default of "0.0.0" or "" something?

If appVersion is required, do we need a default? Though, a default would make existing operators without appVersion work with the new schema.

What do we do when a user does not provide an --appVersion but the repository shows us having multiple appVersions? Do we just require a specified --appVersion in this case? Simple lexicographical ordering does not work.

My suggestion is to first try to sort operator versions as semver, then select the latest operator version. If that doesn't work, the installation will be aborted due to ambiguous operator versions and the user will be prompted to also supply an operator version.

[ANeumann82]

Changing appVersion to a required field would make existing packages potentially invalid unless we specify a default.

Parsing the appVersion as semVer and bailing if it doesn't work seems like a reasonable approach, although I would prefer some other solution. But everything I can think of would have a lot more complexity.

One thing to keep in mind when implementing this: We currently generate the operatorVersion CRs from operatorName + operatorVersion. As we're now changing the uniqueness of the operatorVersion, we need to adjust the generated name of the operatorVersion CRs to include the appVersion. Which will be a breaking change. Additionally, the generated CR name has a limited charset and length, which we need to keep in mind.

[nfnt]

@ANeumann82, I agree, making appVersion required is a breaking change. This concern was raised in the team and it was agreed to proceed, given that the current landscape of KUDO operators is still small and we're in v1beta1.

Good point regarding the version in OperatorVersion names. Let me update the KEP to include this.

[gerred]

Since this KEP is so sensitive, we're waiting on a supermajority of approvals to merge.

[zen-dog]

Godspeed! 🚢

[zmalik]

LGTM!

[kensipe]

nice work! 🚢

[mpereira]

Thanks @nfnt.

[nfnt]

Thanks everyone for the healthy discussion and suggestions and helping to get this KEP into shape. I'm merging this now and will have an implementation soon.