[operator] support multiple version streams

[jmazzitelli]

Right now, we have a single stream of versions. In OLM, the metadata has a single channel where this stream of versions is found. The OLM channel is called "stable". See the community metadata here. (note: ignore the alpha channel - that is just for future use but it tracks the same version as "stable" and we might remove "alpha" soon since it really isn't used anyway).

The same channel name "stable" is used in the kiali-ossm package (this is the Red Hat certified operator) - as you see here in the golden copy of the metadata.

However, we can't have just a single stream/channel because the upstream Istio versions that are supported by Kiali differ. And the way OLM works for automatic updates is once a new operator is released in a channel, the previous operator is automatically updated. But this will break certain users. For example, suppose we have a release compatibility chart consisting of the following fictitious versions (this isn't reality - these version numbers are just used for illustration):

Kiali 1.0 through Kiali 1.9 supports Istio 1.0 and 1.1
Kiali 1.10 through Kiali 1.19 supports Istio 1.1 and 1.2

Suppose a user has installed Istio 1.0 and Kiali 1.9. Kiali then releases 1.10. The user SHOULD NOT install this latest Kiali release because Kiali 1.10 does not support Istio 1.0.

But the way OLM and automatic updates work is, as soon as 1.10 is released in the "stable" channel, the user will get upgraded to 1.10 automatically, thus breaking the user due to the incompatibility.

What would work is we release Kiali to channels based on the upstream Istio releases. So, for example (following the same fictitious compatibility chart above), suppose we had two channels. We no longer have "stable" but instead we have two channels named "istio10" and "istio11". In the "istio10" channel, we release Kiali up to and including Kiali 1.9. But then the channel stops there. In channel "istio11" we start the stream of versions at Kiali 1.10 and the stream can continue up to Kiali 1.19. Thus when Kiali 1.19 is released, the user that subscribed to channel "kiali11" will be able to automatically update his Kiali from 1.10 all the way through Kiali 1.19. But then if Kiali 1.20 is released (and presumably that will not support Istio 1.1), the "istio11" channel will not include Kiali 1.20 - it stops at Kiali 1.19. Thus the user that is using Istio 1.1 will never auto-upgrade to Kiali 1.20 and thus avoid installing a bad Kiali version.

The problem here is what to name the channels and how to support overlapping support. In the example above, all versions from Kiali 1.0 through Kiali 1.19 support Istio 1.1. So technicaly, a user that is using Istio 1.1 can install any version of Kiali from 1.0 to 1.19 and still work. But if the channels are defined as explained, if the user was an original user (originally subscribed to the "istio10" channel), that user will stop at Kiali 1.9. Somehow the user would have to switch subscriptions to the new "istio11" channel to get the latest versions.

So how do we want to define the channels? What are the names? What versions to the channel streams include? How do we support overlapping version support?

[jmazzitelli]

One easy solution is just to remove Kiali from the community/upstream OLM - I'm not sure many (if any) are using that anyway.

Still might require some thinking with respect to the Red Hat certified operator.

[jmazzitelli]

I'm still thinking about how it would work if we used different api versions for the CRD, but I don't think it is enough to just have a different CRD versions (while maintaining a single channel). Right now we are at kiali.io/v1alpha1. If we were to say that CRD indicates support for Istio 1.0 and we bump up to kiali.io/v2 to indicate support for Istio 1.1, that doesn't help the versions of Kiali that support both 1.0 and 1.1.

We need a way to stop automatic installs of a later Kiali version if the user's version of Istio is not supported by that later Kiali version.

[jmazzitelli]

Here is an email I sent to some OLM folks - I wanted to document it here because it may have additional examples/details useful for the discussion:

====

The Kiali Operator vX.Y installs operand Kiali vX.Y. Kiali is a UI console for upstream Istio. Istio has several versions 1.X (I'll be avoiding actual, real-world version numbers - it's a general question not specific to any one version).

Upstream Istio is installed separately (outside of OLM - some Istio versions are even installed via helm, not even via operators).

Suppose I have a support matrix such as:

• Kiali 1.0 thru 1.9 supports Istio 1.0 and 1.1
• Kiali 1.10 thru 1.19 supports Istio 1.1 and 1.2

So notice there is an overlap - Istio 1.1 is supported across all Kiali versions (but Istio 1.0 support ended at Kiali 1.9, and Istio 1.2 support only started at Kiali 1.10).

Kiali Operator's OLM metadata defines a single channel today, "stable", and so a single version stream.

Suppose a user has installed Istio 1.0 and Kiali 1.9. Kiali then releases Kiali 1.10. The user SHOULD NOT install this latest Kiali release because Kiali 1.10 does not support Istio 1.0. But if automatic updates is enabled, OLM will see that Kiali 1.10 is released in the "stable" channel, and thus upgrade the user to Kiali 1.10 automatically, thus breaking the user's Kiali due to the incompatibility with the older Istio.

To solve that problem, the answer is to use different channels (I think) - one channel for those Kiali versions that support Istio 1.0 and another channel for those Kiali versions that do NOT support Istio 1.0.

However, suppose the user had Istio 1.1 installed instead. Well, in that case, it's perfectly fine that OLM auto-updates the Kiali 1.9 to Kiali 1.10, because that version of Istio 1.1 is supported. If different channels were used, the user will have been stopped at Kiali 1.9 and no further -- even though the user could get Kiali 1.10, OLM will not upgrade because the channel for Istio 1.0 support stopped at Kiali 1.9.

That said, would it still be appropriate to define the channels based on Istio version support?

For example, should I have 3 channels like this:

• "istio10" that contains CSVs for Kiali 1.0 through 1.9 (notice it stops at 1.9)
• "istio11" that contains CSVs for Kiali 1.0 through 1.19 (notice the overlap - contains all versions found in channel "istio10" and "istio12", too)
• "istio12" that contains CSVs for Kiali 1.10 through 1.19 (notice it starts at 1.10 - does not contain anything 1.9 and earlier)

This seems to make sense. But suppose a user upgrades their Istio from 1.0 to 1.1. Can that user easily switch their Kiali subscription to use a different channel (they would have originally been subscribed to channel "istio10" but now need to change the subscription to channel "istio11")?

Finally, I thought about using different CRD api versions to dictate which Kiali to install - but that doesn't seem the right thing to do here. The version of the CRD really doesn't track the Istio version that is supported (e.g. the CRD schema could change from Kiali 1.15 to Kiali 1.16, even though the Istio versions that Kiali supports is the same). This does add a wrinkle in that we do eventually plan on upgrading CRD versions (the schema to the Kiail CR will potentially change from version to version - wholly separate from this Istio-compatibility issue). How do different CRDs affect this multi-channel scenario?

[jshaughn]

@jmazzitelli As for upstream I feel like a single channel could be sufficient. The "stable" Kiali must just support the supported versions of Istio (typically the current and previous 2 releases). Users using older versions of Istio would need to upgrade Istio, or just not install Kiali via the operator (use the bundled Kiali for example). Does that make any sense or simplify anything?

[jmazzitelli]

Does that make any sense or simplify anything?

It does simplify things alot. We just have a single channel called "stable" and have the latest release in that single stream support Istio X and Y. Users that want older versions can still install them via OLM but must not turn on auto-upgrade in their subscription.

[jmazzitelli]

The other thought is to keep the "stable" channel as-is in the community metadata - and in documentation we just tell the user what version they need if they want to run older, unsupported Istio versions. This would make the community metadata much more easier to maintain and manage - just have a single stream of latest versions that maintain compatibility with the latest version of Istio.

For productized versions of Kiali, those are a different package in OLM and thus can have their own set of channels.

[jmazzitelli]

BTW: when removing "stable" channel and creating new channels to replace it, in order to receive updates, users already subscribed to the "stable" channel will need to manually update their Subscriptions to point to either of the new channels.

[jshaughn]

I personally like the idea of a single channel for upstream, and given the relatively short window of support for upstream istio versions I feel like it could be a viable alternative.

[jmazzitelli]

This is the current PoC that I'm tinkering with: #2219

We keep the same "stable" OLM channel as before and we have a single stream of operator version releases. The operator will be able to be told to install multiple versions of Kiali. That PR will have three different versions: default (which is the master or latest release), "v1.0" (which is the release off of the v1.0 branch), and "v1.12" (which is the release off the v1.12 branch). To tell the operator which one you want, you put "spec.version" in the Kiali CR and set it to one of those three values ("default" or "v1.0" or "v1.12"). If you do not specify a spec.version, you will get the "default" version.

The thinking is that if there is any breaking change such that one version cannot run with a particular past version of Istio, we fork off another supported version. "default" will always install the latest-and-greatest release.

We still release the operator in lock step with a kiali release. So a vX.Y.Z release of Kiali will have a vX.Y.Z release of Kiali operator.