Drop support for Kubernetes version < 1.21 ?

[erikgb]

With Kubernetes 1.25 right around the corner, I created #221 - as it turns out that some APIs currently used in trivy-operator will be removed with the release of K8s 1.25. Since I am pretty sure we want to support K8s 1.25, I think we are left with two choice:

1. Drop support for Kubernetes version < 1.21
2. Determine the Kubernetes version in the running cluster on startup, and implement conditional logic based on what is supported/not-supported. Suggested by @josedonizetti in Prepare support for K8s 1.25 APIs and removal of PSP #221 (comment).

Also relevant to this discussion is the proposed solution to #228 suggested in #228 (comment).

My personal opinion is that we could drop support for Kubernetes versions that are past EOL, which means that we should drop support for K8s version < 1.21, ref. https://endoflife.date/kubernetes. I fear that the code required to extend support for older versions in a controller with such a big fan-out as trivy-operator, will add significant complexity that will result in annoying bugs and make things a lot harder to maintain and improve!

WDYT? Are you running clusters based on Kubernetes pre version 1.21?

[chen-keinan]

@erikgb I would take under consideration the cloud providers support for k8s versions as well , here are the current supported k8s releases :
EKS : 1.22.10 , 1.21.13 1.20.15 ,1.19.16
AKS : 1.21 , 1.22 1.23. 1.24
GKE: 1.21, 1.22, 1.23

[NissesSenap]

Where did you find the versions for aks?
Aks currently supports 1.22, 1.23, 1.24
https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar

The eks versions that you point to is correct https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html
But it's super strange that they haven't even started to support 1.23 (currently waiting for it) but some time soon they will have to go to 1.23 and 1.24 so i wouldn't consider skipping everything from 1.21 straight away.
1.21 is end of life any way even if aws currently is sleeping at the wheel.

[chen-keinan]

correct ,my mistake ,list has ben updated

[NissesSenap]

Totally agree @erikgb. If the users need to support the old clusters they can use the starboard operator untill they have upgraded there k8s version.

[erikgb]

@chen-keinan So you think it is more important to support 1.20 than supporting the upcoming 1.25? I am pretty sure that users on old kubernetes versions, at least the EOL ones, are lagging behind in general and not particularly interested in the new features in trivy-operator.

Our latest release (0.1.0) should still work on old clusters. What if we plan for dropping support for version < 1.21 in our 0.2.0 release? And keep a 0.1.x branch open for eventual bugfixes that needs to be back-portedand released? That is a common way to do this...

We (my team) are particularly interested in some improvements to trivy-operator that are a lot easier to implement if requiring Kubernetes 1.21 or newer.

[chen-keinan]

Can you elaborate on what is preventing us from supporting both 1.20 and 1.25? and what are your team needs which require 1.21 and above , it will help us better to understand this req.

[erikgb]

Can you elaborate on what is preventing us from supporting both 1.20 and 1.25? and what are your team needs which require 1.21 and above , it will help us better to understand this req.

Sure, I have already tried to explain this in the start of this discussion, but I will try to make it even clearer.

1. To make the operator work on 1.25, we cannot use batch/v1beta1 CronJob, as this API will no longer be served in 1.25 according to https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-25. We have to use batch/v1 CronJob, that was introduced in 1.21, so not available in 1.20. Also PodSecurityPolicy is removed in 1.25, so we cannot use that resource at all, and currently trivy-operator is watching this resource.
2. The simple fix to feat: opt-in TTL on trivy-operator managed jobs #228 (comment), involves using jobs .spec.ttlSecondsAfterFinished property to ensure the scan jobs are deleted by Kubernetes. Since the feature gate enabling the TTL-controller only was enabled by default in 1.21, we risk that scan jobs are kept around "forever" if we rely on the jobs TTL property.

I hope this made it a bit clearer! In general, we will be blocked from taking advantage of new Kubernetes features if lagging behind on upstream versions that are EOL. And Kubernetes seems to use every 4th release to move things forward. So supporting the 4 most recent versions makes sense to me!

[chen-keinan]

OK , I understand the complexity , the argument is valid. supporting batch/v1 CronJob is most important out of items above in order to move to 1.25
FYI: we track PodSecurityPolicy however it not in use , we can delete it anytime.

[erikgb]

I've created #275 to track removal of PSP usage.