Ignore deprecated APIs in last-applied-configuration

[mkemmerz]

kube-no-trouble also dectects deprecated APIs in configurations that are not directly applied at the moment. The example below should show this. kube-no-trouble detectes the in Kubernetes 1.22 depcreated API networking.k8s.io/v1beta1 but the Ingress in running on a valid API networking.k8s.io/v1.

Therefore my feature request is to ignore deprecated API entries (and the warning) in kubectl.kubernetes.io/last-applied-configuration if the object is running on a non-deprecated API.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  namespace: xxx
  uid: xxxx
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.k8s.io/v1beta1","kind":"

[davem-git]

I'm running in to this issue too. this took isn't helpful because of it

[mkemmerz]

Unfortunately it seems like this is exactly what kube-no-trouble was designed for: https://github.com/doitintl/kube-no-trouble/blob/master/pkg/collector/cluster.go#L115

Can someone explain, maybe the author himself @stepanstipl, why it scans this configuration?

last-applied-configuration only represents the object state of the creation (see https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/):

This sets the kubectl.kubernetes.io/last-applied-configuration: '{...}' annotation on each object. 
The annotation contains the contents of the object configuration file that was used to create the object.

so if I make changes to the object after its creation kube-no-trouble will not be able to scan the new configuration and only detects outdated APIs.

[stepanstipl]

Hi @mkemmerz, I think this is a bit of a misunderstanding of how K8S works internally regarding API versions. The last-applied-configuration annotation is how kubent is actually able to detect old versions (or one of the ways). You can't rely on the returned apiVersion from the live resource, as this is governed by the client, independently how the resource was created in the first place (that information is actually not stored anywhere, and internally the resource is stored in API version-independent format). I.e. you can get the same resource in all the supported API versions. I've explained a bit more about it in this comment #230 (comment).

TL;DR: If you have old apiVersion in the last-applied-configuration, it is likely the resource was created using this old API version. And as this would fail once you upgrade to the next minor K8s version, as the API won't be supported anymore, it is flagged by kubent.

I hope this makes sense. But let me know if you have any questions or if have misunderstood the issue. 👍

[davem-git]

say that again? the last applied version is what was applied before the latest one. We'll have a cluster upgrade to 1.22. shortly i'll respond if it causes any issues

[mkemmerz]

@stepanstipl thanks a lot for the clarification! As this was the only documenttion I was able to find it was a misunderstanding from my side 😄

@davem-git good luck! I am also interested if you ran into any problems. We are going to upgrade soon too.

As this ticket makes no sense now I will close.

[stepanstipl]

@davem-git Not quite - what I was trying to say is that the API version of how the resource is created is not related to how it's stored internally, and how it's retrieved by the client.

Let me give a little example:

1. You create Ingress named test from the following manifest in your K8S 1.21.x cluster. And let's say you do this as part of your CI/CD run, for the sake of this example:

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: test
   ...

   So far this works fine, as 1.21 still supports networking.k8s.io/v1beta1 version of Ingress.

2. Now you're able to retrieve this Ingress both in networking.k8s.io/v1beta1 and networking.k8s.io/v1 versions, as both are supported:

   $ kubectl get ingress.v1beta1.networking.k8s.io test -o yaml
   Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     ...

   and also

   $ kubectl get ingress.v1.networking.k8s.io test -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     ...

   Notice the two different apiVersions above. This shows that the returned version is controlled by the client (and independent of the internally stored version). The information in which version the resource was created is actually lost, and that's why you need to use some other forms, such as checking the last-applied-configuration annotation`, to try to retrieve this info.

   Also, the version you get when you run just kubectl get ingress test, i.e. without the full version spec, is going to be v1, as it's marked as the preferred version. But it is kubectl that does some clever discovery before making the call, and it actually requests the full networking.k8s.io/v1 in the end - you can see this by using the --v flag:

   $ kubectl get ingress test --v=6
   ...
   I0309 12:26:54.406990   83758 round_trippers.go:454] GET https://35.204.91.100/apis/networking.k8s.io/v1/namespaces/default/ingresses/test 200 OK in 155 milliseconds

3. You upgrade your cluster to 1.22. All the existing resources would be just fine (they would be auto-converted to newer internal version if needed, but in this case they already were stored in the newer version, so no conversion was actually necessary).

   Note: there's always only one internal version in which a given resource is stored, while there are potentially multiple API versions in which resource can be saved/retrieved.

4. And now if you would try to apply the original manifest, it would fail with the error that K8S does not recognize Ingress resource in networking.k8s.io/v1beta1. The same would go for retrieving it, and of course, it does not apply only to manually using kubectl or from your CI/CD - any application or tool still using the old version would not work anymore.

I hope this makes it a bit more clear 😃

@mkemmerz np, I should probably document this in the README or somewhere, as you're right - it's not well documented anywhere AFAIK, and often a point of confusion. 🤯 ☮️

[davem-git]

ok i guess i miss understood what this tool did. i was trying to use this tool to ensure we were safe to upgrade, with these results though this tool cannot be used that way

[stepanstipl]

@davem-git I believe it can - maybe it's a bit of misunderstanding what the issue is with K8S API deprecations.

So because you have a resource with the annotation with the old API version, it is likely that the resource was created/last updated using the old API version. This won't matter if you have manually created it as a one-off action, and never change it again. But if it's part of your CI/CD pipeline, or even just a manifest in your repo that you try to use next time, it will fail.

And this is typically the case with K8S API deprecations - resources that are already created in the cluster will be upgraded just fine (automatically, as part of K8S migration process; leaving aside edge cases when the behaviour can change as a result of such upgrade and cases when the resource is removed completely). But once any tool/app/script tries to use the old API after the upgrade, it will fail.

I hope this makes sense 😅

[SaberStrat]

This is a really great explanation @stepanstipl of how this stuff works, thank you very much!