deprecatedAPI scanning in K8s cluster not working as expected

[chen-keinan]

Discussed in #4778

^{Originally posted by jkleinlercher July 5, 2023}

Description

I applied a cronjob described in https://github.com/suxess-it/deprecated-k8s-api-testcase/blob/main/cronjob-deprecated.yaml and when scanning the yaml with "trivy conf cronjob-deprecated.yaml" it correctly says

apiVersion 'batch/v1beta1' and kind 'CronJob' has been deprecated on: 'v1.21' and planned for removal on:'v1.25'

However, after applying the manifest to the K8s cluster and run "trivy kubernetes --report all -n dep-apis-test all" I don't get any deprecatedAPI misconfiguration.

Desired Behavior

trivy should show "apiVersion 'batch/v1beta1' and kind 'CronJob' has been deprecated on: 'v1.21' and planned for removal on:'v1.25'" alert while scanning the cluster.

Actual Behavior

trivy doesn't recognize the deprecated API.

Reproduction Steps

1. create or use a K8s cluster with version 1.24
2. scan the yaml in https://github.com/suxess-it/deprecated-k8s-api-testcase/blob/main/cronjob-deprecated.yaml with "trivy conf cronjob-deprecated.yaml" and you will see the deprecated API alert.
3. apply the manifest with "kubectl apply -f cronjob-deprecated.yaml -n <your-ns>"
4. scan cluster with "trivy kubernetes --report all -n <your-ns> all" --> you wont't see the deprecated API alert

Target

Kubernetes

Scanner

Misconfiguration

Output Format

None

Mode

Client/Server

Debug Output

trivy kubernetes --report all -n dep-apis-test all --debug
2023-07-05T16:52:28.880+0200    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-07-05T16:52:30.309+0200    DEBUG   cache dir:  /home/johannes/.cache/trivy
2023-07-05T16:52:30.309+0200    DEBUG   DB update was skipped because the local DB is the latest
2023-07-05T16:52:30.309+0200    DEBUG   DB Schema: 2, UpdatedAt: 2023-07-05 12:11:04.500000989 +0000 UTC, NextUpdate: 2023-07-05 18:11:04.500000289 +0000 UTC

Operating System

ubuntu

Version

trivy --version
Version: 0.43.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-07-05 12:11:04.500000989 +0000 UTC
  NextUpdate: 2023-07-05 18:11:04.500000289 +0000 UTC
  DownloadedAt: 2023-07-05 12:46:24.380787155 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2023-07-03 00:53:46.661061023 +0000 UTC
  NextUpdate: 2023-07-06 00:53:46.661060523 +0000 UTC
  DownloadedAt: 2023-07-03 13:04:33.70268622 +0000 UTC
Policy Bundle:
  Digest: sha256:30ca89c908eac67d337f8b393262de10ccc2b4f486e7b47bf7828167ad6840b5
  DownloadedAt: 2023-07-05 07:02:24.202668804 +0000 UTC

Checklist

• Run trivy image --reset
• Read the troubleshooting

[chen-keinan]

@jkleinlercher I found the issue , I'll add a fix , could you please also please run the following command and confirm job version has not been converted from betav1to v1

some clusters conver resource from betav1 to to v1

kubectl get job <cronjob-deprecated> -o yaml

[jkleinlercher]

yes that is exactly the case:

kubectl get cronjob demo-deprecated-cron-job -n dep-apis-test -o yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"batch/v1beta1","kind":"CronJob","metadata":{"annotations":{},"name":"demo-deprecated-cron-job","namespace":"dep-apis-test"},"spec":{"concurrencyPolicy":"Replace","jobTemplate":{"spec":{"template":{"spec":{"containers":[{"args":["-e","console.log(new Date().toString());"],"image":"node:14-alpine","imagePullPolicy":"Always","name":"demo-cron-job"}],"restartPolicy":"OnFailure"}}}},"schedule":"*/1 * * * *","successfulJobsHistoryLimit":10}}
  creationTimestamp: "2023-07-05T14:43:01Z"
  generation: 1
  name: demo-deprecated-cron-job
  namespace: dep-apis-test
  resourceVersion: "659627"
  uid: 31940221-3fad-433f-8a6d-d7a21166249d

and I am afraid that is the normal behaviour of all K8s distributions. You can test that also on your enviroments. That is also why tools like kubent and pluto don't evaluate the apiVersion of the resources but the annotation "last-applied-configuration", where the original manifest apiVersion is stored.

kubent:

https://github.com/doitintl/kube-no-trouble/blob/df9a0174afa2f13cee0c3f5bcc3bf3421166b57d/pkg/collector/cluster.go#L141C33-L141C33

pluto:

https://github.com/FairwindsOps/pluto/blob/802f6791164ade42d9875954ca1449e62e02f9e0/pkg/discovery-api/discovery_api.go#L140

I don't know if this is the only way to really search for deprecated APIs. However, also the K8s documentation points to this behaviour:

https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning

For example, suppose there are two API versions, v1 and v1beta1, for the same resource. If you originally created an object using the v1beta1 version of its API, you can later read, update, or delete that object using either the v1beta1 or the v1 API version, until the v1beta1 version is deprecated and removed. At that point you can continue accessing and modifying the object using the v1 API.

this behaviour is also described here: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#operational-overview

[chen-keinan]

@jkleinlercher Thank you for this input I'll take a look at it

[chen-keinan]

@jkleinlercher I have included the support for kubectl.kubernetes.io/last-applied-configuration in the PR #4786 and tested it and it works good.
Again thank you for helping us improving trivy capabilities
it will be included with next trivy release

[jkleinlercher]

@chen-keinan that‘s great! Thank you so much for your speedy response! I would love to use this feature then also on the cluster with trivy-operator. However, I do not understand if all features of the trivy cli are also available on trivy-operator. Can you shed some light on this please?

[chen-keinan]

@chen-keinan that‘s great! Thank you so much for your speedy response! I would love to use this feature then also on the cluster with trivy-operator. However, I do not understand if all features of the trivy cli are also available on trivy-operator. Can you shed some light on this please?

Of course! it is well understood , already opened an issue in trivy-operator aquasecurity/trivy-operator#1345 and it will be addressed soon 😄