AKS multiple False Positives

[chen-keinan]

IDs

AVD-KCV-0083, AVD-KCV-0088, AVD-KCV-0089, AVD-KCV-0090, AVD-KCV-0091, AVD-KCV-0092

Description

AVD-KCV-0083: verified that my AKS Kubelet does run with the flag --protect-kernel-defaults set to true
AVD-KCV-0088: --tls-cert-file is correctly set (/etc/kubernetes/certs/kubeletserver.crt)
AVD-KCV-0089: this one is a bit different, --tls-key-file is not set, but --tls-private-key-file is (/etc/kubernetes/certs/kubeletserver.key)
AVD-KCV-0090: --rotate-certificates is correctly set to true
AVD-KCV-0091: the mentioned config file (/etc/kubernetes/kubelet.conf) is not existing / used by AKS
AVD-KCV-0092: cipher suites are set (--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256)

Reproduction Steps

1. Install Trivy Operator on AKS cluster
2. Observe `cis` `ClusterComplianceReport`.

Target

Kubernetes

Scanner

Misconfiguration

Target OS

Ubuntu

Debug Output

N/A

Version

trivy-operator:0.19.2

Checklist

• Read the documentation regarding wrong detection
• Ran Trivy with -f json that shows data sources and confirmed that the security advisory in data sources was correct

[Pionerd]

Could this be related? Another error related to different OSes, but apparently it fails to do something with the k8s-cluster SBOM, is that a special one?

{"level":"error","ts":"2024-04-03T15:42:17Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-6cccfb67dd","container":"k8s-cluster","status.reason":"E │
│ rror","status.message":"2024-04-03T15:42:13.871Z\t\u001b[31mFATAL\u001b[0m\tsbom scan error: scan error: scan failed: failed analysis: SBOM decode error: failed to decode: failed to decode components: mul │
│ tiple OS components are not supported\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operato │
│ r/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).rec │
│ oncileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod │
│ /sigs.k8s.io/controller-runtime@v0.17.2/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-run │
│ time@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.1 │
│ 7.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pk │
│ g/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/c │
│ ontroller/controller.go:227"}

[chen-keinan]

@Pionerd its a bug with latest trivy versions here is the issue ref.
should be fixed soon.

[chen-keinan]

@Pionerd I have opened a separate issue #2016 2016 for sbom multi os problem

[Pionerd]

@chen-keinan This one is not yet completely solved, KVD-KCV-0091 still pops up:

│     - checks:                                                                                                                                                                                                │
│       - checkID: ""                                                                                                                                                                                          │
│         severity: ""                                                                                                                                                                                         │
│         success: true                                                                                                                                                                                        │
│       description: Enable kubelet client certificate rotation                                                                                                                                                │
│       id: 4.2.11                                                                                                                                                                                             │
│       name: Ensure that the --rotate-certificates argument is not set to false                                                                                                                               │
│       severity: CRITICAL                                                                                                                                                                                     │
│     - checks:                                                                                                                                                                                                │
│       - category: Kubernetes Security Check                                                                                                                                                                  │
│         checkID: AVD-KCV-0091                                                                                                                                                                                │
│         description: Enable kubelet server certificate rotation.                                                                                                                                             │
│         messages:                                                                                                                                                                                            │
│         - Enable kubelet server certificate rotation.                                                                                                                                                        │
│         remediation: Edit the kubelet service file /etc/kubernetes/kubelet.conf and                                                                                                                          │
│           set --feature-gates=RotateKubeletServerCertificate=true                                                                                                                                            │
│         severity: HIGH                                                                                                                                                                                       │
│         success: false                                                                                                                                                                                       │
│         target: /node-aks-spot-21258674-vmss00000d                                                                                                                                                           │
│         title: Verify that the RotateKubeletServerCertificate argument is set to true

[chen-keinan]

@Pionerd can you please run the following command and let me know which value you get for kubeletconfig.featureGates.RotateKubeletServerCertificate

kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz"

[Pionerd]

The only featureGate I get is "featureGates":{"CSIMigrationAzureFile":true}

[Pionerd]

However, what may be interesting is this one:

{
   "kubeletconfig":{
      "enableServer":true,
[...]
      "rotateCertificates":true, <--
[...]

[chen-keinan]

However, what may be interesting is this one:

{
   "kubeletconfig":{
      "enableServer":true,
[...]
      "rotateCertificates":true, <--
[...]

you should have both:
1st: kubelet Rotate Kubelet Server Certificate Argument Set:kubeletconfig.featureGates.RotateKubeletServerCertificate
2nd: kubelet Rotate Certificates Argument Set: kubeletconfig.rotateCertificates

[Pionerd]

Apparantly AKS is complying with this control, but not in the way CIS scans expect it:

• [Question] Kubelets Serving Certificate Rotation Azure/AKS#4009
• [Feature] Kube-Bench returns: 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true Azure/AKS#3166
• Azure AKS v1.21.9 and v1.22.6 not properly reporting RotateKubeletServerCertificate kube-bench#1147