Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-bench doesn't take account of settings in kubelet --config file (affects EKS) #195

Open
lizrice opened this issue Jan 11, 2019 · 5 comments

Comments

Projects
None yet
4 participants
@lizrice
Copy link
Collaborator

commented Jan 11, 2019

As of 1.10, Kubelet may be configured using a YAML or JSON config file. kube-bench does not take these config settings into account.

On EKS, the kubelet process looks something like this:

root      2811     1  1 Jan10 ?        00:24:56 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --allow-privileged=true --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --network-plugin cni --node-ip=192.168.193.173 --pod-infra-container-image=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause-amd64:3.1

The contents of the config file kubelet-config.json:

{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "address": "0.0.0.0",
  "authentication": {
    "anonymous": {
      "enabled": false
    },
    "webhook": {
      "cacheTTL": "2m0s",
      "enabled": true
    },
    "x509": {
      "clientCAFile": "/etc/kubernetes/pki/ca.crt"
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "clusterDomain": "cluster.local",
  "cgroupDriver": "cgroupfs",
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "serverTLSBootstrap": true,
  "clusterDNS": [
    "10.100.0.10"
  ],
  "maxPods": 17
}

@lizrice lizrice changed the title kube-bench doesn kube-bench doesn't take account of settings in kubelet --config file (affects EKS) Jan 11, 2019

@lizrice lizrice added the bug label Jan 11, 2019

@lizrice

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 15, 2019

Strictly speaking, a host would fail if you ran the CIS checks manually exactly as described in the benchmark, as none of the commands specified in it take notice of these config file settings, and that's what kube-bench implements. But I think it's fair to say that the spirit of the benchmark would be that the settings should be as recommended, even if they are set using this config file method.

@christopherhein

This comment has been minimized.

Copy link

commented Feb 28, 2019

Do we have a timeline for when kube-bench will be able to handle using config files?

/cc @lizrice

@djsly

This comment has been minimized.

Copy link

commented Mar 20, 2019

Same problem here, the kubelet tests are all failing even though we are setting 80% of the kubelet config a yaml config.

We have the KUBELET_ARGS defined here

KUBELET_ARGS=" --hostname-override=kn-infra-0 --config=/etc/kubernetes/configs/kubelet-config.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256  --cloud-provider=azure --cloud-config=/etc/kubernetes/azure.json  --network-plugin=kubenet --azure-container-registry-config=/etc/kubernetes/azure.json --non-masquerade-cidr=10.244.0.0/16 --kubeconfig=/var/lib/kubelet/kubeconfig --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap.kubeconfig --pod-infra-container-image=<private-repo>:443/sspi-docker/kubernetes/pause:3.1"

and the /etc/kubernetes/configs/kubelet-config.yaml

# Further documentation on values for this file are here: https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
staticPodPath: "/etc/kubernetes/manifests"
authentication:
  x509:
    clientCAFile: /srv/kubernetes/ca.crt
  webhook:
    enabled: true
  anonymous:
    enabled: false
authorization:
  mode: "Webhook"
clusterDomain: "cluster.local"
clusterDNS:
  - "192.168.0.10"
hairpinMode: "promiscuous-bridge"
nodeStatusUpdateFrequency: "1m"
evictionHard:
  nodefs.inodesFree: "20%"
evictionMinimumReclaim:
  nodefs.inodesFree: "30%"
rotateCertificates: true
featureGates:
  RotateKubeletClientCertificate: true
  RotateKubeletServerCertificate: true
readOnlyPort: 0
protectKernelDefaults: false
eventRecordQPS: 0
failSwapOn: false
kubeReserved:
  cpu: 100m
  memory: 500Mi
systemReserved:
  cpu: 100m
  memory: 1536Mi
tlsCertFile: /srv/kubernetes/kubeletserver.cert
tlsPrivateKeyFile: /srv/kubernetes/kubeletserver.key
@lizrice

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 20, 2019

This is being worked on under #239

@lizrice

This comment has been minimized.

Copy link
Collaborator Author

commented May 13, 2019

So now we do have this support but it requires running a separate config file. It would be better to be able to combine the two, or use auto-detection to decide which to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.