Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reconfigure a Node's Kubelet in a Live Cluster does not work #67580

Closed
jiuchongxiao opened this issue Aug 20, 2018 · 6 comments
Closed

Reconfigure a Node's Kubelet in a Live Cluster does not work #67580

jiuchongxiao opened this issue Aug 20, 2018 · 6 comments
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@jiuchongxiao
Copy link

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
Reconfigure a Node's Kubelet in a Live Cluster does not work
1.kubectl proxy --port=8001 &
2.NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}

{
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "0.0.0.0",
  "port": 10250,
  "tlsCertFile": "/var/lib/kubelet/pki/kubelet.crt",
  "tlsPrivateKeyFile": "/var/lib/kubelet/pki/kubelet.key",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/kube-ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "AlwaysAllow",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 5,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "dinfo.cn",
  "clusterDNS": [
    "10.43.0.10"
  ],
  "streamingConnectionIdleTimeout": "4h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "cgroupsPerQOS": true,
  "cgroupDriver": "systemd",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 110,
  "podPidsLimit": -1,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 5,
  "kubeAPIBurst": 10,
  "serializeImagePulls": true,
  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "failSwapOn": false,
  "containerLogMaxSize": "10Mi",
  "containerLogMaxFiles": 5,
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication-token-webhook": true,
  "authorization-mode": "Webhook",
  "anonymous-auth": false,
  "volume-plugin-dir": "/var/lib/kubelet/volumeplugins",
  "cluster-dns": "10.43.0.10",
  "v": "2",
  "cni-bin-dir": "/opt/cni/bin",
  "root-dir": "/var/lib/kubelet",
  "fail-swap-on": false,
  "register-with-taints": "node-role.kubernetes.io/controlplane=true:NoSchedule",
  "read-only-port": "0",
  "allow-privileged": true,
  "kubeconfig": "/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
  "resolv-conf": "/etc/resolv.conf",
  "cluster-domain": "dinfo.cn",
  "address": "0.0.0.0",
  "cadvisor-port": "0",
  "cni-conf-dir": "/etc/cni/net.d",
  "cgroups-per-qos": "True",
  "network-plugin": "cni",
  "pod-infra-container-image": "rancher/pause-amd64:3.1",
  "cgroup-driver": "systemd" 
}

ps aux | grep kubelet

[rancher@node158 ~]$ ps aux | grep kubelet
rancher  19408  0.0  0.0 112724   984 pts/0    S+   13:57   0:00 grep --color=auto kubelet
root     23560  4.9  0.9 1827920 153652 ?      Ssl  8月17 220:46 kubelet --pod-infra-container-image=rancher/pause-amd64:3.1 --cloud-provider= --v=2
 --address=0.0.0.0 --cadvisor-port=0 --cni-conf-dir=/etc/cni/net.d --authentication-token-webhook=true 
 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem 
 --fail-swap-on=false --anonymous-auth=false --cluster-domain=dinfo.cn --cluster-dns=10.43.0.10 --read-only-port=0 --cgroups-per-qos=True
 --enforce-node-allocatable= --cni-bin-dir=/opt/cni/bin --allow-privileged=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml 
 --network-plugin=cni --resolv-conf=/etc/resolv.conf --hostname-override=192.168.181.158 --root-dir=/var/lib/kubelet --cgroup-driver=systemd

3.edit the file
change authorization.mode="Webhook"

{
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "0.0.0.0",
  "port": 10250,
  "tlsCertFile": "/var/lib/kubelet/pki/kubelet.crt",
  "tlsPrivateKeyFile": "/var/lib/kubelet/pki/kubelet.key",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/kube-ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 5,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "dinfo.cn",
  "clusterDNS": [
    "10.43.0.10"
  ],
  "streamingConnectionIdleTimeout": "4h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "cgroupsPerQOS": true,
  "cgroupDriver": "systemd",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 110,
  "podPidsLimit": -1,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 5,
  "kubeAPIBurst": 10,
  "serializeImagePulls": true,
  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "failSwapOn": false,
  "containerLogMaxSize": "10Mi",
  "containerLogMaxFiles": 5,
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1"
}

4.kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml

it will produce a configmap copy it name.

5.kubectl edit node 192.168.191.128

paste the config info to it

apiVersion: v1
kind: Node
metadata:
  annotations:
    flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"86:61:01:d9:7c:56"}'
    flannel.alpha.coreos.com/backend-type: vxlan
    flannel.alpha.coreos.com/kube-subnet-manager: "true"
    flannel.alpha.coreos.com/public-ip: 192.168.191.128
    node.alpha.kubernetes.io/ttl: "0"
    rke.cattle.io/external-ip: 192.168.191.128
    rke.cattle.io/internal-ip: 192.168.191.128
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: 2018-08-17T04:18:07Z
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/fluentd-ds-ready: "true"
    beta.kubernetes.io/os: linux
    kubernetes.io/hostname: 192.168.191.128
    node-role.kubernetes.io/worker: "true"
  name: 192.168.191.128
  resourceVersion: "618388"
  selfLink: /api/v1/nodes/192.168.191.128
  uid: 8b435900-a1d4-11e8-8c8a-000c29541fa1
spec:
  configSource:
    configMap:
      kubeletConfigKey: kubelet
      name: my-node-config-2m72htkctb
      namespace: kube-system
  podCIDR: 10.42.3.0/24
status:
  addresses:
  - address: 192.168.191.128
    type: InternalIP
  - address: 192.168.191.128
    type: Hostname
  allocatable:
    cpu: "2"
    ephemeral-storage: "142792246650"
    hugepages-2Mi: "0"
    memory: 16330060Ki
    pods: "110"
  capacity:
    cpu: "2"
    ephemeral-storage: 154939504Ki
    hugepages-2Mi: "0"
    memory: 16432460Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: 2018-08-20T06:10:24Z
    lastTransitionTime: 2018-08-17T04:18:07Z
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: 2018-08-20T06:10:24Z
    lastTransitionTime: 2018-08-17T04:18:07Z
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: 2018-08-20T06:10:24Z
    lastTransitionTime: 2018-08-17T04:18:07Z
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: 2018-08-20T06:10:24Z
    lastTransitionTime: 2018-08-17T04:18:07Z
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: 2018-08-20T06:10:24Z
    lastTransitionTime: 2018-08-17T04:18:47Z
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - dinfo.cn/ubuntu/herokuish@sha256:0af374a1b25424bb1fd490d2102fb17de0b60208fad2b414e745a33b9de8c901
    - dinfo.cn/ubuntu/herokuish:0.4.2
    sizeBytes: 1296308975
  - names:
    - docker.io/rancher/hyperkube@sha256:4ef65d3eda5a56db46da3aeff6f67bf34f2b03c344c1f86aab2847ba3ffd187f
    - docker.io/rancher/hyperkube:v1.11.1-rancher1
    sizeBytes: 939028696
  - names:
    - dinfo.cn/nlpapi/comment-api-service@sha256:e705df1fa84b8a1a27a9091674651efa465c82dc0375c30f8db538778be682f7
    - dinfo.cn/nlpapi/comment-api-service:3.1
    sizeBytes: 827504947
  - names:
    - dinfo.cn/nlpapi/pipeline-algorithm@sha256:166e0a58d19936baa1791bfc3128055cc818e93bf130cc788c63b15765a81a58
    - dinfo.cn/nlpapi/pipeline-algorithm:3.0
    sizeBytes: 811032489
  - names:
    - dinfo.cn/nlpapi/distance-algorithm@sha256:dc60b51099590095410339e4cf4aac017b9f08ce745cca316d3988350e44e28e
    - dinfo.cn/nlpapi/distance-algorithm:3.0
    sizeBytes: 811027181
  - names:
    - dinfo.cn/nlpapi/semantic-algorithm@sha256:91203dcf4d11a192cc2a50dd5cb7386d611f1fb364fe69b33fa601e1a698d681
    - dinfo.cn/nlpapi/semantic-algorithm:3.0
    sizeBytes: 810588185
  - names:
    - docker.io/istio/proxy@sha256:89ea24cbd96e4c6d15dcde0680cccaadc0c9262deec1ae21787af6b603437d4c
    - docker.io/istio/proxy:0.7.1
    sizeBytes: 769154240
  - names:
    - quay.io/coreos/hyperkube@sha256:699fc03fccb1c4662fee9d996cf30106aea55a6f594d16c1504cc7334dadcee4
    - quay.io/coreos/hyperkube:v1.7.6_coreos.0
    sizeBytes: 699456364
  - names:
    - docker.elastic.co/kibana/kibana-oss@sha256:61c80292ce97b05f460efe26409f176c20d8f493b9ec4bdd9da0a8fa78ff8f0a
    - docker.elastic.co/kibana/kibana-oss:6.3.0
    sizeBytes: 542783838
  - names:
    - docker.io/istio/examples-bookinfo-reviews-v1@sha256:920d46b3c526376b28b90d0e895ca7682d36132e6338301fcbcd567ef81bde05
    - docker.io/istio/examples-bookinfo-reviews-v1:1.8.0
    sizeBytes: 525074812
  - names:
    - docker.io/istio/examples-bookinfo-reviews-v2@sha256:d2483dcb235b27309680177726e4e86905d66e47facaf1d57ed590b2bf95c8ad
    - docker.io/istio/examples-bookinfo-reviews-v2:1.8.0
    sizeBytes: 525074812
  - names:
    - docker.io/istio/examples-bookinfo-reviews-v3@sha256:8c0385f0ca799e655d8770b52cb4618ba54e8966a0734ab1aeb6e8b14e171a3b
    - docker.io/istio/examples-bookinfo-reviews-v3:1.8.0
    sizeBytes: 525074812
  - names:
    - dinfo.cn/alpine/codeclimate-sonar-python@sha256:e9bd8e6ae17ea6b3a5454840c88b935847246f16b1b1f51723a0e28ad5c9af27
    - dinfo.cn/alpine/codeclimate-sonar-python:1.0
    sizeBytes: 480146860
  - names:
    - dinfo.cn/alpine/codelimate-sonar-java@sha256:35e0d992f47d9683a10379cac9ec92da38bb42f23d0d77a7dd5bf7ab457e839f
    - dinfo.cn/alpine/codelimate-sonar-java:1.0
    sizeBytes: 442182642
  - names:
    - docker.io/istio/examples-bookinfo-reviews-v2@sha256:300d21576ee414a0616584e5846660a4f82bc0cfc2d9c1c2ce42bca4d7b3af9d
    - docker.io/istio/examples-bookinfo-reviews-v2:1.5.0
    - istio/examples-bookinfo-reviews-v2:1.5.0
    sizeBytes: 431362026
  - names:
    - docker.io/argoproj/argoexec@sha256:c94cf6a7b76ad5cc0a7038f045df981a3c2a1dc790fdc0673fa7e8fb0ca24b17
    - docker.io/argoproj/argoexec:v2.1.1
    sizeBytes: 389061178
  - names:
    - docker.io/istio/examples-bookinfo-mongodb@sha256:07ddf2f96c7287b761747ff5fd64e8364e009bf1e4932fb5a2ce9457f5eee9a2
    - docker.io/istio/examples-bookinfo-mongodb:1.8.0
    sizeBytes: 379080429
  - names:
    - docker.io/istio/proxyv2@sha256:77915a0b8c88cce11f04caf88c9ee30300d5ba1fe13146ad5ece9abf8826204c
    - docker.io/istio/proxyv2:1.0.0
    sizeBytes: 351037946
  - names:
    - gcr.io/istio-release/proxyv2:release-1.0-latest-daily
    sizeBytes: 350373050
  - names:
    - docker.io/istio/proxyv2@sha256:5c89848a4418cd4e493312fa85135e5f797c5b3e5a72be0f0dda45d70fea8b6b
    - docker.io/istio/proxyv2:1.0.0-snapshot.1
    sizeBytes: 349588545
  - names:
    - docker.io/istio/grafana@sha256:9788bf01cb550e8cec648a56d50da653cbd5a310dc6cecd86274da50573b3e1b
    - docker.io/istio/grafana:0.7.1
    sizeBytes: 305776814
  - names:
    - docker.io/grafana/grafana@sha256:9c86e0950726eb2d38dba6a0fa77e8757b76782a9a3cf56b65fcb689fcfd3b9e
    - docker.io/grafana/grafana:5.0.0
    sizeBytes: 300902580
  - names:
    - docker.io/rancher/rancher-agent@sha256:8b456ed80e6e21017d49b9f45c8528e347f0e683982c1cb5b0f622866a9f6fa0
    - docker.io/rancher/rancher-agent@sha256:fe710fccaa9f046f4ea8249355f0bcfde9b55da98f8eed8397dd41a0ab3d83c6
    - docker.io/rancher/rancher-agent:v2.0.6
    sizeBytes: 296191287
  - names:
    - docker.io/rancher/rancher-agent@sha256:b46d16bfe8d703b1fda91a399d406667a9c23462e191a806e739dfb4984b9f14
    - docker.io/rancher/rancher-agent:v2.0.7
    sizeBytes: 293895130
  - names:
    - docker.io/istio/pilot@sha256:db61fb30f321d28c57452bd3fa4ec1592f3c89f5e1d1dc2eb6ef2289c6dd097d
    - docker.io/istio/pilot:1.0.0
    sizeBytes: 289317147
  - names:
    - gcr.io/istio-release/pilot:release-1.0-latest-daily
    sizeBytes: 289157211
  - names:
    - docker.io/istio/pilot@sha256:7ec46d12d7b5bc54802b8e1dedfa99c880bb605ba25f05091a812ff3eb00acf4
    - docker.io/istio/pilot:1.0.0-snapshot.1
    sizeBytes: 288934972
  - names:
    - docker.io/kiali/kiali@sha256:4140573e9d717449c452e781d80410ebf106cb5c28a4829abd81ed0c5bac7e31
    - docker.io/kiali/kiali@sha256:e3e4e0364a90132654ac6cfb063e0fb57db5327e46a9ccda4dca477e6dfad71e
    - docker.io/kiali/kiali:latest
    sizeBytes: 265680533
  - names:
    - dinfo.cn/alpine/nexus@sha256:63535f095d3cd9265062ddcee81397efcb392ba0329bb5ba449171d2a4e1989c
    - dinfo.cn/alpine/nexus:2.0
    sizeBytes: 255809804
  - names:
    - docker.io/istio/examples-bookinfo-details-v1@sha256:73e1de909cd387cf377bb51ddd90167d0f44cf0659746d1d0e50276e8f1c9df3
    - docker.io/istio/examples-bookinfo-details-v1:1.8.0
    sizeBytes: 253849570
  - names:
    - docker.io/rancher/calico-node@sha256:21d581d7356f2dba648f2905502a38fd4ae325fd079d377bcf94028bcfa577a3
    - docker.io/rancher/calico-node:v3.1.1
    sizeBytes: 248203187
  - names:
    - dinfo.cn/jiabao/service-c@sha256:48419f7b784d423aa13134625a10444290c8235da74603d94c2483b13c2b6cd4
    - dinfo.cn/jiabao/service-c:0.0.3
    sizeBytes: 242188880
  - names:
    - quay.io/pires/docker-elasticsearch-kubernetes@sha256:7b4b0b9b63ac658c1055113468ab9658f7530362ecc89e7f29ca0f126fae6660
    - quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
    sizeBytes: 237103317
  - names:
    - quay.io/pires/docker-elasticsearch-kubernetes@sha256:dcd3e9db3d2c6b9a448d135aebcacac30a4cca655d42efaa115aa57405cd22f3
    - quay.io/pires/docker-elasticsearch-kubernetes:6.3.0
    sizeBytes: 236003421
  - names:
    - docker.io/istio/examples-bookinfo-ratings-v1@sha256:f84302e53e62a8a12510c7f8f437a7a34949be3dfb37f4eb9d054a76436fa4d7
    - docker.io/istio/examples-bookinfo-ratings-v1:1.8.0
    sizeBytes: 217946166
  - names:
    - dinfo.cn/jiabao/service-a@sha256:9d88662831b9e54180c2eda302fa9c45e5af2684512373481acdc0dc22276f26
    - dinfo.cn/jiabao/service-a:0.0.3
    sizeBytes: 217005284
  - names:
    - dinfo.cn/jiabao/service-b@sha256:16ba92521ff7a9e0d8f7a05f56ddddef35493596dcf1be5500957721886ea5e0
    - dinfo.cn/jiabao/service-b:0.0.3
    sizeBytes: 217005236
  - names:
    - dinfo.cn/jiabao/portal@sha256:e3710b58e831f33c1f0495f9ee70bd0708789577ff35ff1fc3e77e83ec0e8263
    - dinfo.cn/jiabao/portal:0.0.3
    sizeBytes: 214638357
  - names:
    - docker.io/rancher/nginx-ingress-controller@sha256:70569898ae816e8351af9b02d2ad836e8066b99b51af02f4fbcbdcc37dba689a
    - docker.io/rancher/nginx-ingress-controller:0.10.2-rancher3
    sizeBytes: 203231435
  - names:
    - docker.io/istio/examples-bookinfo-productpage-v1@sha256:b012b9545d7883cb214a16547a4456f160cfe5cf82362a412cad8107fa55c542
    - docker.io/istio/examples-bookinfo-productpage-v1:1.5.0
    - istio/examples-bookinfo-productpage-v1:1.5.0
    sizeBytes: 161141475
  - names:
    - docker.io/argoproj/argoui@sha256:3ec72558018902b814f164fc0c61dc069166c7b8ccafc3e3b2516cbe7393715c
    - docker.io/argoproj/argoui:v2.1.1
    sizeBytes: 157995512
  - names:
    - docker.io/istio/examples-bookinfo-productpage-v1@sha256:ab56bedb0195aafc985cb54ed2d69d0eb083da212bc4d17a017ebd8d72af513a
    - docker.io/istio/examples-bookinfo-productpage-v1:1.8.0
    sizeBytes: 143976735
  - names:
    - docker.io/argoproj/workflow-controller@sha256:ac75f47cb304111d6a021f35f7354a0a405836a3d668ceb9ec7d023e2a460954
    - docker.io/argoproj/workflow-controller:v2.1.1
    sizeBytes: 135792860
  - names:
    - dinfo.cn/jiabao/fluentd-elasticsearch@sha256:379eee3e144c4bfb94c01573a4693d8b5e445d2737e21b64899fa0bdb67b2a3e
    - dinfo.cn/jiabao/fluentd-elasticsearch:v2.0.4
    sizeBytes: 135716379
  - names:
    - docker.io/rancher/rke-tools@sha256:90f9403f8f9ce05dfa1984183d3e4e7c4da2b9ea883027a2d2e6c0643c7a45e4
    - docker.io/rancher/rke-tools:v0.1.13
    sizeBytes: 124983324
  - names:
    - docker.io/rancher/rke-tools@sha256:669b829817ca0fde14dd5cbeb7a4609016312c40241e934b77da79bf67766109
    - docker.io/rancher/rke-tools:v0.1.10
    sizeBytes: 124368144
  - names:
    - docker.io/prom/prometheus@sha256:aac607c45c0340e8734bba09f76e8220b876ad76121e9747eb369fc601283dc9
    - docker.io/prom/prometheus:latest
    sizeBytes: 118822274
  - names:
    - docker.io/prom/prometheus@sha256:0283ae2509e1ccc71830edf382713cc3906aa55ca9418c50911854223829bf0b
    - docker.io/prom/prometheus:v2.3.1
    sizeBytes: 118818909
  - names:
    - docker.io/istio/proxy_init@sha256:062220ba6415536cd5b2f7029f1d27a3f024f45eed0582705a0a62e07029a92a
    - docker.io/istio/proxy_init:1.0.0
    sizeBytes: 118776809
  - names:
    - gcr.io/istio-release/proxy_init:release-1.0-latest-daily
    sizeBytes: 118503562
  nodeInfo:
    architecture: amd64
    bootID: 2585ba31-5781-4d79-bb64-7dcf9e724036
    containerRuntimeVersion: docker://1.13.1
    kernelVersion: 3.10.0-862.6.3.el7.x86_64
    kubeProxyVersion: v1.11.1
    kubeletVersion: v1.11.1
    machineID: 8c859f2bde9e41155bca40d2d85aebb5
    operatingSystem: linux
    osImage: CentOS Linux 7 (Core)
    systemUUID: 564DE8DB-5012-A0F2-6A51-66C4E7CC5E76

What you expected to happen:

ps aux | grep kubelet should see the parameter --authorization-mode=Webhook

[rancher@node158 ~]$ ps aux | grep kubelet
rancher  19408  0.0  0.0 112724   984 pts/0    S+   13:57   0:00 grep --color=auto kubelet
root     23560  4.9  0.9 1827920 153652 ?      Ssl  8月17 220:46 kubelet --pod-infra-container-image=rancher/pause-amd64:3.1 --cloud-provider= --v=2
 --address=0.0.0.0 --cadvisor-port=0 --cni-conf-dir=/etc/cni/net.d --authentication-token-webhook=true 
 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem 
 --fail-swap-on=false --anonymous-auth=false --cluster-domain=dinfo.cn --cluster-dns=10.43.0.10 --read-only-port=0 --cgroups-per-qos=True
 --enforce-node-allocatable= --cni-bin-dir=/opt/cni/bin --allow-privileged=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml 
 --network-plugin=cni --resolv-conf=/etc/resolv.conf --hostname-override=192.168.181.158 --root-dir=/var/lib/kubelet --cgroup-driver=systemd

Environment:

  • Kubernetes version (use kubectl version):
[root@node128 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
[rancher@node93 ~]$ ./rke --version
rke version bff7e0a-dirty
  • OS (e.g. from /etc/os-release):
[root@node128 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a):
[root@node128 ~]# uname -a
Linux node128 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    rancher2.0.7
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Aug 20, 2018
@neolit123
Copy link
Member

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 20, 2018
@ipuustin
Copy link
Contributor

If you change the kubelet configuration with a ConfigMap, the changes will not come to the kubelet command line flags, so you can't see the changes with ps aux. The values just change internally according to the configuration. You can verify that the values have changed by reading the configuration again from the configz endpoint.

The command line flags can be used to override the values set form the ConfigMap. However, if you have set parameters using --config, then those values will be overridden by the ConfigMap values (even though you will still see the same flags on the command line).

@jiuchongxiao
Copy link
Author

@ipuustin thank you

@workhardcc
Copy link

@ipuustin I have the same issue.
"The values just change internally according to the configuration. You can verify that the values have changed by reading the configuration again from the configz endpoint."
After kubectl edit ${node} added configuration configSource: configMap: kubeletConfigKey: kubelet name: my-node-config-kg6kcht9c9 namespace: kube-system
Then check the configuration again from the configz endpoint. there is nothing changed.

@ipuustin
Copy link
Contributor

ipuustin commented Jan 3, 2019

@workhardcc I think the best way to debug this is to check the kubelet logs with journalctl. The kubelet on the node should restart when the configuration change is applied and the kubelet logs should show any errors and also the configuration source that it tries to use.

@workhardcc
Copy link

@workhardcc I think the best way to debug this is to check the kubelet logs with journalctl. The kubelet on the node should restart when the configuration change is applied and the kubelet logs should show any errors and also the configuration source that it tries to use.

I forgot add --dynamic-config-dir flag in kubelet config file and there isn't any error logs.
Thanks for reply~

@ipuustin ipuustin mentioned this issue Feb 5, 2020
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@neolit123 @ipuustin @workhardcc @jiuchongxiao @k8s-ci-robot