Describe scenario
When multiple pod starts pulling docker images at the same time the image pulls fail with pull QPS exceeded.
Warning Failed 52s kubelet Failed to pull image "a.b.c/image-0.1": pull QPS exceeded
Warning Failed 52s kubelet Error: ErrImagePull
Normal BackOff 45s (x2 over 46s) kubelet Back-off pulling image "a.b.c/image-0.1"
Warning Failed 45s (x2 over 46s) kubelet Error: ImagePullBackOff
The nodes run with the default configuration for registryPullQPS and registryBurst, curl "localhost:8080/api/v1/nodes/aks--38921313-vmss000000/proxy/configz":
"registryPullQPS": 5,
"registryBurst": 10,
-
registryPullQPS is the limit of registry pulls per second.
The value must not be a negative number.
Setting it to 0 means no limit. Default: 5
-
registryBurst is the maximum size of bursty pulls, temporarily allows
pulls to burst to this number, while still not exceeding
registryPullQPS. The value must not be a negative number.
Only used if registryPullQPS is greater than 0. Default: 10
Question
How to change registryPullQPS and registryBurst in the kubelet configuration?
I could not find anything on this in the documentation.
Further information
The kubelet has --registry-qps and --registry-burst command line arguments, but they are already deprecated. Probably the --config /etc/kubernetes/kubelet-config.json option could be used and the configuration values could be defined in the config file.
I checked my node how kubelet is managed there:
root@aks-38921313-vmss000000:/# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubelet
ConditionPathExists=/usr/local/bin/kubelet
Wants=network-online.target
After=network-online.target
[Service]
Restart=always
EnvironmentFile=/etc/default/kubelet
SuccessExitStatus=143
ExecStartPre=/bin/bash /opt/azure/containers/kubelet.sh
ExecStartPre=/bin/mkdir -p /var/lib/kubelet
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/bash -c "if [ $(mount | grep \"/var/lib/kubelet\" | wc -l) -le 0 ] ; then /bin/mount --bind /var/lib/kubelet /var/lib/kubelet ; fi"
ExecStartPre=/bin/mount --make-shared /var/lib/kubelet
ExecStartPre=-/sbin/ebtables -t nat --list
ExecStartPre=-/sbin/iptables -t nat --numeric --list
ExecStart=/usr/local/bin/kubelet \
--enable-server \
--node-labels="${KUBELET_NODE_LABELS}" \
--v=2 \
--volume-plugin-dir=/etc/kubernetes/volumeplugins \
$KUBELET_TLS_BOOTSTRAP_FLAGS \
$KUBELET_CONFIG_FILE_FLAGS \
$KUBELET_CONTAINERD_FLAGS \
$KUBELET_FLAGS
[Install]
WantedBy=multi-user.target
Describe scenario
When multiple pod starts pulling docker images at the same time the image pulls fail with
pull QPS exceeded.The nodes run with the default configuration for registryPullQPS and registryBurst,
curl "localhost:8080/api/v1/nodes/aks--38921313-vmss000000/proxy/configz":registryPullQPS is the limit of registry pulls per second.
The value must not be a negative number.
Setting it to 0 means no limit. Default: 5
registryBurst is the maximum size of bursty pulls, temporarily allows
pulls to burst to this number, while still not exceeding
registryPullQPS. The value must not be a negative number.
Only used if registryPullQPS is greater than 0. Default: 10
Question
How to change registryPullQPS and registryBurst in the kubelet configuration?
I could not find anything on this in the documentation.
Further information
The kubelet has --registry-qps and --registry-burst command line arguments, but they are already deprecated. Probably the --config /etc/kubernetes/kubelet-config.json option could be used and the configuration values could be defined in the config file.
I checked my node how kubelet is managed there: