Failed to pull image: failed to pull and unpack image

[ysicing]

Environmental Info:
K3s Version: k3s version v1.28.6+k3s2 (c9f49a3)
go version go1.20.13

Node(s) CPU architecture, OS, and Version: Linux qcloud.bj 6.1.0-16-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.67-1 (2023-12-12) x86_64 GNU/Linux

Cluster Configuration: 3servers, 8agents ,

k3s args

ExecStart=/usr/local/bin/k3s agent \
                --server https://100.90.80.70:6443 \
                --token token123 \
                --prefer-bundled-bin \
                --vpn-auth="name=tailscale,joinKey=tskey-auth-token,extraArgs=--accept-dns=false" \
                --pause-image ccr.ccs.tencentyun.com/k7scn/mirrored-pause:3.6 \
                --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \
                --kube-proxy-arg "metrics-bind-address=0.0.0.0"

Describe the bug:

All of a sudden, there's no operation. ctr or crictl pulling images by hand is no problem

Additional context / logs:

  Normal   Pulling    24s (x3 over 64s)  kubelet            Pulling image "docker.io/bitnami/os-shell:11-debian-11-r91"
  Warning  Failed     22s (x3 over 63s)  kubelet            Failed to pull image "docker.io/bitnami/os-shell:11-debian-11-r91": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "docker.io/bitnami/os-shell:11-debian-11-r91": failed commit on ref "layer-sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c": unexpected commit digest sha256:425329cb125bd43cf57e19612d273907f091f566ec9856943355f964218e4c91, expected sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c: failed precondition
  Warning  Failed     22s (x3 over 63s)  kubelet            Error: ErrImagePull
  Normal   BackOff    11s (x3 over 62s)  kubelet            Back-off pulling image "docker.io/bitnami/os-shell:11-debian-11-r91"
  Warning  Failed     11s (x3 over 62s)  kubelet            Error: ImagePullBackOff

k3s.log

Mar 09 10:49:43 qcloud.bj  k3s[397499]: E0309 10:49:43.845849  397499 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = FailedPrecondition desc = failed to pull and unpack image \"docker.io/bitnami/os-shell:11-debian-11-r91\": failed commit on ref \"layer-sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c\": unexpected commit digest sha256:425329cb125bd43cf57e19612d273907f091f566ec9856943355f964218e4c91, expected sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c: failed precondition" image="docker.io/bitnami/os-shell:11-debian-11-r91"
Mar 09 10:49:43 qcloud.bj  k3s[397499]: E0309 10:49:43.845891  397499 kuberuntime_image.go:53] "Failed to pull image" err="rpc error: code = FailedPrecondition desc = failed to pull and unpack image \"docker.io/bitnami/os-shell:11-debian-11-r91\": failed commit on ref \"layer-sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c\": unexpected commit digest sha256:425329cb125bd43cf57e19612d273907f091f566ec9856943355f964218e4c91, expected sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c: failed precondition" image="docker.io/bitnami/os-shell:11-debian-11-r91"
Mar 09 10:49:43 qcloud.bj  k3s[397499]: E0309 10:49:43.845975  397499 kuberuntime_manager.go:1261] init container &Container{Name:volume-permissions,Image:docker.io/bitnami/os-shell:11-debian-11-r91,Command:[/bin/bash -ec chown -R 1001:1001 /bitnami/mariadb
Mar 09 10:49:43 qcloud.bj  k3s[397499]: ],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:data,ReadOnly:false,MountPath:/bitnami/mariadb,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod bjmariadb-0_pangu(d426569d-a6b9-400a-8f69-c996921c6dd1): ErrImagePull: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "docker.io/bitnami/os-shell:11-debian-11-r91": failed commit on ref "layer-sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c": unexpected commit digest sha256:425329cb125bd43cf57e19612d273907f091f566ec9856943355f964218e4c91, expected sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c: failed precondition
Mar 09 10:49:43 qcloud.bj  k3s[397499]: E0309 10:49:43.846024  397499 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"volume-permissions\" with ErrImagePull: \"rpc error: code = FailedPrecondition desc = failed to pull and unpack image \\\"docker.io/bitnami/os-shell:11-debian-11-r91\\\": failed commit on ref \\\"layer-sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c\\\": unexpected commit digest sha256:425329cb125bd43cf57e19612d273907f091f566ec9856943355f964218e4c91, expected sha256:832a6e76f1bb7c935336cfba4100933a5964edf819a0b56eaaf756bd9c209d4c: failed precondition\"" pod="pangu/bjmariadb-0" podUID="d426569d-a6b9-400a-8f69-c996921c6dd1"