failed to assign quota after kubelet was restarted

[mchtech]

What happened?

After kubelet was restarted, kubelet starts to output quota related errors and the total count of entries in /etc/projects and /etc/projid decreased.

I0416 17:38:13.086019  157568 empty_dir.go:306] Set quota on /var/lib/kubelet/pods/e5149231-3a81-41e0-87be-579846f6caea/volumes/kubernetes.io~configmap/config failed assign quota FAILED exit status 1
I0416 17:38:13.086051  157568 operation_generator.go:838] "MountVolume.markVolumeErrorState leaving volume uncertain" volumeName=kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config
E0416 17:38:13.086130  157568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config podName:e5149231-3a81-41e0-87be-579846f6caea nodeName:}" failed. No retries permitted until 2024-04-16 17:40:15.086106798 +0800 CST m=+376.261198810 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config") pod "xxx-yyy" (UID: "e5149231-3a81-41e0-87be-579846f6caea") : assign quota FAILED exit status 1
I0416 17:38:13.086156  157568 event.go:294] "Event occurred" object="xxx/yyy" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="FailedMount" message="MountVolume.SetUp failed for volume \"config\" : assign quota FAILED exit status 1"

Similar issues: #115309

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

1. ext4 fs with quota enabled and enforced tune2fs -O project -Q prjquota /dev/xxx; mount -o prjquota ...
2. enable featuregate LocalStorageCapacityIsolationFSQuotaMonitoring in kubelet config
3. register this node to k8s (start kubelet): works well
4. restart kubelet: reproduced

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:23:27Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:16:45Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

IDC

OS version

# On Linux:
$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

$ uname -a
Linux xxx 5.14.0-362.24.2.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 30 14:11:54 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

containerd 1.6.31

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mchtech]

/sig node

[kannon92]

/remove-sig node
/sig storage

[kannon92]

/sig storage

[kannon92]

cc @pacoxu

[xing-yang]

cc @jingxu97