[WARNING] InvalidDiskCapacity warning thrown in node events by Kubelet

[furkatgofurov7]

What happened?

Kubelet keeps throwing InvalidDiskCapacity kubelet invalid capacity 0 on image filesystem in the nodes status

  Normal   NodeHasNoDiskPressure    29m (x5 over 29m)  kubelet  Node cp-host-1 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     29m (x5 over 29m)  kubelet  Node cp-host-1 status is now: NodeHasSufficientPID
  Normal   NodeHasSufficientMemory  29m (x6 over 29m)  kubelet  Node host-1 status is now: NodeHasSufficientMemory
  Normal   NodeAllocatableEnforced  29m                kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 29m                kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      29m                kubelet  invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  29m                kubelet  Node cp-host-1 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    29m                kubelet  Node cp-host-1 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     29m                kubelet  Node cp-host-1 status is now: NodeHasSufficientPID
  Normal   Starting                 29m                kubelet  Starting kubelet.

What did you expect to happen?

No warnings thrown in output of node events

How can we reproduce it (as minimally and precisely as possible)?

n/a

Anything else we need to know?

I have checked couple of reported issues open/closed ones in the repo and most of them have this in logs when node logs were provided, can this be considered benign or disregarded?
Also, the same has been reported some time ago (2018) but that was seen in windows nodes and was fixed here. Maybe, this should be OS agnostic and fix should be extended?

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-11-04T09:36:56Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Baremetal

OS version

SLES 15 SP2

Install tools

Container runtime (CRI) and and version (if applicable)

containerd version:  containerd github.com/containerd/containerd v1.5.6 1a1b383ad5b520349f13f9715e0cd1e2f132c087

Related plugins (CNI, CSI, ...) and versions (if applicable)

CNI is not deployed at that point

[furkatgofurov7]

/cc @feiskyer

[MadhavJivrajani]

/sig node

[pacoxu]

kubernetes/pkg/kubelet/stats/cadvisor_stats_provider.go

Lines 246 to 271 in ea07644

	func (p *cadvisorStatsProvider) ImageFsStats() (*statsapi.FsStats, error) {
	imageFsInfo, err := p.cadvisor.ImagesFsInfo()
	if err != nil {
	return nil, fmt.Errorf("failed to get imageFs info: %v", err)
	}
	imageStats, err := p.imageService.ImageStats()
	if err != nil || imageStats == nil {
	return nil, fmt.Errorf("failed to get image stats: %v", err)
	}
	var imageFsInodesUsed *uint64
	if imageFsInfo.Inodes != nil && imageFsInfo.InodesFree != nil {
	imageFsIU := *imageFsInfo.Inodes - *imageFsInfo.InodesFree
	imageFsInodesUsed = &imageFsIU
	}
	return &statsapi.FsStats{
	Time: metav1.NewTime(imageFsInfo.Timestamp),
	AvailableBytes: &imageFsInfo.Available,
	CapacityBytes: &imageFsInfo.Capacity,
	UsedBytes: &imageStats.TotalStorageBytes,
	InodesFree: imageFsInfo.InodesFree,
	Inodes: imageFsInfo.Inodes,
	InodesUsed: imageFsInodesUsed,
	}, nil
	}

It is most likely a cadvisor related issue on SLES 15 SP2.
/area cadvisor

[SergeyKanzhelev]

/triage accepted
/cc @jackfrancis
/assign @ehashman

[jackfrancis]

@pacoxu do you think that #101882 will help?

Upon first glance, I don't think this is related

[pacoxu]

@pacoxu do you think that #101882 will help?

Maybe.
I need to do some deep digging.

[ehashman]

/remove-triage accepted
/triage needs-information

Reading over this more closely, this may be an install environment specific issue.

Can you please attach a copy of the kubelet logs (preferably at v=4 or higher) for the affected node?

[k8s-ci-robot]

@furkatgofurov7: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ehashman]

/unassign

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[furkatgofurov7]

Can you please attach a copy of the kubelet logs (preferably at v=4 or higher) for the affected node?

Hey, sorry I was not able to come back to this issue earlier, unfortunately, but this can be closed since it is a "WARNING" thrown without any real impact on the node.

/close

[k8s-ci-robot]

@furkatgofurov7: Closing this issue.

In response to this:

Can you please attach a copy of the kubelet logs (preferably at v=4 or higher) for the affected node?

Hey, sorry I was not able to come back to this issue earlier, unfortunately, but this can be closed since it is a "WARNING" thrown without any real impact on the node.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.