Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubelet does not work with busybox du command #95172

Closed
ncopa opened this issue Sep 30, 2020 · 7 comments · Fixed by #96115
Closed

kubelet does not work with busybox du command #95172

ncopa opened this issue Sep 30, 2020 · 7 comments · Fixed by #96115
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage.

Comments

@ncopa
Copy link
Contributor

ncopa commented Sep 30, 2020

What happened:
Running sonobouy test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the du command.

E0928 14:39:57.194066    2455 cri_stats_provider.go:591] Unable to fetch container log stats for path /var/log/pods/kube-system_metrics-server-7d4bcb75dd-gvqcf_107aa715-8c80-491c-b679-532578bda180/metrics-server: failed command 'du' ($ nice -n 19 du -x -s -B 1) on path /var/log/pods/kube-system_metrics-server-7d4bcb75dd-gvqcf_107aa715-8c80-491c-b679-532578bda180/metrics-server with error exit status 1

What you expected to happen:
I expected the du call to exit with success.

How to reproduce it (as minimally and precisely as possible):
Install and run kublet on Alpine Linux without installing coreutils.

Anything else we need to know?:
It failed due to the -B 1 option which is not implemented in busybox.

The -B option is not specified in POSIX so I don't think it is reasonable to expect it to implemented in busybox.

Environment:

  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"archive", BuildDate:"2020-09-21T06:09:21Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-09-14T11:56:23Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    QEMU/KVM virtual machine
  • OS (e.g: cat /etc/os-release):
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.12.0
PRETTY_NAME="Alpine Linux v3.12"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
  • Kernel (e.g. uname -a):
Linux aworker0 5.4.43-1-virt #2-Alpine SMP Thu, 28 May 2020 20:13:48 UTC x86_64 GNU/Linux

  • Install tools:
    custom

  • Network plugin and version (if this is a network-related bug):
    not network-related

  • Others:
@ncopa ncopa added the kind/bug Categorizes issue or PR as related to a bug. label Sep 30, 2020
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 30, 2020
@ncopa
Copy link
Contributor Author

ncopa commented Sep 30, 2020

/sig Storage

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 30, 2020
This was referenced Sep 30, 2020
Closed
Closed
ncopa added a commit to ncopa/kubernetes that referenced this issue Oct 28, 2020
@ncopa
Fix du command to be portable (kubernetes#95172)
e8970cf 
The `-B 1` argument is not defiend in POSIX du[1] and does not work with
busybox du. Fix this by using `-k` and append "Ki" for the ParseQuantity
call.

`resource.ParseQuantity()` will set the format to `BinarySI` when there
is a "Ki" suffix so we can remove the explicit set of the format.

[1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/du.html

Signed-off-by: Natanael Copa <ncopa@mirantis.com>
@ncopa ncopa mentioned this issue Nov 2, 2020
Merged
algitbot pushed a commit to alpinelinux/aports that referenced this issue Nov 19, 2020
@ncopa
testing/kubernetes: enable on armv7 fix deps and linker options
7ab399e 
enable on armv7

build without providers and strip ELF symbols and debug info.

kubelet does not have a hard dependency on cni-plugins. They can be
installed from a pod.

kubelet need GNU du[1] from coreutils and GNU find[2].

[1]: kubernetes/kubernetes#95172
[2]: kubernetes/kubernetes#95186
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 29, 2020
@ncopa
Copy link
Contributor Author

ncopa commented Jan 1, 2021

/remove-lifecycle stale

#96115 fixes this

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 1, 2021
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 1, 2021
ncopa added a commit to ncopa/kubernetes that referenced this issue Apr 6, 2021
@ncopa
Get inodes and disk usage via pure go
d45ba64 
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (kubernetes#95172)

This is similar to what cadvisor does since commit
google/cadvisor@046818d

This solves three problems:
- Counts number of inodes correct when there are hardlinks (kubernetes#96114)
- Makes kubelet work without GNU findutils (kubernetes#95186)
- Makes kubelet work without GNU coreutils (kubernetes#95172)
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 1, 2021
@ncopa
Copy link
Contributor Author

ncopa commented May 3, 2021

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 3, 2021
@ncopa
Copy link
Contributor Author

ncopa commented May 3, 2021

Still waiting for PR #96115 to be merged

@ncopa ncopa mentioned this issue May 4, 2021
Closed
249043822 pushed a commit to 249043822/kubernetes that referenced this issue Jul 30, 2021
@ncopa @249043822
Get inodes and disk usage via pure go
4743b03 
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (kubernetes#95172)

This is similar to what cadvisor does since commit
google/cadvisor@046818d

This solves three problems:
- Counts number of inodes correct when there are hardlinks (kubernetes#96114)
- Makes kubelet work without GNU findutils (kubernetes#95186)
- Makes kubelet work without GNU coreutils (kubernetes#95172)
249043822 pushed a commit to 249043822/kubernetes that referenced this issue Jul 30, 2021
@ncopa @249043822
Get inodes and disk usage via pure go
5b13c8f 
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (kubernetes#95172)

This is similar to what cadvisor does since commit
google/cadvisor@046818d

This solves three problems:
- Counts number of inodes correct when there are hardlinks (kubernetes#96114)
- Makes kubelet work without GNU findutils (kubernetes#95186)
- Makes kubelet work without GNU coreutils (kubernetes#95172)
249043822 pushed a commit to 249043822/kubernetes that referenced this issue Jul 30, 2021
@ncopa @249043822
Get inodes and disk usage via pure go
5a15256 
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (kubernetes#95172)

This is similar to what cadvisor does since commit
google/cadvisor@046818d

This solves three problems:
- Counts number of inodes correct when there are hardlinks (kubernetes#96114)
- Makes kubelet work without GNU findutils (kubernetes#95186)
- Makes kubelet work without GNU coreutils (kubernetes#95172)
249043822 pushed a commit to 249043822/kubernetes that referenced this issue Jul 30, 2021
@ncopa @249043822
Get inodes and disk usage via pure go
d18234f 
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (kubernetes#95172)

This is similar to what cadvisor does since commit
google/cadvisor@046818d

This solves three problems:
- Counts number of inodes correct when there are hardlinks (kubernetes#96114)
- Makes kubelet work without GNU findutils (kubernetes#95186)
- Makes kubelet work without GNU coreutils (kubernetes#95172)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Milestone
No milestone
3 participants
@ncopa @k8s-ci-robot @fejta-bot