@pwittrock What is the best way to document these requirements moving forward?

@luxas Classy digging, and this is super-useful to have. +100

We should decide at what level we want to document transitive deps. For example, cadvisor is going to begin calling thin_ls to monitor containers backed by devicemapper thin pools. Since disk eviction is going to use this, I would want to know about that specific dep, though it is not strictly a dependency of the kubelet itself.

@vishh I think adding a page to the "Administering Clusters" at http://kubernetes.io/docs/ would be the best place

@pmorie Thanks :) It took some time to dig out.

When we document it, I think we should present them in groups, e.g. always required, required with option --xxx, required when using yyy as the storage driver, and nice-to-haves.

But we should also get rid of some:

• glibc: Make all Kubernetes binaries statically linked #26028
• touch when using rkt. Seems simple enough to remove.
• iptables we shouldn't call NewFlannelOverlay() regardless of the option, only if it's enabled.

Some things I've learned so far (will update the list)

• iptables is effectively required for the kubelet, although flannel is deactivated: Do not call NewFlannelServer() unless flannel overlay is enabled #26264
• journalctl shouldn't be a dependency to cAdvisor, that's a bug: OOM detection logic doesn't work inside a docker container. google/cadvisor#1313

There's a dependency on dmidecode in the vSphere volume plugin.

@pmorie Added. Do you remember any others right now?

@luxas Nope, I just saw that one during a code review.

In addition to documenting these, I think it would be useful to check for all of them on kubelet startup. Hard dependencies should log fatal errors, and soft dependencies could log warnings.

nfs-utils (at least on Fedora/Red Hat/CentOS based systems) is needed by the nfs volume plugin.

ceph-common (at least on Fedora/Red Hat/CentOS based systems) is needed for the ceph volume plugin

glusterfs-fuse should be all that is needed for using the gluster storage plugin (at least on Fedora/Red Hat/CentOS based systems).

iscsi-initiator-utils (Fedora/Red Hat/CentOS based systems) is needed by the iscsi volume plugin

@timstclair I'm not sure you would want a fatal error if a storage plugin dependency isn't available, I would expect them to just not be loaded (and an obvious event and log message generated) if that is the case.

Thanks @detiber! (And for anyone out there that knows even more deps of the kubelet, comment!)

I think @timstclair meant that all storage drivers are soft deps, so kubelet would log a warning.

Is this part of NodeSpec? I am still unclear what kubernetes/enhancements#56 is trying to describe.

cc @vishh @dchen1107 @yujuhong @aronchick @aaronlevy

Just noticed that pkg/kubelet/container_bridge.go has a lot of dependencies on binaries in $PATH.

• brctl
• ip
• pkill
• (service|systemd|docker-babysitter)

Summarizing the PATH requirements for kubelet (oh my!):

• pidof in pkg/kubelet/cm/container_manager_linux.go
• uname -n in pkg/util/node/node.go
• mount and umount in pkg/util/mount/mount_linux.go
• lsblk, fsck, mkfs in pkg/util/mount/mount_linux.go
• nsenter in a lot of places!
• ethtool in pkg/kubelet/network/hairpin/hairpin.go
• socat in pkg/kubelet/dockertools/docker_manager.go and pkg/kubelet/rkt/rkt.go
• touch in pkg/kubelet/rkt/rkt.go
• env (inside the container) in pkg/kubelet/dockertools/exec.go
• iptables, iptables-restore, iptables-save in pkg/util/iptables/iptables.go and many other places
• brctl in pkg/kubelet/container_bridge.go
• pkill in pkg/kubelet/container_bridge.go
• ip in pkg/kubelet/container_bridge.go, pkg/kubelet/network/plugins.go, pkg/kubelet/rkt/rkt.go
• modprobe in pkg/kubelet/network/plugins.go
• tc in pkg/util/bandwidth/linux.go
• journalctl in vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go, but it's not a hard requirement
• thin_ls in vendor/github.com/google/cadvisor/devicemapper/thin_ls_client.go for the devicemapper storage driver
• nice, du, dmsetup in vendor/github.com/google/cadvisor/fs/fs.go, are these important?

Hard requirements (kubelet should fail without these):

• root access (is there an easy way to check for CAP_SYS_ADMIN?)
• cpu, cpuset, cpuacct, memory cgroups mounted
• iptables, ip must be present for kubelet networking
• nsenter must be present for kubelet container management
• mount and umount is required for nearly all types of volumes
• syscall.Mount() is a wrapper for mount(2), but if we strictly need mount(8), that's a hard dependency @kubernetes/sig-storage
• the br-netfilter module. kubelet should try to enable it on linux >= 3.18 with modprobe, and if it isn't enabled earlier or if modprobe doesn't exist, kubelet should fail (or should it?)
• write access to /sys, /var/lib/docker and /var/lib/kubelet

I'm not sure if these should be hard or soft deps:

• socat might be replaced by the gRPC protocol when Exec-ing, but still it's used for port-forwarding

How to remove some dependencies (before v1.4):

• uname -n should be replaced with os.Hostname
• brctl should be replaced by calls to the netlink go package
• pkill with syscall.Kill
• pidof, if we think it's worth it, we can do this
• touch can be replaced with os.Chtimes link
• ethtool used for hairpin, but ip link might be used instead

A dedicated pkg/util/nsenter package should be made, now it's called in ~10 different places in 10 different ways.

I can start to implement a pkg/kubelet/validator package that checks for these dependencies.

WDYT?
@kubernetes/sig-cluster-lifecycle @kubernetes/sig-node @kubernetes/sig-network

• mount and umount is required for nearly all types of volumes
• syscall.Mount() is a wrapper for mount(2), but if we strictly need mount(8), that's a hard dependency @kubernetes/sig-storage

Not only we need /bin/mount, we need all mount utilities like mount.nfs, mount.ceph and mount.glusterfs. Simple mount(2) syscall is not enough.

cc @andrewrynhard

I'm not sure if that's the right place to drop that (perhaps new issue should be created?), but for people running kubelet as a container while deploying Kubernetes, it would be super useful to have a list of capabilities, host directories etc where kubelet needs to have access on host in order to function properly.

For my project, I run kubelet as a Docker container. I've tried looking for some documentation on how to do that, but I couldn't find any, so I had to do that simply with trials and errors, starting with simply running kubelet from hyperkube image via Docker, looking into error messages and googling each one for a solution.

I documented it here, even though it's probably incomplete, as I didn't test everything yet.

I documented it here, even though it's probably incomplete, as I didn't test everything yet.

this type of documentation improvement is nice to have.
you can get feedback on this in the #sig-node channel on k8s slack or if you join their meeting:
https://github.com/kubernetes/community/tree/master/sig-node

Just a note here - I had my own version of nsenter installed (via pip3 install) on ubuntu20. This caused a continual kubelet crash which took a while to diagnose - kubelet of course wanted nsenter from util-linux.

It sure would be nice if the preflight called out the dependency on the distro version of nsenter.

Oct 09 18:45:59 faucet kubelet[296741]: E1009 18:45:59.180219 296741 pod_workers.go:191] Error syncing pod db24ffc3-ef58-45b8-87f9-6da3dd701ce7 ("coredns-f9fd979d6-7d488_kube-system(db24ffc3-ef58-45b8-87f9-6da3dd701ce7)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 40s restarting failed container=coredns pod=coredns-f9fd979d6-7d488_kube-system(db24ffc3-ef58-45b8-87f9-6da3dd701ce7)"
Oct 09 18:45:59 faucet kubelet[296741]: W1009 18:45:59.259650 296741 docker_sandbox.go:402] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "coredns-f9fd979d6-mm6kj_kube-system": unexpected command output usage: nsenter [-h] --target PID [--proc PROCFS] [--user] [--net] [--ipc]
Oct 09 18:45:59 faucet kubelet[296741]: [--uts] [--pid] [--mnt] [--all]
Oct 09 18:45:59 faucet kubelet[296741]: [command [command ...]]
Oct 09 18:45:59 faucet kubelet[296741]: nsenter: error: argument --net: ignored explicit argument '/proc/327613/ns/net'
Oct 09 18:45:59 faucet kubelet[296741]: with error: exit status 2
Oct 09 18:45:59 faucet kubelet[296741]: E1009 18:45:59.260055 296741 pod_workers.go:191] Error syncing pod 5b915605-4d4a-4e22-967c-aaeee184f0cd ("coredns-f9fd979d6-mm6kj_kube-system(5b915605-4d4a-4e22-967c-aaeee184f0cd)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 40s restarting failed container=coredns pod=coredns-f9fd979d6-mm6kj_kube-system(5b915605-4d4a-4e22-967c-aaeee184f0cd)"

/joke

@XuanYang-cn: A termite walks into a bar and asks “Is the bar tender here?”

/triage accepted

/priority important-longterm

/remove-priority backlog

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

HI @thockin,

Could you please share why did you close this one?
Was it not accepted in the triage? If so, what is the reason?

Mostly backlog grooming. This is 7 years old and the last meaningful update was 3 years ago. Does it serve a purpose to leave it open? Is someone going to do it? It's a wonderful idea....if someone will act on it, let's reopen?