Kubernetes requires conntrack binary

[cyrus-mc]

The conntrack binary is required for correct cleanup of network namespaces (https://github.com/kubernetes/kubernetes/blob/95a3cd54cf739019b1211163add7247bd31c0ed7/pkg/kubelet/dockershim/network/hostport/hostport_manager.go#L69).

If CoreOS is to be used as an operating system to run Kubernetes this component should be included.

[root@ip-10-36-8-249 kubelet]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:

• ostree://fedora:fedora/x86_64/coreos/stable
  Version: 31.20200210.3.0 (2020-02-24T16:48:02Z)
  Commit: 4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505
  GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[lucab]

Thanks for the report. I'm not against adding it, but I have some further questions below.

This is pointing out a dependency from kubelet (dockershim), but the OS doesn't ship a kubelet binary. How are you installing and running it?

I think in general there are two good cases here:

• kubelet installed from RPM: in this case conntrack is pulled in via package dependencies.
• kubelet run in a container: in this case conntrack is part of the container image and doesn't need to land in the host OS.

[cyrus-mc]

@lucab

In my case I am building an AMI via Packer and copy the kubelet binary to the OS (following the pattern laid out here - https://github.com/awslabs/amazon-eks-ami).

If you pull in via RPM how would you choose the version of Kubernetes you want to run?

I am not aware of an official Image for kubelet and was looking to not have to manage more component myself and is the reason I copy the binary. CoreOS use to provide images but I can't seem to find those anymore (and not sure conntrack was included in the image).

[dustymabe]

We discussed this very briefly during the meeting today.

In the meeting it was decided we would ask @cyrus-mc this:

@cyrus-mc
In my case I am building an AMI via Packer and copy the kubelet binary to the OS (following the pattern laid out here - https://github.com/awslabs/amazon-eks-ami).

Could you provide conntrack in this same step where you provide kubectl ?

I did test out package layering it and it seems it does have some other deps other than just the one RPM so if you do add it in as a binary in the packer step you might have to add more than just the conntrack binary.

Added:
  conntrack-tools-1.4.5-4.fc31.x86_64
  libnetfilter_cthelper-1.0.0-16.fc31.x86_64
  libnetfilter_cttimeout-1.0.0-14.fc31.x86_64
  libnetfilter_queue-1.0.2-14.fc31.x86_64

[lucab]

Additionally, conntrack-tools brings in both the conntrack utility and a new conntrackd daemon. The daemon does not seem to understand config fragments/dropins, and it ships with a pre-populated conntrackd.conf.

Is the daemon needed too? And anyway, should we ask for split binary packages in order to consume the utility and the daemon separately?

[cyrus-mc]

We discussed this very briefly during the meeting today.

In the meeting it was decided we would ask @cyrus-mc this:

@cyrus-mc
In my case I am building an AMI via Packer and copy the kubelet binary to the OS (following the pattern laid out here - https://github.com/awslabs/amazon-eks-ami).

Could you provide conntrack in this same step where you provide kubectl ?

I did test out package layering it and it seems it does have some other deps other than just the one RPM so if you do add it in as a binary in the packer step you might have to add more than just the conntrack binary.

Added:
  conntrack-tools-1.4.5-4.fc31.x86_64
  libnetfilter_cthelper-1.0.0-16.fc31.x86_64
  libnetfilter_cttimeout-1.0.0-14.fc31.x86_64
  libnetfilter_queue-1.0.2-14.fc31.x86_64

@dustymabe That is what I am currently doing. When I install kubectl I also install/copy conntrack.

I guess it would make more sense to run kubelet via a container. Which is what I use to do on Container Linux when they supplied kubelet images. Don't believe they do that anymore. And I didn't want to get into having to manage my own builds of kubelet images.

What I do now works well for me.

[cyrus-mc]

Additionally, conntrack-tools brings in both the conntrack utility and a new conntrackd daemon. The daemon does not seem to understand config fragments/dropins, and it ships with a pre-populated conntrackd.conf.

Is the daemon needed too? And anyway, should we ask for split binary packages in order to consume the utility and the daemon separately?

Daemon is not needed. Just the CLI utility.

[dghubble]

Kubernetes publishes the k8s.gcr.io/hyperkube container image containing the kubelet binary, along with the conntrack (and other) dependencies. Its used in the community to run the Kubelet on Fedora CoreOS today, without a need for packages to be added to the minimal base OS.

[lucab]

Thanks both for the feedback.
I've requested a package split at https://bugzilla.redhat.com/show_bug.cgi?id=1814190. I'd suggest to re-evaluate this when that is done.

While I still believe that "all kubelet transitive dependencies on-host" is a bad plan, I have the feeling that conntrack specifically may have a few more consumers that we will discover soon via bug reports. I'd keep this open in the meanwhile to attract those kind of reports, if any.

[hakman]

This would also benefit kubernetes/kops. At the moment Kops is using an old static binary for conntrack but seems like the way it was built doesn't work anymore.

[lucab]

@MartinForReal @hakman my understanding is that both kops and kubeadm are just trying to mirror/satisfy the kubelet dependency, is that right? Or are they also using conntrack internally?

[hakman]

In the case ok Kops, it just satisfies the kubelet dependency. I mentioned this here in case this gets some higher priority.
Though I agree that the contrackd service is not needed in this case and would be nice to have it separate, I have a feeling that it's a longer term thing. :)

[MartinForReal]

In the case of kubeadm, it just satisfies the kubelet dependency. FYI

[dghubble]

Update to my comment above, Kubernetes v1.19+ will stop publishing k8s.gcr.io/hyperkube and leave Kubelet to traditional or container image packagers.

In Typhoon, the Kubelet image was spun out of hyperkube and spruced up. Its actively used on Fedora CoreOS clusters. No need for conntrack or the many other Kubelet dependencies to be installed on-host, if you're interested in borrowing those approaches or just awareness.

[ecky-l]

@dghubble I think most people will do their first k8s installation with kubeadm and the native kubelet binary, since this is the topmost officially documented way. There is even a copy/paste section for Container Linux that can be used on Fedora CoreOS with minor adoptions.

Therefore I think it makes sense to have conntrack included. Also conntrack is so far the only dependency that is needed and not included (hopefully it will stay that way...).

[dghubble]

Its unfortunately not the only dependency. Once the Kubelet runs, next you'd want to use it in a conformant manner. For NFS, Gluster, Ceph, and Azure files, the Kubelet environment requires packages (my links above). The Kubelet iptables tool has to align with kube-proxy too. Its a moving target and could change. Imo, its preferable to tightly control that Kubelet environment as a container image when using a container optimized OS.

I view kubeadm as geared for traditional package-based OSes. That curl'd kubelet binary described in their docs would have somewhat worked on CoreOS Container Linux. Because some deps and kubelet-wrapper were shipped in the OS (something we later regretted as I recall). But CoreOS clusters used the hyperkube image in CoreOS supported clusters (e.g. Tectonic).

[ecky-l]

I understand... guess I'll have to look into running kubelet as container, I like the idea anyway. Until then I can personally live with rpm-ostree install conntrack --reboot in my ignition job, or extending that with more packages when the need arises.

It would be nice to have a documented way to run containerized kubelet on Fedora CoreOS, or at least some tutorial to get started.

[ecky-l]

It would be nice to have a documented way to run containerized kubelet on Fedora CoreOS, or at least some tutorial to get started.

Oh, its already done. Nevermind :)

[cgwalters]

In the short term, one option is to add the package by default but strip out the systemd unit. This pattern currently breaks anyone who wants the daemon though, as discussed in e.g. the networkd bits coreos/fedora-coreos-config#648