conntrack util not installed

[johanneswuerbach]

kubernetes uses the conntrack binary to perform some UDP related cleanup operations like kubernetes/kubernetes#59286 or kubernetes/kubernetes#22573

It seems that kops is not explicitly installing the conntrack tool, causing those cleanup tasks to never be performed.

Examples of others installing it
kubernetes/kubernetes#26839, kubernetes/kubernetes#64640

1. What kops version are you running? The command kops version, will display
this information.

Version 1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS / CoreOS nodes

4. What commands did you run? What is the simplest way to reproduce this issue?

conntrack

5. What happened after the commands executed?

conntrack: command not found

6. What did you expect to happen?

conntrack to be installed

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-06-22T07:08:46Z
  labels:
    kops.k8s.io/cluster: xxx
  name: master-a
spec:
  image: coreos.com/CoreOS-stable-*-hvm
  machineType: t2.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    beta.kubernetes.io/fluentd-ds-ready: "true"
  role: Master
  subnets:
  - a

[geojaz]

@johanneswuerbach
Thanks for the report, sorry you're running into this problem.
Could you please clarify the exact coreos version string and AMI that you're using?

Also, it would be super helpful if you could post the logs that show where k8s tried to call out to conntrack and failed.

I'd need more information to make a definitive call, but I'm wondering if talking to coreos/redhat is the right place to push this forward? It sounds to me like a problem with the AMI, not a problem with how it was deployed. We (kops team: ie-> @justinsb ) create the debian images but I'm not sure if we're involved in the others.

Is that fair and/or useful?

[johanneswuerbach]

We are running the latest stable CoreOS AMIs in us-east-1, currently 1800.7.0 / ami-00cc4337762ba4a52

The addition of conntrack to CoreOS itself was rejected previously, coreos/bugs#485

Going forward with this I see currently three options:

1. Build custom AMIs, which is kind of a bummer as that will cause problems with CoreOS auto-updating
2. Modify the launch config user-data to fetch conntrack on boot (this is already done for the kubelet binary)
3. Create a systemd one-shot hook, which fetches conntrack

We will go with 3. now, but I would prefer to upstream this into kops and would happily work on 2., if this is accepted. Option 2 is also similar, to what is proposed for GCI in upstream k8s kubernetes/kubernetes#64640

[geojaz]

Ok that's useful context. Thanks for the additional details.

1. I can't speak for everyone, but I would say that baking coreos images is not something that we're going to support today and probably don't want to support longer term.

2. We could theoretically hack the launch config and grab it during node bootstrapping. I think this solution is not too terrible... but I don't love hacking the launchconfig for everyone's special case. I really wish Cores would reconsider Rename to kops #1

3. This is probably the current path of least resistance. I was doing something similar in my apparmor hook solution here kope.io/k8s-1.9-jessie and stretch images have a broken dbus #5450. It's not ideal, but it works. I think it gets you going most quickly.

Thinking a bit deeper, option 1 is not possible this evening, but I think it's something that we may want to do as our release process automation matures.

[johanneswuerbach]

I would strongly speak against creating custom CoreOS images as this breaks tooling like https://github.com/coreos/container-linux-update-operator, which is the main reason for us to use CoreOS (no more manual security upgrades)

There are already other customisation for CoreOS (like pulling socat) and I would just see that inline with this

kops/upup/pkg/fi/cloudup/apply_cluster.go

Lines 1084 to 1085 in 56ccfac

	// At this time we just copy the socat binary to all distros. Most distros will be there own
	// socat binary. Container operating systems like CoreOS need to have socat added to them.

Overall the recommended approach for CoreOS is running the kubelet itself inside a container, but thats a whole different story.