Bad things happen when a DaemonSet with hostNetwork:true is deployed

[jpweber]

When deploying a DaemonSet with hostNetwork set to true an assortment of network related issues happened. kube-apiserver was not reachable. Hosts could not be SSHed in to, conduit gui could not connect to conduit components etc.

I'm raising this issue not because I want to deploy a DaemonSet with conduit injected in to and use hostNetwork. But I don't think it should effectively take down a cluster either. There has to be a way to protect from this. To replicate the issue deploy the manifest below.

This occurred in the following environment.
k8s 1.9.3
ubuntu 16.04

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter
    name: node-exporter
  name: node-exporter
spec:
  clusterIP: None
  ports:
  - name: scrape
    port: 9100
    protocol: TCP
  selector:
    app: node-exporter
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  template:
    metadata:
      labels:
        app: node-exporter
      name: node-exporter
    spec:
      containers:
      - image: prom/node-exporter
        name: node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100 #comment these out for it to work
          name: scrape
      hostNetwork: true #comment these out for it to work
      hostPID: true
      tolerations:
        - effect: NoSchedule
          operator: Exists

[briansmith]

Is this caused buy the init container rewriting the iptables rules for the entire host when hostNetwork: true?

I believe conduit inject needs to either ignore hostNetwork: true pods (with a warning to stderr?) or exit with an error message and non-zero exit code when a hostNetwork: true pod is used.

See istio/istio.io#655.

/cc @pcalcado

[briansmith]

Also, are we really intending to inject into deamonsets by default? We have code that appears to do so, but I don't know that we actually intend to do that. I don't think the UI is expecting injections into non-pod objects, at least. For example, the UI doesn't discern the type of object and might even group a pod and a deamonset with the same name together or do other bad things. I think we should restrict injection to pods exclusively and file a follow-up issue about spec'ing what, if anything, to do about deamonsets.

[olix0r]

Wow, @jpweber, nice find!

---

@briansmith

are we really intending to inject into deamonsets by default?

the UI doesn't discern the type of object and might even group a pod and a deamonset with the same name together or do other bad things

This isn't how I understand daemonsets -- daemonsets are like replicacontrollers or deployments, essentially, in that they are a parent resource that creates pods. So I think daemonsets are very much analogous to what we refer to as "deployments" in the UI at the moment.

There's no reason that I'm aware of that we wouldn't want to support injecting daemonsets.

However, it seems we absolutely must avoid adding proxy-init into any resource that has hostNetwork: true.

[wmorgan]

@olix0r should we try to get a fix in for 0.3? "conduit inject may effectively take down a cluster" is a pretty big caveat for a release.

[wmorgan]

@jpweber We'll fix this for 0.3, scheduled for next week.