Use an init container for CNI when in k8s api datastore mode

[caseydavenport]

Description

I'd like to do this for etcd as well, but it needs more thought first. There's some code which runs and updates certs if they change. I'm not sure if anyone uses that code, so maybe we can just drop it.

I say we do it for kdd anyway, since it's an easy win.

• One fewer docker containers per node required (which takes up resources).
• kubectl logs no longer requires specifying container name.
• simpler kubectl describe output, since it's only for one container.

Todos

• Tests
• Documentation
• Release note

Release Note

The CNI plugin is now installed using a Kubernetes init container rather than a long-lived sidecar.

[fasaxc]

Any way we can avoid the duplication in the template? I see the value in simplifying for the user and reducing our resource use but if the template gets duplicated we're more likely to introduce bugs in maintenance.

[fasaxc]

Think I'd prefer making such a change at the start of a release cycle; just to give us time to spot any unexpected quirk of initContainers.

[caseydavenport]

Think I'd prefer making such a change at the start of a release cycle; just to give us time to spot any unexpected quirk of initContainers.

Yep, SGTM. Master is now v3.4 dev, so we should be OK now.

Any way we can avoid the duplication in the template? I see the value in simplifying for the user and reducing our resource use but if the template gets duplicated we're more likely to introduce bugs in maintenance.

Hm, I'll think about it. There may be a way. Honestly, I'd like to move etcd over to this approach as well anyway, so maybe we should just do that.