[Wip] Kubernetes talks to flannel-vxlan #13877
Conversation
|
GCE e2e build/test failed for commit b75d42ee3eeb438b690d427ba3fa9e224bf4df72. |
k8s-github-robot
added
needs-rebase
|
Labelling this PR as size/XL |
|
Will this ensure that the proper firewall ports are open on the hosts when it configures flannel for you? |
|
Hmm.. firewall for the vxlan port? For external access or inside the cluster? |
|
@bprashanth - by default the port used by vxlan and flannel was blocked on my host and blocked pod-to-pod communication, so if kube is configuring things, it may want to open the firewall port on the host by doing the proper iptables rule. |
|
cc @kubernetes/rh-cluster-infra |
|
@kubernetes/rh-networking |
| @@ -98,16 +98,13 @@ readonly KUBE_CLIENT_PLATFORMS=( | |||
| linux/amd64 | |||
| linux/386 | |||
| linux/arm | |||
| darwin/amd64 | |||
| darwin/386 | |||
| windows/amd64 | |||
There was a problem hiding this comment.
can't build on non-linux any more?
There was a problem hiding this comment.
ah, heh, that's just a hack. One of the packages flannel includes (netlink i think) had documented problems building on non-linux. Punted on it for a while.
There was a problem hiding this comment.
ah, heh, that's just a hack. One of the packages flannel includes (netlink i think) had documented problems building on non-linux. Punted on it for a while.
netlink is Linux specific. And flannel uses netlink to create the vxlan interfaces at this point. So yeah, anything using flannel + the VXLAN backend would be Linux-only.
|
So is the point of this PR to implement the flannel server API in kubernetes so that the cluster doesn't have to run a master flanneld alongside the apiserver on the master? |
|
Roughly, yes. Flannel doesn't have access to etcd, which is the important part. Kube pushed subnets down to the flannel daemon, not the other way aroud. No new resources are created to manage subnets etc (normal flanneld will create new kvs under --etcd-prefix subdir). This pr is just a proof of concept at the moment. |
bprashanth
force-pushed
the
network_vxlan
branch
from
b75d42e
to
667b873
Compare
|
@wojtek-t rebased, latest patch set includes patched flannel godep |
|
GCE e2e build/test failed for commit 667b873. |
k8s-github-robot
added
size/XS
|
Labelling this PR as size/XS |
k8s-github-robot
added
the
needs-rebase
bprashanth
force-pushed
the
network_vxlan
branch
from
667b873
to
2429be4
Compare
|
GCE e2e build/test failed for commit 2429be4741018ca90612138b6299670a0620c574. |
k8s-github-robot
removed
the
needs-rebase
|
IMO adding flannel bits into Kubernetes would be the wrong solution to the issue. Instead, the solution would be more along the lines of:
We're doing almost exactly the same thing for openshift-sdn, except we'll be running a flannel master that talks to the kubernetes etcd instance, rather than stuffing all of flannel into the Kubernetes process itself. |
|
Only reason it is the way it is, is because nothing else is ready yet. Even getting this for 1.2 and churning a little after that won't be too bad, it's isolated enough to split out eitherway. Exposing etcd to flannel probably won't fly, though. |
bprashanth
force-pushed
the
network_vxlan
branch
from
2429be4
to
4bf6f07
Compare
|
GCE e2e build/test failed for commit 4bf6f07b20b42fd1a95353ea99cd2e7edf15f90f. |
* Node compare through LeaseClient store * Watch doesn't break for each node update * Logging
bprashanth
force-pushed
the
network_vxlan
branch
from
4bf6f07
to
8b92564
Compare
|
GCE e2e build/test failed for commit 8b92564. |
@bprashanth Why is that, especially if using flannel's listen/remote mode so that only the master flannel instance can talk to etcd? |
k8s-github-robot
added
the
needs-rebase
|
@dcbw directly accessing etcd is verboten for a few reasons (api server checks schema of a database, the db is pluggable and can be mysql, we shard/replicate the apiserver not the db etc). It certainly is possible, and listen/remote makes it less egregious, but it will take some convincing before everyone just agress to writing to etcd. i.e we have a higher chance of getting stuff done by going through the apiserver, and an even higher chance if we just let the master assign the subnets like it does today (though I'll concede there are advantages to letting flannel do the latter, and that's an easier sell than the former). |
@bprashanth I see some of the arguments here; though personally (and I'm not authoritative on the matter) I really think anything like this, especially adding a whole new API surface, should be done in a network provider once the network provider proposals land. That way the API is contained in your external network provider, and not added to Kubernetes itself. |
|
We already hacked in flannel integration in for 1.2, hopefully it will get cleaned up through CNI plugins for 1.3 |
Requires flannel as a godep with flannel-io/flannel#308
Invoked via USE_OVERLAY="true" ./hack/local-up-cluster.sh or e2e.go -pushup
Only uploading so I can reference in networking discussions. Won't build because the godep isn't included, and including it will make the pr less referencable because of diff size. Godep commit hash 8b260bc.