-
Notifications
You must be signed in to change notification settings - Fork 39.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubelet network plugin for client/server container runtimes #28667
Comments
I don't think so.
Precisely
I think the only model is that we define what we expect from a runtime wrt network, which WILL eventually include a notion of multiple-networks, and then we expect the runtime to do the work. This means that our "docker" runtime is really "docker + CNI".
I think we also need to spec this. What is required in order for the out-of-the-box kube-proxy to function. |
Have we discussed having both? the runtime declares upfront if it wants kubernetes to handle networking or do it privately (in which case the kubelet just uses the noOp plugin) |
"let's do both" is the trap we always fall into. It's part of what makes On Fri, Jul 8, 2016 at 10:03 AM, Prashanth B notifications@github.com
|
I mean |
If most runtimes work with net namespaces, and only a few hypervisor-based runtimes are exceptions, it might be worth handling the plugins in kubelet for the majority of the runtimes. This saves those runtimes a lot of pain to integrate. |
Given that we intend to let this API handle other OSes too in the future and as per @thockin, there might be an user who wants to just use docker networking, I'd prefer integrating network config into the new runtime API. |
cc @kubernetes/sig-rktnetes |
NS-based:
Incompatible with NS:
It's not looking like "most" working with NS, it's more like 50/50 |
There are proposals/discussions to integrate runc (oci) runtime, kurma, etc. It's the question of whether we want to make integration of non-hypervisor based runtime easier. If they can easily reuse/import what we maintain for the docker integration, that might be enough. On the other hand, it's probably not that easy to reuse our network-related code as it is today. |
Proposal of letting kubelet handle networking by CNI pluginBenifits of this proposal
The propoposed method involves changes in CNI partCNI needs to support hypervisor-based container runtime first, which is on progress:
type RuntimeConf struct {
// If set, a tap device instead of veth pair should be created
UsesTapDevice bool
....
}
Kubelet partKubelet needs to update CNI version and support hypervisor-based container runtimes:
Container Runtime API partSince container runtime type (hypervisor or netns) is required for CNI network plugin, the type should be got from runtime API. A new field message VersionResponse {
....
// If set, the container runtime is based-on hypervisor
optional bool is_hypervisor = 5;
} Since CNI network plugin do not setup interfaces inside hypervisor, kubelet should pass the network information (interface name, MAC address, ip address and so on) got from CNI network plugin back to runtime. A new interface
Disadvantage compared to let runtimes handle networking
|
I commented on the CNI proposal, but this just feels backwards to me. It Do we really expect that every CNI plugin will work for every runtime? I On Mon, Aug 1, 2016 at 6:26 AM, Pengfei Ni notifications@github.com wrote:
|
@feiskyer, why can't we just have one CNI plugin that understands VM networking and passes the plugin name to kubelet if user wants hypervisor-based runtimes? That way, we don't have to force every cni plugin to understand the new semantics. |
@thockin Yes, I'm expecting all core plugins inside CNI for this. I think @lukasredynk and @jellonek also expect this since they are working on the implementation of this inside CNI. But other external plugins may be opt-in for this.
@yujuhong Same with above, a general CNI for all runtimes is expected, so there is no need to handle networking differently for different runtimes. And this could give users more options of network plugins. |
I think you misunderstood my point. I don't think that is a reasonable On Mon, Aug 1, 2016 at 6:40 PM, Pengfei Ni notifications@github.com wrote:
|
@thockin Yes, we can definitely implement a specific plugin to handle HyperContainer's network, but so do the rkt and other hypervisor container users (like Mirantis) have to. Since hypervisor containers' networks share lots in common, it will be great if we can (actually we are eager to) collaborate on one project instead of re-inventing wheel on each side. Obviously, this proposal is not asking for CNI to support all kind of vendors. In fact, it does not have (and shouldn't have) any hyper/rkt specific refactoring, it is just asking for support another kind of container runtime (since Otherwise, hyper & rkt & others may have to build another interface under CNI, well, a standard under a standard ... |
Using host networking we are already gathering this information but because of not passing this info to execution driver - we are cutting this information from exec drivers, removing possibility to tune config for specific driver, or showing this info on itself. Also host networking is incompatible with VM based runtimes.
IMO no. I think that if there is no interest in some possibilities (ex. @lukasredynk to list of NS incompatible runtimes you can add https://github.com/fabiand/vocker because @fabiand showed interest in this area also, so |
Nothing says that the individual runtimes can't all use CNI or even use a I really think that having a docker-native driver mode will appeal to some
If you mean docker and rocket as two, those should only count as one. They On Tue, Aug 2, 2016 at 2:01 AM, harry notifications@github.com wrote:
|
Sorry, I mean we had one (ns container), and will make it two, I have updated the comment. I can see one of your concerns: if you are arguing I can't see that this proposal is implying ns containers should know or aware about tap devices. Is there anything misleading? I can only see the core idea is: kubelet choose different workflow (method chain to call) according to its runtime option. |
Yes. In the long run, it is essential to cooperate network plugins with networking inside kubernetes, e.g. define a network object inside k8s and let plugins to do real connecting things. But currently, we could keep network plugins into runtimes, and pass in network related information in the future. |
Right now there is a single implied Network, but eventually I believe there On Tue, Aug 9, 2016 at 10:43 PM, Pengfei Ni notifications@github.com
|
We still need to pass some information to the runtime via CRI, e.g., pod CIDR, so that the runtime can setup everything properly. @freehan I assume you will be working on augmenting CRI for this. Let me know if that's not the case, thanks! |
@yujuhong I will pick that up. Thanks for summarizing. |
Note that PodCIDR is sort of a legacy artifact. We don't have a way to get On Wed, Aug 10, 2016 at 11:39 AM, Yu-Ju Hong notifications@github.com
|
Hi there, rkt developer here, Will the CRI support both kubelet and runtime invocation of CNI? On the flip side, the |
@squeed, @euank and I discussed last week on whether kubelet or rkt should setup the CNI networks,
The information returned by CRI's |
@yifan-gu I agree with you - lots of focus seems to be on the kubelet CRI driver. It's not clear to me if direction should be shifted from that. |
@yifan-gu I disagree with the first two bullet points mattering here. With the no-op driver, I think the ip returned in
I think an important point the kubelet could care about is resource isolation. The way to manage that reliably is for kubelet to manage a network namespace (similar to the pod cgroup parent for cpu/memory). This namespace can be used as a point for bandwidth QoS and network stats. Of course, the runtime could still in theory do different network setup within that namespace (make tap devices or more bridges or whatever), but it could also just reuse it if the kubelet set it up appropriately. I'm not totally convinced on either whether the kubelet or the runtime should actually be calling network plugins. If kubenet is moved into dockershim, I'm pretty confident rktshim will have an almost totally unmodified copy of that code for setup / gc / etc, so in practice some will be shared. |
My personal preference (especially given #31307) is to have Kubelet call CNI, but I don't really care either way. I just want to see whatever the decision is captured in the CRI spec. |
@euank @yifan-gu last net-sig meeting @thockin was suggesting that the network setup would all live within the runtime code, so would be under control of docker/rkt/etc instead of in kubelet. If that's still the planned direction, then I would say that the CNI driver code should be in some kind of utility packet in kubernetes git or under the kubernetes project that the runtimes could then import and use, instead of duplicating all that themselves. WRT the stuff that kubelet currently passes into the plugins, all that can be handled instead in the runtime or the plugin itself by watching the apiserver. You don't need kubelet to pass that stuff down. For example. podCIDR can be grabbed from the node object, bandwidth annotations can be grabbed from the Pod's annotations, etc. Not sure if that's desirable or if the CRI runtimes are currently expected to watch the apiserver or just use info passed from kubelet. @thockin what's the plan of record here? |
@dcbw based on @freehan's PR over here #34276 it seems like that indeed is the plan of action. I'm not sure that we wanted CRI runtimes to talk to the api-server (out of band) vs it being explicitly expressed in kubelet (in-band). I'm in favor of the latter even though it might mean fairly bulky CRI interfaces. |
IMO kubelet should be a proxy for passing this data and CRI should not access api server by self. |
+1 for moving network plugins to a separate project for reusing. |
I agree that moving the CRI->CNI logic into a lib of its own probably makes On Tue, Oct 11, 2016 at 5:18 PM, Pengfei Ni notifications@github.com
|
Right now, the node team is exploring to how to include network plugin invocation in-runtime (or shim) via the CRI API. The conclusion is that it's hard to predict if any nasty complexities (or needless leaky abstractions) will crop up without actually trying it. In designing this API, it's also worth looking over any node-affecting proposals to the networking spec. The first ones that come to mind are NetworkPolicy (#22469) and NetworkEgressPolicy (#26720). Will implementing these require a change to CRI and the corresponding implementations? If so, that results in a nasty support matrix scenario (where different runtimes support different sets of features) - something to be avoided! |
I think sharing kubenet and cni-related code might not be premature, since for the sake of backwards compatibility I assume at least docker and rkt will want to have implementations that are functionally identical to the existing code. @squeed Wouldn't that matrix exist in either scenario? The runtimes have a natural desire to be compatible so switching is painless, but they might be forced to be incompatible because of e.g. the limitations of VMs. Regardless where the logic lives, you'll have the same pair of (networkPluginConfig, runtime) which fails, and handling that failure just ends up in different places. |
To summarize the discussion: With container runtime interface, runtime/shim are responsible for pod networking. This includes set_up, tear_down, get_network_status and other background mechanism for maintaining network state (e.g. keep track of allocated IPs and hostports). Kubelet is responsible for supplying information to shim/runtime. From kubelet point of view, it pushes necessary config information (e.g. podCIDR), triggers create/delete podSandbox and expect shim/runtime to implement/maintain pod network state. Why let runtime/shim handle networking?
Some Additional points:
Plan to move forward:
|
@freehan Could you elaborate what's included in the From what I understand, today's runtimes can be categorized into two types:
We already have a solution for the namespace based runtimes, which rkt and docker are sharing the same kubelet network logic. What I want to point out is, I am not expecting a one single uniform solution, that's why plugin comes in.
I think this is solvable by IMHO, we cannot avoid such logic, it either lives in kubelet and shared by all runtimes, or implemented in all runtimes (in which case, the runtime has to clean up resources first in
I think today kubelet only pass information only once when it creates the sandbox, after that, it gets the network information from the network code, and does the clean up itself, so it't not really touching the runtime for network a lot.
This is another example that it's duplicating the shareable logic. Besides isn't it contradicting what you said above:
Since kubelet is making assumption about the shaper. To sum up, my two cents are:
|
I want to echo what @thockin has said:
Reading through all of this I see the argument for doing this to fix a bug and give more power to the runtimes. But, I don't understand why we need to do it in this release with everything else we are trying to accomplish in proving out the CRI model. We know the CRI will need to evolve overtime as @freehan said above:
Also, are there other things the kubelet allocates and might leak? For example how do we ensure that kubelet allocated cgroups don't leak? It would be great to support everything with CRI in v1.5 but I question if we can get everything 100% with net-ns based runtimes in this release. There are already plenty of challenges on the plate (tty, port-forward, etc). Is networking the only potential place that the CRI and kubelet might get out of sync on a kubelet restart? It does feel like we need to reflect on @thockin's comment about checkpointing:
What is the urgency for #34780? |
@philips, could you clarify a little bit whether you are disagreeing with the general direction of delegating the networking to the runtime, or just questioning the timeline?
kubelet checks the current state of all containers and sandboxes, and create/remove them if necessary. If runtime can faithfully report that there are still remaining resources allocated to the container/sandbox, kubelet will remove them. For example, if an infra container has been removed but the IP addressed allocated to it remains, runtime can report the podsandbox is not ready, so that kubelet can call For resources managed by kubelet directly, kubelet is responsible for ensuring that there is no leakage. |
I think there is some misunderstanding here. We are NOT doing #34780 to fix a bug. This issue was created back in July so that we could reach a consensus on how to handle networking in CRI. PR #34780 is simply implementing the that.
I think @thockin was saying that each runtime can implement its own networking logic, and if there is ever a need to factor out a common library, we can do it (but not now). I don't see how this contradicts with the decision and implementation. |
@yifan-gu I think kubelet is not going to dictate what network plugins you use, whether it be CNI or not. For example, @thockin also mentioned in #28667 (comment) that docker native driver may appeal to some users. |
To summarize, pod/container networking is delegated to the container runtime completely in CRI as of v1alpha1. Closing this issue. |
@yujuhong could you expand on this a bit? I don't really understand, since the CRI still only contains minimal networking related information (only pod CIDR at this stage). |
Right now, CRI doesn't even care about CNI or !CNI. Our implementation of dockershim probes for plugins and translates pod sandbox setup/teardown calls to CNI ADD/DEL calls. The kubelet just passes a struct of plugin settings down to the shim (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go#L509) and then, sometime later, passes down the podCIDR through an rpc (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_network.go#L221). The 6 kubelet command line args that are passed through are:
We'll probably rip these out of the kubelet alltogether when we default to a standalone shim. |
In short, kubelet doesn't care how pod networking is set up. It assumes the runtime (shim) handles all the necessary steps to set up networking for a pod. The only thing it passes to the runtime via CRI as of now is the pod CIDR, and that's only because pod CIDR is written to the node spec by a controller. Not all network plugins use the pod CIDR, so this extra bit of information may be ignored by the runtime/plugin completely. AFAIK, @freehan is working on a documentation for CRI networking. |
Proposal #17048 introduces client/server container runtime into kubelet, it supposes container's networking is processed by kubelet network plugins.
As suggested by @thockin, we should have more discussions about the role of network plugins in all of this. @thockin provides two options here:
Concerns on Option 1):
Concerns on Option 2):
CC @thockin @freehan @dchen1107 @kubernetes/sig-node @kubernetes/sig-network
The text was updated successfully, but these errors were encountered: