-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Network Drivers #9983
Comments
This looks great, thanks for working on this! Maybe this is the wrong proposal to comment on (if so, please tell me where to go), but I'm curious specifically about the driver/extension mechanism. Is there any working code that shows I'm reminded of this tweet from @shykes: https://twitter.com/solomonstre/status/542027779198701568 😄 |
There is! We are retooling to create a separate binary since we have reached our initial PoC. The PoC is here: https://github.com/shykes/docker/tree/extensions/extensions/simplebridge -- please keep in mind this is seriously alpha code and some corners were cut to accomodate our PoC. To answer the extensions question; yes, it is being worked on as a separate project. For now, we're clarifying the interfaces terminology-wise only specifically for this reason -- the other proposals are not ready yet. I hope this helps, even if it is probably not the answer you were looking for. :) |
@erikh Awesome! Thanks for sharing this. This looks like really good progress towards the extensions architecture we talked about at DockerCon Europe. I look forward to seeing the extensions proposals when they're ready. I'll leave the rest of this thread for the networking folks to feed back on. |
Will do, probably later today. Will update with a post then. On Fri, 2015-01-09 at 06:00 -0800, Dave Tucker wrote:
|
I understood that this model would allow multi-host networking. But what wasn't clear was whether docker would still be implementing multi-host networking, or if it was simply designing this driver scheme so that someone else could implement it. |
@erikh Any links to the "simplebridge" implementation (to get an idea about how it all fits together) ? |
My comments as I read it through: "network's globally unique identifier" - how global? a machine? a cluster? UUID? Is Driver.Link(foo, ...) just an alias to Driver.GetNetwork(foo).Link(...)? why are the args in a different order between them? Is Endpoint Name() the same as the netif name (e.g. ifconfig) ? The existence of Expose() as an API suggests that ports are not exposed by default. Is that a requirement or a suggestion? What does it mean to create/destroy networks in a dumb L2 model where each container gets an SRIOV interface (or the moral equivalent - macvlan, ipvlan, etc)? Is it expected that 'docker net create' reaches out into the network fabric (off-host) and configures things? How do other nodes learn about the existence of new networks? Or is the concept of a Network purely single-node-centric? Where do I specify things like "use this specific IP address" or "use this MAC address" or "set MTU to X" or "use IPv6"? What is a sandbox? Is it a net namespace? How do I allow multiple containers to join the same net namespace (as we do with --net=container:id)? How are drivers going to be implemented? The simplest answer for just trying things out would be something like an "exec driver" which looks in a directory, e.g. /var/lib/docker/net-drivers/exec. If it finds any executable file, it creates a network driver with the same name. Whenever a driver API has to be called, simply exec that file with some args indicating the API call and params. That way, we can |
@thockin has written:
I would like to realize that via a docker run network-config string that is passed to the driver. e.g. Yet I have not understood what Update
then |
Maybe I'm missing something, but re: 'Docker's new system will allow for N networks, but a 'default' network will be I don't see how consensus is maintained across a cluster. |
@timothysc |
My takeway from this proposal is that it is intended for a single user that owns multiple hosts/VMs. The VMs should have IP reachability with each other. The user deploys docker containers across multiple hosts to get the benefits of redundancy. The containers that run on these hosts should completely trust each other. Docker network daemon peer with each other to create overlay tunnels. For a multi-tenant environment (i.e people that don't trust each other, but use the same infrastructure/cloud to deploy docker), then a very simplistic workflow like this would be needed.
If the proposal in question allows extension of 'docker net' tool for different drivers/plugins, |
overall – a very good proposal! Quick thoughts and comments: Concepts:
CNM:
Workflow for a network driver:
Network abstract interface:
docker net tool:
|
I am strongly in favor of these axioms and the rest looks pretty good but it will take some time to fully understand. The fact that each container will have multiple IP addresses will ripple through service discovery since you need to use a different IP depending on where you're coming from. @shettyg There's already authentication between docker and dockerd so we should not need networking-specific auth. Multi-tenant details should mostly be left to each driver, but I think there are some common-sense solutions here; e.g. the network ID namespace is implicitly scoped per tenant and is not globally visible, but inter-tenant communication can be allowed if both sides opt in. |
@mrunalp https://github.com/shykes/docker/tree/extensions/extensions/simplebridge is most of it. the rest of the branch is set up to support it. |
@thockin we're working on a way to deal with parameters, they're just not quite ready yet. |
@thewmf
May be I am missing something, so do correct me. Assume there is a cloud-provider 'X' that wants to let its multiple customers run docker containers on its infrastructure. So 'X' needs to create a network plugin which is specific to its infrastructure. Customer1(dev) has credentials with docker but not with the infrastructure provider's network controller. You will let Customer1 create a network-X (using 'docker net create') with the network controller and start his containers on that network. Now Customer2 (QA) can come in and ask to create his containers on network-X too. You need to prevent that. What I am saying is that, the plugin should be able to get some additional CLI parameters than what is needed for the single customer case. So if 'docker net' tool will allow additional CLI commands to be passed to the plugin hiding beneath (So, 'X' will need a set of key-value pair, 'Y' may need a different set), it will help a customer specific workflow. For e.g., 'docker net' may not have a 'add-acl' cli command in its default driver. But a infrastructure provider may want to have that command for people using his infrastructure. So, his customers should be able to pass additional commands that only his plugin understands. |
Guys, just letting you know that I'm watching and preparing responses to some of the questions in this thread. I hope to have them ready tomorrow or Monday depending on availability. I'm also keeping a list of things requested so we can discuss and review them at a later point, especially to the questions I don't have answers to. |
It seems like |
I think there might be some confusion around "sandbox". Is the sandbox the netns or is it the docker container itself? The proposal mentions "iptable rules", can you clarify how and when iptables will be used? Assuming other networks want to populate the iptable rules as well, is there a way to co-ordinate or is that not an issue? |
Maybe I missed it, but what is the proposal around policies? Firewall, QOS etc |
Also, when endpoints supposed to be removed? Can endpoint exist without sandbox? |
@NetCubist, @shettyg - I am not sure why QoS/ACL need to be brought as an API. Of course that has to be handled by the driver based on the network name, or a label/network-profile. It is not something that the driver APIs need to expose unless you believe that the ACLs/QoS (tc, iptables, or OVS rules) must be communicated using docker apis. Note that I already asked earlier about ability to run the commands within netns of the container and so long as driver can do it, it can do all the operations on the netdev from driver. Feel free to correct me if I understood you wrong on this. |
@jainvipin - Is a ACL per network or QoS per network useful? Don't you want to add an ACL (or Qos or firewall or port mirroring) per endpoint? When one endpoint could be a webserver and another endpoint of the same network could be a database, I see them having different needs. May be doing it per network is good enough for majority of the use cases of the container world. If that is the case, you are right. Also I agree with you that QoS and ACL etc are just labels that the driver can interpret anyway it chooses. With the assumption that we need to associate that label per end-point (and not network), I was saying that we can pass it to the driver through a docker-network command extension. Note that based on the below code, I have been assuming that the 'add network', 'del network', 'add endpoint', 'del endpoint' etc are part of the docker-network binary and not part of docker itself. I have also been assuming that different plugins will be part of docker-network project. Something similar to what docker machine does here with digital-ocean, AWS, Azure etc and can take additional command line options. If different plugins will be independent binaries and not part of docker-network project, then my points are moot. |
@discordianfish @vieux and I are talking about how swarm and docker networking could cooperate but there's no immediate plan to share this. I will let you know as I know more. |
@discordianfish libpack intends to have a distributed implementation, but we're still figuring out if libpack is the right tool for us. |
I've added a diagram which goes into how Extensions, Drivers, Networks and Endpoints relate to each other. Please inform me all if this is insufficient. |
Hello again folks, For starters, please review the proposal again for those of you heavily invested. I have made several updates based on your comments which I hope will clarify things. There's quite a few edits so I will not make note of them here. If you were not mentioned, please, please check the proposal to see if I answered your questions. @shettyg to answer your questions, we have no intention of mandating ACL, QoS, or other multi-tenancy solutions for drivers at this time. These things will be expected to be implemented by custom drivers with custom configuration. @MalteJ I really like this flags/configuration system and am working on a way to incorporate it. Someone asked if Ports were exposed by default. Our feeling is that As for compose, machine, swarm integration I would like to point most of you at @bfirsh or @shykes for now, but realize we have no plans to directly integrate with them immediately, but we are evaluating the notion of sharing a state/discovery implementation with the swarm project. I will have more answers on discovery and shared state in a week or two. I hope this helps! Please let me know if you have additional questions. |
Collected thoughts as I read again:
On Mon, Jan 19, 2015 at 11:15 AM, Erik Hollensbe notifications@github.com
|
Is it your intention to mandate both of these in the spec? If a user desires to join a container to several networks, implemented by drivers from different providers, how will these operations be ordered?
|
This is already the way it works now, so I presumed it and overlooked it in the proposal. Sorry about that! I will correct it now.
|
Currently, Docker writes the IP address it has allocated for a container into the container's /etc/hosts file. How will this be achieved in the new model? Related: when a container is attached to several networks, which address will go first in /etc/hosts? Some programs take this as 'my' IP address. Disclosure: I work on Weave. |
@erikh, I would like to get clarity on few questions primarily from usage point of view
|
I am very interested in the networking plugin.How's it going now? |
closed by libnetwork |
@shettyg a question: thanks very much! |
This issue is almost 2 years old. If you want to discuss kubernetes and
multi-tenancy, you should do that on the kubernetes repo, where we have
many interested parties.
…On Thu, Dec 29, 2016 at 11:46 PM, JunneYang ***@***.***> wrote:
@shettyg <https://github.com/shettyg>
very good suggestion
but i think it's hard to accomplish this goal
multi-tenant need much more works
and kubernetes also doesn't support this , kubernetes simplify the network
model as well
a question:
if we comply with the norm of kubernetes, how do we achieve the goal of
multi-tenant?
thanks very much!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9983 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVMf2uwo5uuUcXC_LKe5Oaq3LFJOGks5rNLbXgaJpZM4DQEb7>
.
|
Proposal: Network Driver
THIS PROPOSAL IS A WORK IN PROGRESS
This proposal brings new functionality and new interfaces to Docker's
networking implementation. Some things are still in flux, but our goal is to
get this approved as there is a sizeable effort internally to make this happen.
We need reviewers and comments on this proposal from the community.
You can see some of the work here
but this is a work in progress, and its implementation (and the maturity of it)
should not reflect the state of this document.
Concepts
Many concepts will be used throughout this and several proposals that are in
process. You may see these repeated with slightly different verbiage or may
contain at-length descriptions of how these interfaces will be implemented.
Here's a basic diagram about how the networking itself operates:
The following includes a description of the interface and its name. The
sub-bullets provide practical examples of these components with the existing
implementation of Docker as an example.
requested.
functionality to Docker.
implementation, but we are also evaluating
ecc and others.
Container Network Model (or CNM)
The container network model is a few axioms about how docker wishes to supply
interoperation between networks and containers.
should be supported by all drivers.
multiple networks.
This has a few consequences:
allows discovery to be scoped and more flexible for external implementations.
Implementation of this is still TBD.
Notion of a Default Network
Since Docker is a tool heavily used by both operations and development
personnel to different goals respective to their skill sets, it is critical to
have a functioning "out of the box" implementation for people to use with ease.
Docker will create a "default" network (named
default
) for the use of basicnetworking.
Networks and Endpoints
Endpoints are a part of a Network. The network is (at least in the simplebridge implementation) isolated, but drivers may implement the notion of a network however they choose.
Docker's new system will allow for N networks, but a 'default' network will be
created as a convenience for users, and with the default driver it will
function similarly to the existing network solution now.
Multiple endpoints can be created for a single container, and bound to them at
the same time. The endpoints may live on different networks and may all belong
to one container.
Again, Endpoints as a part of different networks should not be able to communicate with each other in the default implementation. It's expected that network operators would program any bridging between two networks.
Workflow for a Network Driver
At boot, a network driver will be given a replay of its state; this will allow
the driver to return to being in sync with the state of the system, to create
new networks, etc. How replays are handled by drivers is intentionally
undefined.
A network can be requested to be created. In the workflow below, the network
assumes to be created already.
A network driver will be asked at
docker run
(in order, for a given network):The driver will also provide port mapping/expose functionality (see below for
API) and communicate with service discovery (TBD).
Network API
Driver abstract interface:
Network abstract interface:
(note that Link and Unlink here are merely for convenience and do not require
an alternative implementation)
Endpoint abstract interface:
docker net
tooldocker net
is the vehicle we're pushing to manipulate networks. Thebasic commands are described below:
create
: create a networkdestroy
: destroy a networkjoin
: join a container to a networkleave
: remove a container from a networkThere will be a forthcoming UI extension to
docker run
which also selects anetwork at run time. This interface is currently to be determined.
Our initial implementation
Our implementation is very similar to docker's existing implementation, which
we are dubbing
simplebridge
. This driver creates a bridge for each network,and a veth pair for each endpoint. Networks may contain a set of vxlan peers
which will be attached to the bridge to ensure network connectivity to get
multi-host links.
The text was updated successfully, but these errors were encountered: