Having multiple NICs (network interfaces) on a Docker image before it boots

[MarkNijhof]

I have a Docker image with ubuntu:latest and a few other dependencies. The script it starts automatically expects 2 NIC's eth0 (which is there by default) and eth1. Because this second NIC isn't there the script crashes and the container is stopped. So using Pipeworks doesn't work as the container isn't there anymore.

I tried adding this to the Dockerfile:

RUN echo "auto eth1" >> /etc/network/interfaces
RUN echo "iface eth1 inet dhcp" >> /etc/network/interfaces
But that didn't work either.

Is there a proper way to achieve this, or else a hack to begin with :-)

Maybe later having something like "NIC eth1 dhcp" in the Dockerfile would be cool.

-Mark

[jpetazzo]

I think I have a way to achieve that with pipework; I'll give it a try.

[MarkNijhof]

That would be awesome, I tried but couldn't get pipework to work during
setup/boot. But then again I don't know what I am doing :-)
On Sep 11, 2013 12:15 AM, "Jérôme Petazzoni" notifications@github.com
wrote:

I think I have a way to achieve that with pipework; I'll give it a try.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24199187
.

[MarkNijhof]

I know I am being impatient, but I was wondering if you got anywhere with this?

[jpetazzo]

This still needs a bit of work to be functional; tentatively scheduling it
for this week.

On Sat, Sep 14, 2013 at 2:46 PM, Mark Nijhof notifications@github.comwrote:

I know I am being impatient, but I was wondering if you got anywhere with
this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24459862
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

[jpetazzo]

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].

Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

[shykes]

Using a script like pipeworks to do things that break the portability of containers (like adding a dependency in your app on dhcp or the presence of 2 NICs) should be a last resort when you really can't find a way to stay within the recommended use of docker.

​

May I ask what your application does and why it needs 2 NICs?

​

​I'm also curious why you need dhcp for your application.

​

@jpetazzo please don't forget to ask these questions and recommend your script as a last resort with a healthy warning on the downsides. Otherwise we will never get the user feedback we need and your custom script will always be necessary, which is bad for Docker.

—
@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 12:00 PM, Jérôme Petazzoni
notifications@github.com wrote:

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].
Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

Reply to this email directly or view it on GitHub:
#1824 (comment)

[solomonstre]

That said I'm curious if we need to add support for multi-NICs in the docker runtime. If there's interest we can start a discussion on the mailing list.

—
@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 1:11 PM, Solomon Hykes notifications@github.com
wrote:

Using a script like pipeworks to do things that break the portability of containers (like adding a dependency in your app on dhcp or the presence of 2 NICs) should be a last resort when you really can't find a way to stay within the recommended use of docker.
​
May I ask what your application does and why it needs 2 NICs?
​
​I'm also curious why you need dhcp for your application.
​
@jpetazzo please don't forget to ask these questions and recommend your script as a last resort with a healthy warning on the downsides. Otherwise we will never get the user feedback we need and your custom script will always be necessary, which is bad for Docker.
—
@solomonstre
@getdocker
On Sat, Sep 21, 2013 at 12:00 PM, Jérôme Petazzoni
notifications@github.com wrote:

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].
Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

Reply to this email directly or view it on GitHub:

#1824 (comment)

Reply to this email directly or view it on GitHub:
#1824 (comment)

[MarkNijhof]

Hi @jpetazzo thank you so much, I will test it tomorrow, but from the sound of it it would!

@solomonstre I think you can say this is a last resort, I have asked on StackOverflow and on the Docker google group, I pointed to the SO post on Twitter and finally I opened a ticket @jpetazzo was the only one replying. So I think the Docker community should have heard about it.

The use case is: Our client has two nic in their machines, one to manage the machine, reboot, install a new version of the software ect. And one nic that is used to do the actual work, this one is also externally visible. Our client works in the broadcast industry wants to create an extra layer of security by completely separating these two networks. Our software checks the nic's on startup and if they are not there it crashes (Erlang) thus the container never comes up, so we couldn't use the normal pipework's way of adding nic's.

The machines are currently Ubuntu machines and upgrading specific parts of the setup is a pain, we would love to run Docker on these machines to help ease the pain. Currently we are talking about 30 machines, but this will be tripled in the future.

I am happy to provide more details, and help test if this becomes part of Docker. Before @jpetazzo solution this was a show stopper for using Docker for this specific scenario.

-Mark

[MarkNijhof]

Sorry dhcp is because these machines will be started and stopped when needed, when they start they just get a free ip address assigned, then they tell the manager they are there, they get configured with the proper ip addresses and software version and reboot. So dhcp on these nic's will prevent duplicate ip's in the network.

[jpetazzo]

Solomon is right; I added a big fat warning to the Pipework documentation
to remind people that they should first contact docker-user.
Also, recently, I pointed many people at Pipework not for its features, but
to show them how to retrieve network metrics... I think I will split that
feature, or write an article explaining how to do that, outside of the
context of Pipework, to reduce the confusion.

Mark, I'll see if I can add DHCP support; that way, you won't have to run
DHCP within your containers (which would be a problem because you would
have different parameters for dev and prod containers). Do your DHCP server
require periodic renewal of the lease, or would it be fine to do a one-shot
negotiation?

On Sat, Sep 21, 2013 at 1:37 PM, Mark Nijhof notifications@github.comwrote:

Sorry dhcp is because these machines will be started and stopped when
needed, when they start they just get a free ip address assigned, then they
tell the manager they are there, they get configured with the proper ip
addresses and software version and reboot. So dhcp on these nic's will
prevent duplicate ip's in the network.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24870151
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

[solomonstre]

Thanks @jpetazzo. To clarify, I think pipeworks is super useful and please continue to point to it when it can help - but let's also gather all the information we can about which parts could be merged in upstream docker.

​

For example would a dhcp plugin make sense to keep it out of the containers? It would be pretty easy to implement. 

—
@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 8:25 PM, Jérôme Petazzoni
notifications@github.com wrote:

Solomon is right; I added a big fat warning to the Pipework documentation
to remind people that they should first contact docker-user.
Also, recently, I pointed many people at Pipework not for its features, but
to show them how to retrieve network metrics... I think I will split that
feature, or write an article explaining how to do that, outside of the
context of Pipework, to reduce the confusion.
Mark, I'll see if I can add DHCP support; that way, you won't have to run
DHCP within your containers (which would be a problem because you would
have different parameters for dev and prod containers). Do your DHCP server
require periodic renewal of the lease, or would it be fine to do a one-shot
negotiation?
On Sat, Sep 21, 2013 at 1:37 PM, Mark Nijhof notifications@github.comwrote:

Sorry dhcp is because these machines will be started and stopped when
needed, when they start they just get a free ip address assigned, then they
tell the manager they are there, they get configured with the proper ip
addresses and software version and reboot. So dhcp on these nic's will
prevent duplicate ip's in the network.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24870151
.

@jpetazzo https://twitter.com/jpetazzo

Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

Reply to this email directly or view it on GitHub:
#1824 (comment)

[MarkNijhof]

Cheers @jpetazzo I think I am happy with the default setting for DHCP, in our case the machine only keeps the DHCP assigned address for maybe 10 minutes until someone has assigned it a new role after which it gets a new IP assigned (not from the DHCP).

@solomonstre I didn't mean to sound mean, but pipework is solving a blocking issue for us to use Docker, I would love to have these things directly in Docker, but if that isn't possible due to time constraints or other reasons then I am happy. So I am not complaining at all.

I would create a pull request if I thought I could implement it myself in Docker, but I am not sure I would be doing you a favor :-)

I just want to be able to use Docker :-)

[jpetazzo]

Hi Mark.

I added DHCP support: https://github.com/jpetazzo/pipework#dhcp

Let me know if that helps.

On Sun, Sep 22, 2013 at 12:53 PM, Mark Nijhof notifications@github.comwrote:

Cheers @jpetazzo https://github.com/jpetazzo I think I am happy with
the default setting for DHCP, in our case the machine only keeps the DHCP
assigned address for maybe 10 minutes until someone has assigned it a new
role after which it gets a new IP assigned (not from the DHCP).

@solomonstre https://github.com/solomonstre I didn't mean to sound
mean, but pipework is solving a blocking issue for us to use Docker, I
would love to have these things directly in Docker, but if that isn't
possible due to time constraints or other reasons then I am happy. So I am
not complaining at all.

I would create a pull request if I thought I could implement it myself in
Docker, but I am not sure I would be doing you a favor :-)

I just want to be able to use Docker :-)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24889180
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

[crosbymichael]

@jpetazzo It sounds like it is about time to integrate pipework into docker. What do you think?

[jamtur01]

+1.

[jpetazzo]

Milos Gajdos (I don't remember his github ID so I e-mailed him separetely) said he wanted to collaborate on a rewrite of Pipework to Go.
That might be a good first step. Do we want to design further design etc. here, or on docker-dev?

[milosgajdos]

Hi Guys, yeah I'll start looking into this stuff at some point today. I'm quite keen on the Pipework Go rewrite. I'll keep you in the loop!

[bodenr]

+1 for multi-nic binding at container boot time... my container has a requirement for a public/private network on separate eths... based on what my google-foo has produced thus far it appears I'm off to pipeworks. i see the port binding via run -p is becoming more robust, but still doesn't seem to suffice for more complex container requirements.

[amirkdv]

👍

[PerilousApricot]

👍 For multiple nics with docker

[dsaard]

👍 For multiple nics with docker

[seanchann]

For multiple nics with docker

[alinz]

For multiple nics with docker

[hibooboo2]

👍 for multiple nics as well. Something rancher would love to have.

[xaionaro]

It's required to use bountysource for this issue, IMHO :)

[jorgemoralespou]

👍 For multiple nics with docker

[amalagaura]

This is necessary because if doing a host network we cannot link to other containers. We need to be on the host network and at the same time link to other containers.

[drbokko]

👍 or +1

[vijayendrabvs]

Has this work been completed? Any pointers to how to plumb multiple NICs inside docker containers?

[cpuguy83]

@vijayendrabvs close, but not quite yet. #13441
Also this will (probably) never be supported for build time since there would be no need to

[cpuguy83]

Closing as #14023 was merged, which provides an API for managing networks and #14051 for discovery

[tangicolin]

It is possible to have some tutorial or pointer about how to set this features ?
I understand that docker network command can create command but i don't understand how to "bind" docker instance to new network (and how to create second NIC).

[kujiy]

+1 For multiple nics with docker

I red the docker network docs then thought the same as @tangicolin .

Should we still use jpetazzo/pipework ?

[mavenugo]

@tangicolin @kujiy please refer to https://github.com/docker/docker/blob/master/docs/userguide/networking/work-with-networks.md.
one can create multiple networks (using bridge or any other driver) and connect a container to multiple networks. As the container is connected to these networks, it will automatically create the necessary interfaces inside the container. In order to have multiple interface to be available before the container starts, our recommendation is to

1. create multiple networks using docker network create command
2. create a container in a particular network using --net : docker create --net=network1 --name=foo ...
3. connect the container to other network : docker network connect network2 foo
4. start the container docker start foo

This will start the container connected to multiple networks with multiple interfaces in the container on each of these networks. Will that satisfy your requirement ?

[nvachhar]

I just gave this a try on docker 1.9.1 and it doesn't seem to work. The network connect on the not-yet-running container fails. Here's the error I get:

$ docker -v
Docker version 1.9.1, build a34a1d5
$ docker network connect mgmt test
Error response from daemon: Container 159304e33634fe4a555b1fa8cb81690de09f0d7ecf2fe7450fc530c8526baf14 is not running

If I launch the container, and then connect, it works fine.

$ docker start test
test
$ docker network connect mgmt test
$ docker inspect test | awk '/Networks/,EOF{print}'
        "Networks": {
            "bridge": {
                "EndpointID": "fa80ff1d2aac2b872736df15a77bc913b6718fd9a8ae782730e052bd6e6cde0b",
                "Gateway": "172.17.42.1",
                "IPAddress": "172.17.0.1",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:01"
            },
            "mgmt": {
                "EndpointID": "c5f013d74e8e3f1a057a218d808ae87349ef0aff2980fa11e541eddaa5144375",
                "Gateway": "172.18.0.1",
                "IPAddress": "172.18.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:12:00:02"
            }
        }
    }
}

Is there anyway to do the connect before the container is running?

[kujiy]

@mavenugo @nvachhar Thank you guys, it seems what I want. I'll try it later.

[thaJeztah]

@nvachhar see this PR, which is scheduled for docker 1.10; #18906