New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Having multiple NICs (network interfaces) on a Docker image before it boots #1824

Closed
MarkNijhof opened this Issue Sep 7, 2013 · 37 comments

Comments

Projects
None yet
@MarkNijhof

MarkNijhof commented Sep 7, 2013

I have a Docker image with ubuntu:latest and a few other dependencies. The script it starts automatically expects 2 NIC's eth0 (which is there by default) and eth1. Because this second NIC isn't there the script crashes and the container is stopped. So using Pipeworks doesn't work as the container isn't there anymore.

I tried adding this to the Dockerfile:

RUN echo "auto eth1" >> /etc/network/interfaces
RUN echo "iface eth1 inet dhcp" >> /etc/network/interfaces
But that didn't work either.

Is there a proper way to achieve this, or else a hack to begin with :-)

Maybe later having something like "NIC eth1 dhcp" in the Dockerfile would be cool.

-Mark

@ghost ghost assigned jpetazzo Sep 10, 2013

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Sep 10, 2013

I think I have a way to achieve that with pipework; I'll give it a try.

@MarkNijhof

This comment has been minimized.

MarkNijhof commented Sep 10, 2013

That would be awesome, I tried but couldn't get pipework to work during
setup/boot. But then again I don't know what I am doing :-)
On Sep 11, 2013 12:15 AM, "Jérôme Petazzoni" notifications@github.com
wrote:

I think I have a way to achieve that with pipework; I'll give it a try.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24199187
.

@MarkNijhof

This comment has been minimized.

MarkNijhof commented Sep 14, 2013

I know I am being impatient, but I was wondering if you got anywhere with this?

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Sep 15, 2013

This still needs a bit of work to be functional; tentatively scheduling it
for this week.

On Sat, Sep 14, 2013 at 2:46 PM, Mark Nijhof notifications@github.comwrote:

I know I am being impatient, but I was wondering if you got anywhere with
this?


Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24459862
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Sep 21, 2013

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].

Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

@shykes

This comment has been minimized.

Collaborator

shykes commented Sep 21, 2013

Using a script like pipeworks to do things that break the portability of containers (like adding a dependency in your app on dhcp or the presence of 2 NICs) should be a last resort when you really can't find a way to stay within the recommended use of docker.

May I ask what your application does and why it needs 2 NICs?

​I'm also curious why you need dhcp for your application.

@jpetazzo please don't forget to ask these questions and recommend your script as a last resort with a healthy warning on the downsides. Otherwise we will never get the user feedback we need and your custom script will always be necessary, which is bad for Docker.


@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 12:00 PM, Jérôme Petazzoni
notifications@github.com wrote:

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].
Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

Reply to this email directly or view it on GitHub:
#1824 (comment)

@solomonstre

This comment has been minimized.

solomonstre commented Sep 21, 2013

That said I'm curious if we need to add support for multi-NICs in the docker runtime. If there's interest we can start a discussion on the mailing list.


@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 1:11 PM, Solomon Hykes notifications@github.com
wrote:

Using a script like pipeworks to do things that break the portability of containers (like adding a dependency in your app on dhcp or the presence of 2 NICs) should be a last resort when you really can't find a way to stay within the recommended use of docker.

May I ask what your application does and why it needs 2 NICs?

​I'm also curious why you need dhcp for your application.

@jpetazzo please don't forget to ask these questions and recommend your script as a last resort with a healthy warning on the downsides. Otherwise we will never get the user feedback we need and your custom script will always be necessary, which is bad for Docker.

@solomonstre
@getdocker
On Sat, Sep 21, 2013 at 12:00 PM, Jérôme Petazzoni
notifications@github.com wrote:

Your wish has been granted: you can now run "pipework --wait" within a
container, and it will wait until the network interface is ready before
continuing[1].
Does this solve your problem?

[1] https://github.com/jpetazzo/pipework#wait-for-the-network-to-be-ready

Reply to this email directly or view it on GitHub:

#1824 (comment)

Reply to this email directly or view it on GitHub:
#1824 (comment)

@MarkNijhof

This comment has been minimized.

MarkNijhof commented Sep 21, 2013

Hi @jpetazzo thank you so much, I will test it tomorrow, but from the sound of it it would!

@solomonstre I think you can say this is a last resort, I have asked on StackOverflow and on the Docker google group, I pointed to the SO post on Twitter and finally I opened a ticket @jpetazzo was the only one replying. So I think the Docker community should have heard about it.

The use case is: Our client has two nic in their machines, one to manage the machine, reboot, install a new version of the software ect. And one nic that is used to do the actual work, this one is also externally visible. Our client works in the broadcast industry wants to create an extra layer of security by completely separating these two networks. Our software checks the nic's on startup and if they are not there it crashes (Erlang) thus the container never comes up, so we couldn't use the normal pipework's way of adding nic's.

The machines are currently Ubuntu machines and upgrading specific parts of the setup is a pain, we would love to run Docker on these machines to help ease the pain. Currently we are talking about 30 machines, but this will be tripled in the future.

I am happy to provide more details, and help test if this becomes part of Docker. Before @jpetazzo solution this was a show stopper for using Docker for this specific scenario.

-Mark

@MarkNijhof

This comment has been minimized.

MarkNijhof commented Sep 21, 2013

Sorry dhcp is because these machines will be started and stopped when needed, when they start they just get a free ip address assigned, then they tell the manager they are there, they get configured with the proper ip addresses and software version and reboot. So dhcp on these nic's will prevent duplicate ip's in the network.

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Sep 22, 2013

Solomon is right; I added a big fat warning to the Pipework documentation
to remind people that they should first contact docker-user.
Also, recently, I pointed many people at Pipework not for its features, but
to show them how to retrieve network metrics... I think I will split that
feature, or write an article explaining how to do that, outside of the
context of Pipework, to reduce the confusion.

Mark, I'll see if I can add DHCP support; that way, you won't have to run
DHCP within your containers (which would be a problem because you would
have different parameters for dev and prod containers). Do your DHCP server
require periodic renewal of the lease, or would it be fine to do a one-shot
negotiation?

On Sat, Sep 21, 2013 at 1:37 PM, Mark Nijhof notifications@github.comwrote:

Sorry dhcp is because these machines will be started and stopped when
needed, when they start they just get a free ip address assigned, then they
tell the manager they are there, they get configured with the proper ip
addresses and software version and reboot. So dhcp on these nic's will
prevent duplicate ip's in the network.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24870151
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

@solomonstre

This comment has been minimized.

solomonstre commented Sep 22, 2013

Thanks @jpetazzo. To clarify, I think pipeworks is super useful and please continue to point to it when it can help - but let's also gather all the information we can about which parts could be merged in upstream docker.

For example would a dhcp plugin make sense to keep it out of the containers? It would be pretty easy to implement. 


@solomonstre
@getdocker

On Sat, Sep 21, 2013 at 8:25 PM, Jérôme Petazzoni
notifications@github.com wrote:

Solomon is right; I added a big fat warning to the Pipework documentation
to remind people that they should first contact docker-user.
Also, recently, I pointed many people at Pipework not for its features, but
to show them how to retrieve network metrics... I think I will split that
feature, or write an article explaining how to do that, outside of the
context of Pipework, to reduce the confusion.
Mark, I'll see if I can add DHCP support; that way, you won't have to run
DHCP within your containers (which would be a problem because you would
have different parameters for dev and prod containers). Do your DHCP server
require periodic renewal of the lease, or would it be fine to do a one-shot
negotiation?
On Sat, Sep 21, 2013 at 1:37 PM, Mark Nijhof notifications@github.comwrote:

Sorry dhcp is because these machines will be started and stopped when
needed, when they start they just get a free ip address assigned, then they
tell the manager they are there, they get configured with the proper ip
addresses and software version and reboot. So dhcp on these nic's will
prevent duplicate ip's in the network.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24870151
.

@jpetazzo https://twitter.com/jpetazzo

Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

Reply to this email directly or view it on GitHub:
#1824 (comment)

@MarkNijhof

This comment has been minimized.

MarkNijhof commented Sep 22, 2013

Cheers @jpetazzo I think I am happy with the default setting for DHCP, in our case the machine only keeps the DHCP assigned address for maybe 10 minutes until someone has assigned it a new role after which it gets a new IP assigned (not from the DHCP).

@solomonstre I didn't mean to sound mean, but pipework is solving a blocking issue for us to use Docker, I would love to have these things directly in Docker, but if that isn't possible due to time constraints or other reasons then I am happy. So I am not complaining at all.

I would create a pull request if I thought I could implement it myself in Docker, but I am not sure I would be doing you a favor :-)

I just want to be able to use Docker :-)

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Sep 23, 2013

Hi Mark.

I added DHCP support: https://github.com/jpetazzo/pipework#dhcp

Let me know if that helps.

On Sun, Sep 22, 2013 at 12:53 PM, Mark Nijhof notifications@github.comwrote:

Cheers @jpetazzo https://github.com/jpetazzo I think I am happy with
the default setting for DHCP, in our case the machine only keeps the DHCP
assigned address for maybe 10 minutes until someone has assigned it a new
role after which it gets a new IP assigned (not from the DHCP).

@solomonstre https://github.com/solomonstre I didn't mean to sound
mean, but pipework is solving a blocking issue for us to use Docker, I
would love to have these things directly in Docker, but if that isn't
possible due to time constraints or other reasons then I am happy. So I am
not complaining at all.

I would create a pull request if I thought I could implement it myself in
Docker, but I am not sure I would be doing you a favor :-)

I just want to be able to use Docker :-)


Reply to this email directly or view it on GitHubhttps://github.com//issues/1824#issuecomment-24889180
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

@crosbymichael

This comment has been minimized.

Member

crosbymichael commented Nov 27, 2013

@jpetazzo It sounds like it is about time to integrate pipework into docker. What do you think?

@jamtur01

This comment has been minimized.

Contributor

jamtur01 commented Nov 27, 2013

+1.

@jpetazzo

This comment has been minimized.

Contributor

jpetazzo commented Dec 2, 2013

Milos Gajdos (I don't remember his github ID so I e-mailed him separetely) said he wanted to collaborate on a rewrite of Pipework to Go.
That might be a good first step. Do we want to design further design etc. here, or on docker-dev?

@milosgajdos83

This comment has been minimized.

milosgajdos83 commented Dec 2, 2013

Hi Guys, yeah I'll start looking into this stuff at some point today. I'm quite keen on the Pipework Go rewrite. I'll keep you in the loop!

@bodenr

This comment has been minimized.

bodenr commented Dec 13, 2013

+1 for multi-nic binding at container boot time... my container has a requirement for a public/private network on separate eths... based on what my google-foo has produced thus far it appears I'm off to pipeworks. i see the port binding via run -p is becoming more robust, but still doesn't seem to suffice for more complex container requirements.

@jpetazzo jpetazzo removed their assignment Apr 12, 2014

@amirkdv

This comment has been minimized.

amirkdv commented May 16, 2014

👍

@unclejack unclejack changed the title from Having two NIC's on a Docker image before it boots to Having multiple NICs (network interfaces) on a Docker image before it boots Jul 16, 2014

@PerilousApricot

This comment has been minimized.

PerilousApricot commented Oct 21, 2014

👍 For multiple nics with docker

1 similar comment
@dsaard

This comment has been minimized.

dsaard commented Nov 7, 2014

👍 For multiple nics with docker

@seanchann

This comment has been minimized.

seanchann commented Dec 22, 2014

For multiple nics with docker

1 similar comment
@alinz

This comment has been minimized.

alinz commented Jan 21, 2015

For multiple nics with docker

@hibooboo2

This comment has been minimized.

hibooboo2 commented Jan 24, 2015

👍 for multiple nics as well. Something rancher would love to have.

@xaionaro

This comment has been minimized.

xaionaro commented Jan 26, 2015

It's required to use bountysource for this issue, IMHO :)

@jorgemoralespou

This comment has been minimized.

jorgemoralespou commented Jan 28, 2015

👍 For multiple nics with docker

@amalagaura

This comment has been minimized.

amalagaura commented Feb 17, 2015

This is necessary because if doing a host network we cannot link to other containers. We need to be on the host network and at the same time link to other containers.

@drbokko

This comment has been minimized.

drbokko commented Apr 9, 2015

👍 or +1

@icecrime icecrime removed the roadmap label Apr 22, 2015

@vijayendrabvs

This comment has been minimized.

vijayendrabvs commented May 26, 2015

Has this work been completed? Any pointers to how to plumb multiple NICs inside docker containers?

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented May 26, 2015

@vijayendrabvs close, but not quite yet. #13441
Also this will (probably) never be supported for build time since there would be no need to

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jul 7, 2015

Closing as #14023 was merged, which provides an API for managing networks and #14051 for discovery

@cpuguy83 cpuguy83 closed this Jul 7, 2015

@tangicolin

This comment has been minimized.

Contributor

tangicolin commented Sep 24, 2015

It is possible to have some tutorial or pointer about how to set this features ?
I understand that docker network command can create command but i don't understand how to "bind" docker instance to new network (and how to create second NIC).

@kujiy

This comment has been minimized.

kujiy commented Jan 7, 2016

+1 For multiple nics with docker

I red the docker network docs then thought the same as @tangicolin .

Should we still use jpetazzo/pipework ?

@mavenugo

This comment has been minimized.

Contributor

mavenugo commented Jan 7, 2016

@tangicolin @kujiy please refer to https://github.com/docker/docker/blob/master/docs/userguide/networking/work-with-networks.md.
one can create multiple networks (using bridge or any other driver) and connect a container to multiple networks. As the container is connected to these networks, it will automatically create the necessary interfaces inside the container. In order to have multiple interface to be available before the container starts, our recommendation is to

  1. create multiple networks using docker network create command
  2. create a container in a particular network using --net : docker create --net=network1 --name=foo ...
  3. connect the container to other network : docker network connect network2 foo
  4. start the container docker start foo

This will start the container connected to multiple networks with multiple interfaces in the container on each of these networks. Will that satisfy your requirement ?

@nvachhar

This comment has been minimized.

nvachhar commented Jan 7, 2016

I just gave this a try on docker 1.9.1 and it doesn't seem to work. The network connect on the not-yet-running container fails. Here's the error I get:

$ docker -v
Docker version 1.9.1, build a34a1d5
$ docker network connect mgmt test
Error response from daemon: Container 159304e33634fe4a555b1fa8cb81690de09f0d7ecf2fe7450fc530c8526baf14 is not running

If I launch the container, and then connect, it works fine.

$ docker start test
test
$ docker network connect mgmt test
$ docker inspect test | awk '/Networks/,EOF{print}'
        "Networks": {
            "bridge": {
                "EndpointID": "fa80ff1d2aac2b872736df15a77bc913b6718fd9a8ae782730e052bd6e6cde0b",
                "Gateway": "172.17.42.1",
                "IPAddress": "172.17.0.1",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:01"
            },
            "mgmt": {
                "EndpointID": "c5f013d74e8e3f1a057a218d808ae87349ef0aff2980fa11e541eddaa5144375",
                "Gateway": "172.18.0.1",
                "IPAddress": "172.18.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:12:00:02"
            }
        }
    }
}

Is there anyway to do the connect before the container is running?

@kujiy

This comment has been minimized.

kujiy commented Jan 8, 2016

@mavenugo @nvachhar Thank you guys, it seems what I want. I'll try it later.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 8, 2016

@nvachhar see this PR, which is scheduled for docker 1.10; #18906

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment