Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unclear how to assign ip address for docker container #209

Closed
macb opened this Issue Oct 3, 2015 · 17 comments

Comments

Projects
None yet
9 participants
@macb
Copy link

commented Oct 3, 2015

I'd like to launch containers and have docker forward all interfaces to the container for the dynamic port. It doesn't seem like that is currently supported as ip isn't exposed in the network config and whatever is configuring it is defaulting to the 10.20.x.x network which is reachable from the host but no where else.

I could easily be missing something though.

Nomad uses port binding to expose services running in containers using the port space on the host's interface. For example, Nomad host running on 1.2.3.4 may allocate port 22333 to a task, so you would access that service via 1.2.3.4:22333.

root@nomad-client-tor1-0:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                         NAMES
a9a8f68d8fd1        macb/repeater       "/bin/sh -c 'socat -v"   11 hours ago        Up 11 hours         10.20.0.166:21470->9000/tcp   drunk_mahavira
8ba64f31bd7f        macb/repeater       "/bin/sh -c 'socat -v"   11 hours ago        Up 11 hours         10.20.0.166:59453->9000/tcp   jolly_franklin
@achanda

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2015

docker selects a IP range to be used on docker0 (in bridge mode). All containers are allocated IPs from that range. Nomad does not provide a way to configure that. You can try to:

  1. Assign a IP to docker0. This can be done by starting the daemon with a --bip parameter as described in https://docs.docker.com/articles/networking/#docker0 This is independent of nomad.
  2. Use host networking mode. This can be done by passing network_mode=host in the configuration for a nomad task.

Please let me know if I misunderstood.

@ghost

This comment has been minimized.

Copy link

commented Oct 4, 2015

You can also start docker in non-masquerading mode, turn on routing in the kernel, and then route traffic through the host IP. You'll run into problems if you have more than one docker host however as you'll have to configure each hosts bridge interface to have a different docker0 subnet. It's doable but ugly to scale and you're pretty much doing an end run around nomad for all your networking.

@macb

This comment has been minimized.

Copy link
Author

commented Oct 4, 2015

Hmm, I'm not sure if we're all talking about the same thing. I'm not really interested in assigning an IP to the container. I was just want port exposed on all interfaces (or at least the node's public ip) instead of only on the container private IP. I think my original post was misleading.

In bridged mode if you start a container like:

root@nomad-client-tor1-4:~# docker run -d -p 1234:9000 macb/repeater
b3506717030c8fa92d4a31ce3c6bd21a3193d80cddb0f7cc4a29ff882bfc14ce
root@nomad-client-tor1-4:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                         NAMES
b3506717030c        macb/repeater       "/bin/sh -c 'socat -v"   4 seconds ago       Up 3 seconds        0.0.0.0:1234->9000/tcp        naughty_galileo
6bbdf4988653        macb/repeater       "/bin/sh -c 'socat -v"   26 hours ago        Up 26 hours         10.20.0.167:32755->9000/tcp   naughty_curie

Then it's available both on localhost as well as via the server's IP:

root@nomad-client-tor1-4:~# nc 127.0.0.1 1234
hi
hi

It appears that when nomad starts the container it finds an IP to apply to it and sets the -p flag as ip:port:port. This is seen in nomad's logs (if log_level="DEBUG" mode is enabled) here.

@ghost

This comment has been minimized.

Copy link

commented Oct 5, 2015

My understanding is you would have to assign dynamic_ports = ["9000"], this will pass a NOMAD_PORT_9000 environment variable through to the container when it's started. This port will be a random port in the service ports range, nomad is expecting the container to register this port in service discovery somewhere (consul or etcd). You can see which one it is by doing docker ps or docker inspect on the container.

As far as I can tell there is no way to set a 1:1 mapping of, "I want docker to expose this container port as this docker host port," through nomad.

I'm having issues with this working inconsistently in ticket #212 but it seems to work properly if you set the port to ["6379"] instead of ["redis"] as it is in the 'nomad init' example.

@macb

This comment has been minimized.

Copy link
Author

commented Oct 5, 2015

This is the job that's being run which has worked well for starting the container and having it map a dynamic port to the port in the container. However it also applies an IP address when I'd rather it use 0.0.0.0 so it's only reachable from the host. nc 10.20.0.167 32755 instead of nc 127.0.0.1 32755 in the docker ps from #209 (comment).

@cbednarski

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2015

@macb Thanks for the report. We are working out the kinks in networking (some bug fixes have landed in master already), and binding to 127.0.0.1 is not intended.

We bind a task to one interface (ip address), potentially with multiple ports on that IP. The scheduler allows you some control over this so you can select a private or public subnet. Our design does not currently support binding to 0.0.0.0 to bind to all interfaces.

@skozin

This comment has been minimized.

Copy link

commented Nov 27, 2015

@cbednarski, could you please tell me how can I select private or public subnet for a specific Docker task? I was unable to find this in the docs and thought it is not possible yet.

@diptanu

This comment has been minimized.

Copy link
Collaborator

commented Nov 28, 2015

@skozin We don't support selecting one of many interfaces in Nomad yet. Support for doing that is going to come fairly soon.

@skozin

This comment has been minimized.

Copy link

commented Nov 28, 2015

@diptanu Thanks for the info. That's the only thing that really prevents us from using Nomad.

We have a cluster of machines, and some of these machines have both public and private interfaces. These two-nic machines run internal services, that need to listen only on private interface, as well as public services, which should listen on public interface.

I really hope that it will be available soon. Currently we're upgrading our infrastructure and we'd really like to give Nomad a try, because it looks very promising.

The ability to listen on all interfaces is also a desirable feature, although not as critical for us as the ability to select one specific interface.

@dontrebootme

This comment has been minimized.

Copy link

commented Dec 9, 2015

I think the point of this thread that @macb is trying to bring up is that Nomad is incorrectly passing an IP with the dynamic port association to Docker and that IP in some cases is wrong (eth0 vs eth1). You can see that Nomad is passing not only the dynamic port, but also an IP when you do a docker inspect.

Perhaps we need more documentation on how to control NOMAD_IP, I would expect the docker port mapping to match the same address as the bind_addr specified in the nomad client config.

@dontrebootme

This comment has been minimized.

Copy link

commented Dec 9, 2015

For example, even when bind and advertise are set to a specific IP, the command sent to docker for the port mapping is using another IP.

vagrant@docker1:~$ cat /etc/nomad/nomad.json
{
  "advertise_addr": "10.7.0.21",
  "bind_addr": "10.7.0.21",
  "data_dir": "/var/lib/nomad",
  "datacenter": "nomad-intro",
  "enable_syslog": true,
  "syslog_facility": "LOCAL0",
  "log_level": "INFO",
  "client": {
    "enabled": true,
    "servers": [ "10.7.0.10:4647" ]
  }
}
vagrant@docker1:~$ docker ps -a
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                              NAMES
7099eebb018f        dontrebootme/microbot:v1   "/bin/sh -c /start_ng"   15 seconds ago      Up 14 seconds       10.0.2.15:50793->80/tcp, 10.0.2.15:50793->80/udp   microbot-4361f96b-fb84-4d25-41d9-d29f2c88d12f
ded4fb62469d        dontrebootme/microbot:v1   "/bin/sh -c /start_ng"   15 seconds ago      Up 14 seconds       10.0.2.15:25027->80/tcp, 10.0.2.15:25027->80/udp   microbot-6fc76072-6be1-587b-f83e-fb701e22e0f0
9415b6d4fa95        dontrebootme/microbot:v1   "/bin/sh -c /start_ng"   15 seconds ago      Up 15 seconds       10.0.2.15:56609->80/tcp, 10.0.2.15:56609->80/udp   microbot-65306089-b2e3-105f-4155-d42b904a379a

docker inspect shows me that a HostIp is being passed that differs from the configuration above.

        "Ports": {
            "80/tcp": [
                {
                    "HostIp": "10.0.2.15",
                    "HostPort": "50793"
                }
            ],
            "80/udp": [
                {
                    "HostIp": "10.0.2.15",
                    "HostPort": "50793"
                }
            ]
        },
@diptanu

This comment has been minimized.

Copy link
Collaborator

commented Dec 9, 2015

@dontrebootme Did you try setting the network_interface option? This is a client specific option which forces Nomad client to fingerprint a specific network interface and not look at anything else on the host on which the Agent is running. This would make Nomad to always choose this interface. The option takes an interface name.

The bind addr can be different than the network interface that Nomad fingerprints and allocates ports to Tasks.

Let me know if that works?

@dontrebootme

This comment has been minimized.

Copy link

commented Dec 9, 2015

I can confirm that setting network_interface in my nomad client configuration did it for me.

Snippet from my config:

  "client": {
    "enabled": true,
    "network_interface": "eth1",
    "servers": [ "10.7.0.10:4647" ]
  }
@skozin

This comment has been minimized.

Copy link

commented Dec 24, 2015

@skozin We don't support selecting one of many interfaces in Nomad yet. Support for doing that is going to come fairly soon.

@diptanu Do you have any updates on this (and #264, which is closely related)? Can I assume that by "fairly soon" you mean 0.3? Should I open separate issue for that use case (selecting specific interface for specific Docker job)?

@skozin

This comment has been minimized.

Copy link

commented Jan 5, 2016

@diptanu, @cbednarski, I've opened a separate issue #646 for my request; but the proposed solution covers listening on all interfaces as well.

@sean- sean- added sync and removed sync labels Apr 29, 2016

@safanaj

This comment has been minimized.

Copy link

commented Jun 7, 2016

Hi, the IP od dockers services registered in consul shouldn't not select dynamically, inspecting the started docker?
here I think nomad should inspect the task (atleast for docker driver) instead just to use NetworkResources

@dadgar

This comment has been minimized.

Copy link
Contributor

commented Jan 3, 2017

Going to close this since the original was a config bug and there is a subsequent issue for multiple networks #646

@dadgar dadgar closed this Jan 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.