Feedback for docker 1.7/1.8 experimental networking features

[mavenugo]

Using this issue to solicit feedback on the exciting new networking experimental features introduced during the Docker 1.7 / 1.8 release time-frame.
https://github.com/docker/docker/blob/master/experimental/networking.md

[abdelrahmanhosny]

I created a network called test using:
docker network create test

Then I ran two containers like that:
docker --default-network=test:bridge run -it --publish-service my-service1.test ubuntu bash
docker --default-network=test:bridge run -it --publish-service my-service2.test ubuntu bash
in two separate terminal windows.

I tried to ping from one container to the other using:
ping my-service1 from the second container, but it is not reachable. What is missing on my side?

[mavenugo]

@abdelrahmanhosny Couple of things.

1. --default-network is a daemon flag, which must be used when docker daemon is brought up.
2. /etc/hosts based service discovery code is not merged in docker experimental channel yet.
   Its on its way via Vendoring libnetwork 83743db8ceb2bdbfa0960d9a54ed2f98df4ea846 #14071. Can you try pinging the container using its IP-address ?

Regardless, we would like to keep this Issue for design questions, rather than discussing specific technical issues. If you have specific issues like these, please reach us in #docker-network IRC or open another issue once you confirm that it is a bug.

[philm]

Excellent feature, thanks!

Since it's possible to publish a service when running a container via the --publish-service flag, it would be nice to be able to optionally deregister the service (in one step) when stopping/removing a container.

[thockin]

A few issues.

1. I want to be able specify the CIDR for the bridge when I create a new bridge network. Is there a way to pass parameters?

2. What are the restrictions on the name format? For example, "_foo" is not a valid container name, but it is a valid network name?

3. When my docker daemon restarts, all my networks are gone. Are those not persisted?

[Stellaverse]

Hi folks,

I would like to see:

Multiple containers on the same host published under the same 'service domain'

sudo docker network create srv
sudo docker run -d --name Hello --publish-service hello.srv stellaverse/hello
sudo docker run -d --name Hello2 --publish-service hello.srv stellaverse/hello
sudo docker run -d --name Hello3 --publish-service hello.srv stellaverse/hello

And running curl http://hello.srv from within any of the containers would be round-robin balanced across all available running containers.

Hello@10.0.0.1$ curl http://hello.srv
Hello from Hello at 10.0.0.1!

Hello@10.0.0.1$ curl http://hello.srv
Hello from Hello2 at 10.0.0.2!

Hello@10.0.0.1$ curl http://hello.srv
Hello from Hello3 at 10.0.0.3!

• I would like to see this extended across an entire swarm, and x-dc too.
• I would also like to see same-host affinity tried first.

Thanks!

[diegomarangoni]

I can't try the multi-host with overlay network, having the following error:

# docker -d --kv-store=etcd:127.0.0.1:2379

WARN[0000] Running experimental build                   
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
INFO[0000] [graphdriver] using prior storage driver "aufs" 
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1 
FATA[0000] Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to initialize vxlan id manager: failed to initialize bit sequence handler: 100: Key not found (/docker) [4] 

What I doing wrong?

[ghost]

@diegomarangoni
this may be related to #14145, I met this problem at first on kernel 3.13, but it's solved now when i switch to 3.19.

[ghost]

@mavenugo
Hi,
I just tried the overlay over two nodes, basic setup is like below:

node-1
usr/bin/docker -d -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --kv-store=consul:10.170.28.3:8500 --label="com.docker.network.driver.overlay.bind_interface=eth0" &
node-2
usr/bin/docker -d -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --kv-store=consul:10.170.28.4:8500 --label="com.docker.network.driver.overlay.bind_interface=eth0" &

And I created overlay network called test:

node-1
root@ubuntu-swarm-1:# docker network ls
NETWORK ID NAME TYPE
ecb6a9f48630 bridge bridge
53e7cde4a447 test overlay
7dc095d5e00f none null
4a78f9bd9d49 host host
node-2
root@ubuntu-swarm-2:# docker network ls
NETWORK ID NAME TYPE
58391b4faa42 host host
ad294b290e39 bridge bridge
53e7cde4a447 test overlay
5e6f4b98d10a none null

And I started the two containers in these two nodes:

node-1
docker run -d -P --publish-service service.test centos/ssh
INFO[0181] POST /v1.20/containers/create
a5a9ea57d9522fc65b78bc6b0c617c66e17efb6fa55c3aca977d6b875746897a
INFO[0182] POST /v1.20/containers/a5a9ea57d9522fc65b78bc6b0c617c66e17efb6fa55c3aca977d6b875746897a/start
Received user event name:jl 10.10.10.229 53e7cde4a447d0e07e5529da738596cfb77829612ce394a7a28d416da2cac449 7a39cb3ee96f9c8b0df2f71be8b9e7771fb4083555a47023147dabd1cd31a2b8, payload:join 172.21.0.1 02:42:e7:16:4f:69
Parsed data = 53e7cde4a447d0e07e5529da738596cfb77829612ce394a7a28d416da2cac449/7a39cb3ee96f9c8b0df2f71be8b9e7771fb4083555a47023147dabd1cd31a2b8/10.10.10.229/172.21.0.1/02:42:e7:16:4f:69

node-2
docker run -d -P --publish-service service2.test centos/ssh
INFO[0924] POST /v1.20/containers/create
67d46f5acb27bceb1bd27db2c839a7e4d4e124536f22e7bdad5ca87c4211b795
INFO[0924] POST /v1.20/containers/67d46f5acb27bceb1bd27db2c839a7e4d4e124536f22e7bdad5ca87c4211b795/start
WARN[0924] Failed adding service host entries to the running container: open : no such file or directory
Received user event name:jl 10.10.10.233 53e7cde4a447d0e07e5529da738596cfb77829612ce394a7a28d416da2cac449 252722b1318395c50a32675f903ad590a78f455a9c40cecbcf3f8230511f02de, payload:join 172.21.0.2 02:42:aa:c6:2a:e5
Parsed data = 53e7cde4a447d0e07e5529da738596cfb77829612ce394a7a28d416da2cac449/252722b1318395c50a32675f903ad590a78f455a9c40cecbcf3f8230511f02de/10.10.10.233/172.21.0.2/02:42:aa:c6:2a:e5

but there are two problems here:
1 "-P" doesn't work, there is no port mapping, is it on purpose?
2 first container has ip "172.21.0.1", second container has ip "172.21.0.2", but they can't ping each other, it means overlay network doesn't work, what's wrong with my operation?

I noticed that there is warning when i start second container on overlay network - "WARN[0924] Failed adding service host entries to the running container: open : no such file or directory", is it because of that? how it happened?

[mavenugo]

@diegomarangoni we are working on a blog post to clearly mention all the pre-req and the configuration details. In addition to the issue with 3.13 kernel, we also have a PR opened in libkv to solve a specific problem with etcd, which got resolved in docker/libkv#14. We will pull that fix in to libnetwork to make etcd backend work.

[mavenugo]

@wf2004081522 you are missing the --label="com.docker.network.driver.overlay.neighbor_ip in the second host, in order for Serf to join the neighbor.
We are working on an end-to-end blog post to get this cleared up.

[ghost]

@mavenugo
thanks very much for explanation, now containers on overlay network can communicate each other, but how about my first question? the container still can't map port with flag '-P' or '-p', is it on purpose? if so, how outside can visit the service on overlay network?

[eckz]

Hi,

I've create a Vagrant setup to test the docker multi host networking setup,
Its at https://github.com/eckz/docker-multihost-example

I hope that can help someone to test it and give feedback, or at least as a setup example.

This is some feedback, I hope it helps:

• Configuring the neighbor_ip it's not clear, (do I need to specify only the first node?, a random node?).
  Couldn't be easier just to create a list of IP's in consul, pretty much like swarm does when configured to use consul?
• When defining --default-network= it's not clear that this network will be created automatically.
• it would be useful to let the docker daemon to start even when the --kv-store is not accessible, (e.g. running consul inside docker ). Of course it would be checked when needed (e.g docker network ls).
• I guess for now the docker client network part is missing, something like docker run --net="overlay:multihost", that would be great to test.

[jgkamat]

@eckz
I think the neighbor_ip needs to be configured on all hosts other than the first one, pointing to the ip address of the first node
If --default-network is set, it will be created automatically if not already present
You could run your kv-store directly on the host, or run a second docker daemon (with normal settings) if you wanted in the meantime
I think the behavior you are looking for is --publish-service

My own feedback:
It would be really nice, as @wf2004081522 said, to have a way to access overlay networks externally. A possible hack to solve this is starting a container to use as a proxy...

I'll continue adding to this post as I think of things...

[rajnemani]

Hi
I am trying to run the secnario at https://github.com/docker/docker/blob/master/experimental/compose_swarm_networking.md. Setting up on AWS. I see two errors.

1. when I try to run the following on Swarm-1 (my first swarm node)
   docker-machine --debug create
   --driver amazonec2 --amazonec2-access-key X --amazonec2-secret-key X --amazonec2-region X --amazonec2-subnet-id "X" --amazonec2-zone X --amazonec2-ami X --engine-install-url="https://experimental.docker.com" --engine-opt="default-network=overlay:multihost"
   --engine-opt="kv-store=consul:$(docker-machine ip consul):8500"
   --engine-label="com.docker.network.driver.overlay.bind_interface=eth0"
   --engine-label="com.docker.network.driver.overlay.neighbor_ip=$(docker-machine ip swarm-0)"
   swarm-1
   I get the following weird error.
   "Maximum number of retries (60) exceeded
   docker: 'Error' is not a docker command.
   See 'docker --help'."
2. When i try to run the "docker run busybox echo hello world" , it pulls the image down but then throws the following error
   "Error response from daemon: Cannot start container 7fc95d4c287db41023c644a01c2761dafcb24b8cd3804f8f2c2d875f8dd93013: network sandbox join failed: could not add vxlan interface inside the network sandbox: failed in prefunc: failed to set namespace on link "vxlan06810e5": invalid argument"

There are some suggestions of upgrading the kernel to 3.16 or 3.19. Can I just upgrade my swarm-0 (master) to 3.19 now or do I need to upgrade the kernel after creating the machine on AWS but before setting up swarm on master and individual nodes.

[mavenugo]

@wf2004081522 @jgkamat external access via overlay network is something that we are exploring. One of the ideas that we are trying out now is to add the container to the bridge network when the user expresses interest using the port-expose / mapping options : --expose or -P or -p. So, if you run a container with both the --publish-service option (in overlay network) and -P / -p, then that container is placed on 2 networks managed by overlay & bridge. The bridge network takes care of exposing the service via NAT and Port-mapping (just as it is done today) and the overlay network takes care of the east-west traffic between the container using the VXLAN overlay tunnel. Please try it out and let us know if that works for you.

[mavenugo]

@eckz Thanks for the feedback. @jgkamat answered a few of them.
We are exploring the possibility of moving away from the need to have an external consistent store (represented by --kv-store) and have an embedded eventually-consistent layer such as Serf that we currently use in the overlay driver. But it needs a lot of work and we will keep you posted.

Also, the --net option is a heavily discussed topic and we will have some concrete answer soon.

[mavenugo]

@rajnemani Yes. we currently have a pre-req of 3.16+ kernel in order to try the overlay driver. Your question is specific to Machine and I would ping @nathanleclaire to give a more authoritative answer.

[rajnemani]

@mavenugo Upgrading to 3.19 resolved my 2nd issue from my earlier post. Thank you for your help. But I still have 1st issue. I cannot add/setup swarm-1 (additional node) to swarm.

@nathanleclaire, any thoughts on the following issue?

when I try to run the following on (my first swarm node, swarm-0 is my master)
docker $(docker-machine config swarm-1) run -d
--restart="always"
--net="bridge"
swarm:latest join
--addr "$(docker-machine ip swarm-1):2376"
"token://$SWARM_TOKEN"

I get the following weird error.

"Maximum number of retries (60) exceeded
docker: 'Error' is not a docker command.
See 'docker --help'."

Note that in my post above I gave the wrong docker command. That command I gave is for creating the swarm-1 node using docker-machine on AWS. That works fine. The problem happens when I run the above docker run command that I provided in this post.

Thanks in advance for help.

[jgkamat]

@mavenugo thats a great solution! It looks like its working great right now! 😄

My only issue with it (so far, and its not really an issue), is that the IP of the container shows the overlay IP when running docker inspect <container>. I personally would prefer the external IP to show up (if running in this dual setup), and maybe the overlay IP in the 'secondary IP section'? I'm not sure if that is it's intended purpose. I have been using docker inspect to get the IP's of containers so far, but I'm not sure if that's the correct way to do so.

EDIT: It would also be nice to activate the dual networks with its own flag, so that we don't have to publish ports if we don't need to. I'm not sure if this behavior should be made default or not.

[smakam]

@mavenugo
Great job by Docker Networking team...
I was able to try out the bridge and overlay network types and basic stuff worked fine and I was able to connect containers.
Few questions:
Question 1:
As I understand endpoint and service is used interchangeably, endpoint and network are mapped 1:1, container can have multiple endpoints.
There can be 2 workflows:
workflow1:
create network
create service and attach to network
create container and attach to service and network
workflow2:
create container with service and network in 1 shot

Rather can container creation getting exposed to network option, can container attach/detach from service/endpoint and let service creation/deletion manage the network? Can we avoid specifying network while creating containers since we are anyway mentioning the service?

Question 2:
Related to previous question, can we create containers with just the network option? I think in 1 of the answers, you mention --net is widely discussed.

Question 3:
Is there a plan to share environment variables like --link option that was available earlier? Not sure if this is covered somewhere? I tried --link option with overlay, was not able to get that to work.

Thanks
Sreenivas Makam

[Lennie]

So yesterday I watched the Docker channel video from DockerCon 2015 about Networking.

And of course I had to try it out and here I my thoughts on what I've seen.

Really glad you all figured out the right abstractions and things are starting to look usable.

At the moment when you want to connect to an overlay-network and you want to connect to the outside world you need to configure 2 things.

For example bridge and overlay.

But you can't specify both on the commandline in 1 command. This is all you get:

docker run --publish-service=test1.multihost.overlay -i -t alpine /bin/sh
or:

docker run --publish-service=test1.bridge.bridge -i -t alpine /bin/sh

It seems you can't use 2 --publish-service-flags or some combined command divided with a comma or something:
docker run --publish-service=test1.bridge.bridge,test1.multihost.overlay -i -t alpine /bin/sh

Also if you create the container with a bridge first and attach the overlay later you don't get the same behaviour as when you create the container with overlay network first and attach bridge later.

With overlay network first you get a working /etc/hosts file, the other way around you don't get that.

That is obviously kind of annoying.

If you have an application that at start up time needs to connect to something on the overlay-network and the outside world then it would probably fail because one isn't configured yet at the start of the application because you can't specify both on the commandline (maybe there is a workaround, but I don't know yet).

Also if you publish a network, you'll always have to unpublish it. While the cool thing about the default network is that all the bahaviour stays the same when just a deploy a container without any extra options. It does mean it will automatically publish and if you are not aware of what you are doing you'll get lots and lots of published networks you don't need anymore. If I'm not mistaken every single published network does generate some traffic.

---

Sreenivas Makam mentioned --link.

Haven't played with --link yet in this context.

But I'm not sure how much sense that would make.

Because as I understand it with overlays:

Every container connect to an overlay can talk to anything else connected to that overlay.

Also every container you deploy with an overlay will have it's published named available to all other container automatically.

So you can also choose the name: --publish-service=db.mynetwork.overlay

Not sure why I would still need --link for that, right ?

---

"We are exploring the possibility of moving away from the need to have an external consistent store (represented by --kv-store) and have an embedded eventually-consistent layer such as Serf that we currently use in the overlay driver. But it needs a lot of work and we will keep you posted."

Maybe I'm wrong, but I believe Serf is not a reliable message transport. If you want to do that you might want to look at the Consul code as an example on how to do that because Consul also uses Serf but in some kind of reliable fashion.

An option is to always use a KV and register the Docker host IP-address in the KV so you don't have to specify it on the commandline to find other hosts. That way it wouldn't need to specify any IP-addresses on the commandline when starting Docker just some way to connect to the KV.

Because now you have to do 2 cluster join operations. One for the KV (like Consul) and one for Docker.

Obviously as mentioned above, if Docker needs consul at start up and you want to run Consul in a container this is also kind of a weird situation. The easiest workaround is to use a 'system-docker' just like RancherOS has, basically 2 docker daemons running on the same host.

---

Have not tried to look at performance either. I would think vxlan, how it ended up in the kernel would be fast. But sometimes how you use it can really make things slower.

Thanks again,
Hope this thoughts are helpful.

[sridhardevarapalli]

Great job Docker networking team! Very useful feature. Have been able to successfully trial the basic overlay networking model with the guidance provided.

Have a couple of questions regarding IPAM -

• By default, every network created using the overlay driver seems to be using the 172.21.0.0/16 CIDR block. And an IP address is allocated to a service/endpoint by the driver at the time of service creation. How does a user specify a desired CIDR block/IP pool to the driver, instead of using the default IP pool? Is this specified on a per-network basis, or is it specified one time during initial driver configuration?
• What is the longer term thinking in terms of using a customer's existing IPAM solutions/tools for IP address management? Are there any thoughts on having a pluggable IPAM module? I understand this may be too early to discuss now, but just want to get a sense for how this might evolve.

[sridhardevarapalli]

Another related question regarding IP addresses -
Now that networks have global scope, would there be a way to assign static IP addresses to some select services/endpoints on a given network?

[dkirrane]

Will there be a way to assign a virtual ip to a service. For instance, 2 Load Balancer containers would share a virtual IP to avoid a single point of failure. When the first LB container (primary) goes down then the virtual IP is transferred to the second LB container (backup).

[McAlister]

I have an experimental swarm going but can't seem to create a network. This happens:

[ec2-user@ip-172-31-47-154 ~]$ docker network create -d overlay intc
Error response from daemon: 404 page not found

I'm guessing the "not found" is from an attempt to access a key/value store that the documents say the overlay driver needs. Since all my nodes are already set up I don't need Machine to make them for me but the only documents for wiring in the key/value store are through machine. Do I have to scrap everything just so Machine can spin it back up and wire the consul store in? Or is there I way I can point the swarm nodes I have to the stand-alone node with the store on it?

Thanks,

[cpuguy83]

@McAlister Swarm does not support docker experimental features.
As such, 404 is happening because swarm does not implement the docker network API endpoints.

[McAlister]

@cpuguy83 thats rather confusing then. This is a red herring?

https://github.com/docker/docker/blob/master/experimental/compose_swarm_networking.md

[cpuguy83]

@McAlister You'll notice the example is not using the top-level networking API's, it's only using docker run and docker engine handles the rest.
docker network create, however, is a full-blown top-level API command that swarm does not implement, because it's not part of docker engine's stable release.

[McAlister]

@cpuguy83 I think I found the proper set of instructions:

https://github.com/docker/libnetwork/blob/master/docs/overlay.md

Of course now I get to curse at EC2 for being on kernel 3.14 ... /shakes fist in the general direction of of the Oregon datacenter.

[jgkamat]

@McAlister You should be able to upgrade the kernel on your machines by changing the repository lists in /etc/apt/sources.list (to utopic?) and apt-get installing the new kernel. If you use ubuntu 14.04:

sed -i 's/trusty/utopic/g' /etc/apt/sources.list
apt-get update
apt-get install linux-image-extra-3.16.0-41-generic
# Reboot your machine

as root

[McAlister]

@jgkamat Amazon EC2 boxes are a bit less cooperative than that ... it is possible to change their kernel but its less straightforward a yum update.

https://aws.amazon.com/blogs/aws/use-your-own-kernel-with-amazon-ec2/

If 3.16 isn't “EC2 compatible” I'll have to bake cookies for our yum repo guardian to convince him to add the rpm for the 3.16 kernel to the corporate repository where my on premise boxes can see them ... since for security reasons we aren't allowed to talk to public repos.

Glory in the freedom of open source work you lucky people =).

[McAlister]

For reference the Debian folks have put a free ec2 ami in the Amazon Marketplace that grubs you up to kernel 3.16.0-4 Image name:

Debian GNU/Linux 8 (Jessie)

The only others I found were $$$ images from Oracle and Redhat. When you release you might add a list of docker-friendly ami's to a faq somewhere for people deploying docker into AWS.

[McAlister]

Hrrmmm ... I'm seeing things in etc/hosts but the traffic doesn't seem to be getting from point A to point B.

admin@ip-172-31-33-164:/docker$ docker network ls
NETWORK ID NAME TYPE
8d03f93006e6 intc overlay
d856ce91c4e8 none null
5ea54da5e8b6 host host
dad21bbeb170 bridge bridge

admin@ip-172-31-33-164:/docker$ docker service ls
SERVICE ID NAME NETWORK CONTAINER
f9a0d1c4f87d ec intc 61c6cd06c88c
0123effc0634 la intc 986d3059dbab

ec.intc is on host 1 and la.intc is on host 2

admin@ip-172-31-33-164:/docker$ consul members
Node Address Status Type Build Protocol DC
ip-172-31-33-164 172.31.33.164:8301 alive server 0.5.2 2 dc1
ip-172-31-44-7 172.31.44.7:8301 alive client 0.5.2 2 dc1
ip-172-31-34-176 172.31.34.176:8301 alive client 0.5.2 2 dc1

Exec into either one and I see the proper etc/hosts file:

Host 1:

admin@ip-172-31-33-164:/docker$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
61c6cd06c88c busybox "top" 15 minutes ago Up 15 minutes mock-ec
admin@ip-172-31-33-164:/docker$ docker exec -it 61c6cd06c88c /bin/sh
/ # cat /etc/hosts
172.21.0.1 61c6cd06c88c
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.21.0.2 la
172.21.0.2 la.intc
172.21.0.1 ec
172.21.0.1 ec.intc

Host 2:

admin@ip-172-31-44-7:/docker$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PO
RTS NAMES
986d3059dbab busybox "top" 14 minutes ago Up 14 minutes
mock-la
admin@ip-172-31-44-7:/docker$ docker exec -it 986d3059dbab /bin/sh
/ # cat /etc/hosts
172.21.0.2 986d3059dbab
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.21.0.2 la.intc
172.21.0.1 ec
172.21.0.1 ec.intc
172.21.0.2 la

However, they can't ping each other and if I open a socket on one the other can't write to it.

Host 1:

/ # nc -l -p 4444

Host 2:

/ # telnet ec.intc 4444
asdfsadf
asdfsdaf

and nothing happens on host 1. If i open another shell on host 1 like so:

admin@ip-172-31-33-164:~$ docker exec -it 61c6cd06c88c /bin/sh
/ # telnet ec.int 4444
asdfasd
asdfsadf
asdfsdf
^]

then I see my listener on host 1 output my asdfasd as expected.

Host 1:

/ # ping ec.intc
PING ec.intc (172.21.0.1): 56 data bytes
64 bytes from 172.21.0.1: seq=0 ttl=64 time=0.034 ms
64 bytes from 172.21.0.1: seq=1 ttl=64 time=0.031 ms

Host 2:

/ # ping ec.intc
PING ec.intc (172.21.0.1): 56 data bytes
^C
--- ec.intc ping statistics ---
381 packets transmitted, 0 packets received, 100% packet loss

If I create another container on Host 1 things work as expected.

Host 1:

admin@ip-172-31-33-164:/docker$ docker run -itd --name mock-aa --publish-service=aa.intc.overlay busybox top
047d242eea177c2e7f2e6d0e3c36912d17e8baba6f4f3948e694a99522d2fbe2
admin@ip-172-31-33-164:/docker$ docker service ls
SERVICE ID NAME NETWORK CONTAINER
f9a0d1c4f87d ec intc 61c6cd06c88c
0123effc0634 la intc 986d3059dbab
95a325181ae6 aa intc 047d242eea17
admin@ip-172-31-33-164:/docker$ docker exec -it 047d242eea17 /bin/sh
/ # nc -l -p 4444

So I'm listening on mock-aa on HOST 1 then in another window in the ec-intc container:

admin@ip-172-31-33-164:~$ docker exec -it 61c6cd06c88c /bin/sh
/ # telnet aa.intc 4444
asdfsdf
asdfasdf
asdfsdf
^]

and I see the asdfasdf appear in the aa-intc container also on host 1 in the shell where it is listening

So if they are on the same host all is good. Across hosts ... not so much. All the amazon hosts in the test are in the same security group and all the tcp ports are open between computers within that group. /etc/hosts files on all the containers update appropriately when I add containers in or take them out. 'docker network ls' and 'docker service ls' commands likewise are staying nicely in sync between all hosts in the network.

I'm not entirely sure how to debug this so I'll sleep on it. I find it very intriguing that the telnet command appears to connect but doesn't get there. I would expect a "connection refused" or a "trying ec.intc ...." that eventually times out. I've never seen telnet appear to connect but not actually connect before.

[felixrabe]

@McAlister Any update on the connectivity issue? I'm also trying to get Docker networking running on AWS EC2.

See https://gist.github.com/felixrabe/bcc8f67d6c262443d2ef for my work-in-progress on a provisioning script. It's mostly Bash.

[felixrabe]

@McAlister I seem to remember that I once got two Docker containers successfully pinging each other on two AWS hosts, but I'm stuck reproducing how I did that right now.

[felixrabe]

I figured the missing part thanks to @mrjana on IRC (opening UDP port 4789 for VXLAN) and rewrote my provisioning script to docker-machine: https://gist.github.com/felixrabe/cf076655f1d0235e28a3 (this works)

[McAlister]

@felixrabe Sorry I didn't respond faster, The UDP port makes sense, I'll try it tomorrow and let you know. Thank you for digging into it too.

[ajaybhatnagar]

I am trying to setup containers which need to talk to some external servers in meshed topology for example I have servers s1,s2 and s3 with their Ip addresses being 10.0.0.1, 10.0.0.2 and 10.0.0.3 and 3 docker containers say c1,c2 and c3 on 3 separate servers s4,s5, and s6 with server IPs being 10.0.0.4, 10.0.0.5 and 10.0.0.6. Containers are using overlay driver and ip associated with containers c1,c2 and c3 are 172.17.0.1, 172.17.0.2 and 172.17.0.3 respectively. What routing mechanism to be used so that all 3 servers s1,s2, and s3 can talk to any container c1, c2 or c3 ?

Do I need to have a route to network 172.17 vlan with one of the server hosting container acting as default gateway to container network or any other mechanism is suggested . So far I am able to establish connectivity only to single container ip from external server by adding route add net command with the container hosting node acting as gateway other containers are not reachable from external nodes.

[ajaybhatnagar]

What is the process of allocating IP addresses to containers when they are launched on different hosts. How to ensure unique IPs across hosts. When launching containers on two different hosts, I get the same ip address 172.17.0.1 allocated with default setup.

[fkautz]

A command to retrieve the types of network types available during creation would be useful.

e.g.

docker network create --help

 # Available Network Types:
 #    - bridge
 #    - host
 #    - weave

or

docker network list-types
# Available Network Types:
#     - bridge
#     - ...

[ryneeverett]

This should probably be closed since #16645 was merged.

[thaJeztah]

Agreed, let's close this (but feel free to comment after it's closed)