dnsrr endpoint mode not working with routing mesh #25016

Open
smakam opened this Issue Jul 25, 2016 · 16 comments

Comments

Projects
None yet
@smakam

smakam commented Jul 25, 2016

Output of docker version:

docker@node2:~$ docker --version
Docker version 1.12.0-rc4, build e4a0dbc, experimental

Output of docker info:

docker@node2:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0-rc4
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay bridge host
Swarm: active
 NodeID: a0j7h4vbc32wmzw1cl56cey9h
 IsManager: No
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.8-boot2docker
Operating System: Boot2Docker 1.11.1 (TCL 7.0); master : 7954f54 - Wed Apr 27 17:59:58 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.9 MiB
Name: node2
ID: PWMA:PMXK:J4VC:Y665:XWJ5:J2AG:U43U:CG6T:VNWA:7USA:V3QU:ZGIM
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 120
 System Time: 2016-07-25T17:02:40.657240432Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Experimental: true
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:

  1. Create 2 node swarm cluster
    docker@node1:~$ docker node ls
    ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
    a0j7h4vbc32wmzw1cl56cey9h node2 Accepted Ready Active
    bfqlqktthzijxk0thf371zxx8 * node1 Accepted Ready Active Leader

Create 2 service with dnsrr endpoint and routing mesh
docker service create --endpoint-mode dnsrr --replicas 1 --name client --network overlay1 smakam/myubuntu:v4 ping docker.com
docker service create --endpoint-mode dnsrr --name vote --network overlay1 --replicas 2 -p 8080:80 instavote/vote
3.

Describe the results you received:
Service gets stuck in new state
docker@node1:$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
4gljblqy8u0t client 1/1 smakam/myubuntu:v4 ping docker.com
4wbpkae9pdsz vote 0/2 instavote/vote
docker@node1:
$ docker service tasks vote
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
53ttlrww3c9cfzlmt4y8jrv9m vote.1 vote instavote/vote New 58 seconds ago Running
0xptv7b3mjbgsy1q1c1rpqnps vote.2 vote instavote/vote New 58 seconds ago Running

Describe the results you expected:
Service should transition to running

Additional information you deem important (e.g. issue happens only occasionally):
dnsrr endpoint works fine without routing mesh. For example, following services got created fine and moved to running state.
docker service create --endpoint-mode dnsrr --replicas 1 --name client --network overlay1 smakam/myubuntu:v4 ping docker.com
docker service create --endpoint-mode dnsrr --name vote --network overlay1 --replicas 2 instavote/vote

docker@node1:~$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
7ctu5us9gu5f vote 2/2 instavote/vote
da5q0xscsw2d client 1/1 smakam/myubuntu:v4 ping docker.com

Also, routing mesh works fine with vip based endpoint mode.

@cpuguy83 cpuguy83 added this to the 1.12.0 milestone Jul 25, 2016

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 25, 2016

Contributor

ping @mavenugo

Contributor

cpuguy83 commented Jul 25, 2016

ping @mavenugo

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jul 25, 2016

Contributor

@smakam @cpuguy83 unfortunately this is the expected behavior and such a configuration is blocked in rc5 : docker/swarmkit#1207

Contributor

mavenugo commented Jul 25, 2016

@smakam @cpuguy83 unfortunately this is the expected behavior and such a configuration is blocked in rc5 : docker/swarmkit#1207

@smakam

This comment has been minimized.

Show comment
Hide comment
@smakam

smakam Jul 26, 2016

@mavenugo
Is it correct that dnsrr with routing mesh does not make sense since there is no concept of service IP with dnsrr? Maybe, its possible that routing mesh can be achieved using a different approach for dnsrr?

Thanks
Sreenivas

smakam commented Jul 26, 2016

@mavenugo
Is it correct that dnsrr with routing mesh does not make sense since there is no concept of service IP with dnsrr? Maybe, its possible that routing mesh can be achieved using a different approach for dnsrr?

Thanks
Sreenivas

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jul 26, 2016

Contributor

@smakam not entirely. I think we should be able to support dnsrr with routing-mesh. But the code-interaction was tricky and it was not handled for 1.12.0. We can see how it can be achieved in subsequent releases.

Contributor

mavenugo commented Jul 26, 2016

@smakam not entirely. I think we should be able to support dnsrr with routing-mesh. But the code-interaction was tricky and it was not handled for 1.12.0. We can see how it can be achieved in subsequent releases.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Aug 1, 2016

Member

With 1.12.0, this is now blocked;

$ docker network create -d overlay overlay1
3k3rzlcp2thy04dzqh48on8mn

$ docker service create --endpoint-mode dnsrr --replicas 1 --name client --network overlay1 smakam/myubuntu:v4 ping docker.com
apqch2ctbv8cjczyk1aos4yzc

$ docker service create --endpoint-mode dnsrr --name vote --network overlay1 --replicas 2 -p 8080:80 instavote/vote
Error response from daemon: rpc error: code = 3 desc = EndpointSpec: ports can't be used with dnsrr mode

I'm removing this from the milestone, as at least this case is handled;

@mavenugo should we close this issue, or want to keep it open for the enhancement?

Member

thaJeztah commented Aug 1, 2016

With 1.12.0, this is now blocked;

$ docker network create -d overlay overlay1
3k3rzlcp2thy04dzqh48on8mn

$ docker service create --endpoint-mode dnsrr --replicas 1 --name client --network overlay1 smakam/myubuntu:v4 ping docker.com
apqch2ctbv8cjczyk1aos4yzc

$ docker service create --endpoint-mode dnsrr --name vote --network overlay1 --replicas 2 -p 8080:80 instavote/vote
Error response from daemon: rpc error: code = 3 desc = EndpointSpec: ports can't be used with dnsrr mode

I'm removing this from the milestone, as at least this case is handled;

@mavenugo should we close this issue, or want to keep it open for the enhancement?

@thaJeztah thaJeztah removed this from the 1.12.0 milestone Aug 1, 2016

@thaJeztah thaJeztah removed the priority/P2 label Aug 1, 2016

@therealpadams

This comment has been minimized.

Show comment
Hide comment
@therealpadams

therealpadams Aug 18, 2016

Do we have a timeline for handling this issue? It's important for users of Crate/Elasticsearch:

  • If I do not enable DNSRR then the Crate/ES nodes cannot communicate with each other
  • If I /do/ enable DNSRR then a Crate/ES cluster can be properly formed, but I no-longer have an exposed service port to connect to

Do we have a timeline for handling this issue? It's important for users of Crate/Elasticsearch:

  • If I do not enable DNSRR then the Crate/ES nodes cannot communicate with each other
  • If I /do/ enable DNSRR then a Crate/ES cluster can be properly formed, but I no-longer have an exposed service port to connect to
@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Aug 18, 2016

Contributor

@therealpadams Can you clarify why Crate/ES nodes cannot communicate if you did not enable DNSRR?

Contributor

mrjana commented Aug 18, 2016

@therealpadams Can you clarify why Crate/ES nodes cannot communicate if you did not enable DNSRR?

@therealpadams

This comment has been minimized.

Show comment
Hide comment
@therealpadams

therealpadams Aug 19, 2016

Honestly, with any level of technical eloquence, no. But I'm happy to go poking around and generate further info for you, if you point me in the right direction. Typically, I was trying to create a Crate service like this:

docker service create
--name crate
--network crate
--mode global
--publish 4200:4200/tcp
--publish 4200:4300/tcp
crate:latest
crate
-Des.discovery.zen.ping.multicast.enabled=false
-Des.discovery.zen.ping.unicast.hosts=crate

Instead of giving me one Crate cluster of n nodes, this would give me n clusters of 1 node each. If I switch to DNSRR I end up with 1 cluster of n nodes (win!) but I am no-longer able to access the cluster via the public-ip:4200 on the swarm members.

If you can point me in the right direction, I am happy to investigate further as to why this does not work with DNSRR disabled.

therealpadams commented Aug 19, 2016

Honestly, with any level of technical eloquence, no. But I'm happy to go poking around and generate further info for you, if you point me in the right direction. Typically, I was trying to create a Crate service like this:

docker service create
--name crate
--network crate
--mode global
--publish 4200:4200/tcp
--publish 4200:4300/tcp
crate:latest
crate
-Des.discovery.zen.ping.multicast.enabled=false
-Des.discovery.zen.ping.unicast.hosts=crate

Instead of giving me one Crate cluster of n nodes, this would give me n clusters of 1 node each. If I switch to DNSRR I end up with 1 cluster of n nodes (win!) but I am no-longer able to access the cluster via the public-ip:4200 on the swarm members.

If you can point me in the right direction, I am happy to investigate further as to why this does not work with DNSRR disabled.

@pascalandy

This comment has been minimized.

Show comment
Hide comment
@pascalandy

pascalandy Sep 23, 2016

+1 !. I want to create a cluster for MySQL via Create.


EDIT - MySQL InnoDB Cluster might be an alternative. https://www.youtube.com/watch?v=JWy7ZLXxtZ4

pascalandy commented Sep 23, 2016

+1 !. I want to create a cluster for MySQL via Create.


EDIT - MySQL InnoDB Cluster might be an alternative. https://www.youtube.com/watch?v=JWy7ZLXxtZ4

@guenhter

This comment has been minimized.

Show comment
Hide comment
@guenhter

guenhter Oct 18, 2016

Also want to create a cluster with Elasticsearch, but this is not possible with VIP.
https://forums.docker.com/t/easy-elasticsearch-cluster-with-docker-1-12-swarm/19648/5
has a description.
Core statement: "Elasticsearch doesn't like get load balanced at all"

Would appreciate this, because the current way how I setup a ES cluster with swarm is ...

Also want to create a cluster with Elasticsearch, but this is not possible with VIP.
https://forums.docker.com/t/easy-elasticsearch-cluster-with-docker-1-12-swarm/19648/5
has a description.
Core statement: "Elasticsearch doesn't like get load balanced at all"

Would appreciate this, because the current way how I setup a ES cluster with swarm is ...

@mavenugo mavenugo added this to the 1.13.0 milestone Oct 25, 2016

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Oct 25, 2016

Contributor

cc @mrjana @al I think we should remove this restriction also to support swarm-mode for windows where routing-mesh will not be enabled and the user should be able to use DNS-RR on the back-end networks with published ports.

Contributor

mavenugo commented Oct 25, 2016

cc @mrjana @al I think we should remove this restriction also to support swarm-mode for windows where routing-mesh will not be enabled and the user should be able to use DNS-RR on the back-end networks with published ports.

@tyoh

This comment has been minimized.

Show comment
Hide comment
@tyoh

tyoh Nov 2, 2016

+1 I'm eager to have this feature to create cluster of Software Defined Storage!

tyoh commented Nov 2, 2016

+1 I'm eager to have this feature to create cluster of Software Defined Storage!

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
Member

thaJeztah commented Nov 2, 2016

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Nov 2, 2016

Contributor

@smakam @tyoh @guenhter @johnharris85 @pascalandy I opened docker/swarmkit#1693 to take care of this issue. But @mrjana has some valid concerns with the use-cases. Could you please comment in docker/swarmkit#1693 so that we understand the use-case better ?

Contributor

mavenugo commented Nov 2, 2016

@smakam @tyoh @guenhter @johnharris85 @pascalandy I opened docker/swarmkit#1693 to take care of this issue. But @mrjana has some valid concerns with the use-cases. Could you please comment in docker/swarmkit#1693 so that we understand the use-case better ?

@RehanSaeed

This comment has been minimized.

Show comment
Hide comment
@RehanSaeed

RehanSaeed Dec 18, 2017

I'm getting the following error when using endpoint-mode: dnsrr using Docker Swarm:

EndpointSpec: port published with ingress mode can't be used with dnsrr mode

Is there a workaround in the latest 17.06 or 17.09 releases? I'm trying to start RabbitMQ which has it's own internal clustering ability with dnsrr and ports exposed.

I'm getting the following error when using endpoint-mode: dnsrr using Docker Swarm:

EndpointSpec: port published with ingress mode can't be used with dnsrr mode

Is there a workaround in the latest 17.06 or 17.09 releases? I'm trying to start RabbitMQ which has it's own internal clustering ability with dnsrr and ports exposed.

@sify21

This comment has been minimized.

Show comment
Hide comment
@sify21

sify21 Feb 23, 2018

@RehanSaeed use --publish target=,published=,mode=host works for me

sify21 commented Feb 23, 2018

@RehanSaeed use --publish target=,published=,mode=host works for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment