Using with Docker stack in Swarm mode

[yogeek]

Hello,

Thank you for sharing this. Is it possible to use the same mechanism but to scale it in a Swarm mode cluster using the 'global' deploy mode with the docker stack deploy command instead of docker-compose on a single host ? I would allow each new node to have a new ES service added to the cluster.
I tried to deploy it but it seems that putting the service name as discovery.zen.ping.unicast.hosts: elasticsearch is not sufficient for ES to detect the other nodes in swarm mode. Here is the error log I have :

java.net.UnknownHostException: elasticsearch: Name or service not known

And here is the docker-stack.yml I use :

version: "3"

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.5
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      transport.host: 0.0.0.0
      discovery.zen.ping.unicast.hosts: elasticsearch
      discovery.zen.minimum_master_nodes: 1
      cluster.name: elasticsearch
      ES_JAVA_OPTS: "-Xms512m -Xmx512m -XX:-AssumeMP"
      xpack.security.enabled: "false"
      xpack.monitoring.enabled: "false"
      xpack.ml.enabled: "false"
      xpack.graph.enabled: "false"
      xpack.watcher.enabled: "false"
      http.cors.enabled: "true"
      http.cors.allow-origin: "*"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    networks:
      spirit1-apis:
    deploy:
      resources:
        limits:
          memory: 1G
      mode: global
      update_config:
        parallelism: 1
        delay: 1m30s
      restart_policy:
        condition: on-failure

volumes:
  elasticsearch_data:

Why can't the DNS round robin of Swarm cannot apply here as it applies in your local docker-compose stack ?

Thank you

[antoineco]

From your description it's unclear how the spirit1-apis network is configured, is it an overlay network?

Also you'll probably want to set endpoint mode: dnsrr in your deploy block, otherwise you'll only get a single IP for all ES nodes so the unicast discovery won't work.

[antoineco]

It also seems like you have to prepend your service name with tasks. in order to get a response from the DNS.

Ref elastic/elasticsearch-docker#91 (comment)

[antoineco]

ping @yogeek
Any success?

[yogeek]

@antoineco Sorry for the delay and thank you for your help.
Indeed, the network is an overlay network. I confirm you that modifying the endpoint mode to "dnsrr" is working but I have to first create 1 replica and then, after the initialization ends, I can scale the service by adding more replicas (I have to ensure that they are not deployed to the same node).
So it works doing this way but without the "tasks." though...
Maybe if I add the "tasks." to the unicast it will allow me to start directly the replicas in global mode... I have to test and I will give you the result
Thanks again

[yogeek]

@antoineco @deviantony Ok, after reading the elastic/elasticsearch-docker#91 (comment) again, I thint I understand.
Here is the conclusion of my tests :

• I an currently in docker 17.06 and the configuration that is working is :

network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: tasks.elasticsearch
deploy:
      endpoint_mode: dnsrr 

• in docker 17.09, the PR Add service virtual IP to sandbox's loopback address moby/libnetwork#1877 made the VIP attached to loopback (I am not sure I understand what it means exactly so I you could detail it, it would be great) so the following configuration allows the services to scale correctly :

network.host = _eth0_
discovery.zen.ping.unicast.hosts: tasks.elasticsearch

And so no more need to be in dnsrr and it becomes possible to publish 9200 !

Am I understanding correctly ? I did not have catch that the PR in libnetwork was the key here

[antoineco]

Hardcore Kubernetes user here :) Not sure if I'm in a good position to answer Swarm questions but I may give 17.09 a look out of curiosity. Stay tuned.

[scmikes]

Same failr with end_point mode

very simple yml file, contributed by some else who verified at their site

version: '3.3'

networks:
  my-net:
    driver: overlay
    attachable: true
           
services:
 
  app1:
    image:  hello-world
    networks:
      - my-net
    deploy:
      endpoint_mode: dnsrr 
    command: powershell -command Start-Sleep 86400
  app2:
    image:  hello-world
    networks:
      - my-net
    deploy:
      endpoint_mode: dnsrr 
    command: powershell -command Start-Sleep 86400  

1. save yml as docker-compose.yml

2. docker swarm init ...
3. docker stack deploy -c docker-compose.yml simple
4. docker ps to get continer name
5. docker exec -it powershell
6. ping other service

PS C:\> ping simple_app2
Ping request could not find host simple_app2. Please check the name and try again.
PS C:\> ping simple_app1
Ping request could not find host simple_app1. Please check the name and try again.

[scmikes]

Note: same failure in simple docker_compose.yml app1 and app2 with or without deploy: endpoint_mode

[antoineco]

@scmikes can you try with #262?

[scmikes]

Sorry Antoineco, somehow, I commented on the wrong bug, I was try for docker/for-win #1476