"Using Docker Swarm when running Zookeeper"?

[bluepuma77]

I found this sentence in the Readme to be really promising:

Consider using Docker Swarm when running Zookeeper in replicated mode.

Is this actual possible to run Zookeeper with Docker Swarm on multiple servers? In "real" swarm mode, where I can easily just scale it up from 3 to 5 servers? How would I do that? Don't I need to set all other Zookeeper server (IP) addresses via environment variable?

So far I found only Elastic to be easily docker-swarm-scalable, thanks to DZone. Would love to see this possible with Zookeeper, Apache Kafka, Apache NiFi and Hadoop/HDFS.

[si-harps]

Is there any update on this? I'm keen to run a zookeeper ensemble across a Docker Swarm. It's unclear how to set the required variables (ZOO_MY_ID, ZOO_SERVERS) when deploying a stack either globally or using replicas. Any feedback would be much appreciated.

[31z4]

The solution for dynamic scaling will be pretty complicated. You would have to use Consul as a service registry and Consul Template to change Zookeeper configuration when new container is added or an existing container removed. And then you also have to restart all your Zookepers because prior to version 3.5 it doesn't support dynamic reconfiguration.

A similar solution for scaling Nginx is described in this article.

But it general it seems like dynamic scaling isn't that useful for Zookeeper for several reasons

1. It doesn't scale for writing. In fact addind new nodes increase reliability at the cost of write speed.
2. Usually you decide how many node failures you'd like to tolerate in advance. And often it's no use of running more than 5 nodes.

[31z4]

I guess that's what folks from Bandsintown did in their fork.

[31z4]

Implementing dynamic scaling has become a bit easier with the introduction of the 3.5.3-beta tag. Because starting from 3.5 Zookeeper now supports dynamic reconfiguration.

[31z4]

Closing it because the issue seems to be inactive for quite some time now.