[wip] [FLINK-2288] [FLINK-2302] Setup ZooKeeper for distributed coordination

[uce]

• FLINK-2288: Setup ZooKeeper for distributed coordination
  • Add FlinkZooKeeperQuorumPeer to wrap ZooKeeper's quorum peers with
    utilities to write required config values (default datadir, myid)
  • Add default conf/zoo.cfg config for ZooKeeper
  • Add startup scripts for ZooKeeper quorum
  • Add conf/masters file for HA masters
  • @rmetzger This PR includes docs ;-)
• FLINK-2302: Allow multiple instances to run on single host
  • Multiple TaskManager and JobManager instances can run on a single
    host.

@tillrohrmann, you can base your changes on this branch. After that we can close this PR. I've added TODOs in TaskManager and JobManager, where you need to integrate your leader election/retrieval service.

From the docs:

Example: Start and stop a local HA-cluster with 2 JobManagers

1. Configure ZooKeeper quorum in conf/flink.yaml:

   ha.zookeeper.quorum: localhost

2. Configure masters in conf/masters:

   localhost
   localhost

3. Configure ZooKeeper server in conf/zoo.cfg (currently it's only possible to run a single ZooKeeper server per machine, because there is a single client port per configuration):

   server.0=localhost:2888:3888

4. Start ZooKeeper quorum:

   $ bin/start-zookeeper-quorum.sh
   Starting zookeeper daemon on host localhost.

5. Start an HA-cluster:

   $ bin/start-cluster-streaming.sh
   Starting HA cluster (streaming mode) with 2 masters and 1 peers in ZooKeeper quorum.
   Starting jobmanager daemon on host localhost.
   Starting jobmanager daemon on host localhost.
   Starting taskmanager daemon on host localhost.

6. Stop ZooKeeper quorum and cluster:

   $ bin/stop-cluster.sh
   Stopping taskmanager daemon (pid: 7647) on localhost.
   Stopping jobmanager daemon (pid: 7495) on host localhost.
   Stopping jobmanager daemon (pid: 7349) on host localhost.
   $ bin/stop-zookeeper-quorum.sh
   Stopping zookeeper daemon (pid: 7101) on host localhost.

[StephanEwen]

Typo: rqeuired -> required

[StephanEwen]

Looks like a good piece of work!

Can we actually get into ZooKeeper version conflicts here? For example, the Kafka connector needs a certain Zookeeper version. Can it conflict with our Zookeeper version?

[StephanEwen]

I will address the comments and merge this...

[StephanEwen]

BTW: It is beautiful how easy it is to start multiple TaskManagers and JobManagers on one machine now :-)

[StephanEwen]

We need to find a solution for the webfrontend, though. Starting it on random ports is not a real solution. Also the user that wants to see progress needs to connect manually to the web server of the leader JobManager.

[mxm]

Did we find a solution for the random port problem?

[StephanEwen]

In HA mode, JobManagers start with a random free port. That is fine, because no one connects to them based on a config value, but only based on ZooKeeper entries.

[mxm]

Yes that makes sense. So the user will always have to connect to the web interface of the leading job manager, right? We could only circumvent that by separating the web interface from the job manager.

[tillrohrmann]

The web interface is, modulo some object which are not serializable,
already independent of the JobManager. It should not be a big problem to
only have one web server which also retrieves the leading JobManager from
ZooKeeper and then serves the information from the leader.

On Thu, Jul 9, 2015 at 4:24 PM, Max notifications@github.com wrote:

Yes that makes sense. So the user will always have to connect to the web
interface of the leading job manager, right? We could only circumvent that
by separating the web interface from the job manager.

—
Reply to this email directly or view it on GitHub
#886 (comment).

[mxm]

Alright, I've opened a JIRA for this: https://issues.apache.org/jira/browse/FLINK-2340