Skip to content

Latest commit

 

History

History
15 lines (12 loc) · 4.34 KB

File metadata and controls

15 lines (12 loc) · 4.34 KB

Clusters Coordination & Configuration Service (also known as Distributed key-value store)

Purpose Description
Store basic metadata of the cluster Metadata like info about brokers, topics, partitions, mapping of partitions to nodes, partition leader/followers, consumer offset etc.
- ZooKeeper and etcd are designed to hold small amounts of data that can fit entirely in memory (although they still write to disk for durability).
- That small amount of data is replicated across all the nodes using a fault-tolerant total order broadcast algorithm.
Leader Election in the cluster Electing leader in the leader-follower replication, through consensus algorithm.
Controller election in the Cluster Through consensus algorithm
Handle new broker/node addition or failure in the cluster Notifies consumers, producers, routing tier of the arrival of new broker or failure of existing broker.
Route requests to partition leader Route all requests to partition's leaders.

img.png

Why separate distributed key-value store?

  • As an application developer, you will rarely need to use ZooKeeper or etcd directly, because it is actually not well suited as a general-purpose database.
  • It is more likely that you will end up relying on it indirectly via some other project - Kafka (uses Zookeeper), Kubernates (uses etcd) etc.