topicctl new action: rebalance

[ssingudasu]

Introduction:

To rebalance a topic in topicctl, We currently use topicctl apply $topic --rebalance / topicctl apply $topic --rebalance --to-remove . In a production scale Kafka cluster, there is a need to rebalance all topics for a cluster.

Goal:

• Add new action - rebalance to topicctl. i.e topicctl rebalance
• Print rebalance progress status (stdout) during rebalance. Refer log [timestamp] INFO Rebalance Progress: {}. (Please refer below screenshots for output)
• Rebalance progress information is two fold. overall topic rebalance status and topic rebalance status for each iteration

Considerations for rebalance:

• expects --cluster-config (i.e cluster.yaml) and --path-prefix (topics folder)
• Uses underlying topicctl pkg/apply to rebalance a topic
• Argument --show-progress-interval to show rebalance progress in intervals
• Arguments --skip-confirm --auto-continue-rebalance are set to true by default
• Argument --retention-drop-step-duration is set to default value. Hence that argument is not considered for rebalance

NOTE: A topic is not considered for rebalance if

1. topic config is inconsistent with cluster config (name, region, environment etc...)
2. partitions of a topic in kafka cluster does not match with topic partition setting in topic config
3. retention.ms of a topic in kafka cluster does not match with topic retentionMinutes setting in topic config
4. topic does not exist in kafka cluster

Test Output:

Tested on local kafka cluster (3 brokers, 1 in each rack) with 3 topics
test-1 -> exists in kafka. But settings mismatch with test-1 topic yaml config
test-2 -> exists in kafka. settings match (replication 2)
test-3 -> does not exist in kafka
test-4 -> exists in kafka. settings match (replication 1)

---

go run cmd/topicctl/main.go -h

---

go run cmd/topicctl/main.go rebalance -h

---

go run cmd/topicctl/main.go rebalance

---

go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640

---

go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640 --to-remove 3

---

go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640 --to-remove 2,3

---

[hhahn-tw]

I think something should also be added to the rebalance section of the readme to emphasize that rebalance is a new verb in addition to apply for this functionality, and briefly explain how it gathers a list of topics based on yaml files found in the specified location.

[hhahn-tw]

I'm assuming you just used the wording from the rebalance section but "full broker rebalance" suggests something else to me. How about "full cluster rebalance" or "all broker rebalance"? This change should be made in all places it appears.

[ssingudasu]

"full cluster rebalance" this makes sense to me..

[ssingudasu]

I am not sure about modifying at other places. But I am modifying atleast in the subcommand rebalance