Skip to content

Commit

Permalink
range balancer: use correct members for div/rem
Browse files Browse the repository at this point in the history
Issue redpanda-data#24 showed that the range balancer was broken. On first glance,
the issue appears to be because the consumerIdx wraps past the end of
potentialConsumers, and the obvious fix is to just inc and mod. This
would be an incorrect fix.

The real problem lies above in determining div and rem. The whole point
is that (div*(potentialConsumers) + rem) == num partitions, thus if we
drain div and one from rem every loop, then we will never increment past
the end of potential consumers, because partitions will be completely
drained on the last consumer.

The prior logic accidentally made div smaller and rem larger, thus
giving too few partitions to each consumer, and then in some scenarios,
would cause us to exhaust consumers while still draining partitions.

I will be adding unit tests later.
  • Loading branch information
twmb committed Mar 5, 2021
1 parent e806126 commit e038916
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pkg/kgo/group_balancer.go
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ func (*rangeBalancer) balance(members []groupMember, topics map[string]int32) ba
partitions[i] = int32(i)
}
numParts := len(partitions)
div, rem := numParts/len(members), numParts%len(members)
div, rem := numParts/len(potentialConsumers), numParts%len(potentialConsumers)

var consumerIdx int
for len(partitions) > 0 {
Expand Down

0 comments on commit e038916

Please sign in to comment.