[improve] replace bookie from different rack of other quorum when do recovery

[TakaHiR07]

Motivation

If we need to do recovery when one bookie down, in the current RackAwarePolicy, it would try picking a candidate from same rack to replace the down bookie firstly. There is a risk that if many bookie in the same rack shutdown, it would result in replacing bookie to the several bookie remain in this rack, making these bookies become hot spot.

Changes

choose the replaced bookie from different rack of other quorum when do recovery or do ensemble change.

[dlg99]

Please make this configurable with current behavior enabled by default + add tests

[hangc0276]

For RackAwarePolicy, it will try its best effort to pick bookies from different racks. When doing recovery, it is safe to replace bookies from the same rack.

[TakaHiR07]

For RackAwarePolicy, it will try its best effort to pick bookies from different racks. When doing recovery, it is safe to replace bookies from the same rack.

In some extreme case, such as 5 bookies in 1 rack, 4 bookies down, 1 bookie survive. The recovery throughput would concentrate on this bookie. After recovery, it would result in data skew in this survived bookie. I think it is not really safe, maybe cause some problem.
And when do recovery, it seems data locality is not exist. Because the recover bk client in ReplicationWorker is random distribution, it would read data from rack 2 and then recover data to rack 1. So it is not neccessary to ensure the replaced bookie in the same rack

[hangc0276]

For RackAwarePolicy, it will try its best effort to pick bookies from different racks. When doing recovery, it is safe to replace bookies from the same rack.

In some extreme case, such as 5 bookies in 1 rack, 4 bookies down, 1 bookie survive. The recovery throughput would concentrate on this bookie. After recovery, it would result in data skew in this survived bookie. I think it is not really safe, maybe cause some problem. And when do recovery, it seems data locality is not exist. Because the recover bk client in ReplicationWorker is random distribution, it would read data from rack 2 and then recover data to rack 1. So it is not neccessary to ensure the replaced bookie in the same rack

@TakaHiR07 If we not ensure the replaced bookie in the same rack, it will break the placement policy in ensemble change and auto recovery.

[TakaHiR07]

For RackAwarePolicy, it will try its best effort to pick bookies from different racks. When doing recovery, it is safe to replace bookies from the same rack.

In some extreme case, such as 5 bookies in 1 rack, 4 bookies down, 1 bookie survive. The recovery throughput would concentrate on this bookie. After recovery, it would result in data skew in this survived bookie. I think it is not really safe, maybe cause some problem. And when do recovery, it seems data locality is not exist. Because the recover bk client in ReplicationWorker is random distribution, it would read data from rack 2 and then recover data to rack 1. So it is not neccessary to ensure the replaced bookie in the same rack

@TakaHiR07 If we not ensure the replaced bookie in the same rack, it will break the placement policy in ensemble change and auto recovery.

@hangc0276 This pr only use the implementation in

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Lines 558 to 562 in e672284

	public BookieNode selectFromNetworkLocation(Set<String> excludeRacks,
	Set<Node> excludeBookies,
	Predicate<BookieNode> predicate,
	Ensemble<BookieNode> ensemble,
	boolean fallbackToRandom)

instead of

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Lines 531 to 536 in e672284

	public BookieNode selectFromNetworkLocation(String networkLoc,
	Set<String> excludeRacks,
	Set<Node> excludeBookies,
	Predicate<BookieNode> predicate,
	Ensemble<BookieNode> ensemble,
	boolean fallbackToRandom)

I think this implementation should not break the RackawarePolicy

[hangc0276]

For RackAwarePolicy, it will try its best effort to pick bookies from different racks. When doing recovery, it is safe to replace bookies from the same rack.

In some extreme case, such as 5 bookies in 1 rack, 4 bookies down, 1 bookie survive. The recovery throughput would concentrate on this bookie. After recovery, it would result in data skew in this survived bookie. I think it is not really safe, maybe cause some problem. And when do recovery, it seems data locality is not exist. Because the recover bk client in ReplicationWorker is random distribution, it would read data from rack 2 and then recover data to rack 1. So it is not neccessary to ensure the replaced bookie in the same rack

@TakaHiR07 If we not ensure the replaced bookie in the same rack, it will break the placement policy in ensemble change and auto recovery.

@hangc0276 This pr only use the implementation in

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Lines 558 to 562 in e672284

	public BookieNode selectFromNetworkLocation(Set<String> excludeRacks,
	Set<Node> excludeBookies,
	Predicate<BookieNode> predicate,
	Ensemble<BookieNode> ensemble,
	boolean fallbackToRandom)

instead of

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Lines 531 to 536 in e672284

	public BookieNode selectFromNetworkLocation(String networkLoc,
	Set<String> excludeRacks,
	Set<Node> excludeBookies,
	Predicate<BookieNode> predicate,
	Ensemble<BookieNode> ensemble,
	boolean fallbackToRandom)

I think this implementation should not break the RackawarePolicy

@TakaHiR07 The replaceBookie will be used in ensemble change and auto recovery.

[StevenLuMT]

rerun failure checks