Skip to content

Commit

Permalink
Add logs for ensemble select failed (apache#3779)
Browse files Browse the repository at this point in the history
### Motivation
We have 3 bookies in the same rack, and configured `E = 2, W = 2, A = 2`. When one bookie restarted, we found the ledger select bookie from the same rack failed in the ensemble change replacing the failed bookie step.

Due to there being no log information in `selectRandomFromRack`, it's hard to debug the root cause.
https://github.com/apache/bookkeeper/blob/02e64a4b97e03afc9993ab227f82a5956965c03f/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L630-L679

### Modification
Add one warn log in `selectRandomFromRack` when selecting a new bookie node failed.

(cherry picked from commit 9ff2954)
  • Loading branch information
hangc0276 committed Jun 26, 2023
1 parent 069b51f commit d51c86e
Showing 1 changed file with 8 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -531,9 +531,10 @@ public BookieNode selectFromNetworkLocation(
networkLoc, excludeBookies);
throw e;
}
LOG.warn("Failed to choose a bookie from {} : "
+ "excluded {}, fallback to choose bookie randomly from the cluster.",
networkLoc, excludeBookies);
LOG.warn("Failed to choose a bookie from network location {}, "
+ "the bookies in the network location are {}, excluded bookies {}, "
+ "current ensemble {}, fallback to choose bookie randomly from the cluster.",
networkLoc, topology.getLeaves(networkLoc), excludeBookies, ensemble);
// randomly choose one from whole cluster, ignore the provided predicate.
return selectRandom(1, excludeBookies, predicate, ensemble).get(0);
}
Expand All @@ -556,6 +557,10 @@ public BookieNode selectFromNetworkLocation(String networkLoc,
* the whole cluster and exclude the racks specified at
* <tt>excludeRacks</tt>.
*/
LOG.warn("Failed to choose a bookie node from network location {}, "
+ "the bookies in the network location are {}, excluded bookies {}, "
+ "current ensemble {}, fallback to choose bookie randomly from the cluster.",
networkLoc, topology.getLeaves(networkLoc), excludeBookies, ensemble);
return selectFromNetworkLocation(excludeRacks, excludeBookies, predicate, ensemble, fallbackToRandom);
}
}
Expand Down

0 comments on commit d51c86e

Please sign in to comment.