Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add logs for ensemble select failed #3779

Merged
merged 5 commits into from
Feb 22, 2023
Merged

Add logs for ensemble select failed #3779

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c6505dd
add logs for ensemble select failed
hangc0276 Feb 14, 2023
374b8b0
address comments
hangc0276 Feb 18, 2023
a4dda6d
format code
hangc0276 Feb 18, 2023
75aad86
change log level
hangc0276 Feb 18, 2023
2f22475
update log information
hangc0276 Feb 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions ...rver/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,9 +519,10 @@ public BookieNode selectFromNetworkLocation(
networkLoc, excludeBookies);
throw e;
}
LOG.warn("Failed to choose a bookie from {} : "
+ "excluded {}, fallback to choose bookie randomly from the cluster.",
networkLoc, excludeBookies);
LOG.warn("Failed to choose a bookie from network location {}, "
+ "the bookies in the network location are {}, excluded bookies {}, "
+ "current ensemble {}, fallback to choose bookie randomly from the cluster.",
networkLoc, topology.getLeaves(networkLoc), excludeBookies, ensemble);
// randomly choose one from whole cluster, ignore the provided predicate.
return selectRandom(1, excludeBookies, predicate, ensemble).get(0);
}
Expand All @@ -544,6 +545,10 @@ public BookieNode selectFromNetworkLocation(String networkLoc,
* the whole cluster and exclude the racks specified at
* <tt>excludeRacks</tt>.
*/
LOG.warn("Failed to choose a bookie node from network location {}, "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you think here it is actually an INFO?
What if we have 5 racks. If we failed choosing from same rack (say rack 1) and we can't choose from rack 2 and 3 since they already have other copies, we can still choose from rack 4, and it's ok, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it still can choose from rack 4. But if we only have 3 racks (rack1, rack2, and rack3), and we failed to choose any bookie nodes from rack1, it will randomly choose one from rack2 or rack3, which will lead to 3 replicas located on 2 racks.

IMO, we'd better use the WARN level, because it will choose one bookie node from the whole bookie cluster randomly and it has the risk of the chosen bookie being located on the same rack with existing replicas.

+ "the bookies in the network location are {}, excluded bookies {}, "
+ "current ensemble {}, fallback to choose bookie randomly from the cluster.",
networkLoc, topology.getLeaves(networkLoc), excludeBookies, ensemble);
return selectFromNetworkLocation(excludeRacks, excludeBookies, predicate, ensemble, fallbackToRandom);
}
}
Expand Down