Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
shuffle sharding package for priority and fairness #83665
What type of PR is this?
What this PR does / why we need it:
Implement package for shuffle sharding algorithm, which will be used in KEP priority and fariness .
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
Implement several shuffle sharding functions including ShuffleAndDeal, ShuffleAndDealToSlice. Add benchmarks and tests for shuffle sharding to test performance, correctness and distribution uniformity. Signed-off-by: Bruce Ma <firstname.lastname@example.org>
Changes following up on shuffle sharding util package. Made the validation checking function return a slice of error messages rather than just a bit. Replaced all the `int32` with `int` because this is intended for more than just the priority-and-faireness feature and so should not be a slave to its configuration datatypes. Introduced ShuffleAndDealIntoHand, to make memory allocation the caller's problem/privilege. Made the hand uniformity tester avoid reflection, evaluate the histogram against the expected range of counts, and run multiple test cases, including one in which the number of hash values is a power of two with four extra bits (as the validation check requires) and one in which the deck size is not a power of two.
1 similar comment
Clean up useless functions, only keep the basic function Deal and the function DealIntoHand which will be used by Priority and Fairness. Improve some comments for constants and functions. Introduce Dealer to combine parameters and methods into a whole. Use fixed-size slice to improve performance. Use math.Ceil and math.Log2 to calculate required entropy bits. Make the given hand adaptive to handSize in DealIntoHand. Signed-off-by: Bruce Ma <email@example.com>
FYI, I wondered whether returning the slice would confuse the compiler's escape analysis. To investigate that, I ran the unit tests with
Looking more carefully at the escape analysis remarks, I see the following. The unit test file makes three calls to DealIntoHand. The last two are coded in such a way that the given hand escapes to the heap regardless of what DealIntoHand does, and the analysis says that the hand escapes. The first one is not coded in that way, but the given hand is always empty (
I extended the last test function with some code that uses a non-empty given hand, and the analysis said it does not leak. Here is the code:
and here is what the analysis said:
[APPROVALNOTIFIER] This PR is APPROVED
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing