New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ddl: Deal with discrete handles #5102
Conversation
ddl/index.go
Outdated
defaultTaskHandleCnt = 128 | ||
maxTaskHandleCnt = 2048 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is too small.
We should also decrease the batch size if the actual count is too large. |
@zimulala |
The threshold to decrease can be 1024 actual keys, but the handle range can be as large as 1 million. |
@coocood PTAL |
ddl/index.go
Outdated
@@ -525,6 +529,13 @@ type handleInfo struct { | |||
endHandle int64 | |||
} | |||
|
|||
func chooseInt64(baseHandle, batch int64) int64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rename it to getEndHandle is better.
ddl/index.go
Outdated
} | ||
} | ||
|
||
func getCountAndHandle(workers []*worker) (int64, int64, bool, error) { | ||
taskAddedCount, nextHandle := int64(0), workers[0].taskRange.startHandle | ||
var err error | ||
var isEnd bool | ||
lessMinWorkers := 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/lessMinWorkers/starvingWorkers/
nextHandle = ret.outOfRangeHandle | ||
isEnd = ret.isAllDone | ||
} | ||
|
||
// Adjust the worker's batch size. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add more comment here to describe why we should increase the batch size. What is the problem we are trying to solve.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
PTAL @shenli
ddl/index.go
Outdated
defaultTaskHandleCnt = 128 | ||
maxTaskHandleCnt = 1 << 20 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add comment here.
LGTM |
/run-all-tests |
/run-unit-test |
* ddl: deal with discrete handles
The test has been added to the function of
TestAddIndex
.