Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

indexMerge with limit may cause tidb server OOM #51188

Closed
wjhuang2016 opened this issue Feb 20, 2024 · 1 comment · Fixed by #51284
Closed

indexMerge with limit may cause tidb server OOM #51188

wjhuang2016 opened this issue Feb 20, 2024 · 1 comment · Fixed by #51284
Assignees

Comments

@wjhuang2016
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

(pprof) list NewHandleHeap
Total: 19.90GB
ROUTINE ======================== github.com/pingcap/tidb/pkg/executor.(*indexMergeProcessWorker).NewHandleHeap in /Users/bba/pc/tidb/pkg/executor/index_merge_reader.go
   19.33GB    19.33GB (flat, cum) 97.13% of Total
         .          .   1001:func (w *indexMergeProcessWorker) NewHandleHeap(taskMap map[int][]*indexMergeTableTask, memTracker *memory.Tracker) *handleHeap {
         .          .   1002:	compareFuncs := make([]chunk.CompareFunc, 0, len(w.indexMerge.byItems))
         .          .   1003:	for _, item := range w.indexMerge.byItems {
         .          .   1004:		keyType := item.Expr.GetType()
         .          .   1005:		compareFuncs = append(compareFuncs, chunk.GetCompareFunc(keyType))
         .          .   1006:	}
         .          .   1007:
         .          .   1008:	requiredCnt := uint64(0)
         .          .   1009:	if w.indexMerge.pushedLimit != nil {
         .          .   1010:		requiredCnt = max(requiredCnt, w.indexMerge.pushedLimit.Count+w.indexMerge.pushedLimit.Offset)
         .          .   1011:	}
         .          .   1012:	return &handleHeap{
         .          .   1013:		requiredCnt: requiredCnt,
         .          .   1014:		tracker:     memTracker,
         .          .   1015:		taskMap:     taskMap,
   19.33GB    19.33GB   1016:		idx:         make([]rowIdx, 0, requiredCnt),
         .          .   1017:		compareFunc: compareFuncs,
         .          .   1018:		byItems:     w.indexMerge.byItems,
         .          .   1019:	}
         .          .   1020:}
         .          .   1021:

2. What did you expect to see? (Required)

No OOM

3. What did you see instead (Required)

19.33GB 19.33GB 1016: idx: make([]rowIdx, 0, requiredCnt),

4. What is your TiDB version? (Required)

1fc92b3

@wjhuang2016 wjhuang2016 added type/bug This issue is a bug. fuzz/randomtest labels Feb 20, 2024
@wjhuang2016 wjhuang2016 changed the title indexMerge with limit may caused tidb server OOM indexMerge with limit may cause tidb server OOM Feb 20, 2024
@Defined2014
Copy link
Contributor

This only happens if the limit number is very large. It would also OOM if we had that many rows. So the problem only occurs when limit number is very large and the number of rows are much less. Change its serverity to moderate rather than major.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants