Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disk-based sort executor #12431

Closed
SunRunAway opened this issue Sep 27, 2019 · 0 comments · Fixed by #14279
Closed

Disk-based sort executor #12431

SunRunAway opened this issue Sep 27, 2019 · 0 comments · Fixed by #14279
Assignees
Labels
epic/memory-management help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. sig/execution SIG execution type/enhancement type/new-feature

Comments

@SunRunAway
Copy link
Contributor

SunRunAway commented Sep 27, 2019

Feature Request

Is your feature request related to a problem? Please describe:

The sort executor is keeping all the chunk data in memory. It should be spilled over to disk when mem-quota-query in tidb.toml is exceeded.

Describe the feature you'd like:

After #11607, we can consider using a disk-based chunk data for sort executor avoiding OOM.

It's some tips here,

  1. Implement the external sorting, https://en.wikipedia.org/wiki/External_sorting
  2. Use chunk.ListInDisk when rowChunks needs to spill over to disk.
  3. Trigger the spilling action, there's a similar implementation in hash join

    tidb/executor/hash_table.go

    Lines 276 to 283 in 510a851

    // spillDiskAction implements memory.ActionOnExceed for chunk.List. If
    // the memory quota of a query is exceeded, spillDiskAction.Action is
    // triggered.
    type spillDiskAction struct {
    once sync.Once
    c *hashRowContainer
    fallbackAction memory.ActionOnExceed
    }
    . I've written a slide to demonstrate how Spilling to disk is triggered in hash join, https://docs.google.com/presentation/d/1Sa9xNbDTPnLwnQHLKfpwksdYXWodisXPEqp-WR5Up0U/edit?usp=sharing

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic/memory-management help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. sig/execution SIG execution type/enhancement type/new-feature
Projects
No open projects
SIG Runtime Kanban
  
Done (Historical)
Development

Successfully merging a pull request may close this issue.

2 participants