Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: implement disk-based sort (Part 1) #13718

Merged
merged 14 commits into from Dec 18, 2019
Merged

Conversation

wshwsh12
Copy link
Contributor

What problem does this PR solve?

part of #12431
This introduces some ListInDisk field and function readPartition to Sort Executor. So that in further pull request, I can implement externalSorting.

What is changed and how it works?

In this pr, I do the following thins.

  1. Spill all data to disk.
  2. Implement readPartition function, and use it to read all data to memory.
  3. Use memory sorting algorithm to sort the data.
  4. Restore the data to disk.
  5. Read data from disk and return.

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

Release note

  • Write release note for bug-fix or new feature.

Support external_sorting.

@wshwsh12
Copy link
Contributor Author

/run-all-tests

@codecov
Copy link

codecov bot commented Nov 25, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@9b7b2c0). Click here to learn what that means.
The diff coverage is 74.6153%.

@@             Coverage Diff             @@
##             master     #13718   +/-   ##
===========================================
  Coverage          ?   80.1468%           
===========================================
  Files             ?        483           
  Lines             ?     121633           
  Branches          ?          0           
===========================================
  Hits              ?      97485           
  Misses            ?      16369           
  Partials          ?       7779

@wshwsh12 wshwsh12 requested a review from qw4990 December 3, 2019 08:19
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qw4990 qw4990 added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 4, 2019
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
Copy link
Contributor

@SunRunAway SunRunAway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SunRunAway SunRunAway added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 18, 2019
@SunRunAway SunRunAway added the status/can-merge Indicates a PR has been approved by a committer. label Dec 18, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Dec 18, 2019

Your auto merge job has been accepted, waiting for 14088, 14111

@sre-bot
Copy link
Contributor

sre-bot commented Dec 18, 2019

/run-all-tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/new-feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants