Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: abstract TransformSortMergeBase to unify merge sort logic. #13927

Merged
merged 2 commits into from Dec 6, 2023

Conversation

RinChanNOWWW
Copy link
Member

@RinChanNOWWW RinChanNOWWW commented Dec 5, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Use TransformSortMergeBase to unify the most logic of TransformSortMerge and TransformSortMergeLimit.

This is also a pre-task of implementing external sort.

TransformSortMergeBase will be wrapped in AccumulatingTransformer. It makes the transform can either outputs block or not while processing the input block:

  • Common path: not output blocks until on_finish.
  • Spill path: output block with spilling information in its block meta.

Tracking in #13889


This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Dec 5, 2023
@RinChanNOWWW RinChanNOWWW marked this pull request as ready for review December 5, 2023 10:27
@RinChanNOWWW RinChanNOWWW added this pull request to the merge queue Dec 6, 2023
Merged via the queue into datafuselabs:main with commit a2c8a80 Dec 6, 2023
68 checks passed
@RinChanNOWWW RinChanNOWWW deleted the sort-merge-base branch December 6, 2023 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants