Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize index creation time #1240

Merged
merged 18 commits into from
May 24, 2024
Merged

Optimize index creation time #1240

merged 18 commits into from
May 24, 2024

Conversation

Ma-cat
Copy link
Contributor

@Ma-cat Ma-cat commented May 23, 2024

What problem does this PR solve?

  • Use loser tree to replace normal heap
  • Optimize io and use mmap to replace fread
  • Reconstruct merge and predic threads using ring queues
  • Reconstruct merge and output threads using output queues
  • Adjust the intermediate file output format and reduce the size of the intermediate file
  • Reconstruct the locks between merge, predic, and output threads to reduce lock granularity

Type of change

  • Refactoring
  • Performance Improvement
  • Test cases

Ma-cat added 13 commits May 10, 2024 16:55
@Ma-cat
optimize external sort comparison and reduce index creation time by 10%
778a3db
@Ma-cat
implement the loser tree and use the loser tree to replace the heap t…
ff3bbc4 
…o achieve multi-way merge time and improve performance
@Ma-cat
refactor the loser tree to reduce the impact of function and copy on …
435ce89 
…performance
@Ma-cat
Optimize io for creating index
45a5dcb
@Ma-cat
spill file write with more buf for creating index
8cbb344
@Ma-cat
remove the predict thread and output thread, use mmap for sequential …
bf8fac9 
…reading and merge execution, and try three mmap reading methods
@Ma-cat
refactor merge and predict, use CycleBuffer read more buf for merge a…
f9b3cb3 
…nd perdict
@Ma-cat
use queue to output merge results and add TermTupleList to reduce the…
93b0362 
… output temporary data size
@Ma-cat
optimize predict thread lock granularity
41a8490
@Ma-cat
delete some useless code
4aabc75
@Ma-cat
delete some useless code
5414da6
@Ma-cat
Merge remote-tracking branch 'origin/create-index-optimize-merge' int…
164dc3e 
…o create-index-optimize-merge
@Ma-cat
merge upstream main
d53a368
@Ma-cat Ma-cat added the ci PR can be test label May 23, 2024
yuzhichang
yuzhichang reviewed May 23, 2024
View reviewed changes
src/storage/invertedindex/column_inverter.cpp Show resolved Hide resolved
src/storage/invertedindex/common/buf_writer.cppm Outdated Show resolved Hide resolved
src/storage/invertedindex/common/buf_writer.cppm Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cpp Outdated Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cpp Outdated Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cpp Outdated Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cppm Outdated Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cppm Show resolved Hide resolved
src/storage/invertedindex/common/loser_tree.cppm Show resolved Hide resolved
src/storage/invertedindex/common/external_sort_merger.cpp Show resolved Hide resolved
@yuzhichang yuzhichang merged commit 42be3d7 into infiniflow:main May 24, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yuzhichang yuzhichang yuzhichang left review comments

Assignees
No one assigned
Labels
ci PR can be test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Ma-cat @yuzhichang