Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize index creation time #1240

Merged
merged 18 commits into from
May 24, 2024
Merged

Optimize index creation time #1240

merged 18 commits into from
May 24, 2024

Commits on May 10, 2024

  1. optimize external sort comparison and reduce index creation time by 10%

    @Ma-cat
    Ma-cat committed May 10, 2024
    Configuration menu
    Copy the full SHA
    778a3db View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. implement the loser tree and use the loser tree to replace the heap t… 

    …o achieve multi-way merge time and improve performance
    @Ma-cat
    Ma-cat committed May 13, 2024
    Configuration menu
    Copy the full SHA
    ff3bbc4 View commit details
    Browse the repository at this point in the history

  2. refactor the loser tree to reduce the impact of function and copy on … 

    …performance
    @Ma-cat
    Ma-cat committed May 13, 2024
    Configuration menu
    Copy the full SHA
    435ce89 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Optimize io for creating index

    @Ma-cat
    Ma-cat committed May 14, 2024
    Configuration menu
    Copy the full SHA
    45a5dcb View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. spill file write with more buf for creating index

    @Ma-cat
    Ma-cat committed May 15, 2024
    Configuration menu
    Copy the full SHA
    8cbb344 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. remove the predict thread and output thread, use mmap for sequential … 

    …reading and merge execution, and try three mmap reading methods
    @Ma-cat
    Ma-cat committed May 17, 2024
    Configuration menu
    Copy the full SHA
    bf8fac9 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. refactor merge and predict, use CycleBuffer read more buf for merge a… 

    …nd perdict
    @Ma-cat
    Ma-cat committed May 20, 2024
    Configuration menu
    Copy the full SHA
    f9b3cb3 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. use queue to output merge results and add TermTupleList to reduce the… 

    … output temporary data size
    @Ma-cat
    Ma-cat committed May 22, 2024
    Configuration menu
    Copy the full SHA
    93b0362 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. optimize predict thread lock granularity

    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    41a8490 View commit details
    Browse the repository at this point in the history

  2. delete some useless code

    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    4aabc75 View commit details
    Browse the repository at this point in the history

  3. delete some useless code

    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    5414da6 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/create-index-optimize-merge' int… 

    …o create-index-optimize-merge
    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    164dc3e View commit details
    Browse the repository at this point in the history

  5. merge upstream main

    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    d53a368 View commit details
    Browse the repository at this point in the history

  6. refactor some code

    @Ma-cat
    Ma-cat committed May 23, 2024
    Configuration menu
    Copy the full SHA
    ada5d1a View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. fix deadlock when parallel create index

    @Ma-cat
    Ma-cat committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c4f7be5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into create-index-optimize-merge

    @Ma-cat
    Ma-cat committed May 24, 2024
    Configuration menu
    Copy the full SHA
    2ddcfe8 View commit details
    Browse the repository at this point in the history

  3. fix infinity only occupies one core during the external_sort stage an… 

    …d delete some useless code
    @Ma-cat
    Ma-cat committed May 24, 2024
    Configuration menu
    Copy the full SHA
    1dcd235 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/create-index-optimize-merge' int… 

    …o create-index-optimize-merge
    @Ma-cat
    Ma-cat committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c9dca99 View commit details
    Browse the repository at this point in the history