Skip to content

feat(agg index): initial impl refresh index - #11634

Merged
ariesdevil merged 9 commits into
mainfrom
refresh-index_ariesdevil
Jun 21, 2023
Merged

feat(agg index): initial impl refresh index#11634
ariesdevil merged 9 commits into
mainfrom
refresh-index_ariesdevil

Conversation

@ariesdevil

@ariesdevil ariesdevil commented May 31, 2023

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Initial impl refresh index.

Closes #issue

@vercel

vercel Bot commented May 31, 2023

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
databend ⬜️ Ignored (Inspect) Visit Preview Jun 21, 2023 3:51am

@mergify mergify Bot added the pr-feature this PR introduces a new feature to the codebase label May 31, 2023
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 3 times, most recently from fa351cc to 7023976 Compare June 7, 2023 18:12
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 2 times, most recently from d48e7ec to 9a031a8 Compare June 12, 2023 12:02
Comment thread src/query/ast/src/ast/statements/index.rs Outdated
Comment thread src/query/sql/src/executor/physical_plan.rs Outdated
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 3 times, most recently from c19e8ea to 783cbcb Compare June 13, 2023 08:27
@ariesdevil
ariesdevil marked this pull request as ready for review June 13, 2023 08:37
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 2 times, most recently from a5ed823 to 3968e9a Compare June 13, 2023 10:26
@bohutang

Copy link
Copy Markdown
Member

Please add the summary about this PR :)

@bohutang

Copy link
Copy Markdown
Member

From the PR codes, it looks like a new feature, is it ready to add logic test for this PR or next?

@ariesdevil

ariesdevil commented Jun 13, 2023

Copy link
Copy Markdown
Contributor Author

From the PR codes, it looks like a new feature, is it ready to add logic test for this PR or next?↳

The logic tests are not suite for refresh index, we need design new test for this feature, this PR will merge first so that the it not stuck the read side by @RinChanNOWWW

I'll continue work on refresh index includes: partial refresh by limit, add new field to index meta, add metrics, and so on.

@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch from 3968e9a to f1cd678 Compare June 13, 2023 12:03
@ariesdevil

Copy link
Copy Markdown
Contributor Author

I have encountered some problems in the development process on the correspondence between blocks and underlying files, and I especially appreciate @sundy-li @RinChanNOWWW @zhang2014 help, although some of the suggestions are not used in the end :)

@RinChanNOWWW

Copy link
Copy Markdown
Contributor

Better to add some unit tests to mock the index writing.

Comment thread src/query/service/src/interpreters/interpreter_index_refresh.rs Outdated
Comment thread src/query/service/src/interpreters/interpreter_index_refresh.rs
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 4 times, most recently from d21d19f to 8642b74 Compare June 18, 2023 15:07
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch 3 times, most recently from 655ab4a to d64cfc6 Compare June 19, 2023 13:46
@ariesdevil
ariesdevil requested a review from RinChanNOWWW June 19, 2023 14:45
Comment thread src/query/storages/fuse/src/operations/agg_index_sink.rs Outdated
Comment thread src/query/service/src/interpreters/interpreter_index_refresh.rs
Comment thread src/query/service/src/pipelines/pipeline_builder.rs Outdated
Comment thread src/query/ee/tests/it/aggregating_index/index_refresh.rs Outdated
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch from d64cfc6 to cec5a43 Compare June 20, 2023 10:36
@ariesdevil
ariesdevil requested a review from RinChanNOWWW June 20, 2023 10:36
Comment thread src/query/storages/fuse/src/operations/agg_index_sink.rs
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch from cec5a43 to 92f0be4 Compare June 20, 2023 14:58
@ariesdevil
ariesdevil requested a review from sundy-li June 20, 2023 14:58
@ariesdevil
ariesdevil force-pushed the refresh-index_ariesdevil branch from 92f0be4 to fff83e7 Compare June 20, 2023 15:23
@ariesdevil
ariesdevil merged commit c08d1f1 into main Jun 21, 2023
@ariesdevil
ariesdevil deleted the refresh-index_ariesdevil branch June 21, 2023 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants