Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(frontend): distribute index by arrange key #2056

Merged
merged 2 commits into from
Apr 22, 2022
Merged

Conversation

skyzh
Copy link
Contributor

@skyzh skyzh commented Apr 22, 2022

Signed-off-by: Alex Chi iskyzh@gmail.com

What's changed and what's your intention?

As title

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

close #2017

Signed-off-by: Alex Chi <iskyzh@gmail.com>
@codecov
Copy link

codecov bot commented Apr 22, 2022

Codecov Report

Merging #2056 (838b039) into main (58b97fa) will decrease coverage by 0.00%.
The diff coverage is 77.27%.

@@            Coverage Diff             @@
##             main    #2056      +/-   ##
==========================================
- Coverage   70.95%   70.95%   -0.01%     
==========================================
  Files         633      633              
  Lines       81175    81192      +17     
==========================================
+ Hits        57601    57609       +8     
- Misses      23574    23583       +9     
Flag Coverage Δ
rust 70.95% <77.27%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/frontend/src/handler/create_index.rs 0.00% <0.00%> (ø)
...tend/src/optimizer/plan_node/stream_materialize.rs 89.30% <80.00%> (-0.37%) ⬇️
src/frontend/src/optimizer/mod.rs 88.57% <81.25%> (-5.22%) ⬇️
src/meta/src/barrier/mod.rs 69.33% <0.00%> (+0.33%) ⬆️
src/connector/src/filesystem/file_common.rs 80.88% <0.00%> (+0.44%) ⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

) -> Result<Self> {
// ensure the same pk will not shuffle to different node
let input = match input.distribution() {
Distribution::Single => input,
_ => Distribution::HashShard(input.pk_indices().to_vec())
.enforce_if_not_satisfies(input, Order::any()),
_ => Distribution::HashShard(if distribute_only_order_by {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks somehow strange to me... I think this should be handled by the caller, not here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Infer distribution is always done inside operators?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. Also makes sense

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact I'm also considering making Materialize to be AnyShard, and let the create mv / index statement to decide the distribution by specifying it in PlanRoot. But in this case, we might need some redesign.

# TODO: support drop index

statement ok
drop index iii_index_t1;
drop materialized view iii_index_t1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May let drop index act as an alias of drop materialized view in the future

Signed-off-by: Alex Chi <iskyzh@gmail.com>
@skyzh skyzh merged commit 2eaa9ff into main Apr 22, 2022
@skyzh skyzh deleted the skyzh/index-distribution branch April 22, 2022 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

frontend: fix distribution of index materialize
2 participants