Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(hash-agg): rename distribution keys to group keys #3492

Merged
merged 2 commits into from
Jun 28, 2022

Conversation

BowenXiao1999
Copy link
Contributor

@BowenXiao1999 BowenXiao1999 commented Jun 27, 2022

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

image

As you can see in the code, when serialize "distribution_keys" in FE, it's logical.group_keys(). However in some cases, the dist key might be different from group keys (See the pic above, the hash agg and hash join are in same fragment). So it's better to distinguish dist keys and group keys.

In future, after we done the refactor of state table, might no need to pass dist keys in each StreamNode. Ideally, we can get dist keys from state table and encode it as vnode.

ref: #3456.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov
Copy link

codecov bot commented Jun 27, 2022

Codecov Report

Merging #3492 (88d3389) into main (321c259) will increase coverage by 0.02%.
The diff coverage is 80.00%.

❗ Current head 88d3389 differs from pull request most recent head 8490115. Consider uploading reports for the commit 8490115 to get more accurate results

@@            Coverage Diff             @@
##             main    #3492      +/-   ##
==========================================
+ Coverage   74.40%   74.43%   +0.02%     
==========================================
  Files         768      769       +1     
  Lines      107647   107511     -136     
==========================================
- Hits        80098    80024      -74     
+ Misses      27549    27487      -62     
Flag Coverage Δ
rust 74.43% <80.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/stream/src/from_proto/hash_agg.rs 0.00% <0.00%> (ø)
...rontend/src/optimizer/plan_node/stream_hash_agg.rs 95.94% <100.00%> (ø)
src/object_store/src/object/mem.rs 84.84% <0.00%> (-1.02%) ⬇️
src/frontend/src/binder/expr/function.rs 81.75% <0.00%> (-0.89%) ⬇️
src/common/src/cache.rs 96.73% <0.00%> (-0.85%) ⬇️
...ge/src/hummock/sstable/forward_sstable_iterator.rs 95.12% <0.00%> (-0.72%) ⬇️
src/storage/src/hummock/iterator/backward_user.rs 95.82% <0.00%> (-0.70%) ⬇️
src/storage/src/storage_failpoints/test_sstable.rs 96.62% <0.00%> (-0.46%) ⬇️
src/connector/src/filesystem/file_common.rs 80.35% <0.00%> (-0.45%) ⬇️
src/storage/src/hummock/compactor.rs 77.67% <0.00%> (-0.43%) ⬇️
... and 24 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@fuyufjh
Copy link
Contributor

fuyufjh commented Jun 27, 2022

Can you paste the query? Why there are so many group keys...

@BowenXiao1999
Copy link
Contributor Author

BowenXiao1999 commented Jun 27, 2022

@BowenXiao1999 BowenXiao1999 added the mergify/can-merge Indicates that the PR can be added to the merge queue label Jun 27, 2022
@mergify
Copy link
Contributor

mergify bot commented Jun 27, 2022

Hey @BowenXiao1999, this pull request failed to merge and has been dequeued from the merge train. If you believe your PR failed in the merge train because of a flaky test, requeue it by commenting with @mergifyio requeue. More details can be found on the Queue: Embarked in merge train check-run.

@BowenXiao1999 BowenXiao1999 enabled auto-merge (squash) June 28, 2022 03:10
@BowenXiao1999 BowenXiao1999 merged commit 6e7f1a1 into main Jun 28, 2022
@BowenXiao1999 BowenXiao1999 deleted the bw/fix-rename-proto branch June 28, 2022 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mergify/can-merge Indicates that the PR can be added to the merge queue type/refactor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

frontend: distribution key of StreamNode::HashAgg should be group key
5 participants