Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hail] Prevent rewriting of TableAggregateByKey after key shortening #7901

Merged
merged 2 commits into from Jan 17, 2020

Conversation

tpoterba
Copy link
Contributor

No description provided.

@@ -711,7 +711,7 @@ object Simplify {
&& child.typ.key.nonEmpty && canRepartition =>
TableAggregateByKey(child, expr)

case TableAggregateByKey(TableKeyBy(child, keys, _), expr) if canRepartition =>
case TableAggregateByKey(TableKeyBy(child, keys, false), expr) if canRepartition && ! child.typ.key.startsWith(keys) =>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If child.typ.key.startsWith(keys), then TableKeyBy will ignore the isSorted flag, so I don't think you want to match on isSorted = false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want both. If we're lengthening the key without shuffling, we should not rewrite this as a shuffling aggregation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I get it now. I still think it's clearer matching on TableKeyBy(child, keys, _), since !x.definitelyDoesNotShuffle already implies that x.isSorted = false, but it's not a big deal. Approving.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@tpoterba tpoterba dismissed patrick-schultz’s stale review January 17, 2020 17:40

I think I clarified the code

@danking danking merged commit e047f54 into hail-is:master Jan 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants