Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query: Add support for GroupBy patterns beyond aggregate #25495

Merged
1 commit merged into from
Aug 17, 2021
Merged

Conversation

smitpatel
Copy link
Member

@smitpatel smitpatel commented Aug 12, 2021

Overall approach:
A grouping element (the range variable you get after applying GroupBy operator) is of type IGrouping<TKey, TElement> which implements IEnumerable<TElement>. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks.
During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted.
Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other.

Bug fixes:

  • Fix a bug in identifying single result in InMemory to convert it to enumerable
  • Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied
  • Throw better error message when translating Min/Max over an entity type for InMemory

@smitpatel smitpatel force-pushed the smit/groupbyabc branch 4 times, most recently from 8c2317f to ab7f83c Compare August 12, 2021 18:18
@smitpatel smitpatel changed the title The magical group by Query: Add support for GroupBy patterns beyond aggregate Aug 12, 2021
@smitpatel smitpatel marked this pull request as ready for review August 12, 2021 19:25
@smitpatel smitpatel requested review from maumar and roji August 12, 2021 19:59
Copy link
Member

@roji roji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was somewhat less bewildered than expected! Here are some nits, LGTM but will let @maumar do final approval.

@ghost
Copy link

ghost commented Aug 16, 2021

Hello @smitpatel!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ghost
Copy link

ghost commented Aug 16, 2021

Apologies, while this PR appears ready to be merged, I've been configured to only merge when all checks have explicitly passed. The following integrations have not reported any progress on their checks and are blocking auto-merge:

  1. Azure Pipelines

These integrations are possibly never going to report a check, and unblocking auto-merge likely requires a human being to update my configuration to exempt these integrations from requiring a passing check.

Give feedback on this
From the bot dev team

We've tried to tune the bot such that it posts a comment like this only when auto-merge is blocked for exceptional, non-intuitive reasons. When the bot's auto-merge capability is properly configured, auto-merge should operate as you would intuitively expect and you should not see any spurious comments.

Please reach out to us at fabricbotservices@microsoft.com to provide feedback if you believe you're seeing this comment appear spuriously. Please note that we usually are unable to update your bot configuration on your team's behalf, but we're happy to help you identify your bot admin.

- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609
- Translate FirstOrDefault over grouping element Fixes #12088
- Add ability to select N element over grouping element Fixes #13805

Overall approach:
A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks.
During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted.
Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other.

Bug fixes:
- Fix a bug in identifying single result in InMemory to convert it to enumerable
- Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied
- Throw better error message when translating Min/Max over an entity type for InMemory
@ghost
Copy link

ghost commented Aug 16, 2021

Apologies, while this PR appears ready to be merged, I've been configured to only merge when all checks have explicitly passed. The following integrations have not reported any progress on their checks and are blocking auto-merge:

  1. Azure Pipelines

These integrations are possibly never going to report a check, and unblocking auto-merge likely requires a human being to update my configuration to exempt these integrations from requiring a passing check.

Give feedback on this
From the bot dev team

We've tried to tune the bot such that it posts a comment like this only when auto-merge is blocked for exceptional, non-intuitive reasons. When the bot's auto-merge capability is properly configured, auto-merge should operate as you would intuitively expect and you should not see any spurious comments.

Please reach out to us at fabricbotservices@microsoft.com to provide feedback if you believe you're seeing this comment appear spuriously. Please note that we usually are unable to update your bot configuration on your team's behalf, but we're happy to help you identify your bot admin.

@ghost ghost merged commit 9c59a77 into main Aug 17, 2021
@ghost ghost deleted the smit/groupbyabc branch August 17, 2021 00:34
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants