Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Preceding with negative indices in window function #15783

Closed
wants to merge 4 commits into from

Conversation

somu-imply
Copy link
Contributor

@somu-imply somu-imply commented Jan 30, 2024

This aims to solve #15739 by introducing a change in the native query plan for Windowing query where a query like

WITH virtual_table as (select m2 ,sum(m1) as summ1
from foo group by 1 order by summ1 DESC)
select m2,summ1,sum(summ1) OVER (order by m2 rows between 1 PRECEDING and 2 FOLLOWING) as sumfinal
from virtual_table order by 1

will be planned in the between 1 PRECEDING and 2 FOLLOWING part as

"processor": {
            "type": "framedAgg",
            "frame": {
              "peerType": "ROWS",
              "lowUnbounded": false,
              "lowOffset": -1,
              "uppUnbounded": false,
              "uppOffset": 2,
              "orderBy": null
            },
            "aggregations": [
              {
                "type": "doubleSum",
                "name": "w0",
                "fieldName": "a0"
              }
            ]
          }

This would allow us to support all the cases which postgres supports such as

between 1 PRECEDING and 2 FOLLOWING
between 2 PRECEDING and 1 PRECEDING
between 1 FOLLOWING and 2 FOLLOWING

This will also handle cases with a meaningful error where the upper offset is of a lesser value than the lower offset such as

between 1 PRECEDING and 2 PRECEDING
between 2 FOLLOWING and 1 FOLLOWING

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@somu-imply somu-imply changed the title Initial commit to mark negative indices for preceding in windows Support Preceding with negative indices in window function Jan 30, 2024
@soumyava soumyava added the WIP label Jan 30, 2024
Copy link
Member

@kgyrtkirk kgyrtkirk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seen the topic of the PR - didn't notice it was a draft and taken a quick look.... :)

Comment on lines 362 to 366
int lowerOffset = figureOutOffset(group.lowerBound);
int upperOffset = figureOutOffset(group.upperBound);
if (group.lowerBound.isPreceding()) {
lowerOffset = -lowerOffset;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of adding if-s for both lower/upper - can we make these changes inside figureOutOffset ?

second = val + " FOLLOWING";
}
throw InvalidInput.exception(
"The first value of range in the window [%s] should be lesser than the second value [%s]",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this check should be in DruidSqlValidator ; it will need similar things like #15746

@@ -75,7 +75,7 @@ public RowsAndColumns aggregateAll(
int lowerOffset = frame.getLowerOffset();
int upperOffset = frame.getUpperOffset();

if (numRows < lowerOffset + upperOffset + 1) {
if (numRows < upperOffset - lowerOffset + 1) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should also probably take a look at: WindowFrame#getLowerOffsetClamped
and the callsites of that method in this file

by the way: I think we should retire the old rows processing logic and leave that as well to the one handling the range stuff - it should be on-par in performance; but could handle some edge cases a bit better.

Copy link

github-actions bot commented Apr 1, 2024

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions bot added the stale label Apr 1, 2024
Copy link

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants