Optimize WindowOperator for pre-sorted input #5437

JkSelf · 2023-06-28T06:59:40Z

Spark planner inserts a Sort operator before a Window operator to sort the data.
Hence, there is no need to sort the data again in the window operator.

Add an option to WindowOperator to skip sorting if inputs are already sorted.
This allow WindowOperator to process data in streaming manner without
accumulating all input in memory.

netlify · 2023-06-28T06:59:50Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`e545fc5`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/652487827004b80008b7796b

velox/exec/Window.cpp

mbasmanova

@JkSelf I assume you have a use case where inputs to Window operator are already partitioned and sorted. In this case, you'd like to optimize Window operator to skip partitioning and sorting. To achieve that, I suggest to add a flag to Window operator that indicates that input is already partitioned and sorted and provide an optimized (streaming) implementation that doesn't accumulate all of the input in memory. See StreamingAggregation for an example.

JkSelf · 2023-07-24T02:05:23Z

@JkSelf I assume you have a use case where inputs to Window operator are already partitioned and sorted. In this case, you'd like to optimize Window operator to skip partitioning and sorting. To achieve that, I suggest to add a flag to Window operator that indicates that input is already partitioned and sorted and provide an optimized (streaming) implementation that doesn't accumulate all of the input in memory. See StreamingAggregation for an example.

Sorry for the delay response. Agree with implementing StreamingWindow to skip the sorting, and also can reduce the memory footprint. I pushed the code and can you help to review? Thanks.

mbasmanova

@aditi-pandit Aditi, would you help take a first pass?

velox/exec/LocalPlanner.cpp

velox/exec/StreamingWindow.h

velox/exec/StreamingWindow.cpp

velox/functions/lib/window/tests/WindowTestBase.cpp

velox/exec/StreamingWindow.cpp

JkSelf · 2023-07-25T09:29:43Z

@aditi-pandit Thanks for your review. I have resolved all the comments. Can you help to review again? Thanks.

JkSelf · 2023-07-25T09:30:18Z

@rui-mo Can you help to review? Thanks.

velox/exec/StreamingWindow.h

velox/exec/StreamingWindow.cpp

JkSelf · 2023-07-26T00:55:39Z

@rui-mo Thanks for your review. I have made the modifications. Could you please help with another review?

rui-mo

Just several tiny issues.

velox/exec/Window.cpp

velox/exec/StreamingWindow.h

velox/functions/lib/window/tests/WindowTestBase.cpp

JkSelf · 2023-07-27T09:51:22Z

@aditi-pandit @rui-mo Thanks for your comments. I have updated and can you help to review again?

mbasmanova

@JkSelf Thank you for working on this optimization. Some initial comments.

velox/docs/develop/operators.rst

velox/core/PlanNode.h

velox/core/PlanNode.cpp

velox/exec/tests/utils/PlanBuilder.h

velox/exec/StreamingWindow.h

velox/exec/StreamingWindow.cpp

velox/functions/sparksql/window/tests/StreamingWindowTest.cpp

velox/exec/StreamingWindow.cpp

velox/exec/tests/utils/PlanBuilder.h

velox/exec/Window.cpp

mbasmanova

Thank you for iterating. Looks good % a few nits.

velox/exec/tests/utils/PlanBuilder.h

velox/exec/tests/utils/PlanBuilder.cpp

velox/functions/lib/window/tests/WindowTestBase.cpp

velox/exec/tests/utils/PlanBuilder.cpp

velox/exec/tests/utils/PlanBuilder.h

velox/docs/develop/operators.rst

mbasmanova

@JkSelf Looks great. Just a couple nits remain.

JkSelf · 2023-09-30T00:11:11Z

@JkSelf Looks great. Just a couple nits remain.

@mbasmanova Thanks for your review. I have resolved all your comment. Can you help to review again? Thanks.

JkSelf · 2023-09-30T08:00:42Z

@mbasmanova It seems that the failed unit tests are not related to this pull request. Could you please help to verify this? Thank you.

mbasmanova

@JkSelf Looks good % some typos and a question.

velox/docs/develop/operators.rst

JkSelf · 2023-10-10T01:27:16Z

@mbasmanova The CI is passed. Do you have any further comment? Thanks.

JkSelf · 2023-10-11T01:37:55Z

@mbasmanova Do you have any further comment? Thanks.

mbasmanova

Thanks.

facebook-github-bot · 2023-10-11T21:15:22Z

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

JkSelf · 2023-10-11T23:16:53Z

@mbasmanova Can you help to look the failed testing? Thanks.

facebook-github-bot · 2023-10-12T08:49:11Z

@mbasmanova merged this pull request in 5a34994.

conbench-facebook · 2023-10-12T09:16:52Z

Conbench analyzed the 1 benchmark run on commit 5a34994d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Summary: Spark planner inserts a Sort operator before a Window operator to sort the data. Hence, there is no need to sort the data again in the window operator. Add an option to WindowOperator to skip sorting if inputs are already sorted. This allow WindowOperator to process data in streaming manner without accumulating all input in memory. Pull Request resolved: facebookincubator#5437 Reviewed By: laithsakka Differential Revision: D50198545 Pulled By: mbasmanova fbshipit-source-id: 05ed9ea26691c00310730ceea0153405291023ab

mbasmanova · 2023-10-24T18:21:30Z

@JkSelf @aditi-pandit Would you help cleanup this code in PlanNode.h:

#ifdef VELOX_ENABLE_BACKWARD_COMPATIBILITY
  WindowNode(
      PlanNodeId id,

aditi-pandit · 2023-10-24T20:24:01Z

@JkSelf @aditi-pandit Would you help cleanup this code in PlanNode.h:
#ifdef VELOX_ENABLE_BACKWARD_COMPATIBILITY
  WindowNode(
      PlanNodeId id,

@mbasmanova : This needs Prestissimo changes to be removed. I can follow up.

aditi-pandit · 2023-10-24T21:06:43Z

Prestissimo code was already updated. Sent PR #7223

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 28, 2023

mbasmanova reviewed Jun 28, 2023

View reviewed changes

velox/exec/Window.cpp Outdated Show resolved Hide resolved

mbasmanova reviewed Jun 28, 2023

View reviewed changes

JkSelf force-pushed the window-op1 branch 2 times, most recently from 1aebf93 to c171c0d Compare July 24, 2023 02:03

JkSelf force-pushed the window-op1 branch from c171c0d to dcfb807 Compare July 24, 2023 02:08

JkSelf changed the title ~~Add flag to control whether need to sort in window op~~ Implement Streaming Window to align with Spark Window operator Jul 24, 2023

mbasmanova requested a review from aditi-pandit July 24, 2023 12:50

mbasmanova reviewed Jul 24, 2023

View reviewed changes

velox/exec/LocalPlanner.cpp Outdated Show resolved Hide resolved

aditi-pandit reviewed Jul 24, 2023

View reviewed changes

rui-mo reviewed Jul 25, 2023

View reviewed changes

velox/exec/StreamingWindow.h Outdated Show resolved Hide resolved

velox/exec/StreamingWindow.cpp Outdated Show resolved Hide resolved

velox/exec/StreamingWindow.cpp Outdated Show resolved Hide resolved

JkSelf force-pushed the window-op1 branch from 505aec4 to 2a856f9 Compare July 26, 2023 01:52

rui-mo reviewed Jul 26, 2023

View reviewed changes

velox/exec/Window.cpp Outdated Show resolved Hide resolved

velox/exec/StreamingWindow.h Outdated Show resolved Hide resolved

velox/exec/StreamingWindow.h Outdated Show resolved Hide resolved

velox/exec/StreamingWindow.h Outdated Show resolved Hide resolved

aditi-pandit reviewed Jul 26, 2023

View reviewed changes

mbasmanova reviewed Jul 27, 2023

View reviewed changes

aditi-pandit reviewed Jul 27, 2023

View reviewed changes

velox/exec/StreamingWindow.cpp Outdated Show resolved Hide resolved

aditi-pandit reviewed Jul 28, 2023

View reviewed changes

velox/exec/tests/utils/PlanBuilder.h Outdated Show resolved Hide resolved

JkSelf changed the title ~~Implement Streaming Window to align with Spark Window operator~~ GLUTEN: Implement Streaming Window to align with Spark Window operator Aug 3, 2023

JkSelf changed the title ~~GLUTEN: Implement Streaming Window to align with Spark Window operator~~ [GLUTEN] Implement Streaming Window to align with Spark Window operator Aug 8, 2023

aditi-pandit mentioned this pull request Aug 23, 2023

Add WindowBuild to abstract the RowContainer used for Window operator #6011

Closed

JkSelf changed the title ~~[GLUTEN] Implement Streaming Window to align with Spark Window operator~~ Implement Streaming Window to align with Spark Window operator Aug 24, 2023

JkSelf force-pushed the window-op1 branch from 4621bb9 to fdc6b86 Compare September 4, 2023 03:28

JkSelf commented Sep 4, 2023

View reviewed changes

velox/exec/Window.cpp Outdated Show resolved Hide resolved

mbasmanova reviewed Sep 28, 2023

View reviewed changes

aditi-pandit reviewed Sep 28, 2023

View reviewed changes

velox/exec/tests/utils/PlanBuilder.cpp Outdated Show resolved Hide resolved

JkSelf force-pushed the window-op1 branch from b1070b8 to e393c35 Compare September 29, 2023 07:07

mbasmanova reviewed Sep 29, 2023

View reviewed changes

velox/exec/tests/utils/PlanBuilder.h Outdated Show resolved Hide resolved

mbasmanova reviewed Sep 29, 2023

View reviewed changes

velox/docs/develop/operators.rst Outdated Show resolved Hide resolved

mbasmanova reviewed Sep 29, 2023

View reviewed changes

JkSelf force-pushed the window-op1 branch from e393c35 to 5a6328c Compare September 30, 2023 00:09

JkSelf force-pushed the window-op1 branch from 5a6328c to c971953 Compare September 30, 2023 05:01

mbasmanova reviewed Oct 2, 2023

View reviewed changes

velox/docs/develop/operators.rst Outdated Show resolved Hide resolved

JkSelf force-pushed the window-op1 branch 2 times, most recently from d182907 to e545fc5 Compare October 9, 2023 23:06

Add StreamingWindowBuild support in Window operator

e545fc5

mbasmanova approved these changes Oct 11, 2023

View reviewed changes

mbasmanova changed the title ~~Add StreamingWindowBuild support~~ Optimize WindowOperator for pre-sorted input Oct 11, 2023

facebook-github-bot closed this in 5a34994 Oct 12, 2023

facebook-github-bot added the Merged label Oct 12, 2023

JkSelf mentioned this pull request Oct 13, 2023

[GLUTEN-2961][VL] Integrate with upstream Velox apache/incubator-gluten#3341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize WindowOperator for pre-sorted input #5437

Optimize WindowOperator for pre-sorted input #5437

JkSelf commented Jun 28, 2023 •

edited by mbasmanova

netlify bot commented Jun 28, 2023 •

edited

mbasmanova left a comment

JkSelf commented Jul 24, 2023

mbasmanova left a comment

JkSelf commented Jul 25, 2023

JkSelf commented Jul 25, 2023

JkSelf commented Jul 26, 2023

rui-mo left a comment

JkSelf commented Jul 27, 2023

mbasmanova left a comment

mbasmanova left a comment

mbasmanova left a comment

JkSelf commented Sep 30, 2023

JkSelf commented Sep 30, 2023

mbasmanova left a comment

JkSelf commented Oct 10, 2023

JkSelf commented Oct 11, 2023

mbasmanova left a comment

facebook-github-bot commented Oct 11, 2023

JkSelf commented Oct 11, 2023

facebook-github-bot commented Oct 12, 2023

conbench-facebook bot commented Oct 12, 2023

mbasmanova commented Oct 24, 2023

aditi-pandit commented Oct 24, 2023

aditi-pandit commented Oct 24, 2023 •

edited

Optimize WindowOperator for pre-sorted input #5437

Optimize WindowOperator for pre-sorted input #5437

Conversation

JkSelf commented Jun 28, 2023 • edited by mbasmanova

netlify bot commented Jun 28, 2023 • edited

✅ Deploy Preview for meta-velox canceled.

mbasmanova left a comment

Choose a reason for hiding this comment

JkSelf commented Jul 24, 2023

mbasmanova left a comment

Choose a reason for hiding this comment

JkSelf commented Jul 25, 2023

JkSelf commented Jul 25, 2023

JkSelf commented Jul 26, 2023

rui-mo left a comment

Choose a reason for hiding this comment

JkSelf commented Jul 27, 2023

mbasmanova left a comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

JkSelf commented Sep 30, 2023

JkSelf commented Sep 30, 2023

mbasmanova left a comment

Choose a reason for hiding this comment

JkSelf commented Oct 10, 2023

JkSelf commented Oct 11, 2023

mbasmanova left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Oct 11, 2023

JkSelf commented Oct 11, 2023

facebook-github-bot commented Oct 12, 2023

conbench-facebook bot commented Oct 12, 2023

mbasmanova commented Oct 24, 2023

aditi-pandit commented Oct 24, 2023

aditi-pandit commented Oct 24, 2023 • edited

JkSelf commented Jun 28, 2023 •

edited by mbasmanova

netlify bot commented Jun 28, 2023 •

edited

aditi-pandit commented Oct 24, 2023 •

edited