[SPARK-34807][SQL] Transpose Window nodes with Project between them #31980

tanelk · 2021-03-27T13:26:39Z

What changes were proposed in this pull request?

Extend the TransposeWindow rule to transpose Window nodes, that have Project between them.

Why are the changes needed?

The analyzer will turn a dataset.withColumn("colName", expressionWithWindowFunction) method call to a Project - Window - Project chain in the logical plan. When this method is called multiple times in a row, then the projects can block the Window nodes from being transposed by the current TransposeWindow rule.

TPCDS q47 and q57 are also improved by this.

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT

tanelk · 2021-03-27T13:27:02Z

cc @wangyum

SparkQA · 2021-03-27T14:14:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41178/

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

SparkQA · 2021-03-27T14:56:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41178/

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

SparkQA · 2021-03-27T17:49:50Z

Test build #136594 has finished for PR 31980 at commit b9c38a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

SparkQA · 2021-03-28T05:14:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41187/

SparkQA · 2021-03-28T05:21:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41187/

SparkQA · 2021-03-28T08:43:17Z

Test build #136604 has finished for PR 31980 at commit bcf0b09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

SparkQA · 2021-03-28T15:09:26Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41198/

SparkQA · 2021-03-28T19:01:32Z

Test build #136616 has finished for PR 31980 at commit 8f15d15.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-04-02T07:49:01Z

This looks fine too but it would be great if we can have a sign-off from @hvanhovell too .. he has much better insight than I have

SparkQA · 2021-04-02T09:24:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41425/

SparkQA · 2021-04-02T09:24:57Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41425/

SparkQA · 2021-04-02T12:57:26Z

Test build #136847 has finished for PR 31980 at commit 194b5bf.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-03T11:26:11Z

Test build #136879 has finished for PR 31980 at commit 194b5bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-24T17:07:12Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43400/

SparkQA · 2021-05-24T20:38:29Z

Test build #138877 has finished for PR 31980 at commit faa17f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…pose_window

SparkQA · 2021-06-11T06:44:53Z

Test build #139691 has started for PR 31980 at commit b0b64a3.

SparkQA · 2021-06-11T11:42:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44215/

SparkQA · 2021-06-11T12:14:14Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44215/

SparkQA · 2021-06-16T16:53:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44411/

SparkQA · 2021-06-16T17:27:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44411/

SparkQA · 2021-06-16T20:10:20Z

Test build #139881 has finished for PR 31980 at commit 1b0553e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tanelk · 2021-06-23T21:09:15Z

@maropu
This is antoher slight improvement to an optimizer rule, that has been stuck in approved state for a quite while.
Could you take another look and perhaps we can merge this?

SparkQA · 2021-06-23T22:22:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44751/

SparkQA · 2021-06-23T22:30:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44751/

SparkQA · 2021-06-24T01:54:47Z

Test build #140223 has finished for PR 31980 at commit 60c6643.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class MergedBlockMetaRequest extends AbstractMessage implements RequestMessage
public class MergedBlockMetaSuccess extends AbstractResponseMessage
public abstract class AbstractFetchShuffleBlocks extends BlockTransferMessage
public class FetchShuffleBlockChunks extends AbstractFetchShuffleBlocks
public class FetchShuffleBlocks extends AbstractFetchShuffleBlocks
case class ShuffleBlockInfo(shuffleId: Int, mapId: Long)
class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String])
class DecimalOps(FractionalOps):
class IntegralExtensionOps(IntegralOps):
class FractionalExtensionOps(FractionalOps):
class StringExtensionOps(StringOps):
class GroupBy(Generic[T_Frame], metaclass=ABCMeta):
class DataFrameGroupBy(GroupBy[DataFrame]):
class SeriesGroupBy(GroupBy[Series]):
class SparkIndexOpsMethods(Generic[T_IndexOps], metaclass=ABCMeta):
class SparkSeriesMethods(SparkIndexOpsMethods[\"ps.Series\"]):
class SparkIndexMethods(SparkIndexOpsMethods[\"ps.Index\"]):
case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression
case class Cast(
class ExpressionContainmentOrdering extends Ordering[Expression]
case class GetTimestampWithoutTZ(
case class ParseToTimestampWithoutTZ(
case class MakeDTInterval(
case class MakeYMInterval(years: Expression, months: Expression)
case class DayTimeIntervalType(startField: Byte, endField: Byte) extends AtomicType
case class YearMonthIntervalType(startField: Byte, endField: Byte) extends AtomicType
final class ParquetReadState
public class ParquetVectorUpdaterFactory
case class CommandResult(
case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomSizeMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomTimingMetric(name: String, desc: String) extends StateStoreCustomMetric
trait TestGroupState[S] extends GroupState[S]

wangyum · 2021-06-24T02:31:50Z

Merged to master.

…995) Extend the `TransposeWindow` rule to transpose `Window` nodes, that have `Project` between them. The analyzer will turn a `dataset.withColumn("colName", expressionWithWindowFunction)` method call to a `Project - Window - Project` chain in the logical plan. When this method is called multiple times in a row, then the projects can block the `Window` nodes from being transposed by the current `TransposeWindow` rule. TPCDS q47 and q57 are also improved by this. No UT Closes #31980 from tanelk/SPARK-34807_transpose_window. Lead-authored-by: tanel.kiis@gmail.com <tanel.kiis@gmail.com> Co-authored-by: Tanel Kiis <tanel.kiis@gmail.com> Signed-off-by: Yuming Wang <yumwang@ebay.com> Co-authored-by: Tanel Kiis <tanel.kiis@gmail.com>

Transpose windows with project between

b9c38a9

github-actions bot added the SQL label Mar 27, 2021

wangyum reviewed Mar 27, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Show resolved Hide resolved

wangyum reviewed Mar 27, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Show resolved Hide resolved

wangyum reviewed Mar 28, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Outdated Show resolved Hide resolved

Address comments

bcf0b09

wangyum approved these changes Mar 28, 2021

View reviewed changes

wangyum reviewed Mar 28, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Show resolved Hide resolved

wangyum reviewed Mar 28, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Show resolved Hide resolved

Address comments

8f15d15

wangyum requested review from cloud-fan and maropu March 29, 2021 02:14

wangyum mentioned this pull request Mar 30, 2021

[SPARK-34565][SQL] Collapse Window nodes with Project between them #31677

Closed

Rerun plan stability suite

194b5bf

tanelk added 2 commits May 24, 2021 18:46

Merge master

58ff0e9

Remove redundant import

faa17f6

Merge remote-tracking branch 'upstream/master' into SPARK-34807_trans…

b0b64a3

…pose_window

Trigger Build

1b0553e

Rerun plan stability suite

60c6643

wangyum closed this in b3a2ceb Jun 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-34807][SQL] Transpose Window nodes with Project between them #31980

[SPARK-34807][SQL] Transpose Window nodes with Project between them #31980

tanelk commented Mar 27, 2021

tanelk commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

HyukjinKwon commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 3, 2021

SparkQA commented May 24, 2021

SparkQA commented May 24, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 16, 2021

SparkQA commented Jun 16, 2021

SparkQA commented Jun 16, 2021

tanelk commented Jun 23, 2021

SparkQA commented Jun 23, 2021

SparkQA commented Jun 23, 2021

SparkQA commented Jun 24, 2021

wangyum commented Jun 24, 2021

[SPARK-34807][SQL] Transpose Window nodes with Project between them #31980

[SPARK-34807][SQL] Transpose Window nodes with Project between them #31980

Conversation

tanelk commented Mar 27, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

tanelk commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 27, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

SparkQA commented Mar 28, 2021

HyukjinKwon commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 3, 2021

SparkQA commented May 24, 2021

SparkQA commented May 24, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 11, 2021

SparkQA commented Jun 16, 2021

SparkQA commented Jun 16, 2021

SparkQA commented Jun 16, 2021

tanelk commented Jun 23, 2021

SparkQA commented Jun 23, 2021

SparkQA commented Jun 23, 2021

SparkQA commented Jun 24, 2021

wangyum commented Jun 24, 2021