Skip to content
Permalink
Browse files

[SPARK-24033][SQL] Fix Mismatched of Window Frame specifiedwindowfram…

…e(RowFrame, -1, -1)

## What changes were proposed in this pull request?

When the OffsetWindowFunction's frame is `UnaryMinus(Literal(1))` but the specified window frame has been simplified to `Literal(-1)` by some optimizer rules e.g., `ConstantFolding`. Thus, they do not match and cause the following error:
```
org.apache.spark.sql.AnalysisException: Window Frame specifiedwindowframe(RowFrame, -1, -1) must match the required frame specifiedwindowframe(RowFrame, -1, -1);
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:41)
at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:91)
at
```
## How was this patch tested?
Added a test

Author: gatorsmile <gatorsmile@gmail.com>

Closes #21115 from gatorsmile/fixLag.

(cherry picked from commit 7bc853d)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
  • Loading branch information...
gatorsmile committed Apr 21, 2018
1 parent 8eb64a5 commit d91410029331ff475843ea660b5f98ba15f23892
@@ -363,7 +363,10 @@ abstract class OffsetWindowFunction
override lazy val frame: WindowFrame = {
val boundary = direction match {
case Ascending => offset
case Descending => UnaryMinus(offset)
case Descending => UnaryMinus(offset) match {
case e: Expression if e.foldable => Literal.create(e.eval(EmptyRow), e.dataType)
case o => o
}
}
SpecifiedWindowFrame(RowFrame, boundary, boundary)
}
@@ -402,4 +402,18 @@ class DataFrameWindowFramesSuite extends QueryTest with SharedSQLContext {
Row(7, 3000) :: Row(8, 3000) :: Row(9, 5500) ::
Row(10, 6000) :: Nil)
}

test("SPARK-24033: Analysis Failure of OffsetWindowFunction") {
val ds = Seq((1, 1), (1, 2), (1, 3), (2, 1), (2, 2)).toDF("n", "i")
val res =
Row(1, 1, null) :: Row (1, 2, 1) :: Row(1, 3, 2) :: Row(2, 1, null) :: Row(2, 2, 1) :: Nil
checkAnswer(
ds.withColumn("m",
lead("i", -1).over(Window.partitionBy("n").orderBy("i").rowsBetween(-1, -1))),
res)
checkAnswer(
ds.withColumn("m",
lag("i", 1).over(Window.partitionBy("n").orderBy("i").rowsBetween(-1, -1))),
res)
}
}

0 comments on commit d914100

Please sign in to comment.
You can’t perform that action at this time.