New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-46442][SQL] DS V2 supports push down PERCENTILE_CONT and PERCENTILE_DISC #44397
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -42,7 +42,7 @@ private[sql] object H2Dialect extends JdbcDialect { | |
|
||
private val distinctUnsupportedAggregateFunctions = | ||
Set("COVAR_POP", "COVAR_SAMP", "CORR", "REGR_INTERCEPT", "REGR_R2", "REGR_SLOPE", "REGR_SXY", | ||
"MODE") | ||
"MODE", "PERCENTILE_CONT", "PERCENTILE_DISC") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need this change? H2 dialect deals with these two functions in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Because H2 dialect overrides the
|
||
|
||
private val supportedAggregateFunctions = Set("MAX", "MIN", "SUM", "COUNT", "AVG", | ||
"VAR_POP", "VAR_SAMP", "STDDEV_POP", "STDDEV_SAMP") ++ distinctUnsupportedAggregateFunctions | ||
|
@@ -270,18 +270,7 @@ private[sql] object H2Dialect extends JdbcDialect { | |
throw new UnsupportedOperationException(s"${this.getClass.getSimpleName} does not " + | ||
s"support aggregate function: $funcName with DISTINCT") | ||
} else { | ||
funcName match { | ||
case "MODE" => | ||
// Support Mode only if it is deterministic or reverse is defined. | ||
assert(inputs.length == 2) | ||
if (inputs.last == "true") { | ||
s"MODE() WITHIN GROUP (ORDER BY ${inputs.head})" | ||
} else { | ||
s"MODE() WITHIN GROUP (ORDER BY ${inputs.head} DESC)" | ||
} | ||
case _ => | ||
super.visitAggregateFunction(funcName, isDistinct, inputs) | ||
} | ||
super.visitAggregateFunction(funcName, isDistinct, inputs) | ||
} | ||
|
||
override def visitExtract(field: String, source: String): String = { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how do we translate ASC/DESC?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please refer
visitSortOrder
.