[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987

dongjoon-hyun · 2016-11-23T04:13:17Z

What changes were proposed in this pull request?

This PR aims to support comparators, e.g. '<', '<=', '>', '>=', again in Apache Spark 2.0 for backward compatibility.

Spark 1.6

scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame = [result: string]

scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
res1: org.apache.spark.sql.DataFrame = [result: string]

Spark 2.0

scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame = []

scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '<' expecting {')', ','}(line 1, pos 42)

After this PR, it's supported.

How was this patch tested?

Pass the Jenkins test with a newly added testcase.

dongjoon-hyun · 2016-11-23T04:22:59Z

Hi, @hvanhovell .
Could you review this PR?

This is the first attempt to use UnresolvedAttribute and Analyzer rule. There are two debatable issues.

Catalyst Analyzer doesn't know AlterTableDropPartitions, so I need to introduce CommandWithExpression trait here.
I added a testcase for atomic types, but we need to change ADD PARTITION, too. ADD PARTITION relates the several more parts. If possible, I want to make it as a separate PR.

SparkQA · 2016-11-23T05:38:14Z

Test build #69050 has finished for PR 15987 at commit 970a904.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CommandWithExpression extends LeafNode

rxin · 2016-11-23T06:20:10Z

Why does it cast it to double? The fix looks pretty weird.

dongjoon-hyun · 2016-11-24T00:15:39Z

Thank you for review, @rxin . In the expression, the partition attribute are assumed String, so Analyzer injects that.

I tried to implement the following recommended way in JIRA issue, but the PR may choose not a clean way. Do you mean CommandWithExpression in command.scala and Analyzer.scala?

The Analyzer is injecting casts because the dataType of the partition attribute is a String:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L223
This is not trivial to solve. If we want to properly fix this, we need to use UnresolvedAttributes and do some add some analysis for this command. Another - hackier - way of getting there would, to explicitly exclude this command from analysis.
I am personally a proponent of option 1.

SparkQA · 2016-11-30T06:53:44Z

Test build #69383 has finished for PR 15987 at commit 9db07af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CommandWithExpression extends LeafNode

SparkQA · 2016-11-30T10:55:29Z

Test build #69397 has finished for PR 15987 at commit c35aeab.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CommandWithExpression extends LeafNode

dongjoon-hyun · 2016-11-30T11:12:39Z

Hi, @hvanhovell .
Could you review this PR? This PR now is the new fix + the reverted commit.

dongjoon-hyun · 2016-11-30T11:17:26Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

+        // We should not do this here in order to support general CommandWithExpression
+        if (catalogTable.tableType == CatalogTableType.VIEW) {
+          throw new AnalysisException(
+            "Cannot alter a view with ALTER TABLE. Please use ALTER VIEW instead")


Previously, run function of Command does every error handlings including this kind of errors.
Now, we should do that here because this exception has higher priority than unresolved attributes.

Except AlterTableDropPartitionCommand, do we have other running commands which need this resolving? If not, I think we may not need to add this to Analyzer and create new abstraction CommandWithExpression.

Yes. There is no such command.
AlterTableDropPartitionCommand is not visible in catalyst. So, it's inevitable to create some corresponding one in catalyst.

This is an real issue. So, @rxin told me it's a weird.
Until now, I didn't find a better way to support expressions on Command.
I'm asking the committers not to use expressions here.

dongjoon-hyun · 2016-11-30T18:25:23Z

Hi, @hvanhovell , @gatorsmile , @rxin .

Should we remove Expression related stuff?
Originally, #15302 does not have this kind of expression issue. It's just an extension of the current Spark.

But, while supporting expressions in partition spec, I made #15704 . but it's reverted now. IMO, the followings can be different issues.

SUPPORT DROP BY FILTER ([SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15302)
SUPPORT EXPRESSION ON PARTITION SPEC

Especially, Apache Spark Command is not designed to carry expression, so it seems to make things complex and weird like in this PR.
Also, for 2, we need to support expressions on ADD PARTITION, too. For example, current Spark do the following, but Hive makes a partition p=10.

scala> sql("create table tbl_x (a int) partitioned by (p int)")
scala> sql("alter table tbl_x add partition (p=10.0)")
scala> sql("show partitions tbl_x").show
+---------+
|partition|
+---------+
|   p=10.0|
+---------+

In short, this is beyond of 1, I think we had better go back the old version (without expression.) How do you think about this?

…port comparators This PR aims to support `comparators`, e.g. '<', '<=', '>', '>=', again in Apache Spark 2.0 for backward compatibility. **Spark 1.6** ``` scala scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") res0: org.apache.spark.sql.DataFrame = [result: string] scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')") res1: org.apache.spark.sql.DataFrame = [result: string] ``` **Spark 2.0** ``` scala scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") res0: org.apache.spark.sql.DataFrame = [] scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')") org.apache.spark.sql.catalyst.parser.ParseException: mismatched input '<' expecting {')', ','}(line 1, pos 42) ``` Pass the Jenkins test with a newly added testcase.

dongjoon-hyun · 2016-12-02T16:00:47Z

Rebased to resolve conflicts.

dongjoon-hyun · 2016-12-02T16:01:30Z

Hi, @hvanhovell .
Could you give me some overall directional advice?

SparkQA · 2016-12-02T18:44:50Z

Test build #69574 has finished for PR 15987 at commit c7c31e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CommandWithExpression extends LeafNode

dongjoon-hyun · 2016-12-07T13:01:29Z

Retest this please

SparkQA · 2016-12-07T15:35:24Z

Test build #69797 has finished for PR 15987 at commit c7c31e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CommandWithExpression extends LeafNode

dongjoon-hyun · 2016-12-13T18:22:50Z

Hi, @hvanhovell .
I will remove the support of expression from this PR. Is it okay for you?

hvanhovell mentioned this pull request Nov 28, 2016

[SPARK-17732][SQL] Revert ALTER TABLE DROP PARTITION should support comparators #16036

Closed

dongjoon-hyun changed the title ~~[SPARK-18515][SQL] AlterTableDropPartitions fails for non-string columns~~ [SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITION should support comparators Nov 30, 2016

dongjoon-hyun commented Nov 30, 2016

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITION should support comparators~~ [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators Nov 30, 2016

dongjoon-hyun closed this Feb 4, 2017

gatorsmile mentioned this pull request Apr 6, 2018

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filters in ALTER TABLE DROP PARTITION #20999

Closed

dongjoon-hyun deleted the SPARK-18515 branch January 7, 2019 07:03

weixiuli mentioned this pull request Oct 28, 2019

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filter in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS #26280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987

dongjoon-hyun commented Nov 23, 2016 •

edited

Loading

dongjoon-hyun commented Nov 23, 2016 •

edited

Loading

SparkQA commented Nov 23, 2016

rxin commented Nov 23, 2016

dongjoon-hyun commented Nov 24, 2016

SparkQA commented Nov 30, 2016

SparkQA commented Nov 30, 2016

dongjoon-hyun commented Nov 30, 2016

dongjoon-hyun Nov 30, 2016

viirya Dec 5, 2016

dongjoon-hyun Dec 5, 2016

dongjoon-hyun commented Nov 30, 2016 •

edited

Loading

dongjoon-hyun commented Dec 2, 2016

dongjoon-hyun commented Dec 2, 2016

SparkQA commented Dec 2, 2016

dongjoon-hyun commented Dec 7, 2016

SparkQA commented Dec 7, 2016

dongjoon-hyun commented Dec 13, 2016

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987

Conversation

dongjoon-hyun commented Nov 23, 2016 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

dongjoon-hyun commented Nov 23, 2016 • edited Loading

SparkQA commented Nov 23, 2016

rxin commented Nov 23, 2016

dongjoon-hyun commented Nov 24, 2016

SparkQA commented Nov 30, 2016

SparkQA commented Nov 30, 2016

dongjoon-hyun commented Nov 30, 2016

dongjoon-hyun Nov 30, 2016

Choose a reason for hiding this comment

viirya Dec 5, 2016

Choose a reason for hiding this comment

dongjoon-hyun Dec 5, 2016

Choose a reason for hiding this comment

dongjoon-hyun commented Nov 30, 2016 • edited Loading

dongjoon-hyun commented Dec 2, 2016

dongjoon-hyun commented Dec 2, 2016

SparkQA commented Dec 2, 2016

dongjoon-hyun commented Dec 7, 2016

SparkQA commented Dec 7, 2016

dongjoon-hyun commented Dec 13, 2016

dongjoon-hyun commented Nov 23, 2016 •

edited

Loading

dongjoon-hyun commented Nov 23, 2016 •

edited

Loading

dongjoon-hyun commented Nov 30, 2016 •

edited

Loading