-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators #15987
Conversation
Hi, @hvanhovell . This is the first attempt to use
|
Test build #69050 has finished for PR 15987 at commit
|
Why does it cast it to double? The fix looks pretty weird. |
Thank you for review, @rxin . In the expression, the partition attribute are assumed I tried to implement the following recommended way in JIRA issue, but the PR may choose not a clean way. Do you mean
|
Test build #69383 has finished for PR 15987 at commit
|
Test build #69397 has finished for PR 15987 at commit
|
Hi, @hvanhovell . |
// We should not do this here in order to support general CommandWithExpression | ||
if (catalogTable.tableType == CatalogTableType.VIEW) { | ||
throw new AnalysisException( | ||
"Cannot alter a view with ALTER TABLE. Please use ALTER VIEW instead") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previously, run
function of Command
does every error handlings including this kind of errors.
Now, we should do that here because this exception has higher priority than unresolved attributes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Except AlterTableDropPartitionCommand
, do we have other running commands which need this resolving? If not, I think we may not need to add this to Analyzer
and create new abstraction CommandWithExpression
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. There is no such command.
AlterTableDropPartitionCommand
is not visible in catalyst
. So, it's inevitable to create some corresponding one in catalyst
.
This is an real issue. So, @rxin told me it's a weird.
Until now, I didn't find a better way to support expressions on Command
.
I'm asking the committers not to use expressions here.
Hi, @hvanhovell , @gatorsmile , @rxin . Should we remove Expression related stuff? But, while supporting expressions in partition spec, I made #15704 . but it's reverted now. IMO, the followings can be different issues.
Especially, Apache Spark scala> sql("create table tbl_x (a int) partitioned by (p int)")
scala> sql("alter table tbl_x add partition (p=10.0)")
scala> sql("show partitions tbl_x").show
+---------+
|partition|
+---------+
| p=10.0|
+---------+ In short, this is beyond of 1, I think we had better go back the old version (without expression.) How do you think about this? |
…port comparators This PR aims to support `comparators`, e.g. '<', '<=', '>', '>=', again in Apache Spark 2.0 for backward compatibility. **Spark 1.6** ``` scala scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") res0: org.apache.spark.sql.DataFrame = [result: string] scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')") res1: org.apache.spark.sql.DataFrame = [result: string] ``` **Spark 2.0** ``` scala scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") res0: org.apache.spark.sql.DataFrame = [] scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')") org.apache.spark.sql.catalyst.parser.ParseException: mismatched input '<' expecting {')', ','}(line 1, pos 42) ``` Pass the Jenkins test with a newly added testcase.
Rebased to resolve conflicts. |
Hi, @hvanhovell . |
Test build #69574 has finished for PR 15987 at commit
|
Retest this please |
Test build #69797 has finished for PR 15987 at commit
|
Hi, @hvanhovell . |
What changes were proposed in this pull request?
This PR aims to support
comparators
, e.g. '<', '<=', '>', '>=', again in Apache Spark 2.0 for backward compatibility.Spark 1.6
Spark 2.0
After this PR, it's supported.
How was this patch tested?
Pass the Jenkins test with a newly added testcase.