[SPARK-12975] [SQL] Throwing Exception when Bucketing Columns are part of Partitioning Columns

[gatorsmile]

When users are using partitionBy and bucketBy at the same time, some bucketing columns might be part of partitioning columns. For example,

        df.write
          .format(source)
          .partitionBy("i")
          .bucketBy(8, "i", "k")
          .saveAsTable("bucketed_table")

However, in the above case, adding column i into bucketBy is useless. It is just wasting extra CPU when reading or writing bucket tables. Thus, like Hive, we can issue an exception and let users do the change.

Also added a test case for checking if the information of sortBy and bucketBy columns are correctly saved in the metastore table.

Could you check if my understanding is correct? @cloud-fan @rxin @marmbrus Thanks!

[rxin]

Does Hive write them out?

[SparkQA]

Test build #49962 has finished for PR 10891 at commit 8c718b3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Will hive throw exception for this?

[gatorsmile]

I am not a Hive expert. I just did a try in Hive 1.2.1:

hive> CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
    > PARTITIONED BY(ds STRING)
    > CLUSTERED BY(ds, user_id) INTO 256 BUCKETS;
FAILED: SemanticException [Error 10002]: Invalid column reference

It sounds like Hive does not allow users use Partitioning columns in Bucketing key. I think this is not an issue in Hive. However, this is not prohibited in our Spark SQL. @rxin @cloud-fan

[cloud-fan]

I think we should follow hive for this case, i.e. throw exception.

[gatorsmile]

Ok, let users change it. Will do the change. This can simplify the logics in bucket pruning. Thanks!

[gatorsmile]

Also updated the description of PR. Code is ready for review. : ) @cloud-fan Thanks!

[viirya]

typo: blockedBy.

[SparkQA]

Test build #49982 has finished for PR 10891 at commit d207813.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49991 has finished for PR 10891 at commit f9a8bdf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

no need to do this?

[cloud-fan]

LGTM exception some minor comments, thanks for working on it!

[gatorsmile]

Thank you for your review! So glad I can provide a help in this great feature! Table bucketing is very useful in real-world scenario.

[SparkQA]

Test build #50003 has finished for PR 10891 at commit 2ace09f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM

[marmbrus]

Thanks, merging to master.