Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-33667][SQL][3.0] Respect the spark.sql.caseSensitive config while resolving partition spec in v1 SHOW PARTITIONS #30626

Closed
wants to merge 2 commits into from
Closed

[SPARK-33667][SQL][3.0] Respect the spark.sql.caseSensitive config while resolving partition spec in v1 SHOW PARTITIONS #30626

wants to merge 2 commits into from

Commits on Dec 6, 2020

  1. [SPARK-33667][SQL] Respect the spark.sql.caseSensitive config while… 

    … resolving partition spec in v1 `SHOW PARTITIONS`

Preprocess the partition spec passed to the V1 SHOW PARTITIONS implementation `ShowPartitionsCommand`, and normalize the passed spec according to the partition columns w.r.t the case sensitivity flag  **spark.sql.caseSensitive**.

V1 SHOW PARTITIONS is case sensitive in fact, and doesn't respect the SQL config **spark.sql.caseSensitive** which is false by default, for instance:
```sql
spark-sql> CREATE TABLE tbl1 (price int, qty int, year int, month int)
         > USING parquet
         > PARTITIONED BY (year, month);
spark-sql> INSERT INTO tbl1 PARTITION(year = 2015, month = 1) SELECT 1, 1;
spark-sql> SHOW PARTITIONS tbl1 PARTITION(YEAR = 2015, Month = 1);
Error in query: Non-partitioning column(s) [YEAR, Month] are specified for SHOW PARTITIONS;
```
The `SHOW PARTITIONS` command must show the partition `year = 2015, month = 1` specified by `YEAR = 2015, Month = 1`.

Yes. After the changes, the command above works as expected:
```sql
spark-sql> SHOW PARTITIONS tbl1 PARTITION(YEAR = 2015, Month = 1);
year=2015/month=1
```

By running the affected test suites:
- `v1/ShowPartitionsSuite`
- `v2/ShowPartitionsSuite`

Closes apache#30615 from MaxGekk/show-partitions-case-sensitivity-test.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 4829781)
Signed-off-by: Max Gekk <max.gekk@gmail.com>
    @MaxGekk
    MaxGekk committed Dec 6, 2020
    Configuration menu
    Copy the full SHA
    88ccb59 View commit details
    Browse the repository at this point in the history

  2. Fix HiveCommandSuite

    @MaxGekk
    MaxGekk committed Dec 6, 2020
    Configuration menu
    Copy the full SHA
    ff7bf27 View commit details
    Browse the repository at this point in the history