Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-29505][SQL] Make DESC EXTENDED <table name> <column name> case insensitive #26927

Closed
wants to merge 1 commit into from
Closed

[SPARK-29505][SQL] Make DESC EXTENDED <table name> <column name> case insensitive #26927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 3 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
View file
Open in desktop
Expand Up @@ -34,7 +34,7 @@ import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.catalyst.plans.DescribeTableSchema
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.util.{escapeSingleQuotedString, quoteIdentifier}
import org.apache.spark.sql.catalyst.util.{escapeSingleQuotedString, quoteIdentifier, CaseInsensitiveMap}
import org.apache.spark.sql.execution.datasources.{DataSource, PartitioningUtils}
import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
import org.apache.spark.sql.execution.datasources.json.JsonFileFormat
Expand Down Expand Up @@ -720,7 +720,8 @@ case class DescribeColumnCommand(
}

val catalogTable = catalog.getTempViewOrPermanentTableMetadata(table)
val colStats = catalogTable.stats.map(_.colStats).getOrElse(Map.empty)
val colStatsMap = catalogTable.stats.map(_.colStats).getOrElse(Map.empty)
val colStats = if (conf.caseSensitiveAnalysis) colStatsMap else CaseInsensitiveMap(colStatsMap)
val cs = colStats.get(field.name)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about just simply writing it like this?


    val catalogStats = catalog.getTempViewOrPermanentTableMetadata(table).stats
    val cs = if (conf.caseSensitiveAnalysis) {
      catalogStats.flatMap { cs => cs.colStats.get(field.name) }
    } else {
      catalogStats.flatMap { cs =>
        cs.colStats.map { case (k, v) => (k.toLowerCase(Locale.ROOT), v) }
          .get(field.name.toLowerCase(Locale.ROOT))
      }
    }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reworked on the change suggested. Could you review?

val comment = if (field.metadata.contains("comment")) {
Expand Down
13 changes: 13 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/describe-table-column.sql
View file
Open in desktop
Expand Up @@ -49,3 +49,16 @@ DROP VIEW desc_col_temp_view;
DROP TABLE desc_col_table;

DROP TABLE desc_complex_col_table;

--Test case insensitive

CREATE TABLE customer(CName STRING);

INSERT INTO customer VALUES('Maria');

ANALYZE TABLE customer COMPUTE STATISTICS FOR COLUMNS cname;

DESC EXTENDED customer cname;

DROP TABLE customer;

51 changes: 50 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/describe-table-column.sql.out
View file
Open in desktop
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 23
-- Number of queries: 28


-- !query 0
Expand Down Expand Up @@ -264,3 +264,52 @@ DROP TABLE desc_complex_col_table
struct<>
-- !query 22 output



-- !query 23
CREATE TABLE customer(CName STRING)
-- !query 23 schema
struct<>
-- !query 23 output



-- !query 24
INSERT INTO customer VALUES('Maria')
-- !query 24 schema
struct<>
-- !query 24 output



-- !query 25
ANALYZE TABLE customer COMPUTE STATISTICS FOR COLUMNS cname
-- !query 25 schema
struct<>
-- !query 25 output



-- !query 26
DESC EXTENDED customer cname
-- !query 26 schema
struct<info_name:string,info_value:string>
-- !query 26 output
col_name cname
data_type string
comment NULL
min NULL
max NULL
num_nulls 0
distinct_count 1
avg_col_len 5
max_col_len 5
histogram NULL


-- !query 27
DROP TABLE customer
-- !query 27 schema
struct<>
-- !query 27 output