Skip to content
Permalink
Browse files

[SPARK-29943][SQL] Improve error messages for unsupported data type

### What changes were proposed in this pull request?
Improve error messages for unsupported data type.

### Why are the changes needed?
When the spark reads the hive table and encounters an unsupported field type, the exception message has only one unsupported type, and the user cannot know which field of which table.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
```create view t AS SELECT STRUCT('a' AS `$a`, 1 AS b) as q;```
current:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>
change:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q

```select * from t,t_normal_1,t_normal_2```
current:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>
change:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q, db: default, table: t

Closes #26577 from cxzl25/unsupport_data_type_msg.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
cxzl25 authored and HyukjinKwon committed Dec 3, 2019
1 parent 68034a8 commit 332e593093460c34fae303a913a862b1b579c83f
Showing with 9 additions and 3 deletions.
  1. +9 −3 sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -440,8 +440,13 @@ private[hive] class HiveClientImpl(
private def convertHiveTableToCatalogTable(h: HiveTable): CatalogTable = {
// Note: Hive separates partition columns and the schema, but for us the
// partition columns are part of the schema
val cols = h.getCols.asScala.map(fromHiveColumn)
val partCols = h.getPartCols.asScala.map(fromHiveColumn)
val (cols, partCols) = try {
(h.getCols.asScala.map(fromHiveColumn), h.getPartCols.asScala.map(fromHiveColumn))
} catch {
case ex: SparkException =>
throw new SparkException(
s"${ex.getMessage}, db: ${h.getDbName}, table: ${h.getTableName}", ex)
}
val schema = StructType(cols ++ partCols)

val bucketSpec = if (h.getNumBuckets > 0) {
@@ -982,7 +987,8 @@ private[hive] object HiveClientImpl {
CatalystSqlParser.parseDataType(hc.getType)
} catch {
case e: ParseException =>
throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
throw new SparkException(
s"Cannot recognize hive type string: ${hc.getType}, column: ${hc.getName}", e)
}
}

0 comments on commit 332e593

Please sign in to comment.
You can’t perform that action at this time.