Skip to content

Commit

Permalink
[SPARK-29943][SQL] Improve error messages for unsupported data type
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Improve error messages for unsupported data type.

### Why are the changes needed?
When the spark reads the hive table and encounters an unsupported field type, the exception message has only one unsupported type, and the user cannot know which field of which table.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
```create view t AS SELECT STRUCT('a' AS `$a`, 1 AS b) as q;```
current:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>
change:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q

```select * from t,t_normal_1,t_normal_2```
current:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>
change:
org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q, db: default, table: t

Closes #26577 from cxzl25/unsupport_data_type_msg.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
cxzl25 authored and HyukjinKwon committed Dec 3, 2019
1 parent 68034a8 commit 332e593
Showing 1 changed file with 9 additions and 3 deletions.
Expand Up @@ -440,8 +440,13 @@ private[hive] class HiveClientImpl(
private def convertHiveTableToCatalogTable(h: HiveTable): CatalogTable = {
// Note: Hive separates partition columns and the schema, but for us the
// partition columns are part of the schema
val cols = h.getCols.asScala.map(fromHiveColumn)
val partCols = h.getPartCols.asScala.map(fromHiveColumn)
val (cols, partCols) = try {
(h.getCols.asScala.map(fromHiveColumn), h.getPartCols.asScala.map(fromHiveColumn))
} catch {
case ex: SparkException =>
throw new SparkException(
s"${ex.getMessage}, db: ${h.getDbName}, table: ${h.getTableName}", ex)
}
val schema = StructType(cols ++ partCols)

val bucketSpec = if (h.getNumBuckets > 0) {
Expand Down Expand Up @@ -982,7 +987,8 @@ private[hive] object HiveClientImpl {
CatalystSqlParser.parseDataType(hc.getType)
} catch {
case e: ParseException =>
throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
throw new SparkException(
s"Cannot recognize hive type string: ${hc.getType}, column: ${hc.getName}", e)
}
}

Expand Down

0 comments on commit 332e593

Please sign in to comment.