Skip to content

Commit

Permalink
[SPARK-32549][PYSPARK] Add column name in _infer_schema error message
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

The current error message from `_infer_type` in `_infer_schema` only includes the unsupported column type but not the column name. This PR adds the column name in the error message to make it easier for users to identify which column should they drop or convert.

### Why are the changes needed?

Improve user experience.

### Does this PR introduce _any_ user-facing change?

Yes. The error message from `_infer_schema` is changed.
Before:
"not supported type: foo"
After:
"Column bar contains not supported type: foo"

### How was this patch tested?

Updated the existing unit test.

Closes #29365 from liangz1/types-error-colname.

Authored-by: Liang Zhang <liang.zhang@databricks.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
liangz1 authored and HyukjinKwon committed Aug 7, 2020
1 parent 4e267f3 commit 2cb48ea
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 2 deletions.
2 changes: 1 addition & 1 deletion python/pyspark/sql/tests/test_types.py
Expand Up @@ -720,7 +720,7 @@ def assertCollectSuccess(typecode, value):
unsupported_types = all_types - set(supported_types)
# test unsupported types
for t in unsupported_types:
with self.assertRaises(TypeError):
with self.assertRaisesRegexp(TypeError, "infer the type of the field myarray"):
a = array.array(t)
self.spark.createDataFrame([Row(myarray=a)]).collect()

Expand Down
7 changes: 6 additions & 1 deletion python/pyspark/sql/types.py
Expand Up @@ -1032,7 +1032,12 @@ def _infer_schema(row, names=None):
else:
raise TypeError("Can not infer schema for type: %s" % type(row))

fields = [StructField(k, _infer_type(v), True) for k, v in items]
fields = []
for k, v in items:
try:
fields.append(StructField(k, _infer_type(v), True))
except TypeError as e:
raise TypeError("Unable to infer the type of the field {}.".format(k)) from e
return StructType(fields)


Expand Down

0 comments on commit 2cb48ea

Please sign in to comment.