Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-18950][SQL] Report conflicting fields when merging two StructTypes #16365
What changes were proposed in this pull request?
Currently, StructType.merge() only reports data types of conflicting fields when merging two incompatible schemas. It would be nice to also report the field names for easier debugging.
How was this patch tested?
Unit test in DataTypeSuite.
This is actually the message users face in some cases. Isn't it :)?
val df1 = spark.range(10).selectExpr("id as intcol", "cast(id as int) as longcol") df1.write.parquet("/tmp/a") val df2 = spark.range(10).selectExpr("id as intcol", "id as longcol") df2.write.parquet("/tmp/b") spark.read.option("mergeSchema", true).parquet("/tmp/a", "/tmp/b").show()
BTW, it looks the test in
Thanks for the review @HyukjinKwon !