[SPARK-22442][SQL] ScalaReflection should produce correct field names for special characters

[viirya]

What changes were proposed in this pull request?

For a class with field name of special characters, e.g.:

case class MyType(`field.1`: String, `field 2`: String)

Although we can manipulate DataFrame/Dataset, the field names are encoded:

scala> val df = Seq(MyType("a", "b"), MyType("c", "d")).toDF
df: org.apache.spark.sql.DataFrame = [field$u002E1: string, field$u00202: string]
scala> df.as[MyType].collect
res7: Array[MyType] = Array(MyType(a,b), MyType(c,d))

It causes resolving problem when we try to convert the data with non-encoded field names:

spark.read.json(path).as[MyType]
...
[info]   org.apache.spark.sql.AnalysisException: cannot resolve '`field$u002E1`' given input columns: [field 2, fie
ld.1];
[info]   at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
...

We should use decoded field name in Dataset schema.

How was this patch tested?

Added tests.

[SparkQA]

Test build #83478 has finished for PR 19664 at commit 319c804.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #83486 has finished for PR 19664 at commit a671a83.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class SpecialCharClass(field.1: String, field 2: String)

[viirya]

cc @cloud-fan for review.

[cloud-fan]

does StaticInvoke have some issue?

[viirya]

Maybe, although I didn't have concrete case causing the issue for now.

[viirya]

Since we use Invoke to access field(s) in object, this is an issue. I didn't found StaticInvoke used similarly. So it should be fine.

[cloud-fan]

why it has backticks?

[viirya]

We need to deliberately wrap backticks around a field name such as field.1 because of the dot character. Otherwise UnresolvedAttribute will parse it as two name parts Seq("field", "1") and then fail resolving later.

[cloud-fan]

good catch!

[cloud-fan]

actually we should collect u.nameParts here. u.name adds backticks if the nameParts contains dot, which makes the test result a little confusing.

[viirya]

Ok. Reasonable.

[SparkQA]

Test build #83622 has finished for PR 19664 at commit 10db6b4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!