[SPARK-10656][SQL]fix selection fails when a column has special characters

[zhichao-li]

Best explained with this example:
val df = sqlContext.read.json(sqlContext.sparkContext.makeRDD(
"""{"a.b": "c", "d": "e" }""" :: Nil))
df.select("").show() //successful
df.select(df("")).show() //throws exception
df.withColumnRenamed("d", "f").show() //also fails

This PR address this by adding back tick for the column names.

[SparkQA]

Test build #42648 has finished for PR 8811 at commit 2fa0aa6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Interaction(override val uid: String) extends Transformer

[zhichao-li]

cc @chenghao-intel @yhuai @liancheng

[chenghao-intel]

A better place to add the back tick is in the class Column?

[yhuai]

For these tests, which ones will fail without your fix?

[zhichao-li]

Just add more test to cover the if branch you mentioned. and the below items would fail if without this patch.

    checkAnswer(df.select(df("*")), Row(10, 11, Row(12)))
    checkAnswer(df.withColumnRenamed("f", "h").select("h"), Row(Row(12)))
    checkAnswer(df.withColumnRenamed("f", "f").select("f"), Row(Row(12)))
    checkAnswer(df.withColumnRenamed("`a.b`", "s").select("s"), Row(10))
    checkAnswer(df.withColumnRenamed("f", "h").select("`a.b`"), Row(10))

[SparkQA]

Test build #42801 has finished for PR 8811 at commit d360b07.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

@chenghao-intel, UnresolvedAttribute.parseAttributeName still lack the ability to solve case like this: name, so we cannot add the back tick to all cases which will do if you make the changes in Column. Here I'm supposing the original field name should not containing any back-tick.

[SparkQA]

Test build #42814 has finished for PR 8811 at commit 626a82b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #44003 has finished for PR 8811 at commit 626a82b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

retest this please

[SparkQA]

Test build #44024 has finished for PR 8811 at commit 626a82b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I think the main problem is: we interpret column name with special handling of . for DataFrame. This enables us to write something like df("a.b") to get the field b of a. However, we don't need this feature in DataFrame.apply("*") or DataFrame.withColumnRenamed. In these 2 cases, the column name is the final name already, we don't need extra process to interpret it.

How about adding a new version of DataFrame.resolve which use the column name directly without interpreting it? And use new version of DataFrame.resolve in DataFrame.apply("*") and DataFrame.withColumnRenamed.