[SPARK-13067][SQL] workaround for a weird scala reflection problem

[cloud-fan]

A simple workaround to avoid getting parameter types when convert a logical plan to json. Note that this bug only exists locally, but ok on jenkins, not sure why...

[cloud-fan]

CC @davies @yhuai @marmbrus

@davies can you try it out locally?

[cloud-fan]

retest this please

[SparkQA]

Test build #50316 has finished for PR 10970 at commit 7e66869.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #50320 has finished for PR 10970 at commit 7e66869.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2472 has finished for PR 10970 at commit 7e66869.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

"。" replace with "." ?

[davies]

Tested it locally, test passed. LGTM, I'm merging this into master, thanks! (fixed the 。)

[rxin]

@cloud-fan is this fixing a bug? If yes, please add a test somewhere.

[marmbrus]

Existing tests were failing locally, but not on Jenkins. We are not sure why.

[rxin]

OK got it. @cloud-fan Please add that to the pull request description next time.

[cloud-fan]

Sorry I thought the JIRA description tells it, will add it to PR description next time.

[rxin]

Yea the pull request description gets merged as commit description, and is the only thing that sticks around forever.

[atronchi]

How can we workaround this problem when using 1.6.1?

[atronchi]

This solution does not seem to work for datasets on tables using complex types. e.g. Array[Int] still throws the exception java.lang.AssertionError: assertion failed: Unsound substitution from List(type T, type U) to List()

[cloud-fan]

@atronchi can you create a JIRA and put the code that can reproduce the bug? thanks!

[atronchi]

The solution mentioned in [SPARK-17424] by @rdblue fixes this issue.