[SPARK-16356][Follow-up][ML] Enforce ML test of exception for local/distributed Dataset.

[yanboliang]

What changes were proposed in this pull request?

#14035 added testImplicits to ML unit tests and promoted toDF(), but left one minor issue at VectorIndexerSuite. If we create the DataFrame by Seq(...).toDF(), it will throw different error/exception compared with sc.parallelize(Seq(...)).toDF() for one of the test cases.

After in-depth study, I found it was caused by different behavior of local and distributed Dataset if the UDF failed at assert. If the data is local Dataset, it throws AssertionError directly; If the data is distributed Dataset, it throws SparkException which is the wrapper of AssertionError. I think we should enforce this test to cover both case.

How was this patch tested?

Unit test.

[yanboliang]

cc @HyukjinKwon

[HyukjinKwon]

Oh, I see. Thanks for looking into this. +1 for this PR.

[HyukjinKwon]

So, this is roughly error from driver-side vs executor side. I see.

[SparkQA]

Test build #65964 has finished for PR 15261 at commit c24021e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

@HyukjinKwon The root cause of this is Spark supported creating local Dataset which may not trigger a Spark job. This satisfied the design of Dataset, and in most case they have same behavior for local and distributed Dataset except whether to wrapper the exception. We need to enforce the test in ML. If Spark SQL guys unified them in the future, we can make corresponding change. Thanks!

[yanboliang]

@srowen Would you mind to have a look when you are available? Thanks.

[srowen]

Maybe off-topic for this review, but is this an assertion error? bad input shouldn't cause an assertion to trip.

[yanboliang]

Yes, this is indeed a problem, let's try to find out all similar cases and resolved them in a separate PR. Thanks.

[yanboliang]

Merged into master, thanks for review.