Use generate sklearn UDT within gapply() [SPARK-16062 blocks this]

[vlad17]

Currently, KeyedModel fitting in KeyedEstimator._fit is implemented by generating an array of a single serialized estimator, requiring an additional pass over the resulting dataframe which deserializes the UDT.

This is necessary because of a pyspark bug, and the circuitous implementation should be straightened out once the UDT issues are resolved (SPARK-16062).

[vlad17]

Also blocking this test from being unskipped:
test_gapply_python_only_udt_val (spark_sklearn.tests.test_gapply.GapplyTests) ... SKIP: python only UDTs can't be nested in arraytypes for now, see SPARK-15989

[thunterdb]

This has been fixed in Spark 2.0.1-SNAPSHOT. We can work on it once I cut the the release of spark-sklearn.

[srowen]

Was this resolved? I see the test was un-skipped in 7583ee1