[SPARK-13594][SQL] remove typed operations(e.g. map, flatMap) from python DataFrame

[cloud-fan]

What changes were proposed in this pull request?

Remove map, flatMap, mapPartitions from python DataFrame, to prepare for Dataset API in the future.

How was this patch tested?

existing tests

[cloud-fan]

cc @rxin @yhuai

[SparkQA]

Test build #52244 has finished for PR 11445 at commit 86ec0ff.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Should we also remove foreach and foreachPartition?

[rxin]

those are fine, since they don't return anything.

[SparkQA]

Test build #52273 has finished for PR 11445 at commit bf4b9d5.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52285 has finished for PR 11445 at commit d0a69fa.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52287 has finished for PR 11445 at commit 5e711e3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging this in master.

[mydpy]

This change surprised me as a user of Pyspark on the 2.0.0-Snapshot. Thanks for documenting this well. Since I usually use the Scala API, it was not clear to me that Pyspark didn't support the Datasets API yet (i.e., df.rdd.flatMap(...) returns a PythonRDD as-opposed to a Dataset)

[maver1ck]

@rxin
As we're not planning to implement DataSets in Python is there a plan to revert this PR?