[SPARK-23060][Python] New feature - apply method to extend rdd's functionality

[gianmarcodonetti]

What changes were proposed in this pull request?

Extend the RDD class with the method apply.
This method should be like the pipe operator, attached to the RDD class itself.
Example:

def foo(rdd): return rdd.map(lambda x: x.split('|')).filter(lambda x: x[0] == 'ERROR')
rdd = sc.parallelize(['ERROR|10', 'ERROR|12', 'WARNING|10', 'INFO|2'])
result = rdd.apply(foo)
result.collect()
[('ERROR', '10'), ('ERROR', '12')]

The idea is to have something like this:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pipe.html?

How was this patch tested?

Manual tests. Easy patch.

Please review http://spark.apache.org/contributing.html before opening a pull request.

[HyukjinKwon]

Isn't it just a helper function?

def apply(self, func):
    return func(self)

I don't think it's quite worth adding it.

[gianmarcodonetti]

@HyukjinKwon in my opinion, it helps a lot.
My goal is to avoid this case:

final_rdd = func_3(func_2(func_1(initial_rdd)))

And admit this:

final_rdd = initial_rdd.apply(func_1).apply(func_2).apply(func_3)

More functional and readable...

[HyukjinKwon]

That resembles pipe as I pointed out in the JIRA. It's just a little trick and I don't think it's worth adding it for an API alone.

BTW, we should consider Java / Scala APIs and how it's going to work with Dataset and DataFrame too.

[ueshin]

Is this similar to Dataset.transform() in Java/Scala API? But we don't have similar APIs for RDDs.

[HyukjinKwon]

Oh, I see! Yea, they look quite same.

[srowen]

At best, the functionality already exists for the new API in a form. This should be closed.

[AmplabJenkins]

Can one of the admins verify this patch?

[holdenk]

I'm +1 to @srowen on this, I don't believe this is a change we're going to make to the API. @gianmarcodonetti please close this PR.