[SPARK-13049] Add First/last with ignore nulls to functions.scala

[hvanhovell]

This PR adds the ability to specify the ignoreNulls option to the functions dsl, e.g:
df.select($"id", last($"value", ignoreNulls = true).over(Window.partitionBy($"id").orderBy($"other"))

This PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6.

cc @yhuai

[yhuai]

@hvanhovell Thanks for the PR. Do you know why expr/callUDF does not work?

[hvanhovell]

@yhuai expr("last(r, true)") would return an UnresolvedFunction(UnresolvedAttribute(r), Literal(true)). The problem is that the WindowSpec does not recognize UnresolvedFunction's.

This is the cleaner fix. We could also add a match to the WindowSpec for unresolved functions.

[hvanhovell]

retest this please

[hvanhovell]

retest this please

[rxin]

Why might this be a bug fix?

[hvanhovell]

A user is trying to get this working on 1.6 using the dataframe api. That doesn't work directly because functions.scala misses the functions implemented in this PR. The indirect approach using expr(...) doesn't work because WindowSpec does not support UnresolvedFunctions.

I guess this is more a feature than a bug fix....

[SparkQA]

Test build #50231 has finished for PR 10957 at commit defcc02.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Actually can you update the Python API as well?

[SparkQA]

Test build #50462 has finished for PR 10957 at commit b002d60.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50463 has finished for PR 10957 at commit 6e4da4f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Can you write something like this to be more clear? And update all the docs (including Python).

"The function by default includes the first value it sees. When ignoreNulls is set to true, then it ignores the null values and includes the first non-null value. If all values are null, then null is returned."

[rxin]

Thanks - only some minor comment on the documentation to make it more clear.

[SparkQA]

Test build #50464 has finished for PR 10957 at commit 809c999.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging this in master.