[SPARK-4819] Remove Guava's "Optional" from public API

[srowen]

Replace Guava Optional with (an API clone of) Java 8 java.util.Optional (edit: and a clone of Guava Optional)

See also #10512

[SparkQA]

Test build #48427 has finished for PR 10513 at commit 1196fcb.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class Optional<T>

[SparkQA]

Test build #48474 has finished for PR 10513 at commit 72f7122.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class Optional<T> implements Serializable

[SparkQA]

Test build #48477 has finished for PR 10513 at commit 283095b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

This looks ok to me if we're ok with breaking source and binary compatibility in 2.0. I guess it's the less bad option to fix this without depending on Java 8...

[rxin]

If we are creating our own Optional, why don't we also support all the methods available on Guava, so we only need to break "import" compatibility?

[srowen]

My theory was that it makes it more compatible with Java 8 Optional. Later, moving to its implementation is a smaller change. We're taking the bigger change now. Or at least that was the motivation.

Another possibility is to add the Guava work-alike methods and deprecate them. Later they'd be removed I guess when Java 8 Optional gets used. I liked that less since it still just half of the breaking change farther down the road, when a major release seems like the time to do as much of the breaking as possible.

[SparkQA]

Test build #48605 has finished for PR 10513 at commit b0e96b4.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class Optional<T> implements Serializable

[rxin]

But we might never switch to Java 8's Optional, or maybe the entire API that's depending Optional will be deprecated in the future. Why should we increase the level of breakage now in anticipation of something that we don't even know we need in the future?

[srowen]

No matter what, there's a break now -- not even source-compatible. In for a penny, in for a pound, if it has a future benefit? but I agree it's unlikely to actually switch to the Java 8 Optional as it's yet another source-incompatible change. I think it's either use Java 8 in Spark 2.x, and then really use java.util.Optional, or else go with the hybrid solution where the new class tries to implement both the Guava and Java 8 API.

[SparkQA]

Test build #48654 has finished for PR 10513 at commit 5b3dbd0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48850 has finished for PR 10513 at commit b2c4b17.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2336 has finished for PR 10513 at commit b2c4b17.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

we should probably have a better user-facing documentation in addition to this ...

[SparkQA]

Test build #48936 has finished for PR 10513 at commit bf7a205.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

LGTM. Anything else you want to do? Otherwise we should remove WIP and merge it.

[srowen]

It's ready as far as I'm concerned.

[rxin]

Thanks - I'm going to merge this.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49027/
Test FAILed.