[SPARK-6004][MLlib] Pick the best model when training GradientBoostedTrees with validation

[viirya]

Since the validation error does not change monotonically, in practice, it should be proper to pick the best model when training GradientBoostedTrees with validation instead of stopping it early.

[SparkQA]

Test build #27948 has finished for PR 4763 at commit ea2fae2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

Please see my comments on the JIRA; I think parts of this change need to be discussed. Thanks!

[viirya]

@jkbradley Modified based on our discussion on JIRA. Please take a look. Thanks!

[SparkQA]

Test build #27979 has finished for PR 4763 at commit 452e049.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

LGTM Merging into master. Thanks for the PR!

Note for posterity: This doesn't have tests, but it would be difficult to create a good test for it. It might be possible by hand-constructing a dataset and using trees of depth 1, where we could judge exactly how much each new feature (= new tree) would help.