[SPARK-3162][MLlib][WIP] Add local tree training for decision tree regressors

[smurching]

What changes were proposed in this pull request?

Based on Yggdrasil, added local training of decision tree regressors.

Some classes/objects largely correspond to Yggdrasil classes/objects.
Specifically:

• class LocalDecisionTreeRegressor --> class YggdrasilRegressor
• object LocalDecisionTree --> object YggdrasilRegression
• object LocalDecisionTreeUtils --> object Yggdrasil

How was this patch tested?

Added unit tests in (ml/tree/impl/LocalTreeTrainingSuite.scala) verifying that local & distributed training of a decision tree regressor produces the same tree.

[jkbradley]

ok to test

[SparkQA]

Test build #64624 has finished for PR 14872 at commit aa4fcc8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

Just get the children from computeBestSplits.

[SparkQA]

Test build #64879 has finished for PR 14872 at commit 8d443ce.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

ok to test

[SparkQA]

Test build #66067 has finished for PR 14872 at commit ee56ffe.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66249 has finished for PR 14872 at commit 6e3be3a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Can one of the admins verify this patch?

[tomlaube]

Hi, is there any progress on this?

[smurching]

Hi,

I've stopped working on this PR - I can go ahead and close it.

[jkbradley]

@smurching Sorry we haven't had time to continue with this. Please don't delete the branch; I'd like to pick it up eventually!

[smurching]

No worries, apologies for being busy on my end -- I'll leave the branch up & try to contribute in other ways when I have the time!

[hollinwilkins]

Hey, would definitely like to see this included in Spark. We are having issues with OOM errors training pretty small RFs. It could have something to do with the cross validation implementation as well, as it stores all of the models in memory at once.