[SPARK-5597] [mllib] Save/load for Decision Trees and ensembles

[jkbradley]

This adds save/load methods for trees and ensembles. For details on the design, please see the design doc.

Notes:

• This also modifies/adds for InformationGainStats and Predict.
• There are a bunch of case classes for model formats. We could potentially change the model classes themselves to be case classes, rather than creating new ones. I'd be OK with that, though it's an API change.

CC: @mengxr

[SparkQA]

Test build #26963 has finished for PR 4444 at commit 45873a2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DecisionTreeModel(val topNode: Node, val algo: Algo) extends Serializable with Saveable
  • case class PredictData(predict: Double, prob: Double)
  • case class InformationGainStatsData(
  • case class SplitData(
  • case class NodeData(
  • case class NodeWithKids(node: Node, leftChildId: Int, rightChildId: Int)
  • case class Metadata(
  • * case class EnsembleNodeData(treeId: Int, node: NodeData),
  • case class NodeData(

[mengxr]

To help review, this is the schema for saving a tree node:

id: Int
predict/
       |- predict: Double
       |- prob: Double
impurity: Double
isLeaf: Boolean
split/
     |- feature: Int
     |- threshold: Double
     |- featureType: Int
     |- categories: Array[Double]
leftNodeId: Integer
rightNodeId: Integer
stats/
     |- gain: Double
     |- impurity: Double
     |- leftImpurity: Double
     |- rightImpurity: Double
     |- leftPredict/
                   |- predict: Double
                   |- prob: Double
     |- rightPredict/
                    |- predict: Double
                    |- prob: Double

[mengxr]

The question is whether we want to save stats, which contains information that is already included in other nodes except gain. (Correct me if I'm wrong.) I'm considering replacing stats by infoGain: Double and re-construct other fields in InformationGainStats at load(). The schema becomes

id: Int
predict/
       |- predict: Double
       |- prob: Double
impurity: Double
isLeaf: Boolean
split/
     |- feature: Int
     |- threshold: Double
     |- featureType: Int
     |- categories: Array[Double]
leftNodeId: Integer
rightNodeId: Integer
infoGain: Double