[SPARK-14985][ML] Update LinearRegression, LogisticRegression summary internals to handle model copy

[BenFradet]

What changes were proposed in this pull request?

The summaries now have a internal copy of the model

How was this patch tested?

Reran the concerned suites

[BenFradet]

@jkbradley

Regarding linear regression, should the tValues, pValues, coefficientStandardErrors and the different metrics be moved to the training summary?

Same thing for logistic regression, should the different metrics be moved to the training summary?

Also, I noticed some discrepancy regarding the visibility of some fields in linear regression compared to generalized linear regression:

• numInstances is public in lr and regression private in glr
• degreesOfFreedom is private in lr and public in glr

Is this intentional?

[SparkQA]

Test build #57482 has finished for PR 12823 at commit 030442c.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

@BenFradet I'm sorry for dropping the ball on this one. Did you close this due to inactivity? If you're willing, it would be nice to do this cleanup.

To answer your questions:

Regarding linear regression, should the tValues, pValues, coefficientStandardErrors and the different metrics be moved to the training summary?
Same thing for logistic regression, should the different metrics be moved to the training summary?

These can go in the summary, not just the training summary, since they can be calculated from the model. The training summary just has values which are specific to the training process.

Also, I noticed some discrepancy regarding the visibility of some fields in linear regression compared to generalized linear regression:

• numInstances is public in lr and regression private in glr
• degreesOfFreedom is private in lr and public in glr

Making numInstances and degreesOfFreedom public sounds good to me.

Thanks!

[BenFradet]

@jkbradley I did close this due to inactivity, I'm reopening it as I now have a bit of time.

[SparkQA]

Test build #75107 has finished for PR 12823 at commit 030442c.

• This patch fails MiMa tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75110 has finished for PR 12823 at commit a996b6d.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75169 has finished for PR 12823 at commit d0a04bb.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75242 has finished for PR 12823 at commit 74e1fa0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BenFradet]

org.apache.spark.storage.BlockManagerProactiveReplicationSuite.proactive block replication - 3 replicas - 2 block manager deletions doesn't seem to be linked, relaunching the tests.

[BenFradet]

Jenkins, retest this please

[SparkQA]

Test build #75245 has finished for PR 12823 at commit 74e1fa0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BenFradet]

ping @jkbradley if you could take a look, that'd be great.

If you have the time, there is also the #17431 segue.

[MLnick]

@BenFradet I think this needs to be updated now - and should also take into account #15435 and #17586

[BenFradet]

@MLnick ok, I'll have a look

[BenFradet]

@MLnick should we wait on those PRs being merged before moving forward with this?

[SparkQA]

Test build #95021 has finished for PR 12823 at commit 74e1fa0.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.