[SPARK-13626] [core] Revert change to SparkConf's constructor.

[vanzin]

It shouldn't be private.

[vanzin]

BTW it's a little worrying that MiMA didn't catch this in the original change.

[rxin]

LGtm

[SparkQA]

Test build #53205 has finished for PR 11734 at commit 7b9a92b.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging

[vanzin]

unrelated test failure. merging to master.

[dongjoon-hyun]

Oh, @vanzin .
Sorry for writing comments at the closed PR.

I've read your concern and investigated some. The root cause was GenerateMIMAIgnore.scala makes .generated-mima-class-excludes from the current code having private class. As a result, it ignores the case : visibility goes from public into private. Do you think I need to make a Jira issue for this?

[vanzin]

Do you think should I make a Jira issue for this?

Please do; this means that APIs that are being changed from public to private might not trigger MiMA failures, when they should. Maybe @JoshRosen could also chime in.

[dongjoon-hyun]

Oh, thank you for fast response. I'll do right now.

[JoshRosen]

I think this problem relates to the fact that the SparkConf companion object is private[spark] and our exclusion generator inappropriately handles cases where classes and their companion objects have different visibility.

I have a good idea of how to fix this, so I'll open a PR in a little bit to address it.

[dongjoon-hyun]

Oh, that's great, @JoshRosen .

I made a Jira issue, but could you change that appropriately with yours?

https://issues.apache.org/jira/browse/SPARK-13948

[JoshRosen]

I don't think that MiMa is necessarily capable of detecting this problem for things which are evolved to become private[spark] because Scala package-private methods become public from Java's point of view and MiMa uses a Java-centric type of binary compatibility; see lightbend-labs/mima#34. My understanding is that private[spark] is a source compatibility issue from Scala's POV and thus is out-of-scope for what MiMa currently detects.

However, I think it's true that we will fail to detect a class which becomes straight-up private, so we should definitely fix that.