[SPARK-13948] MiMa check should catch if the visibility changes to private

[JoshRosen]

MiMa excludes are currently generated using both the current Spark version's classes and Spark 1.2.0's classes, but this doesn't make sense: we should only be ignoring classes which were private in the previous Spark version, not classes which became private in the current version.

This patch updates dev/mima to only generate excludes with respect to the previous artifacts that MiMa checks against. It also updates MimaBuild so that excludeClass only applies directly to the class being excluded and not to its companion object (since a class and its companion object can have different accessibility).

[JoshRosen]

Here were the errors that I ignored (we should follow up on these later):

[error]  * method initializeLogIfNecessary(Boolean)Unit in trait org.apache.spark.Logging is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.Logging.initializeLogIfNecessary")
[error]  * deprecated method lookupTimeout(org.apache.spark.SparkConf)scala.concurrent.duration.FiniteDuration in object org.apache.spark.util.RpcUtils does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.util.RpcUtils.lookupTimeout")
[error]  * deprecated method askTimeout(org.apache.spark.SparkConf)scala.concurrent.duration.FiniteDuration in object org.apache.spark.util.RpcUtils does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.util.RpcUtils.askTimeout")
[error]  * method logEvent()Boolean in trait org.apache.spark.scheduler.SparkListenerEvent is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.scheduler.SparkListenerEvent.logEvent")
[info] spark-mllib: found 4 potential binary incompatibilities while checking against org.apache.spark:spark-mllib_2.11:1.6.0  (filtered 151)
[error]  * method transform(org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame in class org.apache.spark.ml.UnaryTransformer's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.sql.Dataset instead of (org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.UnaryTransformer.transform")
[error]  * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel in class org.apache.spark.ml.classification.DecisionTreeClassifier's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.DecisionTreeClassifier.train")
[error]  * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel in class org.apache.spark.ml.classification.LogisticRegression's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.LogisticRegression.train")
[error]  * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel in class org.apache.spark.ml.regression.DecisionTreeRegressor's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.regression.DecisionTreeRegressor.train")
[info] spark-sql: found 10 potential binary incompatibilities while checking against org.apache.spark:spark-sql_2.11:1.6.0  (filtered 658)
[error]  * method toDF()org.apache.spark.sql.DataFrame in class org.apache.spark.sql.Dataset has a different result type in current version, where it is org.apache.spark.sql.Dataset rather than org.apache.spark.sql.DataFrame
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.Dataset.toDF")
[error]  * method groupBy(org.apache.spark.api.java.function.MapFunction,org.apache.spark.sql.Encoder)org.apache.spark.sql.GroupedDataset in class org.apache.spark.sql.Dataset in current version does not have a correspondent with same parameter signature among (java.lang.String,scala.collection.Seq)org.apache.spark.sql.GroupedData, (java.lang.String,Array[java.lang.String])org.apache.spark.sql.GroupedData
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.Dataset.groupBy")
[error]  * method groupBy(scala.collection.Seq)org.apache.spark.sql.GroupedDataset in class org.apache.spark.sql.Dataset has a different result type in current version, where it is org.apache.spark.sql.GroupedData rather than org.apache.spark.sql.GroupedDataset
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.Dataset.groupBy")
[error]  * method groupBy(scala.Function1,org.apache.spark.sql.Encoder)org.apache.spark.sql.GroupedDataset in class org.apache.spark.sql.Dataset in current version does not have a correspondent with same parameter signature among (java.lang.String,scala.collection.Seq)org.apache.spark.sql.GroupedData, (java.lang.String,Array[java.lang.String])org.apache.spark.sql.GroupedData
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.Dataset.groupBy")
[error]  * method groupBy(Array[org.apache.spark.sql.Column])org.apache.spark.sql.GroupedDataset in class org.apache.spark.sql.Dataset has a different result type in current version, where it is org.apache.spark.sql.GroupedData rather than org.apache.spark.sql.GroupedDataset
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.Dataset.groupBy")
[error]  * method select(scala.collection.Seq)org.apache.spark.sql.DataFrame in class org.apache.spark.sql.Dataset has a different result type in current version, where it is org.apache.spark.sql.Dataset rather than org.apache.spark.sql.DataFrame
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.Dataset.select")
[error]  * method select(Array[org.apache.spark.sql.Column])org.apache.spark.sql.DataFrame in class org.apache.spark.sql.Dataset has a different result type in current version, where it is org.apache.spark.sql.Dataset rather than org.apache.spark.sql.DataFrame
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.Dataset.select")
[error]  * method toDS()org.apache.spark.sql.Dataset in class org.apache.spark.sql.Dataset does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.Dataset.toDS")
[error]  * abstract method newInstance(java.lang.String,org.apache.spark.sql.types.StructType,org.apache.hadoop.mapreduce.TaskAttemptContext)org.apache.spark.sql.sources.OutputWriter in class org.apache.spark.sql.sources.OutputWriterFactory does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.sources.OutputWriterFactory.newInstance")
[error]  * abstract method newInstance(java.lang.String,scala.Option,org.apache.spark.sql.types.StructType,org.apache.hadoop.mapreduce.TaskAttemptContext)org.apache.spark.sql.sources.OutputWriter in class org.apache.spark.sql.sources.OutputWriterFactory is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.sources.OutputWriterFactory.newInstance")

[SparkQA]

Test build #53378 has finished for PR 11774 at commit fbb93ca.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

merging in master. this is a great change!

[JoshRosen]

I've filed https://issues.apache.org/jira/browse/SPARK-13959 as a followup to make sure to audit the new excludes before we ship 2.0.