Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

Closed
wants to merge 3 commits into from
Closed

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
35b6c71
SPARK-3433 Fix for Mima false-positives with @DeveloperAPI and @Exper…
ScrapCodes Sep 5, 2014
59012a8
Update mima
ScrapCodes Sep 9, 2014
093c76f
Update mima
ScrapCodes Sep 9, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions dev/mima
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,15 @@ FWDIR="$(cd `dirname $0`/..; pwd)"
cd "$FWDIR"

echo -e "q\n" | sbt/sbt oldDeps/update
rm -f .generated-mima*

./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need to be called twice? It's now called here and below as well. If it is needed twice, it would be good to document why,


export SPARK_CLASSPATH=`find lib_managed \( -name '*spark*jar' -a -type f \) | tr "\\n" ":"`
echo "SPARK_CLASSPATH=$SPARK_CLASSPATH"

./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore

echo -e "q\n" | sbt/sbt mima-report-binary-issues | grep -v -e "info.*Resolving"
ret_val=$?

Expand Down
6 changes: 6 additions & 0 deletions project/MimaBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ object MimaBuild {

def excludeMember(fullName: String) = Seq(
ProblemFilters.exclude[MissingMethodProblem](fullName),
// Sometimes excluded methods have default arguments and
// they are translated into public methods/fields($default$) in generated
// bytecode. It is not possible to exhustively list everything.
// But this should be okay.
ProblemFilters.exclude[MissingMethodProblem](fullName+"$default$2"),
ProblemFilters.exclude[MissingMethodProblem](fullName+"$default$1"),
ProblemFilters.exclude[MissingFieldProblem](fullName),
ProblemFilters.exclude[IncompatibleResultTypeProblem](fullName),
ProblemFilters.exclude[IncompatibleMethTypeProblem](fullName),
Expand Down
8 changes: 1 addition & 7 deletions project/MimaExcludes.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,8 @@ object MimaExcludes {
Seq(
MimaBuild.excludeSparkPackage("deploy"),
MimaBuild.excludeSparkPackage("graphx")
) ++
// This is @DeveloperAPI, but Mima still gives false-positives:
MimaBuild.excludeSparkClass("scheduler.SparkListenerApplicationStart") ++
Seq(
// This is @Experimental, but Mima still gives false-positives:
ProblemFilters.exclude[MissingMethodProblem](
"org.apache.spark.api.java.JavaRDDLike.foreachAsync")
)

case v if v.startsWith("1.1") =>
Seq(
MimaBuild.excludeSparkPackage("deploy"),
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ object OldDeps {
Some("org.apache.spark" % fullId % "1.1.0")
}

def oldDepsSettings() = Defaults.defaultSettings ++ Seq(
def oldDepsSettings() = Defaults.coreDefaultSettings ++ Seq(
name := "old-deps",
scalaVersion := "2.10.4",
retrieveManaged := true,
Expand Down
14 changes: 10 additions & 4 deletions tools/src/main/scala/org/apache/spark/tools/GenerateMIMAIgnore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import scala.collection.mutable
import scala.collection.JavaConversions._
import scala.reflect.runtime.universe.runtimeMirror
import scala.reflect.runtime.{universe => unv}
import scala.util.Try

/**
* A tool for generating classes to be excluded during binary checking with MIMA. It is expected
Expand Down Expand Up @@ -121,12 +122,17 @@ object GenerateMIMAIgnore {
}

def main(args: Array[String]) {
import scala.tools.nsc.io.File
val (privateClasses, privateMembers) = privateWithin("org.apache.spark")
scala.tools.nsc.io.File(".generated-mima-class-excludes").
writeAll(privateClasses.mkString("\n"))
val previousContents = Try(File(".generated-mima-class-excludes").lines()).
getOrElse(Iterator.empty).mkString("\n")
File(".generated-mima-class-excludes")
.writeAll(previousContents + privateClasses.mkString("\n"))
println("Created : .generated-mima-class-excludes in current directory.")
scala.tools.nsc.io.File(".generated-mima-member-excludes").
writeAll(privateMembers.mkString("\n"))
val previousMembersContents = Try(File(".generated-mima-member-excludes").lines)
.getOrElse(Iterator.empty).mkString("\n")
File(".generated-mima-member-excludes").writeAll(previousMembersContents +
privateMembers.mkString("\n"))
println("Created : .generated-mima-member-excludes in current directory.")
}

Expand Down