[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

ScrapCodes · 2014-09-05T08:29:15Z

Actually false positive reported was due to mima generator not picking up the new jars in presence of old jars(theoretically this should not have happened.). So as a workaround, ran them both separately and just append them together.

ScrapCodes · 2014-09-05T08:30:53Z

project/SparkBuild.scala

@@ -187,7 +187,7 @@ object OldDeps {
    Some("org.apache.spark" % fullId % "1.0.0")
  }

-  def oldDepsSettings() = Defaults.defaultSettings ++ Seq(
+  def oldDepsSettings() = Defaults.coreDefaultSettings ++ Seq(


This just fixes a deprecated warning.

ScrapCodes · 2014-09-05T08:37:05Z

dev/mima

@@ -25,12 +25,16 @@ FWDIR="$(cd `dirname $0`/..; pwd)"
 cd "$FWDIR"

 echo -e "q\n" | sbt/sbt oldDeps/update
+rm -f .generated-mima*
+
+./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore new


run with just new jars first.

I don't see the args array being used in GenerateMIMAIgnore.scala, where are the "new" and "old" arguments being used anywhere?

SparkQA · 2014-09-05T09:39:12Z

QA tests have started for PR 2285 at commit 24f3381.

This patch merges cleanly.

SparkQA · 2014-09-05T10:41:02Z

QA tests have finished for PR 2285 at commit 24f3381.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AddJar(path: String) extends LeafNode with Command

rxin · 2014-09-06T06:12:40Z

Can you change an old API to make sure this is reporting correctly?

ScrapCodes · 2014-09-08T05:12:25Z

test this please.

SparkQA · 2014-09-08T05:47:05Z

QA tests have started for PR 2285 at commit be0d196.

This patch merges cleanly.

SparkQA · 2014-09-08T06:56:32Z

QA tests have finished for PR 2285 at commit be0d196.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

ScrapCodes · 2014-09-08T07:04:27Z

@rxin The test failed as expected. I will fix it once you confirm the code change.

rxin · 2014-09-08T07:05:32Z

LGTM (although I don't really understand this part very well).

…imental

…imental annotations.

ScrapCodes · 2014-09-08T09:54:13Z

I will let @JoshRosen take a look too, since he reported this issue as well.

SparkQA · 2014-09-08T10:47:47Z

QA tests have started for PR 2285 at commit 35b6c71.

This patch merges cleanly.

SparkQA · 2014-09-08T11:47:46Z

QA tests have finished for PR 2285 at commit 35b6c71.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2014-09-08T19:39:54Z

dev/mima

@@ -25,11 +25,15 @@ FWDIR="$(cd `dirname $0`/..; pwd)"
 cd "$FWDIR"

 echo -e "q\n" | sbt/sbt oldDeps/update
+rm -f .generated-mima*
+
+./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore


does this need to be called twice? It's now called here and below as well. If it is needed twice, it would be good to document why,

SparkQA · 2014-09-09T12:49:56Z

QA tests have started for PR 2285 at commit 093c76f.

This patch merges cleanly.

SparkQA · 2014-09-09T13:43:07Z

QA tests have finished for PR 2285 at commit 093c76f.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

ScrapCodes · 2014-09-10T06:47:38Z

Jenkins, retest this please.

SparkQA · 2014-09-12T22:04:32Z

QA tests have started for PR 2285 at commit 093c76f.

This patch merges cleanly.

SparkQA · 2014-09-12T23:03:21Z

QA tests have finished for PR 2285 at commit 093c76f.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- throw new IllegalStateException("The main method in the given main class must be static")
- case class CreateTableAsSelect(
- case class CreateTableAsSelect(

rxin · 2014-09-16T02:14:34Z

@ScrapCodes is this good to merge now?

ScrapCodes · 2014-09-16T02:28:15Z

Yes, it should fix this problem.

rxin · 2014-09-16T02:58:28Z

Jenkins, retest this please.

SparkQA · 2014-09-16T03:04:15Z

QA tests have started for PR 2285 at commit 093c76f.

This patch merges cleanly.

SparkQA · 2014-09-16T04:12:21Z

QA tests have finished for PR 2285 at commit 093c76f.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ArrayConstructor extends net.razorvine.pickle.objects.ArrayConstructor
- class SCCallSiteSync(object):

rxin · 2014-09-16T04:13:45Z

Ok merging this one. Thanks!

JoshRosen · 2015-01-13T00:43:38Z

I've backported this to branch-1.1 (1.1.2) in order to fix a false-positive MiMa failure in that branch.

@experimental

@experimental annotations. Actually false positive reported was due to mima generator not picking up the new jars in presence of old jars(theoretically this should not have happened.). So as a workaround, ran them both separately and just append them together. Author: Prashant Sharma <prashant@apache.org> Author: Prashant Sharma <prashant.s@imaginea.com> Closes #2285 from ScrapCodes/mima-fix and squashes the following commits: 093c76f [Prashant Sharma] Update mima 59012a8 [Prashant Sharma] Update mima 35b6c71 [Prashant Sharma] SPARK-3433 Fix for Mima false-positives with @DeveloperAPI and @experimental annotations. (cherry picked from commit ecf0c02) Signed-off-by: Josh Rosen <joshrosen@databricks.com> Conflicts: project/MimaExcludes.scala

ScrapCodes reviewed Sep 5, 2014
View reviewed changes

ScrapCodes mentioned this pull request Sep 5, 2014

SPARK-2895: Add mapPartitionsWithContext related support on Spark Java API. #2194

Closed

ScrapCodes reviewed Sep 5, 2014
View reviewed changes

ScrapCodes changed the title ~~Fix for false positives reported by mima on PR 2194.~~ [BUILD] Fix for false positives reported by mima on PR 2194. Sep 5, 2014

ScrapCodes force-pushed the mima-fix branch from 71d9deb to be0d196 Compare September 8, 2014 05:14

ScrapCodes force-pushed the mima-fix branch from be0d196 to cd4ec76 Compare September 8, 2014 09:45

SPARK-3433 Fix for Mima false-positives with @DeveloperAPI and @exper…

35b6c71

…imental annotations.

ScrapCodes force-pushed the mima-fix branch from cd4ec76 to 35b6c71 Compare September 8, 2014 09:47

ScrapCodes changed the title ~~[BUILD] Fix for false positives reported by mima on PR 2194.~~ [SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. Sep 8, 2014

pwendell reviewed Sep 8, 2014
View reviewed changes

ScrapCodes added 2 commits September 9, 2014 17:50

Update mima

59012a8

Update mima

093c76f

asfgit closed this in ecf0c02 Sep 16, 2014

ScrapCodes deleted the mima-fix branch June 3, 2015 05:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

ScrapCodes commented Sep 5, 2014

ScrapCodes Sep 5, 2014

ScrapCodes Sep 5, 2014

vanzin Sep 5, 2014

SparkQA commented Sep 5, 2014

SparkQA commented Sep 5, 2014

rxin commented Sep 6, 2014

ScrapCodes commented Sep 8, 2014

SparkQA commented Sep 8, 2014

SparkQA commented Sep 8, 2014

ScrapCodes commented Sep 8, 2014

rxin commented Sep 8, 2014

ScrapCodes commented Sep 8, 2014

SparkQA commented Sep 8, 2014

SparkQA commented Sep 8, 2014

pwendell Sep 8, 2014

SparkQA commented Sep 9, 2014

SparkQA commented Sep 9, 2014

ScrapCodes commented Sep 10, 2014

SparkQA commented Sep 12, 2014

SparkQA commented Sep 12, 2014

rxin commented Sep 16, 2014

ScrapCodes commented Sep 16, 2014

rxin commented Sep 16, 2014

SparkQA commented Sep 16, 2014

SparkQA commented Sep 16, 2014

rxin commented Sep 16, 2014

JoshRosen commented Jan 13, 2015

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

[SPARK-3433][BUILD] Fix for Mima false-positives with @DeveloperAPI and @Experimental annotations. #2285

Conversation

ScrapCodes commented Sep 5, 2014

ScrapCodes Sep 5, 2014

Choose a reason for hiding this comment

ScrapCodes Sep 5, 2014

Choose a reason for hiding this comment

vanzin Sep 5, 2014

Choose a reason for hiding this comment

SparkQA commented Sep 5, 2014

SparkQA commented Sep 5, 2014

rxin commented Sep 6, 2014

ScrapCodes commented Sep 8, 2014

SparkQA commented Sep 8, 2014

SparkQA commented Sep 8, 2014

ScrapCodes commented Sep 8, 2014

rxin commented Sep 8, 2014

ScrapCodes commented Sep 8, 2014

SparkQA commented Sep 8, 2014

SparkQA commented Sep 8, 2014

pwendell Sep 8, 2014

Choose a reason for hiding this comment

SparkQA commented Sep 9, 2014

SparkQA commented Sep 9, 2014

ScrapCodes commented Sep 10, 2014

SparkQA commented Sep 12, 2014

SparkQA commented Sep 12, 2014

rxin commented Sep 16, 2014

ScrapCodes commented Sep 16, 2014

rxin commented Sep 16, 2014

SparkQA commented Sep 16, 2014

SparkQA commented Sep 16, 2014

rxin commented Sep 16, 2014

JoshRosen commented Jan 13, 2015