Update version to 1.6.0-SNAPSHOT. #8350

rxin · 2015-08-21T06:16:18Z

No description provided.

SparkQA · 2015-08-21T06:38:44Z

Test build #41352 has finished for PR 8350 at commit a7be5b6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-21T07:10:03Z

Test build #41356 has finished for PR 8350 at commit 4f3b8f6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-08-21T08:10:28Z

pom.xml

@@ -295,6 +295,18 @@
        <enabled>false</enabled>
      </snapshots>
    </repository>
+    <!-- TODO: This can be deleted after Spark 1.5 is posted -->


All this looks fine; does this particular change belong only in the 1.5 branch though?

I'm actually not sure -- I followed Patrick's last pr of this kind.

What happened without this is that Mima would fail and complain about all classes we marked as exclude for 1.5. I actually don't know exactly how mima works. Adding this doesn't seem to help much either.

shivaram · 2015-09-10T00:20:45Z

@rxin Now that 1.5 has been released, we should merge this ?

SparkQA · 2015-09-10T00:52:36Z

Test build #1732 has finished for PR 8350 at commit 4f3b8f6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-09-10T12:27:14Z

Yes, I think this needs MIMA entries for 1.6 now too.

SparkQA · 2015-09-12T07:37:07Z

Test build #42369 has finished for PR 8350 at commit b7887b3.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-09-14T11:08:50Z

I started on figuring out what MiMa excludes have to be brought forward and got this far before being tripped up by more local SBT errors:

    case v if v.startsWith("1.6") =>
      Seq(
        MimaBuild.excludeSparkPackage("network"),
        MimaBuild.excludeSparkPackage("deploy"),
        // These are needed if checking against the sbt build, since they are part of
        // the maven-generated artifacts in 1.3.
        excludePackage("org.spark-project.jetty"),
        MimaBuild.excludeSparkPackage("unused"),
        // JavaRDDLike is not meant to be extended by user programs
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.java.JavaRDDLike.partitioner"),
        // Modification of private static method
        ProblemFilters.exclude[IncompatibleMethTypeProblem](
          "org.apache.spark.streaming.kafka.KafkaUtils.org$apache$spark$streaming$kafka$KafkaUtils$$leadersForRanges"),
        // Mima false positive (was a private[spark] class)
        ProblemFilters.exclude[MissingClassProblem](
          "org.apache.spark.util.collection.PairIterator"),
        // Removing a testing method from a private class
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.streaming.kafka.KafkaTestUtils.waitUntilLeaderOffset")
      ) ++ Seq(
        // SPARK-6797 Support YARN modes for SparkR
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.r.PairwiseRRDD.this"),
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.r.RRDD.createRWorker"),
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.r.RRDD.this"),
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.r.StringRRDD.this"),
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.api.r.BaseRRDD.this")
      ) ++ Seq(
        // SPARK-8701 Add input metadata in the batch page.
        ProblemFilters.exclude[MissingClassProblem](
          "org.apache.spark.streaming.scheduler.InputInfo$"),
        ProblemFilters.exclude[MissingClassProblem](
          "org.apache.spark.streaming.scheduler.InputInfo")
      ) ++ Seq(
        // SPARK-7292 Provide operator to truncate lineage cheaply
        ProblemFilters.exclude[AbstractClassProblem](
          "org.apache.spark.rdd.RDDCheckpointData"),
        ProblemFilters.exclude[AbstractClassProblem](
          "org.apache.spark.rdd.CheckpointRDD")
      ) ++ Seq(
        // SPARK-4751 Dynamic allocation for standalone mode
        ProblemFilters.exclude[MissingMethodProblem](
          "org.apache.spark.SparkContext.supportDynamicAllocation")
      )

Try adding this, running ./dev/mima and then seeing what else needs to be copied forward.

pwendell · 2015-09-14T21:40:35Z

I think if you update previousVersion in MimaBuild.scala many of these should go away. I'm happy to look at the error output after doing that.

SparkQA · 2015-09-14T21:59:07Z

Test build #42450 has finished for PR 8350 at commit ca9ff77.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-09-14T23:19:36Z

I tested it locally, you'll need something like this:

       version match {
+        case v if v.startsWith("1.6") =>
+          Seq(
+            MimaBuild.excludeSparkPackage("deploy"),
+            // These are needed if checking against the sbt build, since they are part of
+            // the maven-generated artifacts in 1.3.
+            excludePackage("org.spark-project.jetty"),
+            MimaBuild.excludeSparkPackage("unused"),
+            ProblemFilters.exclude[MissingClassProblem](
+              "org.apache.spark.sql.execution.datasources.DefaultSource")
+          )
+

SparkQA · 2015-09-14T23:54:36Z

Test build #42455 has finished for PR 8350 at commit ad4e6bd.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-15T01:16:08Z

Test build #42460 has finished for PR 8350 at commit b99703a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-15T05:29:11Z

Test build #1754 has finished for PR 8350 at commit ce5b5bb.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MinMaxScaler(JavaEstimator, HasInputCol, HasOutputCol):
- class MinMaxScalerModel(JavaModel):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(
- case class IntersectNode(conf: SQLConf, left: LocalNode, right: LocalNode)
- case class SampleNode(
- case class TakeOrderedAndProjectNode(

pwendell · 2015-09-15T05:36:34Z

Hm - this is a test file that is in the source folder, so MIMA is
complaining about changes. The error message gives you a filter you can add
to the MIMA file to ignore it.

On Mon, Sep 14, 2015 at 10:29 PM, Apache Spark QA notifications@github.com
wrote:

Test build #1754 has finished
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1754/console
for PR 8350 at commit ce5b5bb
ce5b5bb
.

This patch fails MiMa tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class MinMaxScaler(JavaEstimator, HasInputCol, HasOutputCol):

class MinMaxScalerModel(JavaModel):

case class Stddev(child: Expression) extends StddevAgg(child)

case class StddevPop(child: Expression) extends StddevAgg(child)

case class StddevSamp(child: Expression) extends StddevAgg(child)

abstract class StddevAgg(child: Expression) extends
AlgebraicAggregate

abstract class StddevAgg1(child: Expression) extends
UnaryExpression with PartialAggregate1

case class Stddev(child: Expression) extends StddevAgg1(child)

case class StddevPop(child: Expression) extends StddevAgg1(child)

case class StddevSamp(child: Expression) extends StddevAgg1(child)

case class ComputePartialStd(child: Expression) extends
UnaryExpression with AggregateExpression1

case class ComputePartialStdFunction (

case class MergePartialStd(

case class MergePartialStdFunction(

case class StddevFunction(

case class IntersectNode(conf: SQLConf, left: LocalNode, right:
LocalNode)

case class SampleNode(

case class TakeOrderedAndProjectNode(

—
Reply to this email directly or view it on GitHub
#8350 (comment).

SparkQA · 2015-09-15T07:51:07Z

Test build #42469 has finished for PR 8350 at commit 2178d25.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-09-15T07:54:07Z

Alright I'm going to merge this.

srowen · 2015-09-15T07:56:11Z

I think that's reasonable since the only failure mode this patch should cause is MiMa failure and that passed now. Thanks @pwendell for reminding me about the previousVersion thing, had totally overlooked how that worked.

pwendell · 2015-09-15T22:18:47Z

No problem - I should have caught it earlier. Hopefully people didn't spin
too many cycles on this.

On Tue, Sep 15, 2015 at 12:56 AM, Sean Owen notifications@github.com
wrote:

I think that's reasonable since the only failure mode this patch should
cause is MiMa failure and that passed now. Thanks @pwendell
https://github.com/pwendell for reminding me about the previousVersion
thing, had totally overlooked how that worked.

—
Reply to this email directly or view it on GitHub
#8350 (comment).

srowen reviewed Aug 21, 2015
View reviewed changes

rxin force-pushed the 1.6 branch from 4f3b8f6 to b7887b3 Compare September 12, 2015 07:16

rxin added 4 commits September 14, 2015 14:36

Update version to 1.6.0-SNAPSHOT.

afea187

1.5 snapshot.

9c7e462

Removed temp repo.

97820c6

Update previous Spark version in Mima.

ca9ff77

rxin force-pushed the 1.6 branch from b7887b3 to ca9ff77 Compare September 14, 2015 21:37

Excludes based on Patrick's comment.

ad4e6bd

Mima fixes.

b99703a

Scala style.

ce5b5bb

More mima.

2178d25

asfgit closed this in 09b7e7c Sep 15, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update version to 1.6.0-SNAPSHOT. #8350

Update version to 1.6.0-SNAPSHOT. #8350

rxin commented Aug 21, 2015

SparkQA commented Aug 21, 2015

SparkQA commented Aug 21, 2015

srowen Aug 21, 2015

rxin Aug 21, 2015

shivaram commented Sep 10, 2015

SparkQA commented Sep 10, 2015

srowen commented Sep 10, 2015

SparkQA commented Sep 12, 2015

srowen commented Sep 14, 2015

pwendell commented Sep 14, 2015

SparkQA commented Sep 14, 2015

pwendell commented Sep 14, 2015

SparkQA commented Sep 14, 2015

SparkQA commented Sep 15, 2015

SparkQA commented Sep 15, 2015

pwendell commented Sep 15, 2015

SparkQA commented Sep 15, 2015

rxin commented Sep 15, 2015

srowen commented Sep 15, 2015

pwendell commented Sep 15, 2015

Update version to 1.6.0-SNAPSHOT. #8350

Update version to 1.6.0-SNAPSHOT. #8350

Conversation

rxin commented Aug 21, 2015

SparkQA commented Aug 21, 2015

SparkQA commented Aug 21, 2015

srowen Aug 21, 2015

Choose a reason for hiding this comment

rxin Aug 21, 2015

Choose a reason for hiding this comment

shivaram commented Sep 10, 2015

SparkQA commented Sep 10, 2015

srowen commented Sep 10, 2015

SparkQA commented Sep 12, 2015

srowen commented Sep 14, 2015

pwendell commented Sep 14, 2015

SparkQA commented Sep 14, 2015

pwendell commented Sep 14, 2015

SparkQA commented Sep 14, 2015

SparkQA commented Sep 15, 2015

SparkQA commented Sep 15, 2015

pwendell commented Sep 15, 2015

SparkQA commented Sep 15, 2015

rxin commented Sep 15, 2015

srowen commented Sep 15, 2015

pwendell commented Sep 15, 2015