[SPARK-8781] Fix variables in published pom.xml are not resolved #7193

andrewor14 · 2015-07-02T18:02:49Z

The issue is summarized in the JIRA and is caused by this commit: 984ad60.

This patch reverts that commit and fixes the maven build in a different way. We limit the dependencies of KinesisReceiverSuite to avoid having to deal with the complexities in how maven deals with transitive test dependencies.

This reverts commit 984ad60.

AmplabJenkins · 2015-07-02T18:03:11Z

Merged build triggered.

AmplabJenkins · 2015-07-02T18:03:19Z

Merged build started.

SparkQA · 2015-07-02T18:04:42Z

Test build #36417 has started for PR 7193 at commit 5cb5f45.

vanzin · 2015-07-02T18:12:31Z

extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala

+// To avoid introducing a dependency on Spark core tests, simply use scalatest's FunSuite
+// here instead of our own SparkFunSuite. Introducing the dependency has caused problems
+// in the past (SPARK-8781) that are complicated by bugs in the maven shade plugin (MSHADE-148).
+import org.scalatest.{BeforeAndAfter, Matchers, FunSuite}


super nit: ordering

ok, I fixed

vanzin · 2015-07-02T18:12:45Z

LGTM.

KinesisReceiverSuite really doesn't need to depend on TestSuiteBase. The issue is that TestSuiteBase transitively depends on Spark core tests (SparkFunSuite), but there is no simple way in maven to express transitive test dependencies. To simplify the build structure of this module, we should just have KinesisReceiverSuite extend directly from FunSuite.

AmplabJenkins · 2015-07-02T18:28:10Z

Merged build triggered.

AmplabJenkins · 2015-07-02T18:28:19Z

Merged build started.

AmplabJenkins · 2015-07-02T18:43:30Z

Merged build finished. Test FAILed.

andrewor14 · 2015-07-02T19:14:09Z

retest this please

AmplabJenkins · 2015-07-02T19:18:10Z

Merged build triggered.

AmplabJenkins · 2015-07-02T19:18:18Z

Merged build started.

SparkQA · 2015-07-02T19:21:37Z

Test build #36425 has started for PR 7193 at commit ca3d5d4.

SparkQA · 2015-07-02T20:11:32Z

Test build #36417 has finished for PR 7193 at commit 5cb5f45.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
- case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression

AmplabJenkins · 2015-07-02T20:12:20Z

Merged build finished. Test PASSed.

tdas · 2015-07-02T20:23:22Z

LGTM.

andrewor14 · 2015-07-02T20:49:16Z

Merging into master, 1.4, and 1.3. Thanks for the reviews.

The issue is summarized in the JIRA and is caused by this commit: 984ad60. This patch reverts that commit and fixes the maven build in a different way. We limit the dependencies of `KinesisReceiverSuite` to avoid having to deal with the complexities in how maven deals with transitive test dependencies. Author: Andrew Or <andrew@databricks.com> Closes #7193 from andrewor14/fix-kinesis-pom and squashes the following commits: ca3d5d4 [Andrew Or] Limit kinesis test dependencies f24e09c [Andrew Or] Revert "[BUILD] Fix Maven build for Kinesis" (cherry picked from commit 82cf331) Signed-off-by: Andrew Or <andrew@databricks.com>

The issue is summarized in the JIRA and is caused by this commit: 984ad60. This patch reverts that commit and fixes the maven build in a different way. We limit the dependencies of `KinesisReceiverSuite` to avoid having to deal with the complexities in how maven deals with transitive test dependencies. Author: Andrew Or <andrew@databricks.com> Closes #7193 from andrewor14/fix-kinesis-pom and squashes the following commits: ca3d5d4 [Andrew Or] Limit kinesis test dependencies f24e09c [Andrew Or] Revert "[BUILD] Fix Maven build for Kinesis" (cherry picked from commit 82cf331) Signed-off-by: Andrew Or <andrew@databricks.com> Conflicts: extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala

SparkQA · 2015-07-02T21:33:34Z

Test build #36425 has finished for PR 7193 at commit ca3d5d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-02T21:34:13Z

Merged build finished. Test PASSed.

This is a workaround for MSHADE-148, which leads to an infinite loop when building Spark with maven 3.3.x. This was originally caused by #6441, which added a bunch of test dependencies on the spark-core test module. Recently, it was revealed by #7193. This patch adds a `-Prelease` profile. If present, it will set `createDependencyReducedPom` to true. The consequences are: - If you are releasing Spark with this profile, you are fine as long as you use maven 3.2.x or before. - If you are releasing Spark without this profile, you will run into SPARK-8781. - If you are not releasing Spark but you are using this profile, you may run into SPARK-8819. - If you are not releasing Spark and you did not include this profile, you are fine. This is all documented in `pom.xml` and tested locally with both versions of maven. Author: Andrew Or <andrew@databricks.com> Closes #7219 from andrewor14/fix-maven-build and squashes the following commits: 1d37e87 [Andrew Or] Merge branch 'master' of github.com:apache/spark into fix-maven-build 3574ae4 [Andrew Or] Review comments f39199c [Andrew Or] Create a -Prelease profile that flags `createDependencyReducedPom` (cherry picked from commit 9eae5fa) Signed-off-by: Andrew Or <andrew@databricks.com>

This is a workaround for MSHADE-148, which leads to an infinite loop when building Spark with maven 3.3.x. This was originally caused by #6441, which added a bunch of test dependencies on the spark-core test module. Recently, it was revealed by #7193. This patch adds a `-Prelease` profile. If present, it will set `createDependencyReducedPom` to true. The consequences are: - If you are releasing Spark with this profile, you are fine as long as you use maven 3.2.x or before. - If you are releasing Spark without this profile, you will run into SPARK-8781. - If you are not releasing Spark but you are using this profile, you may run into SPARK-8819. - If you are not releasing Spark and you did not include this profile, you are fine. This is all documented in `pom.xml` and tested locally with both versions of maven. Author: Andrew Or <andrew@databricks.com> Closes #7219 from andrewor14/fix-maven-build and squashes the following commits: 1d37e87 [Andrew Or] Merge branch 'master' of github.com:apache/spark into fix-maven-build 3574ae4 [Andrew Or] Review comments f39199c [Andrew Or] Create a -Prelease profile that flags `createDependencyReducedPom`

This is a workaround for MSHADE-148, which leads to an infinite loop when building Spark with maven 3.3.x. This was originally caused by #6441, which added a bunch of test dependencies on the spark-core test module. Recently, it was revealed by #7193. This patch adds a `-Prelease` profile. If present, it will set `createDependencyReducedPom` to true. The consequences are: - If you are releasing Spark with this profile, you are fine as long as you use maven 3.2.x or before. - If you are releasing Spark without this profile, you will run into SPARK-8781. - If you are not releasing Spark but you are using this profile, you may run into SPARK-8819. - If you are not releasing Spark and you did not include this profile, you are fine. This is all documented in `pom.xml` and tested locally with both versions of maven. Author: Andrew Or <andrew@databricks.com> Closes #7219 from andrewor14/fix-maven-build and squashes the following commits: 1d37e87 [Andrew Or] Merge branch 'master' of github.com:apache/spark into fix-maven-build 3574ae4 [Andrew Or] Review comments f39199c [Andrew Or] Create a -Prelease profile that flags `createDependencyReducedPom` Conflicts: dev/create-release/create-release.sh pom.xml

andrewor14 · 2015-07-07T02:29:23Z

Alright, I reverted this patch in master, 1.4 and 1.3 because #7219 proposes a nicer fix.

Revert "[BUILD] Fix Maven build for Kinesis"

f24e09c

This reverts commit 984ad60.

vanzin reviewed Jul 2, 2015
View reviewed changes

andrewor14 force-pushed the fix-kinesis-pom branch from 5cb5f45 to ca3d5d4 Compare July 2, 2015 18:24

asfgit closed this in 82cf331 Jul 2, 2015

andrewor14 deleted the fix-kinesis-pom branch July 2, 2015 20:53

tarekbecker mentioned this pull request Jul 3, 2015

[HOTFIX][BUILD] Cannot build because of: [INFO] Dependency-reduced POM written at: … #7217

Closed

andrewor14 mentioned this pull request Jul 4, 2015

[SPARK-8819] Fix build for maven 3.3.x #7219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8781] Fix variables in published pom.xml are not resolved #7193

[SPARK-8781] Fix variables in published pom.xml are not resolved #7193

andrewor14 commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

vanzin Jul 2, 2015

andrewor14 Jul 2, 2015

vanzin commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

andrewor14 commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

tdas commented Jul 2, 2015

andrewor14 commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

andrewor14 commented Jul 7, 2015

[SPARK-8781] Fix variables in published pom.xml are not resolved #7193

[SPARK-8781] Fix variables in published pom.xml are not resolved #7193

Conversation

andrewor14 commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

vanzin Jul 2, 2015

Choose a reason for hiding this comment

andrewor14 Jul 2, 2015

Choose a reason for hiding this comment

vanzin commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

andrewor14 commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

tdas commented Jul 2, 2015

andrewor14 commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

andrewor14 commented Jul 7, 2015