[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

JoshRosen · 2015-12-23T23:46:33Z

This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath.

This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs.

This patch is based on @pwendell's work in #8531.

Closes #8531.

Conflicts: dev/run-tests-jenkins dev/sparktestsupport/modules.py

…atching)

JoshRosen · 2015-12-23T23:47:18Z

/cc @pwendell @srowen @brennonyork @nchammas

JoshRosen · 2015-12-30T08:10:55Z

Uh oh, I'd hold off on merging this pending investigation of issues with the dummy change.

SparkQA · 2015-12-30T09:06:22Z

Test build #48452 has finished for PR 10461 at commit 9932c80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T19:22:15Z

Spotted the problem: the original logic for checking should_run_build_tests was wrong: it needs to loop over all test_modules (using any) and the root module needs to have the build tests enabled.

SparkQA · 2015-12-30T19:30:12Z

Test build #48503 has finished for PR 10461 at commit 1454f5f.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T19:38:27Z

Woohoo, that worked! However, I'm not a fan of how vanilla diff formats its output, so I'm going to quickly try and see if I can use git diff for this.

SparkQA · 2015-12-30T20:10:14Z

Test build #48505 has finished for PR 10461 at commit 329bdf2.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T20:46:25Z

Going to merge this into master and will email the dev list with a summary describing the changes + impact on process.

SparkQA · 2015-12-30T22:04:43Z

Test build #48507 has finished for PR 10461 at commit cc99de6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in #10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closes #10543 from JoshRosen/dep-script-fixes.

…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closes apache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closes apache#10461 from JoshRosen/SPARK-10359.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in apache#10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closes apache#10543 from JoshRosen/dep-script-fixes.

…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closes apache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closes apache#10461 from JoshRosen/SPARK-10359.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in apache#10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closes apache#10543 from JoshRosen/dep-script-fixes.

… branch-1.6 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.6. Author: Josh Rosen <joshrosen@databricks.com> Closes #10680 from JoshRosen/test-deps-16-backport.

…o branch-1.5 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.5. Author: Josh Rosen <joshrosen@databricks.com> Closes #10679 from JoshRosen/test-deps-15-backport and squashes the following commits: 7633793 [Josh Rosen] Fix merge conflicts. 1d60cb6 [Josh Rosen] [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching 889c355 [Josh Rosen] [SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script 4e48f14 [Josh Rosen] [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests

pwendell and others added 26 commits August 28, 2015 17:18

Adding build test module

9cf442d

Adding new deps file without spark

20ae791

Moving build tests earlier

5670b6a

Adding license header

5b2910b

Adding RAT exclude

0c8153d

Merge remote-tracking branch 'apache/master' into dependency-audits

f4243b9

Conflicts: dev/run-tests-jenkins dev/sparktestsupport/modules.py

Ignore spark itself in deps file

17970f7

Style fix

16adab6

Adding test block

8b973bf

Only include compile dependencies when generating list

d181e3d

Adding Hadoop 1 profile

84cab2c

Adding new hadoop versions

5c8ccf6

Bug fix

fcea19f

Adding new hadoop deps

2f4d3e5

More RAT excludes

37230f0

Testing new dependencies

1d91634

Merge remote-tracking branch 'origin/master' into SPARK-10359

1e3ff47

Refactor to avoid compile in install.

b013514

Don't generate version backup POMs.

8f7007e

Use bash exit traps to ensure proper cleanup

258e4f7

Whoops, missed another -DgenerateBackupPoms=false

e8af274

Support multiple Hadoop profiles.

8507ffd

Roll back test dependency change.

2711252

Fix RAT excludes (CLI RAT doesn't seem to support full path pattern m…

4ad3ae4

…atching)

dev/test-dependencies.sh is part of the build module.

7c86eb7

Use runtime scope (per Sean's comments)

87918d8

JoshRosen mentioned this pull request Dec 23, 2015

[SPARK-4123][Project Infra]: Show new dependencies added in pull requests #5093

Closed

JoshRosen added 2 commits December 23, 2015 15:48

Remove old dependency diffing code.

15a0e68

Remove accidentally-added files

5acdd9a

JoshRosen changed the title ~~[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests~~ [SPARK-10359][DONOTMERGE] Enumerate dependencies in a file and diff against it for new pull requests Dec 30, 2015

Changes to test module system

1454f5f

JoshRosen added 3 commits December 30, 2015 11:53

Use git diff for nicer diff formatting.

329bdf2

Minor change in exit trap

367c8ed

Roll back test dependency change.

cc99de6

JoshRosen changed the title ~~[SPARK-10359][DONOTMERGE] Enumerate dependencies in a file and diff against it for new pull requests~~ [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests Dec 30, 2015

asfgit closed this in 27a42c7 Dec 30, 2015

JoshRosen deleted the SPARK-10359 branch December 30, 2015 20:50

JoshRosen mentioned this pull request Jan 1, 2016

[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script #10543

Closed

JoshRosen mentioned this pull request Jan 10, 2016

[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.5 #10679

Closed

JoshRosen mentioned this pull request Jan 10, 2016

[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.6 #10680

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

JoshRosen commented Dec 23, 2015

JoshRosen commented Dec 23, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

Conversation

JoshRosen commented Dec 23, 2015

JoshRosen commented Dec 23, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015

JoshRosen commented Dec 30, 2015

SparkQA commented Dec 30, 2015