Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7485] [build] Remove pyspark files from assembly. #6022

Closed
wants to merge 2 commits into from
Closed

[SPARK-7485] [build] Remove pyspark files from assembly. #6022

wants to merge 2 commits into from

Conversation

vanzin
Copy link
Contributor

@vanzin vanzin commented May 9, 2015

The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.

[SPARK-7485] [build] Remove pyspark files from assembly.
4893622 
The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 9, 2015

Test build #32287 has started for PR 6022 at commit 4893622.

@SparkQA
Copy link

SparkQA commented May 9, 2015

Test build #32287 has finished for PR 6022 at commit 4893622.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32287/
Test PASSed.

@srowen
Copy link
Member

srowen commented May 9, 2015

LGTM, certainly the Maven parts. I take your word on the SBT part.

@lianhuiwang lianhuiwang mentioned this pull request May 10, 2015
Closed
lianhuiwang
lianhuiwang reviewed May 10, 2015
View reviewed changes
project/SparkBuild.scala
@@ -388,7 +387,8 @@ object PySparkAssembly {
if (!dst.isDirectory()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think these codes also can be removed.

@tgravescs
Copy link
Contributor

looks good other then the one comment about removing the dst directory stuff from sbt

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32406 has started for PR 6022 at commit 22c1e04.

@andrewor14
Copy link
Contributor

LGTM. Great to see so much code removed!

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32406 has finished for PR 6022 at commit 22c1e04.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32406/
Test PASSed.

@mengxr
Copy link
Contributor

mengxr commented May 11, 2015

@vanzin Does it mean that Maven users have to use sbt to include Python support?

@vanzin
Copy link
Contributor Author

vanzin commented May 11, 2015

No. See the bug referenced in this PR.

@andrewor14
Copy link
Contributor

Merging into master 1.4.

asfgit pushed a commit that referenced this pull request May 12, 2015
[SPARK-7485] [BUILD] Remove pyspark files from assembly.
afe54b7 
The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #6022 from vanzin/SPARK-7485 and squashes the following commits:

22c1e04 [Marcelo Vanzin] Remove unneeded code.
4893622 [Marcelo Vanzin] [SPARK-7485] [build] Remove pyspark files from assembly.

(cherry picked from commit 82e890f)
Signed-off-by: Andrew Or <andrew@databricks.com>
@asfgit asfgit closed this in 82e890f May 12, 2015
@vanzin vanzin deleted the SPARK-7485 branch May 12, 2015 20:25
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
[SPARK-7485] [BUILD] Remove pyspark files from assembly.
0624add 
The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#6022 from vanzin/SPARK-7485 and squashes the following commits:

22c1e04 [Marcelo Vanzin] Remove unneeded code.
4893622 [Marcelo Vanzin] [SPARK-7485] [build] Remove pyspark files from assembly.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-7485] [BUILD] Remove pyspark files from assembly.
478d359 
The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#6022 from vanzin/SPARK-7485 and squashes the following commits:

22c1e04 [Marcelo Vanzin] Remove unneeded code.
4893622 [Marcelo Vanzin] [SPARK-7485] [build] Remove pyspark files from assembly.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-7485] [BUILD] Remove pyspark files from assembly.
024659a 
The sbt part of the build is hacky; it basically tricks sbt
into generating the zip by using a generator, but returns
an empty list for the generated files so that nothing is
actually added to the assembly.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#6022 from vanzin/SPARK-7485 and squashes the following commits:

22c1e04 [Marcelo Vanzin] Remove unneeded code.
4893622 [Marcelo Vanzin] [SPARK-7485] [build] Remove pyspark files from assembly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
8 participants
@vanzin @AmplabJenkins @SparkQA @srowen @tgravescs @andrewor14 @mengxr @lianhuiwang