Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-6811] Copy SparkR lib in make-distribution.sh #6373

Closed
wants to merge 1 commit into from
Closed

[SPARK-6811] Copy SparkR lib in make-distribution.sh #6373

wants to merge 1 commit into from

Conversation

shivaram
Copy link
Contributor

This change also remove native libraries from SparkR to make sure our distribution works across platforms

Tested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)

I will also test this with YARN soon and update this PR.

@shivaram
Remove native libraries from SparkR
ae41b5c 
Also include the built SparkR package in make-distribution.sh
@shivaram
Copy link
Contributor Author

cc @pwendell

@SparkQA
Copy link

SparkQA commented May 23, 2015

Test build #33389 has finished for PR 6373 at commit ae41b5c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@pwendell
Copy link
Contributor

make-distribution changes LGTM

@shivaram
Copy link
Contributor Author

Thanks @pwendell. I'm running the YARN test right now and will merge if that looks good.

Also cc @davies

@shivaram
Copy link
Contributor Author

The YARN test worked out fine. I might leave this open in case @davies also has time to test this out.
@pwendell Feel free to merge this if its getting close to another RC

@shivaram
Copy link
Contributor Author

Tested this a bit more and I think merging this now will make it easier for others to test this.

asfgit pushed a commit that referenced this pull request May 23, 2015
@shivaram
[SPARK-6811] Copy SparkR lib in make-distribution.sh
c8eb76b 
This change also remove native libraries from SparkR to make sure our distribution works across platforms

Tested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)

I will also test this with YARN soon and update this PR.

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes #6373 from shivaram/sparkr-binary and squashes the following commits:

ae41b5c [Shivaram Venkataraman] Remove native libraries from SparkR Also include the built SparkR package in make-distribution.sh

(cherry picked from commit a40bca0)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
@asfgit asfgit closed this in a40bca0 May 23, 2015
@dragos
Copy link
Contributor

dragos commented May 23, 2015

@shivaram I'm getting this error when I run make-distribution.sh. Can't get around it. What am I missing?

+ mkdir -p /Volumes/Thunderbolt_SSD/dragos/workspace/Spark/dev/spark/dist/R/lib
+ cp -r /Volumes/Thunderbolt_SSD/dragos/workspace/Spark/dev/spark/R/lib/SparkR /Volumes/Thunderbolt_SSD/dragos/workspace/Spark/dev/spark/dist/R/lib
cp: /Volumes/Thunderbolt_SSD/dragos/workspace/Spark/dev/spark/R/lib/SparkR: No such file or directory

@shivaram shivaram mentioned this pull request May 23, 2015
Closed
@shivaram
Copy link
Contributor Author

@dragos The R/lib/SparkR gets created if the maven flag -Psparkr is passed to make-distribution.sh
However this broke backwards compatibility for users who dont need SparkR in their build -- so I've created a hot fix for this in #6379

asfgit pushed a commit that referenced this pull request May 23, 2015
@shivaram
[HOTFIX] Copy SparkR lib if it exists in make-distribution
fbc4480 
This is to fix an issue reported in #6373 where the `cp` would fail if `-Psparkr` was not used in the build

cc dragos pwendell

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes #6379 from shivaram/make-distribution-hotfix and squashes the following commits:

08eb7e4 [Shivaram Venkataraman] Copy SparkR lib if it exists in make-distribution

(cherry picked from commit b231baa)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
asfgit pushed a commit that referenced this pull request May 23, 2015
@shivaram
[HOTFIX] Copy SparkR lib if it exists in make-distribution
b231baa 
This is to fix an issue reported in #6373 where the `cp` would fail if `-Psparkr` was not used in the build

cc dragos pwendell

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes #6379 from shivaram/make-distribution-hotfix and squashes the following commits:

08eb7e4 [Shivaram Venkataraman] Copy SparkR lib if it exists in make-distribution
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@shivaram @jeanlyn
[SPARK-6811] Copy SparkR lib in make-distribution.sh
e867b71 
This change also remove native libraries from SparkR to make sure our distribution works across platforms

Tested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)

I will also test this with YARN soon and update this PR.

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6373 from shivaram/sparkr-binary and squashes the following commits:

ae41b5c [Shivaram Venkataraman] Remove native libraries from SparkR Also include the built SparkR package in make-distribution.sh
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@shivaram @jeanlyn
[HOTFIX] Copy SparkR lib if it exists in make-distribution
d5a3bcb 
This is to fix an issue reported in apache#6373 where the `cp` would fail if `-Psparkr` was not used in the build

cc dragos pwendell

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6379 from shivaram/make-distribution-hotfix and squashes the following commits:

08eb7e4 [Shivaram Venkataraman] Copy SparkR lib if it exists in make-distribution
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@shivaram @jeanlyn
[SPARK-6811] Copy SparkR lib in make-distribution.sh
e8547a3 
This change also remove native libraries from SparkR to make sure our distribution works across platforms

Tested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)

I will also test this with YARN soon and update this PR.

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6373 from shivaram/sparkr-binary and squashes the following commits:

ae41b5c [Shivaram Venkataraman] Remove native libraries from SparkR Also include the built SparkR package in make-distribution.sh
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@shivaram @jeanlyn
[HOTFIX] Copy SparkR lib if it exists in make-distribution
da721ea 
This is to fix an issue reported in apache#6373 where the `cp` would fail if `-Psparkr` was not used in the build

cc dragos pwendell

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6379 from shivaram/make-distribution-hotfix and squashes the following commits:

08eb7e4 [Shivaram Venkataraman] Copy SparkR lib if it exists in make-distribution
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@shivaram @nemccarthy
[SPARK-6811] Copy SparkR lib in make-distribution.sh
74ff595 
This change also remove native libraries from SparkR to make sure our distribution works across platforms

Tested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)

I will also test this with YARN soon and update this PR.

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6373 from shivaram/sparkr-binary and squashes the following commits:

ae41b5c [Shivaram Venkataraman] Remove native libraries from SparkR Also include the built SparkR package in make-distribution.sh
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@shivaram @nemccarthy
[HOTFIX] Copy SparkR lib if it exists in make-distribution
3aedc1a 
This is to fix an issue reported in apache#6373 where the `cp` would fail if `-Psparkr` was not used in the build

cc dragos pwendell

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#6379 from shivaram/make-distribution-hotfix and squashes the following commits:

08eb7e4 [Shivaram Venkataraman] Copy SparkR lib if it exists in make-distribution
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@shivaram @SparkQA @pwendell @dragos