New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

Closed
wants to merge 4 commits into
base: master
from

Conversation

Projects
3 participants
@fnothaft
Member

fnothaft commented Jan 1, 2018

Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full, self-contained pip install-able Spark by finding the JARs and bin scripts and packaging them up as packages which are deployed to pip. We then needed to modify the bin scripts to find the pip installed JARs.

To test this out, from inside of a virtualenv, do:

pip install pyspark
pip install --index-url https://test.pypi.org/simple/ bdgenomics.adam==0.23.0rc18 

You should be able to run adam-shell, adam-submit, etc. You can see the release docs here.

As you can tell, this took a bit of trial and error...

[ADAM-1847] Update ADAM scripts to support self-contained pip install.
Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full,
self-contained pip install-able Spark by finding the JARs and bin scripts and
packaging them up as packages which are deployed to pip. We then needed to
modify the bin scripts to find the pip installed JARs.

@fnothaft fnothaft added this to the 0.23.0 milestone Jan 1, 2018

@fnothaft fnothaft requested review from heuermh and akmorrow13 Jan 1, 2018

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2533/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 0c5c82b # timeout=10Checking out Revision 0c5c82b (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0c5c82bf0c49ab5bdb11c84e4ca569153278db14First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2533/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 0c5c82b # timeout=10Checking out Revision 0c5c82b (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0c5c82bf0c49ab5bdb11c84e4ca569153278db14First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Show outdated Hide outdated adam-python/setup.py Outdated
Show outdated Hide outdated adam-python/setup.py Outdated
Show outdated Hide outdated adam-python/setup.py Outdated
Show outdated Hide outdated adam-python/setup.py Outdated
Show outdated Hide outdated adam-python/version.py Outdated
Show outdated Hide outdated bin/find-adam-home Outdated
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 1, 2018

Member

Thanks for the quick review @heuermh! I've addressed your comments, and the commits are OK to squash down. Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Member

fnothaft commented Jan 1, 2018

Thanks for the quick review @heuermh! I've addressed your comments, and the commits are OK to squash down. Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

rm -rf release-venv
virtualenv release-venv
. release-venv/bin/activate
pip install pyspark

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018

Member

I don't have pip installed by default, does that come in the virtualenv?

@heuermh

heuermh Jan 1, 2018

Member

I don't have pip installed by default, does that come in the virtualenv?

This comment has been minimized.

@fnothaft

fnothaft Jan 1, 2018

Member

Correct; virtualenvs include their own pip.

@fnothaft

fnothaft Jan 1, 2018

Member

Correct; virtualenvs include their own pip.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 1, 2018

Member

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server? How are those credentials handled by the release script?

Member

heuermh commented Jan 1, 2018

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server? How are those credentials handled by the release script?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 1, 2018

Member

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server?

Yup; you'll need to set up an account following these directions.

How are those credentials handled by the release script?

They can be stored in a plaintext file (urgh), but the twine command in the release script prompts for credentials before beginning the upload.

Member

fnothaft commented Jan 1, 2018

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server?

Yup; you'll need to set up an account following these directions.

How are those credentials handled by the release script?

They can be stored in a plaintext file (urgh), but the twine command in the release script prompts for credentials before beginning the upload.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2534/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains bb54419 # timeout=10Checking out Revision bb54419 (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bb54419315a3505d657508a61346769225bbddc4First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2534/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains bb54419 # timeout=10Checking out Revision bb54419 (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bb54419315a3505d657508a61346769225bbddc4First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 2, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2536/
Test PASSed.

AmplabJenkins commented Jan 2, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2536/
Test PASSed.

@fnothaft fnothaft removed the request for review from akmorrow13 Jan 3, 2018

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 3, 2018

Member

Closing in favor of this merging in as a part of #1849.

Member

fnothaft commented Jan 3, 2018

Closing in favor of this merging in as a part of #1849.

@fnothaft fnothaft closed this Jan 3, 2018

@fnothaft fnothaft deleted the fnothaft:issues/1847-pip-release branch Jan 3, 2018

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment