Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1847] Update ADAM scripts to support self-contained pip install. #1848

Closed

Conversation

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jan 1, 2018

Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full, self-contained pip install-able Spark by finding the JARs and bin scripts and packaging them up as packages which are deployed to pip. We then needed to modify the bin scripts to find the pip installed JARs.

To test this out, from inside of a virtualenv, do:

pip install pyspark
pip install --index-url https://test.pypi.org/simple/ bdgenomics.adam==0.23.0rc18 

You should be able to run adam-shell, adam-submit, etc. You can see the release docs here.

As you can tell, this took a bit of trial and error...

Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full,
self-contained pip install-able Spark by finding the JARs and bin scripts and
packaging them up as packages which are deployed to pip. We then needed to
modify the bin scripts to find the pip installed JARs.
@fnothaft fnothaft added this to the 0.23.0 milestone Jan 1, 2018
@fnothaft fnothaft requested review from heuermh and akmorrow13 Jan 1, 2018
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2533/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 0c5c82b # timeout=10Checking out Revision 0c5c82b (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0c5c82bf0c49ab5bdb11c84e4ca569153278db14First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.


# Provide guidance about how to use setup.py
incorrect_invocation_message = """
If you are installing PyADAM from ADAM's source, you must first build ADAM and

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

We don't actually call this PyADAM anywhere, right? Rather bdgenomics.adam, or Python support for ADAM, or somesuch.

if len(JARS_PATH) == 1:
JARS_PATH = JARS_PATH[0]
elif len(JARS_PATH) > 1:
print("Assembly jars exist for multiple scalas ({0}), please cleanup assembly/target".format(

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

scalas → Scala versions

ADAM_HOME = os.path.abspath("../")
TEMP_PATH = "deps"

JARS_PATH = glob.glob(os.path.join(ADAM_HOME, "adam-assembly/target/adam-assembly*.jar"))

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

Do we need the same regex mess to avoid -sources and -javadoc jars as in find-adam-assembly.sh here?

import pypandoc
long_description = pypandoc.convert('README.md', 'rst')
except ImportError:
print("Could not import pypandoc - required to package PySpark", file=sys.stderr)

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

PySpark?

@@ -16,7 +16,7 @@
# limitations under the License.
#

version = '0.23.0-SNAPSHOT'
version = '0.23.0rc18'

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

Evidence of trial and error noted. ;) Was this meant to be committed?

exit 0
elif [ ! -f "$FIND_ADAM_HOME_PYTHON_SCRIPT" ]; then
# If we are not in the same directory as find_adam_home.py we are not pip installed so we don't
# need to search the different Python directories for a Adam installation.

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

for a Adam → for an ADAM

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jan 1, 2018

Thanks for the quick review @heuermh! I've addressed your comments, and the commits are OK to squash down. Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

rm -rf release-venv
virtualenv release-venv
. release-venv/bin/activate
pip install pyspark

This comment has been minimized.

@heuermh

heuermh Jan 1, 2018
Member

I don't have pip installed by default, does that come in the virtualenv?

This comment has been minimized.

@fnothaft

fnothaft Jan 1, 2018
Author Member

Correct; virtualenvs include their own pip.

@heuermh
Copy link
Member

@heuermh heuermh commented Jan 1, 2018

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server? How are those credentials handled by the release script?

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jan 1, 2018

Can you run through the scripts on your side and make sure that releasing to the test PyPI server works for you as well?

Sure, do I need an account for pushing to the test PyPI server?

Yup; you'll need to set up an account following these directions.

How are those credentials handled by the release script?

They can be stored in a plaintext file (urgh), but the twine command in the release script prompts for credentials before beginning the upload.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2534/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1848/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains bb54419 # timeout=10Checking out Revision bb54419 (origin/pr/1848/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f bb54419315a3505d657508a61346769225bbddc4First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jan 2, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2536/
Test PASSed.

@fnothaft fnothaft removed the request for review from akmorrow13 Jan 3, 2018
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jan 3, 2018

Closing in favor of this merging in as a part of #1849.

@fnothaft fnothaft closed this Jan 3, 2018
@fnothaft fnothaft deleted the fnothaft:issues/1847-pip-release branch Jan 3, 2018
@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.