New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1783] Resolve check issues that block pushing to CRAN. #1849

Merged
merged 2 commits into from Jan 4, 2018

Conversation

Projects
3 participants
@fnothaft
Member

fnothaft commented Jan 2, 2018

Based on #1848. Resolves #1783 and #1847. Cleans up a host of documentation issues that would cause warnings when submitting to CRAN. TODO for tomorrow:

- [ ] Make sure that JAR is packaged in properly
- [ ] Test on linux
- [ ] Possibly test on Windows via CRAN windows test server
- [ ] Add CRAN markdown readme

  • Integrate into release flow

@fnothaft fnothaft added this to the 0.23.0 milestone Jan 2, 2018

@fnothaft fnothaft requested a review from heuermh Jan 2, 2018

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 2, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2535/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1849/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 7c4f9a3 # timeout=10Checking out Revision 7c4f9a3 (origin/pr/1849/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 7c4f9a3328d1e79afb3b23f453e4ec0a1c7b338bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 2, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2535/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1849/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 7c4f9a3 # timeout=10Checking out Revision 7c4f9a3 (origin/pr/1849/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 7c4f9a3328d1e79afb3b23f453e4ec0a1c7b338bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Show outdated Hide outdated adam-r/bdgenomics.adam/R/rdd.R Outdated
Show outdated Hide outdated adam-r/bdgenomics.adam/R/rdd.R Outdated
Show outdated Hide outdated adam-r/bdgenomics.adam/R/rdd.R Outdated
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 2, 2018

Member

As far as I know, SparkR ran into trouble with the CRAN submission process because Spark dropped temporary files in places it shouldn't. We might need to look out for the same. I had some Spark JIRA issues on the topic bookmarked but apparently not on this computer.

Member

heuermh commented Jan 2, 2018

As far as I know, SparkR ran into trouble with the CRAN submission process because Spark dropped temporary files in places it shouldn't. We might need to look out for the same. I had some Spark JIRA issues on the topic bookmarked but apparently not on this computer.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 3, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2537/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1849/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 9628690 # timeout=10Checking out Revision 9628690 (origin/pr/1849/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 962869088b4d0d35fc231fca86e3b0aa120fb6c0First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 3, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2537/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1849/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 9628690 # timeout=10Checking out Revision 9628690 (origin/pr/1849/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 962869088b4d0d35fc231fca86e3b0aa120fb6c0First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 3, 2018

Member

Jenkins, retest this please.

Member

fnothaft commented Jan 3, 2018

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2538/
Test PASSed.

AmplabJenkins commented Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2538/
Test PASSed.

@heuermh

heuermh approved these changes Jan 3, 2018

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 3, 2018

Member

As per discussion with @heuermh, we don't think we can push to CRAN until SparkR is back in CRAN due to:

Packages on which a CRAN package depends should be available from a mainstream
repository: if any mentioned in ‘Suggests’ or ‘Enhances’ fields are not from
such a repository, where to obtain them at a repository should be specified in an
‘Additional_repositories’ field of the DESCRIPTION file (as a comma-separated list
of repository URLs) or for other means of access, described in the ‘Description’ field.
A package listed in ‘Suggests’ or ‘Enhances’ should be used conditionally in examples
or tests if it cannot straightforwardly be installed on the major R platforms

With SparkR being temporarily removed from CRAN, it is not available from a mainstream repository. The above snippet is from the CRAN policy guide. In lieu of CRAN, we will distribute the R tarball.

Member

fnothaft commented Jan 3, 2018

As per discussion with @heuermh, we don't think we can push to CRAN until SparkR is back in CRAN due to:

Packages on which a CRAN package depends should be available from a mainstream
repository: if any mentioned in ‘Suggests’ or ‘Enhances’ fields are not from
such a repository, where to obtain them at a repository should be specified in an
‘Additional_repositories’ field of the DESCRIPTION file (as a comma-separated list
of repository URLs) or for other means of access, described in the ‘Description’ field.
A package listed in ‘Suggests’ or ‘Enhances’ should be used conditionally in examples
or tests if it cannot straightforwardly be installed on the major R platforms

With SparkR being temporarily removed from CRAN, it is not available from a mainstream repository. The above snippet is from the CRAN policy guide. In lieu of CRAN, we will distribute the R tarball.

@fnothaft fnothaft referenced this pull request Jan 3, 2018

Open

Push to CRAN #1851

0 of 5 tasks complete

fnothaft added some commits Dec 30, 2017

[ADAM-1847] Update ADAM scripts to support self-contained pip install.
Resolves #1847. Cribs heavily from PySpark's script flow for supporting a full,
self-contained pip install-able Spark by finding the JARs and bin scripts and
packaging them up as packages which are deployed to pip. We then needed to
modify the bin scripts to find the pip installed JARs.
[ADAM-1783] Resolve check issues that block pushing to CRAN.
Resolves #1783. Cleans up a host of documentation issues that would cause
warnings when submitting to CRAN.
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2539/
Test PASSed.

AmplabJenkins commented Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2539/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2540/
Test PASSed.

AmplabJenkins commented Jan 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2540/
Test PASSed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

Are twine and pypandoc supposed to be installed as part of the virtualenv?

...
adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ virtualenv release-venv
Using base prefix '/Users/mheuer2/miniconda3'
New python executable in /Users/mheuer2/working/adam/adam-python/release-venv/bin/python
Installing setuptools, pip, wheel...done.

adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ . release-venv/bin/activate

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ pip install pyspark
Collecting pyspark
  Downloading pyspark-2.2.0.post0.tar.gz (188.3MB)
    100% |████████████████████████████████| 188.3MB 9.3kB/s
Collecting py4j==0.10.4 (from pyspark)
  Downloading py4j-0.10.4-py2.py3-none-any.whl (186kB)
    100% |████████████████████████████████| 194kB 4.4MB/s
Building wheels for collected packages: pyspark
  Running setup.py bdist_wheel for pyspark ... done
  Stored in directory: /Users/mheuer2/Library/Caches/pip/wheels/5f/0b/b3/5cb16b15d28dcc32f8e7ec91a044829642874bb7586f6e6cbe
Successfully built pyspark
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.4 pyspark-2.2.0

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ rm -rf dist

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ make sdist
python2.7 setup.py sdist
Could not import pypandoc - required to package bdgenomics.adam
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:360: UserWarning: The version specified ('0.23.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running sdist
running egg_info
creating bdgenomics.adam.egg-info
writing requirements to bdgenomics.adam.egg-info/requires.txt
writing bdgenomics.adam.egg-info/PKG-INFO
writing top-level names to bdgenomics.adam.egg-info/top_level.txt
writing dependency_links to bdgenomics.adam.egg-info/dependency_links.txt
writing manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
package init file 'deps/jars/__init__.py' not found (or not a regular file)
package init file 'deps/bin/__init__.py' not found (or not a regular file)
reading manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
running check
creating bdgenomics.adam-0.23.0-SNAPSHOT
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps/jars
copying files to bdgenomics.adam-0.23.0-SNAPSHOT...
copying MANIFEST.in -> bdgenomics.adam-0.23.0-SNAPSHOT
copying README.md -> bdgenomics.adam-0.23.0-SNAPSHOT
copying setup.py -> bdgenomics.adam-0.23.0-SNAPSHOT
copying version.py -> bdgenomics.adam-0.23.0-SNAPSHOT
copying bdgenomics/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics
copying bdgenomics.adam.egg-info/PKG-INFO -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/SOURCES.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/dependency_links.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/requires.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/top_level.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics/adam/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/adamContext.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/find_adam_home.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/rdd.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/stringency.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/test/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/adamContext_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/alignmentRecordRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/featureRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/genotypeRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/variantRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying deps/bin/adam-shell -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/adam-submit -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/adamR -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-assembly.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-egg.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-home -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-spark.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/pyadam -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/jars/adam.jar -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/jars
Writing bdgenomics.adam-0.23.0-SNAPSHOT/setup.cfg
creating dist
Creating tar archive
removing 'bdgenomics.adam-0.23.0-SNAPSHOT' (and everything under it)

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload dist/*.tar.gz
-bash: twine: command not found

Then I suppose I should update the version in version.py to an rc before attempting to deploy to test pypi again.

Member

heuermh commented Jan 3, 2018

Are twine and pypandoc supposed to be installed as part of the virtualenv?

...
adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ virtualenv release-venv
Using base prefix '/Users/mheuer2/miniconda3'
New python executable in /Users/mheuer2/working/adam/adam-python/release-venv/bin/python
Installing setuptools, pip, wheel...done.

adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ . release-venv/bin/activate

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ pip install pyspark
Collecting pyspark
  Downloading pyspark-2.2.0.post0.tar.gz (188.3MB)
    100% |████████████████████████████████| 188.3MB 9.3kB/s
Collecting py4j==0.10.4 (from pyspark)
  Downloading py4j-0.10.4-py2.py3-none-any.whl (186kB)
    100% |████████████████████████████████| 194kB 4.4MB/s
Building wheels for collected packages: pyspark
  Running setup.py bdist_wheel for pyspark ... done
  Stored in directory: /Users/mheuer2/Library/Caches/pip/wheels/5f/0b/b3/5cb16b15d28dcc32f8e7ec91a044829642874bb7586f6e6cbe
Successfully built pyspark
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.4 pyspark-2.2.0

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ rm -rf dist

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ make sdist
python2.7 setup.py sdist
Could not import pypandoc - required to package bdgenomics.adam
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:360: UserWarning: The version specified ('0.23.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running sdist
running egg_info
creating bdgenomics.adam.egg-info
writing requirements to bdgenomics.adam.egg-info/requires.txt
writing bdgenomics.adam.egg-info/PKG-INFO
writing top-level names to bdgenomics.adam.egg-info/top_level.txt
writing dependency_links to bdgenomics.adam.egg-info/dependency_links.txt
writing manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
package init file 'deps/jars/__init__.py' not found (or not a regular file)
package init file 'deps/bin/__init__.py' not found (or not a regular file)
reading manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'bdgenomics.adam.egg-info/SOURCES.txt'
running check
creating bdgenomics.adam-0.23.0-SNAPSHOT
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
creating bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
creating bdgenomics.adam-0.23.0-SNAPSHOT/deps/jars
copying files to bdgenomics.adam-0.23.0-SNAPSHOT...
copying MANIFEST.in -> bdgenomics.adam-0.23.0-SNAPSHOT
copying README.md -> bdgenomics.adam-0.23.0-SNAPSHOT
copying setup.py -> bdgenomics.adam-0.23.0-SNAPSHOT
copying version.py -> bdgenomics.adam-0.23.0-SNAPSHOT
copying bdgenomics/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics
copying bdgenomics.adam.egg-info/PKG-INFO -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/SOURCES.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/dependency_links.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/requires.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics.adam.egg-info/top_level.txt -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics.adam.egg-info
copying bdgenomics/adam/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/adamContext.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/find_adam_home.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/rdd.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/stringency.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam
copying bdgenomics/adam/test/__init__.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/adamContext_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/alignmentRecordRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/featureRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/genotypeRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying bdgenomics/adam/test/variantRdd_test.py -> bdgenomics.adam-0.23.0-SNAPSHOT/bdgenomics/adam/test
copying deps/bin/adam-shell -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/adam-submit -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/adamR -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-assembly.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-egg.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-adam-home -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/find-spark.sh -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/bin/pyadam -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/bin
copying deps/jars/adam.jar -> bdgenomics.adam-0.23.0-SNAPSHOT/deps/jars
Writing bdgenomics.adam-0.23.0-SNAPSHOT/setup.cfg
creating dist
Creating tar archive
removing 'bdgenomics.adam-0.23.0-SNAPSHOT' (and everything under it)

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload dist/*.tar.gz
-bash: twine: command not found

Then I suppose I should update the version in version.py to an rc before attempting to deploy to test pypi again.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

Regarding version.py, should that be committed as "0.23.0" before the release? Or do we want to modify it in place in release.sh?

Member

heuermh commented Jan 3, 2018

Regarding version.py, should that be committed as "0.23.0" before the release? Or do we want to modify it in place in release.sh?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 3, 2018

Member

@heuermh I'd commit version.py as 0.23.0 before the release.

Are twine and pypandoc supposed to be installed as part of the virtualenv?

No, you'll need to pip install twine pypandoc, and yum install pandoc (or whatever is correct for your system) iirc.

Member

fnothaft commented Jan 3, 2018

@heuermh I'd commit version.py as 0.23.0 before the release.

Are twine and pypandoc supposed to be installed as part of the virtualenv?

No, you'll need to pip install twine pypandoc, and yum install pandoc (or whatever is correct for your system) iirc.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

ok, will try again this afternoon, and if it deploys to testpypi correctly, then I'll merge this pr and perform the release.

Member

heuermh commented Jan 3, 2018

ok, will try again this afternoon, and if it deploys to testpypi correctly, then I'll merge this pr and perform the release.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

Back to my workstation, installed Miniconda for Python 2.7, sourced ~/.bash_profile, then

$ virtualenv release-venv
New python executable in /Users/heuermh/working/adam-tmp/adam-python/release-venv/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /Users/heuermh/worki...ease-venv/bin/python - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/heuermh/.miniconda2/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/heuermh/.miniconda2/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: dlopen(/Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
  Referenced from: /Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so
  Expected in: flat namespace
 in /Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/Users/heuermh/.miniconda2/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /Users/heuermh/worki...ease-venv/bin/python - setuptools pip wheel failed with error code 1
Member

heuermh commented Jan 3, 2018

Back to my workstation, installed Miniconda for Python 2.7, sourced ~/.bash_profile, then

$ virtualenv release-venv
New python executable in /Users/heuermh/working/adam-tmp/adam-python/release-venv/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /Users/heuermh/worki...ease-venv/bin/python - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/heuermh/.miniconda2/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/heuermh/.miniconda2/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: dlopen(/Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
  Referenced from: /Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so
  Expected in: flat namespace
 in /Users/heuermh/working/adam-tmp/adam-python/release-venv/lib/python2.7/lib-dynload/_io.so
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/Users/heuermh/.miniconda2/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/Users/heuermh/.miniconda2/lib/python2.7/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /Users/heuermh/worki...ease-venv/bin/python - setuptools pip wheel failed with error code 1
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

Next tried installing pip, virtualenv, twine and pypandoc against system python, pandoc is /usr/local/bin/pandoc. Got farther, then

...
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.4 pyspark-2.2.0

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ rm -rf dist

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ make sdist
python2.7 setup.py sdist
Could not import pypandoc - required to package bdgenomics.adam
Member

heuermh commented Jan 3, 2018

Next tried installing pip, virtualenv, twine and pypandoc against system python, pandoc is /usr/local/bin/pandoc. Got farther, then

...
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.4 pyspark-2.2.0

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ rm -rf dist

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ make sdist
python2.7 setup.py sdist
Could not import pypandoc - required to package bdgenomics.adam
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 3, 2018

Member

Ran pip install pypandoc inside the virtualenv.

Now having trouble with twine, created .pypirc with [testpypi] credentials, then

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload dist/*.tar.gz
KeyError: Missing 'pypi' section from the configuration file
or not a complete URL in --repository.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload -r testpypi dist/*.tar.gz
KeyError: Missing 'testpypi' section from the configuration file
or not a complete URL in --repository.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*.tar.gz
Uploading distributions to https://test.pypi.org/legacy/
Enter your username: ***
Enter your password: 
Uploading bdgenomics.adam-0.23.0rc19.tar.gz
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload --repository-url https://test.pypi.org/ dist/*.tar.gz
Uploading distributions to https://test.pypi.org/
Enter your username: ***
Enter your password: 
Uploading bdgenomics.adam-0.23.0rc19.tar.gz
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))
Member

heuermh commented Jan 3, 2018

Ran pip install pypandoc inside the virtualenv.

Now having trouble with twine, created .pypirc with [testpypi] credentials, then

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload dist/*.tar.gz
KeyError: Missing 'pypi' section from the configuration file
or not a complete URL in --repository.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload -r testpypi dist/*.tar.gz
KeyError: Missing 'testpypi' section from the configuration file
or not a complete URL in --repository.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*.tar.gz
Uploading distributions to https://test.pypi.org/legacy/
Enter your username: ***
Enter your password: 
Uploading bdgenomics.adam-0.23.0rc19.tar.gz
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))

(release-venv) adam-python (HEAD detached at fnothaft/issues/1783-cran)!
$ twine upload --repository-url https://test.pypi.org/ dist/*.tar.gz
Uploading distributions to https://test.pypi.org/
Enter your username: ***
Enter your password: 
Uploading bdgenomics.adam-0.23.0rc19.tar.gz
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 3, 2018

Member

Run this:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Member

fnothaft commented Jan 3, 2018

Run this:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

@heuermh heuermh merged commit 2986cf7 into bigdatagenomics:master Jan 4, 2018

1 of 2 checks passed

Codacy/PR Quality Review Not so good... This pull request quality could be better.
Details
default Merged build finished.
Details
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 4, 2018

Member

DO
IT
LIVE

Member

heuermh commented Jan 4, 2018

DO
IT
LIVE

@fnothaft fnothaft deleted the fnothaft:issues/1783-cran branch Jan 4, 2018

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 4, 2018

Member

w00t!

Member

fnothaft commented Jan 4, 2018

w00t!

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment