Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use ParsedLoci in loadIndexedBam #1277

Closed
wants to merge 1 commit into from

Conversation

@ryan-williams
Copy link
Member

ryan-williams commented Nov 17, 2016

I recently factored a small library for parsing string-representations of (possibly open-ended, e.g. chr1, chr1:100-) loci ranges out of guacamole and over to https://github.com/hammerlab/genomic-loci, because I was preparing to use it across a few of my projects, and in particular wanted it plus adam-core's loadIndexedBam functionality, which this PR adds.

@fnothaft you mentioned there is some similar loci-parsing code laying around somewhere in BDG, lmk if I should find/consider it instead / in addition to this.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 17, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1620/

Build result: FAILURE

[...truncated 38 lines...]Triggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@ryan-williams ryan-williams mentioned this pull request Nov 17, 2016
@ryan-williams
Copy link
Member Author

ryan-williams commented Nov 17, 2016

Jenkins, retest this please.

SPARK_VERSION=2.0.0 SCALAVER=2.11 HADOOP_VERSION=2.6.0 ./scripts/jenkins-test

passes for me locally. In particular, the error in the logs:

[ERROR] Failed to execute goal on project adam-core-spark2_2.11: Could not resolve dependencies for project org.bdgenomics.adam:adam-core-spark2_2.11:jar:0.20.1-SNAPSHOT: The following artifacts could not be resolved: org.spire-math:spire-macros_2.11:jar:0.11.0, org.typelevel:machinist_2.11:jar:0.4.1: Could not find artifact org.spire-math:spire-macros_2.11:jar:0.11.0 -> [Help 1]

seems erroneous, as org.spire-math:spire-macros_2.11:jar:0.11.0 exists on Maven Central.

@heuermh
Copy link
Member

heuermh commented Nov 17, 2016

Open ended range support in was added in #1252.

@ryan-williams
Copy link
Member Author

ryan-williams commented Nov 17, 2016

Cool, gtk; the ParsedLoci class I'm using here comes with some facilities for parsing to/from strings/files that I think are still useful for what I am after here, fwiw.

@heuermh
Copy link
Member

heuermh commented Nov 21, 2016

While I am supportive of the functionality proposed here, I would have to -1 it as is with the new dependency. For better or worse, we need to cross-build for Spark 1.x/2.x and Scala 2.10/2.11 and so need upstream dependencies to do the same.

Would there be much of a performance hit to transform
ParsedLociIterable[ReferenceRegion]Iterable[LocatableReferenceRegion]

as compared to
ParsedLociLociSetlociSet.toHtsJDKIntervals?

@ryan-williams
Copy link
Member Author

ryan-williams commented Nov 21, 2016

The genomic-loci library is published for Scala 2.1[01] and I'm not sure uses anything that would differ between Spark 1 and 2 but I'll dual-publish on that axis as well if I find that that's the case.

As it currently stands it needs some guava-shading that I haven't worked out yet to run happily inside a Spark app so this PR is not ready to merge, closing for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.