Add reference broadcasting to BQSR #928

laserson · 2015-09-25T22:37:36Z

please review any of @droazen @tomwhite @davidadamsphd

laserson · 2015-09-25T22:39:15Z

(Also note this adds .2bit chr20/21 reference to LFS which is about 27 MB)

droazen · 2015-09-25T23:00:44Z

Will do a review pass first thing Monday -- @tomwhite and/or @davidadamsphd should also have a look.

tomwhite · 2015-09-27T23:33:05Z

src/main/java/org/broadinstitute/hellbender/engine/spark/BroadcastJoinReadsWithRefBases.java

+public class BroadcastJoinReadsWithRefBases {
+    public static JavaPairRDD<GATKRead, ReferenceBases> addBases(final ReferenceDataflowSource referenceDataflowSource,
+                                                                 final JavaRDD<GATKRead> reads) {
+        JavaSparkContext ctx = new JavaSparkContext(reads.context());


It's odd that reads.context() doesn't return a JavaSparkContext. This is fine though since it's reusing the underlying context.

I agree. Strange API choice.

tomwhite · 2015-09-27T23:48:54Z

I added a few fairly minor comments. Curious to see the performance implications.

davidadamsphd · 2015-09-28T13:20:31Z

src/main/java/org/broadinstitute/hellbender/engine/spark/AddContextDataToReadSpark.java

-
-    }
-
-    private static void assertSameReads(final JavaRDD<GATKRead> reads,


We'd like to keep this function around to verify we're not losing reads.

It's no longer being used. If anything, this should be moved into a test somewhere.

Moving to test sounds good to me.

davidadamsphd · 2015-09-28T16:48:21Z

@laserson, any chance you can add support for fasta files in a follow up? (I remember @tomwhite did some work on this, but I don't recall the current state).

droazen · 2015-09-28T16:50:57Z

.../java/org/broadinstitute/hellbender/engine/dataflow/datasources/ReferenceDataflowSource.java

@@ -35,7 +34,13 @@
    public ReferenceDataflowSource(final PipelineOptions pipelineOptions, final String referenceURL,
                                   final SerializableFunction<GATKRead, SimpleInterval> referenceWindowFunction) {
        Utils.nonNull(referenceWindowFunction);
-        if (isFasta(referenceURL)) {
+        if (referenceURL.endsWith(".2bit")) {


Declare this extension as a named constant at the top of the class.

See note above about similar issue.

Answered above

droazen · 2015-09-28T17:58:00Z

src/test/java/org/broadinstitute/hellbender/engine/spark/AddContextDataToReadSparkUnitTest.java

@@ -14,6 +14,7 @@
 import org.broadinstitute.hellbender.engine.dataflow.datasources.ReferenceDataflowSource;
 import org.broadinstitute.hellbender.utils.SimpleInterval;
 import org.broadinstitute.hellbender.utils.read.GATKRead;
+import org.broadinstitute.hellbender.utils.spark.JoinStrategy;
 import org.broadinstitute.hellbender.utils.test.BaseTest;
 import org.broadinstitute.hellbender.utils.test.FakeReferenceSource;
 import org.broadinstitute.hellbender.utils.variant.Variant;


Add a new unit test to AddContextDataToReadSparkUnitTest that tests the BROADCAST case.

Made the DataProvider simply add a case for BROADCAST strategy as well. It will probably be inefficient, but for small data sets, all the ReferenceSources should work in a broadcast strategy as well I think.

droazen · 2015-09-28T18:01:48Z

Have a couple of test failures still -- do we understand what's causing them?

JoinReadsWithVariantsSparkUnitTest. pairReadsAndVariantsTest[0]([org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@dc61e602, org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@784984f1, org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@4985b350, org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@497fa2e4, org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@4985b429], [MinimalVariant -- interval(1:170-180), snp(true), indel(false), MinimalVariant -- interval(1:210-220), snp(false), indel(true), MinimalVariant -- interval(1:10000-10000), snp(true), indel(false), MinimalVariant -- interval(1:29998-30002), snp(false), indel(true), MinimalVariant -- interval(2:10000-10000), snp(false), indel(true)], [KV(org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@784984f1, [MinimalVariant -- interval(1:210-220), snp(false), indel(true), MinimalVariant -- interval(1:170-180), snp(true), indel(false)]), KV(org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@4985b350, [MinimalVariant -- interval(1:10000-10000), snp(true), indel(false)]), KV(org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@497fa2e4, [MinimalVariant -- interval(1:29998-30002), snp(false), indel(true)]), KV(org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@4985b429, [MinimalVariant -- interval(2:10000-10000), snp(false), indel(true)]), KV(org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter@dc61e602, [null])])
JoinReadsWithVariantsSparkUnitTest. pairReadsAndVariantsTest[1]([1 50b aligned read., 2 100b aligned read., 3 10b aligned read., 4 10b aligned read., 5 10b aligned read.], [MinimalVariant -- interval(1:170-180), snp(true), indel(false), MinimalVariant -- interval(1:210-220), snp(false), indel(true), MinimalVariant -- interval(1:10000-10000), snp(true), indel(false), MinimalVariant -- interval(1:29998-30002), snp(false), indel(true), MinimalVariant -- interval(2:10000-10000), snp(false), indel(true)], [KV(2 100b aligned read., [MinimalVariant -- interval(1:210-220), snp(false), indel(true), MinimalVariant -- interval(1:170-180), snp(true), indel(false)]), KV(3 10b aligned read., [MinimalVariant -- interval(1:10000-10000), snp(true), indel(false)]), KV(4 10b aligned read., [MinimalVariant -- interval(1:29998-30002), snp(false), indel(true)]), KV(5 10b aligned read., [MinimalVariant -- interval(2:10000-10000), snp(false), indel(true)]), KV(1 50b aligned read., [null])])
BaseRecalibratorSparkFnUnitTest. testApply[0](/home/travis/build/broadinstitute/hellbender/src/test/resources/human_g1k_v37.chr17_1Mb.fasta, src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.bam, src/test/resources/org/broadinstitute/hellbender/tools/BQSR/dbsnp_132.b37.excluding_sites_after_129.chr17_69k_70k.vcf, src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.NA12878.chr17_69k_70k.txt)

droazen · 2015-09-28T18:03:15Z

Review complete -- back to @laserson

laserson · 2015-10-06T07:55:27Z

Ok, @droazen, I think just a few comments for you to address, and we're almost finished.

droazen · 2015-10-06T16:46:36Z

@laserson Ok, responded to the latest round of comments. Ping me when this is rebased/finalized and passing tests, and I'll take a last look and merge.

laserson · 2015-10-06T16:47:56Z

Thanks! One issue is that the tests here will fail until the newest ADAM is released later this week :-/ I don't mind waiting if you don't. Otherwise, I could disable the tests for now.

droazen · 2015-10-06T16:52:18Z

I vote to disable the test(s) in question in the interests of getting this merged (@akiezun feel free to chime in if you disagree) -- let's open a ticket to remind us to turn them back on when we rev ADAM, though.

akiezun · 2015-10-06T17:02:32Z

fine, disable and enter a ticket to reenable.

Adds a new file to LFS: human_g1k_v37.20.21.2bit. It was by applying the faToTwoBit binary from the "kent" UCSC utilities package on the similarly named fasta reference already in the repo.

laserson · 2015-10-07T06:40:56Z

I anticipate this should pass, and I believe all issues have been addressed.

droazen · 2015-10-07T15:27:08Z

👍 merging!

Add reference broadcasting to BQSR

laserson · 2015-10-07T17:13:13Z

hallelujah. sorry that took so long

droazen self-assigned this Sep 25, 2015

tomwhite reviewed Sep 27, 2015
View reviewed changes

davidadamsphd reviewed Sep 28, 2015
View reviewed changes

droazen reviewed Sep 28, 2015
View reviewed changes

droazen assigned laserson and unassigned droazen Sep 28, 2015

akiezun modified the milestone: alpha Oct 3, 2015

laserson force-pushed the tw_ul_spark_bqsr branch from 3b39dfa to 9e1a222 Compare October 5, 2015 15:46

laserson force-pushed the tw_ul_spark_bqsr branch from 9e1a222 to bf363ad Compare October 6, 2015 07:58

akiezun added the Spark label Oct 7, 2015

laserson force-pushed the tw_ul_spark_bqsr branch from bf363ad to 9713727 Compare October 7, 2015 05:43

Add reference broadcasting to BQSR

ac321f4

Adds a new file to LFS: human_g1k_v37.20.21.2bit. It was by applying the faToTwoBit binary from the "kent" UCSC utilities package on the similarly named fasta reference already in the repo.

laserson force-pushed the tw_ul_spark_bqsr branch from 9713727 to ac321f4 Compare October 7, 2015 06:38

laserson mentioned this pull request Oct 7, 2015

Reenable ReferenceTwoBitSourceUnitTest after ADAM 0.18 is released #957

Closed

droazen added a commit that referenced this pull request Oct 7, 2015

Merge pull request #928 from broadinstitute/tw_ul_spark_bqsr

a4cadc3

Add reference broadcasting to BQSR

droazen merged commit a4cadc3 into master Oct 7, 2015

droazen deleted the tw_ul_spark_bqsr branch October 7, 2015 15:27

droazen mentioned this pull request Oct 7, 2015

Support reference broadcasting for fasta files #956

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add reference broadcasting to BQSR #928

Add reference broadcasting to BQSR #928

laserson commented Sep 25, 2015

laserson commented Sep 25, 2015

droazen commented Sep 25, 2015

tomwhite Sep 27, 2015

laserson Sep 29, 2015

tomwhite commented Sep 27, 2015

davidadamsphd Sep 28, 2015

laserson Oct 5, 2015

davidadamsphd Oct 5, 2015

davidadamsphd commented Sep 28, 2015

droazen Sep 28, 2015

laserson Oct 5, 2015

droazen Oct 6, 2015

laserson Oct 7, 2015

droazen Sep 28, 2015

laserson Oct 5, 2015

droazen commented Sep 28, 2015

droazen commented Sep 28, 2015

laserson commented Oct 6, 2015

droazen commented Oct 6, 2015

laserson commented Oct 6, 2015

droazen commented Oct 6, 2015

akiezun commented Oct 6, 2015

laserson commented Oct 7, 2015

droazen commented Oct 7, 2015

laserson commented Oct 7, 2015


		}

		private static void assertSameReads(final JavaRDD<GATKRead> reads,

Add reference broadcasting to BQSR #928

Add reference broadcasting to BQSR #928

Conversation

laserson commented Sep 25, 2015

laserson commented Sep 25, 2015

droazen commented Sep 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomwhite commented Sep 27, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidadamsphd commented Sep 28, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droazen commented Sep 28, 2015

droazen commented Sep 28, 2015

laserson commented Oct 6, 2015

droazen commented Oct 6, 2015

laserson commented Oct 6, 2015

droazen commented Oct 6, 2015

akiezun commented Oct 6, 2015

laserson commented Oct 7, 2015

droazen commented Oct 7, 2015

laserson commented Oct 7, 2015