Definine GenomicsDB "partitions" over the span of the input intervals in #5540
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #5540 +/- ##
===============================================
+ Coverage 87.068% 87.075% +0.007%
- Complexity 31324 31332 +8
===============================================
Files 1921 1921
Lines 144579 144598 +19
Branches 15949 15952 +3
===============================================
+ Hits 125882 125908 +26
+ Misses 12905 12896 -9
- Partials 5792 5794 +2
|
order to dramatically improve exome performance
ldgauthier
force-pushed
the
ldg_GDBintervalMagic
branch
from
db1038b
to
cb54aad
Compare
|
@lbergelson you were totally right. I pulled out a flag to turn on interval merging. |
lbergelson
requested changes
Dec 20, 2018
There was a problem hiding this comment.
@ldgauthier Looks good, needs a test for the new method. Thank you for checking into the behavior. We could potentially drive this down into the interval argument collection itself as a new merging mode, but I'm not sure if that's useful in general or a liability when other people incorrectly specify it.
| shortName = MERGE_INPUT_INTERVALS_LONG_NAME, | ||
| doc = "Boolean flag to import all data in between intervals. Improves performance using large lists of " + | ||
| "intervals, as in exome sequencing, especially if GVCF data only exists for specified intervals.") | ||
| private boolean mergeInputIntervals = false; |
There was a problem hiding this comment.
This looks good... I guess the other alternative would be to add a new interval merging rule to the interval argument collection and push this behavior into the engine.
| final ArgumentsBuilder args = new ArgumentsBuilder(); | ||
| args.addArgument(GenomicsDBImport.WORKSPACE_ARG_LONG_NAME, workspace); | ||
| intervals.forEach(interval -> args.addArgument("L", IntervalUtils.locatableToString(interval))); | ||
| vcfInputs.forEach(vcf -> args.addArgument("V", vcf)); | ||
| args.addArgument("batch-size", String.valueOf(batchSize)); | ||
| args.addArgument(GenomicsDBImport.VCF_INITIALIZER_THREADS_LONG_NAME, String.valueOf(threads)); | ||
| if (useBufferSize) | ||
| if (mergeIntervals) { |
There was a problem hiding this comment.
you could use args.addBooleanArgument(GenomicsDBImport.MERGE_INPUT_INTERVALS_LONG_NAME, mergeIntervals) and drop the if
| @@ -151,14 +151,16 @@ public void testGenomicsDBImportFileInputsWithMultipleIntervals() throws IOExcep | |||
| testGenomicsDBImporter(LOCAL_GVCFS, MULTIPLE_INTERVALS, COMBINED_MULTI_INTERVAL, b38_reference_20_21, true, 1); | |||
| } | |||
|
|
|||
| private void testGenomicsDBImportWith1000Intervals() throws IOException { | |||
| @Test | |||
| public void testGenomicsDBImportWith1000Intervals() throws IOException { | |||
There was a problem hiding this comment.
I'd rename this to say something about the interval merging.
src/main/java/org/broadinstitute/hellbender/tools/genomicsdb/GenomicsDBImport.java
Show resolved
Hide resolved
ldgauthier
force-pushed
the
ldg_GDBintervalMagic
branch
from
d5d4564
to
e745f5e
Compare
|
Comments addressed -- back to you @lbergelson |
droazen
pushed a commit
that referenced
this pull request
Mar 28, 2019
… exome input (#5741) Rather than start off the exome genotyping task with a step to merge the query intervals for better GDB performance, make it an option as for Import in #5540 This is necessary to get the exome joint calling pipeline onto official GATK release dockers. GenotypeGVCFs also performs better reading from a GenomicsDB as a single interval
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
order to dramatically improve exome performance
This makes an 81X speedup in the 1000interval test I put back in. Exomes are just about unrunnable without some sort of interval manipulation. The 65 sample exome joint calling callset output was exactly the same with this version. I also did a comparison for a chr1 and chr20 import (so that we were looking at two intervals that were far apart) and the runtime was the same. This is a huge improvement for some typical use cases.