Fix writing variants to GCS buckets #3485
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #3485 +/- ##
=============================================
+ Coverage 79.94% 79.94% +<.001%
Complexity 17897 17897
=============================================
Files 1198 1199 +1
Lines 64980 64986 +6
Branches 10120 10120
=============================================
+ Hits 51945 51950 +5
+ Misses 9002 9001 -1
- Partials 4033 4035 +2
|
lbergelson
reviewed
Aug 23, 2017
| @@ -118,7 +118,7 @@ private static void writeVariantsSingle( | |||
| } | |||
|
|
|||
| final JavaRDD<VariantContext> sortedVariants = sortVariants(variants, header, numReducers); | |||
| final String outputPartsDirectory = outputFile + ".parts"; | |||
There was a problem hiding this comment.
Would you mind adding a test that writes variants to gcs so we don't break this again? There's a PrintReadsSparkIntegrationTest that does it for bams, a ReadSparkSinkUnit test would be ideal.
lbergelson
previously requested changes
Aug 23, 2017
There was a problem hiding this comment.
@tomwhite It would be good to add a test so we never the break the vcf writing again without knowing. JP added the gcs connector to our test dependencies so we can use it to write to gcs via spark during local tests as long permissions are configured correctly. (which they are on travis)
tomwhite
force-pushed
the
tw_write_variants_gcs
branch
2 times, most recently
from
f900d54
to
4217b1b
Compare
lbergelson
reviewed
Aug 30, 2017
| * For this to work, the settings in src/main/resources/core-site.xml must be correct, | ||
| * and the project name and credential file it points to must be present. | ||
| */ | ||
| @Test(dataProvider = "gcsTestingData", groups = "bucket") |
tomwhite
force-pushed
the
tw_write_variants_gcs
branch
from
4217b1b
to
f375452
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Writing reads was fixed in 73f2a62, but unfortunately the same problem occurs with variants.
This commit (a861a23) fixes the problem for variants, when deployed with a Hadoop-BAM fix (HadoopGenomics/Hadoop-BAM#143).