New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1793] Adding vcf2adam and adam2vcf that handle separate variant and genotype data. #1794

Closed
wants to merge 1 commit into
base: master
from

Conversation

3 participants
@heuermh
Member

heuermh commented Nov 7, 2017

Fixes #1793

Rough first cut, needs review.

@heuermh heuermh requested a review from fnothaft Nov 7, 2017

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Nov 7, 2017

Merged build finished. Test FAILed.

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Nov 7, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2466/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1794/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains a27e59b # timeout=10Checking out Revision a27e59b (origin/pr/1794/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a27e59b59f2aa0fdea51c4927ad63e11b541e8f0First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

One small change, otherwise LGTM.

args.inputPath,
stringency = stringency)
// todo: cache variantContexts? sort variantContexts first?

This comment has been minimized.

@fnothaft

fnothaft Nov 7, 2017

Member

I would present this as an option. It's fine to convert twice if you're working with a small dataset, but you'll want to persist if you're working with a large data.

stringency = stringency)
val join = variants.shuffleRegionJoin(genotypes)
val updatedGenotypes = join.rdd.map(pair => {

This comment has been minimized.

@fnothaft

fnothaft Nov 7, 2017

Member

Reading through this a second time, what I'd do here is not a shuffleRegionJoin (think of the multiallelic sites) but instead a join on contig/start/end/ref/alt. You might be able to do this through the dataset API for better (?) performance.

This comment has been minimized.

@heuermh

heuermh Nov 7, 2017

Member

Agreed, will push this post-0.23.0 as to not have to conditionally build against the Spark 1.x vs 2.x Dataset API.

@heuermh heuermh added this to the 0.24.0 milestone Nov 7, 2017

@heuermh heuermh added this to Triage in Release 0.24.0 Jan 4, 2018

@heuermh

This comment has been minimized.

Member

heuermh commented Jan 9, 2018

Move to close this in favor of #1865.

@heuermh

This comment has been minimized.

Member

heuermh commented Jan 11, 2018

Closing in favor of #1865.

@heuermh heuermh closed this Jan 11, 2018

@heuermh heuermh moved this from Triage to Completed in Release 0.24.0 Feb 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment