New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to create a single RDD[Genotype] object out of multiple VCF files? #2025

Closed
rezacsedu opened this Issue Aug 1, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@rezacsedu
Copy link

rezacsedu commented Aug 1, 2018

In Scala, I can read a single VCF file as follows:

val allGenotypes: RDD[Genotype] = sc.loadGenotypes(genotypeFile).rdd

However, I'm wondering if I can read multiple VCF files to create a single RDD[Genotype] object?

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Aug 1, 2018

Hello @rezacsedu!

You may either use globbed paths on load

val genotypes = sc.loadGenotypes("hdfs://data/sample*.vcf")
val rdd: RDD[Genotype] = genotypes.rdd

or union multiple GenotypeRDDs together

val genotypes0 = sc.loadGenotypes("sample0.vcf")
val genotypes1 = sc.loadGenotypes("sample1.vcf")
val union = genotypes0.union(genotypes1)
val rdd: RDD[Genotype] = union.rdd
@rezacsedu

This comment has been minimized.

Copy link

rezacsedu commented Aug 1, 2018

@heuermh thanks so much for prompt reply. Really appreciated!

@rezacsedu rezacsedu closed this Aug 1, 2018

@heuermh heuermh added this to the 0.24.1 milestone Aug 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment