New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding loadPairedFastqAsFragments method #1828

Closed
wants to merge 7 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@ffinfo
Copy link
Contributor

ffinfo commented Dec 12, 2017

During some talk o gitter I missing this method and looks simple enough to add it myself ;).

Some open issue here are how to do validation if the 2 files are indeed in sync. I can use stringency: ValidationStringency like in loadPairedFastq. Will follow up this on gitter also.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Dec 12, 2017

Can one of the admins verify this patch?

ffinfo added some commits Dec 12, 2017

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Dec 12, 2017

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Dec 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2511/
Test PASSed.

ffinfo added some commits Dec 13, 2017

@ffinfo

This comment has been minimized.

Copy link
Contributor

ffinfo commented Dec 13, 2017

This should be done for now, could someone look at this?

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Dec 13, 2017

I'm +1 on adding the new method, but would like some time to benchmark the implementation here against a few other possible ones. Will take a look after cutting the 0.23.0 release.

val rdd = recordsR1.zip(recordsR2)
.map {
case (r1, r2) =>
val pairText = new Text(r1._2.toString + r2._2.toString)

This comment has been minimized.

@heuermh

heuermh Jan 9, 2018

Member

Note zipping the records together assumes that they are sorted in the same order in each file.

This comment has been minimized.

@ffinfo

ffinfo Feb 8, 2018

Contributor

Yes indeed, this should always the case when a pair of fastq files. When this is not the case the fastq files are corrupt, at least as paired reads.
Adding a check add more computation, seems like a waste

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Jul 4, 2018

Thank you for the contribution, @ffinfo! Closing this in favor of #1866, which as you point out makes fewer assumptions at some performance cost.

@heuermh heuermh closed this Jul 4, 2018

@heuermh heuermh added this to the 0.24.1 milestone Jul 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment