Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding loadPairedFastqAsFragments method #1828

Closed

Conversation

@ffinfo
Copy link
Contributor

@ffinfo ffinfo commented Dec 12, 2017

During some talk o gitter I missing this method and looks simple enough to add it myself ;).

Some open issue here are how to do validation if the 2 files are indeed in sync. I can use stringency: ValidationStringency like in loadPairedFastq. Will follow up this on gitter also.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Dec 12, 2017

Can one of the admins verify this patch?

ffinfo added 2 commits Dec 12, 2017
@heuermh
Copy link
Member

@heuermh heuermh commented Dec 12, 2017

Jenkins, test this please.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Dec 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2511/
Test PASSed.

ffinfo added 4 commits Dec 13, 2017
@ffinfo
Copy link
Contributor Author

@ffinfo ffinfo commented Dec 13, 2017

This should be done for now, could someone look at this?

@heuermh
Copy link
Member

@heuermh heuermh commented Dec 13, 2017

I'm +1 on adding the new method, but would like some time to benchmark the implementation here against a few other possible ones. Will take a look after cutting the 0.23.0 release.

val rdd = recordsR1.zip(recordsR2)
.map {
case (r1, r2) =>
val pairText = new Text(r1._2.toString + r2._2.toString)

This comment has been minimized.

@heuermh

heuermh Jan 9, 2018
Member

Note zipping the records together assumes that they are sorted in the same order in each file.

This comment has been minimized.

@ffinfo

ffinfo Feb 8, 2018
Author Contributor

Yes indeed, this should always the case when a pair of fastq files. When this is not the case the fastq files are corrupt, at least as paired reads.
Adding a check add more computation, seems like a waste

@heuermh
Copy link
Member

@heuermh heuermh commented Jul 4, 2018

Thank you for the contribution, @ffinfo! Closing this in favor of #1866, which as you point out makes fewer assumptions at some performance cost.

@heuermh heuermh closed this Jul 4, 2018
@heuermh heuermh added this to the 0.24.1 milestone Jul 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants