New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: speed up 2bit file extract #1426

Merged
merged 1 commit into from Mar 14, 2017

Conversation

Projects
None yet
5 participants
@Blaok
Contributor

Blaok commented Mar 7, 2017

Rewrite the extract logic.
On a single node with 20 workers, BaseRecalibration time is reduced from 5.8 min to 1.7 min.
Pass existing tests.

feat: speed up 2bit file extract
On a single node with 20 workers, BaseRecalibration time is reduced from 5.8 min to 1.7 min.
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 7, 2017

Member

Wonderful! Thank you for submitting the patch; I will take a look at this later today or tomorrow.

Member

fnothaft commented Mar 7, 2017

Wonderful! Thank you for submitting the patch; I will take a look at this later today or tomorrow.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 7, 2017

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 7, 2017

Member

Jenkins, test this please.

Member

fnothaft commented Mar 7, 2017

Jenkins, test this please.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Mar 7, 2017

Coverage Status

Coverage decreased (-0.003%) to 76.396% when pulling 1a53ce5 on Blaok:master into 07c1982 on bigdatagenomics:master.

coveralls commented Mar 7, 2017

Coverage Status

Coverage decreased (-0.003%) to 76.396% when pulling 1a53ce5 on Blaok:master into 07c1982 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 7, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1844/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1844/
Test PASSed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 8, 2017

Member

Is there an ε we can provide to coveralls to prevent it from failing a build in silly cases like this one? Coverage decreased by 0.003%? Come on.

Member

heuermh commented Mar 8, 2017

Is there an ε we can provide to coveralls to prevent it from failing a build in silly cases like this one? Coverage decreased by 0.003%? Come on.

@fnothaft

LGTM! Thanks for the change!

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 9, 2017

Member

@laserson would you be able to review this today or tomorrow, since you wrote the original 2bit code? If not, I think this is good to go and I will merge as is.

Member

fnothaft commented Mar 9, 2017

@laserson would you be able to review this today or tomorrow, since you wrote the original 2bit code? If not, I think this is good to go and I will merge as is.

@heuermh

heuermh approved these changes Mar 9, 2017

@fnothaft fnothaft merged commit 1eed8e8 into bigdatagenomics:master Mar 14, 2017

1 of 2 checks passed

coverage/coveralls Coverage decreased (-0.003%) to 76.396%
Details
default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 14, 2017

Member

Merged! Thanks @Blaok for the contribution! We greatly appreciate it.

Member

fnothaft commented Mar 14, 2017

Merged! Thanks @Blaok for the contribution! We greatly appreciate it.

@Blaok Blaok referenced this pull request Mar 20, 2017

Closed

investigate performance of ReadsPipelineSpark #1657

0 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment