New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1576] Allow translation between two different GenomicRDD types. #1598

Merged
merged 1 commit into from Jul 17, 2017

Conversation

Projects
5 participants
@fnothaft
Member

fnothaft commented Jul 11, 2017

Resolves #1576. Adds transmute function, which is analogous to transform, but which allows any return type. This uses an implicit function a la pipe to build the output GenomicRDD.

Can I get a quick review pass on this? If this general direction looks good, I will flesh out the implicits for all the allowable transformations.

  • Extend to all primary types
  • Add docs

@fnothaft fnothaft added this to the 0.23.0 milestone Jul 11, 2017

@fnothaft fnothaft requested review from heuermh and devin-petersohn Jul 11, 2017

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

coveralls commented Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

1 similar comment
@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

coveralls commented Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2203/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 6b5f30589ede34f153dbac174c5b234aff21bb64 # timeout=10Checking out Revision 6b5f30589ede34f153dbac174c5b234aff21bb64 (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6b5f30589ede34f153dbac174c5b234aff21bb64First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jul 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2203/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 6b5f30589ede34f153dbac174c5b234aff21bb64 # timeout=10Checking out Revision 6b5f30589ede34f153dbac174c5b234aff21bb64 (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6b5f30589ede34f153dbac174c5b234aff21bb64First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@devin-petersohn

Nice! Looks great.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 11, 2017

Member

OK great! I'll flesh this out today. Don't merge as is!

Member

fnothaft commented Jul 11, 2017

OK great! I'll flesh this out today. Don't merge as is!

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 11, 2017

Coverage Status

Coverage increased (+0.2%) to 84.29% when pulling f27ee3d on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

coveralls commented Jul 11, 2017

Coverage Status

Coverage increased (+0.2%) to 84.29% when pulling f27ee3d on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2206/
Test PASSed.

AmplabJenkins commented Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2206/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 12, 2017

Member

This is good to go from my end.

Member

fnothaft commented Jul 12, 2017

This is good to go from my end.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 12, 2017

Coverage Status

Coverage increased (+0.5%) to 84.525% when pulling e25d5a6 on fnothaft:issues/1576-generic-rdd-to-specific into 238e044 on bigdatagenomics:master.

coveralls commented Jul 12, 2017

Coverage Status

Coverage increased (+0.5%) to 84.525% when pulling e25d5a6 on fnothaft:issues/1576-generic-rdd-to-specific into 238e044 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2207/
Test PASSed.

AmplabJenkins commented Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2207/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 13, 2017

Member

Rebased and cleared conflicts. This is good to go from my side.

Member

fnothaft commented Jul 13, 2017

Rebased and cleared conflicts. This is good to go from my side.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 13, 2017

Coverage Status

Coverage increased (+0.4%) to 84.533% when pulling 30a6119 on fnothaft:issues/1576-generic-rdd-to-specific into 607cd50 on bigdatagenomics:master.

coveralls commented Jul 13, 2017

Coverage Status

Coverage increased (+0.4%) to 84.533% when pulling 30a6119 on fnothaft:issues/1576-generic-rdd-to-specific into 607cd50 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 13, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2215/
Test PASSed.

AmplabJenkins commented Jul 13, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2215/
Test PASSed.

@heuermh

This is pretty cool and fairly powerful. It also may lead to confusion, as it allows users to create GenomicRDDs with records that break preconditions and validation checks.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 17, 2017

Member

Rebased and cleared conflicts.

Member

fnothaft commented Jul 17, 2017

Rebased and cleared conflicts.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 17, 2017

Member

Also, WRT @heuermh's comment, there's nothing that stops an enterprising user from breaking our preconditions today. This might make it easier, but not really any easier than just using transform today is.

Member

fnothaft commented Jul 17, 2017

Also, WRT @heuermh's comment, there's nothing that stops an enterprising user from breaking our preconditions today. This might make it easier, but not really any easier than just using transform today is.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

coveralls commented Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2227/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c24fe77ef226e0041a2f34d4212526c893402e9c # timeout=10Checking out Revision c24fe77ef226e0041a2f34d4212526c893402e9c (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c24fe77ef226e0041a2f34d4212526c893402e9cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jul 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2227/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c24fe77ef226e0041a2f34d4212526c893402e9c # timeout=10Checking out Revision c24fe77ef226e0041a2f34d4212526c893402e9c (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c24fe77ef226e0041a2f34d4212526c893402e9cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 17, 2017

Member

Looks like a transient s3 failure.

Member

fnothaft commented Jul 17, 2017

Looks like a transient s3 failure.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 17, 2017

Member

Jenkins, test this please.

Member

fnothaft commented Jul 17, 2017

Jenkins, test this please.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

coveralls commented Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 17, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2230/
Test PASSed.

AmplabJenkins commented Jul 17, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2230/
Test PASSed.

@heuermh heuermh merged commit 961f90a into bigdatagenomics:master Jul 17, 2017

2 of 3 checks passed

codacy/pr Not so good... This pull request quality could be better.
Details
coverage/coveralls Coverage increased (+0.4%) to 84.58%
Details
default Merged build finished.
Details
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 17, 2017

Member

Thank you, @fnothaft!

Member

heuermh commented Jul 17, 2017

Thank you, @fnothaft!

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment