Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1576] Allow translation between two different GenomicRDD types. #1598

Merged

Conversation

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Jul 11, 2017

Resolves #1576. Adds transmute function, which is analogous to transform, but which allows any return type. This uses an implicit function a la pipe to build the output GenomicRDD.

Can I get a quick review pass on this? If this general direction looks good, I will flesh out the implicits for all the allowable transformations.

  • Extend to all primary types
  • Add docs
@fnothaft fnothaft added this to the 0.23.0 milestone Jul 11, 2017
@fnothaft fnothaft requested review from heuermh and devin-petersohn Jul 11, 2017
@coveralls
Copy link

@coveralls coveralls commented Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

1 similar comment
@coveralls
Copy link

@coveralls coveralls commented Jul 11, 2017

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2203/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 6b5f30589ede34f153dbac174c5b234aff21bb64 # timeout=10Checking out Revision 6b5f30589ede34f153dbac174c5b234aff21bb64 (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6b5f30589ede34f153dbac174c5b234aff21bb64First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Copy link
Member

@devin-petersohn devin-petersohn left a comment

Nice! Looks great.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 11, 2017

OK great! I'll flesh this out today. Don't merge as is!

@fnothaft fnothaft force-pushed the fnothaft:issues/1576-generic-rdd-to-specific branch from 2b72fff to f27ee3d Jul 11, 2017
@coveralls
Copy link

@coveralls coveralls commented Jul 11, 2017

Coverage Status

Coverage increased (+0.2%) to 84.29% when pulling f27ee3d on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2206/
Test PASSed.

@fnothaft fnothaft force-pushed the fnothaft:issues/1576-generic-rdd-to-specific branch from f27ee3d to e25d5a6 Jul 12, 2017
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 12, 2017

This is good to go from my end.

@coveralls
Copy link

@coveralls coveralls commented Jul 12, 2017

Coverage Status

Coverage increased (+0.5%) to 84.525% when pulling e25d5a6 on fnothaft:issues/1576-generic-rdd-to-specific into 238e044 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 12, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2207/
Test PASSed.

@fnothaft fnothaft force-pushed the fnothaft:issues/1576-generic-rdd-to-specific branch from e25d5a6 to 30a6119 Jul 13, 2017
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 13, 2017

Rebased and cleared conflicts. This is good to go from my side.

@coveralls
Copy link

@coveralls coveralls commented Jul 13, 2017

Coverage Status

Coverage increased (+0.4%) to 84.533% when pulling 30a6119 on fnothaft:issues/1576-generic-rdd-to-specific into 607cd50 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 13, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2215/
Test PASSed.

Copy link
Member

@heuermh heuermh left a comment

This is pretty cool and fairly powerful. It also may lead to confusion, as it allows users to create GenomicRDDs with records that break preconditions and validation checks.

@fnothaft fnothaft force-pushed the fnothaft:issues/1576-generic-rdd-to-specific branch from 30a6119 to 751da13 Jul 17, 2017
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 17, 2017

Rebased and cleared conflicts.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 17, 2017

Also, WRT @heuermh's comment, there's nothing that stops an enterprising user from breaking our preconditions today. This might make it easier, but not really any easier than just using transform today is.

@coveralls
Copy link

@coveralls coveralls commented Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2227/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c24fe77ef226e0041a2f34d4212526c893402e9c # timeout=10Checking out Revision c24fe77ef226e0041a2f34d4212526c893402e9c (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c24fe77ef226e0041a2f34d4212526c893402e9cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 17, 2017

Looks like a transient s3 failure.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 17, 2017

Jenkins, test this please.

@coveralls
Copy link

@coveralls coveralls commented Jul 17, 2017

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 17, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2230/
Test PASSed.

@heuermh heuermh merged commit 961f90a into bigdatagenomics:master Jul 17, 2017
2 of 3 checks passed
2 of 3 checks passed
codacy/pr Not so good... This pull request quality could be better.
Details
coverage/coveralls Coverage increased (+0.4%) to 84.58%
Details
default Merged build finished.
Details
@heuermh
Copy link
Member

@heuermh heuermh commented Jul 17, 2017

Thank you, @fnothaft!

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.