New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Efficient Joins and (re)Partitioning #1324

Merged
merged 4 commits into from Jun 5, 2017

Conversation

Projects
7 participants
@devin-petersohn
Member

devin-petersohn commented Dec 23, 2016

Ready for review.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 23, 2016

Can one of the admins verify this patch?

AmplabJenkins commented Dec 23, 2016

Can one of the admins verify this patch?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 23, 2016

Member

Jenkins, add to whitelist.

Member

fnothaft commented Dec 23, 2016

Jenkins, add to whitelist.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 23, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1700/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1bbc8ab # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 1bbc8ab (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1bbc8abae508530dab55ae88be11935a74b594e0First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Dec 23, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1700/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1bbc8ab # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 1bbc8ab (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1bbc8abae508530dab55ae88be11935a74b594e0First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 23, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1701/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 58cfe15 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 58cfe15 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 58cfe152a16c84931ba087d38ffa53a9397b5bbfFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Dec 23, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1701/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 58cfe15 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 58cfe15 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 58cfe152a16c84931ba087d38ffa53a9397b5bbfFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

Generally LGTM! Just made a first review pass. For more detailed review, I need more inline docs.

In addition to the changes I suggested, please remove the .parquet test resource files from this PR. If needed for tests, they should be generated inside the test. We try to avoid checking in binary files whenever possible.

Linking to #1216.

Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated ...ore/src/main/scala/org/bdgenomics/adam/rdd/read/AlignmentRecordRDD.scala Outdated
Show outdated Hide outdated ...ore/src/main/scala/org/bdgenomics/adam/rdd/read/AlignmentRecordRDD.scala Outdated
Show outdated Hide outdated ...c/test/scala/org/bdgenomics/adam/models/RecordGroupDictionarySuite.scala Outdated
Show outdated Hide outdated ...-core/src/test/scala/org/bdgenomics/adam/rdd/SortedGenomicRDDSuite.scala Outdated
@@ -0,0 +1,72 @@
package org.bdgenomics.adam.rdd

This comment has been minimized.

@fnothaft

fnothaft Dec 23, 2016

Member

Add license header.

@fnothaft

fnothaft Dec 23, 2016

Member

Add license header.

This comment has been minimized.

@fnothaft

fnothaft Dec 28, 2016

Member

Running ./scripts/format-source should add the license header.

@fnothaft

fnothaft Dec 28, 2016

Member

Running ./scripts/format-source should add the license header.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 28, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1709/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 86150cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 86150cf (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 86150cfa21599654d53606727c562e3ae5c93ec3First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Dec 28, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1709/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 86150cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 86150cf (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 86150cfa21599654d53606727c562e3ae5c93ec3First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

Made an additional review pass. Let's chat about the boundary case where we have a record that is duplicated across partition boundaries.

Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated ...ore/src/main/scala/org/bdgenomics/adam/rdd/read/AlignmentRecordRDD.scala Outdated
@@ -0,0 +1,72 @@
package org.bdgenomics.adam.rdd

This comment has been minimized.

@fnothaft

fnothaft Dec 28, 2016

Member

Running ./scripts/format-source should add the license header.

@fnothaft

fnothaft Dec 28, 2016

Member

Running ./scripts/format-source should add the license header.

Show outdated Hide outdated ...-core/src/test/scala/org/bdgenomics/adam/rdd/SortedGenomicRDDSuite.scala Outdated
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jan 4, 2017

Member

ok to set the milestone for this as 0.22.0?

Member

heuermh commented Jan 4, 2017

ok to set the milestone for this as 0.22.0?

@fnothaft fnothaft added this to the 0.22.0 milestone Jan 4, 2017

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 4, 2017

Member

+1 @heuermh, triaged to 0.22.0.

Member

fnothaft commented Jan 4, 2017

+1 @heuermh, triaged to 0.22.0.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1729/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains e244fcfc34d1580be73e9c76e6dc72ceced60383 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision e244fcfc34d1580be73e9c76e6dc72ceced60383 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e244fcfc34d1580be73e9c76e6dc72ceced60383First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1729/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains e244fcfc34d1580be73e9c76e6dc72ceced60383 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision e244fcfc34d1580be73e9c76e6dc72ceced60383 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e244fcfc34d1580be73e9c76e6dc72ceced60383First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1730/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8f803afa8e862dff1626eb9002de0cb27175465a # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 8f803afa8e862dff1626eb9002de0cb27175465a (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8f803afa8e862dff1626eb9002de0cb27175465aFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1730/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8f803afa8e862dff1626eb9002de0cb27175465a # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 8f803afa8e862dff1626eb9002de0cb27175465a (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8f803afa8e862dff1626eb9002de0cb27175465aFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@devin-petersohn

This comment has been minimized.

Show comment
Hide comment
@devin-petersohn

devin-petersohn Jan 11, 2017

Member

@fnothaft @heuermh How do we handle cross strand joins in the shuffleRegionJoin code? From what I can tell, nothing special is done about it, but the bug I encountered involved the ReferenceRegion.overlap() skipping things that were not on the same strand.

Member

devin-petersohn commented Jan 11, 2017

@fnothaft @heuermh How do we handle cross strand joins in the shuffleRegionJoin code? From what I can tell, nothing special is done about it, but the bug I encountered involved the ReferenceRegion.overlap() skipping things that were not on the same strand.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 11, 2017

Member

Cross-strand joins are handled by

.

ReferenceRegion.overlap() skipping things that were not on the same strand.

Ah yes, things that are not on the same strand do not overlap. ;) This is intentional. IIRC, we clarified a lot of this in e98ee2d. I think the question here is less "what behavior is correct in ReferenceRegion.overlaps" and more "should stranded or unstranded reference regions be input into the region join"? Would you agree? CC @laserson who might have opinions as well.

Member

fnothaft commented Jan 11, 2017

Cross-strand joins are handled by

.

ReferenceRegion.overlap() skipping things that were not on the same strand.

Ah yes, things that are not on the same strand do not overlap. ;) This is intentional. IIRC, we clarified a lot of this in e98ee2d. I think the question here is less "what behavior is correct in ReferenceRegion.overlaps" and more "should stranded or unstranded reference regions be input into the region join"? Would you agree? CC @laserson who might have opinions as well.

@laserson

This comment has been minimized.

Show comment
Hide comment
@laserson

laserson Jan 11, 2017

Contributor

Seems like we should support both stranded and un-stranded. Could this simply be passed through to the operation that determines whether there is an overlap?

Contributor

laserson commented Jan 11, 2017

Seems like we should support both stranded and un-stranded. Could this simply be passed through to the operation that determines whether there is an overlap?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jan 11, 2017

Member

Seems like we should support both stranded and un-stranded. Could this simply be passed through to the operation that determines whether there is an overlap?

I think there's a couple of ways we could do it. I think the simplest would be to generate unstranded reference regions as the keys to the join (after e98ee2d, this is v. easy). Perhaps we'd just pass this as a switch on the GenomicRDD methods?

Member

fnothaft commented Jan 11, 2017

Seems like we should support both stranded and un-stranded. Could this simply be passed through to the operation that determines whether there is an overlap?

I think there's a couple of ways we could do it. I think the simplest would be to generate unstranded reference regions as the keys to the join (after e98ee2d, this is v. easy). Perhaps we'd just pass this as a switch on the GenomicRDD methods?

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1731/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8ccd18912301e91573c5245dc5c09edcf11fc337 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 8ccd18912301e91573c5245dc5c09edcf11fc337 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8ccd18912301e91573c5245dc5c09edcf11fc337First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1731/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8ccd18912301e91573c5245dc5c09edcf11fc337 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 8ccd18912301e91573c5245dc5c09edcf11fc337 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8ccd18912301e91573c5245dc5c09edcf11fc337First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1733/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2b68245 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 2b68245 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2b682452ac6d6453870022697b535357b023771dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 11, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1733/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2b68245 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 2b68245 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2b682452ac6d6453870022697b535357b023771dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 16, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1736/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains c9aac611a0d99b8f3e2e6266369d1fcf6467ef46 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision c9aac611a0d99b8f3e2e6266369d1fcf6467ef46 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c9aac611a0d99b8f3e2e6266369d1fcf6467ef46First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 16, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1736/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains c9aac611a0d99b8f3e2e6266369d1fcf6467ef46 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision c9aac611a0d99b8f3e2e6266369d1fcf6467ef46 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c9aac611a0d99b8f3e2e6266369d1fcf6467ef46First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1737/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b7204f839e6a69208114c630cf6cd025c13a6096 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision b7204f839e6a69208114c630cf6cd025c13a6096 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b7204f839e6a69208114c630cf6cd025c13a6096First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1737/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b7204f839e6a69208114c630cf6cd025c13a6096 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision b7204f839e6a69208114c630cf6cd025c13a6096 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b7204f839e6a69208114c630cf6cd025c13a6096First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

val c = jRdd0.rdd.collect
val c = jRdd.rdd.collect

This comment has been minimized.

@devin-petersohn

devin-petersohn Jan 17, 2017

Member

This seemed to be incorrect previously. Let me know if I should revert.

@devin-petersohn

devin-petersohn Jan 17, 2017

Member

This seemed to be incorrect previously. Let me know if I should revert.

@devin-petersohn

This comment has been minimized.

Show comment
Hide comment
@devin-petersohn

devin-petersohn Jan 17, 2017

Member

Pinging @fnothaft @heuermh for review. Please ignore the whitespace issues for now, I will push something shortly that fixes that.

Member

devin-petersohn commented Jan 17, 2017

Pinging @fnothaft @heuermh for review. Please ignore the whitespace issues for now, I will push something shortly that fixes that.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1738/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ec88406f303d32bcdaefa004bfd2c31ad748d080 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision ec88406f303d32bcdaefa004bfd2c31ad748d080 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ec88406f303d32bcdaefa004bfd2c31ad748d080First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1738/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ec88406f303d32bcdaefa004bfd2c31ad748d080 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision ec88406f303d32bcdaefa004bfd2c31ad748d080 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ec88406f303d32bcdaefa004bfd2c31ad748d080First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1739/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 46af637fa97d2727a0077052d50d55270b18b707 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 46af637fa97d2727a0077052d50d55270b18b707 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 46af637fa97d2727a0077052d50d55270b18b707First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 17, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1739/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 46af637fa97d2727a0077052d50d55270b18b707 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision 46af637fa97d2727a0077052d50d55270b18b707 (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 46af637fa97d2727a0077052d50d55270b18b707First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jan 18, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1740/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ab4cefe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision ab4cefe (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ab4cefeabf657914e554f7acd41e735144cfd5b2First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jan 18, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1740/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1324/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ab4cefe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1324/merge^{commit} # timeout=10Checking out Revision ab4cefe (origin/pr/1324/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ab4cefeabf657914e554f7acd41e735144cfd5b2First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 26, 2017

Member

Thanks @devin-petersohn! I'm getting on a plane right now but should have time tomorrow AM to review.

Member

fnothaft commented May 26, 2017

Thanks @devin-petersohn! I'm getting on a plane right now but should have time tomorrow AM to review.

Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMInputFormat.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
* @param filename the filename for the metadata
* @return a partition map if the data was written sorted, or an empty Seq if unsorted
*/
private[rdd] def extractPartitionMap(

This comment has been minimized.

@fnothaft

fnothaft May 26, 2017

Member

IMO, we should move this into an avro schema or something more structured, instead of parsing JSON. That said, let's not do it here and now. Can you open a ticket to refactor this into avroland post 0.23.0?

@fnothaft

fnothaft May 26, 2017

Member

IMO, we should move this into an avro schema or something more structured, instead of parsing JSON. That said, let's not do it here and now. Can you open a ticket to refactor this into avroland post 0.23.0?

This comment has been minimized.

@heuermh

heuermh May 26, 2017

Member

+1

@heuermh

heuermh May 26, 2017

Member

+1

Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala Outdated
Show outdated Hide outdated ...ore/src/main/scala/org/bdgenomics/adam/rdd/ManualRegionPartitioner.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala Outdated
Show outdated Hide outdated adam-core/src/main/scala/org/bdgenomics/adam/rdd/ShuffleRegionJoin.scala Outdated
*
* @param rdd The underlying AlignmentRecord RDD.
* @return A new AlignmentRecordRDD.
*/
def unaligned(rdd: RDD[AlignmentRecord]): AlignmentRecordRDD = {
AlignmentRecordRDD(rdd,
SequenceDictionary.empty,

This comment has been minimized.

@heuermh

heuermh May 26, 2017

Member

This could call AlignmentRecordRDD(rdd, sequences, recordGroupDictionary, None) directly

@heuermh

heuermh May 26, 2017

Member

This could call AlignmentRecordRDD(rdd, sequences, recordGroupDictionary, None) directly

This comment has been minimized.

@fnothaft

fnothaft May 26, 2017

Member

Or even better, it could call the constructor.

@fnothaft

fnothaft May 26, 2017

Member

Or even better, it could call the constructor.

This comment has been minimized.

@heuermh

heuermh May 26, 2017

Member

That's what I meant. Or is there a different syntax?

@heuermh

heuermh May 26, 2017

Member

That's what I meant. Or is there a different syntax?

This comment has been minimized.

@devin-petersohn

devin-petersohn May 26, 2017

Member

See next 2 lines for solution

@devin-petersohn

devin-petersohn May 26, 2017

Member

See next 2 lines for solution

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh May 26, 2017

Member

All the apply refactoring looks good to me, thanks!

Member

heuermh commented May 26, 2017

All the apply refactoring looks good to me, thanks!

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 26, 2017

Coverage Status

Coverage decreased (-0.07%) to 81.967% when pulling 7805a7f on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

coveralls commented May 26, 2017

Coverage Status

Coverage decreased (-0.07%) to 81.967% when pulling 7805a7f on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 26, 2017

Coverage Status

Coverage increased (+0.4%) to 82.418% when pulling 7805a7f on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

coveralls commented May 26, 2017

Coverage Status

Coverage increased (+0.4%) to 82.418% when pulling 7805a7f on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins May 26, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2067/
Test PASSed.

AmplabJenkins commented May 26, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2067/
Test PASSed.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 26, 2017

Coverage Status

Coverage decreased (-0.08%) to 81.96% when pulling f15fe55 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

coveralls commented May 26, 2017

Coverage Status

Coverage decreased (-0.08%) to 81.96% when pulling f15fe55 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 26, 2017

Coverage Status

Coverage increased (+0.6%) to 82.599% when pulling f15fe55 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

coveralls commented May 26, 2017

Coverage Status

Coverage increased (+0.6%) to 82.599% when pulling f15fe55 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins May 26, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2069/
Test PASSed.

AmplabJenkins commented May 26, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2069/
Test PASSed.

@devin-petersohn

This comment has been minimized.

Show comment
Hide comment
@devin-petersohn

devin-petersohn May 30, 2017

Member

@fnothaft, @heuermh, I think we're good to go for another review. At your earliest convenience of course.

Member

devin-petersohn commented May 30, 2017

@fnothaft, @heuermh, I think we're good to go for another review. At your earliest convenience of course.

@heuermh

Boom!

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 31, 2017

Coverage Status

Coverage increased (+0.6%) to 82.599% when pulling 536b936 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

coveralls commented May 31, 2017

Coverage Status

Coverage increased (+0.6%) to 82.599% when pulling 536b936 on devin-petersohn:partitioner into 3ea4f18 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins May 31, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2076/
Test PASSed.

AmplabJenkins commented May 31, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2076/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 31, 2017

Member

@devin-petersohn you've got a conflict in adam-core/src/main/scala/org/bdgenomics/adam/rdd/variant/VariantContextRDD.scala. Can you rebase and clear the conflict?

Member

fnothaft commented May 31, 2017

@devin-petersohn you've got a conflict in adam-core/src/main/scala/org/bdgenomics/adam/rdd/variant/VariantContextRDD.scala. Can you rebase and clear the conflict?

devin-petersohn added some commits Apr 5, 2017

Deep rewrite of shuffleRegionJoin, adding sorted knowledge to GenomicRDD
Addressing reviewer comments

Performance improvements related to decreasing compute

Major refactor to clean up ShuffleRegionJoin

Fixing serialization issue

Cleaning up tests and code

More cleanup

Addressing reviewer comments

Addressing reviewer comments, cleaning up after intellij

Testing that data stays sorted on Jenkins

Still debugging the sorted persistance

Testing that we can persist sort

Checking output to hopefully see what is happening

Testing more related to sort persist

Adding ADAMInputFormat class to ensure ordering is maintained

Adding docs and cleanup

Cleaning up code a bit

Cleaning up after IntelliJ

Addressing reviewer comments

Addressing reviewer comments

Addressing reviewer comments

Clean up docs, cutting at 80 characters.

Addressing reviewer comments

Addressing reviewer comments. sorted to isSorted

Fixing some spacing issues

Addressing reviwer comments
@devin-petersohn

This comment has been minimized.

Show comment
Hide comment
@devin-petersohn

devin-petersohn May 31, 2017

Member

@fnothaft Rebased, thanks!

Member

devin-petersohn commented May 31, 2017

@fnothaft Rebased, thanks!

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 31, 2017

Coverage Status

Coverage decreased (-0.2%) to 82.569% when pulling 7a9503c on devin-petersohn:partitioner into b7762c2 on bigdatagenomics:master.

coveralls commented May 31, 2017

Coverage Status

Coverage decreased (-0.2%) to 82.569% when pulling 7a9503c on devin-petersohn:partitioner into b7762c2 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins May 31, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2077/
Test PASSed.

AmplabJenkins commented May 31, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2077/
Test PASSed.

@fnothaft fnothaft merged commit e5ae270 into bigdatagenomics:master Jun 5, 2017

1 of 3 checks passed

codacy/pr Not so good... This pull request quality could be better.
Details
coverage/coveralls Coverage decreased (-0.2%) to 82.569%
Details
default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jun 5, 2017

Member

Merged! Thanks @devin-petersohn!

Member

fnothaft commented Jun 5, 2017

Merged! Thanks @devin-petersohn!

@devin-petersohn devin-petersohn referenced this pull request Jul 7, 2017

Merged

[ADAM-1018] Add support for Spark SQL Datasets. #1391

4 of 4 tasks complete

@heuermh heuermh moved this from Triage to Completed in Release 0.23.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment