New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc shuffle join fixes. #1253

Merged
merged 4 commits into from Dec 6, 2016

Conversation

Projects
3 participants
@fnothaft
Member

fnothaft commented Nov 8, 2016

Resolves #1220, #1222. Could use some more unit test coverage; I'll make a pass back and get that.

@fnothaft fnothaft added this to the 0.21.0 milestone Nov 8, 2016

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 8, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1570/
Test PASSed.

AmplabJenkins commented Nov 8, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1570/
Test PASSed.

@heuermh

This eyeballs ok for me and builds fine locally. ok to merge now, or wait for more unit test coverage?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 28, 2016

Member

<ping> Ready to go or wait for more unit tests?

Member

heuermh commented Nov 28, 2016

<ping> Ready to go or wait for more unit tests?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Nov 28, 2016

Member

Not quite ready to go yet, will get to the tests today.

Member

fnothaft commented Nov 28, 2016

Not quite ready to go yet, will get to the tests today.

fnothaft added some commits Nov 8, 2016

[ADAM-1222] Fix loss of instrumentation in ShuffleRegionJoin.
Resolves #1222. ShuffleRegionJoin was importing
`org.apache.spark.SparkContext._` instead of
`org.apache.spark.rdd.MetricsContext._`.
[ADAM-1220] Fix optPartitions parameter in shuffle region join hooks …
…in GenomicRDD.

Resolves #1220. Adds a function called in each of the shuffle join implementations
that calculates the sequence dictionary after the join, as well as the partition sizes
to request.
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 1, 2016

Member

Added commit c01b8a7, which adds a LOT of tests (and fixes some bugs that the tests uncovered). This is good to go now from my side. I think c01b8a7 should get squashed into 11725a8 before merging, though.

Member

fnothaft commented Dec 1, 2016

Added commit c01b8a7, which adds a LOT of tests (and fixes some bugs that the tests uncovered). This is good to go now from my side. I think c01b8a7 should get squashed into 11725a8 before merging, though.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 1, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1648/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1253/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1253/merge^{commit} # timeout=10Checking out Revision 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 (origin/pr/1253/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2d7ed52ffbfae554560f53937f3edd9fd72b9b54First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Dec 1, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1648/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1253/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1253/merge^{commit} # timeout=10Checking out Revision 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 (origin/pr/1253/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2d7ed52ffbfae554560f53937f3edd9fd72b9b54First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 1, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1654/
Test PASSed.

AmplabJenkins commented Dec 1, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1654/
Test PASSed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 2, 2016

Member

Could you squash these commits as you'd like? Then I'll merge using the Rebase and Merge button.

Member

heuermh commented Dec 2, 2016

Could you squash these commits as you'd like? Then I'll merge using the Rebase and Merge button.

Improved test coverage of RegionJoin code.
Added tests to all of the GenomicRDD test suites to run all of the available
broadcast and shuffle region join implementations across all of the various
GenomicRDD types. This caught several bugs, which were fixed in this commit.
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 6, 2016

Member

@heuermh I decided not to squash any of the commits down, but I amended the last commit message to make it more clear about what the last commit does. I've just pushed.

Member

fnothaft commented Dec 6, 2016

@heuermh I decided not to squash any of the commits down, but I amended the last commit message to make it more clear about what the last commit does. I've just pushed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 6, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1660/
Test PASSed.

AmplabJenkins commented Dec 6, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1660/
Test PASSed.

@heuermh heuermh merged commit 1478739 into bigdatagenomics:master Dec 6, 2016

1 check passed

default Merged build finished.
Details
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 6, 2016

Member

Thank you, @fnothaft!

Member

heuermh commented Dec 6, 2016

Thank you, @fnothaft!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment