[ADAM-1334] Clean up serialization issues in Broadcast region join. #1336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1336/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 9b8e3ed # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1336/merge^{commit} # timeout=10Checking out Revision 9b8e3ed (origin/pr/1336/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 9b8e3edf595ec4270648b72a986c9726ba1085afFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
|
Jenkins, retest this please. |
|
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1336/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 9b8e3ed # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1336/merge^{commit} # timeout=10Checking out Revision 9b8e3ed (origin/pr/1336/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 9b8e3edf595ec4270648b72a986c9726ba1085afFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
|
I assume you would like this in 0.21.0? Feel free to set the milestone. |
|
Ah, yes! We need it for the Variant DB challenge. I've just set the milestone to 0.21.0. I'm going to fix the build failure (it's a small issue where the move_to_xyz scripts need to be updated), and once the build passes, I'll cut a utils release so that we can remove the snapshot dependency. |
|
Sounds good, thanks! |
fnothaft
force-pushed
the
issues/1334-tree-serialization-erasure
branch
from
ef72554
to
949970b
Compare
|
Just pushed a fix for the move_to_xyz scripts. This will not be ready to merge though, until the utils release is cut (which I will do after this passes). |
|
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1336/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains fa154df # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1336/merge^{commit} # timeout=10Checking out Revision fa154df (origin/pr/1336/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f fa154dfaa75c1ab94a9583693abb1703a9803f21First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
heuermh
approved these changes
Jan 5, 2017
| import scala.collection.JavaConversions._ | ||
| import scala.math.max | ||
| import scala.reflect.ClassTag | ||
|
|
||
| private[adam] case class NucleotideContigFragmentArray( |
There was a problem hiding this comment.
I think I know the reason already, why do we need concrete classes for each of these?
There was a problem hiding this comment.
Type erasure at serialization time. Since we previously only had generics, we were getting errors that were "last registered IntervalArray class wins". It was great. Uplifting, really.
| @@ -74,6 +82,15 @@ trait TreeRegionJoin[T, U] { | |||
| */ | |||
| case class InnerTreeRegionJoin[T: ClassTag, U: ClassTag]() extends RegionJoin[T, U, T, U] with TreeRegionJoin[T, U] { | |||
|
|
|||
| def broadcastAndJoin(tree: IntervalArray[ReferenceRegion, T], | |||
fnothaft
force-pushed
the
issues/1334-tree-serialization-erasure
branch
from
949970b
to
43e2519
Compare
|
Test PASSed. |
|
I will cut a bdg-utils release tomorrow AM. |
|
Is this on HEAD related? |
|
Just cut the new bdg-utils release. Once that pushes to maven, I will update this, and we should be able to merge it and cut the ADAM release. |
|
bdg-utils 0.2.11 is now available on Maven Central. Since posting the stack trace above, I've been looking into other things. Should I investigate it further this evening? |
Resolves bigdatagenomics#1334. Eliminates type erasure problems by having different concrete implementations per each broadcast. Depends on bigdatagenomics/utils#97.
fnothaft
force-pushed
the
issues/1334-tree-serialization-erasure
branch
from
43e2519
to
9c0668b
Compare
I've just updated to point at the 0.2.11 release. Let me see if I can repro that issue you ran into on my side. |
|
Oh, I see what's going on in your example. If you want to call to Or, change: to either or, more simply: ;) |
|
Test PASSed. |
|
Of course, that does it! I look forward to wiping out the |
|
Thank you, @fnothaft! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #1334. Eliminates type erasure problems by having different concrete implementations per each broadcast. Depends on bigdatagenomics/utils#97.