New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor variant and genotype annotations #1078

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@heuermh
Member

heuermh commented Jul 7, 2016

Update per bigdatagenomics/bdg-formats#90. Currently compiles but fails 14 unit tests.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1329/

Build result: FAILURE

GitHub pull request #1078 of commit abe149f automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1078/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1f177ed # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1078/merge^{commit} # timeout=10Checking out Revision 1f177ed (origin/pr/1078/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1f177ed3fd266495962381d8fb918c66d7dcd4a8First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jul 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1329/

Build result: FAILURE

GitHub pull request #1078 of commit abe149f automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1078/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1f177ed # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1078/merge^{commit} # timeout=10Checking out Revision 1f177ed (origin/pr/1078/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1f177ed3fd266495962381d8fb918c66d7dcd4a8First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Genotype,
VariantCallingAnnotations
}
import org.bdgenomics.formats.avro.{ Genotype, GenotypeAnnotation, VariantAnnotation }

This comment has been minimized.

@fnothaft

fnothaft Jul 19, 2016

Member

How long is this line? I generally break the imports into multiple lines if I'm importing more than 4 things, or if the import line is >80 characters.

@fnothaft

fnothaft Jul 19, 2016

Member

How long is this line? I generally break the imports into multiple lines if I'm importing more than 4 things, or if the import line is >80 characters.

* Mapping between VCF format field names and field IDs in the Genotype schema.
*/
lazy val VCF2GenotypeAnnotations: Map[String, (Int, Object => Object)] =
lazy val VCF2Genotype: Map[String, (Int, Object => Object)] =

This comment has been minimized.

@fnothaft

fnothaft Jul 19, 2016

Member

lower caps on vcf and s/2/To/g?

@fnothaft

fnothaft Jul 19, 2016

Member

lower caps on vcf and s/2/To/g?

*/
lazy val VCF2DatabaseAnnotations: Map[String, (Int, Object => Object)] = createFieldMap(EXTERNAL_DATABASE_KEYS,
DatabaseVariantAnnotation.getClassSchema)
lazy val VCF2VariantAnnotation: Map[String, (Int, Object => Object)] =

This comment has been minimized.

@fnothaft

fnothaft Jul 19, 2016

Member

Ditto here RE: caps, 2.

@fnothaft

fnothaft Jul 19, 2016

Member

Ditto here RE: caps, 2.

@jpdna

This comment has been minimized.

Show comment
Hide comment
@jpdna

jpdna Jul 26, 2016

Member

When trying to build the code at huerermh:annotations abe149f I get an error:

[ERROR] /home/jp/Berk/work/issues/PR1078/adam/adam-core/src/main/scala/org/bdgenomics/adam/converters/VariantContextConverter.scala:498: error: value setVariantIsPassing is not a member of org.bdgenomics.formats.avro.GenotypeAnnotation.Builder

because there is no variantIsPassing field in GenotypeAnnotation in bdg-formats master at 5e81a9c.
Is there a different version of bdg-formats I should build/install first?

Member

jpdna commented Jul 26, 2016

When trying to build the code at huerermh:annotations abe149f I get an error:

[ERROR] /home/jp/Berk/work/issues/PR1078/adam/adam-core/src/main/scala/org/bdgenomics/adam/converters/VariantContextConverter.scala:498: error: value setVariantIsPassing is not a member of org.bdgenomics.formats.avro.GenotypeAnnotation.Builder

because there is no variantIsPassing field in GenotypeAnnotation in bdg-formats master at 5e81a9c.
Is there a different version of bdg-formats I should build/install first?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 26, 2016

Member

I think it needs to be updated for bigdatagenomics/bdg-formats#88.

Member

fnothaft commented Jul 26, 2016

I think it needs to be updated for bigdatagenomics/bdg-formats#88.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 26, 2016

Member

Right, #1075 needs to be applied first

Member

heuermh commented Jul 26, 2016

Right, #1075 needs to be applied first

@jpdna

This comment has been minimized.

Show comment
Hide comment
@jpdna

jpdna Jul 26, 2016

Member

@heuermh do you have a VCF file with the ANN tag that you have used for testing?
I made an annotated file using SNPEFF over the ClinVar dataset, and then tried running:

val x1 = sc.loadVcfAnnotations("file:///home/jp/Berk/work/issues/PR1078/run1/snpeff.vcf")
val test = x1.take(1)

and I'm getting a NullPointerException that I am investigating.
It would be useful to me to have a positive control to check that the above sc.loadVcfAnnotations works for well formed input VCF with ANN tag

Member

jpdna commented Jul 26, 2016

@heuermh do you have a VCF file with the ANN tag that you have used for testing?
I made an annotated file using SNPEFF over the ClinVar dataset, and then tried running:

val x1 = sc.loadVcfAnnotations("file:///home/jp/Berk/work/issues/PR1078/run1/snpeff.vcf")
val test = x1.take(1)

and I'm getting a NullPointerException that I am investigating.
It would be useful to me to have a positive control to check that the above sc.loadVcfAnnotations works for well formed input VCF with ANN tag

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 26, 2016

Member

As mentioned above, this branch isn't ready to be reviewed: Currently compiles but fails 14 unit tests.

Member

heuermh commented Jul 26, 2016

As mentioned above, this branch isn't ready to be reviewed: Currently compiles but fails 14 unit tests.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 26, 2016

Member

For the record, I'm not too good with having lots of different things open and unresolved at the same time, and I currently have four open pull requests in the same code path. If I could get #1060 and #1069 merged soon, that would be helpful.

Then #1075 and this pull request need a 0.10.0 release of bdg-formats, which is waiting on doc fixes, see bigdatagenomics/bdg-formats#92.

Member

heuermh commented Jul 26, 2016

For the record, I'm not too good with having lots of different things open and unresolved at the same time, and I currently have four open pull requests in the same code path. If I could get #1060 and #1069 merged soon, that would be helpful.

Then #1075 and this pull request need a 0.10.0 release of bdg-formats, which is waiting on doc fixes, see bigdatagenomics/bdg-formats#92.

@jpdna

This comment has been minimized.

Show comment
Hide comment
@jpdna

jpdna Jul 26, 2016

Member

As mentioned above, this branch isn't ready to be reviewed:

Ah, no problem, ping when ready - I'll plan to stress test with annotations from snpEff/VEP

Member

jpdna commented Jul 26, 2016

As mentioned above, this branch isn't ready to be reviewed:

Ah, no problem, ping when ready - I'll plan to stress test with annotations from snpEff/VEP

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 26, 2016

Member

Rebased and cherry-picked commit from #1075. Now there are 15 unit test failures. :)

Member

heuermh commented Jul 26, 2016

Rebased and cherry-picked commit from #1075. Now there are 15 unit test failures. :)

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Jul 26, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1362/

Build result: FAILURE

GitHub pull request #1078 of commit 26a84bc automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1078/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains da21402 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1078/merge^{commit} # timeout=10Checking out Revision da21402 (origin/pr/1078/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f da21402573d19edaafedcbd931e50a0e2605b6a1First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins commented Jul 26, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1362/

Build result: FAILURE

GitHub pull request #1078 of commit 26a84bc automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1078/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains da21402 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1078/merge^{commit} # timeout=10Checking out Revision da21402 (origin/pr/1078/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f da21402573d19edaafedcbd931e50a0e2605b6a1First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Aug 31, 2016

Member

We're closing this in favor of #1135, right @heuermh? Please reopen if I've incorrectly closed.

Member

fnothaft commented Aug 31, 2016

We're closing this in favor of #1135, right @heuermh? Please reopen if I've incorrectly closed.

@fnothaft fnothaft closed this Aug 31, 2016

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 31, 2016

Member

Yes, thank you.

Member

heuermh commented Aug 31, 2016

Yes, thank you.

@heuermh heuermh deleted the heuermh:annotations branch Aug 31, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment