Add pipes API #1114

Merged
merged 2 commits into from Oct 14, 2016

Conversation

Projects
None yet
3 participants
@fnothaft
Member

fnothaft commented Aug 12, 2016

WIP towards #1112, resolves #1113. Will post TODOs later.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Aug 12, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1380/

Build result: FAILURE

GitHub pull request #1114 of commit da43795 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 38c84c0 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 38c84c0 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 38c84c0499ad2b4ca2eaaf0090e6048c95eeb63eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1380/

Build result: FAILURE

GitHub pull request #1114 of commit da43795 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 38c84c0 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 38c84c0 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 38c84c0499ad2b4ca2eaaf0090e6048c95eeb63eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh heuermh referenced this pull request Sep 7, 2016

Closed

Release ADAM version 0.20.0 #1048

47 of 61 tasks complete
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Sep 15, 2016

Member

I believe this is good to go from my side; pending testing on the cluster tomorrow. Ping for review.

Member

fnothaft commented Sep 15, 2016

I believe this is good to go from my side; pending testing on the cluster tomorrow. Ping for review.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Sep 15, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1494/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1494/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Sep 15, 2016

Member

Rebased and squashed.

Member

fnothaft commented Sep 15, 2016

Rebased and squashed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Sep 15, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1496/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1496/
Test PASSed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Sep 19, 2016

Member

Will test this tomorrow.

Member

heuermh commented Sep 19, 2016

Will test this tomorrow.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Sep 19, 2016

Member

I adapted the unit test in VariantContextRDDSuite to a new project at https://github.com/heuermh/adam-snpeff (it has to be separate due to LGPL version 3.0 licensing).

The new VCF formatter classes are private[variation] resulting in

[INFO] Compiling 1 Scala source to adam-snpeff/target/scala-2.10.5/classes...
[ERROR] adam-snpeff/src/main/scala/com/github/heuermh/adam/snpeff/AdamSnpEff.scala:56: class VCFInFormatter in package variation cannot be accessed in package org.bdgenomics.adam.rdd.variation
[ERROR]     val output: VariantContextRDD = input.pipe[VariantContext, VariantContextRDD, VCFInFormatter](snpEffCommand)
[ERROR]                                                                                   ^
[ERROR] one error found

Is that intentional, or am I using the public API incorrectly?

Member

heuermh commented Sep 19, 2016

I adapted the unit test in VariantContextRDDSuite to a new project at https://github.com/heuermh/adam-snpeff (it has to be separate due to LGPL version 3.0 licensing).

The new VCF formatter classes are private[variation] resulting in

[INFO] Compiling 1 Scala source to adam-snpeff/target/scala-2.10.5/classes...
[ERROR] adam-snpeff/src/main/scala/com/github/heuermh/adam/snpeff/AdamSnpEff.scala:56: class VCFInFormatter in package variation cannot be accessed in package org.bdgenomics.adam.rdd.variation
[ERROR]     val output: VariantContextRDD = input.pipe[VariantContext, VariantContextRDD, VCFInFormatter](snpEffCommand)
[ERROR]                                                                                   ^
[ERROR] one error found

Is that intentional, or am I using the public API incorrectly?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Sep 19, 2016

Member

Is that intentional, or am I using the public API incorrectly?

No, that's unintentional and was a goof on my side! I ran into another similar goof when testing locally. Since the tests are in the packages that are protected, this slips through easily. I'll fix that and push.

Member

fnothaft commented Sep 19, 2016

Is that intentional, or am I using the public API incorrectly?

No, that's unintentional and was a goof on my side! I ran into another similar goof when testing locally. Since the tests are in the packages that are protected, this slips through easily. I'll fix that and push.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 3, 2016

Member

@heuermh I've just pushed a new commit (https://github.com/bigdatagenomics/adam/pull/1114/commits/117155a5cf0bc1eec687f5401df5013d25155e52a) that should resolve the issue you were seeing. Let me know if this fixed things for you downstream, and I'll squash that commit down into it's predecessor.

Member

fnothaft commented Oct 3, 2016

@heuermh I've just pushed a new commit (https://github.com/bigdatagenomics/adam/pull/1114/commits/117155a5cf0bc1eec687f5401df5013d25155e52a) that should resolve the issue you were seeing. Let me know if this fixed things for you downstream, and I'll squash that commit down into it's predecessor.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 3, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1514/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1514/
Test PASSed.

+ * @tparam X The type of the record created by the piped command.
+ * @tparam Y A GenomicRDD containing X's.
+ */
+ def pipe[X, Y <: GenomicRDD[X, Y], V <: InFormatter[T, U, V]](cmd: String,

This comment has been minimized.

@heuermh

heuermh Oct 3, 2016

Member

Could you add a Map parameter with values to add to the ProcessBuilder.environment?

@heuermh

heuermh Oct 3, 2016

Member

Could you add a Map parameter with values to add to the ProcessBuilder.environment?

+case class VCFOutFormatter() extends OutFormatter[VariantContext] {
+
+ /**
+ * Reads alignment records from an input stream. Autodetects SAM/BAM format.

This comment has been minimized.

@heuermh

heuermh Oct 3, 2016

Member

Wrong doc comment, this is VCFOutFormatter

@heuermh

heuermh Oct 3, 2016

Member

Wrong doc comment, this is VCFOutFormatter

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 3, 2016

Member

This works for me in the downstream repo https://github.com/heuermh/adam-snpeff.

Or may work for me, since while it runs and produces VCF output, the output contains none of the SnpEff annotations.

$ spark-submit --class com.github.heuermh.adam.snpeff.AdamSnpEff target/adam-snpeff_2.10-0.19.1-SNAPSHOT.jar ../adam/adam-core/src/test/resources/small.vcf small-snpeff.vcf

$ cat small-snpeff.vcf/part-r-00000
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1       14397   .       CTGT    C       .       .       .       GT:AD:DP:GQ:PL  0/1:16,4:20:99:120,0,256        0/1:8,2:10:60:60,0,256  0/0:39,0:39:99:0,116,256
1       14522   .       G       A       .       .       .       GT:AD:DP:GQ:PL  0/1:10,5:15:99:99,0,256 0/1:2,5:7:34:128,0,34   0/0:26,0:26:78:0,78,256
1       19190   .       GC      G       .       .       .       GT:AD:DP:GQ:PL  0/1:8,14:22:99:256,0,256        0/1:18,13:31:99:256,0,256       0/1:5,15:20:99:256,0,107
1       63735   .       CCTA    C       .       .       .       GT:AD:DP:GQ:PL  0/0:27,0:27:79:0,79,256 0/0:40,0:40:99:0,117,256        0/1:23,74:97:99:256,0,256
1       752721  .       A       G       .       .       .       GT:AD:DP:GQ:PL  1/1:0,27:27:81:256,81,0 1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0

The same happens when run outside of the Pipes API

$ cat ../adam/adam-core/src/test/resources/small.vcf | snpEff -download GRCh38.82 > small-snpeff-cmdline.vcf
$ cat small-snpeff-cmdline.vcf
...
##SnpEffVersion="4.2 (build 2015-12-05), by Pablo Cingolani"
##SnpEffCmd="SnpEff  GRCh38.82 "
##INFO=<ID=ANN,Number=.,Type=String,Description="Functional annotations: 'Allele | Annotation | Annotation_Impact | Gene_Name | Gene_ID | Feature_Type | Feature_ID | Transcript_BioType | Rank | HGVS.c | HGVS.p | cDNA.pos / cDNA.length | CDS.pos / CDS.length | AA.pos / AA.length | Distance | ERRORS / WARNINGS / INFO' ">
##INFO=<ID=LOF,Number=.,Type=String,Description="Predicted loss of function effects for this variant. Format: 'Gene_Name | Gene_ID | Number_of_transcripts_in_gene | Percent_of_transcripts_affected' ">
##INFO=<ID=NMD,Number=.,Type=String,Description="Predicted nonsense mediated decay effects for this variant. Format: 'Gene_Name | Gene_ID | Number_of_transcripts_in_gene | Percent_of_transcripts_affected' ">
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1   14397   .   CTGT    C   139.12  IndelQD AC=2;AF=0.333;AN=6;BaseQRankSum=1.800;ClippingRankSum=0.138;DP=69;FS=7.786;MLEAC=2;MLEAF=0.333;MQ=26.84;MQ0=0;MQRankSum=-1.906;QD=1.55;ReadPosRankSum=0.384;ANN=C|downstream_gene_variant|MODIFIER|WASH7P|ENSG00000227232|transcript|ENST00000488147.1|unprocessed_pseudogene||n.*4_*6delACA|||||6|,C|downstream_gene_variant|MODIFIER|MIR6859-1|ENSG00000278267|transcript|ENST00000619216.1|miRNA||n.*2969_*2971delACA|||||2971|,C|downstream_gene_variant|MODIFIER|DDX11L1|ENSG00000223972|transcript|ENST00000450305.2|transcribed_unprocessed_pseudogene||n.*728_*730delTGT|||||728|,C|non_coding_exon_variant|MODIFIER|DDX11L1|ENSG00000223972|transcript|ENST00000456328.2|processed_transcript|3/3|n.1648_1650delTTG||||||INFO_REALIGN_3_PRIME  GT:AD:DP:FT:GQ:PL   0/1:16,4:20:rd:99:120,0,827 0/1:8,2:10:dp;rd:60:60,0,414    0/0:39,0:39:PASS:99:0,116,2114
...

$ adam-submit vcf2adam small-snpeff-cmdline.vcf small-snpeff-cmdline.adam
$ adam-submit adam2vcf small-snpeff-cmdline.adam small-snpeff-cmdline-adam.vcf
$ cat small-snpeff-cmdline-adam.vcf/part-r-00000
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=.,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12891 NA12892 NA12878
1   14397   .   CTGT    C   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:8,2:10:IndelQD:60:60,0,256  0/0:39,0:39:IndelQD:99:0,116,256    0/1:16,4:20:IndelQD:99:120,0,256
1   19190   .   GC  G   .   .   .   GT:AD:DP:GQ:PL  0/1:18,13:31:99:256,0,256   0/1:5,15:20:99:256,0,107    0/1:8,14:22:99:256,0,256
1   14522   .   G   A   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34   0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256 0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256
1   63735   .   CCTA    C   .   .   .   GT:AD:DP:GQ:PL  0/0:40,0:40:99:0,117,256    0/1:23,74:97:99:256,0,256   0/0:27,0:27:79:0,79,256
1   752721  .   A   G   .   .   .   GT:AD:DP:GQ:PL  1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0 1/1:0,27:27:81:256,81,0
Member

heuermh commented Oct 3, 2016

This works for me in the downstream repo https://github.com/heuermh/adam-snpeff.

Or may work for me, since while it runs and produces VCF output, the output contains none of the SnpEff annotations.

$ spark-submit --class com.github.heuermh.adam.snpeff.AdamSnpEff target/adam-snpeff_2.10-0.19.1-SNAPSHOT.jar ../adam/adam-core/src/test/resources/small.vcf small-snpeff.vcf

$ cat small-snpeff.vcf/part-r-00000
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1       14397   .       CTGT    C       .       .       .       GT:AD:DP:GQ:PL  0/1:16,4:20:99:120,0,256        0/1:8,2:10:60:60,0,256  0/0:39,0:39:99:0,116,256
1       14522   .       G       A       .       .       .       GT:AD:DP:GQ:PL  0/1:10,5:15:99:99,0,256 0/1:2,5:7:34:128,0,34   0/0:26,0:26:78:0,78,256
1       19190   .       GC      G       .       .       .       GT:AD:DP:GQ:PL  0/1:8,14:22:99:256,0,256        0/1:18,13:31:99:256,0,256       0/1:5,15:20:99:256,0,107
1       63735   .       CCTA    C       .       .       .       GT:AD:DP:GQ:PL  0/0:27,0:27:79:0,79,256 0/0:40,0:40:99:0,117,256        0/1:23,74:97:99:256,0,256
1       752721  .       A       G       .       .       .       GT:AD:DP:GQ:PL  1/1:0,27:27:81:256,81,0 1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0

The same happens when run outside of the Pipes API

$ cat ../adam/adam-core/src/test/resources/small.vcf | snpEff -download GRCh38.82 > small-snpeff-cmdline.vcf
$ cat small-snpeff-cmdline.vcf
...
##SnpEffVersion="4.2 (build 2015-12-05), by Pablo Cingolani"
##SnpEffCmd="SnpEff  GRCh38.82 "
##INFO=<ID=ANN,Number=.,Type=String,Description="Functional annotations: 'Allele | Annotation | Annotation_Impact | Gene_Name | Gene_ID | Feature_Type | Feature_ID | Transcript_BioType | Rank | HGVS.c | HGVS.p | cDNA.pos / cDNA.length | CDS.pos / CDS.length | AA.pos / AA.length | Distance | ERRORS / WARNINGS / INFO' ">
##INFO=<ID=LOF,Number=.,Type=String,Description="Predicted loss of function effects for this variant. Format: 'Gene_Name | Gene_ID | Number_of_transcripts_in_gene | Percent_of_transcripts_affected' ">
##INFO=<ID=NMD,Number=.,Type=String,Description="Predicted nonsense mediated decay effects for this variant. Format: 'Gene_Name | Gene_ID | Number_of_transcripts_in_gene | Percent_of_transcripts_affected' ">
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1   14397   .   CTGT    C   139.12  IndelQD AC=2;AF=0.333;AN=6;BaseQRankSum=1.800;ClippingRankSum=0.138;DP=69;FS=7.786;MLEAC=2;MLEAF=0.333;MQ=26.84;MQ0=0;MQRankSum=-1.906;QD=1.55;ReadPosRankSum=0.384;ANN=C|downstream_gene_variant|MODIFIER|WASH7P|ENSG00000227232|transcript|ENST00000488147.1|unprocessed_pseudogene||n.*4_*6delACA|||||6|,C|downstream_gene_variant|MODIFIER|MIR6859-1|ENSG00000278267|transcript|ENST00000619216.1|miRNA||n.*2969_*2971delACA|||||2971|,C|downstream_gene_variant|MODIFIER|DDX11L1|ENSG00000223972|transcript|ENST00000450305.2|transcribed_unprocessed_pseudogene||n.*728_*730delTGT|||||728|,C|non_coding_exon_variant|MODIFIER|DDX11L1|ENSG00000223972|transcript|ENST00000456328.2|processed_transcript|3/3|n.1648_1650delTTG||||||INFO_REALIGN_3_PRIME  GT:AD:DP:FT:GQ:PL   0/1:16,4:20:rd:99:120,0,827 0/1:8,2:10:dp;rd:60:60,0,414    0/0:39,0:39:PASS:99:0,116,2114
...

$ adam-submit vcf2adam small-snpeff-cmdline.vcf small-snpeff-cmdline.adam
$ adam-submit adam2vcf small-snpeff-cmdline.adam small-snpeff-cmdline-adam.vcf
$ cat small-snpeff-cmdline-adam.vcf/part-r-00000
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=.,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12891 NA12892 NA12878
1   14397   .   CTGT    C   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:8,2:10:IndelQD:60:60,0,256  0/0:39,0:39:IndelQD:99:0,116,256    0/1:16,4:20:IndelQD:99:120,0,256
1   19190   .   GC  G   .   .   .   GT:AD:DP:GQ:PL  0/1:18,13:31:99:256,0,256   0/1:5,15:20:99:256,0,107    0/1:8,14:22:99:256,0,256
1   14522   .   G   A   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34   0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256 0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256
1   63735   .   CCTA    C   .   .   .   GT:AD:DP:GQ:PL  0/0:40,0:40:99:0,117,256    0/1:23,74:97:99:256,0,256   0/0:27,0:27:79:0,79,256
1   752721  .   A   G   .   .   .   GT:AD:DP:GQ:PL  1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0 1/1:0,27:27:81:256,81,0
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 3, 2016

Member

After returning from my afternoon walk, I realize I'm not converting to/from DatabaseVariantAnnotationRDD above, either in the adam-snpeff code or via the command line. Even though I've been heads down refactoring variant annotation stuff, I never fully understood how to use what is there.

Member

heuermh commented Oct 3, 2016

After returning from my afternoon walk, I realize I'm not converting to/from DatabaseVariantAnnotationRDD above, either in the adam-snpeff code or via the command line. Even though I've been heads down refactoring variant annotation stuff, I never fully understood how to use what is there.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1525/

Build result: FAILURE

GitHub pull request #1114 of commit b5fa59c.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse b5fa59c^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b5fa59c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/head^{commit} # timeout=10Checking out Revision b5fa59c (origin/pr/1114/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b5fa59c5dcf6785c60e766de2b5bfc991dcb39aeFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1525/

Build result: FAILURE

GitHub pull request #1114 of commit b5fa59c.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse b5fa59c^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b5fa59c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/head^{commit} # timeout=10Checking out Revision b5fa59c (origin/pr/1114/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b5fa59c5dcf6785c60e766de2b5bfc991dcb39aeFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 7, 2016

Member

@heuermh

Could you add a Map parameter with values to add to the ProcessBuilder.environment?

Added in b56f448.

I've also rebased and cleaned up the conflicts. Once this is reviewed, the commits will have to be squashed down. Otherwise, this is good for merge.

Member

fnothaft commented Oct 7, 2016

@heuermh

Could you add a Map parameter with values to add to the ProcessBuilder.environment?

Added in b56f448.

I've also rebased and cleaned up the conflicts. Once this is reviewed, the commits will have to be squashed down. Otherwise, this is good for merge.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1526/

Build result: FAILURE

GitHub pull request #1114 of commit b56f448 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 538bf74fb5db1645f79caf61c34ea8ec54dd1321 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 538bf74fb5db1645f79caf61c34ea8ec54dd1321First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1526/

Build result: FAILURE

GitHub pull request #1114 of commit b56f448 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 538bf74fb5db1645f79caf61c34ea8ec54dd1321 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 538bf74fb5db1645f79caf61c34ea8ec54dd1321First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 7, 2016

Member

Jenkins, retest this please

Member

heuermh commented Oct 7, 2016

Jenkins, retest this please

@@ -174,10 +180,19 @@ trait GenomicRDD[T, U <: GenomicRDD[T, U]] {
})
}
- // split command and run
+ // split command and create proces builder

This comment has been minimized.

@heuermh

heuermh Oct 7, 2016

Member

minor typo, procesprocess

@heuermh

heuermh Oct 7, 2016

Member

minor typo, procesprocess

*/
def pipe[X, Y <: GenomicRDD[X, Y], V <: InFormatter[T, U, V]](cmd: String,
files: Seq[String] = Seq.empty,
+ environment: Map[String, String] = Map.empty,

This comment has been minimized.

@heuermh

heuermh Oct 7, 2016

Member

nice, thank you!

@heuermh

heuermh Oct 7, 2016

Member

nice, thank you!

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1527/

Build result: FAILURE

GitHub pull request #1114 of commit b56f448 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 538bf74fb5db1645f79caf61c34ea8ec54dd1321 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 538bf74fb5db1645f79caf61c34ea8ec54dd1321 > /home/jenkins/git2/bin/git rev-list 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1527/

Build result: FAILURE

GitHub pull request #1114 of commit b56f448 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 538bf74fb5db1645f79caf61c34ea8ec54dd1321 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 538bf74fb5db1645f79caf61c34ea8ec54dd1321 > /home/jenkins/git2/bin/git rev-list 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 7, 2016

Member

Fixed the +x permissions. For whatever reason, that didn't get picked up by git the first time I pushed.

Member

fnothaft commented Oct 7, 2016

Fixed the +x permissions. For whatever reason, that didn't get picked up by git the first time I pushed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1528/

Build result: FAILURE

GitHub pull request #1114 of commit 71e82bd automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 18d4144 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 18d4144 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 18d4144 > /home/jenkins/git2/bin/git rev-list 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1528/

Build result: FAILURE

GitHub pull request #1114 of commit 71e82bd automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 18d4144 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 18d4144 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 18d4144 > /home/jenkins/git2/bin/git rev-list 538bf74fb5db1645f79caf61c34ea8ec54dd1321 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

+ val records = ardd.rdd.count
+
+ implicit val tFormatter = BAMInFormatter
+ implicit val uFormatter = new AnySAMOutFormatter

This comment has been minimized.

@heuermh

heuermh Oct 7, 2016

Member

Could you explain a bit how these implicits work? And why the second one needs the new? Do they have to be implicit? Does them being implicit prevent calling the pipe method from Java?

@heuermh

heuermh Oct 7, 2016

Member

Could you explain a bit how these implicits work? And why the second one needs the new? Do they have to be implicit? Does them being implicit prevent calling the pipe method from Java?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 7, 2016

Member

I see two minor doc issues outstanding, and I've asked for clarification on the implicit in & out formatter parameters. Then I'd be ready to merge.

Member

heuermh commented Oct 7, 2016

I see two minor doc issues outstanding, and I've asked for clarification on the implicit in & out formatter parameters. Then I'd be ready to merge.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

I tried a second VCF pipe example here
https://github.com/heuermh/adam-dishevelled-bio

It fails with
object not serializable (class: org.bdgenomics.adam.models.VariantContext, value: org.bdgenomics.adam.models.VariantContext@2216cb52)

Member

heuermh commented Oct 10, 2016

I tried a second VCF pipe example here
https://github.com/heuermh/adam-dishevelled-bio

It fails with
object not serializable (class: org.bdgenomics.adam.models.VariantContext, value: org.bdgenomics.adam.models.VariantContext@2216cb52)

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 10, 2016

Member

Do you have org.bdgenomics.adam.serialization.ADAMKryoRegistrator as your Kryo serializer registrator, because if not, you will get that error since VariantContext is (for reasons known only to God) not a serializable class.

Member

fnothaft commented Oct 10, 2016

Do you have org.bdgenomics.adam.serialization.ADAMKryoRegistrator as your Kryo serializer registrator, because if not, you will get that error since VariantContext is (for reasons known only to God) not a serializable class.

@heuermh

This comment has been minimized.

Show comment
Hide comment
Member

heuermh commented Oct 10, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 10, 2016

Member

I'm extending from BDGSparkCommand, which should take care of registration for me, right?

Nyet, no dice. Kryo registrator config is done on the spark-submit command line.

Member

fnothaft commented Oct 10, 2016

I'm extending from BDGSparkCommand, which should take care of registration for me, right?

Nyet, no dice. Kryo registrator config is done on the spark-submit command line.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

Yeah just noticed I'm using spark-submit in the doc.

Member

heuermh commented Oct 10, 2016

Yeah just noticed I'm using spark-submit in the doc.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

Sorry for spamming this issue with my own WIP... hope you don't mind. input.vcf is small.vcf from the adam-core test resources.

If I run with a filter, the remaining record is getting lost when read back in

$ spark-submit     --class com.github.heuermh.adam.dishevelledbio.AdamDshFilterVcf     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer     --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator     target/adam-dishevelled-bio_2.10-0.19.1-SNAPSHOT.jar     --id rs201888535     --single     input.vcf     output.vcf

2016-10-10 10:46:23 WARN  AdamDshFilterVcf:76 - Loaded 5 variant context records
2016-10-10 10:46:24 WARN  AdamDshFilterVcf:92 - Saving 0 variant context records after filtering
$ cat input.vcf | dsh-filter-vcf --id rs201888535
##fileformat=VCFv4.1
##FILTER=<ID=IndelFS,Description="FS > 200.0">
...
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1   63735   rs201888535 CCTA    C   2994    PASS    AC=[1];MQRankSum=[0.636];MQ=[31.06];AF=[0.167];MLEAC=[1];BaseQRankSum=[1.138];MLEAF=[0.167];DP=[176];ReadPosRankSum=[-1.180];AN=[6];FS=[13.597];MQ0=[0];QD=[9.98];ClippingRankSum=[0.448]   GT:AD:DP:FT:GQ:PL   0/1:23,74:97:rd:99:3034,0,942   0/0:40,0:40:PASS:99:0,117,2120  0/0:27,0:27:PASS:79:0,79,1425

If I run without a filter, the stream being read back in can't be processed

$ spark-submit     --class com.github.heuermh.adam.dishevelledbio.AdamDshFilterVcf     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer     --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator     target/adam-dishevelled-bio_2.10-0.19.1-SNAPSHOT.jar     --single     input.vcf     output.vcf

2016-10-10 10:57:51 WARN  AdamDshFilterVcf:76 - Loaded 5 variant context records
2016-10-10 10:57:52 ERROR Executor:95 - Exception in task 0.0 in stage 2.0 (TID 2)
java.util.concurrent.ExecutionException: htsjdk.tribble.TribbleException: The provided VCF file is malformed at approximately line number 21: The VCF specification requires a valid filter status: filter was 
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at org.bdgenomics.adam.rdd.GenomicRDD$$anonfun$12.apply(GenomicRDD.scala:219)
    at org.bdgenomics.adam.rdd.GenomicRDD$$anonfun$12.apply(GenomicRDD.scala:169)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.tribble.TribbleException: The provided VCF file is malformed at approximately line number 21: The VCF specification requires a valid filter status: filter was 
    at htsjdk.variant.vcf.AbstractVCFCodec.generateException(AbstractVCFCodec.java:783)
    at htsjdk.variant.vcf.VCFCodec.parseFilters(VCFCodec.java:138)
    at htsjdk.variant.vcf.AbstractVCFCodec.parseVCFLine(AbstractVCFCodec.java:319)
    at htsjdk.variant.vcf.AbstractVCFCodec.decodeLine(AbstractVCFCodec.java:279)
    at htsjdk.variant.vcf.AbstractVCFCodec.decode(AbstractVCFCodec.java:257)
    at org.bdgenomics.adam.rdd.variation.VCFOutFormatter.convertIterator$1(VCFOutFormatter.scala:60)
    at org.bdgenomics.adam.rdd.variation.VCFOutFormatter.read(VCFOutFormatter.scala:66)
    at org.bdgenomics.adam.rdd.OutFormatterRunner.call(OutFormatter.scala:27)
    at org.bdgenomics.adam.rdd.OutFormatterRunner.call(OutFormatter.scala:23)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
$ cat input.vcf | dsh-filter-vcf
##fileformat=VCFv4.1
##FILTER=<ID=IndelFS,Description="FS > 200.0">
...
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1   14397   .   CTGT    C   139 IndelQD AC=[2];MQRankSum=[-1.906];MQ=[26.84];AF=[0.333];MLEAC=[2];BaseQRankSum=[1.800];MLEAF=[0.333];DP=[69];ReadPosRankSum=[0.384];AN=[6];FS=[7.786];MQ0=[0];QD=[1.55];ClippingRankSum=[0.138] GT:AD:DP:FT:GQ:PL   0/0:39,0:39:PASS:99:0,116,2114  0/1:8,2:10:dp;rd:60:60,0,414    0/1:16,4:20:rd:99:120,0,827
1   14522   .   G   A   195 VQSRTrancheSNP99.95to100.00 AC=[2];MQRankSum=[-0.063];MQ=[25.89];AF=[0.333];MLEAC=[2];BaseQRankSum=[2.044];MLEAF=[0.333];DP=[48];ReadPosRankSum=[0.952];AN=[6];FS=[13.179];MQ0=[0];culprit=[MQ];QD=[8.87];VQSLOD=[-3.333e+00];ClippingRankSum=[-2.196]  GT:AD:DP:FT:GQ:PL   0/0:26,0:26:PASS:78:0,78,783    0/1:2,5:7:dp;rd:34:128,0,34 0/1:10,5:15:dp:99:99,0,233
1   19190   .   GC  G   1186    PASS    AC=[3];MQRankSum=[0.195];MQ=[22.26];AF=[0.500];MLEAC=[3];BaseQRankSum=[4.157];MLEAF=[0.500];DP=[74];ReadPosRankSum=[-4.072];AN=[6];FS=[37.037];MQ0=[0];QD=[16.04];ClippingRankSum=[3.666]   GT:AD:DP:FT:GQ:PL   0/1:5,15:20:rd:99:457,0,107 0/1:18,13:31:PASS:99:353,0,503  0/1:8,14:22:PASS:99:416,0,201
1   63735   rs201888535 CCTA    C   2994    PASS    AC=[1];MQRankSum=[0.636];MQ=[31.06];AF=[0.167];MLEAC=[1];BaseQRankSum=[1.138];MLEAF=[0.167];DP=[176];ReadPosRankSum=[-1.180];AN=[6];FS=[13.597];MQ0=[0];QD=[9.98];ClippingRankSum=[0.448]   GT:AD:DP:FT:GQ:PL   0/1:23,74:97:rd:99:3034,0,942   0/0:40,0:40:PASS:99:0,117,2120  0/0:27,0:27:PASS:79:0,79,1425
1   752721  rs3131972   A   G   2486    PASS    AC=[6];MQ=[60.00];AF=[1.00];MLEAC=[6];MLEAF=[1.00];DP=[69];AN=[6];FS=[0.000];MQ0=[0];culprit=[QD];QD=[31.67];VQSLOD=[18.94] GT:AD:DP:FT:GQ:PL   1/1:0,22:22:PASS:66:831,66,0    1/1:0,19:19:dp:57:661,57,0  1/1:0,27:27:PASS:81:1021,81,0

I wonder if maybe I can tee the input pipe to dsh-filter-vcf to see if it is malformed?

Member

heuermh commented Oct 10, 2016

Sorry for spamming this issue with my own WIP... hope you don't mind. input.vcf is small.vcf from the adam-core test resources.

If I run with a filter, the remaining record is getting lost when read back in

$ spark-submit     --class com.github.heuermh.adam.dishevelledbio.AdamDshFilterVcf     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer     --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator     target/adam-dishevelled-bio_2.10-0.19.1-SNAPSHOT.jar     --id rs201888535     --single     input.vcf     output.vcf

2016-10-10 10:46:23 WARN  AdamDshFilterVcf:76 - Loaded 5 variant context records
2016-10-10 10:46:24 WARN  AdamDshFilterVcf:92 - Saving 0 variant context records after filtering
$ cat input.vcf | dsh-filter-vcf --id rs201888535
##fileformat=VCFv4.1
##FILTER=<ID=IndelFS,Description="FS > 200.0">
...
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1   63735   rs201888535 CCTA    C   2994    PASS    AC=[1];MQRankSum=[0.636];MQ=[31.06];AF=[0.167];MLEAC=[1];BaseQRankSum=[1.138];MLEAF=[0.167];DP=[176];ReadPosRankSum=[-1.180];AN=[6];FS=[13.597];MQ0=[0];QD=[9.98];ClippingRankSum=[0.448]   GT:AD:DP:FT:GQ:PL   0/1:23,74:97:rd:99:3034,0,942   0/0:40,0:40:PASS:99:0,117,2120  0/0:27,0:27:PASS:79:0,79,1425

If I run without a filter, the stream being read back in can't be processed

$ spark-submit     --class com.github.heuermh.adam.dishevelledbio.AdamDshFilterVcf     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer     --conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator     target/adam-dishevelled-bio_2.10-0.19.1-SNAPSHOT.jar     --single     input.vcf     output.vcf

2016-10-10 10:57:51 WARN  AdamDshFilterVcf:76 - Loaded 5 variant context records
2016-10-10 10:57:52 ERROR Executor:95 - Exception in task 0.0 in stage 2.0 (TID 2)
java.util.concurrent.ExecutionException: htsjdk.tribble.TribbleException: The provided VCF file is malformed at approximately line number 21: The VCF specification requires a valid filter status: filter was 
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at org.bdgenomics.adam.rdd.GenomicRDD$$anonfun$12.apply(GenomicRDD.scala:219)
    at org.bdgenomics.adam.rdd.GenomicRDD$$anonfun$12.apply(GenomicRDD.scala:169)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.tribble.TribbleException: The provided VCF file is malformed at approximately line number 21: The VCF specification requires a valid filter status: filter was 
    at htsjdk.variant.vcf.AbstractVCFCodec.generateException(AbstractVCFCodec.java:783)
    at htsjdk.variant.vcf.VCFCodec.parseFilters(VCFCodec.java:138)
    at htsjdk.variant.vcf.AbstractVCFCodec.parseVCFLine(AbstractVCFCodec.java:319)
    at htsjdk.variant.vcf.AbstractVCFCodec.decodeLine(AbstractVCFCodec.java:279)
    at htsjdk.variant.vcf.AbstractVCFCodec.decode(AbstractVCFCodec.java:257)
    at org.bdgenomics.adam.rdd.variation.VCFOutFormatter.convertIterator$1(VCFOutFormatter.scala:60)
    at org.bdgenomics.adam.rdd.variation.VCFOutFormatter.read(VCFOutFormatter.scala:66)
    at org.bdgenomics.adam.rdd.OutFormatterRunner.call(OutFormatter.scala:27)
    at org.bdgenomics.adam.rdd.OutFormatterRunner.call(OutFormatter.scala:23)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
$ cat input.vcf | dsh-filter-vcf
##fileformat=VCFv4.1
##FILTER=<ID=IndelFS,Description="FS > 200.0">
...
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1   14397   .   CTGT    C   139 IndelQD AC=[2];MQRankSum=[-1.906];MQ=[26.84];AF=[0.333];MLEAC=[2];BaseQRankSum=[1.800];MLEAF=[0.333];DP=[69];ReadPosRankSum=[0.384];AN=[6];FS=[7.786];MQ0=[0];QD=[1.55];ClippingRankSum=[0.138] GT:AD:DP:FT:GQ:PL   0/0:39,0:39:PASS:99:0,116,2114  0/1:8,2:10:dp;rd:60:60,0,414    0/1:16,4:20:rd:99:120,0,827
1   14522   .   G   A   195 VQSRTrancheSNP99.95to100.00 AC=[2];MQRankSum=[-0.063];MQ=[25.89];AF=[0.333];MLEAC=[2];BaseQRankSum=[2.044];MLEAF=[0.333];DP=[48];ReadPosRankSum=[0.952];AN=[6];FS=[13.179];MQ0=[0];culprit=[MQ];QD=[8.87];VQSLOD=[-3.333e+00];ClippingRankSum=[-2.196]  GT:AD:DP:FT:GQ:PL   0/0:26,0:26:PASS:78:0,78,783    0/1:2,5:7:dp;rd:34:128,0,34 0/1:10,5:15:dp:99:99,0,233
1   19190   .   GC  G   1186    PASS    AC=[3];MQRankSum=[0.195];MQ=[22.26];AF=[0.500];MLEAC=[3];BaseQRankSum=[4.157];MLEAF=[0.500];DP=[74];ReadPosRankSum=[-4.072];AN=[6];FS=[37.037];MQ0=[0];QD=[16.04];ClippingRankSum=[3.666]   GT:AD:DP:FT:GQ:PL   0/1:5,15:20:rd:99:457,0,107 0/1:18,13:31:PASS:99:353,0,503  0/1:8,14:22:PASS:99:416,0,201
1   63735   rs201888535 CCTA    C   2994    PASS    AC=[1];MQRankSum=[0.636];MQ=[31.06];AF=[0.167];MLEAC=[1];BaseQRankSum=[1.138];MLEAF=[0.167];DP=[176];ReadPosRankSum=[-1.180];AN=[6];FS=[13.597];MQ0=[0];QD=[9.98];ClippingRankSum=[0.448]   GT:AD:DP:FT:GQ:PL   0/1:23,74:97:rd:99:3034,0,942   0/0:40,0:40:PASS:99:0,117,2120  0/0:27,0:27:PASS:79:0,79,1425
1   752721  rs3131972   A   G   2486    PASS    AC=[6];MQ=[60.00];AF=[1.00];MLEAC=[6];MLEAF=[1.00];DP=[69];AN=[6];FS=[0.000];MQ0=[0];culprit=[QD];QD=[31.67];VQSLOD=[18.94] GT:AD:DP:FT:GQ:PL   1/1:0,22:22:PASS:66:831,66,0    1/1:0,19:19:dp:57:661,57,0  1/1:0,27:27:PASS:81:1021,81,0

I wonder if maybe I can tee the input pipe to dsh-filter-vcf to see if it is malformed?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 10, 2016

Member

I wonder if maybe I can tee the input pipe to dsh-filter-vcf to see if it is malformed?

Yeah, I'd probably tee both the input and the output.

Also, I'd just try to load the file you get from cat input.vcf | dsh-filter-vcf with Vcf2ADAM. If that reproduces the TribbleException, then that narrows down the problem.

Member

fnothaft commented Oct 10, 2016

I wonder if maybe I can tee the input pipe to dsh-filter-vcf to see if it is malformed?

Yeah, I'd probably tee both the input and the output.

Also, I'd just try to load the file you get from cat input.vcf | dsh-filter-vcf with Vcf2ADAM. If that reproduces the TribbleException, then that narrows down the problem.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

I fixed a problem formatting INFO values (note []s in output above) that would choke vcf2adam.

Will have to try the tee next, I think adam VCF output might be breaking one of the filter values.

Member

heuermh commented Oct 10, 2016

I fixed a problem formatting INFO values (note []s in output above) that would choke vcf2adam.

Will have to try the tee next, I think adam VCF output might be breaking one of the filter values.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 10, 2016

Member

Will have to try the tee next, I think adam VCF output might be breaking one of the filter values.

We're using HTSJDK which—like it or not—is the reference Java VCF imp'l, so... I don't think we're breaking the filter value. :P ;)

Member

fnothaft commented Oct 10, 2016

Will have to try the tee next, I think adam VCF output might be breaking one of the filter values.

We're using HTSJDK which—like it or not—is the reference Java VCF imp'l, so... I don't think we're breaking the filter value. :P ;)

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

It may be the reference implementation, but it does not follow the specification. :)

Member

heuermh commented Oct 10, 2016

It may be the reference implementation, but it does not follow the specification. :)

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

The tees don't seem to work as part of the command given to pipe, or at least I can't figure it out. Perhaps I need to create a bash script and do the tee'ing there?

Member

heuermh commented Oct 10, 2016

The tees don't seem to work as part of the command given to pipe, or at least I can't figure it out. Perhaps I need to create a bash script and do the tee'ing there?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

That worked. The filter header lines and values are missing

$ cat tee-out.vcf
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1       14397   .       CTGT    C       .               .       GT:AD:DP:FT:GQ:PL       0/0:39,0:39:IndelQD:99:0,116,256        0/1:8,2:10:IndelQD:60:60,0,256  0/1:16,4:20:IndelQD:99:120,0,256
1       14522   .       G       A       .               .       GT:AD:DP:FT:GQ:PL       0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256     0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34       0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256
1       19190   .       GC      G       .               .       GT:AD:DP:GQ:PL  0/1:5,15:20:99:256,0,107        0/1:18,13:31:99:256,0,256       0/1:8,14:22:99:256,0,256
1       63735   .       CCTA    C       .               .       GT:AD:DP:GQ:PL  0/1:23,74:97:99:256,0,256       0/0:40,0:40:99:0,117,256        0/0:27,0:27:79:0,79,256
1       752721  .       A       G       .               .       GT:AD:DP:GQ:PL  1/1:0,22:22:66:256,66,0 1/1:0,19:19:57:256,57,0 1/1:0,27:27:81:256,81,0

Similar happens with vcf2adam and adam2vcf round trip, the filter headers go missing, and it appears that some of the filter values are mistakenly moved to genotype filters.

$ adam-submit vcf2adam input.vcf test.adam

$ adam-submit adam2vcf -single test.adam test.vcf

$ cat test.vcf 
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1   14397   .   CTGT    C   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:16,4:20:IndelQD:99:120,0,256    0/1:8,2:10:IndelQD:60:60,0,256  0/0:39,0:39:IndelQD:99:0,116,256
1   14522   .   G   A   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256 0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34   0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256
1   63735   .   CCTA    C   .   .   .   GT:AD:DP:GQ:PL  0/0:27,0:27:79:0,79,256 0/0:40,0:40:99:0,117,256    0/1:23,74:97:99:256,0,256
1   752721  .   A   G   .   .   .   GT:AD:DP:GQ:PL  1/1:0,27:27:81:256,81,0 1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0
1   19190   .   GC  G   .   .   .   GT:AD:DP:GQ:PL  0/1:8,14:22:99:256,0,256    0/1:18,13:31:99:256,0,256   0/1:5,15:20:99:256,0,107
Member

heuermh commented Oct 10, 2016

That worked. The filter header lines and values are missing

$ cat tee-out.vcf
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  NA12892 NA12891 NA12878
1       14397   .       CTGT    C       .               .       GT:AD:DP:FT:GQ:PL       0/0:39,0:39:IndelQD:99:0,116,256        0/1:8,2:10:IndelQD:60:60,0,256  0/1:16,4:20:IndelQD:99:120,0,256
1       14522   .       G       A       .               .       GT:AD:DP:FT:GQ:PL       0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256     0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34       0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256
1       19190   .       GC      G       .               .       GT:AD:DP:GQ:PL  0/1:5,15:20:99:256,0,107        0/1:18,13:31:99:256,0,256       0/1:8,14:22:99:256,0,256
1       63735   .       CCTA    C       .               .       GT:AD:DP:GQ:PL  0/1:23,74:97:99:256,0,256       0/0:40,0:40:99:0,117,256        0/0:27,0:27:79:0,79,256
1       752721  .       A       G       .               .       GT:AD:DP:GQ:PL  1/1:0,22:22:66:256,66,0 1/1:0,19:19:57:256,57,0 1/1:0,27:27:81:256,81,0

Similar happens with vcf2adam and adam2vcf round trip, the filter headers go missing, and it appears that some of the filter values are mistakenly moved to genotype filters.

$ adam-submit vcf2adam input.vcf test.adam

$ adam-submit adam2vcf -single test.adam test.vcf

$ cat test.vcf 
##fileformat=VCFv4.2
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles in the order listed">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Approximate read depth (reads with MQ=255 or with bad mates are filtered)">
##FORMAT=<ID=FT,Number=1,Type=String,Description="Genotype-level filter">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=MIN_DP,Number=1,Type=Integer,Description="Minimum DP observed within the GVCF block">
##FORMAT=<ID=PL,Number=G,Type=Integer,Description="Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification">
##FORMAT=<ID=PQ,Number=1,Type=Float,Description="Read-backed phasing quality">
##FORMAT=<ID=PS,Number=1,Type=Integer,Description="Phase set">
##FORMAT=<ID=SB,Number=4,Type=Integer,Description="Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.">
##INFO=<ID=FS,Number=1,Type=Float,Description="Phred-scaled p-value using Fisher's exact test to detect strand bias">
##INFO=<ID=MQ,Number=1,Type=Float,Description="RMS Mapping Quality">
##INFO=<ID=MQ0,Number=1,Type=Integer,Description="Total Mapping Quality Zero Reads">
##INFO=<ID=MQRankSum,Number=1,Type=Float,Description="Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities">
##INFO=<ID=ReadPosRankSum,Number=1,Type=Float,Description="Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias">
##INFO=<ID=VQSLOD,Number=1,Type=Float,Description="Log odds ratio of being a true variant versus being false under the trained gaussian mixture model">
##INFO=<ID=culprit,Number=1,Type=String,Description="The annotation which was the worst performing in the Gaussian mixture model, likely the reason why the variant was filtered out">
##contig=<ID=1,length=249250621>
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NA12878 NA12891 NA12892
1   14397   .   CTGT    C   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:16,4:20:IndelQD:99:120,0,256    0/1:8,2:10:IndelQD:60:60,0,256  0/0:39,0:39:IndelQD:99:0,116,256
1   14522   .   G   A   .   .   .   GT:AD:DP:FT:GQ:PL   0/1:10,5:15:VQSRTrancheSNP99.95to100.00:99:99,0,256 0/1:2,5:7:VQSRTrancheSNP99.95to100.00:34:128,0,34   0/0:26,0:26:VQSRTrancheSNP99.95to100.00:78:0,78,256
1   63735   .   CCTA    C   .   .   .   GT:AD:DP:GQ:PL  0/0:27,0:27:79:0,79,256 0/0:40,0:40:99:0,117,256    0/1:23,74:97:99:256,0,256
1   752721  .   A   G   .   .   .   GT:AD:DP:GQ:PL  1/1:0,27:27:81:256,81,0 1/1:0,19:19:57:256,57,0 1/1:0,22:22:66:256,66,0
1   19190   .   GC  G   .   .   .   GT:AD:DP:GQ:PL  0/1:8,14:22:99:256,0,256    0/1:18,13:31:99:256,0,256   0/1:5,15:20:99:256,0,107
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 10, 2016

Member

Perhaps I need to create a bash script and do the tee'ing there?

Correct, ProcessBuilder doesn't run the processes in a shell.

It may be the reference implementation, but it does not follow the specification. :)

Preaching to the choir, but file the bugs upstream.

The filter header lines and values are missing

They're not in SupportedHeaderLines; we need to find a better way to deal with that, but I'm personally not sure what that is. Perhaps some sort of HeaderLineRegistrator?

Member

fnothaft commented Oct 10, 2016

Perhaps I need to create a bash script and do the tee'ing there?

Correct, ProcessBuilder doesn't run the processes in a shell.

It may be the reference implementation, but it does not follow the specification. :)

Preaching to the choir, but file the bugs upstream.

The filter header lines and values are missing

They're not in SupportedHeaderLines; we need to find a better way to deal with that, but I'm personally not sure what that is. Perhaps some sort of HeaderLineRegistrator?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

They're not in SupportedHeaderLines; we need to find a better way to deal with that, but I'm personally not sure what that is. Perhaps some sort of HeaderLineRegistrator?

A clue to why it's taking so long for #1144 . . . The present code is all sorts of not quite correct.

Member

heuermh commented Oct 10, 2016

They're not in SupportedHeaderLines; we need to find a better way to deal with that, but I'm personally not sure what that is. Perhaps some sort of HeaderLineRegistrator?

A clue to why it's taking so long for #1144 . . . The present code is all sorts of not quite correct.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 10, 2016

Member

Well, I've tried twice to use the pipe API for variants and failed, though not because of the code here in this pull request.

My comments above in #1114 (comment) still hold.

Member

heuermh commented Oct 10, 2016

Well, I've tried twice to use the pipe API for variants and failed, though not because of the code here in this pull request.

My comments above in #1114 (comment) still hold.

@heuermh heuermh modified the milestone: 0.20.0 Oct 13, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 13, 2016

Member

Rebased. @heuermh I added commentary about the In/Out formatters to the Scaladoc for the pipe method. I believe the two doc comments you had made were addressed earlier; at the least, I don't see them in my code now. I think GitHub f00bar00ed the code review tool when they introduced the latest round of changes; I've seen this issue on several PRs that I've reviewed...

Member

fnothaft commented Oct 13, 2016

Rebased. @heuermh I added commentary about the In/Out formatters to the Scaladoc for the pipe method. I believe the two doc comments you had made were addressed earlier; at the least, I don't see them in my code now. I think GitHub f00bar00ed the code review tool when they introduced the latest round of changes; I've seen this issue on several PRs that I've reviewed...

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 13, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1535/

Build result: FAILURE

GitHub pull request #1114 of commit 2c19433 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list 18d4144 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1535/

Build result: FAILURE

GitHub pull request #1114 of commit 2c19433 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list 18d4144 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 13, 2016

Member

Jenkins, retest this please.

Member

fnothaft commented Oct 13, 2016

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 13, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1536/

Build result: FAILURE

[...truncated 2 lines...][EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbCloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1536/

Build result: FAILURE

[...truncated 2 lines...][EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbCloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

Jenkins, test this please.

Member

fnothaft commented Oct 14, 2016

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1537/

Build result: FAILURE

[...truncated 2 lines...][EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Cleaning workspace > /home/jenkins/git2/bin/git rev-parse --verify HEAD # timeout=10Resetting working tree > /home/jenkins/git2/bin/git reset --hard # timeout=10 > /home/jenkins/git2/bin/git clean -fdx # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1537/

Build result: FAILURE

[...truncated 2 lines...][EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Cleaning workspace > /home/jenkins/git2/bin/git rev-parse --verify HEAD # timeout=10Resetting working tree > /home/jenkins/git2/bin/git reset --hard # timeout=10 > /home/jenkins/git2/bin/git clean -fdx # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

Jenkins, test this please.

Member

fnothaft commented Oct 14, 2016

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1538/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1538/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d56b76d19f400f94f9018ef16642d0e1a8231d4c (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d56b76d19f400f94f9018ef16642d0e1a8231d4c > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1539/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision aaf9dd15e44c675dd1b02b257756710263ac0f77 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f aaf9dd15e44c675dd1b02b257756710263ac0f77 > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1539/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision aaf9dd15e44c675dd1b02b257756710263ac0f77 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f aaf9dd15e44c675dd1b02b257756710263ac0f77 > /home/jenkins/git2/bin/git rev-list d56b76d19f400f94f9018ef16642d0e1a8231d4c # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

Jenkins, test this please.

Member

fnothaft commented Oct 14, 2016

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1540/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision aaf9dd15e44c675dd1b02b257756710263ac0f77 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f aaf9dd15e44c675dd1b02b257756710263ac0f77 > /home/jenkins/git2/bin/git rev-list aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1540/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision aaf9dd15e44c675dd1b02b257756710263ac0f77 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f aaf9dd15e44c675dd1b02b257756710263ac0f77 > /home/jenkins/git2/bin/git rev-list aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1541/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 > /home/jenkins/git2/bin/git rev-list aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1541/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 > /home/jenkins/git2/bin/git rev-list aaf9dd15e44c675dd1b02b257756710263ac0f77 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

This is killing me!

[issues/1112-pipes-api 5b3b262] Adding back test script with +x.
 1 file changed, 7 insertions(+)
 create mode 100755 adam-core/src/test/resources/env_test_command.sh

But, no +x on Jenkins! Argh!

Member

fnothaft commented Oct 14, 2016

This is killing me!

[issues/1112-pipes-api 5b3b262] Adding back test script with +x.
 1 file changed, 7 insertions(+)
 create mode 100755 adam-core/src/test/resources/env_test_command.sh

But, no +x on Jenkins! Argh!

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

Gah! Looks OK locally...

adam-upstream fnothaft$ mkdir adam-clean-clone
adam-upstream fnothaft$ cd adam-clean-clone/
adam-clean-clone fnothaft$ git clone git@github.com:bigdatagenomics/adam.git
Cloning into 'adam'...
remote: Counting objects: 20155, done.
remote: Compressing objects: 100% (69/69), done.
Receiving objects:  26% (5241/20155), 2.25 MiB | 1.41 MiB/s   MiB/s   
remote: Total 20155 (delta 23), reused 3 (delta 3), pack-reused 20061
Receiving objects: 100% (20155/20155), 5.69 MiB | 1.39 MiB/s, done.
Resolving deltas: 100% (9320/9320), done.
Checking connectivity... done.
adam-clean-clone fnothaft$ cd adam/
adam fnothaft$ git checkout -b fnothaft-issues/1112-pipes-api master
Switched to a new branch 'fnothaft-issues/1112-pipes-api'
adam fnothaft$ git pull https://github.com/fnothaft/adam.git issues/1112-pipes-api
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 80 (delta 23), reused 12 (delta 12), pack-reused 14
Unpacking objects: 100% (80/80), done.
From https://github.com/fnothaft/adam
 * branch            issues/1112-pipes-api -> FETCH_HEAD
Updating df00eff..5b3b262
Fast-forward

<snip />

 create mode 100755 adam-core/src/test/resources/env_test_command.sh
 create mode 100755 adam-core/src/test/resources/fastq_to_usam.py
 create mode 100644 adam-core/src/test/resources/small.1.sam
 create mode 100755 adam-core/src/test/resources/test_command.sh
 create mode 100644 adam-core/src/test/scala/org/bdgenomics/adam/rdd/GenomicRDDSuite.scala
 create mode 100644 adam-core/src/test/scala/org/bdgenomics/adam/rdd/fragment/FragmentRDDSuite.scala
adam fnothaft$ ls -altr adam-core/src/test/resources/env_test_command.sh
-rwxr-xr-x+ 1 fnothaft  staff  133 Oct 13 19:43 adam-core/src/test/resources/env_test_command.sh
adam fnothaft$ 
Member

fnothaft commented Oct 14, 2016

Gah! Looks OK locally...

adam-upstream fnothaft$ mkdir adam-clean-clone
adam-upstream fnothaft$ cd adam-clean-clone/
adam-clean-clone fnothaft$ git clone git@github.com:bigdatagenomics/adam.git
Cloning into 'adam'...
remote: Counting objects: 20155, done.
remote: Compressing objects: 100% (69/69), done.
Receiving objects:  26% (5241/20155), 2.25 MiB | 1.41 MiB/s   MiB/s   
remote: Total 20155 (delta 23), reused 3 (delta 3), pack-reused 20061
Receiving objects: 100% (20155/20155), 5.69 MiB | 1.39 MiB/s, done.
Resolving deltas: 100% (9320/9320), done.
Checking connectivity... done.
adam-clean-clone fnothaft$ cd adam/
adam fnothaft$ git checkout -b fnothaft-issues/1112-pipes-api master
Switched to a new branch 'fnothaft-issues/1112-pipes-api'
adam fnothaft$ git pull https://github.com/fnothaft/adam.git issues/1112-pipes-api
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 80 (delta 23), reused 12 (delta 12), pack-reused 14
Unpacking objects: 100% (80/80), done.
From https://github.com/fnothaft/adam
 * branch            issues/1112-pipes-api -> FETCH_HEAD
Updating df00eff..5b3b262
Fast-forward

<snip />

 create mode 100755 adam-core/src/test/resources/env_test_command.sh
 create mode 100755 adam-core/src/test/resources/fastq_to_usam.py
 create mode 100644 adam-core/src/test/resources/small.1.sam
 create mode 100755 adam-core/src/test/resources/test_command.sh
 create mode 100644 adam-core/src/test/scala/org/bdgenomics/adam/rdd/GenomicRDDSuite.scala
 create mode 100644 adam-core/src/test/scala/org/bdgenomics/adam/rdd/fragment/FragmentRDDSuite.scala
adam fnothaft$ ls -altr adam-core/src/test/resources/env_test_command.sh
-rwxr-xr-x+ 1 fnothaft  staff  133 Oct 13 19:43 adam-core/src/test/resources/env_test_command.sh
adam fnothaft$ 
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1542/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 4f62389 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 4f62389 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 4f62389 > /home/jenkins/git2/bin/git rev-list d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1542/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 4f62389 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 4f62389 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 4f62389 > /home/jenkins/git2/bin/git rev-list d50837bf208059cc3ac3c0e2b51bd4a18c891ac8 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

Jenkins, retest this please.

Member

fnothaft commented Oct 14, 2016

Jenkins, retest this please.

[ADAM-1112] Add interface for piping commands.
Resolves #1112. Adds interfaces for piping SAM/BAM and VCF to subprocesses that
are run in parallel under Apache Spark.
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1543/

Build result: ABORTED

[...truncated 5 lines...]Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 4f62389 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 4f62389 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 4f62389 > /home/jenkins/git2/bin/git rev-list 4f62389 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosAbortedAborted by fnothaftInterrupting #1543Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1543/

Build result: ABORTED

[...truncated 5 lines...]Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1114/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 4f62389 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1114/merge^{commit} # timeout=10Checking out Revision 4f62389 (origin/pr/1114/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 4f62389 > /home/jenkins/git2/bin/git rev-list 4f62389 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosAbortedAborted by fnothaftInterrupting #1543Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

OK, appears that was passing, so I've aborted the build and squashed on down.

Member

fnothaft commented Oct 14, 2016

OK, appears that was passing, so I've aborted the build and squashed on down.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 14, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1544/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1544/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

This is good to go. Ping @heuermh for merge.

Member

fnothaft commented Oct 14, 2016

This is good to go. Ping @heuermh for merge.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 14, 2016

Member

This is good to go. Ping @heuermh for merge.

Member

fnothaft commented Oct 14, 2016

This is good to go. Ping @heuermh for merge.

@heuermh heuermh merged commit babf5cf into bigdatagenomics:master Oct 14, 2016

1 check passed

default Merged build finished.
Details
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 14, 2016

Member

Thank you, @fnothaft!

Member

heuermh commented Oct 14, 2016

Thank you, @fnothaft!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment