[CANNOLI-33] Use ADAM tab5 formatter for bowtie #42

heuermh · 2017-05-17T13:13:05Z

Fixes #33
Depends on bigdatagenomics/adam#1522

No reads are processed however

$ ADAM_MAIN=org.bdgenomics.cannoli.Cannoli \
  adam-submit \
  --jars target/cannoli-spark2_2.11-0.1-SNAPSHOT.jar \
  -- \
  bowtie \
  -bowtie_index GCA_000001405.25_GRCh38.p10_genomic \
  -print_metrics -single \
  NIST7035_TAAGGCGA_L002.ifq \
  NIST7035_TAAGGCGA_L002.sam

Using ADAM_MAIN=org.bdgenomics.cannoli.Cannoli
Using SPARK_SUBMIT=/usr/local/bin/spark-submit
...
# reads processed: 0
# reads with at least one reported alignment: 0 (0.00%)
# reads that failed to align: 0 (0.00%)
No alignments
# reads processed: 0
# reads with at least one reported alignment: 0 (0.00%)
# reads that failed to align: 0 (0.00%)
No alignments

Unless I'm doing something wrong, --12 - doesn't seem to work

$ cat fragments.tab5 
H06HDADXX130110:2:2116:3345:91806	GTTAGGGTTAGGGTTGGGTTAGGGTTAGGGTTAGGGTTAGGGGTAGGGTTAGGGTTAGGGGTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGGTAGGGCTAGGGTTAAGGGTAGGGTTAGCGAAAGGGCTGGGGTTAGGGGTGCGGGTACGCGTAGCATTAGGGCTAGAAGTAGGATCTGCAGTGCCTGACCGCGTCTGCGCGGCGACTGCCCAAAGCCTGGGGCCGACTCCAGGCTGAAGCTCAT	>=<=???>?>???=??>>8<?>=2=<===1194<?;:?>>?#3==>###########################################################################################################################################################################################################	TAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTACCCCTAACCCTAACCCTAACCCTAACCCGTACCCTAAACCCAACCCTAACCACAAAGCAAATCCCAACCTTAACCGGAACCCGAAATCTCGCAGCAAATCTGCAGTAGAGACGCAGACTCAACCATGCGTCTATTAGTACGCATTATCATTGCCTCATGCTTCTTAAGTACAGAGAGATGAC	==;<?>@@@<>>@??<>>???<=>>?>:><@?4=:>7=5=>:=@;'@A?########################################################################################################################################################################################################
...

$ bowtie -S GCA_000001405.25_GRCh38.p10_genomic --12 fragments.tab5 
@HD	VN:1.0	SO:unsorted
@SQ	SN:CM000663.2	LN:248956422
...
@PG	ID:Bowtie	VN:1.2	CL:"bowtie-align --wrapper basic-0 -S GCA_000001405.25_GRCh38.p10_genomic --12 fragments.tab5"
H06HDADXX130110:2:2116:3345:91806	77	*	0	0	*	*	0	0	GTTAGGGTTAGGGTTGGGTTAGGGTTAGGGTTAGGGTTAGGGGTAGGGTTAGGGTTAGGGGTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGGTAGGGCTAGGGTTAAGGGTAGGGTTAGCGAAAGGGCTGGGGTTAGGGGTGCGGGTACGCGTAGCATTAGGGCTAGAAGTAGGATCTGCAGTGCCTGACCGCGTCTGCGCGGCGACTGCCCAAAGCCTGGGGCCGACTCCAGGCTGAAGCTCAT	>=<=???>?>???=??>>8<?><=2=<===1194<?;:?>>?#3==>###########################################################################################################################################################################################################	XM:i:0
H06HDADXX130110:2:2116:3345:91806	141	*	0	0	*	*	0	0	TAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTACCCCTAACCCTAACCCTAACCCTAACCCGTACCCTAAACCCAACCCTAACCACAAAGCAAATCCCAACCTTAACCGGAACCCGAAATCTCGCAGCAAATCTGCAGTAGAGACGCAGACTCAACCATGCGTCTATTAGTACGCATTATCATTGCCTCATGCTTCTTAAGTACAGAGAGATGAC	==;<?>@@@<>>@??<>>???<=>>?>:><@?4=:>7=5=>:<=@;'@A?########################################################################################################################################################################################################	XM:i:0
...

$ cat fragments.tab5 | bowtie -S GCA_000001405.25_GRCh38.p10_genomic --12 -
@HD	VN:1.0	SO:unsorted
@SQ	SN:CM000663.2	LN:248956422
...
@PG	ID:Bowtie	VN:1.2	CL:"bowtie-align --wrapper basic-0 -S GCA_000001405.25_GRCh38.p10_genomic --12 -"
# reads processed: 0
# reads with at least one reported alignment: 0 (0.00%)
# reads that failed to align: 0 (0.00%)
No alignments

AmplabJenkins · 2017-05-17T16:59:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/25/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/42/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ee6e51f # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/42/merge^{commit} # timeout=10Checking out Revision ee6e51f (origin/pr/42/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ee6e51fd572cb3622c51a62971e5789e6c95ae48First time build. Skipping changelog.Triggering cannoli-prb ? 2.6.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.6.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.10,2.0.0,centoscannoli-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREcannoli-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2017-06-20T17:08:11Z

Coverage increased (+33.0%) to 62.617% when pulling 236d570 on heuermh:bowtie-tab5 into d5e9577 on bigdatagenomics:master.

AmplabJenkins · 2017-06-20T17:09:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/26/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/cannoli.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/42/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains a65c216 # timeout=10Checking out Revision a65c216 (origin/pr/42/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a65c2168f6c9440ac4715021392bb171b37994d1First time build. Skipping changelog.Triggering cannoli-prb ? 2.6.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.6.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.11,2.0.0,centoscannoli-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESScannoli-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESScannoli-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-06-22T14:43:30Z

Jenkins, retest this please

coveralls · 2017-06-22T14:48:08Z

Coverage decreased (-0.8%) to 28.879% when pulling 236d570 on heuermh:bowtie-tab5 into d5e9577 on bigdatagenomics:master.

coveralls · 2017-06-22T14:48:09Z

Coverage decreased (-0.8%) to 28.879% when pulling 236d570 on heuermh:bowtie-tab5 into d5e9577 on bigdatagenomics:master.

AmplabJenkins · 2017-06-22T14:48:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/27/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/cannoli.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/42/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains a65c216 # timeout=10Checking out Revision a65c216 (origin/pr/42/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a65c2168f6c9440ac4715021392bb171b37994d1First time build. Skipping changelog.Triggering cannoli-prb ? 2.3.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.10,2.0.0,centosTriggering cannoli-prb ? 2.6.0,2.11,2.0.0,centosTriggering cannoli-prb ? 2.3.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.3.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.11,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.10,1.6.1,centosTriggering cannoli-prb ? 2.6.0,2.10,2.0.0,centoscannoli-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESScannoli-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESScannoli-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESScannoli-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREcannoli-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

devin-petersohn

Looks good, just a couple of comments.

devin-petersohn · 2017-06-22T21:08:00Z

scripts/publish_scaladoc.sh

@@ -0,0 +1 @@
+#!/bin/bash


Is this the extent of the script?

This is intentionally a stub for later.

devin-petersohn · 2017-06-22T21:08:06Z

src/main/scala/org/bdgenomics/cannoli/util/QuerynameGrouper.scala

@@ -31,6 +31,7 @@ private[cannoli] object QuerynameGrouper extends Serializable {
  def apply(rdd: AlignmentRecordRDD): FragmentRDD = {
    new FragmentRDD(apply(rdd.rdd),
      rdd.sequences,
-      rdd.recordGroups)
+      rdd.recordGroups,
+      optPartitionMap = None)


We lose some information here about whether or not the data is sorted. This should be fixed in bigdatagenomics/adam#1561. Under the new method, it does force a recompute on the optPartitionMap, but at least we would keep track of whether or not the data is sorted.

By definition, the data is not coordinate sorted here.

heuermh · 2017-07-01T18:15:03Z

I'm not sure how, I've git-borked the commits on this one and can't seem to unsuck them. The problem with bowtie itself has been fixed in version 1.2.1.1, and is available in homebrew-science. I'd like to close this unmerged and try again with separate pull requests.

fnothaft · 2017-07-05T13:34:30Z

Closing as unmerged, @heuermh will reopen as separate PRs.

heuermh added 4 commits May 17, 2017 08:01

Use tab5 informatter for bowtie, update for ADAM version 0.23.0.

35a6485

Add placeholder publish_scaladoc.sh script

c1934fa

Use --interleaved for bowtie2

3b80adc

Remove unnecessary cache transform.

44b89a2

heuermh mentioned this pull request May 24, 2017

Adding fragment InFormatter for Bowtie tab5 format bigdatagenomics/adam#1522

Merged

Update per recent changes upstream.

236d570

heuermh mentioned this pull request Jun 22, 2017

Spark 1.x Scala 2.10 snapshot artifacts missing since 31 March 2017 bigdatagenomics/adam#1578

Closed

devin-petersohn reviewed Jun 22, 2017

View reviewed changes

heuermh changed the title ~~Use ADAM tab5 formatter for bowtie~~ [CANNOLI-33] Use ADAM tab5 formatter for bowtie Jul 1, 2017

fnothaft closed this Jul 5, 2017

heuermh deleted the bowtie-tab5 branch November 1, 2017 16:34

heuermh added this to the 0.1.0 milestone Jan 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CANNOLI-33] Use ADAM tab5 formatter for bowtie #42

[CANNOLI-33] Use ADAM tab5 formatter for bowtie #42

heuermh commented May 17, 2017

AmplabJenkins commented May 17, 2017

coveralls commented Jun 20, 2017 •

edited

AmplabJenkins commented Jun 20, 2017

heuermh commented Jun 22, 2017

coveralls commented Jun 22, 2017

coveralls commented Jun 22, 2017 •

edited

AmplabJenkins commented Jun 22, 2017

devin-petersohn left a comment

devin-petersohn Jun 22, 2017

fnothaft Jun 22, 2017

devin-petersohn Jun 22, 2017

fnothaft Jun 22, 2017

heuermh commented Jul 1, 2017

fnothaft commented Jul 5, 2017

		@@ -0,0 +1 @@
		#!/bin/bash

[CANNOLI-33] Use ADAM tab5 formatter for bowtie #42

[CANNOLI-33] Use ADAM tab5 formatter for bowtie #42

Conversation

heuermh commented May 17, 2017

AmplabJenkins commented May 17, 2017

Build result: FAILURE

coveralls commented Jun 20, 2017 • edited

AmplabJenkins commented Jun 20, 2017

Build result: FAILURE

heuermh commented Jun 22, 2017

coveralls commented Jun 22, 2017

coveralls commented Jun 22, 2017 • edited

AmplabJenkins commented Jun 22, 2017

Build result: FAILURE

devin-petersohn left a comment

Choose a reason for hiding this comment

devin-petersohn Jun 22, 2017

Choose a reason for hiding this comment

fnothaft Jun 22, 2017

Choose a reason for hiding this comment

devin-petersohn Jun 22, 2017

Choose a reason for hiding this comment

fnothaft Jun 22, 2017

Choose a reason for hiding this comment

heuermh commented Jul 1, 2017

fnothaft commented Jul 5, 2017

coveralls commented Jun 20, 2017 •

edited

coveralls commented Jun 22, 2017 •

edited