Added check for bam index files in loadIndexedBam #1831

akmorrow13 · 2017-12-13T20:21:28Z

No description provided.

AmplabJenkins · 2017-12-13T20:29:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2512/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1831/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 665e131 # timeout=10Checking out Revision 665e131 (origin/pr/1831/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 665e131385df3b695eb244735b2242cd385a4948First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2017-12-13T20:53:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2513/
Test PASSed.

akmorrow13 · 2017-12-13T21:39:44Z

Fixes #1830

heuermh · 2017-12-13T21:55:57Z

We hadn't checked in binary files as test resources before, as then diffs in git aren't meaningful. If we're going to break that convention, and this is as good a time as any, then we should add an index for small.bam and unit test(s) for the happy paths.

Do you need this for the 0.23.0 release?

AmplabJenkins · 2017-12-13T22:05:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2514/
Test PASSed.

AmplabJenkins · 2017-12-14T19:09:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2515/
Test PASSed.

fnothaft

Looks good! A few small nits.

fnothaft · 2017-12-14T20:19:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    // todo: can this method handle SAM and CRAM, or just BAM?
    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


Use getFsAndFilesWithFilter.

fnothaft · 2017-12-14T20:19:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+      new Path(pathName + ".bai")
+    else
+      path
+
    // todo: can this method handle SAM and CRAM, or just BAM?


Can you update this comment to // currently only supports BAM and cross reference #1833.

fnothaft · 2017-12-14T20:19:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a directory of bam files, scan all files to
+    // select index files.
+    val indexPath = if (pathName.endsWith(".bam"))


Use {}'s.

Also, its not clear that this code and the comment match up. Can you add more to the comment to clarify how this bit of code is necessary for directory support?

fnothaft · 2017-12-14T20:19:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    // Make sure each bam file has an index
+    bamFiles.foreach(file => {


Nit: if multiple files are missing an index, this will just throw for the first one. Instead, I'd suggest:

val missingIndices = bamFiles.filter(f => indexFiles.contains(f.toString + ".bai")) require(missingIndices.isEmpty, "Missing indices for BAMs:\n%s".format(missingIndices.mkString("\n"))

fnothaft · 2017-12-14T20:19:12Z

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Can you add a test case for the directory/glob case?

Ping! Also, add a test case for the myFile.bai case.

akmorrow13 · 2017-12-15T19:28:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


@fnothaft getFsAndFilesWith filter wasn't able to find .bai files unless just the directory was specified

Yeah makes sense.

AmplabJenkins · 2017-12-15T19:49:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2516/
Test PASSed.

fnothaft · 2017-12-23T06:12:37Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a single file or *.bam, append .bai to find all bam indices.
+    // Otherwise, pathName is a directory and the entire path must be searched
+    // for indices.


Can you note:

// TODO: support BAMs where the index is located at s/.bam/.bai/g

These aren't common (myFile.bam + myFile.bai vs myFile.bam.bai) but you do see them in the wild occasionally.

Maybe create an issue for that after this merges.

Actually, if you do new Path(pathName.toString.replace(".bam", "*.bai")) here and make a change below (will drop a comment), I think you can handle both cases.

fnothaft · 2017-12-23T06:12:37Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    val missingIndices = bamFiles.filterNot(f => {
+      indexFiles.contains(f.toString + ".bai")


If you do indexFiles.contains(f.toString + ".bai") || indexFiles.contains(f.toString.dropRight(3) + "bai")) you should handle both the myFile.bam.bai and myFile.bai cases.

fnothaft · 2017-12-23T06:12:37Z

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Ping! Also, add a test case for the myFile.bai case.

fnothaft

Thanks @akmorrow13! LGTM.

AmplabJenkins · 2017-12-24T03:43:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2527/
Test PASSed.

AmplabJenkins · 2017-12-24T04:48:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2528/
Test PASSed.

akmorrow13 · 2018-01-03T16:04:27Z

@heuermh and @fnothaft can we merge this?

heuermh · 2018-01-03T16:10:05Z

I'll give it one last review check after the 0.23.0 release is cut

fnothaft · 2018-01-03T16:50:00Z

Yeah, this is good to go from my side but I have a strong preference for it being in 0.24.0 instead of 0.23.0.

heuermh · 2018-01-09T19:43:32Z

Thank you, @akmorrow13!

akmorrow13 force-pushed the akmorrow13:indexedbam-index branch from 786643d to 6a4859f Dec 13, 2017

akmorrow13 force-pushed the akmorrow13:indexedbam-index branch from 6a4859f to 3380265 Dec 14, 2017

fnothaft requested changes Dec 14, 2017

View changes

akmorrow13 reviewed Dec 15, 2017

View changes

fnothaft requested changes Dec 23, 2017

View changes

fnothaft requested a review from heuermh Dec 23, 2017

fnothaft approved these changes Dec 24, 2017

View changes

akmorrow13 added 2 commits Dec 13, 2017

Added check for bam index files in loadIndexedBam

bbc4782

added check for.bam.bai and .bai index files

Loading status checks…

ee06eda

akmorrow13 force-pushed the akmorrow13:indexedbam-index branch from 5cfe378 to ee06eda Dec 24, 2017

akmorrow13 added this to the 0.24.0 milestone Jan 6, 2018

heuermh approved these changes Jan 9, 2018

View changes

heuermh merged commit 87f4667 into bigdatagenomics:master Jan 9, 2018
2 checks passed

2 checks passed

Codacy/PR Quality Review Good work! A positive pull request.
Details

default Merged build finished.
Details

heuermh added this to Completed in Release 0.24.0 Jan 9, 2018

heuermh mentioned this pull request Jan 9, 2018

loadIndexed bam lazily throws Exception if index does not exist #1830

Closed

bigdatagenomics / adam

Added check for bam index files in loadIndexedBam #1831

Added check for bam index files in loadIndexedBam #1831

akmorrow13 commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

akmorrow13 commented Dec 13, 2017

heuermh commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 14, 2017

fnothaft left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

AmplabJenkins commented Dec 15, 2017

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

fnothaft left a comment

AmplabJenkins commented Dec 24, 2017

AmplabJenkins commented Dec 24, 2017

akmorrow13 commented Jan 3, 2018

heuermh commented Jan 3, 2018

fnothaft commented Jan 3, 2018

heuermh commented Jan 9, 2018

		val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

		val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))

bigdatagenomics / adam

Added check for bam index files in loadIndexedBam #1831

Added check for bam index files in loadIndexedBam #1831

Conversation

akmorrow13 commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

Build result: FAILURE

AmplabJenkins commented Dec 13, 2017

akmorrow13 commented Dec 13, 2017

heuermh commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 14, 2017

fnothaft left a comment

This comment has been minimized.

fnothaft Dec 14, 2017 Member

This comment has been minimized.

fnothaft Dec 14, 2017 Member

This comment has been minimized.

fnothaft Dec 14, 2017 Member

This comment has been minimized.

fnothaft Dec 14, 2017 Member

This comment has been minimized.

fnothaft Dec 14, 2017 Member

This comment has been minimized.

fnothaft Dec 23, 2017 Member

This comment has been minimized.

akmorrow13 Dec 15, 2017 Author Contributor

This comment has been minimized.

fnothaft Dec 23, 2017 Member

AmplabJenkins commented Dec 15, 2017

This comment has been minimized.

fnothaft Dec 23, 2017 Member

This comment has been minimized.

fnothaft Dec 23, 2017 Member

This comment has been minimized.

fnothaft Dec 23, 2017 Member

This comment has been minimized.

fnothaft Dec 23, 2017 Member

This comment has been minimized.

fnothaft Dec 23, 2017 Member

fnothaft left a comment

AmplabJenkins commented Dec 24, 2017

AmplabJenkins commented Dec 24, 2017

akmorrow13 commented Jan 3, 2018

heuermh commented Jan 3, 2018

fnothaft commented Jan 3, 2018

heuermh commented Jan 9, 2018

fnothaft Dec 14, 2017
Member

fnothaft Dec 14, 2017
Member

fnothaft Dec 14, 2017
Member

fnothaft Dec 14, 2017
Member

fnothaft Dec 14, 2017
Member

fnothaft Dec 23, 2017
Member

akmorrow13 Dec 15, 2017
Author Contributor

fnothaft Dec 23, 2017
Member

fnothaft Dec 23, 2017
Member

fnothaft Dec 23, 2017
Member

fnothaft Dec 23, 2017
Member

fnothaft Dec 23, 2017
Member

fnothaft Dec 23, 2017
Member