Added check for bam index files in loadIndexedBam #1831

akmorrow13 · Dec 13, 2017

No description provided.

AmplabJenkins · Dec 13, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2512/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1831/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 665e131 # timeout=10Checking out Revision 665e131 (origin/pr/1831/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 665e131385df3b695eb244735b2242cd385a4948First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · Dec 13, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2513/
Test PASSed.

akmorrow13 · Dec 13, 2017

Fixes #1830

heuermh · Dec 13, 2017

We hadn't checked in binary files as test resources before, as then diffs in git aren't meaningful. If we're going to break that convention, and this is as good a time as any, then we should add an index for small.bam and unit test(s) for the happy paths.

Do you need this for the 0.23.0 release?

AmplabJenkins · Dec 13, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2514/
Test PASSed.

AmplabJenkins · Dec 14, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2515/
Test PASSed.

fnothaft · Dec 14, 2017

fnothaft requested changes Dec 14, 2017

View changes

Looks good! A few small nits.

fnothaft · Dec 14, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    // todo: can this method handle SAM and CRAM, or just BAM?
    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


Use getFsAndFilesWithFilter.

fnothaft · Dec 14, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+      new Path(pathName + ".bai")
+    else
+      path
+
    // todo: can this method handle SAM and CRAM, or just BAM?


Can you update this comment to // currently only supports BAM and cross reference #1833.

fnothaft · Dec 14, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a directory of bam files, scan all files to
+    // select index files.
+    val indexPath = if (pathName.endsWith(".bam"))


Use {}'s.

Also, its not clear that this code and the comment match up. Can you add more to the comment to clarify how this bit of code is necessary for directory support?

fnothaft · Dec 14, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    // Make sure each bam file has an index
+    bamFiles.foreach(file => {


Nit: if multiple files are missing an index, this will just throw for the first one. Instead, I'd suggest:

val missingIndices = bamFiles.filter(f => indexFiles.contains(f.toString + ".bai")) require(missingIndices.isEmpty, "Missing indices for BAMs:\n%s".format(missingIndices.mkString("\n"))

fnothaft · Dec 14, 2017

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Can you add a test case for the directory/glob case?

Ping! Also, add a test case for the myFile.bai case.

akmorrow13 · Dec 15, 2017

akmorrow13 reviewed Dec 15, 2017

View changes

akmorrow13 · Dec 15, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


@fnothaft getFsAndFilesWith filter wasn't able to find .bai files unless just the directory was specified

Yeah makes sense.

AmplabJenkins · Dec 15, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2516/
Test PASSed.

fnothaft · Dec 23, 2017

fnothaft requested changes Dec 23, 2017

View changes

fnothaft · Dec 23, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a single file or *.bam, append .bai to find all bam indices.
+    // Otherwise, pathName is a directory and the entire path must be searched
+    // for indices.


Can you note:

// TODO: support BAMs where the index is located at s/.bam/.bai/g

These aren't common (myFile.bam + myFile.bai vs myFile.bam.bai) but you do see them in the wild occasionally.

Maybe create an issue for that after this merges.

Actually, if you do new Path(pathName.toString.replace(".bam", "*.bai")) here and make a change below (will drop a comment), I think you can handle both cases.

fnothaft · Dec 23, 2017

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    val missingIndices = bamFiles.filterNot(f => {
+      indexFiles.contains(f.toString + ".bai")


If you do indexFiles.contains(f.toString + ".bai") || indexFiles.contains(f.toString.dropRight(3) + "bai")) you should handle both the myFile.bam.bai and myFile.bai cases.

fnothaft · Dec 23, 2017

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Ping! Also, add a test case for the myFile.bai case.

fnothaft · Dec 24, 2017

fnothaft approved these changes Dec 24, 2017

View changes

Thanks @akmorrow13! LGTM.

AmplabJenkins · Dec 24, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2527/
Test PASSed.

AmplabJenkins · Dec 24, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2528/
Test PASSed.

akmorrow13 · Jan 3, 2018

@heuermh and @fnothaft can we merge this?

heuermh · Jan 3, 2018

I'll give it one last review check after the 0.23.0 release is cut

fnothaft · Jan 3, 2018

Yeah, this is good to go from my side but I have a strong preference for it being in 0.24.0 instead of 0.23.0.

heuermh · Jan 9, 2018

heuermh approved these changes Jan 9, 2018

View changes

heuermh · Jan 9, 2018

Thank you, @akmorrow13!

fnothaft requested a review from heuermh Dec 23, 2017

akmorrow13 added some commits Dec 13, 2017

Added check for bam index files in loadIndexedBam

bbc4782

added check for.bam.bai and .bai index files

Loading status checks…

ee06eda

akmorrow13 added this to the 0.24.0 milestone Jan 6, 2018

heuermh merged commit 87f4667 into bigdatagenomics:master Jan 9, 2018
2 checks passed

2 checks passed

Codacy/PR Quality Review Good work! A positive pull request.
Details

default Merged build finished.
Details

heuermh added this to Completed in Release 0.24.0 Jan 9, 2018

heuermh referenced this pull request Jan 9, 2018
Closed
loadIndexed bam lazily throws Exception if index does not exist #1830

bigdatagenomics/adam

Join GitHub today

Added check for bam index files in loadIndexedBam #1831

Conversation

akmorrow13 commented Dec 13, 2017

This comment has been minimized.

AmplabJenkins commented Dec 13, 2017

Build result: FAILURE

This comment has been minimized.

AmplabJenkins commented Dec 13, 2017

This comment has been minimized.

akmorrow13 commented Dec 13, 2017

This comment has been minimized.

heuermh commented Dec 13, 2017

This comment has been minimized.

AmplabJenkins commented Dec 13, 2017

This comment has been minimized.

AmplabJenkins commented Dec 14, 2017

fnothaft requested changes Dec 14, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

akmorrow13 reviewed Dec 15, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

AmplabJenkins commented Dec 15, 2017

fnothaft requested changes Dec 23, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

fnothaft requested a review from heuermh Dec 23, 2017

fnothaft approved these changes Dec 24, 2017 View changes

This comment has been minimized.

AmplabJenkins commented Dec 24, 2017

akmorrow13 added some commits Dec 13, 2017

This comment has been minimized.

AmplabJenkins commented Dec 24, 2017

This comment has been minimized.

akmorrow13 commented Jan 3, 2018

This comment has been minimized.

heuermh commented Jan 3, 2018

This comment has been minimized.

fnothaft commented Jan 3, 2018

akmorrow13 added this to the 0.24.0 milestone Jan 6, 2018

heuermh approved these changes Jan 9, 2018 View changes

Hide details View details heuermh merged commit 87f4667 into bigdatagenomics:master Jan 9, 2018 2 checks passed

2 checks passed

This comment has been minimized.

heuermh commented Jan 9, 2018

heuermh added this to Completed in Release 0.24.0 Jan 9, 2018

heuermh referenced this pull request Jan 9, 2018

fnothaft requested changes Dec 14, 2017

View changes

akmorrow13 reviewed Dec 15, 2017

View changes

fnothaft requested changes Dec 23, 2017

View changes

fnothaft approved these changes Dec 24, 2017

View changes

heuermh approved these changes Jan 9, 2018

View changes

heuermh merged commit `87f4667` into bigdatagenomics:master Jan 9, 2018
2 checks passed