Next on empty iterator in BroadcastRegionJoin #661

Closed
fnothaft opened this Issue Apr 29, 2015 · 5 comments

Comments

Projects
3 participants
@fnothaft
Member

fnothaft commented Apr 29, 2015

Not sure why this is happening, but I'm getting this:

java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
        at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
        at scala.collection.IterableLike$class.head(IterableLike.scala:91)
        at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$head(ArrayBuffer.scala:47)
        at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
        at scala.collection.mutable.ArrayBuffer.head(ArrayBuffer.scala:47)
        at org.bdgenomics.adam.rdd.BroadcastRegionJoin$$anonfun$4.apply(BroadcastRegionJoin.scala:114)
        at org.bdgenomics.adam.rdd.BroadcastRegionJoin$$anonfun$4.apply(BroadcastRegionJoin.scala:114)

It is coming from this line:

// each element of the left-side RDD should have exactly one partition.
    val smallerKeyed: RDD[(ReferenceRegion, (ReferenceRegion, T))] =
      baseRDD.map(t => (regions.value.regionsFor(t).head, t))

I'm not sure exactly what's going on here. I've made a local hack to get my code to run that changes this to an optional flatMap, which lets things run forward; I'll take more of a look later this week. @tdanford do you have any intuition as to what the single-partition invariant is here? I'm thinking that this should be changed to either a flatMap with a log warning, or at least a more clear assert message (the stack trace isn't terribly helpful).

@fnothaft fnothaft added the bug label Apr 29, 2015

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Apr 29, 2015

Member

@pgrosu Sorry if things were unclear; that's not what I'm asking. There's an "exactly-one" invariant in the code that is being violated. There's no assert checking the "exactly-one" invariant, but the invariant is documented in the code. So yes, the exception is thrown because we're calling head on an empty iterator, but the bug is that the iterator is supposed to be guaranteed to have a single element in it.

Member

fnothaft commented Apr 29, 2015

@pgrosu Sorry if things were unclear; that's not what I'm asking. There's an "exactly-one" invariant in the code that is being violated. There's no assert checking the "exactly-one" invariant, but the invariant is documented in the code. So yes, the exception is thrown because we're calling head on an empty iterator, but the bug is that the iterator is supposed to be guaranteed to have a single element in it.

@pgrosu

This comment has been minimized.

Show comment
Hide comment
@pgrosu

pgrosu Apr 30, 2015

Ah, okay - no problem :)

Thanks for the clarification,
~p

pgrosu commented Apr 30, 2015

Ah, okay - no problem :)

Thanks for the clarification,
~p

@fnothaft fnothaft added this to the 0.21.0 milestone Jul 20, 2016

@heuermh heuermh modified the milestones: 0.21.0, 0.22.0 Oct 13, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 2, 2017

Member

The section of code impacted by this bug was removed in #1234. Closing.

Member

fnothaft commented Mar 2, 2017

The section of code impacted by this bug was removed in #1234. Closing.

@fnothaft fnothaft closed this Mar 2, 2017

@heuermh heuermh added this to Completed in Release 0.23.0 Mar 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment