Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next on empty iterator in BroadcastRegionJoin #661

Closed
fnothaft opened this issue Apr 29, 2015 · 5 comments
Closed

Next on empty iterator in BroadcastRegionJoin #661

fnothaft opened this issue Apr 29, 2015 · 5 comments

Comments

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Apr 29, 2015

Not sure why this is happening, but I'm getting this:

java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
        at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
        at scala.collection.IterableLike$class.head(IterableLike.scala:91)
        at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$head(ArrayBuffer.scala:47)
        at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
        at scala.collection.mutable.ArrayBuffer.head(ArrayBuffer.scala:47)
        at org.bdgenomics.adam.rdd.BroadcastRegionJoin$$anonfun$4.apply(BroadcastRegionJoin.scala:114)
        at org.bdgenomics.adam.rdd.BroadcastRegionJoin$$anonfun$4.apply(BroadcastRegionJoin.scala:114)

It is coming from this line:

// each element of the left-side RDD should have exactly one partition.
    val smallerKeyed: RDD[(ReferenceRegion, (ReferenceRegion, T))] =
      baseRDD.map(t => (regions.value.regionsFor(t).head, t))

I'm not sure exactly what's going on here. I've made a local hack to get my code to run that changes this to an optional flatMap, which lets things run forward; I'll take more of a look later this week. @tdanford do you have any intuition as to what the single-partition invariant is here? I'm thinking that this should be changed to either a flatMap with a log warning, or at least a more clear assert message (the stack trace isn't terribly helpful).

@fnothaft fnothaft added the bug label Apr 29, 2015
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Apr 29, 2015

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Apr 29, 2015

@pgrosu Sorry if things were unclear; that's not what I'm asking. There's an "exactly-one" invariant in the code that is being violated. There's no assert checking the "exactly-one" invariant, but the invariant is documented in the code. So yes, the exception is thrown because we're calling head on an empty iterator, but the bug is that the iterator is supposed to be guaranteed to have a single element in it.

@pgrosu
Copy link

@pgrosu pgrosu commented Apr 30, 2015

Ah, okay - no problem :)

Thanks for the clarification,
~p

@fnothaft fnothaft added this to the 0.21.0 milestone Jul 20, 2016
@heuermh heuermh modified the milestones: 0.21.0, 0.22.0 Oct 13, 2016
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Mar 2, 2017

The section of code impacted by this bug was removed in #1234. Closing.

@fnothaft fnothaft closed this Mar 2, 2017
@heuermh heuermh added this to Completed in Release 0.23.0 Mar 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.