[query] Fix 0-partition matrix table writes. #10038
Conversation
| val rbBuilder = new BoxedArrayBuilder[Interval]() | ||
| partitioner.rangeBounds.foreach { b => | ||
| val idx = partitioner.lowerBoundInterval(b) | ||
| if (idx < partitioner.numPartitions) { |
There was a problem hiding this comment.
This is confusing me. I don't see how this can ever be false: idx must be at most the index of b. If so, the None case looks equivalent to before. What was the intention here?
There was a problem hiding this comment.
This is not always true -- the test case I added here writes a table with a single partition whose bound is: ([0]-[0])
lowerBoundInterval returns 1 in this case.
There was a problem hiding this comment.
Ah, I see. Thanks. I was assuming non-degenerate intervals.
| val rb = partitioner.rangeBounds(i) | ||
| rb.isDisjointFrom(iOrd, rb) | ||
| }.toArray | ||
| (includedIndices.map(specLeft.partFiles), partitioner.copy(rangeBounds = includedIndices.map(partitioner.rangeBounds))) |
There was a problem hiding this comment.
I actually think the None case could just be (specLeft.partFiles, partitioner), but I'm okay with dropping degenerate partitions too.
There was a problem hiding this comment.
Oh! Yes, this is better. The root problem is that the lower bound is not meaningful, but this is fine.
|
Conflicts. |
|
Fixed conflicts |
|
I think there's still a bug. Will get to it at some point... |
4 participants
CHANGELOG: Fix crashes when writing/reading matrix tables with 0 partitions.