Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] Fix 0-partition matrix table writes. #10038

Merged
merged 5 commits into from Jul 1, 2021
Merged

Conversation

tpoterba
Copy link
Contributor

CHANGELOG: Fix crashes when writing/reading matrix tables with 0 partitions.

val rbBuilder = new BoxedArrayBuilder[Interval]()
partitioner.rangeBounds.foreach { b =>
val idx = partitioner.lowerBoundInterval(b)
if (idx < partitioner.numPartitions) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusing me. I don't see how this can ever be false: idx must be at most the index of b. If so, the None case looks equivalent to before. What was the intention here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not always true -- the test case I added here writes a table with a single partition whose bound is: ([0]-[0])

lowerBoundInterval returns 1 in this case.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. Thanks. I was assuming non-degenerate intervals.

val rb = partitioner.rangeBounds(i)
rb.isDisjointFrom(iOrd, rb)
}.toArray
(includedIndices.map(specLeft.partFiles), partitioner.copy(rangeBounds = includedIndices.map(partitioner.rangeBounds)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think the None case could just be (specLeft.partFiles, partitioner), but I'm okay with dropping degenerate partitions too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! Yes, this is better. The root problem is that the lower bound is not meaningful, but this is fine.

@johnc1231
Copy link
Contributor

Conflicts.

@johnc1231
Copy link
Contributor

Fixed conflicts

@tpoterba
Copy link
Contributor Author

I think there's still a bug. Will get to it at some point...

CHANGELOG: Fix crashes when writing/reading matrix tables with 0 partitions.
@danking danking merged commit afb38c8 into hail-is:main Jul 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants