Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] When Validating, ensure DenseUnionArray offsets are increasing #26543

Closed
asfimport opened this issue Nov 13, 2020 · 6 comments
Closed

[C++] When Validating, ensure DenseUnionArray offsets are increasing #26543

asfimport opened this issue Nov 13, 2020 · 6 comments

Comments

@asfimport
Copy link

asfimport commented Nov 13, 2020

#8652 (comment)

The format documentation states that "The respective offsets for each child value array must be in order / increasing."
https://arrow.apache.org/docs/format/Columnar.html#dense-union

However this is not currently checked in Array::ValidateFull

Reporter: Ben Kietzman / @bkietz
Assignee: Antoine Pitrou / @pitrou

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-10580. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
The problem is that we currently have tests that deliberately use non-increasing offsets. @wesm should we fix them?

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
Something else: if we mandate strictly increasing values, we can't collapse contiguous null or empty slots together.

@asfimport
Copy link
Author

Wes McKinney / @wesm:
We should discuss on the mailing list. In principle I don't have an issue with non-monotonic offsets, but this change is fully backward compatible and should be forward compatible in principle but it's a gray area so we should discuss and see if we want to make a change to the language in the columnar specification

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
Which change? The change to forbid non-increasing offsets?

@asfimport
Copy link
Author

Wes McKinney / @wesm:
Possibly allowing non-increasing offsets in the columnar spec

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
Issue resolved by pull request 9794
#9794

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants