Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GetByteRangesArray doesn't support run-end encoded arrays #36502

Closed
sjperkins opened this issue Jul 6, 2023 · 3 comments · Fixed by #36521
Closed

GetByteRangesArray doesn't support run-end encoded arrays #36502

sjperkins opened this issue Jul 6, 2023 · 3 comments · Fixed by #36521
Assignees
Milestone

Comments

@sjperkins
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

Environment:

OS: Ubuntu 20.04
Python: 3.10
Arrow: 12.0.0

Reproducer:

import numpy as np
import pyarrow as pa
import pyarrow.compute as pac

if __name__ == "__main__":
   pa.Table.from_pydict({"A": pac.run_end_encode(np.zeros(1000))}).nbytes

Example error message:

In [10]: pa.Table.from_pydict({"A": pac.run_end_encode(np.zeros(1000))}).nbytes
---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
Cell In[10], line 1
----> 1 pa.Table.from_pydict({"A": pac.run_end_encode(np.zeros(1000))}).nbytes

File ~/venv/venv3.10/lib/python3.10/site-packages/pyarrow/table.pxi:4474, in pyarrow.lib.Table.nbytes.__get__()

File ~/venv/venv3.10/lib/python3.10/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/venv/venv3.10/lib/python3.10/site-packages/pyarrow/error.pxi:123, in pyarrow.lib.check_status()

ArrowTypeError: Extracting byte ranges not supported for type run_end_encoded<run_ends: int32, values: double>

Component(s)

C++

@mapleFU
Copy link
Member

mapleFU commented Jul 6, 2023

cc @felipecrv

@felipecrv
Copy link
Contributor

@mapleFU @sjperkins I will fix it now.

@felipecrv felipecrv self-assigned this Jul 6, 2023
pitrou pushed a commit that referenced this issue Jul 11, 2023
…nges (#36521)

### Rationale for this change

Fix for #36502.

### What changes are included in this PR?

Fix and C++ tests.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Addition of a new function to the `ree_util` namespace.
* Closes: #36502

Lead-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
@pitrou pitrou added this to the 13.0.0 milestone Jul 11, 2023
@sjperkins
Copy link
Contributor Author

Thank you for the fix @felipecrv.

@raulcd raulcd modified the milestones: 13.0.0, 14.0.0 Jul 13, 2023
R-JunmingChen pushed a commit to R-JunmingChen/arrow that referenced this issue Aug 20, 2023
…ByteRanges (apache#36521)

### Rationale for this change

Fix for apache#36502.

### What changes are included in this PR?

Fix and C++ tests.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Addition of a new function to the `ree_util` namespace.
* Closes: apache#36502

Lead-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants