Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Check for max_chunksize > 0 in Table.to_batches() #39788

Closed
kylebarron opened this issue Jan 24, 2024 · 1 comment · Fixed by #39796
Closed

Python: Check for max_chunksize > 0 in Table.to_batches() #39788

kylebarron opened this issue Jan 24, 2024 · 1 comment · Fixed by #39796

Comments

@kylebarron
Copy link
Contributor

kylebarron commented Jan 24, 2024

Describe the bug, including details regarding any error messages, version, and platform.

Table.to_batches should check for max_chunksize > 0 and raise an exception if not true. This is hard to debug because it creates an infinite loop where Jupyter just hangs.

import pyarrow as pa
table = pa.table({'a': [1, 2, 3, 4]})
table.to_batches(max_chunksize=0)
# hangs forever

It also led to OOM.
image

Component(s)

Python

@jorisvandenbossche
Copy link
Member

Good catch! Did a quick fix in #39796

raulcd pushed a commit that referenced this issue Feb 2, 2024
### Rationale for this change

Validating the keyword to be strictly positive, to avoid an infinite loop.

* Closes: #39788

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
@raulcd raulcd added this to the 16.0.0 milestone Feb 2, 2024
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
…pache#39796)

### Rationale for this change

Validating the keyword to be strictly positive, to avoid an infinite loop.

* Closes: apache#39788

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
…pache#39796)

### Rationale for this change

Validating the keyword to be strictly positive, to avoid an infinite loop.

* Closes: apache#39788

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
…pache#39796)

### Rationale for this change

Validating the keyword to be strictly positive, to avoid an infinite loop.

* Closes: apache#39788

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants