PyArrow 19.0.0 raises exception with `to_pyarrow_table()`

[rustyconover]

Environment

Delta-rs version: 0.24.0

---

Bug

What happened:

When calling to_pyarrow_table() with PyArrow version 19.0.0, an exception is raised with the following traceback:

Traceback (most recent call last):
  File "/Users/rusty/deltalake-experiments/.venv/bin/deltalake_experiment", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/Users/rusty/deltalake-experiments/src/deltalake_experiments/__init__.py", line 38, in run
    t = dt.to_pyarrow_table()
        ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rusty/deltalake-experiments/.venv/lib/python3.12/site-packages/deltalake/table.py", line 1236, in to_pyarrow_table
    ).to_table(columns=columns, filter=filters)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 574, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3865, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Repetition level histogram size mismatch

What you expected to happen:

The to_pyarrow_table() method should execute without raising any exceptions, returning the corresponding PyArrow table as expected.

How to reproduce it:

Here is a short test case

def run() -> int:
    example_schema = pa.schema(
        [
            pa.field("name", pa.string()),
        ]
    )
    example_table = pa.Table.from_pylist(
        [{"name": "Tintin"}],
        schema=example_schema,
    )

    deltalake.write_deltalake(
        "./test-table",
        example_table,
        mode="overwrite",
    )

    dt = deltalake.DeltaTable("./test-table")
    t = dt.to_pyarrow_table()
    print(t)

Additional Information

The issue appears to be specific to PyArrow version 19.0.0.

Thank you for your assistance in investigating this issue! Let me know if I can provide additional details or perform further tests.

[ion-elgreco]

@rustyconover this is a known issue of pyarrow: apache/arrow#45283

[rustyconover]

Cool thanks!

[amoeba]

I posted this over in #3208 too but we've published a patch release to address this, 19.0.1, so you should be able to adjust any version pins accordingly. If you're still running into this, please feel free to file an issue at https://github.com/apache/arrow/issues.