When trying to read deltatable returns OSError: Repetition level histogram size mismatch

[felipbizz]

Environment

Python 3.12.5

Delta-rs version:
"deltalake==0.24.0"

Binding:

Environment:

• Cloud provider:
• OS: wsl
• Other:

---

Bug

What happened: When trying to read deltatable returns OSError: Repetition level histogram size mismatch

What you expected to happen: Excepted to read the deltalake

How to reproduce it:

from deltalake import write_deltalake, DeltaTable
import pandas as pd
df = pd.DataFrame({'teste': [1, 2, 3]})
write_deltalake('teste', df, mode='append')
df = pd.DataFrame({'teste': [4, 5, 6]})
write_deltalake('teste', df, mode='append')

df = DeltaTable('./teste')    ## -> This step returns the error
df = df.to_pandas()

More details:

When I install deltalake, the pyarrow dependency is as pyarrow>=16, however as of version 19.0.0 this error is occurring.
To work around this, the dependency can be stuck by limiting it to version 18.1.0

More details about this error can be viewed here: apache/arrow#45283

[ion-elgreco]

This is a pyarrow issue, it will be resolved in the next pyarrow version.

[riziles]

For anybody coming here looking for answers: pinning pyarrow to a version < 19 resolves this.

[amoeba]

We've published a patch release to address this, 19.0.1, so you should be able to adjust any version pins accordingly. If you're still running into this, please feel free to file an issue at https://github.com/apache/arrow/issues.