Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(python, rust): use new schema for stats parsing instead of old #2480

Conversation

ion-elgreco
Copy link
Collaborator

Description

In some edge cases where we schema evolve, it would parse the stats with the old schema result in these kind of errors:
Exception: Json error: whilst decoding field 'minValues': whilst decoding field 'foo': failed to parse 1000000000000 as Int8

import polars as pl
from deltalake import write_deltalake

pl.DataFrame({
    "foo": [1]
}, schema={"foo": pl.Int8}).write_delta("TEST_TABLE_BUG")


write_deltalake("TEST_TABLE_BUG", data = pl.DataFrame({
    "foo": [1000000000000]
}, schema={"foo": pl.Int64}).to_arrow(), mode='overwrite', overwrite_schema=True,engine='rust')

Instead of taking the old schema, I added an optional schema to be passed in the logMapper

@ion-elgreco ion-elgreco force-pushed the fix/using_old_schema_to_parse_new_stats branch from d958fc9 to 0b4f5c9 Compare May 3, 2024 22:30
@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels May 3, 2024
@ion-elgreco ion-elgreco changed the title fix(python, rust): using old schema for stats parsing fix(python, rust): use new schema for stats parsing instead of old May 3, 2024
@ion-elgreco ion-elgreco force-pushed the fix/using_old_schema_to_parse_new_stats branch from 0b4f5c9 to bf812ad Compare May 6, 2024 16:22
@ion-elgreco ion-elgreco enabled auto-merge (squash) May 6, 2024 16:23
@ion-elgreco ion-elgreco merged commit e25aed7 into delta-io:main May 6, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants