Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(snowflake): ensure that timestamp conversion from parquet files is correct #9181

Merged

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented May 13, 2024

Fixes a bug where the scale was incorrectly left up to Snowflake, which can produce incorrectly scaled values if a timestamp is read in as a variant integer and then cast to string.

@cpcloud cpcloud added this to the 9.1 milestone May 13, 2024
@cpcloud cpcloud added bug Incorrect behavior inside of ibis snowflake The Snowflake backend labels May 13, 2024
@cpcloud cpcloud requested a review from gforsyth May 13, 2024 12:33
@cpcloud cpcloud force-pushed the fix-snowflake-parquet-timestamps branch from 8a81a74 to dfe215f Compare May 13, 2024 12:39
@cpcloud
Copy link
Member Author

cpcloud commented May 13, 2024

Oh, it looks like they may have actually fixed this upstream using a new parquet format parameter USE_LOGICAL_TYPE (https://community.snowflake.com/s/article/How-to-load-logical-type-TIMESTAMP-data-from-Parquet-files-into-Snowflake).

@cpcloud cpcloud force-pushed the fix-snowflake-parquet-timestamps branch from dfe215f to 1ffd0fd Compare May 13, 2024 12:46
@cpcloud cpcloud force-pushed the fix-snowflake-parquet-timestamps branch from 1ffd0fd to 5cf91e4 Compare May 13, 2024 12:47
@cpcloud
Copy link
Member Author

cpcloud commented May 13, 2024

Snowflake is all green:

…/ibis on  fix-snowflake-parquet-timestamps is 📦 v9.0.0 via 🐍 v3.12.2 via ❄️   impure (ibis-3.12.2-env)
❯ pytest -m snowflake -n auto --dist loadgroup -q
bringing up nodes...
....x....x.............x..............................x...........xx...x.....x..............x......x........x......................x.xx.....x.................x........................xx..... [ 11%]
.................................................................................x...........x...x...................x.....x...........x.......x...................xx.............x....x.....x [ 22%]
....................xx.......................x..x.....s.x...............xx......................................xxx....x................x.xx..........x.xx..........x.......x.............x..x [ 33%]
..x........x..x....x.....xx.........x..........x....x..............xx..........x..x............s..................x.............s.....................................s................x...... [ 44%]
x.............x...x........s..........................................................x.......x........x..............x....................xx..............xx...xx......x.......x............. [ 55%]
.x.........xx..x.x......x..x.x.x.....x....x.............xx.x.xx.....x..x..x..xxx..x.xx..xx.xx.xx.xx.xxx..x...x.x.x.xxxx..x..xx..x.xx.xxx..xxxxx.x.x.x.x..xx.xxxxxxxxx..x..x.xxxx..x...xx.x..x. [ 67%]
x..xx.xxxx.x..........xxx................x.....................................x..x.....x....................x.......x.....................x...x......x...x.....x.........x.x....xx...x.xx.xx.. [ 78%]
xx..x..xx...xx..x..........x.x.........x......x...x.....x.x..........x...xxx.x.x.x...xx....x.x....x...s.....x.....x...............x.........................xx.........xx..................... [ 89%]
.......x...........................................................................................................s.......................s......s..............................              [100%]
1458 passed, 9 skipped, 231 xfailed in 116.53s (0:01:56)

Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gforsyth gforsyth merged commit 1ba4c32 into ibis-project:main May 13, 2024
81 checks passed
@cpcloud cpcloud deleted the fix-snowflake-parquet-timestamps branch May 13, 2024 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis snowflake The Snowflake backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants