Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIG-40455] fix arrow batch int64 to ntz/ltz #151

Merged
merged 2 commits into from
Aug 5, 2023

Conversation

Yifeng-Sigma
Copy link

@Yifeng-Sigma Yifeng-Sigma commented Aug 4, 2023

Fix https://sigma-computing.sentry.io/issues/4369020647
panic: interface conversion: arrow.Array is *array.Int64, not *array.Timestamp
sometimes snowflake sends us int64 array and we want to interpret as timestamp.
underlying array.Timestamp is just int64.

@Yifeng-Sigma Yifeng-Sigma temporarily deployed to gosnowflake-ci-env August 4, 2023 23:57 — with GitHub Actions Inactive
@Yifeng-Sigma Yifeng-Sigma requested a review from a team August 4, 2023 23:59
converter.go Outdated
} else if col.DataType().ID() == arrow.INT64 {
for i, t := range col.(*array.Int64).Int64Values() {
if !col.IsNull(i) {
val := time.Unix(0, t*int64(math.Pow10(9-int(srcColumnMeta.Scale)))).UTC()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other calls to time.Unix under case timestampLtzType aren't followed by .UTC(), is this deviation intentional?

Copy link
Author

@Yifeng-Sigma Yifeng-Sigma Aug 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!
It's copy and paste mistake. I should follow line 1049.

@Yifeng-Sigma Yifeng-Sigma temporarily deployed to gosnowflake-ci-env August 5, 2023 01:02 — with GitHub Actions Inactive
@Yifeng-Sigma Yifeng-Sigma requested a review from aldld August 5, 2023 01:07
@@ -941,6 +941,29 @@ func TestArrowToRecord(t *testing.T) {
return -1
},
},
{
logical: "timestamp_ntz",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add a test for this case with timestmap_ltz?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed this earlier. Follow up in #152

@Yifeng-Sigma Yifeng-Sigma merged commit dcd8a27 into master Aug 5, 2023
3 of 4 checks passed
@Yifeng-Sigma Yifeng-Sigma deleted the yifeng/fix_timestamp_i64 branch August 5, 2023 01:48
@github-actions github-actions bot locked and limited conversation to collaborators Aug 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
2 participants