Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: wrong timezone processing in snowpipe #186

Closed

Conversation

taskooh
Copy link

@taskooh taskooh commented Feb 3, 2023

Description & motivation

Now it is using current_timestamp::timestamp function to get current timestamp, but this is wrong because type of current_timestamp is TIMESTAMP_LTZ(local time zone) and when converted to timestamp_ntz(no time zone) by ::timestamp timezone is forcibly removed and showing wrong timestamp.
We should use sysdate() instead.
https://docs.snowflake.com/en/sql-reference/functions/sysdate.html

image

How i checked:

Look at dbt log and generated raw sql. Run that sql with this update manually on snowflake, and check that timezone issue is fixed.

Checklist

  • I have verified that these changes work locally
  • I have updated the README.md (if applicable)
  • I have added an integration test for my fix/feature (if applicable)

@taskooh taskooh changed the title fix: wrong timezone processing fix: wrong timezone processing in snowpipe Feb 3, 2023
@github-actions
Copy link

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Aug 15, 2023
@yingyingqiqi
Copy link

yingyingqiqi commented Aug 16, 2023

The snowflake documentation does not recommend using SYSDATE/CURRENT_TIMESTAMP in Snowpipe. i am using METADATA$START_SCAN_TIME

https://docs.snowflake.com/en/user-guide/data-load-snowpipe-ts#load-times-inserted-using-current-timestamp-earlier-than-load-time-values-in-copy-history-view

@github-actions github-actions bot removed the Stale label Aug 17, 2023
Copy link

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Feb 13, 2024
Copy link

Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers.

@github-actions github-actions bot closed this Feb 21, 2024
@AndyHuang1991
Copy link

We're also encounter this issue, we then modify system timezone to 'UTC' to address... Agree on @yingyingqiqi 's reply #186 (comment) to get more accurate timestamp, but as per snowflake document METADATA$START_SCAN_TIME data type is also TIMESTAMP_LTZ. convert to TIMESTAMP (TIMESTAMP_NTZ) would still cause problem like origin post.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants