Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Ensure BigQuery queries are executed in UTC #1174

Merged
merged 3 commits into from
Jun 21, 2024

Conversation

nj1973
Copy link
Contributor

@nj1973 nj1973 commented Jun 20, 2024

This PR ensures BigQuery queries are in UTC which overrides any local project settings.

I don't think we can easily have automated tests for this, unless we create a new internal project and set it to non-UTC.

Tested manually with:

ALTER PROJECT `my-project-01`
SET OPTIONS (`region-eu.default_time_zone` = 'Europe/Helsinki');

Test using our standard table then failed:

data-validation -v validate row -sc=snowflake -tc=bq -tbls=pso_data_validator.dvt_core_types --primary-keys=id --concat='id,col_datetime'

source_agg_value     │ target_agg_value
═════════════════════╪═════════════════════
11970-01-01 00:00:01 │ 11969-12-31 22:00:01
─────────────────────┼─────────────────────
21970-01-02 00:00:02 │ 21970-01-01 22:00:02
─────────────────────┼─────────────────────
31970-01-03 00:00:03 │ 31970-01-02 22:00:03

Switched to this branch and the command above succeeded.

source_agg_value     │ target_agg_value
═════════════════════╪═════════════════════
11970-01-01 00:00:01 │ 11970-01-01 00:00:01
─────────────────────┼─────────────────────
21970-01-02 00:00:02 │ 21970-01-02 00:00:02
─────────────────────┼─────────────────────
31970-01-03 00:00:03 │ 31970-01-03 00:00:03

@nj1973 nj1973 requested a review from a team as a code owner June 20, 2024 10:10
@nj1973 nj1973 linked an issue Jun 20, 2024 that may be closed by this pull request
@nj1973
Copy link
Contributor Author

nj1973 commented Jun 21, 2024

/gcbrun

Copy link
Collaborator

@nehanene15 nehanene15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nj1973 nj1973 merged commit a77cdd9 into develop Jun 21, 2024
4 of 5 checks passed
@nj1973 nj1973 deleted the 1160-bigquery-timezone-control branch June 21, 2024 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snowflake to BigQuery comparison bugs
2 participants