Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Partitioning Improvements #887

Merged

Conversation

Pipboyguy
Copy link
Collaborator

@Pipboyguy Pipboyguy commented Jan 11, 2024

Adds support to partition on datetime, date and bigint types.

Changes

  • Control flow to test each data type and the appropriate partition logic to apply.
  • Added tests for each type.
  • My research suggests bigquery can't partition on FLOAT64, hence I removed this claim from the docs.
  • Minor grammar fixes, (subjective) code cleanup

Caveats

It is worth nothing that in order to partition on INT64 we need to know the bounds upfront. Please see the workaround and limitations that come with it.

Fixes #852

Pipboyguy and others added 29 commits January 7, 2024 01:16
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
@Pipboyguy Pipboyguy requested a review from sh-rp January 11, 2024 20:20
@Pipboyguy Pipboyguy self-assigned this Jan 11, 2024
@Pipboyguy Pipboyguy added the bug Something isn't working label Jan 11, 2024
@Pipboyguy Pipboyguy linked an issue Jan 11, 2024 that may be closed by this pull request
Copy link

netlify bot commented Jan 11, 2024

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit be60ecf
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/65a7f3d9fc58420009abb502
😎 Deploy Preview https://deploy-preview-887--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
…nation' and 'devel' of github.com:dlt-hub/dlt into 852-cannot-create-partitioned-table-in-bigquery-destination
@Pipboyguy Pipboyguy marked this pull request as draft January 15, 2024 11:33
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
@Pipboyguy Pipboyguy marked this pull request as ready for review January 17, 2024 15:20
@Pipboyguy Pipboyguy requested a review from sh-rp January 17, 2024 15:20
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Copy link
Collaborator

@sh-rp sh-rp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@sh-rp sh-rp merged commit 3921d61 into devel Jan 18, 2024
56 checks passed
@sh-rp sh-rp deleted the 852-cannot-create-partitioned-table-in-bigquery-destination branch January 18, 2024 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Cannot create partitioned table in BigQuery destination
3 participants