PLUGIN 517- fix issue with validation of partition field property#553
Conversation
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
| } | ||
| PartitionType partitioningType = getPartitioningType(); | ||
| if (Strings.isNullOrEmpty(columnName)) { | ||
| if (partitioningType == PartitionType.INTEGER) { |
There was a problem hiding this comment.
Is this check not require if partitioningType == PartitionType.TIME?
There was a problem hiding this comment.
Also, can we add a unit test for this in BigQuerySinkTest?
There was a problem hiding this comment.
BigQuery has 3 types of partitions:
- Ingestion time: Tables are partitioned based on the data's ingestion (load) time or arrival time.
- Date/timestamp/datetime: Tables are partitioned based on a TIMESTAMP, DATE, or DATETIME column.
- Integer range: Tables are partitioned based on an integer column.
https://cloud.google.com/bigquery/docs/partitioned-tables
In Sink for Time Partition if there is no field specified, the table will partition by ingestion time, so partition columnName can be empty.
We have added a unit test where partition type is TIME and the partition field is provided
There was a problem hiding this comment.
I see, thanks for explaining!
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
rmstar
left a comment
There was a problem hiding this comment.
lgtm, please squash commits
1. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table does not exist 2. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table exist 3. Throw validation issue in prepare run stage when partition field or partition type is macro and partition field is empty
db85ae0 to
8a770a0
Compare
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
|
/gcbrun |
Fixed: Validate partition field property when partition type is integer:
Jira Ticket: https://cdap.atlassian.net/browse/PLUGIN-517