Skip to content

PLUGIN 517- fix issue with validation of partition field property#553

Merged
CuriousVini merged 1 commit intodata-integrations:developfrom
AdaptiveScale:bugfix/PLUGIN-517
Feb 26, 2021
Merged

PLUGIN 517- fix issue with validation of partition field property#553
CuriousVini merged 1 commit intodata-integrations:developfrom
AdaptiveScale:bugfix/PLUGIN-517

Conversation

@ardian4
Copy link
Copy Markdown
Contributor

@ardian4 ardian4 commented Feb 15, 2021

Fixed: Validate partition field property when partition type is integer:

1. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table does not exist
2. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table exist
3. Throw validation issue in prepare run stage when partition field or partition type is macro and partition field is empty

Jira Ticket: https://cdap.atlassian.net/browse/PLUGIN-517

@google-cla
Copy link
Copy Markdown

google-cla Bot commented Feb 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

Copy link
Copy Markdown
Contributor

@flakrimjusufi flakrimjusufi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

}
PartitionType partitioningType = getPartitioningType();
if (Strings.isNullOrEmpty(columnName)) {
if (partitioningType == PartitionType.INTEGER) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this check not require if partitioningType == PartitionType.TIME?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can we add a unit test for this in BigQuerySinkTest?

Copy link
Copy Markdown
Contributor Author

@ardian4 ardian4 Feb 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BigQuery has 3 types of partitions:

  1. Ingestion time: Tables are partitioned based on the data's ingestion (load) time or arrival time.
  2. Date/timestamp/datetime: Tables are partitioned based on a TIMESTAMP, DATE, or DATETIME column.
  3. Integer range: Tables are partitioned based on an integer column.

https://cloud.google.com/bigquery/docs/partitioned-tables

In Sink for Time Partition if there is no field specified, the table will partition by ingestion time, so partition columnName can be empty.

We have added a unit test where partition type is TIME and the partition field is provided

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for explaining!

@google-cla
Copy link
Copy Markdown

google-cla Bot commented Feb 24, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

Copy link
Copy Markdown
Contributor

@rmstar rmstar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, please squash commits

1. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table does not exist
2. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table exist
3. Throw validation issue in prepare run stage when partition field or partition type is macro and partition field is empty
@google-cla
Copy link
Copy Markdown

google-cla Bot commented Feb 26, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@CuriousVini CuriousVini added the build Trigger unit test build label Feb 26, 2021
@CuriousVini
Copy link
Copy Markdown
Member

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants