PLUGIN-557: BigQuerySink fix for upsert operation in existing table but in different project

[flakrimjusufi]

BigQuery sink pipeline fails doing upsert operation to existing table in different project. The table is partitioned by ingestion-time (day). Partition field is left empty.

JIRA: https://cdap.atlassian.net/browse/PLUGIN-557

[ardian4]

lgtm

[CuriousVini]

what if config.getProject() is provided? In that case we will rely on config.getDatasetProject()?

[flakrimjusufi]

If config.getProject() and config.getDatasetProject() are provided, then datasetProject will get priority.

If datasetProject is not defined, then config.getProject() will be set up in configuration.
That's how it is defined in AbstractBigQuerySinkConfig

google-cloud/src/main/java/io/cdap/plugin/gcp/bigquery/sink/AbstractBigQuerySinkConfig.java

Lines 114 to 120 in 86f4b91

	@Nullable
	public String getDatasetProject() {
	if (GCPConfig.AUTO_DETECT.equalsIgnoreCase(datasetProject)) {
	return ServiceOptions.getDefaultProjectId();
	}
	return Strings.isNullOrEmpty(datasetProject) ? getProject() : datasetProject;
	}

[CuriousVini]

Is there any unit test or integration test that can be added for this bug?

[flakrimjusufi]

Is there any unit test or integration test that can be added for this bug?

We created this JIRA Ticket https://cdap.atlassian.net/browse/PLUGIN-606 in order to add integration test in the future because currently it would be hard to add without having another project for integration test.

[CuriousVini]

/gcbrun

[CuriousVini]

Has this fix been tested with dataset id being a macro?

[flakrimjusufi]

Is there any unit test or integration test that can be added for this bug?

Has this fix been tested with dataset id being a macro?

Yes, this fix has been tested with dataset id being a macro.
There's no exception or anything like that.
If datasetId is a macro, the validation will not proceed.

This is the block of code that prevents any validation exception if any of the fields contains macro:

google-cloud/src/main/java/io/cdap/plugin/gcp/bigquery/sink/BigQuerySink.java

Lines 251 to 253 in 86f4b91

	if (!config.shouldConnect()) {
	return;
	}

And this is the code of shouldConnect():

google-cloud/src/main/java/io/cdap/plugin/gcp/bigquery/sink/BigQuerySinkConfig.java

Lines 611 to 619 in 86f4b91

	boolean shouldConnect() {
	return !containsMacro(BigQuerySinkConfig.NAME_DATASET) &&
	!containsMacro(BigQuerySinkConfig.NAME_TABLE) &&
	!containsMacro(NAME_SERVICE_ACCOUNT_TYPE) &&
	!(containsMacro(NAME_SERVICE_ACCOUNT_FILE_PATH) || containsMacro(NAME_SERVICE_ACCOUNT_JSON)) &&
	!containsMacro(BigQuerySinkConfig.NAME_PROJECT) &&
	!containsMacro(BigQuerySinkConfig.DATASET_PROJECT_ID) &&
	!containsMacro(BigQuerySinkConfig.NAME_SCHEMA);
	}

[CuriousVini]

/gcbrun