[Bug]: PR 17365 breaks BQ loads in some cases

[steveniemitz]

What happened?

In beam 2.39, this used to work:

BigQueryIO
   .write()
   .optimizedWrites()
   .to(OutputTable)
   .withAvroFormatFunction(abc)
   .withAvroSchemaFactory(xyz)
   .withCreateDisposition(CreateDisposition.CREATE_NEVER)
   .withWriteDisposition(WriteDisposition.WRITE_TRUNCATE))

but now throws an exception if you end up in MultiPartitionsWriteTables:

Caused by: java.lang.IllegalArgumentException: Unless create disposition is CREATE_NEVER, a schema must be specified, i.e. DynamicDestinations.getSchema() may not return null. However, create disposition is CREATE_IF_NEEDED, and org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinationsHelpers$ConstantTableDestinations@52e95e1 returned null for destination tableSpec: <table>
	at org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull(Preconditions.java:436)
	at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:207)

The reason seems like now, the create disposition is set to CreateDisposition.CREATE_IF_NEEDED in WriteTables no matter what the user sets it to.

cc @pabloem @MarcoRob

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

[steveniemitz]

Additionally, the new UpdateSchemaDestination breaks if the source format is set to avro due to it trying to load an empty file (a 0 length file is not a valid avro file).

[pabloem]

@johnjcasey @chamikaramj could you help look into this? I will be on vacation for a couple weeks from today.

[johnjcasey]

ack

[johnjcasey]

Agreed, this looks like that original PR changed how we load dynamic destinations for MultiPartitionsWrite tables. @MarcoRob can you take a look at this?

[MarcoRob]

Agreed, this looks like that original PR changed how we load dynamic destinations for MultiPartitionsWrite tables. @MarcoRob can you take a look at this?

Sure, let me check, tks

[Abacn]

Hi, just curious about what is the current status of this bug? Is the working example fixed by #22390 ? @steveniemitz @MarcoRob