[BEAM-876] Support schemaUpdateOption in BigQueryIO

[ziel]

[BEAM-876] Support schemaUpdateOption in BigQueryIO

This adds schemaUpdateOptions to BigQueryIO.Write so that one can specify these for BiqQuery when writing in batch mode.

Usage example:

Write<TableRow> writer = BigQueryIO.writeTableRows()
  .to("name-goes-here")
  .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
  .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
  .withSchemaUpdateOptions(EnumSet.of(BigQueryIO.Write.SchemaUpdate.Option.ALLOW_FIELD_ADDITION));

Implementation Notes

Hi, hello:
Hi all. I haven't contributed to this code base before, and am not super familiar with it. Style advice and such is super welcome.

Load vs Query Jobs in BigQuery:
BigQuery supports schema update side effects for load and query jobs. This implements support for load jobs only. In the context of Apache Beam, it doesn't seem like there's much utility to supporting queries which write to a table directly (as opposed to reading via a query and writing later in the pipeline). (Open to discussion/correction of course).

Avoiding Changing WriteTables Constructor:
I've added this to WriteTables with a withSchemaUpdateOptions method so as not to perturb the public constructor with an optional parameter. There's some awkwardness there -- but it seemed preferable to breaking the api or adding a secondary constructor.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[ziel]

R: @lukecwik @chamikaramj

[chamikaramj]

Thanks for the contribution.

R: @pabloem will you be able to review ?

[pabloem]

Yes, I'll be glad to review.

[pabloem]

Thanks for the contribution! BigQuery adds features often, and it's great to support them.
I think this looks fine to me, except for the one comment. Also, have you tried running this transform?

It may be good to have an integration test, just to be sure that it'll work fine.... do you know how to add one? I wouldn't say it's a must, but it would be amazing to have one.

[pabloem]

Also, sorry about the delay. Don't hesitate to ping on the PR / email me.

[pabloem]

Should I review once more?

[ziel]

I haven't had a chance to write the integration test yet... but was hoping to take a shot at it in the next week or so.

[pabloem]

thanks!

[ziel]

@pabloem I found some time to write up an integration test.

I'm seeing a failing check (org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety) which I think may be unrelated? I may be missing the connection though :-S There are a number of lines like this in the output like: java.lang.SecurityException: User name [test_user] or password is invalid. ...so I suspect this may be some sort of CI setup thing..

[pabloem]

Run Java PreCommit

[ziel]

I'm seeing a timeout fail here: https://builds.apache.org/job/beam_PreCommit_Java_Commit/8731/testReport/org.apache.beam.runners.flink.translation.wrappers.streaming.io/UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest/testWatermarkEmission_numTasks___4__numSplits_2_/

going to retry

[ziel]

Run Java PreCommit

[pabloem]

Thanks for writing the test! That's going above and beyond : ) - the changes LGTM. I'll try to run the test to check.

[pabloem]

r: @chamikaramj would you like to do an extra pass?

[pabloem]

Okay, bringing in now. Thanks @ziel