[BEAM-7173] Avoiding schema autodetection by default in WriteToBigQuery #8473
Conversation
|
Run Python PreCommit |
|
Run Python PostCommit |
|
Run Python Postcommit |
| | "WriteWithMultipleDests2" >> beam.io.gcp.bigquery.WriteToBigQuery( | ||
| table=value_provider.StaticValueProvider( | ||
| str, '%s:%s' % (self.project, output_table_2)), | ||
| schema=bigquery.SCHEMA_AUTODETECT, |
There was a problem hiding this comment.
Note that SCHEMA_AUTODETECT does not work if BQ has to ingest bytes data into a table that has bytes field.
Also, should we have a test with and without autodetect behavior?
There was a problem hiding this comment.
There are tests with and without this feature running in that file. WDYT?
There was a problem hiding this comment.
I think that should be fine and we don't need to duplicate every test twice, but you might want to do a one-off run of integration tests where the value of autodetection flag is changed to the opposite value, to make sure there are no surprises.
|
I think we should loop in Cham, who reviewed previous BQ PRs, perhaps once the tests pass. |
|
Run Python PostCommit |
3 similar comments
|
Run Python PostCommit |
|
Run Python PostCommit |
|
Run Python PostCommit |
|
Run Portable_Python PreCommit |
|
Run Python Precommit |
|
Run Python PostCommit |
2 similar comments
|
Run Python PostCommit |
|
Run Python PostCommit |
|
Run Python PostCommit |
|
LGTM, thank you! Please squash commits before the merge. |
2 participants
@tvalentyn
cc: @aaltay