[BEAM-13945] (FIX) Update Java BQ connector to support new JSON type #17492
Conversation
|
Can one of the admins verify this patch? |
1 similar comment
|
Can one of the admins verify this patch? |
|
Run PostCommit_Java_DataflowV2 |
|
retest this please |
|
R: @pabloem Fixing the way we look for JSON column types in the schema. |
| + "json-data#ingest_json_data"); | ||
|
|
||
| if (field.getString("type").equals("STRUCT")) { | ||
| validateNoJsonTypeInSchema(field); |
There was a problem hiding this comment.
Looks basically good. Are we at all concerned with recursion depth here?
There was a problem hiding this comment.
BigQuery docs show that the limit is 15 nested structs. If the user sticks to the BigQuery guidelines it shouldn't be a problem.
Ofc this doesn't stop the user from making a mistake. Do you think it's necessary for us to handle that here?
There was a problem hiding this comment.
Probably not, if the bq limit is 15. LGTM
|
Run Java PreCommit |
|
@ahmedabu98 I think this change breaks this test: https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4995/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryServicesImplTest/testInsertWithinRowCountLimits/ You can run it locally with |
|
@pabloem the |
|
What is the next step on this PR? |
|
@pabloem precommit test fails are unrelated |
|
retest this please |
|
Run Kotlin_Examples PreCommit |
|
Run Java PreCommit |
2 similar comments
|
Run Java PreCommit |
|
Run Java PreCommit |
|
Run Java PostCommit |
|
Run Java PreCommit |
|
LGTM. |
|
I believe this is the cause for the permared Java PreCommit which is failing due to: |
…e Java BQ connector to support new JSON type " This reverts commit 12be69d.
…BEAM-13945] (FIX) Update Java… Revert "Merge pull request #17492 from [BEAM-13945] (FIX) Update Java…
…X) Update Java BQ connector to support new JSON type "" This reverts commit 688d183.
… for BQIO * Revert "Revert "Merge pull request #17492 from [BEAM-13945] (FIX) Update Java BQ connector to support new JSON type "" This reverts commit 688d183. * cherry-pick @johnjcasey fix
Follow up to #17209.
More thorough way to look for JSON column type in schema and throw an error when the user is writing to BigQuery with the FILE_LOADS write method. The error is thrown because JSON insertion is currently not supported with batch loads.