Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor write client code to use the generated instead of the handwritten BigQuery client #1249

Closed
stephaniewang526 opened this issue Aug 19, 2021 · 0 comments · Fixed by #1295
Closed
Assignees
Labels
api: bigquerystorage type: cleanup

Comments

@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Aug 19, 2021

The fact that google-cloud-bigquerystorage Write client (v1beta2) is using google-cloud-bigquery as a dependency means it would introduce a cyclic dependency when we integrate it into google-cloud-bigquery for high throughput Write in the future. This is also currently blocking us from using google-cloud-bigquerystorage client in google-cloud-bigquery for the Read API features (v1). We have found a temporary workaround to unblock ourselves on the Read side but this refactoring is necessary before we GA this client.

Specifically, 3 classes require full refactoring:

  1. /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:
  2. /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:
  3. /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/JsonStreamWriter.java:

Full trace:

 Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project google-cloud-bigquerystorage: Compilation failure: Compilation failure: 
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[18,33] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[19,33] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[20,33] cannot find symbol
[ERROR]   symbol:   class StandardSQLTypeName
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[25,36] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[31,31] cannot find symbol
[ERROR]   symbol:   class StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[53,48] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[67,53] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[18,33] cannot find symbol
[ERROR]   symbol:   class BigQuery
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[19,33] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[20,33] cannot find symbol
[ERROR]   symbol:   class LegacySQLTypeName
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[21,33] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[22,33] cannot find symbol
[ERROR]   symbol:   class Table
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[23,33] cannot find symbol
[ERROR]   symbol:   class TableId
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[24,41] package com.google.cloud.bigquery.testing does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[46,11] cannot find symbol
[ERROR]   symbol:   class BigQuery
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[74,31] cannot find symbol
[ERROR]   symbol:   class BigQuery
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[99,49] cannot find symbol
[ERROR]   symbol:   class BigQuery
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[104,11] cannot find symbol
[ERROR]   symbol:   class TableId
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[257,47] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[310,7] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[409,7] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/JsonStreamWriter.java:[22,33] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: package com.google.cloud.bigquery
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/JsonStreamWriter.java:[208,62] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[27,16] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[28,16] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[29,16] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[44,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[43,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[42,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[41,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[40,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[39,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[38,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[37,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[36,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[35,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[34,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[33,16] cannot find symbol
[ERROR]   symbol:   variable StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[32,32] cannot find symbol
[ERROR]   symbol:   class StandardSQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.BQV2ToBQStorageConverter
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/BQV2ToBQStorageConverter.java:[70,46] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[86,7] cannot find symbol
[ERROR]   symbol:   class RemoteBigQueryHelper
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[86,45] cannot find symbol
[ERROR]   symbol:   variable RemoteBigQueryHelper
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[109,12] cannot find symbol
[ERROR]   symbol:   variable TableId
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[264,12] cannot find symbol
[ERROR]   symbol:   variable REPEATED
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[274,12] cannot find symbol
[ERROR]   symbol:   variable REQUIRED
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[284,12] cannot find symbol
[ERROR]   symbol:   variable NULLABLE
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[318,5] cannot find symbol
[ERROR]   symbol:   class LegacySQLTypeName
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[365,13] cannot find symbol
[ERROR]   symbol:   variable Schema
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[421,10] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[442,10] cannot find symbol
[ERROR]   symbol:   class Field
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[450,59] package Field does not exist
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[512,5] cannot find symbol
[ERROR]   symbol:   class TableId
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[513,5] cannot find symbol
[ERROR]   symbol:   class Table
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] /usr/local/google/home/stephwang/Desktop/stephdev/java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/SchemaCompatibility.java:[514,5] cannot find symbol
[ERROR]   symbol:   class Schema
[ERROR]   location: class com.google.cloud.bigquery.storage.v1beta2.SchemaCompatibility
[ERROR] -> [Help 1]
@product-auto-label product-auto-label bot added the api: bigquerystorage label Aug 19, 2021
@stephaniewang526 stephaniewang526 changed the title fix: refactor write client code to use the generated instead of the handwritten BigQuery client Refactor write client code to use the generated instead of the handwritten BigQuery client Aug 24, 2021
@meredithslota meredithslota added the type: cleanup label Aug 28, 2021
@stephaniewang526 stephaniewang526 self-assigned this Sep 1, 2021
stephaniewang526 added a commit that referenced this issue Sep 9, 2021
Towards #1249
BQToBQStorageSchemaConverter.java will go into google-cloud-bigquery eventually. For now it acts as a resource for developers who want to use default stream to write.
gcf-merge-on-green bot pushed a commit that referenced this issue Sep 10, 2021
Towards #1249
BQToBQStorageSchemaConverter.java will go into google-cloud-bigquery eventually. For now it acts as a resource for developers who want to use default stream to write.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquerystorage type: cleanup
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants