Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add schema update support in JsonStreamWriter #1447

Merged
merged 12 commits into from
Dec 29, 2021
Merged

feat: add schema update support in JsonStreamWriter #1447

merged 12 commits into from
Dec 29, 2021

Conversation

stephaniewang526
Copy link
Contributor

No description provided.

@stephaniewang526 stephaniewang526 requested review from a team and loferris December 15, 2021 20:49
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Dec 15, 2021
@stephaniewang526 stephaniewang526 requested review from yirutang and removed request for loferris December 15, 2021 20:49
@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 16, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 16, 2021
@stephaniewang526 stephaniewang526 requested a review from a team as a code owner December 16, 2021 12:30
yirutang
yirutang reviewed Dec 16, 2021
View reviewed changes
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved
...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java Outdated Show resolved Hide resolved
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java Show resolved Hide resolved
...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java Outdated Show resolved Hide resolved
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved
yirutang
yirutang reviewed Dec 22, 2021
View reviewed changes
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java Show resolved Hide resolved
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java Outdated Show resolved Hide resolved
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java
try {
response2 = jsonStreamWriter.append(jsonArr2);
next.getAndIncrement();
} catch (IOException | DescriptorValidationException e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully with ignore unknown fields, we could keep writing ccc and ddd to the table and expect ddd eventually to show up in the table. This is good for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Will keep this in mind.

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java Show resolved Hide resolved
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java Show resolved Hide resolved
stephaniewang526 and others added 4 commits December 23, 2021 11:24
@stephaniewang526
Merge branch 'main' into schema-update
f43ac20
@stephaniewang526
update getUpdatedSchema() in StreamWriter.java to use lock; update co…
d3bf37f 
…ncurrent test case
@stephaniewang526
Merge branch 'main' into schema-update
1802c98 
# Conflicts:
#	google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java
@stephaniewang526
simplify logic
ee1d976
@stephaniewang526 stephaniewang526 added automerge Merge the pull request once unit tests and other checks pass. owlbot:run Add this label to trigger the Owlbot post processor. labels Dec 24, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 24, 2021
@stephaniewang526 stephaniewang526 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed automerge Merge the pull request once unit tests and other checks pass. labels Dec 24, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 24, 2021
@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 28, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 28, 2021
shollyman
shollyman approved these changes Dec 28, 2021
View reviewed changes
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java
// Create a new underlying StreamWriter with the updated TableSchema and Descriptor
this.streamWriter =
streamWriterBuilder
.setWriterSchema(ProtoSchemaConverter.convert(this.descriptor))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to propagate anything else to the new writer like stream id, existing trace id, etc or does it all come from properties stored on the JsonStreamWriter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, those things are needed and are already being populated in the streamWriterBuilder.

@stephaniewang526 stephaniewang526 merged commit 973afcc into googleapis:main Dec 29, 2021
@stephaniewang526 stephaniewang526 mentioned this pull request Dec 29, 2021
Merged
4 tasks
gcf-merge-on-green bot pushed a commit that referenced this pull request Jan 4, 2022
@release-please
chore: release 2.8.0 (#1448)
2b756e0 
🤖 I have created a release \*beep\* \*boop\*
---
## [2.8.0](https://www.github.com/googleapis/java-bigquerystorage/compare/v2.7.0...v2.8.0) (2021-12-30)


### Features

* Add ignoreUnknownField support in JsonWriter ([#1455](https://www.github.com/googleapis/java-bigquerystorage/issues/1455)) ([4616adb](https://www.github.com/googleapis/java-bigquerystorage/commit/4616adb30d07d7be522b74837adae080eca103a6))
* Add reconnect support to v1 client lib. ([#1446](https://www.github.com/googleapis/java-bigquerystorage/issues/1446)) ([a5157fa](https://www.github.com/googleapis/java-bigquerystorage/commit/a5157faf37354ee48a0af930a68edf4823b5be6f))
* add schema update support in JsonStreamWriter ([#1447](https://www.github.com/googleapis/java-bigquerystorage/issues/1447)) ([973afcc](https://www.github.com/googleapis/java-bigquerystorage/commit/973afccc22507b8109b3920d3fc183692dcc66c2))
* support `append()` without offset in `StreamWriter` ([#1452](https://www.github.com/googleapis/java-bigquerystorage/issues/1452)) ([cb8b0ad](https://www.github.com/googleapis/java-bigquerystorage/commit/cb8b0ad25fbf07c3509b3d94244a999b9cb90e54))


### Bug Fixes

* allow all fields to be null values ([#1450](https://www.github.com/googleapis/java-bigquerystorage/issues/1450)) ([e47ac79](https://www.github.com/googleapis/java-bigquerystorage/commit/e47ac791765486f02f027d3c440645ef2a379ba9))
* finish BIGNUMERIC support ([#1449](https://www.github.com/googleapis/java-bigquerystorage/issues/1449)) ([d9d51cd](https://www.github.com/googleapis/java-bigquerystorage/commit/d9d51cd203733a972c80e312e9e279b50fb260fa))
* fix a NullPtr when user closes a writer without connection being ever established ([#1454](https://www.github.com/googleapis/java-bigquerystorage/issues/1454)) ([b774f5d](https://www.github.com/googleapis/java-bigquerystorage/commit/b774f5d0d00275278708d1299b4912b455be4bed))
* update storageError support due to server side enhancement ([#1456](https://www.github.com/googleapis/java-bigquerystorage/issues/1456)) ([6243ad5](https://www.github.com/googleapis/java-bigquerystorage/commit/6243ad5cba61d4dae7f4ceb60b09c625e7589215))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@release-please release-please bot mentioned this pull request Jan 4, 2022
Closed
@bphenriques bphenriques mentioned this pull request Jan 6, 2022
Closed
@damccorm damccorm mentioned this pull request Jun 4, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yirutang yirutang yirutang approved these changes

@shollyman shollyman shollyman approved these changes

@yayi-google yayi-google Awaiting requested review from yayi-google

Assignees
No one assigned
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@stephaniewang526 @shollyman @yirutang @yoshi-kokoro