Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: improve json to proto conversion by caching schema #2179

Merged
merged 18 commits into from Jul 13, 2023
Merged

feat: improve json to proto conversion by caching schema #2179

merged 18 commits into from Jul 13, 2023

Conversation

GaoleMeng
Copy link
Contributor

By caching schema across rows during conversion, we could save ~2x conversion speed for wide schema and when the row count is large

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

GaoleMeng and others added 16 commits March 30, 2023 10:52
@GaoleMeng
feat: add public api to stream writer to set the maximum wait time
1726c85
@GaoleMeng
Merge branch 'googleapis:main' into main
9b9e420
@gcf-owl-bot @GaoleMeng
🦉 Updates from OwlBot post-processor
3f0c5eb 
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
@GaoleMeng
modify back the readme change from owl post processor
a448c8b
@gcf-owl-bot
🦉 Updates from OwlBot post-processor
b15b140 
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
@GaoleMeng
Merge branch 'googleapis:main' into main
4f5c0ad
@GaoleMeng
Merge branch 'googleapis:main' into main
aa6d9c6
@GaoleMeng
Merge branch 'googleapis:main' into main
7055043
@GaoleMeng
Merge branch 'googleapis:main' into main
ea2e8fb
@GaoleMeng
fix: Reduce the timeout to 5 minutes for the requests wait time in qu…
affa11a 
…eue.

Since in write api server side we have total timeout of 2 minutes, it
does not make sense to wait 15 minutes to determine whether we have met
dead connection, let's reduce the timeout here
@gcf-owl-bot
🦉 Updates from OwlBot post-processor
60869f7 
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
@GaoleMeng
Merge branch 'googleapis:main' into main
fc565e0
@GaoleMeng
Merge branch 'googleapis:main' into main
e86b484
@GaoleMeng
fix: 1.disable refresh of stream writer when the table schema is
687ef3a 
explicitly provided 2. fix location string matching for multiplexing
@GaoleMeng
Merge branch 'googleapis:main' into main
de1a1b0
@GaoleMeng
Merge branch 'googleapis:main' into main
b5ebe90
@GaoleMeng GaoleMeng requested a review from a team as a code owner July 7, 2023 23:10
@GaoleMeng GaoleMeng requested a review from tswast July 7, 2023 23:10
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Jul 7, 2023
@GaoleMeng GaoleMeng requested a review from yirutang July 7, 2023 23:10
yirutang
yirutang reviewed Jul 10, 2023
View reviewed changes
...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java Outdated Show resolved Hide resolved
...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java Outdated Show resolved Hide resolved
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ToProtoConverter.java Show resolved Hide resolved
...gquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessageTest.java Show resolved Hide resolved
...gquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessageTest.java Outdated Show resolved Hide resolved
google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/Exceptions.java Outdated Show resolved Hide resolved
google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/Exceptions.java Show resolved Hide resolved
...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java Outdated Show resolved Hide resolved
@GaoleMeng GaoleMeng requested a review from a team as a code owner July 11, 2023 01:20
@GaoleMeng GaoleMeng force-pushed the main branch 5 times, most recently from fb52950 to 403c6b3 Compare July 11, 2023 05:35
@GaoleMeng GaoleMeng force-pushed the main branch 11 times, most recently from a6f49fa to 561cdf1 Compare July 11, 2023 23:54
@GaoleMeng GaoleMeng force-pushed the main branch 3 times, most recently from ec44394 to a856a68 Compare July 12, 2023 23:03
@GaoleMeng
feat: improve json stream writer json to proto conversion speed by
c307e4b 
caching the schema. This will introduce approximately 2x improvement to
append speed
@GaoleMeng GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 12, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 12, 2023
@GaoleMeng GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 13, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 13, 2023
@GaoleMeng GaoleMeng added external This issue is blocked on a bug with the actual product. owlbot:run Add this label to trigger the Owlbot post processor. labels Jul 13, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 13, 2023
@GaoleMeng GaoleMeng merged commit afc550a into googleapis:main Jul 13, 2023
17 checks passed
@release-please release-please bot mentioned this pull request Jul 13, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yirutang yirutang yirutang approved these changes

@tswast tswast Awaiting requested review from tswast tswast is a code owner automatically assigned from googleapis/api-bigquery

Assignees
No one assigned
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. external This issue is blocked on a bug with the actual product. size: l Pull request size is large.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@GaoleMeng @yirutang