Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: use protobuf for metadata to reduce type conversions #325

Merged
merged 3 commits into from
Apr 29, 2021

Conversation

larkee
Copy link
Contributor

@larkee larkee commented Apr 28, 2021

To fix performance issues in #207, the streaming logic switched to using the underlying protobuf objects instead of the proto-plus objects which have an type conversion overhead when accessing values. However, the metadata proto was kept in the proto-plus format. This metadata object contains the type information of the different fields which means it is frequently accessed when iterating over arrays. This causes major performance issues for large scale arrays due to the type conversion overhead. Using the underlying protobuf for the metadata removes this overhead and brings the performance in line with v1.17.1.

Fixes #310

@larkee larkee requested a review from a team as a code owner April 28, 2021 07:19
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner API. label Apr 28, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Apr 28, 2021
@larkee larkee added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Apr 28, 2021
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Apr 29, 2021
Copy link
Contributor

@vi3k6i5 vi3k6i5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@larkee larkee merged commit 5110b9b into googleapis:master Apr 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/python-spanner API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Performance issues for queries returning larger arrays
4 participants