Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve streaming performance #240

Merged
merged 2 commits into from Feb 23, 2021
Merged

perf: improve streaming performance #240

merged 2 commits into from Feb 23, 2021

Conversation

larkee
Copy link
Contributor

@larkee larkee commented Feb 22, 2021

This PR fixes performance issues that were introduced during the mirgration to v2 and optimizes several streaming methods.

From v2 onwards, the library uses proto-plus to produce user friendly Python object to represent proto messages. Unfortunately, it significantly impacted the performance of streaming for cases with many columns as the library must unnecessarily parse the types many times. By reverting the streaming changes to use the raw protobuf messages, the streaming speed is greatly improved.

Fixes #207, fixes #234

@product-auto-label product-auto-label bot added the api: spanner label Feb 22, 2021
@google-cla google-cla bot added the cla: yes label Feb 22, 2021
@larkee larkee marked this pull request as ready for review Feb 23, 2021
@larkee larkee requested a review from as a code owner Feb 23, 2021
@larkee larkee merged commit 3e35d4a into googleapis:master Feb 23, 2021
9 checks passed

def _merge_values(self, values):
"""Merge values into rows.
:type values: list of :class:`~google.protobuf.struct_pb2.Value`
:param values: non-chunked values from partial result set.
"""
width = len(self.fields)
print(self.fields)
Copy link

@harmon harmon Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this make it into the final code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner cla: yes
Projects
None yet
3 participants