Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery: complex repeated schema fields are slow #6

Closed
grooveygr opened this issue Jan 15, 2020 · 1 comment · Fixed by #26
Closed

BigQuery: complex repeated schema fields are slow #6

grooveygr opened this issue Jan 15, 2020 · 1 comment · Fixed by #26
Assignees
@tswast
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@grooveygr
Copy link
Contributor

When converting rows to JSON, repeated schema fields are deepcopied causing noticeable slowdowns in complex schemas.
Since conversion is performed recursively, complex schemas containing multiple levels of repeated fields (e.g. repeated list of structs each containing multiple fields of repeated structs) are copied multiple times when converting a single row - causing even further slowdown due to redundant copying.

I've created a proposal for removing the deepcopy operation. Will reference relevant PR.
@grooveygr
Copy link
Contributor Author

Just to demonstrate the issue, these are the sort of profiling results one may expect when inserting a row into a complex-schema table:
image

@plamut plamut transferred this issue from googleapis/google-cloud-python Feb 4, 2020
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Feb 4, 2020
@plamut plamut added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Feb 4, 2020
@grooveygr grooveygr mentioned this issue Feb 6, 2020
Merged
@tswast tswast closed this as completed in #26 Oct 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tswast tswast

Labels
api: bigquery Issues related to the googleapis/python-bigquery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

perf: remove redundant array deepcopy grooveygr/python-bigquery
3 participants
@tswast @plamut @grooveygr