Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Cache the BigQuery table definition instead of calling tables.get() from every worker #34076

Open
2 of 17 tasks
PraveenAkunuru opened this issue Feb 26, 2025 · 2 comments · May be fixed by #34135
Open
2 of 17 tasks

[Feature Request]: Cache the BigQuery table definition instead of calling tables.get() from every worker #34076

PraveenAkunuru opened this issue Feb 26, 2025 · 2 comments · May be fixed by #34135
Assignees
@Strikerrx01
Labels
good first issue new feature P2 python

Comments

@PraveenAkunuru
Copy link

PraveenAkunuru commented Feb 26, 2025 •
edited by liferoad
Loading

What would you like to happen?

When calling BigQuery's tables.get method, today, we are calling it from every Dataflow worker. This is resulting in customers hitting Quota issues. Please can we cache the Table definition and re-use for all workers instead of calling the method from every worker.

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

beam/sdks/python/apache_beam/io/gcp/bigquery.py

Lines 902 to 903 in 9a1e7bf

table = bq.get_table(
table_ref.projectId, table_ref.datasetId, table_ref.tableId)

beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py

Line 565 in 9a1e7bf

response = self.client.tables.Get(request)

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@tamannakakkar93
Copy link

.take-issue

@Strikerrx01
Copy link

.take-issue

@Strikerrx01 Strikerrx01 mentioned this issue Feb 27, 2025
Closed
10 tasks
@Strikerrx01 Strikerrx01 linked a pull request Mar 1, 2025 that will close this issue
Open
Strikerrx01 added a commit to Strikerrx01/beam that referenced this issue Mar 4, 2025
@Strikerrx01
[Python] Fix BigQuery table definition caching implementation
8c5d460 
- Fix field_type.upper() redundant call in beam_row_from_dict
- Fix temp dataset handling logic in BigQueryWrapper.__init__
- Add comprehensive test coverage for table definition caching
- Add proper thread safety with RLock
- Add documentation and comments

Fixes apache#34076
Strikerrx01 added a commit to Strikerrx01/beam that referenced this issue Mar 4, 2025
@Strikerrx01
[Python] Improve BigQuery table definition caching implementation
7128d38 
- Reorder methods for better readability (get_query_location before get_table)
- Use _get_temp_dataset() instead of inlining logic
- Add comprehensive test coverage for temp dataset handling
- Fix method organization as per review feedback

Part of apache#34076
Strikerrx01 added a commit to Strikerrx01/beam that referenced this issue Mar 9, 2025
@Strikerrx01
[Python] Add caching for BigQuery table definitions to reduce API cal…
c496455 
…ls - Fixes apache#34076
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Strikerrx01 Strikerrx01

Labels
good first issue new feature P2 python
Projects
None yet
Milestone
No milestone
4 participants
@PraveenAkunuru @liferoad @tamannakakkar93 @Strikerrx01