BigQuery Storage API: RuntimeError: ("can't start new thread", 'occurred at index 0') #9
Labels
api: bigquery
Issues related to the googleapis/python-bigquery API.
needs more info
This issue needs more information from the customer to proceed.
type: question
Request for information or clarification. Not an issue.
Issue
I am running a spark script that needs to perform a
count(*)
query 30x for every row in a dataframe. The dataframe on average has 25000 rows, which means after completing the script should have made 750000 requests/queries to the BigQuery table.For some reason with a large amount of executors I ran into the
RuntimeError
detailed in the stacktrace below, where it seems the google api core is unable to create a bigquery client. Is this because my script is creating too many clients? Apologies if my code is incorrect or I am using the client wrong, I am new to BigQuery and have not used this api before. What would be the best way to use the BIgQuery storage api in this use case?Environment details
Steps to reproduce
Code Example Of PySpark Script
Stack trace
The text was updated successfully, but these errors were encountered: