Feature Request: Add support for 'Allow Large Results' to BigQuery connector

[jreback]

xref pandas-dev/pandas#10474

gbq.py currently returns an error if the result of a query is what Google considers to be 'Large'. The google api allows jobs to be sent with a flag to allow large results. It would be very beneficial to provide this as an option in the BigQuery connector.

[tswast]

People have posted some pretty elaborate workarounds on StackOverflow. https://stackoverflow.com/questions/34201923/python-bigquery-allowlargeresults-with-pandas-io-gbq/34203369

[jasonqng]

Can now do this in open PR by passing it via configuration setting: #25

read_gbq(sql, configuration={"allow_large_results":True})

This uses new google-cloud-python api.

[yahyamortassim]

@jasonqng you also have to add destinationTable.

[tswast]

Actually, I think the current API does support this, even without #25. The read_gbq function accepts a configuration keyword argument which is a job configuration resource, so to allow large results one would do either

Standard SQL:

pd.read_gbq(
    query,
    'my-project',
    dialect='standard',
    configuration={
        'query': {
            'destinationTable': {
                'projectId': 'my-project',
                'datasetId': 'mydataset',
                'tableId': 'mytable'
             }
        }
    })

Legacy SQL:

pd.read_gbq(
    query,
    'my-project',
    dialect='standard',
    configuration={
        'query': {
            'allowLargeResults': True,
            'destinationTable': {
                'projectId': 'my-project',
                'datasetId': 'mydataset',
                'tableId': 'mytable'
             }
        }
    })

Admittedly this is a bit onerous to do. We may wish to provide a friendlier interface for options such as these.

[Gitman-code]

The updated answer on stack overflow suggests just using dialect='standard' like tswast did but more simply as
pd.read_gbq(query, 'my-super-project', dialect='standard')

and notes AllowLargeResults: For standard SQL queries, this flag is ignored and large results are always allowed. This worked for me but maybe it is not generic.

[tswast]

I'm glad that worked for you. I believe there may be some size threshold where a destination table is required, even with standard SQL, but perhaps the threshold is larger than it was for legacy SQL.

[tswast]

Closing, as this can be passed in via the configuration argument to read_gbq.