Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Add max_results to read_gbq #34639

Closed
charlesdong1991 opened this issue Jun 8, 2020 · 1 comment · Fixed by #34641
Closed

ENH: Add max_results to read_gbq #34639

charlesdong1991 opened this issue Jun 8, 2020 · 1 comment · Fixed by #34641
Labels
Enhancement good first issue
Milestone
1.1

Comments

@charlesdong1991
Copy link
Member

charlesdong1991 commented Jun 8, 2020 •
edited
Loading

For queries for which the purpose is to create/replace tables: CREATE OR REPLACE TABLE, if the table is huge, it will eat up a lot of memory currently using read_gbq.

And it turns out that since pandas-gbq==0.12.0, a new argument max_results was implemented to limit the number of rows in result dataframe for e.g. setting it to 0 for DDL queries, which will help save memory.

So it will be nice for pandas to add it for read_gbq as pandas-gbq does.
@charlesdong1991 charlesdong1991 added Enhancement Needs Triage Issue that has not been reviewed by a pandas team member good first issue IO Google and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 8, 2020
@charlesdong1991 charlesdong1991 added this to the Contributions Welcome milestone Jun 8, 2020
@JohnPaton
Copy link
Contributor

I can take a swing at this!

JohnPaton added a commit to JohnPaton/pandas that referenced this issue Jun 8, 2020
@JohnPaton
Add max_results kwarg to read_gbq (pandas-dev#34639)
03e2458 
Since max_results is a new kwarg (added in pandas-gbq 0.12.0), it
is handled and tested in the same way as use_bqstorage_api,
using the "new kwargs" mechanism to maintain backwards
compatibility with older pandas-gbq versions.
@JohnPaton JohnPaton mentioned this issue Jun 8, 2020
Merged
5 tasks
@jreback jreback modified the milestones: Contributions Welcome, 1.1 Jun 9, 2020
jreback pushed a commit that referenced this issue Jun 9, 2020
@JohnPaton
Add max_results kwarg to read_gbq (#34639) (#34641)
b024e93 
Since max_results is a new kwarg (added in pandas-gbq 0.12.0), it
is handled and tested in the same way as use_bqstorage_api,
using the "new kwargs" mechanism to maintain backwards
compatibility with older pandas-gbq versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Enhancement good first issue
Projects
None yet
Milestone
1.1
Development

Successfully merging a pull request may close this issue.

ENH: Add max_results kwarg to read_gbq JohnPaton/pandas
3 participants
@jreback @charlesdong1991 @JohnPaton