Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support use of `fields`, and use it to minimize data transfer for some paging/polling operations. #153

Closed
craigcitro opened this issue Apr 22, 2017 · 3 comments

Comments

@craigcitro
Copy link
Collaborator

commented Apr 22, 2017

(This is an offshoot of #58.)

List operations in the BQ API generally return all information about each object by default -- for trying to examine lists of objects, this is great. However, if we're (say) polling for job completion, or looking for a specific table/dataset, it's often wasteful to pull all the data every time. (In particular, it's annoying for manual investigation/debugging.)

IWBN to use fields (which is a global query param in the API) to reduce the set of fields we're pulling: docs.

@hadley

This comment has been minimized.

Copy link
Member

commented Mar 28, 2018

Is this something I should systematically review for every function? Or are there a couple of main offenders? (e.g. wait_for())

@hadley hadley added the api 🕸 label Mar 28, 2018

hadley added a commit that referenced this issue Apr 11, 2018

More news updates
Closes #150. Closes #153.

@hadley hadley closed this in fd8108a Apr 12, 2018

@jennybc

This comment has been minimized.

Copy link
Collaborator

commented Apr 12, 2018

I am also vexed by how to handle fields. In googledrive, we ultimately decided to not handle them, i.e. to use the same fields everywhere (I think the default set?), until that proved to be problematic. So far, so good.

However in googlesheets4, I must handle fields, so am thinking about it once again.

@hadley You closed this with a really huge commit. Did you do anything clever with fields that I should study?

@hadley

This comment has been minimized.

Copy link
Member

commented Apr 12, 2018

I don't think there will be any take aways for google sheets here because the way in which I need to use fields is quite different. For example, most bigquery operations take place by inserting a job into a queue, and then regularly polling until it's done. I'm now using fields so that when I poll I just get whether or not it's complete, not a bunch of other information that I don't use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.