New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GBQ: Handle deprecation of oauth2client.tools.run() #8327

Closed
jtratner opened this Issue Sep 19, 2014 · 14 comments

Comments

Projects
None yet
6 participants
@jtratner
Contributor

jtratner commented Sep 19, 2014

Right now, when you use GbqConnector, you get the following warning:

WARNING:root:This function, oauth2client.tools.run(), and the use of the gflags library are deprecated and will be removed in a future version of the library.

I'm not sure how quickly this deprecation will happen or if there are alternatives to keep things the way that they are; but at least worthwhile to have an issue here to address it (in the future).

As an aside, I've been glancing at the source for the bq client that Google provides as part of Google Cloud SDK and I'm wondering how public the methods are (i.e., if we could adapt some of the code or functions that bq.py uses to authenticate and be able to get credentials set up from gcloud auth login rather than requiring a separate oauth flow). Not sure though.

Again, cc @jacobschaer , I'd be interested in your opinion on whether this is worth investigating at all. I've been finding read_gbq and to_gbq quite useful - thanks for putting it together!

@jreback jreback added the IO Google label Sep 19, 2014

@jacobschaer

This comment has been minimized.

Contributor

jacobschaer commented Sep 20, 2014

Using bq sounds like what we were doing back in v1. In discussions with Google, it seemed that their idea of bq is simply a reference, and the implementation is not to be relied upon in future versions. The decision was made to move away since we had continuous issues with bugs we were finding. At the time, none of the methods were "public" (to the degree with Python allows), and everything was wrapped by Google's apputils making it difficult to access as a library. I'd have to look to see what they intend to use to replace oauth2client.tools.run() - last we were told, Google was committed to a minimum of 1 year support for their API's.

@jtratner

This comment has been minimized.

Contributor

jtratner commented Sep 20, 2014

gotcha - maybe it's just that we need to invoke it differently or something.

On Fri, Sep 19, 2014 at 8:47 PM, Jacob Schaer notifications@github.com
wrote:

Using bq sounds like what we were doing back in v1. In discussions with
Google, it seemed that their idea of bq is simply a reference, and the
implementation is not to be relied upon in future versions. The decision
was made to move away since we had continuous issues with bugs we were
finding. At the time, none of the methods were "public" (to the degree with
Python allows), and everything was wrapped by Google's apputils making it
difficult to access as a library. I'd have to look to see what they intend
to use to replace oauth2client.tools.run() - last we were told, Google
was committed to a minimum of 1 year support for their API's.


Reply to this email directly or view it on GitHub
#8327 (comment).

@azbones

This comment has been minimized.

azbones commented Sep 22, 2014

@jtratner I would be careful with any of the Google client libraries including gcloud. I've been testing a Java library they have which uses gcloud and it apparently uses some custom interval version of boto which conflicted with my desktop python distro's boto library. So, the only way I could get it to work was to run their java code from within a clean python virtual environment. As Jacob said, I would suggest sticking to the APIs.

I'm not sure about the need, but another potential feature for your hackathon might also be some method for authenticating in a server environment. Also, if you want to talk about testing or how we set up the pypandas.t​est@gmail.​com for Oauth, let us know.

@jtratner

This comment has been minimized.

Contributor

jtratner commented Sep 22, 2014

@azbones @jacobschaer I agree - we definitely don't want to rely on some internal and undocumented method to get to authentication.

Right now, it looks like we might actually be able to just pull local credentials like gcloud does, but we could probably just let people specify a path to their credentials file and avoid the need to authorize their credentials to an external application, rather than using Google's helpers for getting to the credentials file. If that's not available, then fall back oauth2client.tools.run

@azbones

This comment has been minimized.

azbones commented Sep 22, 2014

If there is anything we can do to help, just let us know. I think analytics focused services like BQ and pandas are great for developing a powerful analysis workflow.

@jreback

This comment has been minimized.

Contributor

jreback commented Sep 22, 2014

@azbones the immediate issue that prompted this was #8107

seems that setuptools is a run-time dep to figure out the versions of the various gbq deps.

We don't require this on windows (where it was discovered), and its not a listed dep.

Just trying to avoid adding something else.

@azbones

This comment has been minimized.

azbones commented Sep 22, 2014

Ah, got it and that makes sense. I'll try catching up with @jacobschaer also given he was looking into it.

@aginiewicz

This comment has been minimized.

aginiewicz commented Apr 17, 2015

Any news on this one? I'm packager of pandas on Arch Linux and gflags is only dependency left that isn't compatible with python 3 - and seems won't be in any time soon. Even google-api-python-client
now has python 3 version.

@jreback

This comment has been minimized.

Contributor

jreback commented Apr 20, 2015

@aginiewicz IIRC gbq is not tested / enabled for py3 at all. I am not sure that status in general.

cc @jacobschaer
cc @azbones

@azbones

This comment has been minimized.

azbones commented Apr 23, 2015

Hey @jtratner, @jacobschaer and I are going to try to look at the following in the next week:

-options for Google's deprecating oauth2client.tools.run() method
-that feature that I think someone was adding to allow server auth
-python 3 support (we had some concern when we first considered this, but we'll look into again)

@azbones

This comment has been minimized.

azbones commented Apr 23, 2015

@jacobschaer @sean-schaefer @jtratner Also, I just saw #9713, so we can look at testing again as we should catch things like that.

@cgrin

This comment has been minimized.

Contributor

cgrin commented Apr 29, 2015

@azbones Just came over to take a look at this and did a bit of digging. It's actually not a huge change, and I've already written a patch to fix it. The replacement for tools.run is tools.run_flow(). Change the import from

from oauth2client.tools import run

to

from oauth2client.tools import run_flow, argparser

and invoke with

credentials = run_flow(flow, storage, argparser.parse_args([]))

Not sure if I should push this to #9722 or wait for that to accept, and make a new pull for this issue.

@jreback @jacobschaer - Thoughts on how to handle this?

@azbones

This comment has been minimized.

azbones commented Apr 29, 2015

@cgrin Cool- that is the same conclusion we had come to! We'll leave it to you, but let us know if you need any help.

The other feature we were thinking about in gbq was #8489 which is generally focused on an improved auth flow for servers. We haven't researched that one yet, but I think someone had taken a cut at it.

cc: @jacobschaer @sean-schaefer

@jreback jreback added this to the 0.16.1 milestone Apr 30, 2015

jreback added a commit that referenced this issue May 1, 2015

Updates to Google BigQuery connector (#9713, #8327)
commit 3d6fdc8751134f6e5bac700519358cf9a700aba1
Author: Chris Grinolds <cgrinolds@mac.com>
Date:   Wed Apr 29 21:43:10 2015 -0700

    Added missing imports that caused tests to fail

commit 6c345d8dbc0c36bd4369dd220eceb495ab5ff2c6
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 16:23:01 2015 -0700

    Updated test suite to handle changes to gbq.py

commit f43e65f924d07f88bbbe308b9ea396b1f2a720f9
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 08:46:00 2015 -0700

    Updated BigQuery connector to no longer use deprecated ```oauth2client.tools.run() (#8327)

commit ea92e200ff341a3025b06868a565b1eea506c4c2
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 08:43:43 2015 -0700

    Import BigQuery dependencies on a per-method basis (#9713)
@jreback

This comment has been minimized.

Contributor

jreback commented May 1, 2015

closed by #9722

@jreback jreback closed this May 1, 2015

CoolRanch29 added a commit to CoolRanch29/pandas that referenced this issue Jun 3, 2015

Updates to Google BigQuery connector (pandas-dev#9713, pandas-dev#8327)
commit 3d6fdc8751134f6e5bac700519358cf9a700aba1
Author: Chris Grinolds <cgrinolds@mac.com>
Date:   Wed Apr 29 21:43:10 2015 -0700

    Added missing imports that caused tests to fail

commit 6c345d8dbc0c36bd4369dd220eceb495ab5ff2c6
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 16:23:01 2015 -0700

    Updated test suite to handle changes to gbq.py

commit f43e65f924d07f88bbbe308b9ea396b1f2a720f9
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 08:46:00 2015 -0700

    Updated BigQuery connector to no longer use deprecated ```oauth2client.tools.run() (pandas-dev#8327)

commit ea92e200ff341a3025b06868a565b1eea506c4c2
Author: Chris Grinolds <cgrinolds@zulily.com>
Date:   Wed Apr 29 08:43:43 2015 -0700

    Import BigQuery dependencies on a per-method basis (pandas-dev#9713)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment