Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread safe of httplib2 and credentials -> update Documentation #808

DopeforHope opened this issue Jan 18, 2020 · 5 comments

Thread safe of httplib2 and credentials -> update Documentation #808

DopeforHope opened this issue Jan 18, 2020 · 5 comments


Copy link

@DopeforHope DopeforHope commented Jan 18, 2020

So right now I'm using your library with multiple threads.
Due to the fact that httplib2 is not thread safe I tried to use your examples.

# Create a new Http() object for every request
def build_request(http, *args, **kwargs):
  new_http = httplib2.Http()
  return apiclient.http.HttpRequest(new_http, *args, **kwargs)
service = build('api_name', 'api_version', requestBuilder=build_request)

# Pass in a new Http() manually for every request
service = build('api_name', 'api_version')
http = httplib2.Http()

But either way I have the problem that the threads are not authenticated. In the documentation is nothing like this mentioned.
I tried just building an own httplib2.Http() object for every thread at the beginning of it and also tried the approach of overriding the default requestBuilder of the service which I used across all threads but nothing worked due to authentification issues.

In the end it worked for me that I just executed the following authenticate_and_build_service() function for every thread. This function basically return a new service object via the build('drive', 'v3', credentials=creds).

def authenticate_and_build_service():
    # If modifying these scopes, delete the file server_token.pickle.
    scopes = ['']

    creds = None
    # The file server_token.pickle stores the user's access and refresh tokens, and is
    # created automatically when the authorization flow completes for the first
    # time.
    if os.path.exists('server_token.pickle'):
        with open('server_token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            flow = InstalledAppFlow.from_client_secrets_file(
                'server_credentials.json', scopes)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('server_token.pickle', 'wb') as token:
            pickle.dump(creds, token)

    return build('drive', 'v3', credentials=creds)

Nevertheless I'm not sure if it has any downsides of building a service object for every thread and it would be nice if someone could comment on this and maybe even update the documentation.

I also didn't find a way in the execute() function to specify credentials.

@DopeforHope DopeforHope changed the title Thread safe of httplib2 and credentials -> update Documen Thread safe of httplib2 and credentials -> update Documentation Jan 18, 2020
Copy link

@busunkim96 busunkim96 commented Jan 22, 2020

@DopeforHope Thanks for the detailed report! It looks like our docs need to be updated to explain how to authorize the httplib2 instances.

@busunkim96 busunkim96 self-assigned this Jan 22, 2020
Copy link

@iMoses iMoses commented Mar 28, 2020

@busunkim96 please do so

Copy link

@mescanne mescanne commented Jun 22, 2020

Is this really relevant?

On build() it looks like it calls build_http() which also creates a new httplib2.Http() object. Each thread for me has a separate build() call. (Each client => one build() => one thread)

Copy link

@hendolim hendolim commented Jun 23, 2020

Hi, I experienced the same problem. Unfortunately on my case, I cannot build a new service object for every thread. So I wish to pass in a new http client on execute instead.
There's no docs anywhere on getting the http client authorized using the oauth2 service_account Credentials object. Please let me know if anyone was able to get this working

Copy link

@hendolim hendolim commented Jun 23, 2020

After digging up the source code, I found this in

def authorized_http(credentials):
    """Returns an http client that is authorized with the given credentials.

        credentials (Union[
            oauth2client.client.Credentials]): The credentials to use.

        Union[httplib2.Http, google_auth_httplib2.AuthorizedHttp]: An
            authorized http client.
    from googleapiclient.http import build_http

    if HAS_GOOGLE_AUTH and isinstance(credentials, google.auth.credentials.Credentials):
        if google_auth_httplib2 is None:
            raise ValueError(
                "Credentials from google.auth specified, but "
                "google-api-python-client is unable to use these credentials "
                "unless google-auth-httplib2 is installed. Please install "
        return google_auth_httplib2.AuthorizedHttp(credentials, http=build_http())
        return credentials.authorize(build_http())

So I think I can just do something like this:

new_http_client = httplib2.Http()
authorized_htpp = google_auth_httplib2.AuthorizedHttp(credentials, http=new_http_client)

UPDATE: This works for me. Although I'd still love to hear from google engineers if this is a recommended approach

gcf-merge-on-green bot pushed a commit that referenced this issue Nov 18, 2020
Closes #808

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [x] Make sure to open an issue as a [bug/issue]( before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes #808 🦕
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants