Update storage client to handle retries and explicit credentials. #632
Conversation
* Set file_loader util to use an explicit credential object so that per thread http object caching is not used for loading and saving files to a GCS bucket.
Codecov Report
@@ Coverage Diff @@
## dev #632 +/- ##
=======================================
Coverage 83.33% 83.33%
=======================================
Files 164 164
Lines 8103 8109 +6
=======================================
+ Hits 6753 6758 +5
- Misses 1350 1351 +1
|
| """ | ||
| # Pass credential in explicitly so that cached credentials are not used. | ||
| credentials = client.GoogleCredentials.get_application_default() | ||
| storage_client = storage.StorageClient(credentials=credentials) |
There was a problem hiding this comment.
I am sure I am missing something, so please help me to understand how this is different from the previous existing code.
So before, we create the storage_client like this:
storage_client = storage.StorageClient()
Which in turn initialize the super _base_repository.BaseRepositoryClient()
The super also end up getting the application default credential in the init().
https://github.com/GoogleCloudPlatform/forseti-security/blob/dev/google/cloud/security/common/gcp_api/_base_repository.py#L130
There was a problem hiding this comment.
The difference is the setting of the internal _use_cached_http flag, which is only set to True when credentials are not passed in. By passing in credentials it forces the client to generate a new http object for each request instead of using a cached one. That's the behavior we want for the file_loader.
Now this is not the only way that could be accomplished, but it does require the least code changes. If you think it's too confusing I can look at a deeper refactor of the logic.
per thread http object caching is not used for loading and saving files
to a GCS bucket.
Thanks for opening a Pull Request!
Here's a handy checklist to ensure your PR goes smoothly.
pylint --rcfile=pylintrcpasses.These guidelines and more can be found in our contributing guidelines.