support passing credentials from memory for google connectors#1888
Conversation
6d59173 to
66dcd03
Compare
| if isinstance(key_path, dict): | ||
| creds = service_account.Credentials.from_service_account_info(key_path) | ||
| elif os.path.isfile(key_path): | ||
| os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = key_path | ||
| creds, _ = default() | ||
| else: | ||
| raise ValueError( | ||
| f"key path not recognized as a dictionary or a file path: " | ||
| f"[{type(key_path)}] {key_path}", | ||
| ) |
There was a problem hiding this comment.
for clarify maybe just have a separate argument for the path versus the value rather than trying to infer. I think given naming folks are already confused on which we expec
|
ohh so token could be a path OR a value?? got it. yea, I think this is all confusing. also...why is it called a token in one place versus service_key in another? personally I think we should just be explicit about what we expect having discrete options for path vs value. regardless seems like these should be the same argument name across gcs and google drive (given that we're passing the same key to both...) per issue description, keep the old options for back compatibility |
74847b3 to
a5e2140
Compare
| service_account_key: str | ||
| token: t.Union[dict, Path] |
There was a problem hiding this comment.
I think in either case we should keep the old key and flag as deprecated (since we have folks actively using these connectors).
let's defer to service_account_key as the one we make common to both, since this is more specific and seems to be more widely referenced in their documents.
There was a problem hiding this comment.
What do we use if both are provided? Raise an error?
There was a problem hiding this comment.
actually, I think google drive is the one folks are really using. so I think fine to not deprecate, but let's just make service_account_key the common one.
a5e2140 to
d8b900b
Compare
| token: t.Optional[str] = None | ||
| service_account_key: t.Optional[t.Union[dict, Path]] = None |
There was a problem hiding this comment.
should also bump the gcs docs page?
ryannikolaidis
left a comment
There was a problem hiding this comment.
tested both by passing the value as the argument, and both worked.
after docs bump this is gtg!
ryannikolaidis
left a comment
There was a problem hiding this comment.
...per discussion, looks like docs use anonymous, so bumping shouldn't be required for this.
lgtm!
fa04a3e to
148a0aa
Compare
d563c92 to
9c83868
Compare
9c83868 to
05ed62d
Compare
Description
Google Drive
The existing service account parameter was expanded to support either a file path or a json value to generate the credentials when instantiating the google drive client.
GCS
Google Cloud Storage already supports the value being passed in, from their docstring:
I tested this locally:
['utic-test-ingest-fixtures/ideas-page.html', 'utic-test-ingest-fixtures/nested-1', 'utic-test-ingest-fixtures/nested-2']