Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CT-1936] Support Google Cloud Auth from AWS Credentials File #86

Closed
siephen opened this issue Dec 9, 2021 · 10 comments · May be fixed by #1041
Closed

[CT-1936] Support Google Cloud Auth from AWS Credentials File #86

siephen opened this issue Dec 9, 2021 · 10 comments · May be fixed by #1041
Labels
enhancement New feature or request good_first_issue Good for newcomers Stale

Comments

@siephen
Copy link

siephen commented Dec 9, 2021

Describe the feature

Support connecting to BigQuery via from an AWS Credentials File.

Describe alternatives you've considered

None

Additional context

Google Cloud Auth AWS Docs

This is how my org does gcloud auth, so we don’t have many alternatives here

Who will this benefit?

Any user/company who runs Google cloud workloads from AWS and keeps credentials in AWS.

Are you interested in contributing this feature?

Yes
@siephen siephen added enhancement New feature or request triage labels Dec 9, 2021
@McKnight-42
Copy link
Contributor

@siephen going to open this issue up and would be more than happy to help in anyway i can.

@McKnight-42 McKnight-42 added good_first_issue Good for newcomers and removed triage labels Jan 5, 2022
@siephen
Copy link
Author

siephen commented Mar 29, 2022

@McKnight-42,

I'm thinking this would manifest as a new method that can be used called aws and it would use the following syntax to generate the credentials for the BigQuery Client from the AWS From File method.

Updating the following IF Block

Adding in the type here

Is this the right track?

@siephen
Copy link
Author

siephen commented Apr 13, 2022

@McKnight-42,

I'm thinking this would manifest as a new method that can be used called aws and it would use the following syntax to generate the credentials for the BigQuery Client from the AWS From File method.

Updating the following IF Block

Adding in the type here

Is this the right track?

@McKnight-42 Bumping my comment.

@McKnight-42
Copy link
Contributor

McKnight-42 commented Apr 15, 2022 •
edited

@siephen so sorry for the late response; yes these the above are both spots you will need to add in some logic, may also need to add some new credentials specifically for aws connection here

dbt-bigquery/dbt/adapters/bigquery/connections.py

Line 89 in 072050e

class BigQueryCredentials(Credentials):

would love to see a draft pr on trying to get this to work and keep helping you build it out, already looking into what we might need to add it to CI/CD testing as well on our end.

@siephen siephen mentioned this issue May 11, 2022
Closed
4 tasks
@siephen
Copy link
Author

siephen commented May 11, 2022

@McKnight-42 , I drafted a PR for this here. I'm working through validating it on local with my internal team. Any initial thoughts? Also, where are the docs that will need to be updated for this new connection method option?

@McKnight-42
Copy link
Contributor

@siephen fantastic! i'll pop over to the PR and leave some comments there when I can. as for docs updating i'd say this page https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile.

If you can I would appreciate any relevant information on what it takes to setup and get running locally yet? so we can test on our end as well to best assist?

@github-actions
Copy link
Contributor

github-actions bot commented Nov 9, 2022

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Nov 9, 2022
@siephen
Copy link
Author

siephen commented Nov 9, 2022

@McKnight-42 , I have some questions over on the linked PR for this around how to write the test cases, can you take a look when you get a chance?

@github-actions github-actions bot removed the Stale label Nov 10, 2022
@Fleid Fleid added the jira label Jan 27, 2023
@github-actions github-actions bot changed the title Support Google Cloud Auth from AWS Credentials File [CT-1936] Support Google Cloud Auth from AWS Credentials File Jan 27, 2023
chrisguitarguy added a commit to chrisguitarguy/dbt-bigquery that referenced this issue Dec 4, 2023
@chrisguitarguy
Use Google Auth's load_credentials_from_* Methods
e673324 
These methods handle services accounts as well any any other credential
method. Got someone using DBT with AWS-based Auth (see dbt-labs#86)? This will
work with AWS Auth.

Typical service account auth with a key file and oauth2 jwt assertion grant
stuff still works and is covered here: https://github.com/googleapis/google-auth-library-python/blob/v2.14.1/google/auth/_default.py#L408

DBT BigQuery requires google-api-core>=2.11 which requires
google-auth>=2.14.1 which supports oauth2 service acounts as well as
other things.

Basically this will just let folks auth bigquery however they would auth
any gcloud resources without any extra effort. And the previous versions
config still works exactly the same.
chrisguitarguy added a commit to chrisguitarguy/dbt-bigquery that referenced this issue Dec 4, 2023
@chrisguitarguy
Use Google Auth's load_credentials_from_* Methods
c9f1b46 
These methods handle services accounts as well any any other credential
method. Got someone using DBT with AWS-based Auth (see dbt-labs#86)? This will
work with AWS Auth.

Typical service account auth with a key file and oauth2 jwt assertion grant
stuff still works and is covered here: https://github.com/googleapis/google-auth-library-python/blob/v2.14.1/google/auth/_default.py#L408

DBT BigQuery requires google-api-core>=2.11 which requires
google-auth>=2.14.1 which supports oauth2 service acounts as well as
other things.

Basically this will just let folks auth bigquery however they would auth
any gcloud resources without any extra effort. And the previous versions
config still works exactly the same.
chrisguitarguy added a commit to chrisguitarguy/dbt-bigquery that referenced this issue Dec 4, 2023
@chrisguitarguy
Use Google Auth's load_credentials_from_* Methods
01f2c1b 
These methods handle services accounts as well any any other credential
method. Got someone using DBT with AWS-based Auth (see dbt-labs#86)? This will
work with AWS Auth.

Typical service account auth with a key file and oauth2 jwt assertion grant
stuff still works and is covered here: https://github.com/googleapis/google-auth-library-python/blob/v2.14.1/google/auth/_default.py#L408

I did add `google-auth>=2.20.0` which is the first version that has
`load_credentials_from_dict`. Previously this lower bound was v2.14.1
(from `google-api-core>=2.11`).

Basically this will just let folks auth bigquery however they would auth
any gcloud resources without any extra effort. And the previous versions
config still works exactly the same.
@chrisguitarguy chrisguitarguy mentioned this issue Dec 4, 2023
Open
4 tasks
chrisguitarguy added a commit to chrisguitarguy/dbt-bigquery that referenced this issue Dec 4, 2023
@chrisguitarguy
Use Google Auth's load_credentials_from_* Methods
c9b60d5 
These methods handle services accounts as well any any other credential
method. Got someone using DBT with AWS-based Auth (see dbt-labs#86)? This will
work with AWS Auth.

Typical service account auth with a key file and oauth2 jwt assertion grant
stuff still works and is covered here: https://github.com/googleapis/google-auth-library-python/blob/v2.14.1/google/auth/_default.py#L408

I did add `google-auth>=2.20.0` which is the first version that has
`load_credentials_from_dict`. Previously this lower bound was v2.14.1
(from `google-api-core>=2.11`).

Basically this will just let folks auth bigquery however they would auth
any gcloud resources without any extra effort. And the previous versions
config still works exactly the same.
@github-actions GitHub Actions
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jan 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good_first_issue Good for newcomers Stale
Projects
None yet
Milestone
No milestone
3 participants
@Fleid @siephen @McKnight-42