Description
Just noticed this while working on #56324 — quota_project_id was added to GoogleBaseHook but none of the Google Cloud operators expose it as a parameter. The only way to use it right now is through connection extras, which is connection-wide and can't be overridden per task. Every operator already does this pattern for impersonation_chain,quota_project_id just needs the same treatment.
I'm planning to implement this; starting with BigQuery operators and then GCS, Dataflow, and Dataproc.
Happy to submit a PR for this! just wanted to check if there's any reason this pattern shouldn't be extended to operators before I start.
Use case/motivation
if you have two BigQuery tasks in the same DAG that need to bill to different projects, there's currently no way to express that at the operator level. Organizations using shared service accounts across multiple billing projects hit this immediately.
Related issues
#56324
Are you willing to submit a PR?
Code of Conduct
Description
Just noticed this while working on #56324 —
quota_project_idwas added toGoogleBaseHookbut none of the Google Cloud operators expose it as a parameter. The only way to use it right now is through connection extras, which is connection-wide and can't be overridden per task. Every operator already does this pattern forimpersonation_chain,quota_project_idjust needs the same treatment.I'm planning to implement this; starting with BigQuery operators and then GCS, Dataflow, and Dataproc.
Happy to submit a PR for this! just wanted to check if there's any reason this pattern shouldn't be extended to operators before I start.
Use case/motivation
if you have two BigQuery tasks in the same DAG that need to bill to different projects, there's currently no way to express that at the operator level. Organizations using shared service accounts across multiple billing projects hit this immediately.
Related issues
#56324
Are you willing to submit a PR?
Code of Conduct