BigQuery: table metadata resolves dataset against ADC project instead of connection URI project (cross-project Workload Identity broken in 6.x)

[bluemalkin]

Bug description

When a BigQuery connection uses Application Default Credentials (in our case: GKE Workload Identity — the pod's Kubernetes Service Account is bound to a Google Service Account that lives in the GKE project but has been granted read access to BigQuery datasets in a different project), adding a physical dataset fails as soon as a table is selected:

{
  "errors": [{
    "message": "GET https://bigquery.googleapis.com/bigquery/v2/projects/<GSA-PROJECT>/datasets/support/tables/api_events?prettyPrint=false: Not found: Dataset <GSA-PROJECT>:support",
    "error_type": "GENERIC_BACKEND_ERROR",
    "level": "error"
  }]
}

The connection URI is bigquery://<DATA-PROJECT>, but the request targets <GSA-PROJECT> — the project the ADC credentials belong to (where GKE runs), not the project where the datasets live.

Notably, the schema and table dropdowns populate correctly (those go through the SQLAlchemy dialect, which honors the URI host); only selecting a table fails ("Unable to load columns for the selected table"). Enabling allow_multi_catalog and selecting the correct catalog in the dataset panel does not help.

This worked in 4.x and 5.x with the same connection URI, the same Workload Identity setup, and the same sqlalchemy-bigquery version (1.17.0).

Root cause

BigQueryEngineSpec._get_client builds the raw Google client without a project= argument (bigquery.py L527-L553 @ 6.1.0):

credentials = google.auth.default()[0]
return bigquery.Client(credentials=credentials)   # project defaults to the ADC project

so client.project is the credentials' project, ignoring the project in the connection URI (engine.url.host).

This bug is present but latent in ≤5.0 — nothing in the dataset-creation flow called _get_client. Superset 6.0's partition-metadata feature added get_time_partition_column (L409-L423 @ 6.1.0), which is invoked from get_extra_table_metadata on every table selection and passes an unqualified two-part table ID to that client:

bq_table = client.get_table(f"{table.schema}.{table.table}")

A two-part ID is resolved against client.project → wrong project → 404. Other _get_client consumers (get_catalog_names, estimate_query_cost) are affected the same way.

How to reproduce

1. Run Superset ≥6.0 on GKE with Workload Identity; the bound GSA lives in project A and has bigquery.dataViewer/bigquery.jobUser on datasets in project B.
2. Create a BigQuery connection with URI bigquery://<project-B> and no credentials_info (ADC only).
3. Datasets → New dataset → pick the connection → schema and table dropdowns list project B's contents correctly.
4. Select a table → error: Not found: Dataset <project-A>:<schema>.

Expected results

The table metadata call should target the project from the connection URI (or the selected catalog), consistent with how the SQLAlchemy dialect resolves it.

Suggested fix

Pass the engine's project to the client, and/or fully qualify the table reference:

# _get_client
return bigquery.Client(credentials=credentials, project=engine.url.host or None)

Workaround we deployed (confirmed working)

Monkeypatch via superset_config.py (FLASK_APP_MUTATOR to avoid circular imports at config-load time):

def FLASK_APP_MUTATOR(app):
    import google.auth
    from google.cloud import bigquery
    from google.oauth2 import service_account
    from superset.db_engine_specs.bigquery import BigQueryEngineSpec

    @classmethod
    def _get_client(cls, engine, database):
        project = engine.url.host or None
        if credentials_info := engine.dialect.credentials_info:
            credentials = service_account.Credentials.from_service_account_info(credentials_info)
            return bigquery.Client(credentials=credentials, project=project)
        try:
            credentials = google.auth.default()[0]
        except google.auth.exceptions.DefaultCredentialsError as ex:
            from superset.db_engine_specs.exceptions import SupersetDBAPIConnectionError
            raise SupersetDBAPIConnectionError("The database credentials could not be found.") from ex
        return bigquery.Client(credentials=credentials, project=project)

    BigQueryEngineSpec._get_client = _get_client

After this patch, cross-project dataset creation works again.

Environment

• Superset: 6.1.0 (official Helm chart 0.17.2); the affected code is unchanged in master
• sqlalchemy-bigquery: 1.17.0 (same version worked on Superset 5.0 — not a driver issue)
• Deployment: GKE, Workload Identity (KSA → GSA), GSA project ≠ dataset project
• Auth to BigQuery: Application Default Credentials (no credentials_info on the connection)

Screenshots/recordings

No response

Superset version

6.1.0

Python version

3.10

Node version

I don't know

Browser

Chrome

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

This issue was identified and fixed in PR #40200, which was merged on June 24, 2026 [1]. That fix updates get_time_partition_column to use the fully qualified "{catalog}.{schema}.{table}" reference when table.catalog is set, preventing the fallback to the credentials/default project.

However, the fix only helps when the Table object has its catalog field populated — which requires allow_multi_catalog to be enabled and the user to have explicitly selected the catalog in the dataset panel. If you're relying on the connection URI's host as the implicit project (without multi-catalog), this won't resolve the issue since table.catalog would be None and the code still falls back to the two-part dataset.table format resolved against client.project.

The underlying problem you've identified — that _get_client doesn't pass project=engine.url.host — remains unaddressed on master [2]. Your suggested fix of passing project=engine.url.host or None to bigquery.Client(...) would be the more complete solution, as it would make all _get_client consumers (get_catalog_names, estimate_query_cost, get_default_catalog, and get_time_partition_column) respect the connection URI's project regardless of whether table.catalog is set.

Your monkeypatch is a solid workaround in the meantime. If you're interested in contributing the upstream fix, it would be a small change to _get_client at lines 473–499 — adding project=engine.url.host or None to both bigquery.Client(...) calls.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[Abhiiishek44]

I reviewed the issue and the related #40200 change. The remaining problem appears to be that _get_client does not pass the project from engine.url.host to bigquery.Client, so two-part table references can still resolve against the ADC project when no catalog is populated.

I’d like to work on a focused fix that passes engine.url.host or None in both client creation paths and adds tests covering ADC and service-account credentials.