Skip to content

fix(bigquery-analytics): route Storage Write API appends to the dataset's region#2

Merged
caohy1988 merged 1 commit into
mainfrom
fix/bqaa-storage-write-regional-routing
Jun 1, 2026
Merged

fix(bigquery-analytics): route Storage Write API appends to the dataset's region#2
caohy1988 merged 1 commit into
mainfrom
fix/bqaa-storage-write-regional-routing

Conversation

@caohy1988
Copy link
Copy Markdown
Owner

Summary

Fixes cross-region BigQuery logging failures reported in GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK#262: logging works when the destination dataset is in the US multiregion (e.g. us-central1) but fails with a "session not found" error in single regions such as northamerica-northeast1.

Root cause

The plugin writes rows via the Storage Write API's streaming AppendRows RPC:

responses = await self.write_client.append_rows(requests_iter())

AppendRows is a client-streaming RPC, so the GAPIC client cannot auto-populate the implicit request-routing header (it has no unary request body to read write_stream from). Without that header, every append is routed to the US multiregion by default:

  • us-central1 resolves within the US multiregion → the _default stream is found → works.
  • northamerica-northeast1 is a single region → the US-routed backend has no knowledge of the stream created there → "session not found" / stream-not-found, the append fails, and no rows are ever written.

This surfaced to users as session_id (and every other column) "failing to propagate" — in reality the rows never landed because the writes were misrouted. The session_id plumbing itself is fine.

Per the Storage Write API docs: "if you write data to any region except the US multiregion, you must include the following header in your requests: x-goog-request-params: write_stream=<stream_name>." The official google.cloud.bigquery_storage_v1.writer.AppendRowsStream helper sets exactly this header (writer.py); the raw async append_rows this plugin calls does not.

Fix

Set the routing header explicitly on the append_rows call so the request reaches the region that owns the write stream:

responses = await self.write_client.append_rows(
    requests_iter(),
    metadata=(
        ("x-goog-request-params", f"write_stream={self.write_stream}"),
    ),
)

US-multiregion behavior is unchanged (the header is correct there too). This is a one-line functional fix that mirrors Google's own client-library helper.

Tests

  • Adds test_append_rows_sets_regional_routing_header (deploys the plugin with location="northamerica-northeast1", asserts the routing header is passed to append_rows).
  • Full plugin suite green: 229 passed.

Notes / out of scope

  • A regional/REP endpoint (bigquerystorage.<location>.rep.googleapis.com) is only required for strict data-residency guarantees; the routing header alone resolves the functional cross-region failure here. Can be a follow-up if data-residency support is desired.
  • The constructor's location param is intentionally not passed to the synchronous bigquery.ClientTestDatasetLocationHandling documents that omitting it lets BigQuery infer location from the dataset and avoids silent view-creation failures for non-US datasets. Left as-is.

@caohy1988 caohy1988 force-pushed the fix/bqaa-storage-write-regional-routing branch 2 times, most recently from 22328fe to f41093d Compare June 1, 2026 20:03
caohy1988 added a commit that referenced this pull request Jun 1, 2026
)

The "Check for hardcoded googleapis.com endpoints" step in
.github/workflows/check-file-contents.yml uses

  grep -lE 'https?://[a-zA-Z0-9.-]+\.googleapis\.com'

to find files that should also declare an `.mtls.googleapis.com`
counterpart for dynamic endpoint selection. The regex matches any
googleapis.com URL — including OAuth 2.0 scope URLs like
https://www.googleapis.com/auth/cloud-platform and
.../auth/bigquery — which are identity strings, not API endpoints.
They don't have mTLS counterparts and never will. Any file that
legitimately declares an OAuth scope (very common for ADK plugins
integrating Google APIs) trips the gate even when no real endpoint
is hardcoded.

Fix: add a second pass that filters the candidate set down to files
that have at least one googleapis.com URL OUTSIDE the OAuth scope
namespace (i.e. not matching `googleapis.com/auth/`). The mTLS check
runs only against that filtered set.

Verified against four synthesized cases:

  only_oauth.py            (only OAuth scopes)        → ignored ✓
  real_endpoint.py         (endpoint, no mTLS)        → flagged ✓
  real_endpoint_with_mtls  (endpoint + mTLS)          → passes  ✓
  mixed.py                 (OAuth + endpoint, no mTLS)→ flagged ✓

No effect on the surrounding `logger`, `from __future__`, or
`cli` import checks. CI policy intent unchanged: real hardcoded
googleapis.com endpoints still must declare their `.mtls`
counterpart.

Refs:
  - #2 (the BQAA Storage Write regional routing
    fix that surfaced this false positive)
  - GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK#262
…et's region

The Storage Write API `AppendRows` streaming RPC does not auto-populate
the request-routing header, so the plugin's writes were always routed to
the US multiregion. Writes to a dataset in any other region (e.g.
northamerica-northeast1) failed with a "session not found" /
stream-not-found error and no rows were ever written — which surfaced to
users as session_id (and every other column) failing to propagate.

Set the `x-goog-request-params: write_stream=<stream>` routing header on
the append_rows call, matching what google.cloud.bigquery_storage_v1.writer
does internally, so requests reach the region that owns the write stream.
US-multiregion behavior is unchanged.

Adds a regression test asserting the routing header is passed.

Fixes GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK#262
@caohy1988 caohy1988 force-pushed the fix/bqaa-storage-write-regional-routing branch from f41093d to 159f185 Compare June 1, 2026 22:04
@caohy1988 caohy1988 marked this pull request as ready for review June 1, 2026 22:17
@caohy1988 caohy1988 merged commit e5c0d8b into main Jun 1, 2026
11 checks passed
@caohy1988 caohy1988 deleted the fix/bqaa-storage-write-regional-routing branch June 1, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant