feat(storage): native GCS backend, keyless via Workload Identity - #2
Merged
Conversation
Add a GCSStorage backend (USE_GCS=1) that reads/writes gs:// via Application Default Credentials and signs presigned upload POST policies + download URLs through the IAM signBlob API, so no service-account key is required. Mirrors the S3Storage interface; the views are unchanged. Claude-Session: https://claude.ai/code/session_01LmQ4REc5zfzYfBZb9ZRdoy
Addresses review of the GCS backend: - Always define GCSStorage (only the S3Storage swap stays gated on USE_GCS) so it is importable for unit tests; move the google-cloud-storage imports to the module top (it is now a base requirement). - Cache ADC credentials + the resolved signing identity in __init__ and refresh the token lazily, instead of re-running ADC discovery and a full token refresh on every presigned URL. - Harden the signBlob signing identity: an explicit GS_SIGNING_SA wins, else the ADC identity, but Workload Identity reports the literal "default" (signing as which is an IAM 400) so fall back to the metadata server. - Share one RFC 5987 Content-Disposition helper between the S3 and GCS backends instead of duplicating it. Most of the storage.py churn is de-indentation from moving GCSStorage to module scope; no behaviour change on the S3 path. Claude-Session: https://claude.ai/code/session_01LmQ4REc5zfzYfBZb9ZRdoy
…ution Unit tests (same mock pattern as the existing S3 expiration tests) for the shared Content-Disposition helper and the GCS signing-identity resolution — the latter pins the subtle Workload Identity "default" case that would otherwise 400 at IAM signBlob. Claude-Session: https://claude.ai/code/session_01LmQ4REc5zfzYfBZb9ZRdoy
Views construct a storage object inside the request handler — `S3Storage( request=request)` appears at 31 call sites — so everything __init__ does runs on every request. It ran ADC discovery twice (once via google.auth .default, once inside the GCS client) and then resolved the signing identity. That last step is the expensive one, and it is the Workload Identity path this backend exists for: those credentials report their email as the literal "default", so resolution falls through to a blocking HTTP call to the metadata server with a 5s timeout. Per request. Hoist both into a process-wide cache and pass the cached credentials to the GCS client so constructing it does not repeat the discovery. The credentials object refreshes its own token in place, so caching it is safe and the lazy refresh in _signing() still applies. Also return False rather than None when delete_files fails, matching the S3 backend — callers should not have to know which backend they have. Tests cover the caching (ADC discovery and the metadata lookup each happen once across repeated calls) and, for the first time, the USE_GCS symbol swap itself: that module-level rebinding is what routes the entire app to this backend, and nothing asserted it fired, stayed off by default, or ignored near-miss values like "true" and "0". Claude-Session: https://claude.ai/code/session_0192wrU7BnxTNUTs3We514aC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds a native Google Cloud Storage backend to the API, selected by
USE_GCS=1, alongside the existing S3 backend. Reads and writes use ApplicationDefault Credentials, and presigned upload POST policies + download URLs
are signed with the IAM signBlob API — so object storage works with no
service-account key of any kind.
GCSStoragemirrors theS3Storagemethod surface (generate_presigned_post,generate_presigned_url,get_object_metadata,copy_object,upload_file,delete_files,url); the module swaps theS3Storagesymbol under the toggleso the asset views (
asset/v2.py) are unchanged.Why keyless is a hard requirement here (not a preference)
The meas-inst GCP org enforces the Google secure-by-default org-policy bundle
— including
iam.disableServiceAccountKeyCreation(+ the managed variant andiam.disableServiceAccountKeyUpload), aligned with our CMMC posture. That policyblocks GCS HMAC keys and service-account JSON keys entirely, so the usual
paths don't exist here:
The only viable path is Workload Identity + IAM signBlob: the pod runs as a
bound service account and asks IAM to sign each URL/policy.
Deploy-side requirements
roles/storage.objectAdminon the bucket androles/iam.serviceAccountTokenCreatoron itself (signBlob).USE_GCS=1,GS_BUCKET_NAME(falls back toAWS_S3_BUCKET_NAME),optional
GS_PROJECT_ID(inferred from ADC on GKE), optionalGS_SIGNING_SA(explicit signing identity; otherwise auto-resolved).
Robustness / correctness notes
generate_signed_post_policy_v4is a Client method (takes the bucketname), not a
Bucketmethod, in google-cloud-storage 2.19.0.report the literal
"default"until refreshed, and signing as"default"isan IAM 400. Resolution order is explicit
GS_SIGNING_SA→ ADC identity →metadata server, never accepting
"default".token is refreshed lazily, rather than re-running ADC discovery on every URL.
Tests
tests/unit/settings/test_storage.py(same mock pattern as the existing S3tests): the shared Content-Disposition helper, and the GCS signing-identity
resolution incl. the
"default"→ metadata fallback.Validated end-to-end
On the live meas-inst deployment (the real view code path): presigned download
200, presigned upload POST204, read-back, metadata, delete — all keylessvia Workload Identity — re-verified on the reviewed build now deployed
(
v1.4.0-worklogs-gcs-7a393ea719); the signing identity resolved to the realGSA (
plane-storage@…), confirming the hardened resolution path.Claude-Session: https://claude.ai/code/session_01LmQ4REc5zfzYfBZb9ZRdoy