fix(storage): make GCS uploads resilient to network/DNS stalls - #309
Merged
Conversation
Wrap every GCS request (auth token, upload, metadata PATCH, stat, list, get, remove, exists) in a per-attempt timeout with bounded retries and exponential backoff. A stalled egress path (e.g. IPv6 black-holing to Google) now aborts and retries instead of hanging the upload request indefinitely, and transient 5xx/429 responses are retried. Upload payloads within the buffer limit are read into memory so a failed attempt can be safely replayed; larger files stream in a single attempt. Re-enable concise HTTP access logging (method/url/status/latency), warning on 5xx or slow (>=2s) requests, so upload failures are visible. Timeouts, retries and buffer limit are configurable via STORAGE_GCS_TIMEOUT_MS, STORAGE_GCS_UPLOAD_TIMEOUT_MS, STORAGE_GCS_MAX_RETRIES and STORAGE_GCS_BUFFER_LIMIT_BYTES.
raaymax
force-pushed
the
fix/gcs-upload-resilience
branch
from
July 24, 2026 11:14
1d19597 to
a2998c9
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Prod file uploads (
chat.codecat.io) intermittently fail: the progress bar fills but never turns green, then the message send fails. It came and went over whole days and self-healed, with no app-level error logs.Root cause
Uploads are streamed to GCS during the request, and the GCS client made 2–3 sequential outbound calls to Google (auth token → object POST → metadata PATCH) with no timeout and no retry. Any transient stall in the container's egress/DNS path (single upstream resolver via Docker's embedded DNS, plus parallel A/AAAA lookups) made those
fetchcalls hang indefinitely — hanging the whole upload with nothing logged.Amplifier: the deployed
maincached the GCS access token for only 2 seconds, so nearly every upload re-authenticated againstoauth2.googleapis.com, multiplying the round-trips that could stall. (devalready caches for 50 min; this PR keeps that and hardens the rest.)Changes
gcs.ts— every GCS request (auth token, upload POST, metadata PATCH, stat, list, get, remove, exists) now goes through agcsFetchwrapper with a per-attemptAbortSignal.timeout+ bounded retries + exponential backoff. A stalled path aborts and retries instead of hanging; transient 429/5xx are retried.http/mod.ts— re-enabled concise HTTP access logging (method url status latency), escalated towarnon 5xx or>=2s, so upload issues are visible instead of silent.Configurable via
STORAGE_GCS_TIMEOUT_MS,STORAGE_GCS_UPLOAD_TIMEOUT_MS,STORAGE_GCS_MAX_RETRIES,STORAGE_GCS_BUFFER_LIMIT_BYTES.Testing
deno fmt/deno lint/deno checkclean on both files.Deployment note
The deployed
mainalso carries the 2-second token-cache bug (fixed ondev), so shipping this viadev → mainalso delivers that fix. For immediate prod relief, this commit can be cherry-picked onto a hotfix branch offmain.Related infra (out of scope for this PR)
Host-level IPv6 was misconfigured (rogue ULA RAs from smart-home/RPi devices); the servers now ignore RAs (
accept_ra=0). That's hygiene — this code change is what makes uploads resilient regardless of the network path.