Skip to content

fix(storage): make GCS uploads resilient to network/DNS stalls - #309

Merged
raaymax merged 1 commit into
devfrom
fix/gcs-upload-resilience
Jul 24, 2026
Merged

fix(storage): make GCS uploads resilient to network/DNS stalls#309
raaymax merged 1 commit into
devfrom
fix/gcs-upload-resilience

Conversation

@raaymax

@raaymax raaymax commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Problem

Prod file uploads (chat.codecat.io) intermittently fail: the progress bar fills but never turns green, then the message send fails. It came and went over whole days and self-healed, with no app-level error logs.

Root cause

Uploads are streamed to GCS during the request, and the GCS client made 2–3 sequential outbound calls to Google (auth token → object POST → metadata PATCH) with no timeout and no retry. Any transient stall in the container's egress/DNS path (single upstream resolver via Docker's embedded DNS, plus parallel A/AAAA lookups) made those fetch calls hang indefinitely — hanging the whole upload with nothing logged.

Amplifier: the deployed main cached the GCS access token for only 2 seconds, so nearly every upload re-authenticated against oauth2.googleapis.com, multiplying the round-trips that could stall. (dev already caches for 50 min; this PR keeps that and hardens the rest.)

Changes

  • gcs.ts — every GCS request (auth token, upload POST, metadata PATCH, stat, list, get, remove, exists) now goes through a gcsFetch wrapper with a per-attempt AbortSignal.timeout + bounded retries + exponential backoff. A stalled path aborts and retries instead of hanging; transient 429/5xx are retried.
    • Upload payloads within a size limit are buffered so a failed attempt can be safely replayed; larger files stream in a single attempt (no unbounded memory).
  • http/mod.ts — re-enabled concise HTTP access logging (method url status latency), escalated to warn on 5xx or >=2s, so upload issues are visible instead of silent.

Configurable via STORAGE_GCS_TIMEOUT_MS, STORAGE_GCS_UPLOAD_TIMEOUT_MS, STORAGE_GCS_MAX_RETRIES, STORAGE_GCS_BUFFER_LIMIT_BYTES.

Testing

  • deno fmt / deno lint / deno check clean on both files.
  • Verified end-to-end against the real GCS bucket from inside the prod container: upload 200, stat, get (bytes match), remove, exists-after-remove — all correct via the new buffered/retry path.

Deployment note

The deployed main also carries the 2-second token-cache bug (fixed on dev), so shipping this via dev → main also delivers that fix. For immediate prod relief, this commit can be cherry-picked onto a hotfix branch off main.

Related infra (out of scope for this PR)

Host-level IPv6 was misconfigured (rogue ULA RAs from smart-home/RPi devices); the servers now ignore RAs (accept_ra=0). That's hygiene — this code change is what makes uploads resilient regardless of the network path.

Wrap every GCS request (auth token, upload, metadata PATCH, stat, list,
get, remove, exists) in a per-attempt timeout with bounded retries and
exponential backoff. A stalled egress path (e.g. IPv6 black-holing to
Google) now aborts and retries instead of hanging the upload request
indefinitely, and transient 5xx/429 responses are retried.

Upload payloads within the buffer limit are read into memory so a failed
attempt can be safely replayed; larger files stream in a single attempt.

Re-enable concise HTTP access logging (method/url/status/latency),
warning on 5xx or slow (>=2s) requests, so upload failures are visible.

Timeouts, retries and buffer limit are configurable via
STORAGE_GCS_TIMEOUT_MS, STORAGE_GCS_UPLOAD_TIMEOUT_MS,
STORAGE_GCS_MAX_RETRIES and STORAGE_GCS_BUFFER_LIMIT_BYTES.
@raaymax
raaymax force-pushed the fix/gcs-upload-resilience branch from 1d19597 to a2998c9 Compare July 24, 2026 11:14
@raaymax
raaymax merged commit e20f8e9 into dev Jul 24, 2026
1 check passed
@raaymax raaymax mentioned this pull request Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant