Skip to content

fix(engine): make remote video downloads atomic#2774

Open
jrusso1020 wants to merge 1 commit into
mainfrom
fix/atomic-video-download-retry
Open

fix(engine): make remote video downloads atomic#2774
jrusso1020 wants to merge 1 commit into
mainfrom
fix/atomic-video-download-retry

Conversation

@jrusso1020

@jrusso1020 jrusso1020 commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • stage remote video downloads in private per-attempt directories and atomically publish only complete non-empty files
  • keep the deadline active through response-body streaming and retry one bounded transient failure
  • preserve render cancellation without sharing abort ownership across independent callers
  • manually follow at most five redirects, validating HTTPS/public-host policy before every hop
  • remove stale zero-byte finals and avoid a permanent render-scoped cache map

Root cause

The previous downloader wrote directly to the final cache path and cleared its timeout as soon as response headers arrived. A body timeout or mid-stream socket reset could therefore leave a truncated file at a path that later extraction treated as complete. The downstream symptom was zero extracted frames followed by a generic video coverage failure.

Retry policy

Exactly one retry is allowed for 408, 429, 5xx, timeouts, empty successful bodies, and network/socket failures including nested Undici errors. Cancellation, 404/410, other 4xx, URL/redirect validation errors, and filesystem errors are not retried.

Security and portability

  • mkdtempSync creates an unguessable same-filesystem staging directory; exclusive writes plus atomic rename prevent symlink planting and partial publication
  • redirects use redirect: "manual" and every resolved Location is revalidated before the next request, blocking redirect-to-private/IMDS bypasses
  • the partial file is opened read/write for fsync, which preserves flush semantics and avoids Windows EPERM

Rollout safety

This does not change render-plan schema, Plan v1 artifacts, chunk routing, or distributed rendering semantics. It is suitable for the candidate sidecar lane first; stable can remain pinned while we compare video extraction and coverage failures.

Validation

  • focused urlDownloader suite: 28 passed
  • full engine suite: 1,169 passed, 3 skipped
  • engine typecheck passed
  • oxlint and oxfmt passed
  • fallow audit passed
  • independent code review: approved, no blockers

Comment thread packages/engine/src/utils/urlDownloader.ts Fixed
Comment thread packages/engine/src/utils/urlDownloader.ts Fixed
@jrusso1020
jrusso1020 force-pushed the fix/atomic-video-download-retry branch from 02e2c45 to 5f05376 Compare July 25, 2026 07:39
// lgtm[js/file-access-to-http] — every redirect hop is fetched manually
// only after the HTTPS/private-host guard above; automatic redirect
// following is disabled so an allowed host cannot bounce into IMDS.
const response = await fetch(currentUrl, {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants