feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)#1844
Conversation
…ssetUpload
Regenerated from experiment-framework `master` at commit `e74815f7af` (the
merge of EF#41085, which added `/v3/assets/direct-uploads` +
`/v3/assets/{asset_id}/complete` to the `TARGET_ENDPOINTS` allowlist in
`scripts/generate_hyperframes_cli_client.py`).
The `sync-hyperframes-codegen.yml` workflow that normally auto-opens this
PR failed with a `gh: Not Found (HTTP 404)` on the PR-creation step (run
28556975483); regenerated manually with:
cd experiment-framework
PYTHONPATH=. python3 scripts/generate_hyperframes_cli_client.py \\
--out /path/to/hyperframes-oss
This commit is codegen-only — no hand edits. The direct-upload wire-up
that consumes the new `createAssetUpload` + `completeAssetUpload` methods
lands in the follow-up commit.
— Jerrai
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…200MB)
Replaces the legacy `client.uploadAsset(...)` multipart POST to
`/v3/assets` (32 MB in-memory proxy path) with the three-step direct-to-
S3 flow that lifts the practical per-project ceiling to 200 MB:
1. `POST /v3/assets/direct-uploads` — declares filename, content-type,
size, and SHA256 checksum; returns `asset_id`, presigned
`upload_url`, and required `upload_headers`.
2. Raw `PUT` to `upload_url` with the zip bytes + `upload_headers`
verbatim. No CLI auth attached — the presigned URL signature carries
authorization, and any extra headers would break the signature.
3. `POST /v3/assets/{asset_id}/complete` — finalizes into a reusable
asset. Retried up to 5x on 409 ("Uploaded object not found yet"), a
documented race between S3 write consistency and the finalize check.
The returned `asset_id` is the same namespace the legacy path produced
(both write into `movio_asset`), so the downstream render submission at
`createRender({project: {type: "asset_id", asset_id}})` is unchanged.
Server-side context (EF#41085): the direct-upload endpoint now accepts
`application/zip` via a scoped `_ZIP_MIME_TO_EXT` map — the shared media/
PDF allowlist stays zip-free. The exact-MIME cross-check at the sniff
step guards against zip<->PDF confusion under the shared 'document'
category. Canonical S3 key layout matches the legacy proxy path
(`document/{asset_id}/original.zip`), so the render-side head_object
gate is transparent to which upload path produced the asset.
The prior codegen commit added the generated createAssetUpload +
completeAssetUpload methods this commit consumes.
— Jerrai
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
miga-heygen
left a comment
There was a problem hiding this comment.
Review — feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)
Verdict: LGTM ✅
Context
Clean CLI-side counterpart to EF#41085 (which I just reviewed — LGTM there too). Moves from the 32 MB in-memory proxy to a three-step direct-to-S3 flow.
Invariant 1: No CLI auth on S3 PUT
putBytesToPresignedUrl uses raw fetchImpl (not client.request which injects auth headers). The presigned URL signature is bound to the headers signed at presign time — extra headers invalidate the signature (S3 returns 403). The test explicitly asserts absence of authorization and x-api-key in the PUT headers. ✅
Invariant 2: 409 retry, non-409 immediate
completeWithRetry traces clean:
err instanceof HyperframesApiError && err.status === 409→ retry with linear backoff (500ms × attempt)- Non-409 →
!retryable→ throw immediately - Last attempt exhausted → throw the 409
- Non-
HyperframesApiError(network error, etc.) →!retryable→ throw immediately
The retry is safe because completeAssetUpload is idempotent (documented in the API). ✅
Invariant 3: SHA256 triple-verified integrity
The checksum is computed once at the top of uploadZipViaDirectUpload, then:
- Passed to
createAssetUpload→ S3 enforces it on PUT (rejects mismatched bytes) - Passed as
upload_headers→ S3 cross-checks at write time - Passed to
completeAssetUpload→ EF's finalize cross-checks against the stored object
Three independent verification points. ✅
Invariant 4: asset_id namespace preserved
uploadZipViaDirectUpload returns asset_id from the complete response. The downstream createRender({project: {type: "asset_id", asset_id}}) at render.ts:581 is unchanged — both upload paths write into movio_asset, same namespace. ✅
upload.ts (154 lines)
Well-structured orchestrator. A few specific observations:
normalizeUploadHeaders:Record<string, unknown>from the OpenAPI spec needs coercion toRecord<string, string>. Defensively handles number/boolean coercion and skips objects. Right tradeoff between defensive and over-engineered.bytes as unknown as BodyInit: Avoids a 200 MB buffer copy for a known TypeScriptUint8Array<ArrayBufferLike>assignability gap. The comment explains why.detail.slice(0, 300): Caps error body in PUT failure messages. Prevents a huge S3 XML error from blowing up the error output.- Progress events: All four (
initialize,upload:0,upload:100,complete) signal "entering this phase", not "phase done". Consistent with the render.ts consumer which logs "initializing…", "uploading to S3…", "finalizing…".
Tests (9 cases)
Complete coverage of the contract:
- Request shape (filename, content_type, size_bytes, SHA256)
- PUT url + headers + body
- No CLI auth on PUT
- Complete threads asset_id + checksum
- 409 retry + succeed
- Non-409 no retry
- PUT failure with detail
- Idempotency key pass-through
- Progress event ordering
Codegen (commits 1)
Pure regeneration from EF master @ e74815f7af. Two new methods (createAssetUpload, completeAssetUpload) + associated types. encodeURIComponent(args.asset_id) in the path template is correct for path parameter encoding.
Ponytail lens
Lean already. Ship.
— Miga
miguel-heygen
left a comment
There was a problem hiding this comment.
Code/CI stamp for #1844.
Audited: packages/cli/src/cloud/upload.ts, packages/cli/src/cloud/upload.test.ts, packages/cli/src/commands/cloud/render.ts, generated endpoint surface in _gen/client.ts / _gen/types.ts, and the EF#41085 server contract for upload_headers / canonical zip keys.
Strengths:
packages/cli/src/cloud/upload.ts:148sends create → raw presigned PUT → complete in the right order, with the checksum threaded through create and complete.packages/cli/src/cloud/upload.ts:83uses rawfetchImplfor S3, so CLI auth headers never touch the presigned PUT.packages/cli/src/cloud/upload.test.ts:37through:232pins the important wire contracts: request shape, header pass-through, no auth on PUT, 409-only complete retry, idempotency key, and progress sequence.
No blocking code findings. CI is green, and local bun run --cwd packages/cli test src/cloud/upload.test.ts passed. I did not run the live >32 MB dev-cluster upload from the PR body; keep that as a manual merge-time validation if it is still required.
— Magi
Verdict: APPROVE
Reasoning: The CLI-side direct-upload orchestration matches the merged EF endpoint contract, preserves the downstream asset_id render path, and has focused tests for the failure-prone S3/header/retry behavior.
Summary
Migrates the HyperFrames CLI
cloud renderupload flow from the legacyPOST /v3/assetsproxy path (32 MB in-memory cap) to the three-step direct-to-S3 flow, lifting the practical per-project ceiling to 200 MB.Server-side counterpart: EF#41085 (merged 2026-07-02) added scoped
application/zipacceptance to the direct-upload endpoint.Flow
POST /v3/assets/direct-uploads— declares filename, content-type, size, and SHA256 checksum; returnsasset_id, presignedupload_url, and requiredupload_headers.PUTtoupload_urlwith the zip bytes +upload_headersverbatim. No CLI auth on this call — the presigned URL signature carries authorization, and any extra headers would invalidate the signature (S3 returns 403).POST /v3/assets/{asset_id}/complete— finalizes into a reusable asset. Retried up to 5x on 409 (documented "Uploaded object not found yet" race between S3 write consistency and finalize).The returned
asset_iduses the same namespace the legacy path produced (both write intomovio_asset), so the downstreamcreateRender({project: {type: "asset_id", asset_id}})submission atrender.ts:581is unchanged.Commits
chore(cli): regenerate cloud client...— pure codegen from EFmaster @ e74815f7af. Addsclient.createAssetUpload+client.completeAssetUpload+ associated request/response types. Regenerated manually viaPYTHONPATH=. python3 scripts/generate_hyperframes_cli_client.py --out ...because thesync-hyperframes-codegen.ymlworkflow failed with agh: Not Found (HTTP 404)on the PR-creation step (run 28556975483). See "Follow-up" below.feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)— the wire-up:packages/cli/src/cloud/upload.ts(new, 154 lines) — the three-step orchestrator +putBytesToPresignedUrl+completeWithRetryhelpers.packages/cli/src/cloud/upload.test.ts(new, 229 lines) — 9 unit tests covering request shape, SHA256 correctness, header pass-through, no-CLI-auth-on-PUT invariant, 409-retry-succeed, non-409-no-retry, PUT-failure detail surfacing, idempotency-key threading, and progress-event ordering.packages/cli/src/commands/cloud/render.ts— swapclient.uploadAssetforuploadZipViaDirectUpload. Header comment updated to describe the new 3-step flow + 200 MB cap.packages/cli/src/cloud/errors.ts— bump thehyperframes_project_too_largehint from "32 MB" to "200 MB".Test plan
bun run --cwd packages/cli typecheck— clean.bun run --cwd packages/cli test src/cloud/upload.test.ts— 9/9 pass.hyperframes cloud renderagainst a project ≤ 32 MB (regression: legacy path was working, new path should be a drop-in) — deferred to reviewer's e2e or dev-cluster validation.hyperframes cloud renderagainst a project > 32 MB and ≤ 200 MB (the new-capability case). Needs a dev API endpoint with EF#41085 deployed. Hold merge until this passes.Follow-up
The
sync-hyperframes-codegen.ymlworkflow that normally auto-opens PRs like this one failed on the merge of EF#41085 (run 28556975483). Codegen itself completed successfully; failure is in the "Create or update PR in hyperframes" shell step (gh: Not Found (HTTP 404)). Feels like a workflow-side infra issue (token scope or the--assignee jrusso1020path 404s on newerghvalidation) — worth a targeted fix so future codegen syncs don't require manual intervention. Filing that separately.Also noted (out of scope for this PR)
Per tai's review of EF#41085, hyperframes render lacks explicit zip-bomb protection (uncompressed-size + entry-count caps). This is a pre-existing gap on the legacy 32 MB path; the 200 MB direct-upload doesn't introduce it but widens the impact. Worth adding an EF-side guard in the render-submit path — separate issue.
Reviewer routing: per James's standing direction, finished HF PRs go to
#hyperframes-squad-internalwith@Migaand@Viaon the review post. Will post there after this PR is up.— Jerrai