Skip to content

feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)#1844

Merged
jrusso1020 merged 2 commits into
mainfrom
feat/cli-direct-upload
Jul 2, 2026
Merged

feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)#1844
jrusso1020 merged 2 commits into
mainfrom
feat/cli-direct-upload

Conversation

@jrusso1020

Copy link
Copy Markdown
Collaborator

Summary

Migrates the HyperFrames CLI cloud render upload flow from the legacy POST /v3/assets proxy path (32 MB in-memory cap) to the three-step direct-to-S3 flow, lifting the practical per-project ceiling to 200 MB.

Server-side counterpart: EF#41085 (merged 2026-07-02) added scoped application/zip acceptance to the direct-upload endpoint.

Flow

  1. POST /v3/assets/direct-uploads — declares filename, content-type, size, and SHA256 checksum; returns asset_id, presigned upload_url, and required upload_headers.
  2. Raw PUT to upload_url with the zip bytes + upload_headers verbatim. No CLI auth on this call — the presigned URL signature carries authorization, and any extra headers would invalidate the signature (S3 returns 403).
  3. POST /v3/assets/{asset_id}/complete — finalizes into a reusable asset. Retried up to 5x on 409 (documented "Uploaded object not found yet" race between S3 write consistency and finalize).

The returned asset_id uses the same namespace the legacy path produced (both write into movio_asset), so the downstream createRender({project: {type: "asset_id", asset_id}}) submission at render.ts:581 is unchanged.

Commits

  1. chore(cli): regenerate cloud client... — pure codegen from EF master @ e74815f7af. Adds client.createAssetUpload + client.completeAssetUpload + associated request/response types. Regenerated manually via PYTHONPATH=. python3 scripts/generate_hyperframes_cli_client.py --out ... because the sync-hyperframes-codegen.yml workflow failed with a gh: Not Found (HTTP 404) on the PR-creation step (run 28556975483). See "Follow-up" below.
  2. feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB) — the wire-up:
    • packages/cli/src/cloud/upload.ts (new, 154 lines) — the three-step orchestrator + putBytesToPresignedUrl + completeWithRetry helpers.
    • packages/cli/src/cloud/upload.test.ts (new, 229 lines) — 9 unit tests covering request shape, SHA256 correctness, header pass-through, no-CLI-auth-on-PUT invariant, 409-retry-succeed, non-409-no-retry, PUT-failure detail surfacing, idempotency-key threading, and progress-event ordering.
    • packages/cli/src/commands/cloud/render.ts — swap client.uploadAsset for uploadZipViaDirectUpload. Header comment updated to describe the new 3-step flow + 200 MB cap.
    • packages/cli/src/cloud/errors.ts — bump the hyperframes_project_too_large hint from "32 MB" to "200 MB".

Test plan

  • bun run --cwd packages/cli typecheck — clean.
  • bun run --cwd packages/cli test src/cloud/upload.test.ts — 9/9 pass.
  • Local hyperframes cloud render against a project ≤ 32 MB (regression: legacy path was working, new path should be a drop-in) — deferred to reviewer's e2e or dev-cluster validation.
  • Local hyperframes cloud render against a project > 32 MB and ≤ 200 MB (the new-capability case). Needs a dev API endpoint with EF#41085 deployed. Hold merge until this passes.

Follow-up

The sync-hyperframes-codegen.yml workflow that normally auto-opens PRs like this one failed on the merge of EF#41085 (run 28556975483). Codegen itself completed successfully; failure is in the "Create or update PR in hyperframes" shell step (gh: Not Found (HTTP 404)). Feels like a workflow-side infra issue (token scope or the --assignee jrusso1020 path 404s on newer gh validation) — worth a targeted fix so future codegen syncs don't require manual intervention. Filing that separately.

Also noted (out of scope for this PR)

Per tai's review of EF#41085, hyperframes render lacks explicit zip-bomb protection (uncompressed-size + entry-count caps). This is a pre-existing gap on the legacy 32 MB path; the 200 MB direct-upload doesn't introduce it but widens the impact. Worth adding an EF-side guard in the render-submit path — separate issue.


Reviewer routing: per James's standing direction, finished HF PRs go to #hyperframes-squad-internal with @Miga and @Via on the review post. Will post there after this PR is up.

— Jerrai

jrusso1020 and others added 2 commits July 2, 2026 00:36
…ssetUpload

Regenerated from experiment-framework `master` at commit `e74815f7af` (the
merge of EF#41085, which added `/v3/assets/direct-uploads` +
`/v3/assets/{asset_id}/complete` to the `TARGET_ENDPOINTS` allowlist in
`scripts/generate_hyperframes_cli_client.py`).

The `sync-hyperframes-codegen.yml` workflow that normally auto-opens this
PR failed with a `gh: Not Found (HTTP 404)` on the PR-creation step (run
28556975483); regenerated manually with:

  cd experiment-framework
  PYTHONPATH=. python3 scripts/generate_hyperframes_cli_client.py \\
    --out /path/to/hyperframes-oss

This commit is codegen-only — no hand edits. The direct-upload wire-up
that consumes the new `createAssetUpload` + `completeAssetUpload` methods
lands in the follow-up commit.

— Jerrai

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…200MB)

Replaces the legacy `client.uploadAsset(...)` multipart POST to
`/v3/assets` (32 MB in-memory proxy path) with the three-step direct-to-
S3 flow that lifts the practical per-project ceiling to 200 MB:

  1. `POST /v3/assets/direct-uploads` — declares filename, content-type,
     size, and SHA256 checksum; returns `asset_id`, presigned
     `upload_url`, and required `upload_headers`.
  2. Raw `PUT` to `upload_url` with the zip bytes + `upload_headers`
     verbatim. No CLI auth attached — the presigned URL signature carries
     authorization, and any extra headers would break the signature.
  3. `POST /v3/assets/{asset_id}/complete` — finalizes into a reusable
     asset. Retried up to 5x on 409 ("Uploaded object not found yet"), a
     documented race between S3 write consistency and the finalize check.

The returned `asset_id` is the same namespace the legacy path produced
(both write into `movio_asset`), so the downstream render submission at
`createRender({project: {type: "asset_id", asset_id}})` is unchanged.

Server-side context (EF#41085): the direct-upload endpoint now accepts
`application/zip` via a scoped `_ZIP_MIME_TO_EXT` map — the shared media/
PDF allowlist stays zip-free. The exact-MIME cross-check at the sniff
step guards against zip<->PDF confusion under the shared 'document'
category. Canonical S3 key layout matches the legacy proxy path
(`document/{asset_id}/original.zip`), so the render-side head_object
gate is transparent to which upload path produced the asset.

The prior codegen commit added the generated createAssetUpload +
completeAssetUpload methods this commit consumes.

— Jerrai

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@miga-heygen miga-heygen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — feat(cli): migrate cloud-render upload to /v3/assets/direct-uploads (200MB)

Verdict: LGTM

Context

Clean CLI-side counterpart to EF#41085 (which I just reviewed — LGTM there too). Moves from the 32 MB in-memory proxy to a three-step direct-to-S3 flow.

Invariant 1: No CLI auth on S3 PUT

putBytesToPresignedUrl uses raw fetchImpl (not client.request which injects auth headers). The presigned URL signature is bound to the headers signed at presign time — extra headers invalidate the signature (S3 returns 403). The test explicitly asserts absence of authorization and x-api-key in the PUT headers. ✅

Invariant 2: 409 retry, non-409 immediate

completeWithRetry traces clean:

  • err instanceof HyperframesApiError && err.status === 409 → retry with linear backoff (500ms × attempt)
  • Non-409 → !retryable → throw immediately
  • Last attempt exhausted → throw the 409
  • Non-HyperframesApiError (network error, etc.) → !retryable → throw immediately

The retry is safe because completeAssetUpload is idempotent (documented in the API). ✅

Invariant 3: SHA256 triple-verified integrity

The checksum is computed once at the top of uploadZipViaDirectUpload, then:

  1. Passed to createAssetUpload → S3 enforces it on PUT (rejects mismatched bytes)
  2. Passed as upload_headers → S3 cross-checks at write time
  3. Passed to completeAssetUpload → EF's finalize cross-checks against the stored object

Three independent verification points. ✅

Invariant 4: asset_id namespace preserved

uploadZipViaDirectUpload returns asset_id from the complete response. The downstream createRender({project: {type: "asset_id", asset_id}}) at render.ts:581 is unchanged — both upload paths write into movio_asset, same namespace. ✅

upload.ts (154 lines)

Well-structured orchestrator. A few specific observations:

  • normalizeUploadHeaders: Record<string, unknown> from the OpenAPI spec needs coercion to Record<string, string>. Defensively handles number/boolean coercion and skips objects. Right tradeoff between defensive and over-engineered.
  • bytes as unknown as BodyInit: Avoids a 200 MB buffer copy for a known TypeScript Uint8Array<ArrayBufferLike> assignability gap. The comment explains why.
  • detail.slice(0, 300): Caps error body in PUT failure messages. Prevents a huge S3 XML error from blowing up the error output.
  • Progress events: All four (initialize, upload:0, upload:100, complete) signal "entering this phase", not "phase done". Consistent with the render.ts consumer which logs "initializing…", "uploading to S3…", "finalizing…".

Tests (9 cases)

Complete coverage of the contract:

  1. Request shape (filename, content_type, size_bytes, SHA256)
  2. PUT url + headers + body
  3. No CLI auth on PUT
  4. Complete threads asset_id + checksum
  5. 409 retry + succeed
  6. Non-409 no retry
  7. PUT failure with detail
  8. Idempotency key pass-through
  9. Progress event ordering

Codegen (commits 1)

Pure regeneration from EF master @ e74815f7af. Two new methods (createAssetUpload, completeAssetUpload) + associated types. encodeURIComponent(args.asset_id) in the path template is correct for path parameter encoding.

Ponytail lens

Lean already. Ship.

— Miga

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code/CI stamp for #1844.

Audited: packages/cli/src/cloud/upload.ts, packages/cli/src/cloud/upload.test.ts, packages/cli/src/commands/cloud/render.ts, generated endpoint surface in _gen/client.ts / _gen/types.ts, and the EF#41085 server contract for upload_headers / canonical zip keys.

Strengths:

  • packages/cli/src/cloud/upload.ts:148 sends create → raw presigned PUT → complete in the right order, with the checksum threaded through create and complete.
  • packages/cli/src/cloud/upload.ts:83 uses raw fetchImpl for S3, so CLI auth headers never touch the presigned PUT.
  • packages/cli/src/cloud/upload.test.ts:37 through :232 pins the important wire contracts: request shape, header pass-through, no auth on PUT, 409-only complete retry, idempotency key, and progress sequence.

No blocking code findings. CI is green, and local bun run --cwd packages/cli test src/cloud/upload.test.ts passed. I did not run the live >32 MB dev-cluster upload from the PR body; keep that as a manual merge-time validation if it is still required.

— Magi

Verdict: APPROVE
Reasoning: The CLI-side direct-upload orchestration matches the merged EF endpoint contract, preserves the downstream asset_id render path, and has focused tests for the failure-prone S3/header/retry behavior.

@jrusso1020 jrusso1020 merged commit a59ff0d into main Jul 2, 2026
41 checks passed
@jrusso1020 jrusso1020 deleted the feat/cli-direct-upload branch July 2, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants