feat (video): add first-class frameImages and inputReferences call options for video generation - #16219
Merged
Merged
Conversation
joshualipman123
marked this pull request as ready for review
June 22, 2026 22:03
Co-authored-by: Cursor <cursoragent@cursor.com>
joshualipman123
force-pushed
the
feat/video-frame-images
branch
from
June 22, 2026 22:07
7f88673 to
5378c2e
Compare
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
gr2m
approved these changes
Jun 22, 2026
shaper
reviewed
Jun 26, 2026
Collaborator
There was a problem hiding this comment.
feedback from claude -- not sure if it is accurate, take a look?
🔴 Blocking — @ai-sdk/google image wire format is broken (regression)
packages/google/src/google-video-model.ts — the new convertFileToGoogleImage emits the Vertex shape:
return { bytesBase64Encoded: base64Data, mimeType: file.mediaType || 'image/png' };
But @ai-sdk/google targets https://generativelanguage.googleapis.com/v1beta (confirmed in google-provider.ts:152), and the Gemini Developer API requires the inlineData shape — { inlineData: { mimeType, data } } — which is exactly what main produced. bytesBase64Encoded is the Vertex-only field name. The two endpoints genuinely differ on this (Gemini API video docs, Vertex Veo reference).
This breaks three paths on the Google (non-Vertex) provider with a 400 "Unable to process input image":
- instance.image — existing i2v start frame (regression from main's inlineData)
- instance.lastFrame — new feature
- instance.referenceImages legacy path — convertProviderReferenceImage now returns bytesBase64Encoded/gcsUri, where main returned inlineData/gcsUri
The tests don't catch it because they were rewritten to assert the new (wrong) shape — main had it('should send image as inlineData'), the PR replaced it with it('should send image as bytesBase64Encoded'). This is the classic case of updated tests masking a wire-format regression; the "manual verification" in the description most likely exercised Vertex, not the generativelanguage endpoint.
Fix: @ai-sdk/google must keep wrapping inline bytes as { inlineData: { mimeType, data } } (and reference images likewise). google-vertex is correct as-is. The shared helper can't be identical across the two packages.
Contributor
Author
There was a problem hiding this comment.
ya, got gemini api & vertex mixed up
Collaborator
There was a problem hiding this comment.
another claude thought, wdyt?
🟡 Silent data loss when frameImages + inputReferences are both passed
In generate-video.ts, effectiveInputReferences is set to undefined whenever frameImages is present, and the test should ignore inputReferences when frameImages is provided locks this in — with no warning. Because the core nulls it before doGenerate, the provider-level "inputReferences ignored" warnings can never fire for this combination either. A user who passes both gets their references dropped silently. This should emit a Warning so it surfaces via logWarnings.
Contributor
Author
There was a problem hiding this comment.
dropping inputRefs is intentional but ya should warn before doing so.
Collaborator
There was a problem hiding this comment.
if below is accurate may be worth a logic edit, take a look and let me know thoughts. I would think we would want the providers to prioritize what we host into prompt.image as well -- is this consistent re: the gist of PR prioritizes in-general-prompt params over provider-options params, or did we go the other direction?
- image vs first_frame precedence: the core copies first_frame into image only when no prompt image exists, but every provider's resolveStartImage is firstFrame ?? image. So if a caller passes both prompt.image and a frameImages first_frame, the core keeps prompt.image in the image field yet providers use first_frame — prompt.image is silently ignored. Worth a comment documenting the intended precedence, or a warning.
Contributor
Author
There was a problem hiding this comment.
prompt.image and frameImages are both first class. i think it makes sense to keep prioritizing firstFrame as this is the more specific/intentional field. added a warning.
Collaborator
There was a problem hiding this comment.
- xai fileToXaiImageUrl builds data:${file.mediaType};base64,...; if mediaType is ever undefined you get data:undefined;base64,. Low risk since type:'file' normally carries it, but a fallback (?? 'image/png') matches the other providers.
Collaborator
There was a problem hiding this comment.
klingai buildMultiImageBody: the start-frame warning uses feature: 'image' while the last-frame warning uses feature: 'frameImages' — slightly inconsistent labeling.
…ntentToUint8Array Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… labeling, handle xai passing data:undefined
shaper
approved these changes
Jun 29, 2026
Contributor
|
🚀 Published in:
|
joshualipman123
added a commit
that referenced
this pull request
Jun 30, 2026
…es call options for video generation (#16503) Backport of #16219 to `release-v6.0`. ## Summary Promotes two image inputs to first-class top-level options on `experimental_generateVideo`, instead of requiring provider-specific passthrough: - **`frameImages`** — role-tagged images (`first_frame` / `last_frame`) for image-to-video and first-last-frame generation. - **`inputReferences`** — reference images for reference-to-video (r2v) generation. Both are normalized once in core and passed through to providers, with warnings when they conflict (e.g. `frameImages` + `inputReferences` supplied together, or a `first_frame` overriding `prompt.image`). ## Backport notes `release-v6.0` is on the **VideoModelV3** spec, whereas the original PR targeted **VideoModelV4** on `main`. Adaptations: - Added `frameImages` / `inputReferences` to the v3 call-options type and a new `video-model-v3-frame-image.ts` (frame-image / frame-type definitions). All v4-only files from the PR were dropped (`v4` call-options, `v4` frame-image, `as-video-model-v4.test.ts`). - `generate-video.ts` keeps the v3 `satisfies Experimental_VideoModelV3CallOptions` shape; all `VideoModelV4*` types swapped to their `V3` equivalents. - Adapted `normalizeImageData` to v6.0's `detectMediaType({ signatures: imageMediaTypeSignatures })` API (the PR used main-only `{ topLevelType: 'image' }`). - The 7 provider implementations (google, google-vertex, alibaba, bytedance, klingai, xai, gateway) keep `specificationVersion = 'v3'` with `VideoModelV4` → `VideoModelV3` renames throughout. - `google-generative-ai-video-model.ts` keeps v6.0's class name `GoogleGenerativeAIVideoModel` (renamed to `GoogleVideoModel` on main). - `google-vertex-video-model.ts` keeps v6.0's `vertexOptions` variable name (renamed to `googleVertexOptions` on main). - Fixed the back-ported vertex examples + docs to use the v6.0 provider name `vertex` (the PR used the main-only `googleVertex`).
joshualipman123
added a commit
that referenced
this pull request
Jul 1, 2026
Rebase the branch onto main now that the parent PR (#16219) has landed, keeping referenceType support while incorporating main's generateAudio and Google inlineData/GCS URI fixes. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements first-class, provider-agnostic media inputs for video generation. Today, last-frame and multi-reference images are buried under inconsistent providerOptions keys (klingai.imageTail vs bytedance.lastFrameImage, klingai.elementList vs bytedance.referenceImages). This PR lifts those concepts to top-level call options.
Manual Verification
Manually verified by running examples to check cases like:
Checklist
pnpm changesetin the project root)