Skip to content

Backport: feat (video): add first-class frameImages and inputReferences call options for video generation - #16503

Merged
joshualipman123 merged 1 commit into
release-v6.0from
backport-pr-16219-to-release-v6.0
Jun 30, 2026
Merged

Backport: feat (video): add first-class frameImages and inputReferences call options for video generation#16503
joshualipman123 merged 1 commit into
release-v6.0from
backport-pr-16219-to-release-v6.0

Conversation

@joshualipman123

Copy link
Copy Markdown
Contributor

Backport of #16219 to release-v6.0.

Summary

Promotes two image inputs to first-class top-level options on experimental_generateVideo, instead of requiring provider-specific passthrough:

  • frameImages — role-tagged images (first_frame / last_frame) for image-to-video and first-last-frame generation.
  • inputReferences — reference images for reference-to-video (r2v) generation.

Both are normalized once in core and passed through to providers, with warnings when they conflict (e.g. frameImages + inputReferences supplied together, or a first_frame overriding prompt.image).

Backport notes

release-v6.0 is on the VideoModelV3 spec, whereas the original PR targeted VideoModelV4 on main. Adaptations:

  • Added frameImages / inputReferences to the v3 call-options type and a new video-model-v3-frame-image.ts (frame-image / frame-type definitions). All v4-only files from the PR were dropped (v4 call-options, v4 frame-image, as-video-model-v4.test.ts).
  • generate-video.ts keeps the v3 satisfies Experimental_VideoModelV3CallOptions shape; all VideoModelV4* types swapped to their V3 equivalents.
  • Adapted normalizeImageData to v6.0's detectMediaType({ signatures: imageMediaTypeSignatures }) API (the PR used main-only { topLevelType: 'image' }).
  • The 7 provider implementations (google, google-vertex, alibaba, bytedance, klingai, xai, gateway) keep specificationVersion = 'v3' with VideoModelV4VideoModelV3 renames throughout.
  • google-generative-ai-video-model.ts keeps v6.0's class name GoogleGenerativeAIVideoModel (renamed to GoogleVideoModel on main).
  • google-vertex-video-model.ts keeps v6.0's vertexOptions variable name (renamed to googleVertexOptions on main).
  • Fixed the back-ported vertex examples + docs to use the v6.0 provider name vertex (the PR used the main-only googleVertex).

@joshualipman123
joshualipman123 requested a review from shaper June 30, 2026 07:00
@joshualipman123 joshualipman123 changed the title Backport: feat (video): add first-class and call options for video … Backport: feat (video): add first-class and call options for video generation Jun 30, 2026
@joshualipman123 joshualipman123 changed the title Backport: feat (video): add first-class and call options for video generation Backport: feat (video): add first-class frameImages and inputReferences call options for video generation Jun 30, 2026
@joshualipman123
joshualipman123 force-pushed the backport-pr-16219-to-release-v6.0 branch from 585b060 to 63a6551 Compare June 30, 2026 07:16
@joshualipman123
joshualipman123 merged commit fa850e6 into release-v6.0 Jun 30, 2026
48 checks passed
@joshualipman123
joshualipman123 deleted the backport-pr-16219-to-release-v6.0 branch June 30, 2026 07:25
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Published in:

Package Version
ai 6.0.217 github npm
@ai-sdk/alibaba 1.0.33 github npm
@ai-sdk/amazon-bedrock 4.0.126 github npm
@ai-sdk/angular 2.0.218 github npm
@ai-sdk/anthropic 3.0.91 github npm
@ai-sdk/anthropic-aws 1.0.13 github npm
@ai-sdk/assemblyai 2.0.40 github npm
@ai-sdk/azure 3.0.82 github npm
@ai-sdk/baseten 1.0.58 github npm
@ai-sdk/black-forest-labs 1.0.42 github npm
@ai-sdk/bytedance 1.0.22 github npm
@ai-sdk/cerebras 2.0.61 github npm
@ai-sdk/cohere 3.0.43 github npm
@ai-sdk/deepgram 2.0.40 github npm
@ai-sdk/deepinfra 2.0.59 github npm
@ai-sdk/deepseek 2.0.43 github npm
@ai-sdk/devtools 0.0.23 github npm
@ai-sdk/elevenlabs 2.0.40 github npm
@ai-sdk/fal 2.0.41 github npm
@ai-sdk/fireworks 2.0.63 github npm
@ai-sdk/gateway 3.0.141 github npm
@ai-sdk/gladia 2.0.40 github npm
@ai-sdk/google 3.0.87 github npm
@ai-sdk/google-vertex 4.0.154 github npm
@ai-sdk/groq 3.0.46 github npm
@ai-sdk/huggingface 1.0.57 github npm
@ai-sdk/hume 2.0.40 github npm
@ai-sdk/klingai 3.0.25 github npm
@ai-sdk/langchain 2.0.225 github npm
@ai-sdk/llamaindex 2.0.217 github npm
@ai-sdk/lmnt 2.0.40 github npm
@ai-sdk/luma 2.0.40 github npm
@ai-sdk/mcp 1.0.56 github npm
@ai-sdk/mistral 3.0.44 github npm
@ai-sdk/moonshotai 2.0.31 github npm
@ai-sdk/open-responses 1.0.23 github npm
@ai-sdk/openai 3.0.79 github npm
@ai-sdk/openai-compatible 2.0.55 github npm
@ai-sdk/perplexity 3.0.40 github npm
@ai-sdk/prodia 1.0.39 github npm
@ai-sdk/provider 3.0.13 github npm
@ai-sdk/provider-utils 4.0.34 github npm
@ai-sdk/quiverai 1.0.7 github npm
@ai-sdk/react 3.0.219 github npm
@ai-sdk/replicate 2.0.40 github npm
@ai-sdk/revai 2.0.40 github npm
@ai-sdk/rsc 2.0.217 github npm
@ai-sdk/svelte 4.0.217 github npm
@ai-sdk/togetherai 2.0.60 github npm
@ai-sdk/valibot 2.0.35 github npm
@ai-sdk/vercel 2.0.57 github npm
@ai-sdk/voyage 1.0.11 github npm
@ai-sdk/vue 3.0.217 github npm
@ai-sdk/xai 3.0.101 github npm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants