Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 107 additions & 2 deletions public-endpoints/ai-sdk.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ description: "Use the @runpod/ai-sdk-provider package to integrate Public Endpoi
tag: "NEW"
---

The `@runpod/ai-sdk-provider` package integrates Runpod Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This gives you a streamlined, type-safe interface for text generation, streaming, and image generation in JavaScript and TypeScript projects.
The `@runpod/ai-sdk-provider` package integrates Runpod Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This gives you a streamlined, type-safe interface for text generation, streaming, image generation, and video generation in JavaScript and TypeScript projects.

The Vercel AI SDK is a popular open-source library for building AI applications. By using the Runpod provider, you can access Runpod's Public Endpoints using the same patterns and APIs you'd use with other AI providers like OpenAI or Anthropic.

## Why use the Vercel AI SDK?

- **Unified interface**: Use the same `generateText`, `streamText`, and `generateImage` functions regardless of which AI provider you're using.
- **Unified interface**: Use the same `generateText`, `streamText`, `generateImage`, and `generateVideo` functions regardless of which AI provider you're using.
- **Type safety**: Full TypeScript support with typed responses and parameters.
- **Streaming built-in**: First-class support for streaming text responses.
- **Framework integrations**: Works seamlessly with Next.js, React, Svelte, and other frameworks.
Expand Down Expand Up @@ -261,6 +261,91 @@ const { image } = await generateImage({
| `maxPollAttempts` | Max polling attempts for async generation |
| `pollIntervalMillis` | Milliseconds between status polls |

## Video generation

Use `experimental_generateVideo` to generate videos from text prompts or images. The Runpod provider supports 15 video models, including Sora, Wan, Seedance, and Kling.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation: Video generation feature and model list sourced from PR #57, which adds videoModel()/video() methods implementing AI SDK v6's Experimental_VideoModelV3 interface. Model IDs, types, and provider options from src/runpod-video-options.ts and README.md updates.
View source


Video generation is asynchronous—the SDK submits a job, polls for completion, and returns the video URL when ready.

### Text-to-video

Generate videos from text prompts:

```typescript
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateVideo as generateVideo } from "ai";

const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A golden retriever running on a sunny beach, cinematic, 4k",
});

console.log(video.url);
```

The response includes:
- `video.url`: URL to the generated video
- `video.mediaType`: Video MIME type (`video/mp4`)

### Image-to-video

Animate an existing image:

```typescript
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateVideo as generateVideo } from "ai";

const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-i2v"),
prompt: "Animate this scene with gentle camera movement",
image: new URL("https://example.com/image.png"),
});

console.log(video.url);
```

### Video generation parameters

Control the video generation with additional parameters:

```typescript
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A serene mountain landscape with flowing water",
duration: 5,
aspectRatio: "16:9",
seed: 42,
});
```

### Video provider options

Pass model-specific parameters using `providerOptions`:

```typescript
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A serene mountain landscape with flowing water",
duration: 5,
aspectRatio: "16:9",
providerOptions: {
runpod: {
negative_prompt: "blurry, low quality",
guidance_scale: 7.5,
},
},
});
```

| Option | Description |
|--------|-------------|
| `negative_prompt` | Elements to exclude from the video |
| `guidance_scale` | How closely to follow the prompt |
| `num_inference_steps` | Number of inference steps |
| `style` | Style preset (model-specific) |
| `maxPollAttempts` | Max polling attempts (default: 120) |
| `pollIntervalMillis` | Milliseconds between status polls (default: 5000) |

## Supported models

### Text models
Expand All @@ -279,6 +364,26 @@ const { image } = await generateImage({
| `google-nano-banana-edit` | [Nano Banana Edit](/public-endpoints/models/nano-banana-edit). Supports multiple reference images. |
| `bytedance-seedream-4-0-t2i` | [Seedream 4.0](/public-endpoints/models/seedream-4-t2i). Text-to-image with good prompt adherence. |

### Video models
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation: 15 video models listed in PR #57 from VIDEO_MODEL_ID_TO_ENDPOINT_URL mapping in src/runpod-provider.ts. Companies and model types verified against PR description table.
View source

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation: Video models table updated to include Resolution, Aspect Ratios, and Duration columns based on changes in README.md. The PR also fixes alibaba/wan-2.2-t2v-720-lora model type and adds images array mapping for kwaivgi/kling-video-o1-r2v model.
View source


| Model ID | Type | Resolution | Aspect Ratios | Duration |
|----------|------|------------|---------------|----------|
| `pruna/p-video` | t2v | 720p, 1080p | 16:9, 9:16 | 5s |
| `vidu/q3-t2v` | t2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
| `vidu/q3-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
| `kwaivgi/kling-v2.6-std-motion-control` | i2v + video | 720p | 16:9, 9:16, 1:1 | 5, 10s |
| `kwaivgi/kling-video-o1-r2v` | i2v | 720p | 16:9, 9:16, 1:1 | 3–10s |
| `kwaivgi/kling-v2.1-i2v-pro` | i2v | 720p | 16:9, 9:16, 1:1 | 5, 10s |
| `alibaba/wan-2.6-t2v` | t2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
| `alibaba/wan-2.6-i2v` | i2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
| `alibaba/wan-2.5` | i2v | 480p, 720p, 1080p | 16:9, 9:16 | 5, 10s |
| `alibaba/wan-2.2-t2v-720-lora` | i2v | 720p | 16:9 | 5, 8s |
| `alibaba/wan-2.2-i2v-720` | i2v | 720p | 16:9 | 5, 8s |
| `alibaba/wan-2.1-i2v-720` | i2v | 720p | 16:9 | 5s |
| `bytedance/seedance-v1.5-pro-i2v` | i2v | 480p, 720p | 21:9, 16:9, 9:16, 1:1, 4:3, 3:4 | 4–12s |
| `openai/sora-2-pro-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |
| `openai/sora-2-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |

For a complete list of available models and their parameters, see the [model reference](/public-endpoints/reference).

## Example: Chat application
Expand Down
Loading