feat(realtime): cap capture FPS at 30 via ideal/max constraint

[AdirAmsalem]

Summary

Realtime models now request capture FPS via an { ideal: 30, max: 30 } constraint instead of a hardcoded per-model rate (was 20 for Lucy 2.1, 22 for Lucy Restyle 2). The camera delivers its native rate (typically 24/25/30) and the server resamples to whatever the model needs.

Why

• Decouples the SDK from server-side FPS support — bumping a model's supported rate no longer requires a client release.
• Avoids client-side frame duplication / dropping when the camera's native rate doesn't match the model's rate. Resampling now happens once, on the server, where it can be done correctly with PTS-aware logic.
• Keeps a steady frame supply ahead of inference so the GPU isn't waiting on the next frame.

User-facing impact

None for the documented pattern — frameRate: model.fps in getUserMedia still compiles and works, because MediaTrackConstraints["frameRate"] natively accepts both number and constraint-object forms:

const model = models.realtime("lucy-2.1");
const stream = await navigator.mediaDevices.getUserMedia({
  video: { frameRate: model.fps, width: model.width, height: model.height },
});

Video/image model fps stays a number (it's output-rate metadata, not a capture constraint); the type narrowing is enforced at the models.video(...) / models.image(...) getter boundary so consumers of those registries see no change.

Test plan

• pnpm typecheck passes
• pnpm test — 199/199 unit tests pass (added regression guard that video/image fps stays typed as number)
• pnpm test:e2e:realtime against staging across all realtime models
• Manual browser check: track.getSettings().frameRate is ≤ 30 across cameras with native 24/25/30/60 rates

---

Note

Medium Risk
Changes the public models.realtime(...).fps shape from a number to a constraint object, which may break consumers doing numeric operations or passing it to APIs expecting a scalar. Runtime behavior changes capture/mirroring FPS handling, so regressions would show up as altered stream cadence or compatibility issues in edge browsers.

Overview
Realtime model FPS is now represented as a capture constraint object rather than a fixed per-model number. The realtime registry entries switch to fps: { ideal: 30, max: 30 }, and modelDefinitionSchema/ModelDefinition are updated to accept either a number or constraint-object form.

Adds resolveFpsNumber() to derive a scalar FPS when needed, and uses it in realtime/client.ts when creating mirrored streams. Tests are updated to assert the new realtime fps shape, add coverage for resolveFpsNumber, and ensure models.video(...)/models.image(...) still expose fps as a number; the realtime E2E synthetic stream now captures at 30 FPS explicitly.

^{Reviewed by Cursor Bugbot for commit 49342d1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@decartai/sdk@141

commit: 49342d1