sam-ui v0.4.0

sam-ui 0.4.0: agents that work alongside you, Find in clip, and a faster SAM 3. macOS 14 or newer, Apple Silicon (M1 or later); 16 GB of memory recommended, and needed for SAM 3. The browser demo needs WebGPU; it's tested in Chrome, Firefox and Safari. After Effects export needs AE MCP Vision v2.2.0.

• Agents (MCP), off by default. Claude Code, Codex or any MCP client can drive sam-ui: open a video, click or describe objects, track, review, capture frames with the masks drawn on, and export. Turn it on with Agents > Allow agents (MCP). The server listens on 127.0.0.1:8793 only, wants the bearer token in ~/.sam-ui/token, and refuses requests from web pages. (#55)
• Agents work in your view. On the video you have open, an agent's clicks, tracks and review marks show up as they happen. Its tracks fill the timeline lane like yours. An Agent chip names each change and lists the last 20 with Go to, and studio's normal undo undoes an agent's click. An agent can also read the frame and layer you're on, so you can ask it about what you're looking at. Changes on other videos raise a toast with Open. (#75, #73)
• Find in clip. With SAM 3, describe an object and sam-ui looks for it across the whole clip. Each appearance becomes a candidate range in the Review queue, so an object that leaves and comes back is two candidates. It writes no seeds and makes no track stale. (#50)
• SAM 3 at half precision on Apple Silicon. About 40% faster, with the app's worst case down from 6.9 to 6.1 GB. Masks stay within 0.96 IoU of full precision on every frame. Set SAM_UI_SAM3_DTYPE=fp32 to go back. (#57)
• Track finished notifications, opt-in. sam-ui > Notify When a Track Finishes shows a system notification when a track ends while sam-ui isn't in front, failures included. Clicking it opens that track's first review stop. (#53, #74)
• Edits that skip the model. Marking an absent range that only shortens a tracked stretch no longer re-tracks the window around it. (#56)
• Renaming. Rename an object from its timeline lane, and a group from its Actions menu on phones and tablets. (#47, #66)
• Keyboard and touch. The Layers list is one Tab stop: Up and Down select layers, and Shift+F10 or a right-click opens a layer's lane actions. Group buttons stay visible on phones. (#63, #51)
• After Effects footage. Linked footage can be removed from Media without touching the file. ProRes and other footage the browser can't play previews through a frame-exact H.264 proxy, while tracking and export read the original. Footage is re-hashed before export, and variable frame rate footage is refused (conform it first). (#48, #54, #52)
• More browsers. The browser demo now works in Firefox 157, about 2.4 times slower than Chrome, and tracks in Safari 27 (tested through Safari's automation). It no longer crashes WebKit on reload, and the Android Chrome preview no longer goes blank. (#64, #65, #49, #61)
• Reliability. A read during a save can no longer break the track store, and retention works when the data folder sits behind a symlink. (#58, #59)
• The app stays intact after first launch. The bundled backend no longer writes Python cache files inside the app, which broke its code signature after the first run. (#72)
• Measured hardware figures. The hardware table is now measured on an M4 Max, for SAM 2.1 large and tiny and SAM 3 at both precisions. (#60)

Unsigned build: macOS may ask you to approve the app in System Settings > Privacy & Security on first launch. See desktop/README.md for installation steps.

sam-ui v0.3.0

sam-ui 0.3.0: corrections, review, and an After Effects round trip. macOS 14 or newer, Apple Silicon (M1 or later); 16 GB of memory recommended, and needed for SAM 3. The browser demo needs WebGPU in Chrome or Edge. After Effects export needs AE MCP Vision v2.2.0.

• Corrections. A correction refines the tracked mask. On SAM 2, a negative click without a positive asks you to add one instead of replacing the mask. “Gone for a while?” lets you mark an absence. (#23)
• Absent ranges. Mark the frames where an object leaves the shot. They stay empty, and each side of the gap tracks from its own seeds. A positive click inside an absence ends it at that frame. (#26)
• Faster re-tracking. A correction re-tracks the stretch it affects. Clearing a seed can blank its affected span without another model run. (#28)
• Undo and restore. Undo or redo seed changes, move a frame's clicks to another object, and restore a kept track version with its seeds. (#37)
• Object groups. Reorder objects and organize them into named, colored groups. Exports keep group folders, with an optional union mask for each group. (#38)
• Candidate ranges. The timeline distinguishes unknown, candidate, present and absent frames. Review a candidate as Present, Absent or Reject. Present and candidate annotations do not change masks. (#39)
• Review queue. Jump through ranked stops where a track may need attention, see the reason, and mark “Looks right” as you go. (#40)
• Find by text. With SAM 3 selected, describe an object on the current frame and use the result as a tracking seed. SAM 3 still requires Meta's license acceptance and downloaded weights. (#27)
• Video decoding. PyAV replaces decord, improving frame selection for edit-list and transport-stream clips while keeping decoded frames bounded in memory. (#25)
• Safer file handling. Sessions only open listed media, and backend exports check their output paths before writing or replacing files. (#29)
• Upload sizing. Uploads now honor configured width and height limits without upscaling, including portrait and rotated footage. (#30)
• Originals and proxies, opt-in. Backend tools can retain an upload's original bytes and exact frame times, then generate a frame-for-frame inference proxy. Both features default off. They do not switch normal playback, tracking or export to the original; normal uploads still use the configured frame rate, 24 fps by default. (#31, #32)
• Phone and tablet layout. Studio adapts to small screens, with touch pan, pinch zoom and long-press point entry. Uploads continue across layout changes. Model support still depends on the device and browser. (#33)
• Update notices. The desktop app checks for releases and offers a link when a newer version is available. Help > Check for Updates checks on demand. Downloads and installation remain manual. (#34)
• Five-minute browser clips. The browser demo now accepts clips up to five minutes, with a benchmark tool for checking long-track memory. It still uses SAM 2.1 tiny and keeps your video in the browser. (#35)
• Refinement comparisons. A benchmark tool compares six refinement strategies against SAM 2 large, with timing and mask metrics. This is developer tooling, not a new automatic refinement mode. (#36)
• SAM 3 memory. The detector shares the tracker's backbone, cache is released after jobs, and idle models unload. Dropped engines no longer stay alive through their idle timers. Warm prompts keep their cache. (#41)
• After Effects. In the desktop app, open supported footage from AE in place and export objects as animated masks in a new comp. Native frame size, frame rate and frame count are checked before export. Requires AE MCP Vision v2.2.0. (#24)
• Redesign. The Compositing Suite brings a dense timeline, grouped object layers and a review dock into one workspace. Layer colors and clearer track states make it easier to follow objects and review corrections. (#42)
• Refine Detail, experimental. Set SAM_UI_REFINE_DETAIL=1 before starting the backend, then choose Experimental > Refine Detail in Studio. Box or click a small detail, refine the crop with SAM 2.1, and apply the result to the current frame only. It crops the working copy; full-resolution originals come later. Details appear in the preview, matte export and AE handoff without changing tracking, and each Apply or Remove can be undone. This tool is off by default and unavailable in the browser-only engine. (#43)

Unsigned build: macOS may ask you to approve the app in System Settings > Privacy & Security on first launch. See desktop/README.md for installation steps.

sam-ui v0.2.0

sam-ui 0.2.0: long clips, new exports, and a browser demo. macOS 14 or newer, Apple Silicon (M1 or later); 16 GB of memory recommended, and needed for SAM 3 (see Hardware in the README).

• Long clips. Memory no longer grows with clip length: frames are decoded as tracking and playback reach them, and tracking state keeps only what the model reads again. Uploads take up to 5 minutes and 2 GB (Meta's demo limit was 10 seconds); a longer clip keeps its start, and studio says so before it uploads.
• Exports. Mask videos (one black and white MP4 per object, in a zip), the PNG roto working folder, and vector JSON (per-frame outlines for After Effects masks). Each export records the engine and model that made it. Export to After Effects is coming.
• Object names. Rename objects in place; exports use the names.
• Engines. The picker lists every model and says why one can't run, with Set up SAM 3 right there when its weights are missing. SAM 3 still needs you to accept Meta's SAM License on Hugging Face; the app never ships its weights.
• Fixes. Tracking two objects first clicked on different frames no longer crashes the backend on Apple GPUs. The app no longer falls back to browser-only mode when its backend is slow to start. A downloaded copy no longer opens as "damaged".
• Browser demo. SAM 2.1 tiny runs in the browser on WebGPU, with no server and nothing uploaded, at https://volksrat71.github.io/sam-ui/ (Chrome or Edge on desktop; clips up to 2 minutes). It is a demo: the desktop app's SAM 2.1 large and SAM 3 give much better masks.

Unsigned build: the first time, macOS says it can't verify the app. On macOS 15 and later, open it once, then click Open Anyway in System Settings > Privacy & Security (on macOS 14, right-click the app and choose Open). Or run xattr -dr com.apple.quarantine /Applications/sam-ui.app. See desktop/README.md.

sam-ui v0.1.0

First desktop release of sam-ui: studio and the SAM 2 / SAM 3 backend in one macOS app (Apple Silicon).

• Runs the full SAM 2.1 large model locally on your GPU (MPS). It downloads the checkpoint on first run (about 900 MB) and verifies its SHA-256.
• Per-object cached tracks: Track only re-runs objects that are new or changed. You can keep clicking while a track runs.
• SAM 3 tracking is optional. Accept Meta's SAM License on Hugging Face, then use SAM 3 > Download SAM 3 with a Hugging Face token. Every file is hash-checked, and the token is never saved. You can also point the app at weights you already have.
• Exports an MP4 with per-object effects, and a rotoscoping working folder.

Unsigned build: right-click the app and choose Open the first time, or run xattr -dr com.apple.quarantine /Applications/sam-ui.app. See desktop/README.md.