Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Do not embed a proprietary general chatbot. BeatDesign must be complete without
- Canvas layout snapshots are a UI persistence exception for drag, resize, and viewport state; external Agents still use semantic Canvas operations.
- Agent changes must become visible in the browser workspace and remain inspectable, reversible where supported, and verifiable. A database revision alone is not proof of a successful user-visible operation.
- Studio, Canvas, Editor, Assets, and MCP share project, task, asset, and generation services rather than duplicating business logic.
- Preview and MP4 export use browser-native WebCodecs and Mediabunny. Do not make system FFmpeg a requirement for the core localhost UI. Node-side MCP frame extraction may use `ffmpeg` from `PATH` or `BEATDESIGN_FFMPEG` and must fail with a clear setup error when unavailable.
- Preview and browser-side MP4 export use browser-native WebCodecs and Mediabunny. Do not make system FFmpeg a requirement for the core localhost UI. Node-side MCP frame extraction and Timeline rendering may use `ffmpeg` from `PATH` or `BEATDESIGN_FFMPEG`; Timeline rendering also uses `ffprobe` from `PATH` or `BEATDESIGN_FFPROBE`. These tools must fail with a clear setup error when a required binary is unavailable.

## Provider and storage boundary

Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to BeatDesign are documented in this file.

- Image overlays in the local Editor, with UI and MCP controls for placement, size, opacity, rotation, fades, and replacement with any project-owned image Asset.
- Four caption style presets plus per-cue text, timing, size, width, and vertical-position controls shared by the UI and MCP command path.
- MCP `bdesign_editor_render` support for rendering the authoritative Timeline to a project-owned MP4 with visible clips, overlays, caption burn-in, and mixed audio.
- Japanese localization across the application and public READMEs.

### Changed
Expand All @@ -21,6 +22,7 @@ All notable changes to BeatDesign are documented in this file.

- Prepared connected local and generated references through the shared upload bridge before remote generation so providers receive public HTTPS media URLs.
- Made static video previews seek past common opening black frames.
- Invalidated stale Timeline renders after render-affecting UI or MCP edits and synchronized the current render state with an existing Canvas Timeline card.

## [0.2.2] - 2026-09-03

Expand Down
2 changes: 1 addition & 1 deletion README.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ pnpm dev

## AgentからBeatDesignを使う

BeatDesignはProject、Asset、Canvas、生成、Editor操作をカバーする26個のローカルMCPツールを提供します。Agentによる変更は同じProjectサービスを通り、ブラウザーのワークスペースに表示されます。
BeatDesignはProject、Asset、Canvas、生成、Editor操作をカバーする27個のローカルMCPツールを提供し、正式なTimelineからのMP4書き出しにも対応します。Agentによる変更は同じProjectサービスを通り、ブラウザーのワークスペースに表示されます。

MCP Hostへ接続した後は、次のように依頼できます。

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Add your own [BeatAPI API key](https://beatapi.io/dashboard/apikeys) only when y

## Use BeatDesign with an Agent

BeatDesign exposes 26 local MCP tools for Projects, Assets, Canvas, generation, and Editor operations. Agent changes use the same project services and become visible in the browser workspace.
BeatDesign exposes 27 local MCP tools for Projects, Assets, Canvas, generation, and Editor operations, including authoritative MP4 timeline rendering. Agent changes use the same project services and become visible in the browser workspace.

After connecting your MCP host, you can ask:

Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ pnpm dev

## 让 Agent 操作 BeatDesign

BeatDesign 提供 26 个本地 MCP 工具,覆盖 Project、Asset、Canvas、生成和 Editor 操作。Agent 修改会经过同一套项目服务,并显示在浏览器工作空间中。
BeatDesign 提供 27 个本地 MCP 工具,覆盖 Project、Asset、Canvas、生成和 Editor 操作,包括从权威时间线导出 MP4。Agent 修改会经过同一套项目服务,并显示在浏览器工作空间中。

连接 MCP Host 后,可以直接提出这样的要求。

Expand Down
25 changes: 18 additions & 7 deletions docs/MCP.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ installation shapes.

## Tool groups

There are **26** tools:
There are **27** tools:

- Project (5): list, get, create, target the current MCP session, and open a
workspace review surface.
Expand All @@ -103,8 +103,8 @@ There are **26** tools:
continue-from-tail-frame.
- Generation (5): list model capabilities, read one model, submit an
asset-first request, refresh status, and list history.
- Editor (7): get, incremental edit, SRT import, semantic snapshot, diagnostics,
deep-link view, and command history.
- Editor (8): get, incremental edit, SRT import, authoritative MP4 render,
semantic snapshot, diagnostics, deep-link view, and command history.

MCP writes use `origin=mcp` assigned inside the server. `canvas.apply` and
`editor.apply` accept stable IDs, revisions, and idempotency keys. The server
Expand Down Expand Up @@ -157,6 +157,14 @@ Use `update_caption` to tune one caption cue's normalized font size, maximum
width, and bottom position without changing later cues; `set_caption_style`
continues to select the shared visual preset.

Use `bdesign_editor_render` to render the authoritative saved timeline to a
project-owned MP4 Asset. The render includes visible video and image clips,
image overlays, caption burn-in, and mixed audio. The tool requires `ffmpeg`
and `ffprobe` on `PATH`, or explicit `BEATDESIGN_FFMPEG` and
`BEATDESIGN_FFPROBE` paths. If the timeline changes while a render is running,
the revision-checked commit rejects the stale output and removes that attempt's
temporary Asset.

For a newly connected Canvas node, append a `place_card` operation after its
`upsert_card`. By default it places the target once to the right of the frames
listed in `sourceCardIds`, or to the right of the card's `referenceCardIds` when
Expand All @@ -171,12 +179,15 @@ generation independently of prompt text; BeatDesign does not insert synthetic
- `bdesign_editor_snapshot` resolves active clips and source times; it does not
rasterize a pixel frame yet.
- `bdesign_asset_extract_frame` and `bdesign_canvas_continue_from_tail` decode
the local video file. MCP/Node uses `ffmpeg` on PATH (or `BEATDESIGN_FFMPEG`);
this is not a required system install for the browser UI.
the local video file. MCP/Node frame extraction and timeline rendering use
`ffmpeg` on PATH (or `BEATDESIGN_FFMPEG`); timeline rendering also uses
`ffprobe` (or `BEATDESIGN_FFPROBE`). These are not required system installs
for the browser UI.
- SRT import validates the whole subtitle document before replacing the current
caption track. Malformed input leaves the saved timeline unchanged.
- Browser-only MP4 export is not exposed as a headless MCP tool yet. Caption
burn-in is included when the browser exports an MP4.
- Browser-native MP4 export remains available without system FFmpeg.
`bdesign_editor_render` provides the corresponding MCP/Node export path and
writes the result back as a project Asset.
- `bdesign_asset_import` copies a local image, video, or audio file into the
project Asset library from an absolute path. It does not place the Asset on
Canvas or Editor; use `bdesign_canvas_apply` or `bdesign_editor_edit` after.
Expand Down
12 changes: 6 additions & 6 deletions docs/PRODUCT_PLAN_AND_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Codex / Claude Code / Other Agent
### 媒体技术

- 使用 WebCodecs + Mediabunny 在浏览器完成媒体检查、编码和 MP4 封装。
- 核心浏览器预览和 MP4 导出不要求系统 FFmpeg;可选的 MCP/Node 视频抽帧使用 `PATH` 中的 `ffmpeg``BEATDESIGN_FFMPEG`。
- 核心浏览器预览和 MP4 导出不要求系统 FFmpeg;MCP/Node 视频抽帧与权威时间线 MP4 导出使用 `PATH` 中的 `ffmpeg`/`ffprobe`,也可通过 `BEATDESIGN_FFMPEG` 和 `BEATDESIGN_FFPROBE` 指定
- OpenReel 只作为时间线术语、文档模型和非破坏式编辑行为的参考;未打包其完整 UI 和应用外壳。
- 媒体 metadata 采用限并发队列并带超时释放;单个损坏媒体不会永久阻塞后续卡片。

Expand All @@ -144,7 +144,7 @@ Codex / Claude Code / Other Agent
- Generation 的 `AssetFirstGenerationRequest` 已成为服务端权威输入:适配器媒体参数由 Asset ID 和当前 generation intent 编译,旧的客户端 URL 字段不再决定引用事实。
- UI 命令入口不再接受客户端 `origin`;服务端固定写入 `ui`,MCP 入口在内核边界固定写入 `mcp`。
- Provider Contract 已将逻辑模型目录与 BeatAPI effectId、上传路径和上游模型名拆开;BeatAPI 是默认实现,Fork 可在源码扩展点注册其他 Provider。
- 本地 stdio MCP Server 提供 26 个工具(Project / Asset / Canvas / Generation / Editor),模型和参数通过 capability discovery 暴露;Canvas / Editor 增量操作使用完整 JSON Schema,Agent 可直接发现操作类型和参数。MCP 生成直接调用当前 Provider,本地产品不重复实现 API Key、余额、计费或限流策略,只透传 Provider 的结果与错误。`bdesign_project_target` 绑定当前会话项目,Project/Canvas/Editor view 工具返回 Codex Browser handoff;`bdesign_asset_import` 把本地文件导入项目 Asset 库;`bdesign_asset_extract_frame` 与 `bdesign_canvas_continue_from_tail` 负责抽帧续写;Editor MCP 可导入 SRT、放置和替换任意项目图片 Overlay,并调整叠层与单条字幕参数
- 本地 stdio MCP Server 提供 27 个工具(Project / Asset / Canvas / Generation / Editor),模型和参数通过 capability discovery 暴露;Canvas / Editor 增量操作使用完整 JSON Schema,Agent 可直接发现操作类型和参数。MCP 生成直接调用当前 Provider,本地产品不重复实现 API Key、余额、计费或限流策略,只透传 Provider 的结果与错误。`bdesign_project_target` 绑定当前会话项目,Project/Canvas/Editor view 工具返回 Codex Browser handoff;`bdesign_asset_import` 把本地文件导入项目 Asset 库;`bdesign_asset_extract_frame` 与 `bdesign_canvas_continue_from_tail` 负责抽帧续写;Editor MCP 可导入 SRT、放置和替换任意项目图片 Overlay、调整叠层与单条字幕参数,并通过 `bdesign_editor_render` 将权威时间线导出为项目内 MP4 Asset
- Codex、Claude Code 与 WorkBuddy 接入包内含 `beatdesign-workspace` Skill,负责项目选择、字幕/续写工具编排、付费生成停点和可视化复核;三者共用同一 MCP 与本地 Project 数据,其中 Claude Code 和 WorkBuddy 使用本机 HTTP MCP。

## 6. v0.2 Phase 1 本地已实现
Expand All @@ -159,7 +159,7 @@ Codex / Claude Code / Other Agent
- Canvas -> Timeline Node -> Editor 连续工作流。
- Editor 自动保存接入命令入口,并补齐冲突三方合并、重复操作保护和稳定播放头时间。
- 图片 Clip、时间线拖拽调整持续时间与图片/视频统一视觉轨。
- 本地 MCP Server 提供 26 个 Project、Asset、Canvas、Generation、Editor 工具;支持会话项目绑定、Canvas/Editor 可视化交接、从绝对路径导入本地素材、抽取尾帧续写、导入和精调 SRT 字幕,以及放置、替换和调整图片 Overlay。
- 本地 MCP Server 提供 27 个 Project、Asset、Canvas、Generation、Editor 工具;支持会话项目绑定、Canvas/Editor 可视化交接、从绝对路径导入本地素材、抽取尾帧续写、导入和精调 SRT 字幕、放置/替换/调整图片 Overlay,以及权威时间线 MP4 导出
- MCP 增量 Canvas/Editor 命令在短暂 revision 竞争时会基于最新权威文档限次自动重放;持续冲突返回最新 revision 和明确重试提示。
- Canvas 与 Editor 每 2 秒并在页面重新聚焦时检查 MCP 写入的新 revision。

Expand Down Expand Up @@ -189,9 +189,9 @@ Codex / Claude Code / Other Agent
- MCP Resources 和更完整的 schema versioning。
- Agent Activity、命令审计和实时 UI 事件桥。
- 外部市场正式审核与上架。仓库已提供 Codex 本地插件、可直接添加的 Claude Code 仓库插件市场,以及符合目录结构的 WorkBuddy MCP + Skill Connector;这些本地接入包不等于已通过第三方市场审核。
- headless 预览/导出本地媒体 Worker。
- 独立的 headless 像素预览与后台媒体 Worker;当前 MCP MP4 导出在本地 MCP Server 进程中完成

当前可以称为“已支持本地 MCP 基础版”,但不能称为完整 Agent 编辑环境:像素级 Snapshot、headless 导出、实时 UI 事件和永久审计仍未实现。本地文件导入桥已完成;UI 当前使用 2 秒 revision 轮询和聚焦检查,而不是实时事件推送。
当前可以称为“已支持本地 MCP 基础版”,但不能称为完整 Agent 编辑环境:像素级 Snapshot、独立后台媒体 Worker、实时 UI 事件和永久审计仍未实现。本地文件导入桥和 MCP 权威时间线 MP4 导出已完成;UI 当前使用 2 秒 revision 轮询和聚焦检查,而不是实时事件推送。

Canvas 的拖拽、缩放、视口和完整布局仍使用 revision-checked Snapshot 自动保存;Canvas/Timeline 业务 operation 已有 Command 合同,Timeline Node 回写也已接入 `/commands`。后续做 MCP parity 时,应继续把可语义化的 Canvas UI 动作迁移为 `canvas.apply`,不能让外部 Agent 调用完整 Snapshot 覆盖。

Expand All @@ -200,7 +200,7 @@ Canvas 的拖拽、缩放、视口和完整布局仍使用 revision-checked Snap
- Electron/Tauri 壳。
- 内置 Node runtime、MCP 和媒体 Worker。
- 原生文件选择、系统集成、签名、公证和自动更新。
- 桌面封装阶段再评估是否内置 FFmpeg;当前开源 localhost 的核心 UI 不依赖它,但 MCP/Node 抽帧工具需要用户提供本地 `ffmpeg`。
- 桌面封装阶段再评估是否内置 FFmpeg;当前开源 localhost 的核心 UI 不依赖它,但 MCP/Node 抽帧需要用户提供本地 `ffmpeg`,权威时间线导出还需要 `ffprobe`。

## 8. 路线图

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,19 @@ opened when the host cannot open it.
`bdesign_canvas_apply`, then focus the same card with
`bdesign_canvas_view`.

## Export the authoritative timeline

- When the user authorizes an export, call `bdesign_editor_get`, then call
`bdesign_editor_render` with the returned revision. The result is a
project-owned MP4 Asset containing visible clips, image overlays, caption
burn-in, and mixed audio.
- If rendering reports that `ffmpeg` or `ffprobe` is unavailable, stop and
explain that the MCP process needs those binaries on `PATH`, or absolute
`BEATDESIGN_FFMPEG` and `BEATDESIGN_FFPROBE` paths.
- Render-affecting edits invalidate the previous current render without
deleting its historical Asset. If the timeline changes while rendering,
read the latest revision and ask before starting another render.

## Verify the visible result

After a write, read the returned revision and changed IDs. Canvas and Editor
Expand Down
2 changes: 1 addition & 1 deletion integrations/codex/beatdesign/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Canvas or Editor URL to Codex's in-app Browser for visible review.

```text
Browser: pnpm dev → http://127.0.0.1:3020 (Canvas/Editor review surface)
Agent: MCP stdio → pnpm mcp (Agent calls 26 tools)
Agent: MCP stdio → pnpm mcp (Agent calls 27 tools)
Both processes share the same local SQLite + project files.
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,12 @@ For Canvas-specific review, call `bdesign_canvas_view` with `cardId` so the work
- After submission, follow generation status until it succeeds, fails, or needs user action. A successful generation creates an Asset but does not place itself. Read the current continuation card, preserve its generation settings, update the returned `generationCardId` with that output through `bdesign_canvas_apply`, then call `bdesign_canvas_view` with the same card ID.
- If continuation returns `ok=false`, preserve its structured conflict or rollback result. Follow a returned retry instruction at most once after reading current state; do not blindly re-extract frames.

## Export the authoritative timeline

- When the user authorizes an export, call `bdesign_editor_get`, then call `bdesign_editor_render` with the returned revision. The result is a project-owned MP4 Asset that includes visible clips, image overlays, caption burn-in, and mixed audio.
- If rendering reports that `ffmpeg` or `ffprobe` is unavailable, stop and explain that the MCP host needs those binaries on `PATH`, or absolute `BEATDESIGN_FFMPEG` and `BEATDESIGN_FFPROBE` paths.
- A render-affecting edit invalidates the previous current render without deleting its historical Asset. If the timeline changes during rendering, read the latest revision and ask before starting another potentially expensive render.

## Verify what the user can see

After a write, read the returned revision and changed IDs, then inspect the already-open Canvas or Editor. Canvas and Editor currently refresh external revisions within about two seconds and on focus. Confirm the intended card, clip, caption, duration, or media is visible; a successful database revision alone is not completion.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ BeatDesign MCP 是结构化控制层,浏览器中的 BeatDesign 是用户审
- 抽帧和放置节点是本地操作;返回的生成请求是单独的远程付费动作,只有用户明确授权后才能提交。
- 生成成功只会创建 Asset。用 `bdesign_canvas_apply` 把输出更新到返回的生成卡片,再调用 `bdesign_canvas_view` 聚焦它。

## 导出权威时间线

- 用户明确授权导出后,先调用 `bdesign_editor_get`,再使用返回的 revision 调用 `bdesign_editor_render`。结果是项目内 MP4 Asset,包含可见 Clip、图片 Overlay、烧录字幕和混合音频。
- 若提示缺少 `ffmpeg` 或 `ffprobe`,请用户把两者加入 MCP 进程的 `PATH`,或设置绝对路径 `BEATDESIGN_FFMPEG` 和 `BEATDESIGN_FFPROBE`;环境未改变前不要重复调用。
- 会影响画面的时间线修改会让上一版当前导出失效,但不会删除历史 Asset。若导出期间时间线发生变化,先读取最新 revision,再询问用户是否重新导出。

## 完成标准

写入后核对返回的 revision 和变更 ID,再调用对应 view tool。Canvas 和 Editor 当前通常会在约两秒内或页面重新聚焦时读到 Agent 修改。最终提供准确的 `workspaceUrl` 供用户审核;仅有数据库 revision 不代表用户已经看到结果。
Expand Down
18 changes: 13 additions & 5 deletions src/core/commands/canvas-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ export function buildTimelineCanvasCard({
name,
durationSec,
clipCount,
lastRenderAssetId = null,
lastRenderUrl = null,
lastRenderAssetId,
lastRenderUrl,
referenceCardIds = [],
}: {
existing?: CanvasCard;
Expand All @@ -65,6 +65,14 @@ export function buildTimelineCanvasCard({
referenceCardIds?: string[];
}): CanvasAssetCard {
const cardId = timelineCanvasCardId(timelineId);
const hasRenderUpdate =
lastRenderAssetId !== undefined || lastRenderUrl !== undefined;
const resolvedRenderAssetId =
hasRenderUpdate
? (lastRenderAssetId ?? null)
: (existing?.lastRenderAssetId ?? existing?.assetId ?? null);
const resolvedRenderUrl =
hasRenderUpdate ? (lastRenderUrl ?? null) : (existing?.url ?? null);
const mergedRefs = Array.from(
new Set([
...(existing?.referenceCardIds ?? []),
Expand All @@ -73,11 +81,11 @@ export function buildTimelineCanvasCard({
);
return {
id: cardId,
assetId: lastRenderAssetId ?? existing?.assetId ?? null,
assetId: resolvedRenderAssetId,
kind: 'asset',
type: 'timeline',
name,
url: lastRenderUrl ?? existing?.url ?? null,
url: resolvedRenderUrl,
prompt: existing?.prompt ?? '',
referenceCardIds: mergedRefs,
workflowTemplateId: existing?.workflowTemplateId ?? null,
Expand All @@ -94,7 +102,7 @@ export function buildTimelineCanvasCard({
durationSec,
timelineId,
clipCount,
lastRenderAssetId: lastRenderAssetId ?? existing?.lastRenderAssetId ?? null,
lastRenderAssetId: resolvedRenderAssetId,
};
}
export type CanvasCommandApplication = {
Expand Down
Loading
Loading