Skip to content

feat(desktop): Composer 直连生图/生视频#75

Merged
N123999 merged 9 commits into
mainfrom
feat/composer-direct-media-generation
Jun 6, 2026
Merged

feat(desktop): Composer 直连生图/生视频#75
N123999 merged 9 commits into
mainfrom
feat/composer-direct-media-generation

Conversation

@N123999

@N123999 N123999 commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • 当当前推理模型匹配默认生图/生视频槽位,或为无 chat 能力的纯 media 模型时,Composer 发送走直连 generate_image / generate_video,绕过 LLM chat completion。
  • 直连轮次写入 LLM 归档(user + assistant tool_calls + tool),并在完成后同步
    untime.history,切回聊天模型后续聊可引用生成结果。
  • Vercel AI Gateway 纯生图模型改用 v3 image-model 协议,避免误走 language-model 端点。
  • Composer 在直连模式下切换生图/生视频占位符;发送后乐观清空输入框,失败时恢复草稿。
  • 移除模型选择器旁的 Image/Video 徽章(占位符仍提示模式)。

Test plan

  • \�pps/desktop/test/host/composer-direct-media.test.mjs\
  • \�pps/desktop/test/host/direct-media-archive.test.mjs\
  • \�pps/desktop/test/host/direct-media-turn.test.mjs\
  • 手动:GPT Image 2 直连生图 → 切换 deepseek → 发送后续消息,/log-session\ 导出应含生图轮三段且模型能感知前文
  • 手动:Gateway 纯生图模型直连不再报 language-model 404
  • 火山 Seedance 直连生视频 E2E(未在本 PR 环境复测)

N123999 added 8 commits June 6, 2026 14:17
抽取 resolveComposerDirectMediaTool(slot_match,双槽同名优先视频)与
buildImageGenerationSubConfig / buildMediaOnlyTransportConfig,并补充单测。
新增 direct-media-turn:媒体专用 transport、generate_image/video 调用、
orchestrator 事件与 archiveHistory 三段写入;附执行器与归档单测。
在 submitUserTurnAfterInitializedCommand 中分流直连路径;有附件时回退聊天;
SessionTurnOrchestratorContext 注入 requireConfig 与 API Key 解析。
直连模式下切换占位符与模型按钮旁「生图/生视频」标签,并补充中英文 i18n。
纯 imageGeneration 模型作为当前推理时自动走直连生图;Vercel AI Gateway
生图改用 provider.image() v3 协议,避免误走 language-model 聊天 API。
直连生图等长耗时回合在 API 返回前即乐观清空草稿;失败时恢复原文,
并在提交时同步 resetAfterSend 以清空富文本 DOM。
直连生图/生视频仅写入 archiveHistory 时,后续 persist(fromRuntime) 或切回聊天模型后的 refreshArchiveFromRuntime 会用空 runtime 覆盖归档;在 executeDirectMediaTurn 完成后将归档灌入 runtime。
占位符文案仍保留,仅去掉模型名右侧 Image/Video 提示卡片及相关 i18n。
@N123999 N123999 marked this pull request as ready for review June 6, 2026 07:05
@N123999

N123999 commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7dc407f. Configure here.

Comment thread apps/desktop/src/App.tsx Outdated
}
}

composerRichInputRef.current?.resetAfterSend(runtime.settings.agentMode);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

发送失败草稿无法恢复

High Severity

submitComposerMessage 在调用 sendMessage 之前就会执行 resetAfterSend 并清空父级 composer;失败时 useDesktopRuntime 虽用 setComposer 恢复文本,但 ComposerRichInputskipExternalValueSyncRefconversationBusy 下会跳过外部值同步,界面常仍为空,用户会误以为草稿已丢失。

Additional Locations (1)
Fix in Cursor Fix in Web

Triggered by project rule: Bugbot

Reviewed by Cursor Bugbot for commit 7dc407f. Configure here.

config,
config.activeModel,
explicitWorkspaceFiles.length,
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

直连模式模型来源不一致

Medium Severity

Composer 占位符用 runtime.settings.activeModel 判断直连生图/生视频,但 submitUserTurnAfterInitializedconfig.activeModel 决定是否走 executeDirectMediaTurn。模型选择后 setActiveModel 会乐观更新 settings,而 updateConfig 异步完成前两者可能不一致,出现占位符提示直连、实际仍走聊天或相反。

Additional Locations (1)
Fix in Cursor Fix in Web

Triggered by project rule: Bugbot

Reviewed by Cursor Bugbot for commit 7dc407f. Configure here.

发送成功后再清空 RichInput 与 composer 草稿,避免失败时 skipExternalValueSync 导致界面无法恢复。直连占位符改用 snapshot.config.activeModel,与宿主 submit 路径一致。
@N123999 N123999 merged commit 44e4057 into main Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant