Skip to content

feat(macos): add native desktop wallpaper host - #50

Open
tocekuma wants to merge 3 commits into
Code-Amadeus:mainfrom
tocekuma:codex/macos-native-wallpaper-host
Open

feat(macos): add native desktop wallpaper host#50
tocekuma wants to merge 3 commits into
Code-Amadeus:mainfrom
tocekuma:codex/macos-native-wallpaper-host

Conversation

@tocekuma

@tocekuma tocekuma commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What and why

The existing desktop integration relies on Windows wallpaper hosts and BrowserWindow.setShape(). macOS has neither the Lively/Wallpaper Engine host path nor Electron window shaping, so the current Electron slice cannot provide the full wallpaper scene and may be placed below the visible desktop surfaces.

This PR adds a native Electron host policy for macOS:

  • host the existing full wallpaper scene in a desktop-type, click-through window;
  • place it between the Dock wallpaper surface and Finder's desktop-icon surface;
  • keep the existing Windows shaped-slice behavior unchanged;
  • host Canvas in a separate transparent window above desktop icons;
  • capture the mouse only over renderer-reported Canvas controls;
  • avoid calling the unsupported macOS setShape() API;
  • expose the existing Canvas presentation calls and fix the Pixi WebGL monitor reference used by the full scene.

Frame-rate caps, lazy SpriteForge loading, static secondary-display backgrounds, backend restart policy, chat/ASR UI changes, and application-menu changes from the local prototype are intentionally excluded.

Linked Issue for product-semantic or public-contract changes: #47

This branch expects the platform-neutral bridge import fix in #48. Runtime validation was performed with that fix applied locally; it is not duplicated here.

Change class

  • Routine fix, documentation, test, maintenance, or presentation-only UI
  • Product-semantic or public-contract change discussed in the linked Issue
  • Isolated, default-off experiment

Owning layer: Electron wallpaper window hosting and the existing web wallpaper scene

User-visible effect, or none: On macOS, Wallpaper opens the full scene on the desktop and keeps Finder icons clickable. Canvas appears in a separate transparent layer and accepts input only over its visible controls.

Compatibility or migration impact, or none: Windows retains the current shaped interactive slice and helper process. macOS remains a source-run path; signing, notarization, packaging, and multi-display policy are out of scope.

Evidence

Commands and manual journeys run:

  • npm test — 9 passed

  • npm run build — passed

  • node --check render/web/wallpaper_scene.js — passed

  • git diff --check — passed

  • Authenticated wallpaper.start smoke test — started, asset port 17778, bridge port 17797

  • Canvas renderer committed interactive hit regions (1 initial, then 3 after state load)

  • Core Graphics window inspection on Apple Silicon:

    • Dock wallpaper layer: -2147483624
    • Amadeus scene layer: -2147483609, on-screen
    • Finder desktop-icon layer: -2147483603
    • Amadeus Canvas layer: -2147483598, on-screen
  • Runtime log contains no undefined Pixi app error and no missing setCanvasPresentation / setAttention bridge methods after the fix

  • Relevant Python tests pass

  • CPU/model-less baseline remains supported

  • Electron npm run build passes when Electron code changed

  • Dependency audit passes when dependencies changed

  • Before/after screenshots are attached for visible UI changes

  • Documentation/examples are updated for changed settings or contracts

Final check

  • This PR addresses one coherent problem without unrelated cleanup
  • It does not add a speculative API, fallback, or compatibility path
  • No secrets, local state, model weights, voice material, or restricted assets are included
  • Third-party notices and provenance are preserved

@Lucas1479 Lucas1479 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

整体设计与 #47 的 B 层拆分一致:macOS 使用全场景桌面窗口,Canvas 使用独立透明输入层,Windows shaped slice 保持原路径。发送者校验、导航限制和窗口关闭时的关联清理方向也正确。

我复核了 npm test(9 项通过)、npm run buildnode --check render/web/wallpaper_scene.js、15 项相关 Python wallpaper 测试和 git diff --check#48#49#46 分别与本提交做合并分析均无冲突;#50 与刚提交的 #51 会在 electron/src/main/index.ts 产生一个很小的文本冲突,后续按合并顺序 rebase 即可。

批准前建议补齐以下边界:

  1. README 目前把 macOS Wallpaper 写成“源码运行兼容路径”,与仓库当前“新增平台在干净安装、CI 和实机旅程完成前保持候选”的发布边界不一致。请改成“社区实机验证候选”并注明依赖/CI 由 #46 承接,同时补充 README_EN.md 对应说明;现阶段无需把它升级为正式支持基线。
  2. wallpaperWindowPolicy.ts 把一次 Core Graphics 观测值直接作为 Electron relativeLevel 固化。Electron 的该参数是相对于命名 level 的层数偏移,当前单测只锁定了魔数,没有表达真正的不变量。请至少提取具名的 macOS 层级常量,记录验证的 macOS/Electron 版本及数值来源,并让测试明确断言 scene < Finder icons < Canvas < 普通窗口这一排序;如果当前实现可以查询 CGWindowLevelForKey,优先使用系统提供的 desktop/desktop-icon level。
  3. 纯函数测试覆盖了坐标命中,但还没有覆盖本次新增的输入与生命周期边界。请增加聚焦测试,至少证明:非 scene/canvas sender 不能提交命中区域;空区域恢复鼠标穿透并隐藏 Canvas;scene 关闭或重载会清理/重置 Canvas、命中区域和定时器。这里关系到 Finder 点击是否被意外截获,属于本 PR 的核心合同。

#48 是明确前置;最终复审应基于包含 #48 的树运行。多显示器、签名/公证和性能策略可以继续保留在本 PR 范围之外。


English

The overall design matches layer B from #47: macOS uses a full-scene desktop window, Canvas uses a separate transparent input window, and Windows retains the shaped-slice path. The sender checks, navigation restrictions, and associated cleanup when the scene window closes are also headed in the right direction.

I reran npm test (9 passed), npm run build, node --check render/web/wallpaper_scene.js, 15 relevant Python wallpaper tests, and git diff --check. Merge analysis was clean between this commit and each of #48, #49, and #46. #50 and the newly opened #51 have a small textual conflict in electron/src/main/index.ts; rebasing according to the eventual merge order is sufficient.

Before approval, please complete these boundaries:

  1. The README currently describes macOS Wallpaper as a “source-run compatibility path,” which is ahead of the repository rule that a new platform remains a candidate until clean installation, CI, and the corresponding real-device journey are complete. Please describe it as a community real-device candidate, state that dependency/CI work is tracked by #46, and add the matching text to README_EN.md. It does not need to be promoted to an officially supported baseline yet.
  2. wallpaperWindowPolicy.ts turns one Core Graphics observation directly into Electron relativeLevel constants. Electron defines this parameter as an offset from a named level, while the current unit test only snapshots the raw numbers instead of expressing the real invariant. Please at minimum extract named macOS level constants, record the tested macOS/Electron version and the source of the values, and make the tests state the ordering scene < Finder desktop icons < Canvas < normal application windows. If the current host can query CGWindowLevelForKey, prefer the system desktop and desktop-icon levels.
  3. The pure-function tests cover coordinate hit testing, but the new input and lifecycle boundaries are not covered. Please add focused tests proving that an unrelated sender cannot commit hit regions, empty regions restore pass-through and hide Canvas, and scene close/reload cleans up or resets the Canvas window, hit regions, and timer. These behaviors determine whether Finder clicks can be intercepted accidentally and are core contracts of this PR.

#48 is an explicit prerequisite, so the final review should run against a tree that contains it. Multi-display behavior, signing/notarization, and performance policy may remain out of scope.

@tocekuma

tocekuma commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

已按本轮 review 逐项补齐,更新提交为 dda5607

  1. README / README_EN 已统一改成“社区实机验证候选 / community real-device candidate”,明确依赖和 CI 由 build: qualify uv profiles with patched cu124 and experimental ROCm #46 承接,当前不包含 signing、notarization 或 installer,也不声明正式 macOS 支持。
  2. 把 macOS 窗口层级提取为具名常量;desktopdesktop-iconnormal 数值来自 macOS 26.6.2 (25G83) 的 CGWindowLevelForKey 查询,Dock wallpaper、Scene、Canvas 层级再通过 Electron 44.0.0 + CGWindowListCopyWindowInfo 实机确认。测试现在锁定 Dock wallpaper < desktop < Scene < Finder icons < Canvas < normal app,而不是只 snapshot 魔数。
  3. 新增 WallpaperCanvasLifecycle,集中持有 Canvas window、bridge key、hit regions、mouse pass-through 和 timer;IPC sender 只接受当前 Scene/Canvas。Scene reload、renderer gone、Scene close 都会重置或关闭关联 Canvas。
  4. 新增聚焦测试覆盖:unrelated sender 拒绝、空 regions 恢复穿透并隐藏 Canvas、reload 清理 regions/timer、close 清理并关闭双窗口关联资源。

验证结果:

  • npm test:14 passed
  • npm run build:passed
  • node --check render/web/wallpaper_scene.js:passed
  • 叠加 fix(macos): lazy-load Windows pointer forwarding #48 后的相关 Python Wallpaper 回归:15 passed;非 Windows import 通过
  • git diff --check:passed
  • arm64 Mac 实机:Wallpaper bridge 启动成功,Canvas IPC / hit regions 成功提交;CoreGraphics 观测层级为 Dock -2147483624、Scene -2147483609、Finder icons -2147483603、Canvas -2147483598。当前验证时前台是 Chrome fullscreen Space,因此 visibleOnFullScreen: false 下 Scene/Canvas 被系统报告为 off-screen,符合现有策略。

#48 仍作为明确前置,没有重复并入本 PR。#51 的小文本冲突也暂未抢跑处理,等实际合并顺序确定后再 rebase。


Addressed each review item in dda5607: candidate-level docs are now aligned in both READMEs; macOS window levels are named and sourced from CGWindowLevelForKey plus CoreGraphics observation; Canvas lifecycle and sender ownership are centralized and covered by focused reload/close/pass-through tests. The Electron suite (14), build, renderer syntax check, combined-tree Python wallpaper suite (15), non-Windows import, and diff check all pass. #48 remains an explicit prerequisite, and the small #51 conflict is intentionally deferred until merge order is known.

@tocekuma

tocekuma commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

补充一个复审前自查发现的 reload recovery 边界,已在 7133049 修正:

  • Scene 首次加载不触发 Canvas reload;Scene 后续 reload 或 renderer recovery 会重载 Canvas renderer。
  • Canvas reload 会先清空 hit regions、停止 timer、恢复鼠标穿透并隐藏窗口;renderer 重新提交 regions 后恢复显示、hit test 和交互。
  • bridge URL 切换已经在加载 Canvas 时,通过 rendererLoadPending 避免 Scene did-start-loading 再重复 reload。
  • Scene 关闭时同步清理 document-loaded/readiness 状态。

新增/更新测试覆盖 reset → renderer reload → fresh region commit → Canvas 恢复,以及 bridge reload in-flight 时不重复加载。当前 npm test 为 15 passed,npm run build、renderer syntax check 和 git diff --check 均通过。

This follow-up closes a recovery gap found during local integration review: Scene reload now reloads the Canvas renderer and verifies that a fresh region commit restores visibility, hit testing, and interaction, while an in-flight bridge navigation is not restarted.

@Mieluoxxx

Copy link
Copy Markdown
Contributor

有没有截图,我看一下

@Lucas1479 Lucas1479 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本轮修改已经满足上一轮三个主要要求:双语文档正确降级为社区候选;macOS 层级值有具名常量、系统/Electron 版本来源和排序不变量;Canvas sender、空区域穿透、reload 与 close 生命周期已经抽出并测试。最新提交的 15 项 Electron 测试、15 项相关 Python wallpaper 测试、构建、renderer syntax 和 diff check 在本地均通过。

复审还发现一个具体的失败恢复缺口:attach() / prepareReload() 会把 rendererLoadPending 设为 true,但 Canvas 主 frame 的 did-fail-loadloadURL(...).catch(...) 目前只记录错误。如果导航在提交任何 regions 前失败,pending 会一直保持 true;之后 Scene reload 调用 reloadRenderer() 时会在“已有导航进行中”的分支提前返回,不再执行 window.reload(),Canvas 可能永久保持隐藏。

请在主 frame 加载失败/Promise rejection 时把该导航明确结算为失败,使下一次 Scene reload 可以真正重载 Canvas,并补一项 failed navigation -> later scene reload -> renderer reload -> fresh region commit 的测试。处理这一点后,本轮代码可以继续批准流程。

当前远程 Electron job 的失败来自 npm audit endpoint 返回 HTTP 400 / Invalid package tree;本地 npm test 和 build 通过,远程 Python job 也通过,因此这次 CI 失败不归因于上述功能代码,我会单独重跑失败 job。


English

This revision satisfies the three main items from the previous review: both READMEs correctly describe a community candidate; the macOS levels now have named constants, macOS/Electron provenance, and an ordering invariant; and Canvas sender ownership, empty-region pass-through, reload, and close lifecycle behavior have been extracted and tested. On the revised head, all 15 Electron tests, 15 relevant Python wallpaper tests, the Electron build, renderer syntax check, and diff check pass locally.

One concrete failure-recovery gap remains. attach() and prepareReload() set rendererLoadPending to true, while the Canvas main-frame did-fail-load handler and the loadURL(...).catch(...) path only log the error. If navigation fails before any region commit, the pending flag remains true indefinitely. A later Scene reload calls reloadRenderer(), takes the “navigation already in flight” early return, and never calls window.reload(), so the Canvas can remain hidden permanently.

Please explicitly settle the Canvas navigation as failed on main-frame load failure/Promise rejection so a later Scene reload can perform a real reload, and add a focused failed navigation -> later scene reload -> renderer reload -> fresh region commit test. After that, this revision should be ready for approval.

The current remote Electron job failed because the npm audit endpoint returned HTTP 400 / Invalid package tree. Local npm test and the build pass, and the remote Python job passes, so I am treating that CI result separately from the functional change and will rerun the failed job.

Lucas1479 added a commit that referenced this pull request Sep 4, 2026
`npm audit` currently makes otherwise successful Electron CI jobs fail
when npm's Bulk Advisory endpoint times out or returns 5xx, after which
npm 10 may fall back to the retiring Quick Audit endpoint. This change
keeps the locked install, renderer tests, and production build on
Windows, and moves pull-request vulnerability gating to GitHub's
dependency review API.

The review blocks high or critical vulnerabilities introduced in
runtime, development, or unknown scopes. License checks are disabled
here because the previous gate only checked vulnerabilities. Snapshot
warnings are retried to tolerate dependency-graph ingestion delays.

Validation:
- parsed the workflow YAML locally;
- checked every configured input against
`actions/dependency-review-action@v4`;
- verified this repository's dependency-review API on PR #50;
- confirmed there are currently no open npm Dependabot alerts.

---

当前 `npm audit` 会在 npm Bulk Advisory 接口超时或返回 5xx 时,让安装、测试和构建均已成功的
Electron CI 误报失败;npm 10 还可能回退到即将退役的 Quick Audit 接口。本改动保留 Windows
上的锁定安装、渲染器测试和生产构建,并将 PR 的依赖漏洞门禁迁移到 GitHub dependency review API。

门禁会拦截 PR 新引入的高危或严重漏洞,覆盖运行时、开发及未知依赖范围。这里关闭许可证检查,因为原门禁只检查漏洞;同时对 dependency
graph 快照延迟启用重试。

验证:
- 本地解析工作流 YAML;
- 对照 `actions/dependency-review-action@v4` 验证全部输入项;
- 用 PR #50 验证本仓库 dependency-review API 可用;
- 确认当前没有打开的 npm Dependabot 漏洞告警。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants