feat(macos): support GPT-SoVITS inference on Apple Silicon - #49
Conversation
There was a problem hiding this comment.
整体方向和拆分方式都很好。我在当前 Windows / Torch 2.5.1+cu124 环境补做了回归：新增 11 项测试通过，相关 TTS/配置测试 86 通过、1 跳过；现有 GPT、SoVITS LoRA/底模均可由新 loader 读取，RTX 4070 Laptop 上的真实流式 TTS 也通过。说明这组改动没有破坏当前 NVIDIA 路径。
继续评审前建议补两个小调整：
electron/src/main/index.ts当前在所有 Darwin 后端进程中强制写入PYTORCH_ENABLE_MPS_FALLBACK=1，会覆盖用户显式设置的0。请至少保留显式环境配置；如果方便，最好只在本地 GPT-SoVITS 选择 MPS 时把它作为默认值。local_tts_infer.py的_is_cuda同时承担设备上下文和 NVIDIA BigVGAN kernel 两种判断。ROCm 也通过torch.cudaAPI 暴露设备，因此建议拆成“使用 Torch CUDA/HIP 设备 API”和“允许 NVIDIA CUDA 扩展”两个判断；后者应排除torch.version.hip，避免后续 AMD profile 尝试加载 NVIDIA kernel。
测试中也可以顺手恢复 _resolve_tts_device() 写入的 os.environ["TTS_DEVICE"]，避免同一 pytest 进程中的顺序污染。
依赖 profile 和 macOS CI 会在 #46 的分级环境方案中统一处理，不需要把 Darwin 依赖加入 local-cu124。Wallpaper/Canvas、CoreAudio/ASR 以及性能策略继续保持独立 PR 即可；#49 保持为 MPS runtime 的 A 层最合适。
English
The overall direction and the split are good. I ran additional regression checks in the current Windows / Torch 2.5.1+cu124 environment: the 11 new tests passed, and the related TTS/configuration suite passed with 86 tests and 1 skip. The existing GPT checkpoint, SoVITS LoRA checkpoint, and SoVITS base checkpoint all load through the new loader, and a real streaming TTS run passed on an RTX 4070 Laptop GPU. These results indicate that the change does not regress the current NVIDIA path.
Before the next review, please make two small adjustments:
electron/src/main/index.tscurrently forcesPYTORCH_ENABLE_MPS_FALLBACK=1for every Darwin backend process and overrides an explicit user value of0. Please preserve an explicit environment setting at minimum. If practical, make this the default only when local GPT-SoVITS is using MPS.- In
local_tts_infer.py,_is_cudacurrently owns both device-context behavior and the NVIDIA BigVGAN kernel decision. ROCm devices are also exposed through thetorch.cudaAPI. Please split “uses the Torch CUDA/HIP device API” from “may load an NVIDIA CUDA extension”; the latter should excludetorch.version.hipso a future AMD profile cannot attempt to load the NVIDIA kernel.
The tests can also restore the os.environ["TTS_DEVICE"] value written by _resolve_tts_device() to avoid order-dependent contamination within the same pytest process.
Dependency profiles and macOS CI will be handled in the tiered environment work in #46. Darwin dependencies should not be added to local-cu124. Wallpaper/Canvas, CoreAudio/ASR, and performance policy should remain separate PRs; #49 is best kept as the MPS runtime layer A.
|
Addressed the first-round review in
Verification after the revision:
No dependency-profile, Wallpaper, ASR, or performance-policy changes were added. |
Lucas1479
left a comment
Lucas1479
left a comment
There was a problem hiding this comment.
复审通过。c13082a 已完整处理第一轮意见：MPS CPU fallback 现在只在 Apple Silicon + 本地 GPT-SoVITS + MPS 设备策略下默认开启，并保留所有显式设置；Torch CUDA/HIP 设备 API 与 NVIDIA BigVGAN 扩展许可已分离；设备选择测试会恢复 TTS_DEVICE。
我在最新提交上复核了 13 项聚焦 Python 测试、8 项 Electron 测试、Ruff、Electron build 和 git diff --check。另外使用现有 Torch 2.5.1+cu124、同一组 GPT/SoVITS 权重和 RTX 4070 Laptop 重新完成真实流式 TTS：4 个块、2.252 秒音频、约 2041 MiB 峰值显存，当前 NVIDIA 路径仍正常。
本次批准覆盖 #49 的 MPS runtime A 层代码，不把 Apple Silicon 升级为正式支持平台；可复现的 local-mps 依赖与 CI 仍由 #46 承接。本次不执行合并。
English
Approved after re-review. Commit c13082a fully addresses the first-round feedback: MPS CPU fallback now defaults on only for Apple Silicon + local GPT-SoVITS + an MPS-resolving device while preserving every explicit setting; Torch CUDA/HIP device-API use is separated from permission to load NVIDIA BigVGAN extensions; and the device-selection tests restore TTS_DEVICE.
On the revised head I reran 13 focused Python tests, 8 Electron tests, Ruff, the Electron build, and git diff --check. I also repeated a real streaming TTS run with the existing Torch 2.5.1+cu124 environment, the same GPT/SoVITS checkpoints, and an RTX 4070 Laptop GPU: 4 chunks, 2.252 seconds of audio, and about 2041 MiB peak allocated memory. The current NVIDIA path remains functional.
This approval covers the MPS runtime layer A implemented by #49. It does not promote Apple Silicon to an officially supported platform; reproducible local-mps dependencies and CI remain tracked by #46. This review does not merge the PR.
) This PR builds on #45 and preserves the original contributor commits. It qualifies one exact uv-managed project environment before changing the maintained installation path. The default capability ladder is L1 core → L2 remote voice → L3 CPU VAD → L4 Windows cu124 local models. Every tier uses the project `.venv`, and CPU, NVIDIA, and ROCm Torch builds are mutually exclusive. CPU VAD and cu124 now use PyTorch/Torchaudio 2.6.0 as the maintained baseline. This is the smallest official cu124 upgrade that fixes critical advisory GHSA-53q9-r3pm-6pq6; the previous 2.5.1 lock was rejected by dependency review. The Windows local-model profile also uses the `pyopenjtalk-plus` CPython 3.12 wheel so a clean install does not depend on compiling pyopenjtalk inside a long checkout path. This branch includes the merged portability, Apple Silicon MPS runtime, native macOS menu, and dependency-review changes from #48, #49, #51, and #52. An opt-in `local-rocm` candidate is also included for Windows. It locks AMD's official ROCm 7.2.1 / Torch 2.9.1 packages as a third build selection in the same `.venv`. Qwen ASR and GPT-SoVITS may run in persistent sidecar processes while using that interpreter; sidecar mode is disabled unless explicitly selected. NVIDIA CUDA Graph, NVIDIA BigVGAN kernels, and unrelated performance tuning are not enabled by this candidate. The latest integration includes public main `f7e57e2` and fixes the offline-loading boundary: inherited online flags, already-imported Hub/Transformers state, and cached/custom Hub sessions are now reset before local voice-model loading. Explicit local-file loading is applied to Qwen ASR, BERT, and BigVGAN. The new regression checks block remote requests while still loading a tiny locally generated BERT model. They ran successfully with the qualified cu124 and ROCm libraries. Current head `f1f6397` has all six remote checks green; the current Windows model-less suite reports 1769 passed / 11 skipped, and the Electron model-less smoke passed. Recorded validation (previous L4 qualification and latest integration checks): - `uv.lock` resolves CPU 2.6.0, cu124 2.6.0, and ROCm 2.9.1/7.2.1 branches; invalid combinations fail closed. - A clean `local-cu124` sync passed the 234-package environment contract and `uv pip check`. - On an RTX 4070 Ti SUPER, PyTorch 2.6.0+cu124 completed real CUDA matrix compute, safely loaded the existing GPT and SoVITS v3 checkpoints, and generated a finite 1.612-second / 24 kHz v3 TTS sample through BERT, CNHubert, GPT, SoVITS LoRA, and BigVGAN. - The full L4 Python suite passed: 1783 passed, 2 skipped. - Exact sync back to L1+dev removed 127 voice/model packages and verified that Torch, Qwen ASR, ONNX Runtime, and pyopenjtalk were absent. - A clean `local-rocm` sync installed the fixed candidate and passed its version/import contract plus `uv pip check`. - Earlier revisions passed the remote Windows model-less, single-venv ladder, macOS voice, ROCm clean-install, and Electron build jobs. The Torch 2.5.1 critical advisory is removed. Dependency review has narrowly documented four temporary exceptions: NLTK GHSA-8mgp-746c-j5xp has no patched release, while qwen-asr 0.0.6 requires Transformers 4.57.6 exactly and therefore cannot consume the fixes for GHSA-29pf-2h5f-8g72, GHSA-fgcw-684q-jj6r, and GHSA-xrqw-3rrv-vx5w. Amadeus does not call the affected NLTK persistence, Transformers LightGlue, or `save_pretrained` paths; ASR resolves a local directory and both ASR and TTS force Transformers/Hugging Face offline. Each exception must be removed when a compatible fix is published. All six remote checks have now passed again for `f1f6397`. - Ruff, workflow YAML validation, lock consistency, focused profile/sidecar tests, and diff checks passed. ROCm evidence remains explicitly experimental. Community history records successful RX 9070 XT ASR/TTS sidecars on another ROCm/PyTorch build. On the maintainer's Radeon 780M, the fixed 7.2.1 build installed and enumerated gfx1103, but its first FP32 tensor operation crashed in `amdhip64_7.dll`; Radeon 780M is absent from AMD's Windows support matrix and is not treated as a supported result. A supported AMD GPU still needs to complete the fixed-combination ASR/TTS, microphone/playback, interruption, lifecycle, and long-running journeys. The PR is ready for maintainer review. ROCm remains experimental until supported AMD hardware completes the remaining real-device acceptance. No model weights, recordings, transcripts, generated audio, credentials, virtual environments, or validation caches are committed. Refs #44 and #45. ### 中文摘要 本 PR 基于 #45，并保留原贡献者提交，用于在切换维护基线前验证由 uv 管理的单一项目环境。 默认能力阶梯是 L1 core → L2 远程语音 → L3 CPU VAD → L4 Windows cu124 本地模型。所有梯级共用项目 `.venv`，CPU、NVIDIA 和 ROCm Torch 构建两两互斥。CPU VAD 与 cu124 现以 PyTorch/Torchaudio 2.6.0 作为正式维护基线；这是仍提供官方 cu124 wheel、同时修复 critical 漏洞 GHSA-53q9-r3pm-6pq6 的最小升级。旧 2.5.1 锁正是 dependency review 失败的原因。Windows 本地模型档也统一采用带 CPython 3.12 wheel 的 `pyopenjtalk-plus`，避免在较长仓库路径中现场编译 pyopenjtalk。 本分支现已合入 #48、#49、#51、#52 的跨平台导入、Apple Silicon MPS runtime、macOS 原生菜单和 dependency review 改动。 Windows `local-rocm` 仍是默认关闭的实验候选。它在同一个 `.venv` 中锁定 AMD 官方 ROCm 7.2.1 / Torch 2.9.1，并与 CPU/cu124 构建互斥。Qwen ASR 与 GPT-SoVITS 可使用同一解释器运行在常驻 sidecar 子进程；sidecar 只表示进程隔离，不额外要求虚拟环境，也不会默认启用 NVIDIA CUDA Graph、NVIDIA BigVGAN kernel 或社区补丁中的其他性能调优。 本轮已接上公开主线 `f7e57e2`，并补齐模型离线加载边界：继承的在线环境变量、已导入的 Hub/Transformers 状态和缓存／自定义 Hub HTTP 会话都会在本地语音模型加载前恢复为离线；Qwen ASR、BERT、BigVGAN 的加载也明确只使用本地文件。新增测试验证远程请求被阻止，同时本地生成的小型 BERT 模型仍能正常加载，已在 cu124 与 ROCm 的实际依赖环境中通过。最新提交 `f1f6397` 的六项远程检查全绿，其中 Windows 无模型完整回归为 1769 passed / 11 skipped，Electron 无模型冒烟测试通过。 验证记录（此前 L4 资格验证与本轮集成检查）： - `uv.lock` 可解析 CPU 2.6.0、cu124 2.6.0 和 ROCm 2.9.1/7.2.1，冲突组合会明确失败； - 全新 `local-cu124` 同步通过 234 包环境合同和 `uv pip check`； - RTX 4070 Ti SUPER 上，2.6.0+cu124 完成真实 CUDA 矩阵计算，安全加载现有 GPT/SoVITS v3 权重，并经 BERT、CNHubert、GPT、SoVITS LoRA、BigVGAN 生成 1.612 秒、24 kHz 的有限值音频； - 完整 L4 Python 回归 1783 通过、2 跳过； - 同一 `.venv` 精确返回 L1+dev 时移除 127 个语音/模型包，并确认 Torch、Qwen ASR、ONNX Runtime 与 pyopenjtalk 均不存在； - 全新 `local-rocm` 同步通过固定候选版本/导入合同和 `uv pip check`； - 此前提交的远程 Windows 无模型、单 `.venv` 阶梯、macOS voice、ROCm clean-install 与 Electron build 均通过；Torch 2.5.1 critical 漏洞已移除。dependency review 现精确记录四条临时豁免：NLTK GHSA-8mgp-746c-j5xp 尚无修复版；qwen-asr 0.0.6 又严格要求 Transformers 4.57.6，暂时无法采用 GHSA-29pf-2h5f-8g72、GHSA-fgcw-684q-jj6r、GHSA-xrqw-3rrv-vx5w 的 5.x 修复。Amadeus 不调用相关 NLTK 持久化、LightGlue 或 `save_pretrained` 路径；ASR 只解析本地目录，ASR/TTS 均强制 Transformers/Hugging Face 离线。兼容修复发布后必须逐条移除。最新提交 `f1f6397` 的六项远程检查已全部通过； - Ruff、工作流 YAML、锁一致性、profile/sidecar 聚焦测试和 diff check 均通过。 ROCm 继续明确标记为实验候选。社区资料记录了 RX 9070 XT sidecar ASR/TTS 历史成功；维护机 Radeon 780M 上，固定 7.2.1 环境可安装并枚举 gfx1103，但首次 FP32 计算在 `amdhip64_7.dll` 中崩溃。780M 不在 AMD Windows 支持矩阵内，这个负结果只限定本机硬件，不否定社区候选。固定组合仍需由受支持 AMD GPU 完成真实 ASR/TTS、麦克风/播放、打断、生命周期和长时间运行验收。 本 PR 已进入维护者审阅；ROCm 在受支持 AMD 硬件完成剩余实机验收前继续保持实验候选。仓库未提交模型、录音、转写、生成音频、凭证、虚拟环境或验证缓存。关联 #44、#45。 --------- Co-authored-by: Morgan Woods <weiyiding0@gmail.com>
2 participants
What and why
The embedded GPT-SoVITS path assumes CUDA in several runtime-only code paths: device selection defaults to CUDA, synchronization and device contexts call
torch.cudaunconditionally, and BigVGAN probes/loads CUDA kernels even when inference runs on another backend. On Apple Silicon this prevents the bundled local TTS backend from running through PyTorch MPS.This PR adds the smallest runtime compatibility layer needed for Apple Silicon:
TTS_DEVICE=autotompson Apple Silicon andcpuon Intel macOS;Performance-policy changes from the local macOS prototype—lower sample steps, frame-rate caps, lazy animation loading, static secondary displays, and backend auto-restart—are intentionally excluded.
Linked Issue for product-semantic or public-contract changes: #47
Related dependency-profile work: #46. This PR intentionally does not add Darwin packages to the CUDA-named
local-cu124extra; the install-profile contract can be integrated with the ongoing dependency migration.Change class
Owning layer: embedded GPT-SoVITS runtime and backend process environment
User-visible effect, or
none: Apple Silicon users can selectTTS_DEVICE=autoand run compatible GPT-SoVITS v3 models through MPS.Compatibility or migration impact, or
none: Explicit indexed CUDA,mps, andcpuvalues remain supported. Apple Silicon now resolvesauto/unindexedcudatomps; Intel macOS resolves them tocpu. A compatible PyTorch/MPS environment is still required; dependency installation is intentionally left to #46.Evidence
Commands and manual journeys run:
npm run build— passedmps_built=True,mps_available=Truegit diff --check— passedThe full macOS collection on current
mainis independently blocked by an eager import of the Windows-only pointer hook; #48 contains the isolated fix. The remaining 23 macOS failures (temporary-path alias expectations and Windows desktop-layer assertions) reproduce unchanged onupstream/main.npm run buildpasses when Electron code changedFinal check