Skip to content

Add macOS (Apple Silicon) support for the interactive demo - #2

Open
zxytim wants to merge 2 commits into
nv-tlabs:mainfrom
zxytim:macos-support
Open

Add macOS (Apple Silicon) support for the interactive demo#2
zxytim wants to merge 2 commits into
nv-tlabs:mainfrom
zxytim:macos-support

Conversation

@zxytim

@zxytim zxytim commented Jul 11, 2026

Copy link
Copy Markdown

Summary

Enables the real-time interactive demo to run on macOS (Apple Silicon) using the MPS backend — no CUDA/TensorRT required. Closes #1.

Changes

Device support

  • Motion model runs on MPS (auto-selected; ARDY_DEVICE override); text encoder runs on CPU. Demo launched with --no-compile (TensorRT unavailable on macOS).
  • Cast float64 skeleton buffers to float32 (float64 is unsupported on MPS).

MPS thread-safety (fixes hard crashes)

  • MPS is not thread-safe: the playback thread (per-frame skinning), the replan thread (generation), and the server thread (model loads, prompt updates) previously issued MPS work concurrently, crashing the Metal driver (command-buffer assertion, or SIGSEGV after extended sessions).
  • Model (re)loads and generation steps now run under a shared GPU lock.
  • All session/visualization state (motion tensors, joints, contacts, reference motion, skin bind data, mesh rest poses) is kept on CPU, so the per-frame playback path never touches the accelerator.
  • Skinning (lbs) follows the pose tensor's device.

Text encoder without gated Llama access

  • Load LLM2Vec from local adapter repos (TEXT_ENCODERS_DIR) with the base model repointed at the ungated, byte-identical mirror NousResearch/Meta-Llama-3-8B-Instruct (the default meta-llama base is gated).
  • Resolve local PEFT-adapter repos explicitly in the vendored llm2vec.py: transformers>=5 no longer follows adapter_config.base_model_name_or_path when a config.json sits beside it.

Usability

  • run_demo_mac.sh: one-command launch; bootstraps text_encoders/ on first run.
  • README: macOS setup + launch section.

Testing (macOS 15, M-series, 128 GB)

  • Core/Horizon-40 model: ~0.3 s per 2 s generation window on MPS (several times real-time); playback holds native 20 fps with auto-replan.
  • Verified stable across: continuous playback, live prompt edits, model reload mid-playback, Core↔G1 skeleton switch (character recreation), target-velocity steering, waypoint placement, and multi-client connect/reconnect.
  • Previously reproducible Metal crashes on model reload / long sessions no longer occur.

Notes

  • [trt] extra is CUDA-only; macOS installs use pip install -e ".[demo]".
  • No behavior change on CUDA: the GPU lock is harmless (re-entrant) and CPU-state conversions are no-ops where tensors were already on CPU.

Xinyu Zhou added 2 commits July 12, 2026 02:06
- Run the motion model on MPS (auto-selected, ARDY_DEVICE override);
  text encoder stays on CPU. Launch with --no-compile (no TensorRT).
- MPS is not thread-safe: serialize accelerator access with a shared
  GPU lock around model (re)loads and generation steps, and keep all
  session/viz state (motion, joints, contacts, reference motion, skin
  bind data, mesh rest poses) on CPU so the per-frame playback thread
  never touches the accelerator. Previously, concurrent MPS use by the
  playback/replan/server threads crashed the Metal driver (assertion
  or SIGSEGV after extended sessions).
- Text encoder: load LLM2Vec from local adapter repos (TEXT_ENCODERS_DIR)
  with the base repointed at the ungated NousResearch mirror of
  Meta-Llama-3-8B-Instruct (the default meta-llama base is gated), and
  resolve local PEFT-adapter repos explicitly since transformers>=5 no
  longer follows adapter_config base_model_name_or_path when a
  config.json sits beside it.
- Cast float64 skeleton buffers to float32 (float64 unsupported on MPS).
- Skinning (lbs) follows the pose tensor's device.
- run_demo_mac.sh: one-command launch; bootstraps text_encoders/ on
  first run (adapters gitignored).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support MacOS

1 participant