[Roadmap] Apple Device Support (2026 Q2)

[jonahbernard]

As of February 21, 2026 (date of initial creation of this roadmap), SGLang has no support for Apple Silicon.

We are excited to announce the initial roadmap for adding Apple Silicon support. We ask for contributors, of all levels of familiarity with SGLang and Apple devices, to pick up tasks on this roadmap and help us run SGLang on Apple Silicon (M-series).

Please comment below to pick up tasks that interest you, and we will assign them to you. Please join the mlx-backend Slack channel as well (via the SGLang Slack workspace).

Installation Instructions

Here are some quick instructions to set up SGLang on your Mac. Currently, SGLang must be built from source to work on your Mac.

Beware: the Mac version of SGLang is currently verified to work only with Python 3.11 and is known to have issues with other Python versions.

Beware 2: make sure the mlx and mlx-lm packages are installed with the LATEST version after you run uv pip install -e "python[all_mps]". They were only added as dependencies in this PR: #22162

# Install ffmpeg
brew install ffmpeg

# Install uv
brew install uv

# Clone the repository
git clone https://github.com/sgl-project/sglang.git
cd sglang

# Create and activate a virtual environment
uv venv -p 3.11 my-venv
source my-venv/bin/activate

# Install the Python packages
uv pip install --upgrade pip
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"

Basic Server Launch Instructions

Once you have built SGLang from source using the above instructions, you can launch a basic server using this command:

SGLANG_USE_MLX=1 python -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B \
  --port 43440

Now, send a request to that server via this curl command from another shell:

curl http://localhost:43440/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Tell me a short joke."}
    ],
    "max_tokens": 10
  }'

Profiling Instructions

When contributing PRs, it is important to profile your changes to make sure they work as intended. There are 4 different ways to profile SGLang at 4 different levels of the stack.

| Tool                       | HTTP Server                                   | Scheduler                               | Use Case                                                                   |
| -------------------------- | --------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------- |
| `bench_serving`            | Yes (async HTTP client to a running server)   | Yes (indirectly, via server)            | Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL) |
| `bench_one_batch_server`   | Yes (sends HTTP requests to a running server) | Yes (indirectly, via server)            | End-to-end single-batch latency including HTTP and scheduler overhead      |
| `bench_offline_throughput` | No                                            | Yes (directly uses `Engine` in-process) | Maximum throughput measurement without HTTP overhead                       |
| `bench_one_batch`          | No                                            | No (directly calls `ModelRunner`)       | Kernel-level latency profiling of a single static batch              

(taken from benchmark_and_profiling.md, read that for more comprehensive information)

Note: MLX and MLX-LM must be updated to the LATEST version to correctly profile the SGLang Apple Silicon backend. If MLX and MLX-LM are not updated, you may end up with very noisy traces.

There are two types of traces: Metal-only and system-wide.

Metal-only profiling

bench_one_batch.py directly runs the prefill and decode phases:

# this is a summary of the bench_one_batch.py file
start_profile
model_runner.extend(...)
stop_profile

start_profile
model_runner.decode(...)
stop_profile

Because bench_one_batch.py directly calls the extend and decode functions, it cannot be used to profile the server or scheduler.

To use bench_one_batch.py to profile on Apple Silicon and get a Metal-only trace, use this command:

SGLANG_TORCH_PROFILER_DIR=./profiler_output_baseline MTL_CAPTURE_ENABLED=1 SGLANG_USE_MLX=1 python -m sglang.bench_one_batch \
  --model-path Qwen/Qwen3-0.6B \
  --trust-remote-code \
  --disable-radix-cache \
  --disable-cuda-graph \
  --tp-size 1 \
  --batch-size 1 \
  --input-len 60 \
  --output-len 10 \
  --port 43440 \
  --profile \
  --profile-start-step 0 \
  --profile-steps 10

This saves a .gputrace file for prefill and a .gputrace file for decode. Double-click either .gputrace file, and it will automatically open in the XCode app.

To use bench_offline_throughput.py to profile the GPU operations on Apple Silicon and get a Metal-only trace, use this command:

SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.bench_offline_throughput \
    --model-path Qwen/Qwen2.5-0.5B-Instruct \
    --num-prompts 1 \
    --profile

To use bench_one_batch_server.py to profile the GPU operations on Apple Silicon and get a Metal-only trace, use this command:

SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.test.bench_one_batch_server_internal \
    --model Qwen/Qwen2.5-0.5B-Instruct \
    --batch-size 2 \
    --input-len 128 \
    --output-len 16 \
    --profile

To use bench_server.py to profile the GPU operations on Apple Silicon and get a Metal-only trace, use these TWO commands in different terminals:

SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-0.5B-Instruct \
    --port 30000

python3 -m sglang.bench_serving \
    --backend sglang \
    --port 30000 \
    --num-prompts 1 \
    --profile

System-wide profiling

To profile the entire system (CPUs, GPU, drivers, etc.), any Python process can be recorded by simply wrapping the process start command in xctrace record --template "Metal System Trace" --launch -- /usr/bin/env. Thus, for example, to profile the entire system during the execution of bench_one_batch.py, this command can be used:

xctrace record --template "Metal System Trace" --launch -- /usr/bin/env SGLANG_USE_MLX=1 $(which python) -m sglang.bench_one_batch \
  --model-path Qwen/Qwen3-0.6B \
  --trust-remote-code \
  --disable-radix-cache \
  --disable-cuda-graph \
  --tp-size 1 \
  --batch-size 1 \
  --input-len 60 \
  --output-len 10 \
  --port 43440 \

This saves a single .trace file that includes prefill and decode. Double-click the .trace file, and it will open in the Instruments app.

Initial Support (Proof-Of-Concept)

• Add initial mps support so that models can run on MacOS. @yeahdongcn [diffusion][llm] macOS support #19549
• Add check-env script for Apple devices. @yeahdongcn [MPS] Support sglang.check_env #20753

Implement Full MLX Backend

• Switch over completely from PyTorch to MLX via skeleton MLX backend @yeahdongcn [MLX] Add native MLX execution backend for Apple Silicon Mac #20342

MLX runner-stub / framework contract redesign

The MLX backend integrates with the framework through MlxModelRunnerStub, which
re-implements the framework's implicit assumptions about ModelRunner. As SGLang
features and server arguments evolve, the stub silently drifts from framework
behavior — recent instances: #30181, #30389, #30547. Rather than continuing to
extend the stub per feature, re-evaluate the integration design for a more
fundamental solution.

• Audit the framework↔stub contract: catalog assumptions the stub satisfies,
  misses, or silently diverges on — @noob-se7en
• Design note with options + trade-offs for a long-term backend contract — @noob-se7en
• Implementation + migration (following design review) — @noob-se7en

KV Cache

• Support radix cache across all batch sizes @yeahdongcn [MLX] Support radix cache #21509
• Adds env to control allowed number of tokens before clearing the MLX cache @changminbark feat: add decode clear steps env var #28160

UMA Memory Management

• [Feature][MPS] Better memory management for Apple Silicon Macs #21443 -> [MPS] Cap memory to Metal recommended max working set size #21539 @karanb192

Scheduler

• [Apple Silicon] Enable overlap scheduling  #22114 -> [Apple Silicon] [MLX] MLX decode partial overlap scheduling for generation (async eval) #22416 @changminbark

Metal Kernels

To enable blazing fast inference, we need to write custom Metal kernels that are designed for inference instead of relying on MLX's general-purpose "fast" kernels.

• Enable sgl-kernel to build custom Metal kernels @yeahdongcn [Apple Silicon] Add Metal kernel support in sgl-kernel #23449
• Add custom Metal PagedAttention implementation
  • Add custom Metal PagedAttention prefill kernel
  • Add custom Metal PagedAttention decode kernel @adityavaid [Apple Silicon] Add (env-gated) MLX paged attention decode #28335
• Add custom Metal FusedMoE implementation @jlee5814
  • Add custom Metal FusedMoe kernel + MetalRunnerCore
• Add custom RMSNorm kernel @SasankYadati
• Add custom RoPE kernel with fused KV cache store operation @adityavaid [Apple Silicon] Add custom Metal RoPE kernel with fused KV cache store #22868

Mixture-of-Experts

• [Apple Silicon] Optimize MoE performance #22283 @jlee5814
  • [Apple Silicon] [MLX] Fuse SwitchGLU up_proj + gate_proj  #24712

Model Support

We must rewrite all of the model architecture files in SGLang's main sglang/python/sglang/srt/models folder using the MLX library instead of PyTorch.

• qwen3.py
  • qwen3.5/3.6
    • @JINO-ROHIT patch mlx hadware backend to run qwen3.5 #25253
    • @yeahdongcn [MLX] Support Qwen3.5 (dense) Model #25754
• qwen3_moe.py
• qwen2.py
• qwen2_moe.py
• gpt_oss.py
• hunyuan.py

Please comment below if there is another model that you want to add support for and I will add it here with your name and PR.

Quantization

• On-the-fly quant (mlx_q4/mlx_q8) @damahua [MLX] Add on-the-fly --quantization mlx_q4 / mlx_q8 for Apple Silicon #24907
• Auto-detect quant method @jlee5814 [Apple Silicon] [MLX] Auto-detect MLX-format quantization_config dict #25191

Speculative Decoding

• Add EAGLE support @Yatogaii

Multi-LoRA Serving

We need to enable LoRA inference that is aligned with the method proposed in Punica: Multi-Tenant LoRA Serving. Our goal is to utilize as much of the current SGLang LoRA as possible.

• TBD

Profiling

• Add MLX profiling hooks to profile using Apple-native profiling tools with bench_one_batch.py Add MLX profiling to bench_one_batch.py #22159 @Jonahcb
• Add MLX profiling hooks to profile using Apple-native profiling tools with bench_serving.py, bench_offline_throughput.py, and bench_one_batch_server.py @LijuanTang94 [MLX] Add Metal profiling hooks to server profiler #28122 [MLX] Fix step-bounded profiling for bench tools on Apple Silicon #29217

CI

• Set up self-hosted MacOS GitHub runners @yeahdongcn @Kangyan-Zhou
• Add unit test workflow @jlee5814 [Apple Silicon] [CI] Add model-free unit-test workflow on macos-26 #29691
• Add accuracy-based correctness tests for popular models [Apple][MLX][Test] Add Qwen3 correctness and accuracy tests for Apple Silicon #21770 @linzhonghong

Documentation

• minor: docs include mac installation #25178 @JINO-ROHIT

Bug Fixes & Bug Tracker

• Python 3.12+ Support: [MPS] Fix Triton stub sub-module imports on Python 3.12+ #21551 @karanb192
• Dependency updates: [Apple Silicon] [MLX] Add mlx and mlx-lm dependencies #22162 @changminbark
• Server crash: [Bug] [Apple Silicon] Server Crash #22466
• Quant config: [Bug] [Apple Silicon] [MLX] _verify_quantization rejects mlx-community models with quantization_config dict #25119
• Qwen3.5/3.6: ValueError: No attention attribute in layer type <class 'mlx_lm.models.qwen3_5.DecoderLayer'> #25017
• canary_manager fix: fix(mlx): set canary_manager and materialize overlap-loop inputs on Apple Silicon #26882
• Orphan process handling: [Fix] Emulate PDEATHSIG on macOS to prevent orphaned worker processes #27190
• Server startup fix: [Apple Silicon] [MLX] Fix MlxModelRunnerStub.initialize() signature desync with base #28660 [MLX] Fix Apple Silicon server startup; align MLX tests with upstream #28770 fix(mlx): default prefill_aware_swa=False on MlxModelRunnerStub #29681
• write_token fix: fix: make write_token dynamic #29271
• Unit test fix: [MLX] Fix FutureMap relay unit test to use RelayPayload #29311

This roadmap will change rapidly as we work on the above tasks and encounter obstacles that require new tasks.

[yeahdongcn]

This document (AI-assisted) describes the technical roadmap for macOS support in SGLang, covering both LLM inference and Diffusion model inference. The overarching goal is to transition from PyTorch MPS (Metal Performance Shaders) fallbacks to native MLX implementations, eliminating the overhead of PyTorch↔MLX synchronization and data conversion.

---

<===Click to expand details===>

Architecture Overview

+-----------------------------------------------------------+
|                     SGLang on macOS                       |
+-----------------------------+-----------------------------+
| LLM Runtime                 | Diffusion Runtime           |
| (sglang/srt/)               | (sglang/multimodal_gen/)    |
|                             |                             |
|  MlxModelRunner             |  MPS Platform               |
|  MlxTpModelWorker           |  SDPA Attention Backend     |
|  ContiguousKVCache          |  MPS Fallback Kernels       |
|  MLXAttentionWrapper        |  mlx_ops.py (shared prims)  |
|  tensor_bridge              |                             |
+-----------------------------+-----------------------------+
|              Compatibility Layer                          |
|  _triton_stub  |  _mps_stub  |  SGLANG_USE_MLX env        |
+-----------------------------------------------------------+
|  MLX (mx.fast.*, mx.compile)  |  PyTorch MPS (fallback)   |
+-----------------------------------------------------------+
|        Metal Performance Shaders - Apple Silicon GPU      |
+-----------------------------------------------------------+

---

Current State

Component	Status	Backend	Notes
LLM inference	✅ Functional	PyTorch MPS via _triton_stub + _mps_stub	Works but suboptimal performance
Diffusion inference	✅ Functional	PyTorch MPS with optional MLX norm acceleration	SGLANG_USE_MLX=1 enables MLX-accelerated norms
Environment diagnostics	🔧 In progress	sglang.check_env / sglang.show_config	PR #20753

How It Works Today

On macOS, SGLang installs two compatibility shims at startup (sglang/__init__.py):

1. _triton_stub — Registers a mock triton package in sys.modules so that modules importing triton at the top level load without error. The actual Triton kernels are never executed; alternative backends (SDPA for attention, PyTorch-native for norms) are used instead.

2. _mps_stub — Monkey-patches torch.mps to add APIs that torch.cuda provides but torch.mps lacks: Stream, Event, set_device, current_device, get_device_properties, memory tracking functions, etc. This lets device-agnostic code paths work without scattered hasattr guards.

Together, these stubs make the entire SGLang codebase importable and runnable on macOS without code changes in the core paths. However, the resulting inference is suboptimal because:

• PyTorch MPS operations require CPU↔GPU synchronization at each kernel launch
• Any interaction between MLX and PyTorch requires explicit data conversion
• Triton-optimized kernels (fused norms, attention) are replaced with multi-step PyTorch decompositions

---

1. LLM Inference

1.1 Foundation: Stubs and Compatibility Layer

Status: ✅ Complete

Component	File	Purpose
Triton stub	sglang/_triton_stub.py	Mock triton package so Triton-dependent modules load on macOS
MPS stub	sglang/_mps_stub.py	Patch torch.mps with missing APIs (Stream, Event, set_device, memory tracking)
Auto-install	sglang/__init__.py	Installs both stubs on darwin + MPS before any downstream imports
SGLANG_USE_MLX env	sglang/srt/environ.py	Opt-in flag for MLX acceleration (EnvBool(False))

These stubs will remain as the fallback path even after MLX-native implementations are complete, ensuring the codebase stays importable on macOS without MLX installed.

1.2 MLX-Native Model Runner

Status: 🔧 In progress (PR #20342)

The standard ModelRunner (sglang/srt/model_executor/model_runner.py) is heavily CUDA-centric — it initializes NCCL, CUDA graphs, FlashInfer/Triton attention backends, and GPU memory pools, none of which exist on macOS.

The MLX-native MlxModelRunner (sglang/srt/hardware_backend/mlx/model_runner.py) replaces the standard runner with:

Aspect	Standard ModelRunner	MlxModelRunner
Model loading	torch weight loading → GPU	mlx_lm.load() or mlx_vlm.load() → MLX arrays
Forward pass	PyTorch on CUDA/MPS	MLX native ops on Metal GPU
KV cache	ReqToTokenPool + paged CUDA memory	ContiguousKVCache (MLX arrays)
Attention	FlashInfer / Triton / SDPA	mx.fast.scaled_dot_product_attention
Sampling	PyTorch-based sampler	Native MLX greedy + PyTorch fallback for advanced sampling
CUDA graphs	Yes (for decode)	N/A — MLX uses lazy evaluation + mx.eval()
Distributed	NCCL	N/A (single-device only for now)

Key design decisions:

• Uses mlx-lm for model loading, leveraging its extensive HuggingFace model support
• Single mx.eval() call after the full forward pass (lazy evaluation = automatic kernel fusion)
• Greedy sampling stays in MLX to avoid round-trip conversion; advanced sampling (top-k, top-p) falls back to PyTorch

Worker integration: MlxTpModelWorker (sglang/srt/hardware_backend/mlx/tp_worker.py) replaces the standard TpModelWorker, wiring the MLX runner into the scheduler loop. MlxModelRunnerStub provides the interface expected by the scheduler without CUDA-specific initialization.

1.3 MLX-Native KV Cache

Status: 🔧 In progress (PR #20561)

The KV cache implementation evolved through two designs:

Design A — Separated caches:

• BS=1: native mlx-lm KVCache + native attention
• BS>1: ContiguousKVCache + batched SDPA
• Required enter_batched_mode / leave_batched_mode cache conversions (~115ms overhead per transition)

Design B — Unified contiguous cache (chosen):

• All batch sizes use ContiguousKVCache + SDPA
• Zero cache conversion overhead
• Single code path for all batch sizes
• SDPA is faster than native mlx-lm even for BS=1

The ContiguousKVCache stores K/V as contiguous MLX arrays with lazy initialization. The MLXAttentionWrapper patches each model layer's attention module to use mx.fast.scaled_dot_product_attention with the contiguous cache.

Components:

Component	Location	Purpose
ContiguousKVCache	hardware_backend/mlx/kv_cache/contiguous_cache.py	Lazy-init contiguous KV storage for all batch sizes
MLXAttentionWrapper	hardware_backend/mlx/kv_cache/attention_wrapper.py	Replaces model attention with SDPA + contiguous cache reads/writes
model_patching	hardware_backend/mlx/kv_cache/model_patching.py	One-time monkey-patch of model layers at init
request_state	hardware_backend/mlx/kv_cache/request_state.py	Per-request KV cache state management

1.4 Attention Backend

Status: 🔧 In progress (part of PR #20561)

Current: mx.fast.scaled_dot_product_attention (MLX's fused Metal SDPA kernel)

This is used for both prefill and decode. For prefill, it handles variable-length sequences. For decode, it processes single tokens against the full KV cache.

Future considerations:

• Paged attention: The vllm-metal reference (vllm-metal/vllm_metal/v1/model_runner.py) implements paged KV cache with reshape_and_cache for MPS tensors. This could be adopted if memory pressure becomes an issue with larger models.
• Packed prefill: Multiple prefill requests can be concatenated with cu_seqlens to build a block-diagonal causal mask, enabling batched prefill in a single forward pass.

1.5 Tensor Bridge

Status: ✅ Complete

Efficient conversion between PyTorch tensors and MLX arrays using Apple Silicon's unified memory:

Direction	Method	Notes
torch → MLX	mx.array(tensor.detach())	Direct path; fastest for all dtypes including bfloat16
MLX → torch	torch.frombuffer(memoryview(mx_array))	Zero-copy via memoryview on contiguous arrays
Sync	mx.synchronize() / torch.mps.synchronize()	Required before cross-framework access

The tensor bridge handles:

• MPS 4GB MPSTemporaryNDArray allocation limit (conservative 1GB threshold)
• Non-contiguous MLX arrays (auto-mx.contiguous() before conversion)
• bfloat16 (numpy doesn't support it; direct mx.array(tensor) bypasses numpy)

Reference implementations:

• LLM: sglang/srt/utils/tensor_bridge.py (in PR [MLX] Add native MLX execution backend for Apple Silicon Mac #20342)
• Diffusion: sglang/jit_kernel/diffusion/triton/mps_fallback.py (inline _torch_to_mlx / _mlx_to_torch)
• vllm-metal: vllm_metal/pytorch_backend/tensor_bridge.py

1.6 Performance Optimization Roadmap

After the MLX-native model runner and KV cache are merged, the following optimizations apply:

1. mx.compile for the model forward pass — MLX's JIT fusion (mx.compile) can fuse multiple operations into single Metal kernels, reducing kernel launch overhead. This is the next major performance win after the basic MLX migration.

2. Quantized KV cache — Reduce per-token memory by 2–4× by storing KV cache entries in lower precision (e.g., 8-bit or 4-bit). This extends the effective context length on memory-constrained Apple Silicon devices.

3. Prefix sharing — Detect common prefixes across requests and reuse contiguous cache segments. The vllm-metal reference has a PrefixCacheManager with content-hash-based lookup that can be adapted.

---

2. Diffusion Model Inference

2.1 MPS Fallback Kernels

Status: ✅ Complete

sglang/jit_kernel/diffusion/triton/mps_fallback.py provides PyTorch-native replacements for all Triton diffusion kernels:

Kernel	Fallback	MLX-accelerated (SGLANG_USE_MLX=1)
rms_norm_fn	Pure PyTorch decomposition	mx.fast.rms_norm (1.4–2.9× faster)
layer_norm_fn	Pure PyTorch decomposition	mx.fast.layer_norm (1.4–2.9× faster)
fuse_scale_shift_kernel	PyTorch broadcast multiply-add	N/A (already efficient)

The MLX-accelerated norms use single fused Metal kernels instead of 7+ separate PyTorch MPS kernel launches, providing significant speedup for medium-to-large tensors.

2.2 Shared MLX Primitives Extraction

Status: 📋 Planned

Extract reusable ops from flux_mlx.py into a shared module:

Primitive	Reusable By
_linear(x, w, b)	All models
_layer_norm(x, eps)	FLUX, Sana, GLM, LTX2
_rms_norm(x, w, eps)	Wan, HunyuanVideo, Qwen
_silu(x)	All models
_gelu_tanh(x)	FLUX, Sana, GLM
_apply_rope(q, k, cos, sin)	All models (style varies)
mx.fast.scaled_dot_product_attention	All models
load_weights(path) → dict	All models
Weight-key lookup helpers	All models
AdaLayerNorm variants	FLUX, Sana, GLM, Qwen

Deliverable: python/sglang/multimodal_gen/runtime/models/mlx_ops.py

2.3 Model-by-Model Migration

Each DiT model has its own block structure, weight naming, attention patterns, and conditioning mechanisms. There is no universal swap — each model's forward pass must be reimplemented using the shared primitives.

Phase 1: Image Models

Model	File	Complexity	Key Characteristics
Sana	sana.py	Low	Uniform blocks, self+cross attention, LayerNorm, GELU, no RoPE (2D pos embed)
ZImage	zimage.py	Low-Med	Dual-stream like FLUX, joint attention, very high structural overlap with FLUX
GLM Image	glm_image.py	Medium	Uniform blocks, joint attention, 2D RoPE (interleaved), same AdaLayerNorm as FLUX
Qwen Image	qwen_image.py	Med-High	Dual-stream with cross-modulation, RMSNorm, SiLU-gated MLP, 2D RoPE

Phase 2: Video Models

Video models add temporal dimensions, 3D RoPE, and larger memory requirements. Smaller models (1.3B) are viable on Apple Silicon.

Model	File	Complexity	Key Characteristics
Wan 2.1 T2V 1.3B	wanvideo.py	High	Self+cross attention, RMSNorm with QK norm, SiLU-gated FFN, 3D RoPE
HunyuanVideo	hunyuanvideo.py	High	Double+single stream (like FLUX), RMSNorm, 3D RoPE, large model (13B needs quantization)
LTX2	ltx_2.py	Very High	Multi-modal (video+audio), custom QKNorm, 3D RoPE, custom patchification

Phase 3: Specialized Models (On Demand)

Helios, MOVA, Hunyuan3D — migrate only if there is demand on Apple Silicon.

Suggested Execution Order

Phase 0:  Extract mlx_ops.py from flux_mlx.py
            ↓
Phase 1a: Sana (simplest, validates mlx_ops)
            ↓
Phase 1b: ZImage (closest to FLUX structurally)
            ↓
Phase 1c: GLM Image
            ↓
Phase 1d: Qwen Image
            ↓
Phase 2a: Wan 1.3B (first video model, most popular)
            ↓
Phase 2b: HunyuanVideo (if memory allows)
            ↓
Phase 2c+: LTX2, Helios, etc. (on demand)

Per-Model Migration Checklist

For each model:

• Create {model}_mlx.py with block-level forward functions using mlx_ops.py
• Map safetensors weight keys to block prefixes
• Implement any model-specific ops not in mlx_ops.py
• Add MLX fast-path routing in the model's forward() method
• Ensure _mlx_model_path is propagated in the denoising stage
• Benchmark: compare per-step latency and peak memory vs PyTorch MPS baseline
• Visual verification: confirm output matches baseline
• Add the model to the benchmark report

---

3. General / Cross-Cutting

3.1 Environment Diagnostics

Status: 🔧 In progress (PR #20753)

sglang.check_env (sglang/check_env.py) provides MPSEnv which gathers:

Information	Source
MPS availability	torch.backends.mps.is_available()
macOS version + build	platform.mac_ver() + sw_vers
Apple Silicon chip	sysctl -n machdep.cpu.brand_string
Metal support level	system_profiler SPDisplaysDataType
GPU core count	system_profiler SPDisplaysDataType
Unified memory	sysctl -n hw.memsize
CPU info	sysctl queries
Package versions	mlx, mlx-lm, mlx-metal + standard packages

PR #20753 introduces sglang show_config as a user-facing command to surface this information for troubleshooting macOS-specific issues (missing MLX, wrong PyTorch version, insufficient memory, etc.).

3.2 CI and Testing

Current: No macOS CI runners in the SGLang GitHub Actions workflow.

Planned:

• Add macOS (Apple Silicon) GitHub Actions runner or self-hosted runner
• Run bench_one_batch with Qwen3-0.6B as a regression test
• Test both SGLANG_USE_MLX=0 (pure MPS) and SGLANG_USE_MLX=1 (MLX-accelerated) paths
• Validate stub installation doesn't break Linux/CUDA paths

Benchmark baselines (Qwen3-0.6B, input_len=60, output_len=10):

Config	Prefill (token/s)	Decode Median (token/s)
BS=1	~952	~40
BS=4	~1181	~47

---

Reference PRs

PR	Title	Category	Status
#20342	MLX-native model runner	LLM	🔧 In progress
#20561	MLX-native KV cache (unified contiguous)	LLM	🔧 In progress
#20753	sglang show_config environment diagnostics	General	🔧 In progress

[changminbark]

@yeahdongcn @Jonahcb Sorry I've been busy recently. I quickly looked over @yeahdongcn 's code and result and it seems like we are getting good performance. I'm curious if overlap scheduling is being used. I haven't dove too deep into the code but it seems that you are replacing the torch/cuda related code with stubs, but the overlap scheduling uses CudaStreams and CudaStreamContexts. Maybe @yeahdongcn can explain this a little bit?

[yeahdongcn]

I'm curious if overlap scheduling is being used.

Not yet. torch.mps does not expose a concept like Stream/StreamContext. Once we switch to MLX backend (see: #20342), we can explore using streams for certain overlaps (see: https://ml-explore.github.io/mlx/build/html/usage/using_streams.html). And there will be a lot of work to do to enable SGLang features.

[changminbark]

@yeahdongcn I can take a look at this feature for overlap scheduling once the MLX backend is merged.

[yeahdongcn]

@yeahdongcn I can take a look at this feature for overlap scheduling once the MLX backend is merged.

Cool! Thanks!

[jonahbernard]

There was an official SGLang "office hours" where @alexnails discusses the MLX backend: SGLang Office Hour 3/25, and the discussion of MLX backend starts at 35:10. Please watch that if you are interested in the team's thought process around March 25 2026.

Additionally, here is follow-up discussion from the mlx-channel on the official SGLang Slack:

@yeahdongcn: Torch MPS exposes quite limited APIs and lacks fused kernels (like those in mlx.core.fast). If we stick with PyTorch, enabling torch.compile might be the main way to improve performance. Moving to MLX would also be challenging. To get precise control, we’d likely need to replicate most parts (ops, layers, modeling, etc.) from the SGLang PyTorch implementation. Using mlx-lm would be a separate path...moving to MLX might be a better choice.

@jlee5814: I'm sure the SGLang Team will end up doings something like this...cause you know 80/20 Pareto principle, where they keep PyTorch but swap in MLX's fast kernels (Attention/GEMM) and then back, using a zero overhead transfer, avoiding copying/re-writing and turning on static total memory. The hardware makes KV cache tricky but tensor bridge easy.

@Jonahcb: The PyTorch MPS scheduler is slow. It doesn't use concurrent queues like MLX. If we do stick with PyTorch we will have to really hack the scheduling

@alexnails:
I imagine there is some distribution of

• CPU work in Torch that will be torch.compile'ed

  • e.g stuff we want to become OpenVINO wayyy in the future

• Naive Support:

  • MLX impls in the MLX Torch Bridge
  • Random nuances (e.g the float64 issue)

• Specialized Kernels of two flavors:

  • Macs as of two weeks ago (lol) can now do RDMA over thunderbolt. There needs to be a whole deep dive (once support is more mature) on their parallelism ecosystem.
  • As Jae said, wrapping fast kernels once naive support is ready ` mx.core.fast.metal_kernel . This metal kernel is still JIT'ed

@alexnails: I think between the PyTorch MLX Tensor Bridge, some tricks we can do with with the low level torch api (At::from_blob / At::to_blob ), and metal kernels integrating (even if you have a fake torch op which is an mlx pass through to a fast core kernel) there is a pretty clear way to get some good performance

@yeahdongcn: Yeah, the overhead of Torch <-> MLX conversion is quite significant. I haven’t found a way to achieve zero-copy. Although mx.metal exposes memory stats and capture utilities, there’s no public API for buffer import (e.g. mlx.from_dlpack(…)). Alex’s suggestion might be worth trying, if we patch all the way down to the ATen level, we might gain some insights into how different frameworks handle buffer management.

@Jonahcb: what is the purpose of using MLX in this “not fight against PyTorch” case? The Metal kernels in MLX are nothing special and I don’t think they have big plans to optimize them further and copying the metal kernels directly into the sglang repo would have very literal maintanence burden as a baseline and get rid of the MLX PyTorch transfer costs

There were no more responses after this point.

[linzhonghong]

Hi! I’d like to pick up the “Add accuracy-based correctness tests for popular models” item for Apple Silicon / MLX. I already have an initial Qwen3-0.6B Apple MLX correctness/accuracy test working locally and plan to open a PR shortly. Please assign this item to me if it’s still available.