Add support for Apple Silicon Metal

[Shawn Cicoria (cicorias)]

Are there any plans for Metal support on Apple Silicon (M1…4)

[Samuel Kemp (samuel100)]

Thanks for raising an issue Shawn Cicoria (@cicorias) ! I'll move this to a discussion (under feature requests). What is your motivation for wanting Metal support - performance? If so, what level of performance over WebGpu?

[Dr. Shawn P Cicoria (shawncicoria)]

Yes, inference performance is the desire.

Ideally for Apple Silicon it uses the GPU -- since I can't see the source code for the foundry service Inference.Service.Agent -- just not sure if Metal is equivalent to webgpu.

For example, in Candle it's clear that Metal (GPU) is used.

Would the best place to look is in the ONNX Runtime Repo?

[Samuel Kemp (samuel100)]

On Apple Silicon we leverage the GPU (not cpu) via the ONNX Runtime WebGpuExecutionProvider. When you run the model you should see that it is pulling model-generic-gpu - what this means is that the ONNX graph has been optimized for any GPU. For some additional providers like CudaExecutionProvider the ONNX graph has some additional optimizations for CUDA that can be leveraged in the ONNX Runtime.

The WebGPU Execution Provider uses a library called Dawn that converts from the WebGPU shader language to Metal. Therefore, to directly answer the question.... "Foundry Local already supports Metal".

[Shawn Cicoria (cicorias)]

Sweet

[Scott McKay (skottmckay)]

WebGPU EP uses a library called Dawn that converts from the WebGPU shader language to Metal.

[Shawn Cicoria (cicorias)]

Sweet x2!