Is there support for Apple MPS?

[sopotc]

Is there support for Apple Metal Performance Shaders (MPS)? If not, are there any plans to support it? If not, any pointers where one would add such support?

MPS is more and more used to do both training and inference on Macbooks as it has a massive speedup wrt Apple CPU.

[decahedron1]

There is no MPS backend, but the CoreML backend is well supported in ort. A list of supported operators for CoreML can be found here - most models should work well.

[sopotc]

Many thanks for the prompt responses and for bearing with me.

I run this code with the debug setup:

fn main() {
    tracing_subscriber::fmt::init();

    let environment = Arc::new(
        Environment::builder()
            .with_name("TEST")
            .with_execution_providers([ExecutionProvider::coreml()])
            .build().unwrap()
    );
    dbg!(ExecutionProvider::coreml().is_available());
    return;

and this is what comes out

2023-04-12T18:23:22.461492Z DEBUG ort::environment: Environment not yet initialized, creating a new one.
2023-04-12T18:23:22.471985Z DEBUG ort::environment: Environment created. env_ptr="0x600000d8dd60"
[src/main.rs:17] ExecutionProvider::coreml().is_available() = true
2023-04-12T18:23:22.472023Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "TEST", env_ptr: 0x600000d8dd60 }, poisoned: false, .. }, execution_providers: [ExecutionProvider { provider: "CoreMLExecutionProvider", options: {} }] }}: ort::environment: Dropping the Environment. global_arc_count=2
2023-04-12T18:23:22.472031Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "TEST", env_ptr: 0x600000d8dd60 }, poisoned: false, .. }, execution_providers: [ExecutionProvider { provider: "CoreMLExecutionProvider", options: {} }] }}: ort::environment: Releasing the Environment. global_arc_count=2

I do not see the phrase you mentioned in the log.

[decahedron1]

Hmm, is the coreml feature enabled for ort in Cargo.toml?

[dependencies]
ort = { version = "1.14", features = [ "coreml" ] }

[sopotc]

Ashamed to say it was not.
In the log I now see DEBUG apply_execution_providers: ort::execution_providers: CoreML execution provider registration Ok(())

However the inference times are unchanged around 200ms per inference both for CPU and CoreML. It might just be that my model is not amenable to CoreML inference, but on Python I do see a difference.

[sopotc]

Also worth adding that the GPU usage does not move an inch although a batch of 100 inference requests takes around 20 seconds to run, but CPU usage spikes.

[simgt]

@sopotc Did you end up finding the issue and manage to make it work on the ANE or GPU?