loadResources took long time to load UNET

[lamfan]

I had copy the model to Bundle.main.path.
It took really long time to loadResources, do we need to loadResources every time?

pipeline = try StableDiffusionPipeline(resourcesAt: resourceURL, controlNet:[], configuration: configuration, reduceMemory: true) print("Pipeline loadResources") try pipeline.loadResources() print("Pipeline loaded in \(Date().timeIntervalSince(beginDate))")

Pipeline loadResources
textEncoder took 17.54743003845215 seconds.
Invalid group id in source layers: var_49_cast <-what's that error?
Invalid group id in source layers: var_50_cast <-what's that error?
unet took 273.5967849493027 seconds.
decoder took 4.415629982948303 seconds.
encoder took 2.887573003768921 seconds.
controlNet took 9.5367431640625e-07 seconds.
safetyChecker took 56.81853806972504 seconds.
Pipeline loaded in 355.45468401908875

Model
6-bit quantized models (suitable for iOS 17 and macOS 14):
CompVis/stable-diffusion-v1-4
coreml-stable-diffusion-1-4-palettized_original_compiled
coreml-stable-diffusion-1-4-palettized_split_einsum_v2_compiled
both model are loading slow

Hardware
MacBook Pro M1 Pro
iPhone12 Pro Max, iOS17

Software
macOS:Sonoma 14.0
Xcode:Version 15.0.1

[MenKuch]

Are you using the CPU+GPU or the Neural Engine for inference? When using the Neural Engine, it takes a really long time to re-compile the model. On your MacBook Pro, you should be able to use the CPU+GPU option which does not need recompilation.

[lamfan]

@MenKuch
After I set computeUnits = .cpuAndGPU, it only take 16s to start the app, thanks!

https://www.photoroom.com/inside-photoroom/core-ml-performance-2022#

ANE + CPU is much more faster than GPU + CPU (about 500% increase), so how can we use it?
we are using coreml-stable-diffusion-v1-5 from huggingface, it is coreml model already, why does it need to re-compile the modal when running on ANE config, can we manual convert it? Many thanks!

[MenKuch]

In my understanding, the initial compilation is necessary because the model has to be compiled for a specific ANE generation (i.e. the Neural Engine for the A14, A15, A16, M1 and so on)

[lamfan]

Is there anyway to speed up the loading when using .cpuAndNeuralEngine?
or other format not mlmodelc? It's not acceptable to compiled a model in 6 mins.....

[MenKuch]

To my knowledge, there is currently no way of speeding up the process for a model. Worse: You seem to cannot get the progress of the conversion process so there is no way of letting the user know how long he has to wait.

[lamfan]

Ok, thank you!