whisper.swiftui example not working

[jeybee]

When running the whisper.swiftui example, compiled in XCode, transcription fails with the following log:

About to run whisper_full
whisper_full_with_state: failed to encode
Failed to run the model

This is using the ggml-base.en.bin model. The whisper.objc sample run on the same machine with the same model works fine.

Tested on a MBP M1 16GB.

[ggerganov]

We need to apply similar fix as we did in: ggml-org/llama.cpp#4754

[ggerganov]

Pinging @singularity-s0 as they fixed the build in llama.cpp

Here it might be more difficult because the examples uses the whisper.cpp Swift.Package which in turn depends on the ggml Swift.Package. I tried to add a similar build rule, but couldn't figure out the details, so help will be appreciated

[singularity-s0]

Xcode build rules don't seem to apply to Swift packages. In fact, this post suggests custom build behavior for Swift packages might not be supported by Xcode at all. Need to find another way around this.

[ggerganov]

Ok, pinging @1-ashraful-islam as well

[jeybee]

Just wanted to flag that even when manually changing ggml to look for the default.metallib that does exist, and having the Metal device successfully initialised, the same error log still occurs.

[ggerganov]

Did you change the ggml-metal.m in whisper.cpp or in ggml? You have to change it in the latter because that is what the Swift package uses

[zshannon]

ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: ggml.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x600002e4e280 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed

So seems like moving ggml to a SPM dependency broke loading the metal file?

[jeybee]

Did you change the ggml-metal.m in whisper.cpp or in ggml? You have to change it in the latter because that is what the Swift package uses

Yes, I changed it in GGML and in the logs I can see it successfully loads and allocates the Metal buffers but transcription still fails.

[ggerganov]

@jeybee I figure it out - there was a divergence in the ggml API because of commit a3d0aa7

After syncing it back to the ggml repo (ggml-org/ggml@9a867f1) the SwiftUI example now works correctly (make sure to update the Swift Packages to latest version: Xcode -> File -> Packages -> Update to latest package version)

Still, the problem with ggml.metallib remains unresolved, so the example will fallback to CPU transcription if it cannot load ggml.metallib

[zshannon]

It might be a problem of SPM that bundle resources can't be copied from dependencies, so eg copying ggml-metal.metal in ggml then depending on ggml in whisper.cpp/llama.cpp doesn't pull the metal file into the final build. You worked around it in the swiftui example in llama.cpp by adding a build step to Xcode, but that was only viable because the metal file is present in the llama.cpp repo, which sorta blows up the value of using SPM to consume the package... (looking into work arounds now)

[zshannon]

@ggerganov will the metal files always be present in the respective whisper/llama repos and sync'd with the ggml repo? Perhaps copying the metal files in each whisper/llama swift package while depending on the ggml swift package's compiled ggml lib (but not metal files) solves the twin problems of needing the metal files + avoiding the duplication of symbols compiler error when using both libs?

[ggerganov]

Yes, the metal files will always be present in the downstream Swift packages. Probably this is the way to go then