Update `whisper.cpp` to 8e46ba8 #85

tazz4843 · 2023-08-28T23:09:26Z

See full diff at ggerganov/whisper.cpp@a5defbc...ggerganov:whisper.cpp:8e46ba80d3c1dcf532a0029f9bcdf99ce9ce7d40

Big features:

Adds OpenVINO support
Adds tdrz support

Will not be merged until we get a stable release to fix to.

cenzovit · 2023-09-05T15:39:53Z

Looking at recent changes to whisper.cpp, it feels like this should be updated sooner rather than later. IMO the biggest "feature" is actually that it appears there were important fixes to the log_mel_spectrogram generation in whisper.cpp which improves WER for the smaller models significantly (ggerganov/whisper.cpp#1148).

tazz4843 · 2023-09-05T16:38:00Z

I found a related issue yesterday and appears if everything goes well upstream should have a new release by end of week. See ggerganov/whisper.cpp#1233 (comment)

tazz4843 · 2023-09-08T04:31:23Z

Figured I'd point out something possibly concerning for those who want to use this branch: ggerganov/whisper.cpp#1220 (comment)

These changes seem to be fine when using normal models, but quantized models produce broken output. For example, the jfk speech outputs only exclamation marks from my testing. I haven't changed any parameters.

It may be worth doing some local testing before upgrading whisper-rs, just in case this affects you.

tazz4843 · 2023-09-14T22:12:46Z

OpenVINO support is broken: see ggerganov/whisper.cpp#1289

tazz4843 · 2023-09-30T05:27:02Z

I think I'm gonna pull out the OpenVINO support into a separate PR (given it's completely borked right now), merge everything else, and then push a prerelease to crates.io since upstream seems to be taking a bit to merge stuff.

chen-rn · 2023-10-03T19:36:57Z

I think I'm gonna pull out the OpenVINO support into a separate PR (given it's completely borked right now), merge everything else, and then push a prerelease to crates.io since upstream seems to be taking a bit to merge stuff.

That would be fantastic :)
As it seems that the performance boost is huge for both inference speed and WER! Esp the smaller models!

This reverts commit 5c140c1

tazz4843 · 2023-10-07T04:16:26Z

0.9.0-rc.1 has been published to crates.io for anyone who would like to use this branch.

cenzovit · 2023-10-07T09:06:31Z

Attempted to give 0.9.0-rc.1 a whirl but ran into a cmake error:
CMake Error: File .cargo/registry/src/index.crates.io-6f17d22bba15001f/whisper-rs-sys-0.7.0/whisper.cpp/ggml-metal.metal does not exist. CMake Error at CMakeLists.txt:149 (configure_file): configure_file Problem configuring file

tazz4843 · 2023-10-07T15:29:38Z

@cenzovit Do a cargo update, I released -sys version 0.7.1 which adds ggml-metal.metal to the list of included files.

cenzovit · 2023-10-08T19:55:37Z

Hmm, tried to update today and running into a new cmake error:

  CMake Error at CMakeLists.txt:462 (add_library):
    Cannot find source file:

      ggml-metal.m

    Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
    .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
    .f95 .f03 .hip .ispc


  CMake Error at CMakeLists.txt:462 (add_library):
    No SOURCES given to target: whisper

tazz4843 · 2023-10-08T19:59:35Z

Only macOS machine I have is a late 2012 MBP running 10.15, but should be recent enough to support whisper.cpp's Metal. Will give it a shot on there.

tazz4843 · 2023-10-08T21:41:38Z

Should be fixed in -sys v0.7.2, compiles on macOS 10.15

cenzovit · 2023-10-09T18:45:43Z

Got further this time, now running into a clang error on compile:

 Undefined symbols for architecture x86_64:
            "_MTLCopyAllDevices", referenced from:
                _ggml_metal_init in libwhisper_rs_sys-3c0021a7bf9aa3b1.rlib(ggml-metal.m.o)
            "_MTLCreateSystemDefaultDevice", referenced from:
                _ggml_metal_init in libwhisper_rs_sys-3c0021a7bf9aa3b1.rlib(ggml-metal.m.o)
            "_OBJC_CLASS_$_MTLComputePassDescriptor", referenced from:
                objc-class-ref in libwhisper_rs_sys-3c0021a7bf9aa3b1.rlib(ggml-metal.m.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

This one may be an issue in whisper.cpp though?

tazz4843 · 2023-10-09T18:47:02Z

That does seem like a whisper.cpp error since all we do is pass it off to upstream. Unless there's another missing file I don't think it's something we can fix.

cenzovit · 2023-10-09T18:52:37Z

Maybe related: ggerganov/whisper.cpp#1344

cenzovit · 2023-10-09T19:09:46Z

I wonder if other language bindings are seeing similar issues with the latest whisper.cpp

cenzovit · 2023-10-09T21:12:15Z

Further notes, if I link the missing frameworks myself (which I think are suppose to be getting linked by ldflags in the whisper.cpp makefile) via config.toml or my build.rs -- for example in build.rs:

    println!("cargo:rustc-link-lib=framework=Foundation");
    println!("cargo:rustc-link-lib=framework=Metal");
    println!("cargo:rustc-link-lib=framework=MetalKit");

I can get my app to build. However, at runtime I run into a different issue as whisper goes to initialize:

ggml_metal_init: allocating
ggml_metal_init: found device: AMD Radeon Pro 5500M
ggml_metal_init: found device: Intel(R) UHD Graphics 630
ggml_metal_init: picking default device: AMD Radeon Pro 5500M
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

Which seems to be occurring because ggml can't find the path to metallib: https://github.com/ggerganov/ggml/blob/6549d12f2e3176050040a86334f17c001e170f13/src/ggml-metal.m#L200C8-L200C8

At this point I am trying to figure out how to just build whisper.cpp without metal via the WHISPER_NO_METAL env variable; however, I can't seem to figure out how to have it set properly at the time whisper.cpp is being compiled.

tazz4843 · 2023-10-09T21:13:50Z

It should just be as simple as setting it for the cargo build command I would think.

cenzovit · 2023-10-09T21:22:27Z

That's what I would have thought, but when I do that it still tries to initialize metal during runtime and fails with the same issue above

tazz4843 · 2023-10-09T21:23:43Z

If you have a local copy you could probably just edit it out of the Makefile as a temporary workaround, but we'd need to find something else for real use.

tazz4843 · 2023-10-09T22:22:54Z

Oh I was able to hack macOS 14 onto this system so I can give it a shot again and see as well. Will do that now.

cenzovit · 2023-10-09T22:37:36Z

Is there a chance that when whisper-rs-sys runs cmake on whisper.cpp that there is a metal file that needs to be copied to out as well (similar to coreml etc)? Separately, would it be possible to make metal configurable via a feature similar to how coreml is configured?

tazz4843 · 2023-10-09T22:41:04Z

With macOS 14 I can indeed reproduce this issue. However I get some extra missing lib errors that it appears you didn't get

Full error

= note: ld: Undefined symbols:
            _MTLCopyAllDevices, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _MTLCreateSystemDefaultDevice, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_MTLComputePassDescriptor, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSBundle, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSObject, referenced from:
                _OBJC_CLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSString, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_METACLASS_$_NSObject, referenced from:
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            ___CFConstantStringClassReference, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                ...
            __objc_empty_cache, referenced from:
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _OBJC_CLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_autoreleasePoolPop, referenced from:
                _ggml_metal_graph_compute in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_autoreleasePoolPush, referenced from:
                _ggml_metal_graph_compute in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_enumerationMutation, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_msgSend, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                ...
            _objc_opt_class, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_release, referenced from:
                _ggml_metal_free in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

tazz4843 · 2023-10-09T22:42:05Z

Separately, would it be possible to make metal configurable via a feature similar to how coreml is configured?

This is what I want to do in the end to align Metal with all other acceleration layers.

tazz4843 · 2023-10-09T23:02:16Z

I think I was able to hack together a Metal patch, will make a new branch and PR for it, you can try it out from there.

cenzovit · 2023-10-10T00:23:28Z

With macOS 14 I can indeed reproduce this issue. However I get some extra missing lib errors that it appears you didn't get

Full error

= note: ld: Undefined symbols:
            _MTLCopyAllDevices, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _MTLCreateSystemDefaultDevice, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_MTLComputePassDescriptor, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSBundle, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSObject, referenced from:
                _OBJC_CLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_CLASS_$_NSString, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _OBJC_METACLASS_$_NSObject, referenced from:
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            ___CFConstantStringClassReference, referenced from:
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                 in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                ...
            __objc_empty_cache, referenced from:
                _OBJC_METACLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _OBJC_CLASS_$_GGMLMetalClass in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_autoreleasePoolPop, referenced from:
                _ggml_metal_graph_compute in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_autoreleasePoolPush, referenced from:
                _ggml_metal_graph_compute in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_enumerationMutation, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_msgSend, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                _ggml_metal_add_buffer in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
                ...
            _objc_opt_class, referenced from:
                _ggml_metal_init in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
            _objc_release, referenced from:
                _ggml_metal_free in libwhisper_rs_sys-5632026198159c2d.rlib[17](ggml-metal.m.o)
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

When you add the libs and get past the undefined symbols errors, do you run into the same runtime issue where it can't find the path to the metal file?

cenzovit · 2023-10-10T00:48:55Z

Another maybe related issue (even though it is in llama.cpp)? ggerganov/llama.cpp#1912

cenzovit · 2023-10-10T00:58:48Z

Interestingly in: ggerganov/whisper.cpp#1270 there is an incomplete todo for "Can't figure out how to make the CMake build look for ggml-metal.metal in the bin folder"

tazz4843 · 2023-10-10T19:27:10Z

whisper.cpp upstream error

ggml_metal_init: allocating
ggml_metal_init: found device: Intel HD Graphics 4000
ggml_metal_init: picking default device: Intel HD Graphics 4000
ggml_metal_init: loading '/Users/niko/whisper.cpp/ggml-metal.metal'
ggml_metal_init: loaded kernel_add                         0x600002cf0680 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_add_row                     0x600002cf7600 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_mul                         0x600002cf7880 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_mul_row                     0x600002cfc880 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_scale                       0x600002cf0c00 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_silu                        0x600002cf1800 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_relu                        0x600002cf7480 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_gelu                        0x600002cf6e80 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_soft_max                               0x0 | th_max =    0 | th_width =    0
ggml_metal_init: load pipeline error: Error Domain=CompilerError Code=2 "Compiler encountered an internal error" UserInfo={NSLocalizedDescription=Compiler encountered an internal error}
whisper_init_state: ggml_metal_init() failed
error: failed to initialize whisper context

However with whisper-rs I get a different error, regardless of features:

whisper_model_load: adding 1608 extra tokens
whisper_model_load: model ctx     =  464.68 MB
fish: Job 1, 'cargo run --example audio_trans…' terminated by signal SIGILL (Illegal instruction)

tazz4843 · 2023-10-10T21:47:08Z

After thinking about it, chances are my error's related to trying to use Metal on a legacy Metal GPU (i5-3210M). Probably can't help further with this, as it appears to not just be an isolated case (the Photos app crashed a few days ago due to an internal shader compilation error as well).

tazz4843 added 6 commits August 28, 2023 16:38

Update whisper.cpp to 8e46ba80d3c1dcf532a0029f9bcdf99ce9ce7d40

66f8aae

Fix typo

776abc3

Add OpenVINO support

5c140c1

Add new nosp token

851fca3

Add support for tdrz and debug mode

6668c4e

cargo fmt

749c18c

This was referenced Aug 28, 2023

4_0, 4_1, and 8_0 Quantized models fail to load - SIGSEGV invalid memory reference #70

Closed

whisper-rs-sys is likely compiled without AVX support #78

Closed

how to use tdrz params? #80

Closed

This was linked to issues Aug 28, 2023

whisper-rs-sys is likely compiled without AVX support #78

Closed

how to use tdrz params? #80

Closed

tazz4843 mentioned this pull request Sep 8, 2023

calculate mel.n_len as mel spectrogram stride len #75

Merged

fix broken build

e6cbc75

tazz4843 mentioned this pull request Oct 7, 2023

No VAD (Voice Activity Detection) #88

Closed

tazz4843 added 3 commits October 6, 2023 21:09

Revert "Add OpenVINO support"

d6da605

This reverts commit 5c140c1

Merge branch 'master' into whisper-8e46ba8

f2d528e

Update whisper.cpp

4b45810

tazz4843 marked this pull request as ready for review October 7, 2023 04:00

tazz4843 merged commit 6e864d9 into master Oct 7, 2023
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `whisper.cpp` to 8e46ba8 #85

Update `whisper.cpp` to 8e46ba8 #85

tazz4843 commented Aug 28, 2023

cenzovit commented Sep 5, 2023

tazz4843 commented Sep 5, 2023

tazz4843 commented Sep 8, 2023

tazz4843 commented Sep 14, 2023

tazz4843 commented Sep 30, 2023

chen-rn commented Oct 3, 2023

tazz4843 commented Oct 7, 2023

cenzovit commented Oct 7, 2023

tazz4843 commented Oct 7, 2023

cenzovit commented Oct 8, 2023

tazz4843 commented Oct 8, 2023

tazz4843 commented Oct 8, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

cenzovit commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 10, 2023

cenzovit commented Oct 10, 2023

cenzovit commented Oct 10, 2023

tazz4843 commented Oct 10, 2023

tazz4843 commented Oct 10, 2023

Update whisper.cpp to 8e46ba8 #85

Update whisper.cpp to 8e46ba8 #85

Conversation

tazz4843 commented Aug 28, 2023

cenzovit commented Sep 5, 2023

tazz4843 commented Sep 5, 2023

tazz4843 commented Sep 8, 2023

tazz4843 commented Sep 14, 2023

tazz4843 commented Sep 30, 2023

chen-rn commented Oct 3, 2023

tazz4843 commented Oct 7, 2023

cenzovit commented Oct 7, 2023

tazz4843 commented Oct 7, 2023

cenzovit commented Oct 8, 2023

tazz4843 commented Oct 8, 2023

tazz4843 commented Oct 8, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

cenzovit commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

tazz4843 commented Oct 9, 2023

cenzovit commented Oct 10, 2023

cenzovit commented Oct 10, 2023

cenzovit commented Oct 10, 2023

tazz4843 commented Oct 10, 2023

tazz4843 commented Oct 10, 2023

Update `whisper.cpp` to 8e46ba8 #85

Update `whisper.cpp` to 8e46ba8 #85