Crash on old CPUs (non AVX, AVX2, etc)

[Macoron]

There was report (#23) that whisper.cpp crashes on old CPUs that doesn't support architectures extensions like AVX, AVX2. This is because compiled libraries for Windows and Linux assumes that hardware supports AVX and AVX2. When compiled library with AVX trying to run on non-AVX CPU it will crash.

This isn't a problem for Mac x86_64, because it compiled without AVX flags. Accelerate works good enough.

I think that in most cases performance for non-AVX CPUs will be very bad. Also according to Steam Survey, 90% of CPUs supports AVX2. So the best way address it is probably just check hardware in runtime and throw error if AVX isn't available.

If you for some reason would need to run this library on non-AVX supported device - recompile whisper.cpp libraries.

[achimmihca]

I think I ran into this issue on my Windows PC with older (but not stone age old) hardware.
See Player.log

...
========== OUTPUTTING STACK TRACE ==================

0x00007FF88BE2195B (libwhisper) ggml_init
0x00007FF88BE45A58 (libwhisper) ggml_view_tensor
0x00007FF88BE4F90B (libwhisper) whisper_init_no_state
0x00007FF88BE4F843 (libwhisper) whisper_init_from_file_no_state
0x00007FF88BE4F74B (libwhisper) whisper_init_from_file
0x0000022150AE8E42 (Mono JIT Code) (wrapper managed-to-native) Whisper.Native.WhisperNative:whisper_init_from_file (string)
0x0000022150AE19B3 (Mono JIT Code) [.\Packages\com.whisper.unity\Runtime\WhisperWrapper.cs:254] Whisper.WhisperWrapper:InitFromFile (string) 
0x0000022150AE14D3 (Mono JIT Code) [.\Packages\com.whisper.unity\Runtime\WhisperWrapper.cs:223] Whisper.WhisperWrapper/<>c__DisplayClass24_0:<InitFromFileAsync>b__0 () 
0x0000022150AE1245 (Mono JIT Code) System.Threading.Tasks.Task`1<TResult_REF>:InnerInvoke ()
0x0000022150AE1084 (Mono JIT Code) System.Threading.Tasks.Task:Execute ()
0x0000022150AE0EF3 (Mono JIT Code) System.Threading.Tasks.Task:ExecutionContextCallback (object)
0x0000022150AE0A8E (Mono JIT Code) System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
0x0000022150AE05BB (Mono JIT Code) System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
0x0000022150AE012B (Mono JIT Code) System.Threading.Tasks.Task:ExecuteWithThreadLocal (System.Threading.Tasks.Task&)
0x0000022150A9F85B (Mono JIT Code) System.Threading.Tasks.Task:ExecuteEntry (bool)
0x0000022150A9F73B (Mono JIT Code) System.Threading.Tasks.Task:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem ()
0x0000022150A9BC1A (Mono JIT Code) System.Threading.ThreadPoolWorkQueue:Dispatch ()
0x0000022150A9B48B (Mono JIT Code) System.Threading._ThreadPoolWaitCallback:PerformWaitCallback ()
0x0000022150A9B5D5 (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_bool (object,intptr,intptr,intptr)
0x00007FF885E7E0E4 (mono-2.0-bdwgc) mono_jit_set_domain
0x00007FF885DBEB84 (mono-2.0-bdwgc) mono_object_get_virtual_method
0x00007FF885DFB9B7 (mono-2.0-bdwgc) mono_thread_has_sufficient_execution_stack
0x00007FF885DFEA50 (mono-2.0-bdwgc) mono_thread_has_sufficient_execution_stack
0x00007FF885DED35B (mono-2.0-bdwgc) mono_profiler_init_etw
0x00007FF885DED536 (mono-2.0-bdwgc) mono_profiler_init_etw
0x00007FF8BEA47614 (KERNEL32) BaseThreadInitThunk
0x00007FF8C07426F1 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

A crash has been intercepted by the crash handler. For call stack and other details, see the latest crash report generated in:
 * C:/Users/andre/AppData/Local/Temp/DefaultCompany/whisper-example/Crashes

So the best way address it is probably just check hardware in runtime and throw error if AVX isn't available

The CPU (i5-2500K) does already support AVX.
But I get the crash anyway.

What is required exactly, just any AVX or AVX2?

Anyway, I think the proper solution to this issue is running the model on the GPU by using Barracuda (#32)

[achimmihca]

What is required exactly, just any AVX or AVX2?

You already answered this, both are required.

compiled libraries for Windows and Linux assumes that hardware supports AVX and AVX2

Then my question is: how can I check for these from C#?

[Macoron]

Then my question is: how can I check for these from C#?

That's the trick! There is no C# API that supported by Unity (at least I didn't found one checking specifically for AVX2). However there is some C++ libs that can check CPU and OS capabilities, like:

https://github.com/Mysticial/FeatureDetector

One way is to compile this C++ library and distribute it along with whisper.cpp.

Other option - merge checks from this library directly to ggml. I created issue for this here:
ggerganov/ggml#182

Last option is to distribute non-AVX2 dlls with whisper.unity. But I would need to check if PC had AVX support in runtime anyway.

[achimmihca]

Thanks for the hints.

I found that GCC has a built-in method to check for CPU capabilities, including a check for AVX support.

This little program works for me

#include <stdio.h>

/**
 * Check whether AVX is supported
 */
int main(int argc, char** argv)
{
    // NOTE: __builtin_cpu_supports is a method of GCC. Other compilers are not supported.
    // You can install GCC on Windows for example via https://msys2.org

    // 0 if not supported
    int avxSupport = __builtin_cpu_supports("avx");
    int avx2Support = __builtin_cpu_supports("avx2");
    printf("AVX: %d\n", avxSupport);
    printf("AVX2: %d\n", avx2Support);
    return 0;
}

I can parse the program output to determine whether AVX2 is supported.

EDIT: I created a repo for this and added a release with the exe. So others do not need install GCC and compile it.

[Macoron]

Considering this, webgl and barracuda, I'm more convinced to move whisper.cpp to custom fork. This will allow to include AVX check code into dll and use it in C# runtime.

[ehabtawfikelbwab]

Is that problem fixed ?