-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
the example is unable to run on iphone 11 pro. #10
Comments
Thank you for the report! TestFlight feedback also showed that iPhone 11 and iPhone XS (the 2 oldest devices still supported by iOS17) are consistently having issues. My current suspicion is that this is due to the Neural Engine specifically. If you are able to rebuild the app after changing the compute units to GPU and test it on your 11 Pro, would be much appreciated :) |
Looking a bit deeper into your debug logs, the model forward passes seem to be successful but model outputs are likely to be corrupt if you are getting |
Yes one thing standing out to me here is that the cache size is only 2. Can you post the full debug logs from the beginning of the loop all the way to the end? That way we can see what the tokens are coming through as - typically for base you would want to see 0: <|startoftranscription|> |
I tried the base model with ios app whisperBoard, which was ok. doUnloadModel:options:qos:error:: model=_ANEModel: { modelURL=file:///var/mobile/Containers/Data/Application/F857AEFB-FBFC-4530-825D-81DA7772EC2D/Library/Caches/com.moody.whisperkit.WhisperAX/com.apple.e5rt.e5bundlecache/21C66/55FDBDC681E9BEC3377E76410A13B6A36D117901B2EACD6CCEC8EF6477BBCE68/F154AEEA75FF5C72ECAEBA3C899508F053C1576E8D7761AF31DBC258BF1ABFC3.bundle/H12.bundle/main/main_eir/ : sourceURL= (null) : key={"isegment":5,"inputs":{"encoder_output_embeds_eir":{"shape":[1500,1,1,512,1]}},"outputs":{"key_cast_fp16":{"shape":[1500,1,1,512,1]},"value_cast_fp16":{"shape":[1500,1,1,512,1]}}} : identifierSource=0 : cacheURLIdentifier=C231E55A5F6BD6E18E60C70EF622BACAF5215E7A45E150A265BFE09C379D6CA7_2393CF36BE99B057765D2CBDB5257F2EF24668A91C43BF3732D9D8E6D0053626 : string_id=0x70ac000000264af7 : program=_ANEProgramForEvaluation: { programHandle=4755813074256 : intermediateBufferHandle=4755813104416 : queueDepth=127 } : state=3 : programHandle=4755813074256 : intermediateBufferHandle=4755813104416 : queueDepth=127 : attr={ [WhisperKit] Loaded audio encoder [WhisperKit] Loaded tokenizer tcp_input [C2.1.1:3] flags=[R.] seq=67868, ack=3630046153, win=32950 state=LAST_ACK rcv_nxt=67868, snd_una=3630046129 [WhisperKit] Current audio size: 64000 samples, most recent buffer: 1600 samples, most recent energy: (0.07724396, 0.0031911524, 0.009436812, 1.4101643e-06) [WhisperKit] Current audio size: 96000 samples, most recent buffer: 1600 samples, most recent energy: (0.48100042, 0.03933922, 0.13287143, 2.9166695e-05) [WhisperKit] Current audio size: 128000 samples, most recent buffer: 1600 samples, most recent energy: (0.293728, 0.011141126, 0.041470844, 5.8486767e-06) [WhisperKit] Current audio size: 160000 samples, most recent buffer: 1600 samples, most recent energy: (0.3946247, 0.022941826, 0.073072806, 3.928726e-06) [WhisperKit] Current audio size: 192000 samples, most recent buffer: 1600 samples, most recent energy: (0.054507904, 0.0029965553, 0.0102628, 8.92207e-07) [WhisperKit] Decoder init time: 0.006031990051269531 |
Here is the problem
The loop is hitting its log prob threshold before end of text, which is likely part of the early stopping mechanism in the app. Normally it's not an issue that early but perhaps on iphone 11 it is. One step to see if it solves the issue is to try editing line 929 in the example app let options = DecodingOptions(
verbose: false,
task: task,
language: languageCode,
temperatureFallbackCount: 3, // limit fallbacks for realtime
sampleLength: Int(sampleLength), // reduced sample length for realtime
usePrefillPrompt: enablePromptPrefill,
usePrefillCache: enableCachePrefill,
skipSpecialTokens: !enableSpecialCharacters,
withoutTimestamps: !enableTimestamps,
clipTimestamps: seekClip,
logProbThreshold: 0 //<-------- add this line
) Will attempt to reproduce this in the meantime, thanks for the report 👍 |
I am sorry, Still same error. |
@woodymoo Checking in on this, it appears to be a device-specific bug. If anyone can reproduce this consistently on an iPhone 11 Pro - please post here with any other info. |
Feel like this is related but why does the Tesflight version run faster than the Xcode one? Debug/Release build or further optimizations? Transcription appears with a 10 second delay with Xcode (same model size) |
No solution yet? Experiencing the same issue on iPad8,8. Tried above "logProbThreshold: 0" tip but no luck.
|
the example is unable to run on iphone 11 pro.
(The example is running good on Mac m1 max )
The following is the screen shot on iphone 11 pro. Base Model
debug log:
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 2 Input Token: 50359
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 0 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] [0.00 --> 14.90]
[WhisperKit] ---- Transcription Timings ----
[WhisperKit] Audio Load: 0.00 ms / 1 runs ( 0.00 ms/run) 0.00%
[WhisperKit] Audio Processing: 0.41 ms / 1 runs ( 0.41 ms/run) 0.03%
[WhisperKit] Mels: 57.57 ms / 1 runs ( 57.57 ms/run) 3.96%
[WhisperKit] Encoding: 1171.59 ms / 1 runs ( 1171.59 ms/run) 80.56%
[WhisperKit] Matrices Init: 5.36 ms / 1 runs ( 5.36 ms/run) 0.37%
[WhisperKit] Prefill: 0.49 ms / 1 runs ( 0.49 ms/run) 0.03%
[WhisperKit] Decoding: 208.06 ms / 4 runs ( 52.01 ms/run) 14.31%
[WhisperKit] Non-inference: 7.49 ms / 4 runs ( 1.87 ms/run) 0.52%
[WhisperKit] - Sampling: 4.13 ms / 4 runs ( 1.03 ms/run) 0.28%
[WhisperKit] - Kv Caching: 3.91 ms / 4 runs ( 0.98 ms/run) 0.27%
[WhisperKit] - Windowing: 0.08 ms / 1 runs ( 0.08 ms/run) 0.01%
[WhisperKit] Fallbacks: 122.98 ms / 0 runs ( 0.00 ms/run) 8.46%
[WhisperKit] Decoding Full Loop: 1448.16 ms / 4 runs ( 362.04 ms/run) 99.57%
[WhisperKit] -------------------------------
[WhisperKit] Model Load Time: 6.60 seconds
[WhisperKit] Inference Duration: 1.45 seconds
[WhisperKit] - Decoding Loop: 1.45 seconds
[WhisperKit] Time to first token: 1.30 seconds
[WhisperKit] Total Tokens: 5
[WhisperKit] Tokens per Second: 2.76 tok/s
[WhisperKit] Real Time Factor: 0.10
[WhisperKit] Fallbacks: 0.0
[WhisperKit] [0.00 --> 14.90] <|endoftext|>
The text was updated successfully, but these errors were encountered: