Audio to Text doesn't function on all systems

[chenlung]

Before the public alpha of Tero Subtitler was released, it became apparent that Audio to Text (which uses Whisper.cpp) only worked on some systems.

What we know:

• Doesn't work on an early-2009 Mac Pro running El Capitan (10.11.6). [my system]
• Doesn't work on a PC running Windows 10 with AMD FX-6300 Six-Core Processor (3.50 GHz); it does if this is used. [another system I have access to]
• Works on a MacBook Air 2022 (model Mac14,2) (M2) running macOS Ventura 13.4. [@GitGianluc]
• Works on a PC running Windows 11 with AMD Ryzen 3 PRO 4350G with Radeon Graphics (3.80 GHz). [@URUWorks]
• Works on MacBook Air Intel Core i5 running macOS Monterey/Ventura. [@URUWorks]

Could people try Audio to Text on their systems and give feedback (please include hardware and operating system)?

We may need to compile Whisper on respective machines for inclusion in Tero Subtitler (this may also have to happen for Linux). I'm not a developer, but I might be able to try with guidance (assuming it and the compiling tools run on my system).

[URUWorks]

@GitGianluc, I have some questions:

• Did you download whisper.cpp from the wizard?
• Does it show any error when trying to use Audio-to-text?

[GitGianluc]

• Did you download whisper.cpp from the wizard?

I downloaded the small model from the wizard, but it didn't work. Then I downloaded manually the ggml-large-v1.bin model file from the Georgi Gerganov’s Hugging face repository, renamed in large-v1.bin and moved it to ~/tero/Whisper/Models directory.

• Does it show any error when trying to use Audio-to-text?

After selecting the language for transcription, I have clicked “Generate” trying both settings (“Add subtitles while transcribing” and “Load subtitles after transcription”); for a brief moment, this window appears

(below Please wait the prompt Extracting… is displayed first) but then nothing happens.

[chenlung]

I asked on Whisper.cpp.

[URUWorks]

Can you try running the whisper executable from terminal, just to see if it runs?

[GitGianluc]

This is what the terminal displays:

~/tero/Whisper/main ; exit;
error: no input files specified

usage: ~/tero/Whisper/main [options] file0.wav file1.wav ...

options:
  -h,        --help              [default] show this help message and exit
  -t N,      --threads N         [4      ] number of threads to use during computation
  -p N,      --processors N      [1      ] number of processors to use during computation
  -ot N,     --offset-t N        [0      ] time offset in milliseconds
  -on N,     --offset-n N        [0      ] segment index offset
  -d  N,     --duration N        [0      ] duration of audio to process in milliseconds
  -mc N,     --max-context N     [-1     ] maximum number of text context tokens to store
  -ml N,     --max-len N         [0      ] maximum segment length in characters
  -sow,      --split-on-word     [false  ] split on word rather than on token
  -bo N,     --best-of N         [2      ] number of best candidates to keep
  -bs N,     --beam-size N       [-1     ] beam size for beam search
  -wt N,     --word-thold N      [0.01   ] word timestamp probability threshold
  -et N,     --entropy-thold N   [2.40   ] entropy threshold for decoder fail
  -lpt N,    --logprob-thold N   [-1.00  ] log probability threshold for decoder fail
  -su,       --speed-up          [false  ] speed up audio by x2 (reduced accuracy)
  -tr,       --translate         [false  ] translate from source language to english
  -di,       --diarize           [false  ] stereo audio diarization
  -nf,       --no-fallback       [false  ] do not use temperature fallback while decoding
  -otxt,     --output-txt        [false  ] output result in a text file
  -ovtt,     --output-vtt        [false  ] output result in a vtt file
  -osrt,     --output-srt        [false  ] output result in a srt file
  -olrc,     --output-lrc        [false  ] output result in a lrc file
  -owts,     --output-words      [false  ] output script for generating karaoke video
  -fp,       --font-path         [/System/Library/Fonts/Supplemental/Courier New Bold.ttf] path to a monospace font for karaoke video
  -ocsv,     --output-csv        [false  ] output result in a CSV file
  -oj,       --output-json       [false  ] output result in a JSON file
  -of FNAME, --output-file FNAME [       ] output file path (without file extension)
  -ps,       --print-special     [false  ] print special tokens
  -pc,       --print-colors      [false  ] print colors
  -pp,       --print-progress    [false  ] print progress
  -nt,       --no-timestamps     [false  ] do not print timestamps
  -l LANG,   --language LANG     [en     ] spoken language ('auto' for auto-detect)
  -dl,       --detect-language   [false  ] exit after automatically detecting language
             --prompt PROMPT     [       ] initial prompt
  -m FNAME,  --model FNAME       [models/ggml-base.en.bin] model path
  -f FNAME,  --file FNAME        [       ] input WAV file path


Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

[URUWorks]

The executable works, can you send me the file you tried to transcribe or tell me the steps?

[GitGianluc]

Sorry, I can't at the moment because we are dealing with some copyrighted material… 😢
However, these are the steps to reproduce the issue:

1. Launch Tero Subtitler.
2. Click on New empty subtitle on startup.
3. Click anywhere on the subtitles' window (just as a workaround of the issue Shortcuts don't immediately work after start-up #66) and press Ctrl + P.
4. Open a video and leave the FPS settings untouched.
5. Go to Video > Audio to text… and the settings' window will open.
6. Leave Track field untouched, select Italian from the dropdown as the language for transcription (if that is the case) and small or large-v1 model from the relevant Model dropdown menu.
7. Leave the “Add subtitles while transcribing” checkbox selected and press the Generate button.

After that, a message Please wait / Extracting… [then] Transcribing… will pop out for a few seconds, the window will disappear, but nothing will happen.

Am I doing something wrong?

[URUWorks]

I seemed to read somewhere that the Tero freezes when using audio-to-text.
Anyway, we can try from terminal and see what whisper tells you;

• First, we need to extract the audio from the video so that whisper interprets it with something similar to:

ffmpeg -i "yourvideofilefullname" -ar 16000 -ac 0 -c:a pcm_s16le -y "youroutputwavefilefullname"

• Last, tell whisper to transcribe the generated wav file:

main -m "modelfilefullname" -l it -pp -osrt -of "outputtranscribedfilefullname" -f "wavefilefullname"

and we see what happens...

[GitGianluc]

I seemed to read somewhere that the Tero freezes when using audio-to-text.

Yes, but only when I have tried to perform audio-to-text on a remote video: see issue #35.

we see what happens...

My input (in the original string, the file paths are correct):

~/tero/Whisper/main -m "~/tero/Whisper/Models/large-v1.bin" -l it -pp -osrt -of "~/tero/Whisper/Transcriptions/transcription-file.txt" -f "~/path/to/audio-file.wav"

The terminal's output:

whisper_init_from_file_no_state: loading model from '~/tero/Whisper/Models/large-v1.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5
whisper_model_load: mem required  = 3557.00 MB (+   71.00 MB per decoder)
whisper_model_load: adding 1608 extra tokens
zsh: illegal hardware instruction  ~/tero/Whisper/main -m  -l it -pp -osrt -of  -f 

I tried also with "~/tero/Whisper/Models/small.bin", but got the same result: no transcription has been done, and no file has been generated…

😭

[chenlung]

Here is what I get when I double-click main on 10.11.6.:
UsersPro:~ User$ /Volumes/HDD\ 1/user/tero/Whisper/main ; exit; dyld: Symbol not found: __ZTTNSt3__114basic_ifstreamIcNS_11char_traitsIcEEEE Referenced from: /Volumes/HDD 1/User/tero/Whisper/main Expected in: /usr/lib/libc++.1.dylib in /Volumes/HDD 1/User/tero/Whisper/main

[URUWorks]

The executable is not compatible with your version of macOS :(

[chenlung]

A Mac Pro with Apple Silicon was finally announced yesterday (having missed their self-imposed transition deadline), but it's M2 Ultra as opposed to Extreme (as rumoured in the months before). I wonder if Apple will bring out the latter later in the year. Surely an M3 Extreme by next year? 😊

[URUWorks]

I really don't know, I suppose that error could be because your Mac is M2?
maybe you can try compile whisper yourself?

https://blog.castopod.org/install-whisper-cpp-on-your-mac-in-5mn-and-transcribe-all-your-podcasts-for-free/

[chenlung]

I'm afraid that nothing can be done for El Capitan…

Yes, that's fine. Just explaining why I haven't tried it yet. 🙂

[GitGianluc]

Yes, that's fine.

No, that is not! 😄
I, too, tried to compile Whisper.cpp on a Mac running El Capitan 10.11.6, but I received 3 warnings and 2 errors generated. It's frustrating that we can't help you out! 😢

[chenlung]

Yes, that's fine.

No, that is not! 😄 I, too, tried to compile Whisper.cpp on a Mac running El Capitan 10.11.6, but I received 3 warnings and 2 errors generated. It's frustrating that we can't help you out! 😢

Thanks for your efforts!

Anyway, don't worry. I might get the M2 Ultra Mac Pro soon (waited over two years now because I knew the Intel one would become increasingly obsolete), but I'm thinking of waiting a little longer in case M2 Extreme is released (maybe later in the year? Hard to say, but I'm reluctant to purchase now in case it does come).

Maybe Tero Subtitler 2.0 will disable support for older macOS to make it clearer what is supported (and simplify testing), but at least Series 1 will be available for them as a better-than-nothing solution (maybe continuing as a branch after 2.0 for a period of time).

If we can get the cpp for certain CPUs on PCs sorted, that would be brilliant. It might be a bit cheeky to ask on Subtitle Edit's forum for help on Tero! ☺️ Do you think you could try compiling on a Windows machine for/with certain CPUs (or whatever)? I don't know if I'm capable, but I can try if there are instructions.

[GitGianluc]

Thanks for your efforts!

Don't mention it!

I'm thinking of waiting a little longer in case M2 Extreme is released (maybe later in the year? Hard to say, but I'm reluctant to purchase now in case it does come).

I don't think it will ever be released: I've read that «Apple is reportedly abandoning plans for a M2 Extreme Mac Pro due to cost and complexity concerns»; that is why, in the end, "only" a M2 Ultra Mac Pro has been announced in the last few days. More likely, the next generation of M3 Extreme (?) / Ultra (?) Macs Pro will appear in 2024. Anyway, do you mean to fly to the moon with such a computer?! 😂

Maybe Tero Subtitler 2.0 will disable support for older macOS to make it clearer what is supported (and simplify testing), but at least Series 1 will be available for them as a better-than-nothing solution (maybe continuing as a branch after 2.0 for a period of time).

That's unpleasant, but reasonable.

Do you think you could try compiling on a Windows machine for/with certain CPUs (or whatever)? I don't know if I'm capable, but I can try if there are instructions.

My knowledge of Redmond's systems goes back (and remains) to Windows XP!
Can this discussion be useful to some extent? By the way, aren't Whisper.cpp executables available for Windows platform?

[chenlung]

That's unpleasant, but reasonable.

I should add it's not a decision for me, I'm just thinking aloud what might end up happening.

My knowledge of Redmond's systems goes back (and remains) to Windows XP! Can this discussion be useful to some extent? By the way, aren't Whisper.cpp executables available for Windows platform?

There are artefacts, but many (presumably over a certain period of time) have expired in terms of download. I don't know if that's where SE got theirs.

[URUWorks]

Great! I'll take your compilation for ARM64 ;)
Also I agree with the suggestions.

[GitGianluc]

Also I agree with the suggestions.

Do you want me to open a new issue, tagged [enhancement], with the 4 improvements suggested above?

[URUWorks]

No need, I already have it ;)

[humble-b]

I am using an Fedora linux on AMD Radeon and am able to compile whisper.cpp. It is only unclear for me where and which file(s) needs to be put for tero subtitler to work and to test if it will work. The wizard doesn't allow to download the needed file so if anyone can tell me how to make it work on linux i can also try this configuration out

[URUWorks]

Hola,
In the next version, you will be able to choose the path of Whisper.cpp 😄