Limiting previous token

[amnike]

Every time a new result is created it captures the previous result and as this goes on it gets to the point it's capturing the previous 50 results then the processing time increases tremendously lol.

I get that it's done like this for accuracy but, do you have a way or can point me in the direction of reducing the previous tokens or results captured?

I'd really appreciate it

[jongwook]

You could truncate the prefix tokens by adding something like

decode_options["prompt"] = decode_options["prompt"][-10:]

after:

whisper/whisper/transcribe.py

Line 180 in 0f39c89

decode_options["prompt"] = all_tokens[prompt_reset_since:]

That said, prompt tokens affect the inference time only during the first forward pass through the decoder, which is usually not very significant compared to the total autoregressive decoding time which usually involves tens or hundreds of forward passes through the decoder.

[amnike]

Thanks a lot! Any tips on reducing that super lag 15 minutes into a "real-time" transcription?

[jongwook]

In case you're referring to #608, I can't comment because I wasn't involved. This repo currently does not support real-time transcription, except an idea I suggested in #117 (comment). In any case, you'd need a fast GPU to run the transcription faster than the audio duration.