Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WhisperX implementation #71

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

WhisperX implementation #71

wants to merge 1 commit into from

Conversation

JonasCz
Copy link
Contributor

@JonasCz JonasCz commented Dec 30, 2023

This adds WhisperX, useful to me because it's faster and has word-level timestamps.

(Word level timestamps are used here in order to "regroup" subtitle lines more logically, e.g. each sentence gets its own subtitle line if possible, and otherwise lines are split in a more sensible way instead of the pretty "random" output of base Whisper).

This probably:

Closes (sort of) #53
Progress #58 (WhisperX supports diarization, although this doesn't do that)

In order to use it, you have to:

  • Add WhisperX to transcription-api/requirements.txt instead of faster-whisper
  • Replace call to FasterWhisperBackend in transcribe.py with WhisperxBackend

The code is quite messy, it's "minimal effort" adapted from one of my other projects.

@pluja
Copy link
Owner

pluja commented Dec 30, 2023

Hey, thanks for this. I think I will work based on your implementation on a better integration. Also, note that Faster-Whisper also has word-level timestamps, but Whishper is not using them (although if you get the JSON subtitles you'll see the word-level distinction).

The idea is that the API should be able to support multiple backends so that the user can choose between WhisperX and Faster-Whisper.

I think I will be experimenting with this, but if WhisperX is better than Faster-Whisper as it seems, and can also run with CPU, maybe it would be better to switch to it by default instead of faster-whisper.

I'll be looking at this and come back here when I have experimented with it.

I'd also like to hear what you (and any other user) thinks!

pluja added a commit that referenced this pull request Mar 6, 2024
@wobsoriano
Copy link

Would love this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants