-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhancement: Possible to determine different speakers? #9
Comments
You could run this https://huggingface.co/spaces/dwarkesh/whisper-speaker-recognition (see |
I am currently working on a robust way to do this, and there are a few ways to do it, you can use pyannote-audio for diarization. Either you merge the outputs, assign words to speakers by looking up the diarization timestamps. |
I'm very much looking forward to this addition. Thanks for all your work. |
Nvidia NeMo has a tutorial that does speaker diarization and ASR, basically it produces tokens with timestamps which are then grouped according to the RTTM timestamps, diarization timestamps are more accurate than pyannote but their ASR is awful compared to whisper, I'm currently experimenting how to merge alligned word-level transcriptions from whisperX with diarization from NeMo |
@MahmoudAshraf97 ah yes I saw this tutorial -- i didnt know their diarization is better! I will test it on my data -- i thought pyannote was the current best, thank you for letting me know. |
@m-bain I made a minimal example of my idea using my fork of this repo, it saves the words+timestamps using pickle and modifies it to fit NeMo format. https://colab.research.google.com/drive/1fhjGIr_S_vERE9_F1UL033g5shVD-q5K |
What do you think about using speech/source separation models to produce an audio track for each speaker. Then use whisperx to transcribe the result with precise timestamps. Finally put all the transcriptions together and sort them according to timestamp. As pointed out in here whisper generally transcribes only one speaker when there is overlap. I think this approach could be a solution to this problem. |
@Fcabla I am not sure if speech separation is needed unless you have a lot of overlapping speakers. I have good results so far using: Run whisperX and diarization separately. For each word, look if its timestamp lies within a diarization segment, if so, assign speaker label to that word. However this assumes the word timestamps are 100% accurate, which is not always the case due to the current whisperX assumption that whisper timestamps are correct +/- 2 seconds |
@m-bain I agree, for most applications it is not necessary. I am currently using a practically identical pipeline with quite good results. However, when there is some overlapping I have encountered the following two problems. Whisper fails to transcribe several dialogues occurring simultaneously, for a given timestamp there is only one token. However, diarization models can identify multiple speakers for a given timestamp. For example, imagine that the diarization model says that SPEAKER_00 speaks from second 5:00 to second 13:46 and SPEAKER_01 speaks from second 7:35 to second 8:02 -> there is overlap. To whom is assigned the token of the transcript that whisper generates and that starts at second 7:50. It would be amazing if whisper were able to transcribe several tokens occurring at the same time, but currently it is not. Hence the idea of using speech separation as an alternative to diarization. |
@Fcabla I see yes overlapping speech is a difficult problem, probably worth using speech separation only for overlapping segments |
Thank you and I am existed with it, I did a quick run and having the error "ModuleNotFoundError: No module named 'whisperx.diarize'" occurred, will do a few more test later on this week. thank you again. ran another one, facing errors as below, I did use not english file, it might be the reason, will try again . |
Failed to align segment: no characters in this segment found in model dictionary, resorting to original... |
I've tested with Russian audio and everything worked extremely well, apart for Diarization. Even when exact number of speakers is provided, the text is often attributed to the wrong speaker. Let me know if I can support work on Diarization. |
I added the diarization to the Text output from utils.write_txt in a fork,
as extracting/viewing it from the other formats wasn't very easy.
I could add a pull request for a more basic version of that, I made my
version only output the speaker's name when the speaker changes.
…On Tue, Feb 7, 2023 at 12:27 PM Alex Gordon ***@***.***> wrote:
I've tested with Russian audio and everything worked extremely well, apart
for Diarization. Even when exact number of speakers is provided, the text
is often attributed to the wrong speaker. Let me know if I can support work
on Diarization.
—
Reply to this email directly, view it on GitHub
<#9 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAXOL2UZ5WB5A4V6DC6YS3WWKOYPANCNFSM6AAAAAATF4KZRA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hey folks! Thanks! |
Also, how does one add the Hugging Face token in the python implementation? Cheers |
Hey ! you can refer transcribe.py file in whisperx git repo if you want to create your own python script for diarization. |
you can add it in this way: hf token can be obtained from : https://huggingface.co/settings/tokens |
i tried the diarization along with VAD-filter (which gives better results). Thanks @m-bain for adding this. |
Thank you for your script! I've had trouble running NeMo locally, and really want to investigate NeMo as an alternative to Pyannote. So I wanted to try your Collab—but also couldn't run this on Colab Pro. NeMo successfully installs but it chokes when it tries to import the method, can't find it. Have you seen any similar issues? Is this notebook still functioning? Thanks!! |
I have exactly this problem, the error when I try to import. Could someone fix it? Thanks |
@ubanning Try this updated notebook, it's working as expected |
@MahmoudAshraf97 Hello, thanks. |
Fails at #Reading timestamps <> Speaker Labels mapping NameError |
|
Thanks, it was loaded so not sure what was going on, but glitch now resolved. However, no matter what I throw at, even files very easily handled by Pyannote, it always shows one speaker. Can't believe Nemo is this bad, something else has to be going on. |
You might find this better than pyannote on your data: But depends, and ought to be constrained to whisperx sentences, i.e. Appendix Sec. A (page 13) of https://www.robots.ox.ac.uk/~vgg/publications/2023/Han23/han23.pdf |
Co-authored-by: github-actions <github-actions@github.com>
Is it possible to add the functions of determining the different speakers in a conversation and identifying them in the subtitles?
The text was updated successfully, but these errors were encountered: