Trainspodder mobile/desktop app displays Whisper transcription of BBC broadcast or YouTube video

[NielsMayer]

Using a trivial extension to Whisper ( #228 ) I extended my still under development Qt-based multi-platform app, Trainspodder, to display the Whisper Transcription of a BBC 6 Broadcast. This demonstrates timings and accuracy of Whisper for both radio disk-jockey banter and song lyrics, alongside animated display of other audio features extracted from an online broadcast/podcast.

This video is made directly from the Screen Recording feature of my Android Mobile phone running the Qt6 version of Trainspodder.

https://www.youtube.com/watch?v=37KqNeBurzA

[NielsMayer]

So the video and picture above are the result of a weekend of hacking to get Whisper working with existing Soundcloud Comments and Annotation GUI in Trainspodder ( https://www.youtube.com/watch?v=kEjU5nsaHdg ). It was meant for much sparser set of comments than you'd have for a transcription, thus the clutter of "comment" icons in the overview display.
In the details display, comments are associated with a start time not a time-span or segment -- and it's clear from usage with Whisper that some of the transcription lines are more anchored to the segment end-time rather than start times.

I've now updated the GUI specifically for Whisper such that the details view shows the transcript line anchored to both start and end times. The overview just shows a "bar" representing whether voice activity occurs at a given time (VAD).

The updated interface showing Trainspodder/Whisper interface overlaying a video under analysis looks like:

... Shortly, I'll upload a new video of Trainspodder transcribing for this video. ...

What's interesting here is that in the original video ( https://www.youtube.com/watch?v=RoqgWN-Z3iw ) is about "Qt" which is pronounced "cute" -- however Whisper seems to have gleaned from the context that "cute" refers to "Qt" not the word "cute" -- and that correct transcription of that technical acronym occurs throughout the transcript! Very impressive!!

[NielsMayer]

https://rumble.com/v1n7cx8-trainspodder-and-whisper-transcribes-radio-w-good-proper-noun-spelling-infe.html
or https://odysee.com/@NielsMayer:4/Trainspodder+Whisper_Transcribes_Radio_w_Good_Proper_Noun_Spelling,_Inferring_Blanked_Words,_Lyrics:c

Showing Trainspodder's updated GUI for displaying Whisper Transcriptions as an additional analysis source, this demonstrates some of the features, both bad and good, of using Whisper for free-form transcription of podcasts and broadcasts within Trainspodder.

Trainspodder's own display of segmentation, speech/music segments, beat-events, BPMs, etc also provides good and consistent timing markers against the transcribed text, potentially including speaker-detection for multi-party conversations, esp if the voices are tonally different.

Good and/or Interesting to Note:

• Excellent accuracy transcribing the artist's names mentioned, band names, song names, etc.
• No problem with Patois accents, Reggae lyrics.
• Hallucinating the word "Hell" in a contextually correct place where it was blanked out for radio broadcast (while significantly worse "rap" lyrics weren't)
• Either phonetic or actual transcription of Ghanaian language lyrics.
• Attempting to transcribe rap lyrics and perhaps hallucinating the N-word, from a very unclear portion, but *ing out most of the letters.

Bad:

• One instances where the timing of a song lyric segment -- off by seconds.
• Saxophone solos trigger repeated short-phrase hallucinations. (Noticed multiple times over 10's of transcriptions, so perhaps Whisper has emergent Beatnik consciousness and will eventually go "cool cat" "that's hep" or "don't be a drag" in response to jazzy groovy music.)

FYI, I'm using the following whisper call to process these media files:
'whisper --language English --model medium --condition_on_previous_text False --compression_ratio_threshold 1.8 $*'

The original broadcast being played is at
https://www.bbc.co.uk/programmes/m001cpzt
(Gilles Peterson -- Joining the musical dots: Maya Delilah & Conor Albert)

[NielsMayer]

https://www.youtube.com/watch?v=AFk5g7NJ1Ko

In this particular example, we see the feature of Whisper effortllessly breaking into multilingual translation, including translating a noncaptioned video speech in Chechen-accented Russian by Ramzan Kadyrov that contains content that would not sit well with the majority of the Convo Couch's liberal audience, and probably was not translated before airing or they wouldn't have shown it...

FYI, I'm using the following whisper call to process these media files:
'whisper --language English --model medium --condition_on_previous_text False --compression_ratio_threshold 1.8 $*'

The original broadcast being played is at https://odysee.com/@TheConvoCouch:3/putin-condemns-crimean-bridge-bombing,:1
(Putin Condemns Crimean Bridge Bombing, Russia Strikes Back At Ukraine, Media Hypocrisy)

[forest428-gmail]

I'm a Whisper's rookie, could you help me with the exact whisper command or process of how to take a youtube video and make it into the final captioned video as you showed? thank

[NielsMayer]

1. use youtube-dl to download the video into MP4 file.
2. use my fork of whisper to output .csv file via command-line whisper --language English --model medium --condition_on_previous_text False --compression_ratio_threshold 1.8 --fp16 True <downloaded-mp4-file>
3. Play the file in #Trainspodder which is not yet published as I'm still working on it, but will be open-source and on Github.

[akashAD98]

@NielsMayer this is amazing work