Real-Time Speech to Text Subtitles Arrive in OvenMediaEngine

[getroot]

Real-Time Speech to Text Subtitles Arrive

We’re excited to announce that real-time speech-to-text (STT) subtitles are now available in OvenMediaEngine.
Powered by whisper.cpp(https://github.com/ggml-org/whisper.cpp), this feature automatically generates subtitles in real time from live audio streams.

Currently, subtitles are supported only in Low-Latency HLS (LLHLS). You can test it now using the latest Master branch. For best performance, NVIDIA GPU acceleration is recommended.

For setup instructions and detailed configuration, please refer to the documentation:
https://docs.ovenmediaengine.com/dev/subtitles/realtime-speech-to-text

---

Player Compatibility

LLHLS subtitle support has been recently added to OvenMediaEngine. #1916

OvenPlayer uses hls.js for LLHLS playback, but there is a known issue in hls.js that prevents subtitles from rendering correctly.
video-dev/hls.js#7522

Until that issue is resolved, we recommend testing LLHLS subtitles using ShakaPlayer, which works properly.
https://shaka-player-demo.appspot.com/

---

Any feedback or testing results would be greatly appreciated.
This is still an early version, so if you have experience using hardware acceleration, please share it with the community — it will be helpful for everyone.

[mlx-solutions]

very cool was actually looking into how to use whisper myself, but you beat me to it 👍

I'm not really looking for subtitles in playback- althrough thats ok to have - Im more looking for a having a transcribe hook, so the same subtitle info is also send over https call to a url provided in config. this way I can implement my own usage of the transcription.

[mlx-solutions]

Setup

I setup GPU server (Nvidia L4), and build OME (not using docker). ran the prerequisite script to install whisper. installed nvidia driver, downloaded the model(s). configured Server.xml to use hardware, setup my transcoding hook added the code changes need to build outputprofile with hardware encoding and subtitles

Outcome

• Result is that the server can stream using the GPU 👍

• The Whisper model loads 👍

• But the transcription does not seem to start 👎 Specifically i'm getting Error Could not find input audio track or output subtitle track for transcription. AudioIndexHint(0), SubtitleLabel(Origin)
  See attached ouputprofiles.json which is what my transcode hook returns.
  Any idea what I did wrong ?

Note about GPU encoding

even on the latest Nvidia L4 GPU's the encoding quality is just not good enough, I still need to use software encoding usable quality. I cannot use GPU as Origin.
Im hoping for a setup where Whisper can run as a standalone service/server and we just duplicate and push the ingress stream to it, offloading the GPU work to separate server, and it then calls a transcribe_hook callback with the VTT.
Running GPU's servers and only using their CPU for encoding, thats just way to expensive.

[getroot]

I'm sorry for the late reply, I've been really busy.

Regarding this error:
Could not find input audio track or output subtitle track for transcription. AudioIndexHint(0), SubtitleLabel(Origin)

I couldn’t reproduce the issue on my side. Could you please share the full log? I think I’ll need it to reproduce the situation properly.