Repository navigation
Replies: 3 comments
|
This feel like it should be a plugin, not part of the core. What model to use, what hardware to use. I have ROCM video hardware that runs some models well and others not so well as well as a NPU that would be great for this, but drivers and wiring for this is an ever changing scene and would not be well served by needing to be baked into OBS. OBS should just provide the hooks for such a filter to be possible. |
|
Good instinct, and it is the proposal's own open question — question 3 in the body above: "Is a Swift shim acceptable for the macOS path, or would you rather it stay out of tree as a plugin?" Nobody has answered it yet, so your comment is the first real position on it rather than a duplicate. Two things worth separating, because they pull in opposite directions: What the code already allows. The hooks you are asking for mostly exist in libobs today: a third-party filter gets a source's audio, and What argues for out-of-tree, in your terms. The whole difficulty is the one you name: model choice, ROCm vs NPU vs Apple's recognizer, and driver churn on a timescale of months. That is a bad fit for a repo whose merge gate is CI on three OSes and a merge queue — an in-tree recognizer would pin one vendor's stack into builds for everyone, and every driver change would be a PR here. Where I would not go: deciding it here. The proposal was moved from #753 and the maintainer's decision is on the issue, so this thread should record the argument, not settle it. If you want the out-of-tree shape to be the recommendation, say so plainly in a sentence or two — "ship it as a plugin, keep only the encoder hooks in core" is a position a decision can be taken on, whereas the current comment is an objection without a proposed boundary. One correction for accuracy, since it bears on your hardware case: the CPU numbers in the chart are M1 Max measurements of Apple's on-device recognizer. They say nothing about ROCm or NPU throughput, and the proposal does not claim they do. A ROCm/NPU path would need its own measurement before anyone can say whether it is cheap enough to be a default. |
|
Sorry for the slow reply. Life has gotten in the way of keeping up with this, and I should have said so instead of leaving threads hanging. On the substance, alerque is right: the recognizer should ship out of tree as a plugin, and core only needs the hooks a filter already has. Our working Mac version already runs that way. It's a plugin that shows captions through a text source and needs no new core API. I'm withdrawing the in-core part of this proposal, and #741 closes with it. |
Uh oh!
There was an error while loading. Please reload this page.
Moved from #753 (opened by @chriscoveries). The maintainer's decision is on the issue.
The top half is an illustration, a mock-up rather than a screenshot. The chart is real: measured on an M1 Max, details below.
The idea
Live captions made entirely on the streamer's own machine: speech in, CEA-608 captions out, with no cloud service and no account. On Macs this can run on Apple's on-device speech recognizer, which barely touches the CPU (we believe the work goes to the Neural Engine, but we haven't measured that). Elsewhere, whisper.cpp is the obvious fallback.
What we've built so far
SpeechAnalyzer/SpeechTranscriberon a recorded clip, fed at real-time speed like a live capture. Each finalized word goes into the rust: Add CEA-608 roll-up emitter spike (experimental) #741 encoder. The 608 output is then decoded back to text and compared with a hand-checked transcript.Measured CPU cost
Measured on an M1 Max with the same 30 s clip, as the median of 3 runs. Process CPU time is user plus system, and it includes Apple's speech system services.
base.enbase.enThat's about 47× less CPU than Whisper on CPU. The comparison isn't perfectly like for like: Apple was fed at live speed, while Whisper ran as fast as it could over the file. We also checked the Apple speech daemons during a live run:
anedadded 0.03 s of CPU and the others added none.What's weak
small.endid badly on it (89% WER), so we left it out of the chart.SpeechAnalyzeris new. Windows and Linux would need whisper.cpp, which costs far more CPU (see the table) unless it runs on a GPU.Questions for maintainers
All reactions