Repository navigation
Replies: 3 comments
Round 1 — landscapeDirective (as understood)Discussion #842 (zackees) asks for research on supporting "DLSS5" / AI enhancement in OBS: make a poor webcam look good and, by extension, improve microphone audio, and decide how to integrate an AI pipeline that runs effect plugins on video/audio. Constraints: the effects should be among the default selectable plugins; model files need not ship in the installer, but there should be a cache system for downloaded models. No explicit success criteria; the expected output is a research outcome and recommended direction, not code. The discussion had no other comments. Current state
Open questions
Research plan
|
Round 2: options and strategiesSeven researchers each covered one of the Round 1 topics. Points marked (unverified) come from a researcher's background knowledge, not a checked source, and need checking before we rely on them. What we found1. DLSS 5 can't do this, but Maxine can (NVIDIA only).
2. Our existing
3. Inference runtime: ONNX Runtime through
4. Models with licences we can redistribute.
5. Model cache design.
6. How the effects fit into OBS: new filter plugins, no change to the libobs ABI.
7. Lessons from prior art and packaging.
Everything we could do
StrategiesA. NVIDIA fast path. Extend
B. Vendor-neutral Rust stack.
C. Smallest safe start. CPU-only effects with tiny models (DeepFilterNet, MediaPipe segmentation, light denoise) on a pure-Rust runtime, with no downloads in v1.
D. Hybrid. Strategy B's framework, with Maxine as an optional backend behind the same filter ids.
E. Extension pack. Keep the core small, and put the runtime and models in an "AI Pack" that registers the effects when installed.
Open questions
|
Round 3: recommended strategyThis combines strategies B, C and D from Round 2.
Architecture
Phases (each one is shippable and RED then GREEN)
Why this combination
Decisions needed before phase 2
If this direction works for you, I'll file phase 1 as an issue (DeepFilterNet mic denoise, |
Uh oh!
There was an error while loading. Please reload this page.
This is a general question and research topic of what do we want to do to support DLSS5. I think that this could be a huge boon because somebody can have a crappy video cam and now suddenly they can look really, really good. The same thing can happen to, let's say, the microphone. The larger discussion here is how do we integrate an AI pipeline and connect to it so that we can run these plugins effects and make the user look good and also make this part of the default selectable plugins. We don't have to have the models downloaded but at least have it in a cache system.
All reactions