Skip to content

docs: add RFC 0001 for native desktop rewrite (Rust + egui/wgpu)#90

Merged
natashaannn merged 2 commits into
mainfrom
docs/rfc-native-desktop-rewrite
Jul 12, 2026
Merged

docs: add RFC 0001 for native desktop rewrite (Rust + egui/wgpu)#90
natashaannn merged 2 commits into
mainfrom
docs/rfc-native-desktop-rewrite

Conversation

@natashaannn

Copy link
Copy Markdown
Member

What this is

Docs-only PR — adds docs/rfcs/0001-native-desktop-rewrite.md, no code changes. It proposes replacing our Remotion-based render pipeline with a native Rust desktop app to fix two confirmed, architectural problems:

  1. Remotion's headless-Chromium render is a hard bottleneck (2-5fps, no GPU path) — this is not fixable within Remotion.
  2. Our hardware/encoder detection is a stub with no real cross-platform GPU consistency (Windows+NVIDIA silently gets zero hardware acceleration today).

Full reasoning, alternatives considered, and a phased build order are in the RFC. This PR is to get both of you to read it and weigh in before we commit engineering time.

What I need from you two specifically

Please read the full RFC, then give a thumbs up/down (or a counter-argument) on each of these — they're the decisions that actually commit us to a direction:

  • Language: Rust over C++. The call was made factoring in that we're JS/Java-strongest and will lean heavily on coding agents — Rust's compiler turns most agent mistakes into a compile error instead of runtime undefined behavior. C++ has much stronger prior art for this category (Shotcut/Kdenlive/MLT, whisper.cpp links natively either way). If either of you has a strong C++ pull, raise it here — this is the highest-leverage decision in the doc.
  • GUI: native (egui + wgpu), not wrapping our existing Next.js timeline editor. This throws away reuse of the current custom canvas timeline editor in favor of one compositor shared between live preview and final render. Confirm you're OK eating that rewrite cost for the parity guarantee.
  • Old codebase stays frozen to critical-fixes-only while the new one is built in parallel. Means no new features land on the current pipeline once this starts in earnest — confirm this is acceptable against our release roadmap.
  • Build order: render engine first → whisper-rs transcription → rest of pipeline → GUI last. Sanity-check this sequencing against what we actually need next for episodes.
  • Three open questions in the RFC still need a call: new repo vs. subdirectory of this one (I lean separate repo), direct FFmpeg bindings vs. gstreamer-rs, and whether to carry over our "Agent Implementation Convention" (implementation doc + per-step commits) into the new repo.

Also flagged in the RFC as reading material only (not a dependency): gausian-AI/Gausian_native_editor, specifically its native-decoder crate's VideoToolbox/GStreamer decode loops — useful reference, but its license is contradictory (Apache-2.0 file vs. MPL-2.0+commercial claimed in the README) and roughly half its GPU-integration code is dead/decorative, so don't treat it as vetted — no action needed, just context if either of you goes looking at it.

Suggested next step

Comment inline on anything you'd change, or approve if you're aligned — once we have agreement on the checklist above, I'll turn the build-order section into implementation-guide docs and we start Phase 1 (render engine).

Proposes replacing the Remotion-based render pipeline with a native
Rust desktop app to fix the confirmed Chromium/Puppeteer render
bottleneck and cross-platform hardware inconsistency. Covers
language choice, GUI strategy, JSON-schema interop boundary, and a
phased, non-blocking migration plan alongside the current pipeline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers the most likely first question reviewers will ask before
engaging with the Rust-vs-C++ decision: explains why the render
bottleneck can't be fixed by staying in the current stack, and why
that necessarily narrows the options to a compiled/native language.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@victoria-lo victoria-lo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

@missabawse missabawse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMT, Thank you for the detailed RFC!

but I would think we should do a quick spike with Rust to see if we are ok with the dev of it. Claude suggested "port the AudioSyncer FFT correlation function to Rust (self-contained, well-defined I/O, no GPU), verify output matches on all three target machines, and get a real feel for the agent-driven Rust development loop.

@natashaannn

Copy link
Copy Markdown
Member Author

LGTMT, Thank you for the detailed RFC!

but I would think we should do a quick spike with Rust to see if we are ok with the dev of it. Claude suggested "port the AudioSyncer FFT correlation function to Rust (self-contained, well-defined I/O, no GPU), verify output matches on all three target machines, and get a real feel for the agent-driven Rust development loop.

Yea that sounds like a good idea, I've been meaning to do a spike too. Will do on separate repo and then we can test on all our machines and see how it goes!

@natashaannn

Copy link
Copy Markdown
Member Author

Tracked in #93 — a self-contained spike to port the AudioSyncer FFT cross-correlation to Rust, validate output parity on Mac M2 / Mac M3 / Windows+NVIDIA, and get a real feel for the agent-driven Rust dev loop before committing to Phase 1 of the rewrite.

@natashaannn

Copy link
Copy Markdown
Member Author

Spike issue created in #93 , to work on spike and come back to amend this RFC. Merging RFC in and closing this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants