RFC: Split FluidVoice into a shared engine and per-platform shells

[BenjaminPrice]

Summary

Split FluidVoice into one cross-platform engine and thin native shells. The engine takes audio and context in and returns transcripts and enhanced text. Each shell owns everything that touches the user's session: hotkeys, mic capture, the overlay and text insertion.

The engine is a Swift library first and a daemon second. macOS and iOS link it in-process, with no behavior change. Linux runs it as a per-user service behind a Qt shell, and any client can point at a remote engine on a GPU box, which is what #615 asks for.

The migration is six steps, each mergeable on its own. The first three are pure refactors that also make the iOS app cheaper to build.

Questions for the maintainers

These answers decide whether this lands upstream or lives elsewhere, so I'm asking before writing code.

1. Windows. The windows-main branch says the Windows source isn't published, and the windows-v0.0.x releases mention CUDA and Vulkan paths. Is that app Swift? Should it share an open engine, or stay out of scope here?
2. Linux. In [✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615 a collaborator said a Windows/Linux version is coming. Is Linux being built privately too? If so, I'd rather contribute there than duplicate it.
3. iOS. You said iOS comes first. Would an engine target with no AppKit or SwiftUI dependency help that work, or is that split already happening somewhere?
4. Fluid Link. Does the remote design here match what you have in mind, or is there a protocol already?
5. Fluid Intelligence. This proposal keeps it private. It plugs into the engine as a provider, the same way it plugs into the app today. Is that boundary acceptable?

Motivation

Three requests need the same engine, and today that engine is tangled with the macOS UI.

• Remote inference ([✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615). Users want a light client to send audio to a stronger machine on the LAN. Every TranscriptionProvider runs on the client today. The local API already has POST /v1/transcribe and /v1/postprocess, but it binds to 127.0.0.1 and the app can't consume a remote endpoint.
• iOS. A second Apple frontend needs the same ASR, formatting, dictionary and enhancement logic without AppKit.
• Linux. There is no Linux build, and Linux is where many users run their GPU servers.

Files importing SwiftUI or AppKit hold 58% of the 175,064 lines in Sources/Fluid. ASRService (8,453 lines) imports AppKit, CoreAudio and FluidAudio. SettingsStore (6,542 lines) imports SwiftUI, AppKit and ServiceManagement. 44 types are Combine ObservableObjects.

The portable core already exists but has no boundary. About 43k lines import only Foundation or modules with direct cross-platform replacements. That includes LLMClient, the spoken-punctuation and dictation-literal formatting, PronunciationDictionaryStore and fv-serve. Full measurements are in the appendix.

Goals and non-goals

Goals

• One engine that builds on macOS, iOS, Linux and Windows, enforced by CI.
• No behavior change for macOS users at any step.
• One versioned protocol shared by local shells, remote clients and third-party tools.
• Remote inference for [✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615, using the same engine.

Non-goals

• Replacing or restyling the SwiftUI app.
• Opening Fluid Intelligence.
• Shipping a public Windows build. That is your call, given the existing Windows app.
• A big-bang rewrite. Every step merges on its own.
• Full feature parity on Linux on day one. Wayland rules out some features (see Frontends).

Design overview

The cut sits at "audio and context in, events out". Anything that touches the user's session stays in the shell.

macOS and iOS link the engine as a library. The Linux shell and remote clients reach the same engine as a daemon over a socket.

Why capture stays in the shell

• A remote engine can't reach the user's microphone.
• On macOS, an engine running as its own launch agent becomes its own TCC identity and needs its own microphone grant. Capturing in the app keeps one permission prompt.
• iOS can't host a separate engine process at all.

Meetings. The shell captures the microphone and system audio as two tracks. Echo cancellation, transcription and summaries run in the engine.

State ownership

• The engine owns models, dictionary, prompts, history and provider API keys. Keys live where inference runs, so a remote engine keeps them on the server.
• The shell owns hotkeys, overlay style and the audio device.
• One shared engine gives every device the same dictionary and history.

Engine delivery

The engine ships as a library first and a daemon second, because iOS can't run a separate engine process and iOS comes first.

• EngineClient: the protocol every shell codes against.
• InProcessEngine: links the library directly. The default on macOS and iOS.
• RemoteEngine: speaks the wire protocol to a daemon. Used by the Linux shell, LAN clients and [✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615.
• The daemon: a thin network host around the same library.

On macOS, a "serve other devices" setting starts the network host inside the app process. The app keeps a single identity, so there are no new permission prompts.

Inference backends

Apple platforms keep CoreML and FluidAudio, which is where FluidVoice's Parakeet speed comes from. Everywhere else, the engine uses transcribe.cpp, which the app already links for Whisper. Both sit behind the existing TranscriptionProvider protocol.

transcribe.cpp covers the models FluidVoice ships today: Parakeet TDT v2 and v3, Nemotron 3.5 streaming, Cohere Transcribe and Whisper. It runs on Metal, Vulkan, CUDA, ROCm or CPU, and ships Silero VAD in the same library. Apple Speech stays Apple-only.

One change is needed. Its Swift package is an xcframework binaryTarget, and upstream marks Linux Swift out of scope. Off Apple platforms, the engine links it as a systemLibrary target through its C API, which keeps Swift's C++ interop out of those builds.

Protocol and transports

One protobuf schema, one HTTP and WebSocket host, two kinds of listener.

Schema. Messages are defined in protobuf and generated with swift-protobuf for the engine and prost for the Rust shell. One schema keeps the two sides from drifting.

Surfaces

• Session protocol, over WebSocket. Binary messages carry PCM frames and control messages in; partial transcripts, final transcripts and enhancement results come out. Live preview needs this bidirectional stream.
• OpenAI-compatible POST /v1/audio/transcriptions, for batch. This is the shape [✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615 asked for. FluidVoice can call other servers that speak it, and other tools can call the FluidVoice engine.
• Existing local API routes. /v1/health, /v1/transcribe, /v1/postprocess, dictionary, history and meetings move to the engine host unchanged.

Context. Session requests carry app identity, window title, selected text and the active prompt set. Per-app prompts key on macOS bundle IDs today. Linux uses desktop file IDs and Windows uses executable names, so per-app config gets an identity layer with per-OS aliases.

Versioning. A handshake exchanges the protocol version and capabilities: available models, streaming support and enhancement providers.

Listeners

• Local: WebSocket over a Unix socket in $XDG_RUNTIME_DIR (AF_UNIX on Windows).
  • File permissions control access, so local clients need no token. SO_PEERCRED identifies the caller.
  • Web pages can't reach it. A localhost TCP port is reachable through DNS rebinding and CSRF, and this API serves transcripts and history.
  • systemd socket activation starts the engine on first connect, so models load only when someone dictates.
• LAN: opt-in TCP with TLS and a token, and mDNS for discovery.

Speed isn't the reason for Unix sockets. Round trips for a 640-byte frame (20 ms of 16 kHz mono PCM) and a 64-byte reply, 50,000 each (method in the appendix):

Transport	p50 (µs)	p99 (µs)
Unix socket, length-prefixed frames	36	92
WebSocket over Unix socket	42	101
TCP loopback, length-prefixed frames	52	105
WebSocket over TCP loopback	54	113

Dictation sends 50 frames a second, so even the slowest option costs under 1 ms per second of speech. WebSocket framing adds about 5 µs, so one protocol stack on every listener is the better trade.

Engine host and platform notes

The host is built directly on SwiftNIO: NIOPosix, NIOHTTP1 and NIOWebSocket. SwiftNIO runs its unit tests on Windows with Swift 6.3, 6.4 and nightlies. Hummingbird, hummingbird-websocket, swift-nio-ssl and AsyncHTTPClient run no Windows CI, so the host avoids them. The routing logic in LocalAPIRouter ports over.

Linux (Swift 6.4, released September 2026)

• Combine ObservableObjects move to Observation, which ships with Swift on Linux.
• CryptoKit becomes swift-crypto. Keychain becomes libsecret.
• Linux URLSession has no bytes(for:). LLMClient streams with it in two places, so streaming moves to a delegate-based reader.
• The Network-framework listener becomes the NIO host.
• zeppelin-embed links with -force_load, an Apple linker flag. Linux needs --whole-archive.
• The vendored WebRTC AEC3 ships as a macOS-only prebuilt. Its sources are vendored and rebuild with GN, or the engine links the system webrtc-audio-processing.
• nixpkgs has Swift 6.3.3, with 6.4.0 merged to staging.

Windows

• The engine runs as a per-user background process, started at login or by the shell. A Windows Service runs in session 0 under another account, and can't read the user's DPAPI-protected keys or model directory without extra work.
• NIO on Windows has no pipe channels and no working UDP. The engine uses AF_UNIX locally and Windows' DNS-SD API for discovery.
• NIO skips StreamChannelTest on Windows, so the engine's own suite covers half-close and backpressure over TCP and Unix sockets.
• The Swift runtime DLLs ship through the toolchain's MSM merge module. Statically linking Foundation was blocked by ICU conflicts in the latest status I found (August 2025).
• LAN serving over TLS waits for swift-nio-ssl to get Windows CI.
• Fallback: if NIO on Windows falls short, the shell links the engine in-process through a C ABI and passes the same protobuf messages as byte buffers.

Frontends

macOS and iOS keep SwiftUI and the in-process engine. Nothing changes for users.

Linux gets a Qt/QML shell written in Rust with CXX-Qt (KDAB, v0.10.0). The shell talks to the engine only through the protocol, so it never touches Swift.

• CXX-Qt declares typed QObjects in Rust (#[qobject], #[qproperty], #[qinvokable], #[qsignal]) that QML binds to directly. A thread handle queues engine events onto the Qt thread.
• Platform glue comes from existing crates: ashpd for the GlobalShortcuts and RemoteDesktop portals, reis for libei text input, ksni for the tray, pipewire or cpal for capture, interprocess for local sockets.
• LayerShellQt places the overlay as a layer-shell surface through a small cxx shim. That shim is the only C++.
• QML bindings and states sit close to SwiftUI's model, and Qt Quick suits the animated overlay.
• Qt is LGPLv3, which is compatible with FluidVoice's GPLv3.

The same Rust shell builds on Windows if you ever want a public Windows frontend.

Wayland limits. Some macOS features change or drop on Linux:

Feature	Linux status
Global hotkey, hold-to-talk	Works through the GlobalShortcuts portal (GNOME, KDE)
Text insertion	libei through the RemoteDesktop portal on GNOME and KDE; wtype on wlroots compositors; XTest on X11
Hold Space to mute	Not possible on Wayland, where apps can't watch arbitrary keys
Notch-style overlay	layer-shell on KDE, Hyprland, sway and niri; GNOME has no layer-shell
Write Mode context	AT-SPI; weak in Electron and Chromium apps unless their accessibility support is on
Command Mode	A new action set over D-Bus, xdg-open and shell commands

Packaging and distribution

Each platform bundles the engine and a shell, and the engine also ships alone for people who host it elsewhere.

Platform	Engine	Shell	Distribution
macOS	In-process, optional network host	SwiftUI	Unchanged
iOS	In-process, optional remote client	SwiftUI	Your plan
Linux desktop	systemd user service with socket activation	Qt/QML in Rust	Nix flake with an engine module; Flatpak optional
Headless server	Daemon only	None	OCI image with CUDA and Vulkan variants
Windows	Per-user process	Your call	Your call

A Flatpak shell must use portal-based text insertion. ydotool needs /dev/uinput, which the sandbox doesn't grant by default.

Migration plan

Six steps, each mergeable on its own, with no user-visible change until step 5.

1. Extract FluidEngine. A SwiftPM target with no AppKit or SwiftUI dependency, so the compiler enforces the boundary. Move in formatting, dictionary, LLMClient, prompt routing, history and the TranscriptionProvider protocol.
2. Route the app through it. Add EngineClient and InProcessEngine, and route ASRService through them. State at the boundary moves from Combine to Observation.
3. Add Linux and Windows CI. Build FluidEngine with the transcribe.cpp backend on both. This is the step that keeps the engine cross-platform.
4. Add the daemon host. The protobuf schema, the session protocol and the SwiftNIO host. Fold in the existing local API routes and add POST /v1/audio/transcriptions.
5. Add RemoteEngine to the macOS app. This closes [✨ FEATURE] Remote transcription backend with the ability to run the speech-to-text model on a separate machine #615 and gives Fluid Link a base.
6. Build the Linux shell and packaging. Qt/QML in Rust, the Nix flake and the OCI image.

Steps 1 to 3 are refactors that also cut the cost of the iOS app.

Risks and open questions

• Churn. The repo cuts release branches often (40+ B/* branches). Refactor PRs must stay small and land fast, or they rot.
• Protocol stability. Once third parties use the wire protocol, changes need versioning discipline. The handshake and capability list exist for that.
• Two toolchains on Linux. Swift builds the engine and Rust builds the shell. The process boundary keeps them apart, and the Nix flake builds both.
• CXX-Qt is pre-1.0 and breaks its API between minor versions. The shell pins it.
• Wayland fragmentation. Text insertion and overlays differ by compositor. The shell needs per-compositor paths and a clipboard fallback.
• Windows networking. NIO's Windows gaps are listed above. The in-process fallback covers them.
• Ownership. Who maintains the Linux shell beyond its first version? I'll build that version and help maintain it. It could live in this repo, a sibling repo under altic-dev, or a community project.

Alternatives considered

Option	Why not
Port the whole app to Linux with SwiftCrossUI and GTK	Rewrites about 63k lines of views in a toolkit its author calls a work in progress, and still needs the core split
Qt Bridge for Swift for the Linux UI	0.2.0-beta, paused since August 2026. Its run() calls QGuiApplication::exec() with nothing draining Swift's main queue on Linux, so @MainActor code stalls after await unless patched
Go with miqt for the shell	No typed QObjects for QML, and no Go library for the GlobalShortcuts or RemoteDesktop portals or libei
Engine in Rust	Throws away the Swift core, and upstream is a Swift team building iOS next
Raw length-prefixed frames locally	Saves about 5 µs per frame and adds a second protocol stack
Hummingbird for the host	No Windows CI; SwiftNIO has it
Engine as a Windows Service	Session 0 isolation; can't read the user's keys or models

Appendix: measurements and references

Codebase. Sources/Fluid on main, October 2026, with each file bucketed by its imports:

Bucket	Lines	Share
Imports SwiftUI or AppKit	100,935	58%
Apple system frameworks (audio, ML, accessibility)	31,237	18%
Foundation only	26,945	15%
Foundation plus direct replacements (Combine, CryptoKit, os, Darwin)	15,947	9%

About 63k lines of the first bucket are pure view code (UI/, Views/, Theme/ and the app root). The rest are services and state that import AppKit for a few calls.

Transport benchmark. Rust with tokio and tokio-tungstenite on a 2-vCPU cloud VM. Server and client ran on separate OS threads with separate runtimes over real kernel sockets: 5,000 warm-up round trips, then 50,000 measured. A second run matched within about 4 µs. Desktop hardware gives lower absolute numbers with the same ordering.

References

• Discussion #615: Remote transcription backend
• transcribe.cpp
• SwiftNIO and its main CI workflow
• CXX-Qt
• ashpd and reis
• Qt Bridge for Swift
• SwiftCrossUI
• miqt
• GNOME Global Shortcuts portal
• Upcoming changes to Windows Swift SDKs
• nixpkgs: swift 6.3.3 to 6.4.0