Skip to content

Architecture

nodemode-ai edited this page Oct 29, 2026 · 1 revision

Architecture

fvx_core                   native/fvx-core        the compositing engine, Rust and PyO3: project, commands, renderer, jobs
  -> engine service        backend/app            FastAPI on 127.0.0.1:8250, one engine thread
    -> /api/live, /api/viewer, /api/rpc, /mcp, /api/terminal
      -> interface         frontend/src           TypeScript and Vite, formvis Studio Dark
      -> agents            MCP tools, stdio relay, CLI
desktop shell              src-tauri              Tauri 2, runs the frozen engine as a sidecar

The compositing engine

native/fvx-core is formvfx's compositing engine, written in Rust for formvfx and loaded by backend/app/native.py as the fvx_core Python extension. Its Engine class gives the engine service gestures for one-step drags, engine events, the UI snapshot the interface reads (including property trees), viewer overlays, graph samples, and render jobs that run without the Python lock.

The GPU compositor (wgpu) is built in. At start the engine opens the adapter once in a child process (formvfx_engine.py --gpu-probe, the result cached in gpu-probe.json per engine version), so a broken driver costs the GPU, never the engine; then it attaches the GPU and every composition renders on whichever of GPU and CPU measures faster for it.

  • Model (src/model) · a project of folders, compositions, footage and solids; layers with one property tree each (transform, masks, effects, shape contents, text, audio) whose properties carry match names for paths like transform/position; keyframes in layer seconds with linear, Bezier, hold and roving interpolation, speed and influence eases, and spatial Bezier tangents; markers with chapters, cue points and protected regions; time remapping as a root property.
  • Commands (src/engine/cmd) · a registry of 700 commands (the keyframe assistants, time, markers, dependencies, animation presets, viewer snapshots, editor timelines, project templates, the watch folder, the Footage panel, the Media Browser, background jobs, the in-app tutorials, work-area lift and extract, Scene Edit Detection, Auto-trace, Create Nulls from Paths, Geometry Options, VR environments, the stereo rig, Media Replacement, viewer processing and scopes, modal ScriptUI dialogs and CEP extensions among them) with JSON parameters, undo by project snapshots (repeated drags and typing merge into one step), and engine.batch with $N.key references between steps.
  • Renderer (src/render) · a premultiplied float compositor on the CPU, with a wgpu path (gpu.rs, gpu.wgsl) that warps and blends layers and runs the GPU-capable tail of a layer's effects on the GPU; each composition goes to whichever of the two measured faster, and both match to 1 of 255: affine and camera projections with mipmapped sampling, 39 blend modes, masks with feather, feather points, expansion and their own motion blur, track mattes, adjustment layers, precomps (with Responsive Design protected regions), depth-sorted 3D layers, motion blur, frame blending, Draft and Wireframe quality and the nine layer styles; shapes and masks through tiny-skia, text through rustybuzz and system fonts, footage and audio through FFmpeg.
  • Text (src/render/layout.rs, src/render/textanim.rs) · layout by style runs with OpenType features and variable axes, paragraph wrapping and justification; text animators with range, wiggly and expression selectors, text on a mask path and per-character 3D; the 169 text animation presets compiled in.
  • 3D (src/render/light.rs, mesh.rs, model3d.rs, three.rs) · lights shade 3D layers per pixel after their warp (Classic Phong terms from Material Options, physically based terms for models), soft shadows from layers by ray tests against their buffers and from models by shadow maps, equirectangular environment maps, depth of field from the camera; glTF, GLB and OBJ read here and rasterised with a depth buffer; the Advanced renderer composites a run of 3D layers per pixel by depth, refracting what is behind transparent surfaces. Text and shape layers extrude and bevel into meshes with their own front, bevel and side materials (extrude.rs); reflections trace other 3D layers and models through a bounding volume hierarchy (mirror.rs); MaterialX documents are read into the physically based material (materialx.rs). The 3D views are cameras kept per composition in the engine state (engine/cmd/view3d.rs).
  • Tracking (src/render/vision.rs, analysis.rs, stabilize.rs, camsolve.rs, face.rs, engine/jobs.rs) · image primitives written for formvfx (pyramids, normalised cross-correlation, corner detection, pyramidal Lucas-Kanade with a forward and backward check, dense optical flow, model fitting with RANSAC from translation to homography) read a layer's own frames before its effects. Analyses run on a background thread against a snapshot of the project and land as one undo step when they finish; the Tracker panel polls track.status. Wrap Stabilization keeps its per-frame motion in the effect and plans the correction at render time, so smoothness, method and framing change without a new analysis; the 3D Camera Tracker keeps its tracks and solve in the effect and places the camera and targets from them.
  • Shapes (src/render/shapes.rs, shapeops.rs, svg.rs) · shape contents evaluated item by item with every standard path operator (booleans and offsets through polygon clipping), dashed, tapered and wavy strokes; SVG read into vector footage or shape layers.
  • Roto, paint and puppet (src/render/roto.rs, sam.rs, sam2.rs, ptload.rs, puppet.rs, fill.rs, src/effects/paint.rs, puppet.rs, roto.rs) · Roto Paint keeps its strokes, span and binary mattes (run-length coded) in its effect and refines the edge at render time with a guided filter; segmentation is a colour-model and edge-aware solve (Classic) or MobileSAM, whose TinyViT encoder, prompt encoder and mask decoder are implemented here and read the official checkpoint directly (a small zip and pickle reader in ptload.rs, AVX2 and AVX-512 matrix kernels when the processor has them); propagation warps the last matte by optical flow and segments again. SAM 2.1 tiny (sam2.rs: the Hiera encoder, the feature pyramid, memory encoder and memory attention with rotary positions and object pointers) propagates instead from a memory of the stroked frames and the frames just before, as the reference video predictor does. Paint strokes are groups inside a Paint effect rendered as brush dabs; Puppet meshes the layer's outline on a grid and solves an as-rigid-as-possible deformation by conjugate gradients; Selection-Aware-Fill aligns frames by homographies of the background and fills each frame's hole from the others, then blends the seam in the gradient domain.
  • Colour (src/render/color.rs, ocio.rs, aces.rs, lut.rs, yaml.rs) · the project's pipeline (Pipe): bit-depth quantisation, the working space with Linearize and 1.0-gamma blending, footage profiles, display and output transforms with HDR handling. Every conversion is a list of operators (Proc); the OCIO side reads .ocio files through a small YAML reader, compiles their transforms into the same operators, implements the ACES 1.x output transforms and ships a built-in ACES 1.3 config. LUT files of every common format load into the same operators.
  • Disk cache (src/render/diskcache.rs) · frames that take a while are stored as PNG under a hash of the composition's closure (items, settings, footage and LUT stamps) and the view, evicted oldest first under a size limit.
  • Effects (src/effects) · 306 effects written for formvfx in 21 categories (keying, paint, colour, channel, blurs, distortions, generators, stylize and perspective, noise, time, transitions, simulation, immersive video, text, utility, obsolete, the 3D Channel set on OpenEXR passes, Curl Noise and the 13 audio effects the mixdown applies) with the nine Expression Controls (Stereo 3D Controls among them), plus WebAssembly effect plug-ins and OpenFX plug-ins (effects/ofx.rs: the host's property and suite tables, an instance pool, a crash guard and the overlay draw suite; docs/PLUGINS.md). Effects reach other times, the original buffer, other layers, masks, audio and a footage layer's EXR passes through an FxHost the compositor supplies.
  • Expressions (src/expr) · formvfx's own interpreter for the standard expression language (JavaScript): a parser to a cached AST and an evaluator with vector maths in its operators, the standard expression object model and helpers; errors fall back to keyframed values and are listed by expr.errors.
  • Files (src/engine/export.rs, rq.rs, lottie.rs, src/engine/ecproj.rs, src/render/exr.rs, psd.rs, pdf.rs, src/engine/cmd/formats.rs) · .fvxproj is the native project file (JSON, formvfx.project/1); .ecproj and .ecprojx projects are read and converted, never written. Footage adds image sequences, OpenEXR layers (through the exr crate), PSD documents read and written here (merged, one layer, or a composition with groups as precomps) and PDF or .ai pages (drawn by hayro); footage and compositions take proxies. The Render Queue keeps Render Settings and several output modules per item with built-in and saved templates, renders each frame once for all modules, and writes H.264, HEVC, AV1, ProRes, WebM and GIF through FFmpeg, and PNG, JPEG, TIFF and EXR sequences, WAV and AIFF itself. Lottie is written and read here (.json and .lottie), and Save Frame As writes layered PSD and EXR frames. AEP projects are read by src/render/rifx.rs (RIFX chunks and the text documents' COS dictionaries) and aep.rs (items, layers, property trees, keyframes, effect definitions), then built into the project by src/engine/cmd/aep.rs: file.importAep, a .mogrt through its inner .aep, and .ffx presets on selected layers.

docs/PARITY.md grades it against the reference compositor and lists what it does not do yet.

The engine service

One engine lives on one thread; everything that touches it is queued there. A tick drains engine events, polls background jobs and autosaves. Every surface (the interface socket, HTTP, MCP, the command line) goes through one JSON-RPC dispatcher, so all of them see the same commands and the same permission rules. Frames stream to the viewer as binary messages, latest request first. Live Link (/api/link/*) serves a composition to another app as a live clip: from the open project through the main engine, or from another saved project through a side engine of its own.

The interface

The interface renders engine state and sends commands; it holds no editing logic. A store pulls the UI snapshot when the revision changes and emits granular events, so each panel redraws only what changed. Panels are built once and patched in place; the design files come verbatim from formvis.

Agents

The MCP tools are a thin, well-described layer over commands (plus rendering to images). Permission is a per-boot token for agents in the Terminal panel and an ask-once dialog for the rest.

Clone this wiki locally