-
Notifications
You must be signed in to change notification settings - Fork 0
DistributedRendering DevelopmentPlan
Multi-session implementation plan for a master/worker rendering cluster on top
of the existing Server/ + Client/ stack. Goal: cut wall-clock time for
expensive Mandelbrot renders (Bird of Paradise, deep zooms, 8K posters,
multi-minute videos) by sharding work across LAN nodes while keeping full
fidelity and lossless output.
Anchor commit at doc creation: c759214 on feature/cross-platform-full.
In scope
- One Master Server (orchestrator + ffmpeg merge node).
- N Worker Servers (rendering nodes, formerly "slaves" — use "worker" in code and UI per modern convention).
- mTLS on every link (worker→master, client→master, admin→master). LAN only; no WAN-facing endpoints.
- Render-job lifecycle: submit → ack + RenderId → poll → fetch.
- Tiled sharding of image, poster, slideshow-frame, and video-frame jobs.
- Reuse the existing
RenderRequestDto/RenderResponseDto/ChunkDtowire format wherever possible; addTile*andJob*envelopes alongside. - Lossless artifact transport (existing chunk-stream + SHA-256 path).
- Admin UI in
UI.Avalonia/for cluster status, worker control, job control.
Out of scope (this iteration)
- WAN / Internet operation. No NAT traversal, no STUN/TURN, no relay.
- Heterogeneous-OS GPU coordination beyond what
Rendering.Skia/Rendering.Silkalready give us. Workers expose what they have; the master does not try to virtualise across vendors. - WinForms UI changes (CLAUDE.md: WinForms is deprecated, Avalonia-only).
- Re-platforming the existing single-server protocol. The single-server path stays operational; distributed mode is additive.
┌──────────────────────────┐
│ Avalonia Admin UI │
│ (master-attached) │
└──────────┬───────────────┘
│ mTLS JSON-RPC
▼
┌────────────┐ mTLS ┌──────────────────────────┐ mTLS ┌────────────┐
│ Client │──submit──▶│ Master Server │◀──register─│ Worker N │
│ (FFClient)│◀──poll────│ - job queue │ │ (FFServer │
│ │ │ - tile planner │ │ extended)│
│ │ │ - merge / ffmpeg │──tile job─▶│ │
└────────────┘ │ - artifact cache │◀──chunks───│ │
└──────────────────────────┘ └────────────┘
▲
│ mTLS register (push/pull)
│
(more workers…)
-
Master wraps a new
FFMasterclass that hosts both (a) the client-facing protocol (extends today'sFFServer) and (b) the worker-facing registration + dispatch protocol. -
Worker is
FFServerfrom today plus a small client component that dials the master and registers itself (FFWorkerAgent). -
Client stays mostly the same. The only change is that the
RenderResponseDtomay carryJobId+Statusinstead of bytes; the client pollsjob.statusand finallyjob.fetch.
- Master: long-lived, single instance, binds two TLS listeners (or one listener that demuxes by client cert role). Recommend two listeners on separate ports — simpler, lets cert role be the listener identity.
- Worker: long-lived. Outbound TCP to master keeps a session open; bi-directional JSON-RPC over that one socket. Worker NEVER listens — this removes the inbound-firewall requirement on every node and naturally scopes attack surface to "the master".
- Client: short-lived per UI session (today's behaviour).
Three client-cert roles, distinguished by an OID extension or by separate issuing CAs:
-
role=worker— may callworker.register,worker.heartbeat,tile.deliver. Cannot calljob.submit. -
role=client— may calljob.submit,job.status,job.fetch,job.cancel. Cannot call worker RPCs. -
role=admin— superset of client, pluscluster.*,worker.kill,worker.quiesce,master.config.*.
Role parsing reuses ServerCertLoader.BuildClientValidator — add a
post-validation hook that reads the OID and routes the dispatcher.
Reuse Server/Wire/JsonRpcFraming.cs + MessageEnvelope. New methods:
| Method | Purpose |
|---|---|
worker.register |
Announce capabilities, hardware, supported fractal types, max tile size, GPU/CPU mix. Returns a WorkerId (UUID, persists across reconnects via cert thumbprint). |
worker.heartbeat |
5 s cadence. Carries current load: tiles in flight, CPU/GPU %, temp, free RAM. Master times out a worker at 3× missed beats. |
tile.deliver |
Streams ChunkDtos back keyed by (JobId, TileId). Reuses existing chunk path + SHA-256. |
tile.error |
Reports a tile that crashed, ran past deadline, or hit a guard rejection. Master retries on a different worker. |
JSON-RPC has no built-in server push, so a worker session opens with a
half-duplex "pull-loop": after worker.register, the worker issues
tile.next and blocks (with a server-side long-poll, e.g. 30 s) until the
master returns a tile job or a timeout. Standard pattern; avoids inventing
a second framing.
| Method (worker calls) | Purpose |
|---|---|
tile.next |
Long-poll for the next assigned tile. Returns TileJobDto or a wait-again signal. |
tile.ack |
Worker confirms it accepted the tile; arms the master's per-tile deadline. |
| Method | Purpose |
|---|---|
job.submit |
Same payload as today's RenderRequestDto, plus optional priority, tilePreferenceHint. Returns JobId immediately. |
job.status |
Poll. Returns {state, progress, tilesTotal, tilesDone, tilesInFlight, etaSeconds, errors[], artifactReady}. |
job.fetch |
Streams the merged artifact (existing chunk path). Server keeps the artifact for ArtifactRetentionMinutes (default 60). |
job.cancel |
Aborts pending tiles, frees workers. |
| Method | Purpose |
|---|---|
cluster.status |
All workers, all in-flight jobs, queue depth. |
worker.quiesce / worker.resume
|
Drain a worker for maintenance. |
worker.kill |
Force-disconnect a misbehaving worker. |
master.config.get / master.config.set
|
Live config edits, persisted. |
job.list / job.cancel (admin) |
Any job, any user. |
public sealed class JobAckDto {
public string JobId { get; set; } = "";
public int TileCount { get; set; }
public long EstimatedBytes { get; set; }
}
public sealed class JobStatusDto {
public string JobState { get; set; } = ""; // queued|planning|rendering|merging|ready|failed|cancelled
public int TilesTotal { get; set; }
public int TilesDone { get; set; }
public int TilesInFlight { get; set; }
public double ProgressPercent { get; set; }
public long ElapsedMs { get; set; }
public long? EtaMs { get; set; }
public bool ArtifactReady { get; set; }
public string? FailReason { get; set; }
}
public sealed class TileJobDto {
public string JobId { get; set; } = "";
public int TileId { get; set; }
public int OffsetX { get; set; }
public int OffsetY { get; set; }
public int Width { get; set; }
public int Height { get; set; }
public RenderRequestDto Render { get; set; } = new(); // tile-scoped render request
public int DeadlineSeconds { get; set; }
public int Attempt { get; set; }
}
public sealed class WorkerRegisterDto {
public string WorkerName { get; set; } = "";
public string OsPlatform { get; set; } = "";
public string CpuModel { get; set; } = "";
public int LogicalCores { get; set; }
public long TotalRamBytes { get; set; }
public List<string> Gpus { get; set; } = new();
public List<string> SupportedFractalTypes { get; set; } = new();
public int MaxConcurrentTiles { get; set; }
public int PreferredTilePixels { get; set; }
}- Partition the output rect into N×M tiles. Default tile target ≈ 512×512 px;
the master adapts per worker's
PreferredTilePixels. - Each tile is a full
RenderRequestDtowith a sub-rect — the existing calculators already accept centre+zoom; the master translates tile offsets into(centerX, centerY, zoom)deltas usingEngine/Calculators/MandelbrotCalculatorcoord math. -
Caveat: perturbation-based deep zoom (
SeriesApproximation) shares a reference orbit across the whole image. Tiles must either (a) recompute the ref orbit each (cheap once, wasted N×) or (b) the master computes the ref orbit once and ships it insideTileJobDtoas a binary blob. Option (b) is correct long-term; option (a) is fine for v1. - Merge: master holds a
MemoryMappedFilefor the final RGBA buffer, each tile writes into its rect, atomic on completion. PNG encode runs once at the end viaPngSequenceWriter/Skia.
- Each slide is an independent render job. Trivial map/reduce — one tile per slide is fine for v1; subdivide per-slide later if a single slide is the long pole.
- Frame-level sharding is the unit. Per-pixel tile sharding inside a frame is an optional v2 escalation (only useful for 8K+ frames).
- Master plans the zoom keyframe schedule via the existing
IVideoZoomControllerand assigns frame ranges (e.g. frames 0..29 to worker A, 30..59 to worker B). Workers return per-frame PNGs (existingPngSequenceWriteroutput). - Master ingests frames into
Engine/Imaging/FfmpegEncoder.cs/FfmpegVideoWriter.cs— must be sequential, so the master gates encoder input on frame N being present before consuming frame N+1. - Lossless presets ("ffv1", "h264hq", "h264") already exist in
BatchOptions. No change.
- Audio analysis stays on the master (the audio source is a single file). Frame plan is computed master-side, then sharded as per video.
submit
│
▼
queued ──admin cancel──▶ cancelled
│
▼
planning ← tile planner runs; computes N, picks workers
│
▼
rendering ─tile fail (retry budget exhausted)─▶ failed
│
▼
merging ← all tiles delivered; ffmpeg / png compositor runs
│
▼
ready ← artifact on disk; client may fetch
│
(TTL expiry)
▼
evicted
Persistence: %APPDATA%\FracturingFog\master\jobs\<jobid>\ holds
-
request.json(originalRenderRequestDto) -
plan.json(tile list + assignment history) -
tiles/<tileid>.bin(tile payload from worker, raw RGBA or PNG) -
artifact.<ext>(final merged output) -
events.ndjson(state transitions, errors, worker swaps — for the troubleshooting requirement)
Crash recovery v1: on master restart, jobs in rendering or merging are
marked failed with reason "master-restart". v2 can resume.
The user explicitly asked for security suggestions. Recommendations:
-
mTLS everywhere — already in place. Keep
RequireTls13 = truein the master config for the cluster CA; client UI cert may stay TLS 1.2+1.3 for compat. Default config ships TLS 1.3. -
Cert role separation — distinct CA per role (workerCA, clientCA,
adminCA). Reuses
ServerCertLoaderinfrastructure; just three trust stores instead of one. Compromise of a worker cert cannot submit jobs. -
Worker cert pinning —
ServerConfig.AllowedClientThumbprintsalready exists. The master persists a per-worker thumbprint after first registration; subsequent reconnects must match. -
No remote code paths — the existing
FractalTypeAllowlistalready refusesUserEquation,UserBulb,Sandbox. Keep this guard on the master and re-check on the worker. Defense in depth. -
Payload validators —
RegionPayloadValidator,ThemePayloadValidator,WatermarkPayloadValidatorare already enforced. Make the master apply them once atjob.submit, then trust the shardedTileJobDto(the master is authenticated to the worker). -
Rate limiting — extend
EndpointRateLimiterto a per-role policy. Client-role: existing per-IP per-minute. Worker-role: lower bound ontile.nextlong-poll churn. Admin-role: unlimited but log every call. -
Bind interface — default master
BindAddress = 127.0.0.1per existing convention. Operator must explicitly set the LAN address. Refuse0.0.0.0unless an explicit--allow-any-interfaceCLI flag is passed; print a stern warning on startup. -
mTLS revocation —
RevocationCheckMode = "offline"once a real CRL pipeline exists; "none" stays the default for the self-signed bundle. -
Artifact eviction —
JobArtifactRetentionMinutesconfig field. Cleanup loop runs at startup + every minute. MirrorsWorkDirSweeper.cs. -
Audit log —
Logging/SessionLogger.csalready writes per-session NDJSON. Add a parallel cluster log on the master that records(jobId, workerId, tileId, event, ts)events. Asks satisfied: "Logging should provide detailed render request information for troubleshooting". -
Sandboxed worker work-dirs —
Server/WorkDirSweeper.csalready isolates per-job. KeepOutputNamederivation server-side; never trust a clientoutputNamecontaining path separators (existing guard). -
DoS surface — the existing connection gate + queue gate apply
per-listener. The master needs a separate
MaxJobscap and aMaxBytesQueuedcap so a flood of poster requests cannot OOM the merge buffer.
-
Avoid base64 on the hot path. Today's
ChunkDto.BytesBase64adds 33 % overhead and CPU on both ends. For the worker→master tile transport, add a binary framing option:MessageEnvelope.PayloadKind = "binary", length-prefixed raw bytes after the JSON header. JSON-RPC stays for control messages; the bulk path is binary. This is the single biggest win for large posters and video frames. - Tile output format. Workers emit raw RGBA (4 bytes/px, no encode cost) for image tiles; the master encodes once at merge time. PNG encode of a 16K poster on a worker is wasted work the master must redo.
- Compression. Optional LZ4 on tile payloads — fast, lossless, often 2–4× on smooth gradient regions of fractals. Worth a feature flag.
- Memory-mapped merge buffer. Already mentioned; keeps master RAM bounded to one full output buffer per active job regardless of tile count.
- Adaptive tile sizing. Track per-worker median tile time; rebalance tile size so each worker finishes in ~1–3 s. Avoids stragglers.
- Work-stealing. If a worker's last tile is still going when the queue empties, allow other workers to "steal" half its remaining rows. Cheap re-submit on the master side.
-
Reference-orbit caching for perturbation. Compute once on the
master, distribute via
TileJobDto.ReferenceOrbitBlob. Saves the N-tile redundancy mentioned in §4. -
TCP tuning.
NoDelay+KeepAlivealready on. AddSO_RCVBUF/SO_SNDBUFraises to 1 MB on the worker↔master sockets for sustained throughput. -
Video frame pipeline backpressure. Workers must not race ahead of
the master's ffmpeg consumption. Master assigns frame ranges, and
throttles by holding
tile.nextlong-polls when its frame queue depth exceeds a threshold (e.g. 64 frames buffered). -
Avoid double-decode of region JSON. The master parses
regionJson/themeJsononce; pass a parsed binary form inTileJobDtoto skip per-tile JSON re-validation on the worker.
New view tree under UI.Avalonia/Views/Cluster/:
-
ClusterDashboardView— workers grid (name, OS, GPU, RAM, current load, tiles in flight, last heartbeat, uptime). Live updates over the existing admin connection. Red/yellow status — colourblind: use yellow (#FFCC00) not red for problem states per the project's standing accessibility note. -
JobListView— active + recent jobs. Per-row: submitter, type, progress bar, ETA, cancel button. -
JobDetailView— tile map (visual grid showing per-tile status: pending/inflight/done/failed, coloured by worker), live events log, per-worker contribution bar. -
WorkerDetailView— capabilities, history, button row: quiesce, resume, kill, force-reload-config. -
MasterConfigView— extends the existingServerAdminView/ServerAdminViewModelto add cluster-only fields (MaxJobs,ArtifactRetentionMinutes,TileTargetPixels, per-role rate limits).
All views are MVVM (the existing pattern); the view-model talks to
FFAdminConnection (a thin wrapper over FFClientConnection that uses
the admin cert).
Each phase is a separate session-sized milestone. Earlier phases unblock later ones; do not parallelise across phases without explicit need.
- New cert roles, OID parsing, dispatcher routing.
-
FFMasterskeleton: extendsFFServer, adds worker listener and an in-memoryWorkerRegistry. -
FFWorkerAgentinServer/Cluster/(worker side): outbound connect, register, heartbeat loop,tile.nextlong-poll harness (no real work yet — returns "wait" forever). - Headless smoke: one master + two workers register, heartbeat, gracefully disconnect.
-
Exit criteria:
Server.Testscovers register + heartbeat + cert role refusal.
- New
JobIdgenerator (cryptographically random 128-bit, base32 string). -
job.submit,job.status,job.fetch,job.cancelon the master. - Tile planner for image renders;
TileJobDtodispatch overtile.next. - Memory-mapped merge buffer; PNG encode at completion.
- Client-side: extend
FFClientConnectionwith the new methods; add a polling helper. -
Exit criteria: a
--batch --remote --clusterrender of a non-trivial image (e.g. 8K Bird of Paradise at default iter budget) completes across 2 workers and matches a single-worker render byte-for-byte (or pixel-identical after a deterministic tie-break).
-
PayloadKind = "binary"framing. - Raw RGBA worker tile output.
- Optional LZ4.
- Adaptive tile sizing, work-stealing on the last 10 % of tiles.
- Exit criteria: 4-worker scale test shows ≥ 3.2× speedup over single-worker on a Bird-of-Paradise 8K render (≥ 80 % parallel efficiency).
- Frame-range planner.
- Master-side ffmpeg sequential ingest with backpressure.
- Slideshow per-slide sharding.
-
Exit criteria: 20-second 1080p30 zoom render at
lossless=ffv1completes across 2 workers, output passesffprobeparity vs. a single-worker render.
- All views from §8 live and wired to the master.
- One-click quiesce + resume of a worker. Force-cancel of a stuck job.
- Live log tail panel.
- Exit criteria: the UI can drive an end-to-end render with no CLI involvement, and can cleanly recover from a worker dying mid-job (kill the worker process during a render; UI shows the tile retry; final artifact is correct).
- Crash recovery (resume
renderingjobs after master restart). - Reference-orbit caching for perturbation.
- Per-role rate limiting.
- Operational doc (
Docs/User/Distributed-UserGuide.md). - Stress tests in
Server.Tests: 50 concurrent client connections, 8 workers, 200 queued jobs.
| # | Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|---|
| 1 | Perturbation/reference-orbit divergence across tiles produces visible seams | Med | High (fidelity is a stated goal) | Master computes single ref orbit, ships in TileJobDto; pixel-parity test in CI. Fall back to per-tile orbit only behind a flag. |
| 2 | Base64 tile transport saturates LAN before CPU saturates | High | Med | Phase D-3 binary framing. Do not ship D-2 to users without D-3. |
| 3 | Video frame backpressure deadlocks (master full, workers idle, ffmpeg slow) | Med | High | Bounded master buffer + worker long-poll throttle. Explicit deadlock test in D-4. |
| 4 | Mixed-OS workers (Win + Linux) produce subtly different pixels | Med | High | Pin to one calculator path; MandelbrotCalculator is portable C# with no JIT-FP-mode differences in our experience, but add a deterministic-output test that hashes output across OSes in CI. |
| 5 | mTLS cert sprawl operationally painful (3 CAs + per-worker pins) | High | Med | Ship a --issue-worker-cert CLI on the master; admin UI button "Approve pending worker" that signs + persists the thumbprint in one click. |
| 6 | Master single point of failure | High | Med | Documented as an explicit non-goal of v1 (no HA). Master state on disk + fast restart is the answer. |
| 7 | Worker quietly returns wrong-coord tiles after a code-version skew | Low | Critical | Include a ProtocolVersion + EngineBuildSha in worker.register; master refuses workers whose engine SHA does not match its own. |
| 8 | A misbehaving client floods job.submit and exhausts master disk |
Med | Med |
MaxJobsPerClientCert quota + disk-watermark check before accepting. |
| 9 | UE editor / user-authored fractals slip through to workers | Low | High |
FractalTypeAllowlist already blocks at the protocol layer; enforce identically on both master and worker (defense in depth). Already partially in place. |
| 10 | Long-poll tile.next exhausts master thread pool |
Med | Med | Use async + a single shared dispatcher channel; never a thread per worker. |
| 11 | Tile retries cause non-determinism (one worker's float results vs. another's) | Low | Med | Pin <RuntimeHostConfigurationOption Include="System.Runtime.Numerics.FloatToStringRoundtrip" Value="true" /> (already default in net10); plus the deterministic-output test from risk #4. |
| 12 | The admin role being too powerful is itself a risk | Med | Med | Separate admin cert, never auto-issued. Admin UI requires explicit cert path at startup — no "remember me". |
This is the layout we'll land in over phases D-1 through D-6. Paths relative to repo root. Existing files are amended; new files are marked.
Server/
FFServer.cs (amend: role-aware dispatch hook)
Cluster/
FFMaster.cs (new) — orchestrator
FFWorkerAgent.cs (new) — worker outbound + long-poll
WorkerRegistry.cs (new) — in-memory + thumbprint pins
JobStore.cs (new) — on-disk job state
TilePlanner.cs (new) — image/video sharding
TileDispatcher.cs (new) — channel-based worker fanout
ArtifactMerger.cs (new) — RGBA mmap merge + PNG encode
VideoFramePipeline.cs (new) — sequential ffmpeg ingest
Protocol/
JobAckDto.cs JobStatusDto.cs TileJobDto.cs
WorkerRegisterDto.cs HeartbeatDto.cs ClusterStatusDto.cs (all new)
Wire/
MessageEnvelope.cs (amend: PayloadKind=binary)
BinaryFraming.cs (new) — length-prefixed binary frames
Tls/
ServerCertLoader.cs (amend: role OID extraction)
Logging/
ClusterLogger.cs (new) — NDJSON cluster events
Client/
FFClientConnection.cs (amend: SubmitJob/PollJob/FetchJob)
ClusterClient.cs (new) — high-level polling helper
UI.Avalonia/
ViewModels/Cluster/
ClusterDashboardViewModel.cs
JobListViewModel.cs
JobDetailViewModel.cs
WorkerDetailViewModel.cs (all new)
Views/Cluster/
ClusterDashboardView.axaml(+.cs)
JobListView.axaml(+.cs)
JobDetailView.axaml(+.cs)
WorkerDetailView.axaml(+.cs) (all new)
ViewModels/ServerAdminViewModel.cs (amend: cluster config fields)
Views/ServerAdminView.axaml (amend: cluster config tab)
Server.Tests/
Cluster/
WorkerRegistrationTests.cs
TilePlannerTests.cs
ArtifactMergerTests.cs
EndToEndImageTests.cs
EndToEndVideoTests.cs
DeterministicOutputAcrossOsTests.cs (all new)
Docs/
Technical/DistributedRendering-DevelopmentPlan.md (this file)
User/Distributed-UserGuide.md (Phase D-6)
Per the project's "S-X*" session-notes convention
(Docs/Technical/S-X10-Session-Notes.md etc.), distributed-rendering work
takes a D-N[suffix] tag in commit subjects:
-
D-1= Phase D-1, etc. -
D-1a,D-1b, … for follow-ups within the same phase. - Session notes per phase live as
Docs/Technical/D-N-Session-Notes.md(one file per phase, appended to across sessions).
Commit subject pattern (matches the repo's style — see git log):
feat: D-1 — worker registration + role-aware mTLS dispatch
fix: D-2b — tile merge mmap leak under cancellation
docs: D-3 — note binary framing on-wire format
The first commit of each session should reference the previous
phase-notes file and the open question it picks up; the last commit
should append a short "next session" block to the phase notes. Same
pattern as S-X8-S-X9-Session-Notes.md.