FrameFleet

Turn one master video into multiple delivery-ready exports through a fault-tolerant worker pipeline.

FrameFleet is a full-stack video delivery engine for creators and small media teams that need the same source video in several resolutions, quality levels, and file-size limits. A user uploads once, configures up to six outputs, and watches each rendition move from queued segments to a downloadable MP4.

The project focuses on the systems problems behind that workflow: concurrent job claiming, durable state, worker failure recovery, stale-result rejection, final-size verification, and measurable encoding performance.

What it does

• Creates several named exports from one source upload.
• Splits each rendition into independently claimable segments.
• Coordinates concurrent FFmpeg workers through PostgreSQL row locks.
• Recovers expired work with leases, heartbeats, retries, and fencing tokens.
• Supports original, 1080p, 720p, and 480p outputs with three quality profiles.
• Calculates bitrate budgets for optional final-size limits.
• Measures the assembled file and automatically performs one adjusted encoding pass when an output exceeds its limit.
• Persists job history, progress, retry counts, output sizes, and performance metrics.
• Supports cancellation and completed-export downloads from the React UI.

Architecture

flowchart LR
    User["Creator"] -->|"Upload once and choose outputs"| UI["React + TypeScript UI"]
    UI -->|"REST API"| API["FastAPI service"]

    API -->|"Jobs, segments, settings"| DB[("PostgreSQL")]
    API -->|"Source and final downloads"| Storage["Shared Docker volume"]
    API -->|"Probe source metadata"| Probe["FFprobe"]

    subgraph Fleet["Encoding worker fleet"]
        W1["Worker A + FFmpeg"]
        W2["Worker B + FFmpeg"]
        WN["Worker N + FFmpeg"]
    end

    W1 <-->|"Claim and renew leases"| DB
    W2 <-->|"Claim and renew leases"| DB
    WN <-->|"Claim and renew leases"| DB

    W1 <-->|"Read and publish segments"| Storage
    W2 <-->|"Read and publish segments"| Storage
    WN <-->|"Read and publish segments"| Storage

Loading

Component	Responsibility	Technology
Web client	Uploads, output configuration, live progress, history, downloads	React, TypeScript, Vite
API	Validation, metadata probing, delivery planning, job responses	FastAPI, Pydantic
Scheduler	Atomic claims, retries, leases, cancellation, recovery	PostgreSQL, SQLAlchemy
Workers	Segment encoding, heartbeat renewal, assembly, size verification	Python, FFmpeg
Runtime	Reproducible API, database, and worker environment	Docker Compose

Job lifecycle

1. Probe: FFprobe reads the source duration, dimensions, codecs, format, and audio availability.
2. Plan: FrameFleet creates one durable job per requested output and divides it into fixed-duration segments.
3. Claim: Workers atomically select available segments with PostgreSQL FOR UPDATE SKIP LOCKED.
4. Encode: FFmpeg writes each attempt to a temporary file while the worker renews its lease.
5. Publish: The database accepts a result only if the worker still owns the same fenced attempt.
6. Assemble: The final worker concatenates completed video segments, restores audio, and publishes the MP4 atomically.
7. Verify: Size-constrained exports are measured and, when necessary, requeued once with a corrected bitrate.

Reliability model

• Row-level locks: Multiple workers cannot claim the same available segment.
• Expiring leases: Work owned by a stopped worker becomes claimable again.
• Heartbeats: Long FFmpeg processes retain ownership while healthy.
• Fencing: A late worker cannot overwrite the output from a newer attempt.
• Bounded retries: Repeated encoding errors eventually fail the job instead of retrying forever.
• Atomic publication: Temporary files are renamed only after ownership is confirmed.
• Durable cancellation: Cancelling a job revokes its active leases and stops workers at their next heartbeat.

Measured results

Measurements were collected on a local development machine and will vary with hardware, codecs, and source complexity.

• Processed an 8-second 720p test video containing 11 segments in 1.662 seconds, or 4.81x realtime.
• Corrected a deliberately oversized export from 1,991,339 bytes to 1,007,874 bytes, meeting a 1 MiB limit after one automatic bitrate adjustment.
• Paused a worker during a real FFmpeg encode, allowed its 6-second lease to expire, and verified that a replacement worker completed the export while the original stale attempt was rejected.

Run locally

Prerequisites

• Docker Desktop
• Node.js 22
• npm

Start PostgreSQL, the FastAPI service, and one worker:

docker compose up --build

In another terminal, start the frontend:

nvm use 22
npm install
npm run dev

Open the following local URLs:

• Application: http://localhost:5173
• FastAPI documentation: http://localhost:8000/docs
• Health check: http://localhost:8000/health

To run several workers on the same Docker host:

docker compose up --build --scale worker=4

Stop the stack without deleting saved jobs or uploads:

docker compose down

Try the workflow

1. Select a browser-supported video.
2. Add or remove delivery outputs.
3. Choose a resolution, quality level, and optional maximum file size for each output.
4. Create the delivery batch and watch segment-level progress.
5. Review elapsed time and realtime throughput when processing finishes.
6. Download each completed MP4 from the live delivery view or job history.

REST API

Method	Endpoint	Purpose
GET	/health	Check API availability
POST	/deliveries	Upload one source and create several outputs
GET	/deliveries/{batch_id}	Retrieve a delivery batch and its output jobs
POST	/jobs	Create a single encoding job
GET	/jobs	List recent encoding jobs
GET	/jobs/{job_id}	Retrieve progress and performance data
POST	/jobs/{job_id}/cancel	Cancel queued or processing work
GET	/jobs/{job_id}/download	Download a completed MP4

Verification

Run the backend unit tests inside the project environment:

docker compose run --no-deps --rm backend \
  python -m unittest backend.test_size_constraints backend.test_performance

Validate the frontend:

nvm use 22
npm run lint
npm run build

Run the worker-recovery integration test:

bash scripts/test-worker-recovery.sh

The recovery test uses an isolated Docker Compose project. It pauses a worker with an active lease, starts a replacement after expiry, verifies reclamation and stale-attempt rejection, downloads the recovered output, and then removes its temporary containers, volumes, and videos.

Project structure

FrameFleet/
├── backend/
│   ├── main.py                 # REST API and delivery planning
│   ├── worker.py               # Claims, heartbeats, FFmpeg, recovery
│   ├── tables.py               # PostgreSQL persistence models
│   ├── probe.py                # FFprobe metadata extraction
│   ├── size_constraints.py     # Bitrate budgeting and correction
│   └── performance.py          # Wall-clock and realtime metrics
├── scripts/
│   └── test-worker-recovery.sh # End-to-end failure test
├── src/
│   ├── api/                    # Typed REST client
│   ├── video/                  # Client-side segment preview planning
│   └── App.tsx                 # Product interface and job dashboard
└── compose.yaml                # API, PostgreSQL, and worker services

Current deployment boundary

The local version uses a Docker volume shared by workers on one host. The scheduler is safe to scale across worker processes, but a true multi-machine deployment would replace the shared volume with object storage such as S3 and run workers against the same PostgreSQL database. Authentication, multi-tenancy, and hosted observability are also intentionally outside the current scope.