Turns a silent screen recording, a product website URL and a user prompt into a narrated, edited product-demo video with an AI presenter.
The system behaves like an AI video director: AI makes the editorial decisions, deterministic code executes them. No LLM ever touches a video file, and no LLM ever authors an ffmpeg command.
Status: complete. All phases are implemented and wired end to end — upload through to a finished MP4. 410 backend tests pass; the frontend type-checks and builds clean. PROJECT_MEMORY.md holds the live state, including an honest account of what has been measured versus assumed. AVATAR_PROVIDERS.md covers the avatar vendor research.
You give it a screen recording with no audio. Here is every process that runs, in order.
USER
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ VIDEO │ │ PRODUCT │ │ PROMPT │
│ (no audio) │ │ URLs │ │ │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
▼ ▼ │
┌───────────────────┐ ┌───────────────────┐ │
│ PREPROCESSOR │ │ WEB AGENT │ │
│ │ │ │ │
│ ffprobe validate │ │ Interfaze fetch │ │
│ proxy encode │ │ extract claims │ │
└─────────┬─────────┘ └─────────┬─────────┘ │
│ │ │
▼ ▼ │
PROXY CLIP PRODUCT CONTEXT │
1280px · 10fps features · claims │
│ │ │
▼ │ │
┌─────────────────────────┐ │ │
│ INTERFAZE PERCEPTION │ │ │
│ │ │ │
│ video │ │ │
│ OCR (visible_text) │ │ │
│ GUI elements │ │ │
│ actions │ │ │
│ evidence │ │ │
└────────────┬────────────┘ │ │
│ │ │
▼ │ │
COARSE TIMELINE │ │
│ │ │
▼ │ │
┌─────────────────────────┐ │ │
│ REFINEMENT │ │ │
│ │ │ │
│ re-watch each segment │ │ │
│ on its own, closely │ │ │
│ + ffmpeg scene detect │ │ │
└────────────┬────────────┘ │ │
│ │ │
▼ │ │
REFINED TIMELINE │ │
│ │ │
▼ │ │
┌─────────────────────────┐ │ │
│ RECONCILER │ │ │
│ (no AI at all) │ │ │
│ │ │ │
│ merge · grade evidence │ │ │
│ detect conflicts │ │ │
└────────────┬────────────┘ │ │
│ │ │
▼ │ │
CANONICAL TIMELINE │ │
│ │ │
└────────────────────────┬────────────────────────┘
▼
┌─────────────────────────────┐
│ DIRECTOR — Gemini 3.1 Pro │
│ │
│ What happened? │
│ What matters? │
│ What should be said? │
│ Is it actually true? │
└──────────────┬──────────────┘
│
▼
EDIT PLAN
validated schema + plan check
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ ffmpeg │ │ ELEVENLABS │ │ HEYGEN │
│ │ │ │ │ │
│ cut · speed │ │ one take per │ │ one render for │
│ redact │ │ batch, with │ │ the whole bed │
│ │ │ char timings │ │ (alpha matte) │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└────────────────────────┬────────────────────────┘
▼
┌─────────────────────────────┐
│ COMPOSITOR — ffmpeg │
│ │
│ de-collide narration │
│ mix · loudness │
│ burn captions │
│ overlay presenter │
└──────────────┬──────────────┘
│
▼
DRAFT
│
▼
┌─────────────────────────────────────┐
│ QA │
│ │
│ deterministic checks (trusted) │
│ Gemini critic (advisory) │
└──────────────────┬──────────────────┘
│
┌──────────────────┼──────────────────┐
▼ ▼
FINAL MP4 revise the PLAN and
render again, bounded
by MAX_REVISIONS
POST /api/videos accepts the upload. Before anything else the file is
validated by ffprobe, not by its extension: container, real duration, real
dimensions, and whether it actually decodes. Over-long or over-large uploads
are rejected outright, never silently truncated. Product URLs and the user
prompt ride along with the same request.
This is where the screen recording stops being pixels. The recording is downscaled to a proxy clip first (a 1440p60 capture is far past the inline upload limit; timings are preserved exactly, so every timestamp still refers to the original).
Interfaze watches it and returns a structured ObservedAnalysis:
| field | what it is |
|---|---|
summary |
what the recording shows, overall |
application |
which product/app is on screen |
events[] |
timestamped segments, each with a start and end |
events[].description |
what happens in that stretch |
events[].actions[] |
what the user did — clicked, typed, scrolled |
events[].ui_elements[] |
named controls, with their type and label |
events[].visible_text[] |
text read off the screen — the OCR-shaped output |
events[].evidence[] |
what was actually observed, with timestamps |
events[].model_confidence |
how sure it is |
visible_text and ui_elements are what make the rest possible: the Director
can only claim something is on screen if perception recorded the text for it.
The coarse pass is good at what and unreliable at when. So every macro segment long enough to matter is cut out and analysed again on its own, several at a time. Short clips are the regime where timestamps are accurate.
In parallel, ffmpeg scene detection finds real cut boundaries. Screen recordings cut softly, so the threshold here is deliberately far lower than the one used for per-event refinement.
Two timelines now disagree. This stage merges them into one canonical timeline in plain code: parent/child containment, evidence grading, conflict detection, and an explicit record of which timestamps are trustworthy. Nothing is smoothed over — a disagreement is reported as a disagreement.
The product URLs are fetched and analysed into a ProductContext: what the
product is, its features, and the claims the marketing makes — each tied back
to the page text that supports it. Cached on disk by content hash.
The only stage that makes editorial judgements, and it runs in named stages rather than one giant prompt:
- Story — what this video is about and what the viewer should leave with
- Editorial — which events survive, which are trimmed, what gets sped up
- Narrative — the actual narration lines, budgeted against a words-per-minute rate
- Claims — every factual statement checked against the canonical timeline and the product context; unsupported claims are cut, not softened
- Duration — the arithmetic
The output is an EditPlan: segments to cut, speeds, redactions, narration
blocks with timings, transitions. It is validated against a Pydantic schema and
a deterministic plan checker before a single frame is rendered. A plan that
cannot be executed is refused here, not discovered halfway through ffmpeg.
Every stage is cached on disk, so re-running the Director is cheap.
POST /api/videos/{id}/generate starts a background job. In order:
| step | what runs | notes |
|---|---|---|
| extract | ffmpeg | cut each segment, apply speed, apply redaction blur |
| join | ffmpeg | one continuous picture; every clip normalised to one profile |
| narrate | ElevenLabs eleven_v3 |
one take for the whole video, cut into blocks on the returned character alignment |
| fit | Director + ffmpeg | overrunning lines shortened, then the whole take re-spoken |
| de-collide | deterministic | no two lines may ever speak at once |
| avatar | HeyGen | one continuous render over the whole narration bed |
| mix | ffmpeg | narration bed laid at exact offsets, two-pass loudness normalisation |
| caption | ffmpeg | cues built from real per-character timings, burned in |
| overlay | ffmpeg | presenter composited as a rounded picture-in-picture bubble |
Two of those steps exist because of defects that were found by listening and looking, not by reading logs:
- Narration is generated in one take, not per line. eleven_v3 draws an
independent performance on every request — the same sentence rendered five
times wandered 32.3 Hz in pitch — which made the narrator sound like two
different people. Request stitching is unsupported on v3 and
seedis silently ignored, so the only lever is fewer requests. - The avatar is one continuous render, not one per block. Rendering per block meant the presenter vanished during every planned pause. One call over the whole bed — silences included — and the provider generates real idle motion: verified frame by frame across a 4.27 s pause.
Deterministic checks run first and are the trustworthy ones: render integrity, audio levels and clipping, timing and drift, narration fit, caption overlap, and whether every sensitive interval is actually covered by a redaction.
Then a Gemini visual critic samples frames and judges them. It is explicitly treated as advisory — it has a recorded history of over-generalising from a small sample, and its claims are checked against the manifest before anything acts on them.
If the result fails, the revision loop amends the plan and renders again,
bounded by MAX_REVISIONS. Every attempt is recorded. The published file is
re-validated independently after it is written.
| Service | Role | Used for |
|---|---|---|
| Interfaze | video understanding | reading the screen recording — events, UI elements, on-screen text, evidence |
| Interfaze | website understanding | fetching and analysing the product pages into evidence-backed context |
| Gemini 3.1 Pro | Director | every editorial decision, narration writing, claim checking, QA critique, line shortening |
| ElevenLabs | voice | eleven_v3, one take per request batch, returns per-character timings used for captions |
| HeyGen | presenter | one continuous avatar render, alpha-matted webm, composited locally |
| ffmpeg / ffprobe | everything else | validation, probing, cutting, speed, blur, mixing, loudness, captions, overlay |
The interface between the AI half and the deterministic half is exactly one
thing: a validated EditPlan.
The React app is a pipeline inspector, not just an upload form. Each panel appears as its stage becomes available:
| Panel | Endpoint | Shows |
|---|---|---|
BackendStatus |
GET /api/health |
binaries, storage, which provider keys are configured |
UploadForm |
POST /api/videos |
drag-drop upload with progress, URLs, prompt |
AnalysisPanel |
POST/GET .../analyze, .../analysis |
the coarse event timeline |
EditorTimelinePanel |
POST .../refine-analysis, GET .../timeline |
the refined, editor-grade timeline |
CanonicalTimelinePanel |
POST/GET .../canonical-timeline |
the reconciled timeline, conflicts and evidence grades |
ProductContextPanel |
POST/GET .../product-context |
features and claims pulled from the website |
GenerationPanel |
POST/GET .../edit-plan, .../generate |
the plan, live job status, and the finished video |
PipelineStrip |
— | how far along the run is |
Long work is never held open on a request: generation returns a job id and the
UI polls GET /api/videos/{id}/generate/{job_id} until the status is terminal.
Fifteen routes, verified against the running OpenAPI schema.
| Method | Path | Purpose |
|---|---|---|
GET |
/ |
service root |
GET |
/api/health |
binaries, storage, limits, which keys are set |
POST |
/api/videos |
upload a source video (+ product URLs, prompt) |
GET |
/api/videos/{id} |
stored metadata and status |
GET |
/api/videos/{id}/file |
stream the source video back |
POST |
/api/videos/{id}/analyze |
run video understanding |
GET |
/api/videos/{id}/analysis |
the stored event timeline |
POST |
/api/videos/{id}/refine-analysis |
deep per-segment refinement |
GET |
/api/videos/{id}/timeline |
the refined timeline |
POST |
/api/videos/{id}/canonical-timeline |
reconcile coarse + refined |
GET |
/api/videos/{id}/canonical-timeline |
the canonical timeline |
POST |
/api/videos/{id}/product-context |
analyse the product website |
GET |
/api/videos/{id}/product-context |
the stored product context |
POST |
/api/videos/{id}/edit-plan |
run the Director |
GET |
/api/videos/{id}/edit-plan |
the stored EditPlan |
POST |
/api/videos/{id}/generate |
start a render job |
GET |
/api/videos/{id}/generate |
the latest job |
GET |
/api/videos/{id}/generate/{job_id} |
poll one job |
GET |
/api/videos/{id}/final-video |
stream the finished MP4 |
storage/
├── uploads/ source videos + their analysis, timeline, product,
│ canonical and editplan JSON, keyed by video id
├── analysis_cache/ per-stage AI results, keyed by content hash
├── product_cache/ website analyses
├── director/ every Director stage, per video
├── renders/ manifests, QA reports, revision history, final MP4
├── work/<job_id>/ one working directory per render:
│ ├── source/ segments/ audio/ narration/ avatar/
│ └── captions/ draft/ qa/ final/ logs/
└── keep/ deliverables held back from cleanup
The execution manifest in renders/ is the audit trail: every segment,
every narration asset with its real measured duration, every avatar asset,
every ffmpeg invocation, and every warning. When QA disagrees with the plan,
this is the file to read.
| Tool | Verified version | Notes |
|---|---|---|
| Python | 3.14.3 | 3.11+ should work |
| Node.js | 24.3.0 | 20+ should work |
| ffmpeg / ffprobe | 7.1.1 | must be on PATH |
Install ffmpeg:
brew install ffmpegOn Debian/Ubuntu use sudo apt install ffmpeg; on Windows install from
ffmpeg.org and add it to PATH. The health
endpoint reports whether both binaries were found, so you can check without
uploading anything.
cp .env.example .envSet INTERFAZE_API_KEY to use video analysis. Everything else has a working
default, and upload/validation runs fine without a key — the health endpoint
reports which provider keys are configured.
cd backend && python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txtRun it:
cd backend && ./.venv/bin/uvicorn app.main:app --reload --port 8000- API root: http://127.0.0.1:8000/
- Health: http://127.0.0.1:8000/api/health
- Interactive docs: http://127.0.0.1:8000/docs
cd frontend && npm installRun it:
cd frontend && npm run devOpen http://localhost:5173 — note localhost, not 127.0.0.1; Vite 8 binds
only the former.
The frontend reaches the backend at VITE_API_BASE_URL, defaulting to
http://127.0.0.1:8000. Override it via frontend/.env:
cp frontend/.env.example frontend/.envcd backend && ./.venv/bin/pip install -r requirements-dev.txt && ./.venv/bin/python -m pytestThe suite generates its own test clips with ffmpeg (tiny, 160x120) and writes to
a throwaway storage directory, so it never touches storage/.
Server-side only — none of these reach the browser.
| Variable | Default | Purpose |
|---|---|---|
MAX_VIDEO_DURATION_SECONDS |
300 |
Hard limit on source video length. Longer uploads are rejected. |
MAX_UPLOAD_SIZE_MB |
512 |
Hard limit on upload size. |
STORAGE_PATH |
./storage |
Relative paths resolve against the repository root. |
ENVIRONMENT |
development |
Reported by the health endpoint. |
HOST / PORT |
127.0.0.1 / 8000 |
Server bind address. |
LOG_LEVEL |
INFO |
Pipeline logging verbosity. |
FFMPEG_PATH / FFPROBE_PATH |
ffmpeg / ffprobe |
Override if the binaries are not on PATH. |
FFPROBE_TIMEOUT_SECONDS |
30 |
Guard against a wedged probe. |
INTERFAZE_API_KEY |
unset | Required for video analysis. |
INTERFAZE_BASE_URL |
unset | Blank uses the SDK default, https://api.interfaze.ai/v1. |
INTERFAZE_MODEL |
interfaze-beta |
Model used for video understanding. |
INTERFAZE_TIMEOUT_SECONDS |
600 |
Analysis is slow; keep this generous. |
INTERFAZE_MAX_ATTEMPTS |
3 |
Total attempts. Only transient failures are retried. |
ANALYSIS_PROXY_WIDTH |
1280 |
Width of the downscaled clip sent for analysis. |
ANALYSIS_PROXY_FPS |
10 |
Frame rate of that clip. Duration is unchanged. |
ANALYSIS_PROXY_CRF |
30 |
x264 quality for that clip. |
ANALYSIS_MAX_INLINE_MB |
14 |
Guard below Interfaze's 20 MB inline cap. |
PRODUCT_CONTEXT_MAX_URLS |
10 |
Most URLs a user may supply. |
PRODUCT_CONTEXT_MAX_PAGES |
6 |
Most pages retrieved in total. |
PRODUCT_CONTEXT_DISCOVER_LINKS |
false |
Follow links off the first page. Off by default. |
PRODUCT_CONTEXT_MAX_TOTAL_CHARS |
14000 |
Shared text budget across all pages. |
PRODUCT_CONTEXT_ANALYSIS_ATTEMPTS |
2 |
Retry once if the model returns an empty context. |
PRODUCT_CONTEXT_CACHE_TTL_SECONDS |
0 |
Disk cache lifetime. 0 means never expire. |
REFINE_MIN_SEGMENT_SECONDS |
6 |
Shorter macro events are not refined. |
REFINE_MAX_SEGMENT_SECONDS |
75 |
Longer ones are split before analysis. |
REFINE_LONG_SEGMENT_SECONDS |
30 |
At/above this, detail is upgraded regardless of importance. |
REFINE_MAX_CONCURRENCY |
3 |
Concurrent detailed analyses. |
BOUNDARY_REFINEMENT_ENABLED |
true |
Snap event starts to real scene changes. |
BOUNDARY_WINDOW_SECONDS |
1.5 |
How far a boundary may move. |
SCENE_CHANGE_THRESHOLD |
0.12 |
ffmpeg scene-detection sensitivity. |
ANALYSIS_CACHE_ENABLED |
true |
Cache analyses and timelines by video content hash. |
RECONCILE_TOLERANCE_SECONDS |
0.75 |
Allowed excursion past a parent macro boundary. |
RECONCILE_SCENE_THRESHOLD |
0.03 |
Scene sensitivity for reconciliation; screen recordings cut softly. |
SCENE_CLUSTER_SECONDS |
0.5 |
Collapse repeated detections of one transition. |
GEMINI_API_KEY |
unset | Required for the Director, claim checking and the QA critic. |
GEMINI_DIRECTOR_MODEL |
gemini-3.1-pro-preview |
Pinned, not an alias: a Director whose model shifts is not reproducible. |
EDIT_SPEAKING_RATE_WPM |
165 |
Word budget for narration. Tracks the TTS, not a human reference. |
| Variable | Default | Purpose |
|---|---|---|
ELEVENLABS_API_KEY |
unset | Required for narration. |
ELEVENLABS_MODEL |
eleven_v3 |
The expressive tier. eleven_multilingual_v2 reads long narration flat. |
ELEVENLABS_VOICE_ID |
cjVigY5qzO86Huf0OWal |
Measured at 165 wpm overall, 130-201 wpm per block. |
TTS_STABILITY |
0.45 |
Lower is more expressive. Note v3 exposes only 0.0 / 0.5 / 1.0. |
TTS_STYLE |
0.30 |
Defaults to 0 upstream — no emphasis at all — so it is set explicitly. |
TTS_SINGLE_TAKE |
true |
Narrate in as few requests as the character limit allows. Per-request generation made the narrator sound like two different people. |
TTS_MAX_CHARS_PER_REQUEST |
2800 |
Short of the ~3000 v3 limit: an overrun fails the whole batch. |
TTS_MAX_RETRIES |
2 |
Take-level, never line-level — re-rendering one line redraws the speaker. |
A line that overruns its shot is cosmetic; a line that overruns into the next line is two voices at once, because the bed sums overlapping clips.
| Variable | Default | Purpose |
|---|---|---|
NARRATION_MIN_GAP_SECONDS |
0.12 |
Minimum silence between two spoken lines. |
NARRATION_MAX_SHIFT_SECONDS |
2.0 |
How far a line may start late to clear the one before it. |
NARRATION_MAX_TEMPO |
1.12 |
Pitch-preserving compression ceiling for an overrunning line. |
NARRATION_PADDING_BEFORE / NARRATION_PADDING_AFTER |
0.15 |
Breathing room so a line never starts on the cut. |
| Variable | Default | Purpose |
|---|---|---|
HEYGEN_API_KEY |
unset | Required for the avatar. Billed from a prepaid USD wallet, not subscription credits. |
HEYGEN_AVATAR_ID |
a v3 look | Must be a v3 look; a legacy id silently falls back to a worse engine. |
HEYGEN_ENGINE |
avatar_v |
Best lip sync. Check the invoice — a downgrade shows up in the rate. |
HEYGEN_RESOLUTION |
720p |
Plenty for a ~320 px bubble. 4k|1080p|720p only. |
HEYGEN_OUTPUT_FORMAT |
webm |
Real alpha. ffmpeg drops it unless -c:v libvpx-vp9 precedes -i. |
AVATAR_ENABLED |
true |
Voice-only is a supported mode, not a degraded one. |
AVATAR_COVERAGE |
full |
One continuous render over the whole bed so the presenter idles through pauses instead of vanishing. intro_outro is the cheaper, chunked mode. |
AVATAR_LAYOUT / AVATAR_CORNER |
picture_in_picture / bottom_right |
Where the presenter sits. |
AVATAR_SCALE / AVATAR_MARGIN |
0.24 / 44 |
Bubble size and inset. |
| Variable | Default | Purpose |
|---|---|---|
MAX_REVISIONS |
2 |
Extra render attempts after QA. Each one re-renders the avatar — set 0 to cap spend. |
CAPTIONS_ENABLED |
true |
Burned-in captions from real per-character timings. |
REDACTION_METHOD |
blur |
Phase 4.7 records when something sensitive is on screen, never where. |
AUDIO_NORMALIZE_ENABLED |
true |
Two-pass loudness; single-pass pumps on speech. |
The frontend reads only VITE_API_BASE_URL. Never put a secret in a VITE_*
variable — they are inlined into the browser bundle.
The route table is in API surface above. This section covers request and error shapes.
multipart/form-data:
| Field | Required | Notes |
|---|---|---|
video |
yes | The source recording. |
website_urls |
no | Product URLs. Repeat the field to supply several. |
website_url |
no | Deprecated single-URL form, still accepted. |
prompt |
no | Stored for later phases; not processed yet. |
201 Created:
{
"video_id": "5789156e952f46c1b688e96bda897305",
"filename": "product_demo.mp4",
"status": "ready",
"metadata": {
"duration": 240.0,
"width": 854,
"height": 480,
"fps": 24.0,
"codec": "h264",
"format": "mov,mp4,m4a,3gp,3g2,mj2",
"size_bytes": 3295258,
"has_audio": false,
"audio_codec": null
}
}Try it from the shell:
curl -F "video=@demo.mp4" -F "website_url=https://example.com" http://127.0.0.1:8000/api/videosEvery failure uses one envelope, so the UI can react to code rather than
parsing prose. Internal detail (ffprobe stderr, paths, tracebacks) is logged
server-side and never returned.
{
"code": "VIDEO_TOO_LONG",
"message": "Videos must be 5 minutes or shorter. Your video is 6 minutes 14 seconds.",
"details": { "duration": 374.0, "max_duration": 300 }
}| Code | HTTP | Meaning |
|---|---|---|
INVALID_REQUEST |
422 | Malformed request. |
INVALID_FILE_TYPE |
415 | Extension not in the allow-list. |
EMPTY_FILE |
400 | Zero bytes. |
FILE_TOO_LARGE |
413 | Over MAX_UPLOAD_SIZE_MB. |
INVALID_VIDEO |
422 | Not a decodable video, whatever it is named. |
VIDEO_TOO_LONG |
422 | Over MAX_VIDEO_DURATION_SECONDS. |
VIDEO_NOT_FOUND |
404 | Unknown or malformed video_id. |
VIDEO_PROCESSING_UNAVAILABLE |
503 | ffprobe missing or timed out. |
VIDEO_NOT_READY |
409 | Analysis is already running for this video. |
VIDEO_FILE_MISSING |
410 | The record exists but the media is gone. |
ANALYSIS_NOT_CONFIGURED |
503 | No/invalid Interfaze API key. |
ANALYSIS_UNAVAILABLE |
503 | Transient provider failure after retries. |
ANALYSIS_FAILED |
502 | Permanent provider error. |
ANALYSIS_INVALID_OUTPUT |
502 | Provider output was not usable JSON. |
ANALYSIS_NOT_FOUND |
404 | This video has not been analysed. |
INVALID_WEBSITE_URL |
400 | Malformed, non-HTTP, or private/internal URL. |
WEBSITE_UNREACHABLE |
502 | No page could be read. |
PRODUCT_CONTEXT_UNAVAILABLE |
503 | Provider unavailable or unconfigured. |
PRODUCT_CONTEXT_INVALID_OUTPUT |
502 | Provider output was not usable. |
PRODUCT_CONTEXT_NOT_FOUND |
404 | No product context stored for this video. |
INTERNAL_ERROR |
500 | Unexpected failure. |
.mp4 · .mov · .webm · .mkv · .m4v (configurable in Settings).
The extension is only a fast pre-filter. Every upload is probed with ffprobe,
so a text file renamed demo.mp4 is rejected as INVALID_VIDEO.
Audio is optional. The expected input is a silent screen recording, so
has_audio: false is a normal, valid result — never a rejection reason.
Source videos must be 300 seconds or shorter. This is a deliberate V1 constraint aligned with the current Interfaze single-request video limitation.
- The limit is read from
MAX_VIDEO_DURATION_SECONDSand lives in exactly one place (backend/app/core/config.py). It is not hard-coded anywhere. - Duration comes from ffprobe, not from the client.
- Over-length videos are rejected with an explanatory message, never
silently trimmed:
Videos must be 5 minutes or shorter. Your video is 6 minutes 14 seconds. - Videos of exactly 300 s are accepted.
POST /api/videos/{video_id}/analyze turns a stored recording into a
timestamped event timeline. It runs synchronously and takes roughly 55 s for a
26-second clip and 100 s for a four-minute one.
stored video -> analysis proxy -> Interfaze -> validate/normalize -> VideoAnalysis
Provider. Interfaze via the official interfaze
Python SDK (1.0.3), model interfaze-beta. Structured output is requested with
chat.completions.create(response_format=response_format(schema, name)) and the
video is attached with inputs.video(...) as a base64 data URL.
The analysis proxy. Interfaze accepts at most 20 MB inline, and a 1440p60
screen recording is far bigger, so ffmpeg encodes a downscaled copy first
(1280 px wide, 10 fps by default — the 237-second test recording came out at
2.0 MB). The fps filter resamples frames without changing duration, so every
timestamp still refers to the original video. The original is never modified.
What comes back. A VideoAnalysis: a factual summary, the application on
screen, and a list of VideoEvents, each with start_time/end_time in
seconds, an event_type from a closed vocabulary (feature_demo,
irrelevant_navigation, potential_sensitive_content, …), a description, user
actions, UI elements, legible on-screen text, importance, evidence, and the
model's self-reported confidence where it offered one.
Deterministic post-processing. The model's timestamps are never trusted as
given. After generation the backend clamps small overshoots, rejects impossible
ranges outright, sorts events by start time, merges near-duplicate overlapping
events, and records every correction in warnings — visible in the debug panel.
What it does not do. Phase 3 observes; it does not narrate, edit, trim, blur or render. It only classifies irrelevant and sensitive segments so a later Director can decide what to do about them.
Timestamp accuracy caveat. Verified against frame captures from the real test recording: on short clips (≤60 s) the timestamps are accurate, but across a full 237-second video they drift by 15–40 s in the back half and one segment was missed entirely. Treat long-video timings as approximate for now. See PROJECT_MEMORY.md §13 for the evidence.
POST /api/videos/{video_id}/refine-analysis turns the coarse Phase 3 analysis
into a timeline an automated editor can cut from. It needs an existing analysis
— the macro events are the plan.
macro events -> segment plan -> per-clip detailed analysis
-> absolute timestamps -> boundary refinement
-> validation, ordering, secret scan -> timeline
Each macro segment is cut out and analysed as its own clip. This is the point of the phase, not just extra detail. Phase 3 measured the model placing events accurately inside a short clip but drifting 15–40 s across a four-minute one, so every second-pass request is kept inside the short-clip regime and the offsets are added back deterministically. On the test recording this corrected errors of ~40 s and recovered a segment the coarse pass missed entirely.
Adaptive detail. High-importance segments are broken down finely, low ones
lightly. A segment longer than REFINE_LONG_SEGMENT_SECONDS is upgraded
regardless of its rating, because a long macro event is exactly where the coarse
timeline says least. Segments longer than REFINE_MAX_SEGMENT_SECONDS are split.
What each event carries. Timestamps in seconds; an event_type from a
closed vocabulary; actions kept separate from results; observations
(literally visible) kept separate from inference; UI elements; visible text;
importance; evidence with timestamps; sensitive_content; an editorial_hint
(keep / compress / likely_trim / likely_blur / flag / unknown); and
a refinement_status.
Hints are hints. Nothing is trimmed, blurred or edited in this phase. The Director decides later.
Deterministic boundary refinement. Model timestamps are treated as
candidates. ffmpeg's scene-change filter is run over a ±BOUNDARY_WINDOW_SECONDS
window around each event start and, when a real visual change is found there,
the boundary snaps to it and records the shift in boundary. No confidence
number is invented; either a change was found nearby or it was not.
Secrets are checked twice. The model is asked to flag sensitive content, and
app/services/secrets.py independently pattern-matches the extracted text for
keys, tokens, JWTs, emails and card-like numbers. A match escalates the hint to
flag. Neither path ever echoes the matched value.
Failures are contained. If one segment fails, its macro event is kept with
refinement_status: fallback_macro and the rest of the video still processes.
Everything is cached on disk by video content hash, under
storage/analysis_cache/. Per-segment results are cached separately from the
assembled timeline, so improving post-processing costs nothing — the timeline
rebuilds from cached segments with zero provider calls. {"refresh": true}
forces a paid re-run.
The timeline is written to storage/uploads/<video_id>.timeline.json and served
from GET /api/videos/{video_id}/timeline.
In the UI, the "Editor-grade timeline" card shows the video player above the event list. Clicking an event expands its actions, results, evidence, UI elements and sensitive-content flags, and seeks the player to its start.
To reopen an already-uploaded video without re-picking the file:
http://localhost:5173/?video=<video_id>
POST /api/videos/{video_id}/canonical-timeline takes the Phase 4.5 timeline
and decides whether to believe it. Entirely deterministic — zero provider
calls — so it is free to re-run, and the right place to fix reconciliation
logic without spending anything.
Phase 4.5 timeline
-> containment & gap reconciliation
-> macro/detail contradiction detection
-> evidence grading
-> observation / inference separation
-> entity grounding against on-screen text
-> sensitivity re-tiering (P0-P3)
-> safe edit boundaries from one scene pass
-> redundancy grouping & merge candidates
-> editorial recommendation
-> validation report + quality score
It catches the pipeline contradicting itself. On the test recording the
macro pass said "the user opens a new tab and navigates to ChatGPT" at
147.5–154.5 s while the detailed pass said "the screen remains static on a
pricing page". A frame capture confirms the detailed pass is right. Phase 4.5
kept both. Phase 4.7 detects the contradiction, resolves it via the evidence
hierarchy (the detailed pass analysed that clip directly; the macro pass
inferred it while summarising four minutes) and marks the event
needs_review.
Observation, inference and editorial are separate. observations are
checkable facts, inference is interpretation with its own
inference_confidence, and speculative intent — "the user is preparing to
click" — is detected and marked unsupported_inference.
Sensitivity is tiered. P0_SECRET and P1_PRIVATE warrant redaction; P2_CONTEXTUAL asks for a human; P3_PUBLIC does not affect the edit at all. On the test recording this took nine "sensitive" findings down to one genuine redaction, leaving documentation placeholders and public UUIDs alone.
Safe cut points are separate from semantic times. temporal.semantic_* is
what the model meant; temporal.safe_* is where a real scene change was found.
Where none exists nearby, the safe times stay null with
boundary_status: uncertain rather than inventing precision.
Nothing is destroyed. The Phase 4.5 timeline is carried through verbatim as
phase_45_raw, so "what did the model originally say?" stays answerable.
Status is earned. ready_for_phase_5 only when no HIGH or CRITICAL
conflict is unresolved; otherwise needs_review. One ambiguous event is
flagged, never fatal.
Output lands in storage/uploads/<video_id>.canonical.json and is shown in the
"Canonical timeline — validation" card in the UI.
POST /api/videos/{video_id}/edit-plan takes the Phase 4.7 canonical timeline,
the product context and the user's instruction, and returns an executable
EditPlan: story, sections, cut, narration, evidence and validation.
The division of labour is the point:
INTERFAZE OBSERVES -> GEMINI DECIDES -> FFMPEG EXECUTES (Phase 6)
Gemini never sees the video. DirectorReasoningProvider.reason() takes a
prompt and a schema — no file path, no bytes. A Director that cannot be handed
a video cannot quietly start re-doing video understanding, and the 44,000-token
timeline Phase 4.7 already produced is a better input than the pixels.
| Provider | Google Gemini Developer API |
| Model | gemini-3.1-pro-preview (version 3.1-pro-preview-01-2026) |
| SDK | google-genai==2.19.0 (google-generativeai is retired) |
| Structured output | response_schema=<pydantic model>, response.parsed |
| Context | 1,048,576 in / 65,536 out |
| Thinking | required; thinking_level LOW (MEDIUM/HIGH available) |
| Sampling | temperature=0.0, seed=7 |
Pinned rather than aliased: gemini-pro-latest moves between runs, and a
Director whose model changes silently is not reproducible. It is the only
Pro-tier Gemini this key can reach — gemini-2.5-pro now returns 404 for new
keys with a message naming gemini-3.1-pro-preview as its replacement.
The Director does one of two genuinely different jobs, and they need different rules:
full_length (default) |
condensed |
|
|---|---|---|
| Output length | the whole source recording | a target, default 90 s |
| Decisions | KEEP, MERGE, BLUR, FLAG | all seven, including SKIP |
| Segments | tile the source edge to edge | select from it |
| Narration | covers ~90% of the running time | only where it adds something |
| It is a | walkthrough | trailer |
A product walkthrough that throws away three quarters of its footage is a
different video, not a shorter one — so full length is the default. Passing
target_duration selects condensed on its own; full_length overrides both.
Six reasoning stages, then two that are arithmetic. Each stage sees only what its decision needs:
1 story product context + user prompt + timeline SHAPE
2 narrative story + timeline shape -> sections + budget
3 evaluation story + FULL event detail -> 8 scores per event
4 editorial story + evaluations + full detail -> the cut
tiling close gaps, split over-long shots, re-derive events (full_length)
revision the cut, with durations MEASURED not estimated (condensed)
up to EDIT_DURATION_PASSES times; a worse revision is discarded
5 narration story + product context + surviving segments only
6 claims each claim beside exactly what it cites -> adversarial audit
---------------------------------------------------------------- application code
7 assembly clamp, snap, re-index, enforce redactions, log every repair
8 validation 32 checks over the plan and the timeline
Splitting the context is not tidiness. Phase 4 measured this model family's output quality collapsing when one request carried too much — 6 KB of input produced five evidenced features, 21 KB produced none.
Gemini is not the final validator of its own work. Stage 7 clamps every
number the model returned, checks every event id against the timeline, snaps
boundaries to Phase 4.7's verified cut points, and forces P0/P1 material to
BLUR whether the model remembered it or not. Every correction is recorded as
a PlanRepair, so a plan that needed twelve repairs stays visibly different
from one that needed none.
Stage 8 then runs 32 checks — event ids resolve, intervals are real, sequence
indexes are unique and dense, skipped events are not silently reused, narration
fits its shot, every claim has evidence, and no claim rests only on an
inference Phase 4.7 refused to stand behind. errors block
ready_for_phase_6; warnings ask for a human.
Three kinds of statement stay distinguishable end to end:
support_type |
Means | Cites |
|---|---|---|
observed |
the timeline records it on screen | canonical event ids |
strongly_inferred |
it follows necessarily from what was seen | canonical event ids |
product_context |
the website documents it | source URLs |
There is deliberately no weakly_inferred. Where Phase 4.7 marked an inference
unsupported — "the user may be preparing to click the result" — the Director
may not restate it as fact. On the test recording stage 6 rejected three claims
for exactly that, and the blocks that still assert them are marked
needs_review rather than silently rewritten into narration nobody has read.
Narration is timed against the edited video's clock, and the silence is planned as explicitly as the speech — a TTS track assembled from blocks alone would have no idea how long to wait.
Every number here is measured against the product's own 358-second launch walkthrough, transcribed with Gemini:
| Reference | This plan | |
|---|---|---|
| Speaking rate | 185 wpm (verbatim 60 s sample) | 185 wpm configured |
| Speech coverage | 93% | 90.7% |
| Subject changes | ~every 12 s | 18 blocks over 237 s |
| Words | 1,211 over 358 s | 662 over 237 s |
EDIT_SPEAKING_RATE_WPM converts words to seconds; the model is never asked to
estimate its own length. Each block carries lead_in_seconds (how long the
narrator waits — for a result to finish loading, say), a derived
speech_seconds, and the trailing_silence_seconds left over. Every gap above
EDIT_MIN_PAUSE_SECONDS becomes a NarrationPause with a reason: lead_in,
tail, between_blocks or unnarrated.
Two things had to be fixed to make this work, and both are measured rather than assumed:
- The model under-writes long blocks. Against a 131-word budget it returned
102 words; against 93 it returned 35. Blocks over ~30 s came back at 38-50% of
budget while blocks under 20 s came back at 90-96%. Full-length mode therefore
splits any segment over
EDIT_MAX_SEGMENT_SECONDS(22 s) at a real canonical event boundary, and the prompt requires exactly one block per segment. Coverage went from 60.6% to 90.7%. - Style is copied from a real narrator, not invented. The prompt carries distilled rules and verbatim phrasings from the reference — problem-first framing, "So" as the transition, "across the board", "under the hood", listing provider variants fast and moving on, closing by inviting feedback. It never invents a personal name or identity.
Every stage is cached on disk under storage/analysis_cache/director_<stage>/,
keyed on the exact prompt it was sent. Re-running after a fix costs only the
stages whose input actually changed; a full re-run of an unchanged plan costs
nothing. Measured: 152 s cold, 6.7 ms warm. {"refresh": true} pays again.
| Variable | Default | Notes |
|---|---|---|
GEMINI_API_KEY |
— | Required. Separate from INTERFAZE_API_KEY. |
GEMINI_DIRECTOR_MODEL |
gemini-3.1-pro-preview |
Pinned, never an alias. |
GEMINI_THINKING_LEVEL |
LOW |
MINIMAL is rejected by this model. |
GEMINI_TEMPERATURE / GEMINI_SEED |
0.0 / 7 |
Low-variance planning. |
EDIT_FULL_LENGTH |
true |
Walkthrough, not trailer. |
EDIT_FULL_LENGTH_MIN_RATIO |
0.92 |
Floor on source coverage. |
EDIT_SPEAKING_RATE_WPM |
185 |
Measured from the reference narrator. |
EDIT_NARRATION_COVERAGE_TARGET |
0.92 |
Reference sits at 0.93. |
EDIT_MAX_SEGMENT_SECONDS |
22 |
Above this the model under-writes. |
EDIT_MAX_LEAD_IN_SECONDS |
3 |
Longest wait before speaking. |
EDIT_MIN_PAUSE_SECONDS |
0.4 |
Below this a silence is a breath. |
EDIT_TARGET_DURATION_SECONDS |
90 |
Condensed mode only. |
EDIT_DURATION_PASSES |
2 |
Condensed retries when the cut misses. |
{
"prompt": "Narrate this recording as the launch walkthrough for OpenWebSearch...",
"full_length": true,
"target_duration": null,
"refresh": false
}All four are optional. prompt falls back to the one supplied at upload, then
to EDIT_DEFAULT_PROMPT. full_length falls back to EDIT_FULL_LENGTH;
supplying target_duration instead selects a condensed cut.
The prompt matters more than any other input. A vague instruction produces a
vague edit, so EDIT_DEFAULT_PROMPT is deliberately specific about what kind
of video this is and how the narrator should sound.
Against the 236.69-second OpenWebSearch recording (35 canonical events), in full-length mode:
| Gemini calls | 6 |
| Latency | 190.6 s cold, 0.0 s fully cached |
| Tokens | 49,421 in / 15,068 out / 8,852 thinking |
| Output | 18 segments, 236.69 s — exactly the source |
| Narration | 18 blocks, 662 words, 90.7% coverage, 13 planned pauses |
| Evidence | 24 claims surviving, 6 rejected by stage 6 |
| Validation | 0 errors, 3 warnings, quality 0.867 |
Segments tile the recording edge to edge and end exactly at 236.69 s. Block fill runs 76–103% of budget (mean 92%), longest silence 5.2 s.
The story it found, unprompted: "Delivering standardized, structured JSON responses for diverse web search queries through a single interface."
The narration opens:
"So, we built OpenWebSearch because managing multiple web search APIs is really messy. You have all these providers with different structures and billing. What we've managed to do is integrate them into a single gateway."
and closes by inviting feedback, both of which track the reference walkthrough's own moves.
Nothing is hardcoded. The same timeline in condensed mode produced a 9-segment, 92.78-second cut that compressed the repetitive scrolling at 2.0–2.5× and dropped the ChatGPT detour and the API-key dashboard entirely. The same timeline with a different prompt produced a different 13-segment cut again. The Director responds to the instruction it is given.
Output lands in storage/uploads/<video_id>.editplan.json, and the five
documented views land in storage/director/<video_id>/:
phase_5_story.json story, sections, strategy
phase_5_editorial_plan.json evaluations, segments, visual edits, decision log
phase_5_narration_plan.json narration blocks and claims
phase_5_validation.json checks, quality, run info
phase_5_final_edit_plan.json the whole plan
- Boundaries. 6 of 18 segments cut on semantic boundaries, because Phase 4.7
only verified 5 of 35. Reported as
needs_boundary_review, never faked. - Determinism is decision-level, not byte-level.
temperature=0plus a seed gives Google's documented best effort. Two runs of the same input produced 13 and 18 segments — the same story, sections and decisions, segmented differently. Do not expect a reproducible byte stream. - Rejected claims are not rewritten. Stage 6 drops the claim and marks the
block
needs_review; the sentence that made it is left alone, because rewriting it automatically would produce narration nobody has read. - Narration coverage lands near, not on, the target. 90.7% against a 92% goal. The remaining gap is blocks the model under-writes even at 22-second segments.
gemini-3.1-pro-previewhas no free tier, unlike the 3.x Flash models.- The reference style is one narrator. The voice rules are distilled from a single 358-second video. They are specific on purpose, and would need replacing for a different product or register.
- No rendering. Phase 5 stops at the plan. No FFmpeg, no TTS, no avatar.
POST /api/videos/{video_id}/generate executes the Phase 5 edit plan and
checks the result against it. It returns a job id immediately — a render takes
minutes — and the frontend polls.
edit plan -> validate -> cut -> narrate -> mix -> caption -> avatar
|
v
draft -> QA -> pass -> final
|
+-> fail -> revise -> draft
Phase 5 decides what is in the video; Phase 6 decides how to build it. The
executor renders what it was given. Where it cannot — a speed factor beyond
what stays watchable — it clamps and records the correction. What it never does
is change which footage appears: the revision vocabulary is closed and
remove_segment is explicitly refused, because dropping footage is an
editorial decision that belongs upstream.
The source video is never modified. Everything lands in
storage/work/<job_id>/, and the original stays exactly as uploaded.
| Narration | ElevenLabs | eleven_multilingual_v2, voice "Eric" |
| Avatar | HeyGen | /v3/videos, avatar_v engine, transparent WebM |
| Visual QA & critic | Gemini | gemini-3.1-pro-preview, inline frames |
| Everything else | ffmpeg 7.1.1 |
Narration uses the /with-timestamps endpoint, which returns the same audio
plus per-character timings at no extra cost. Captions are built from those
timings rather than by dividing a line evenly, which is what makes them land on
the syllable.
The presenter appears for the intro and the outro, not throughout. Phase 5
already labels every narration block with a purpose, so AVATAR_COVERAGE= intro_outro renders a presenter only for the blocks labelled intro and
outro — reading the Director's own structure rather than guessing at
timestamps. A talking head parked in the corner for four minutes covers the
product it is meant to be demonstrating, and HeyGen bills per second of render:
on the test recording this is 26.5 s of avatar instead of 237 s. Set
AVATAR_COVERAGE=full for a continuous presenter.
Each moment is driven by that block's own narration audio, which is already on disk from the TTS pass, so the avatar is lip-synced to exactly the take that is in the video.
Lip-sync quality is an engine choice, not a provider choice. HeyGen's
avatar_v engine — "the most natural motion and lip-sync" — runs only on
their v3 look catalogue. A legacy v2 avatar id is accepted but silently falls
back to an older engine, and that is what made the first renders look dubbed.
The pipeline names the engine explicitly and uses a v3 look.
The presenter is matted, not cropped out of a room. output_format: "webm"
returns the person on a real alpha channel, which is composited onto a rounded
card. One trap: ffmpeg discards VP9 alpha unless the decoder is named with
-c:v libvpx-vp9 before -i — without it the file probes as yuv420p and
looks like the matting failed.
Each of these exits 0 and produces a plausible file of the wrong length. All were measured, and each is now a test:
-towith an input-side-ssis relative.-ss 100 -i in -to 105produced 20 seconds, not 5. Durations are always computed and passed as-t.-tbounds the output, which a speed change shortens. A 20 s window at 4x with a synthesised silent track came out as a 20 s file holding 5 s of video and 15 s of silence.setptsalone does not resample frames. It raises the frame rate instead, yielding things liker_frame_rate=15360/1. Every speed change is followed byfps=.- The concat demuxer cannot fix mismatched clips, even re-encoding. It locks to the first file's parameters. Every clip is normalised to one profile first, and only then joined.
sidechaincompresstruncates when its sidechain ends. Two 10 s inputs produced 9.2 s. The sidechain is alwaysapadded.amixdefaults tonormalize=1, dividing every input by the input count. An eighteen-line narration bed would come out inaudible.force_divisible_by=2or libx264 hard-fails on an odd width;setsar=1or the player re-stretches a correctly-sized frame.
Deterministic checks run first and can stop the expensive ones (§50) — a vision model asked to review a file that is the wrong length will comment on the content and miss that the render is broken.
1 ffprobe playable, resolution, fps, codec, pixel format
2 ffmpeg filters black frames, frozen frames, loudness, true peak
3 arithmetic duration vs plan, segments present, narration fits
4 frame sampling 14 targeted frames: opening, sections, cuts, redactions, end
5 edge energy did each redaction actually obscure its frame?
6 Gemini vision do the frames look right?
7 Gemini critic does this render implement the plan?
Redaction verification is deterministic, not a model's opinion. A blurred region has almost no high-frequency energy, so comparing edge energy inside the redaction window against a normal frame answers §16 the same way every time. Measured on the real render: 0.03 against a 2.87 baseline — 1%.
Only an observed problem triggers a re-render. The critic must mark each
finding observed_problem or possible_improvement; a loop driven by "the
pacing could be tighter" never terminates.
Bounded at MAX_REVISIONS (default 2). It also stops early when the revision
engine found nothing it is allowed to change, because the next draft would be
byte-identical. After the bound the best draft is kept and the job ends
needs_manual_review — never complete, and never another loop.
Revisions are the smallest change that could fix the issue: one narration line shortened, one redaction widened, one segment slowed. Where QA finds a secret that upstream graded harmless and there is nothing to widen, a new redaction is created — covering the product is recoverable, publishing a credential is not.
Against the 236.69-second recording, full-length plan, 18 segments:
| Execution | 130 s (voice only), 1 iteration |
| Output | 236.72 s, 1920x1080, h264, aac 48 kHz stereo, 19.7 MB |
| ffmpeg calls | 40 |
| Narration | 18 lines, 213 s of speech, 1 shortened to fit |
| Captions | 56 cues, all from real character alignment |
| Redactions | 2 applied, both verified at 1–2% edge energy |
| QA | PASS on render, timing, audio, content, sensitive |
| Quality | 0.929 overall |
The critic's verdict: "The video faithfully implements the plan, demonstrating the OpenWebSearch API, its unified response format, and the dashboard. The redactions are correctly applied to sensitive information."
Two QA false positives were found and fixed, not suppressed. The vision
model twice filed Bearer ${process.env.OPENWEBSEARCH_API_KEY} as a critical
credential leak. It is an environment-variable reference in a documentation
code sample, and Phase 4.7 had already graded it P2_CONTEXTUAL correctly.
Blurring it would have ruined the demo. The fix was to hand the vision pass the
upstream classification so it checks that work rather than redoing it.
- Job state is in memory. A backend restart loses the ability to poll an in-flight render; the artefacts on disk survive. One job per video at a time, one process. Deliberate for this stage — a queue with one consumer does not need Redis.
- Redactions have no bounding box. Phase 4.7 records when something sensitive is on screen, never where, so the only guaranteed region is the whole frame. A 1.5-second full-frame blur is the honest cost of that.
montagerenders as a plain cut. Recognised, reported as degraded, not faked.- The avatar is slow and metered. HeyGen took 310 s for an 18-second clip,
and bills per second of render at a rate that makes a full-length presenter
expensive — a 237-second track was refused for insufficient credit at both
1080p and 720p.
intro_outrocoverage exists because of that, and it is the better editorial choice anyway. - Credit failures surface late. HeyGen accepts the job at submit and only checks credit during processing, so an affordable-looking submit can still fail minutes later. The pipeline treats it as non-fatal and falls back to voice-only.
- 480p is not offered. The resolution enum is
4k | 1080p | 720p, so 720p is the cheapest tier available. At a ~320 px bubble it is still roughly twice the pixels that reach the screen. - Caption placement is fixed.
move_captionis recorded but not applied — position is an ASS style setting rather than a plan field. - Cost is an estimate at list price, not an invoice.
POST /api/videos/{video_id}/product-context reads one or more product URLs and
returns an evidence-backed ProductContext. Body is optional — with none, the
URLs captured at upload are used:
{ "website_urls": ["https://interfaze.ai/", "https://openwebsearch.ai/"] }urls -> validate -> retrieve (Interfaze scraper) -> analyse each page
-> merge -> resolve citations -> ProductContext
Retrieval and understanding are separate, both as interfaces
(WebsiteRetriever, ProductAnalyzer) and as calls. Retrieval uses the
Interfaze scraper task; understanding is a structured-output completion over
the text already retrieved. They cannot be one call — the SDK refuses to combine
task= with a response_format.
Interfaze does the fetching. This backend never issues an HTTP request to a
user-supplied host, which removes most of the SSRF surface. What remains is
handled by services/web/urls.py, which rejects non-HTTP schemes, embedded
credentials, non-standard ports, localhost, .internal/.local hostnames and
any private, loopback, link-local or reserved IP literal — including
169.254.169.254.
One page per analysis call. Measured against the two test sites: a single page yields a full context with evidenced features, while two pages sent together consistently returned a product name and nothing else. Pages are analysed individually and merged deterministically, which also keeps evidence honest.
Evidence cannot be fabricated. The model never sees a URL — pages are
labelled PAGE 1, PAGE 2, and it cites the number. The backend maps the
number back to the real URL, and a number outside the range is dropped with a
warning. URLs are also stripped from the page text before sending, because
Interfaze fetches any URL it finds in a message.
Limits and cost. At most PRODUCT_CONTEXT_MAX_URLS URLs and
PRODUCT_CONTEXT_MAX_PAGES pages, with a shared character budget across pages.
Link discovery is off by default — only the URLs you supply are read.
Upload reuses a cached context automatically. Re-uploading the same
recording produces a new video_id, so on upload the backend looks up the
cache for that video's URLs and, on a hit, attaches the stored context to the
new id. In practice: enter the same URLs, click Upload & continue, and the
product context is already on screen — no provider call, nothing to click.
Results are cached on disk. A URL set is analysed once; after that the
context is read back from storage/product_cache/<hash>.json in milliseconds
with no provider call, and it survives a restart. The key is order-independent,
so the same two URLs in either order hit the same entry. Entries never expire by
default (PRODUCT_CONTEXT_CACHE_TTL_SECONDS=0). To pay for a fresh analysis:
{ "website_urls": ["https://interfaze.ai/"], "refresh": true }backend/ FastAPI service — orchestration, AI calls, ffmpeg
app/
main.py app factory, CORS, lifespan
core/ config, structured errors, logging
api/ HTTP routes, dependencies, exception handlers
models/ Pydantic domain models
services/
video/ ffmpeg.py · metadata.py · validation.py · processor.py
repository.py · transcode.py · boundaries.py · fingerprint.py
ai/ provider.py (interfaces) · interfaze.py (implementations)
video_understanding.py · timeline.py · schema.py
website_agent.py · product_schema.py
refinement.py · segment_schema.py · analysis_cache.py
prompts/{video_analysis,product_context,segment_analysis}.py
web/ urls.py (validation, SSRF guard, link ranking)
secrets.py deterministic secret patterns + P0-P3 tiering
timeline_intel/ reconcile.py · conflicts.py · editorial.py
canonicalize.py (Phase 4.7, no provider calls)
voice/ avatar/ (empty until later phases)
workers/ background pipeline execution (later)
tests/
frontend/ React + TypeScript + Vite + Tailwind
storage/ uploads, frames, audio, avatar, renders, temporary,
product_cache, analysis_cache
storage/ contents are git-ignored; only the directory structure is tracked.
Stored uploads are two files: uploads/<video_id>.<ext> (the media) and
uploads/<video_id>.json (the record), plus uploads/<video_id>.analysis.json
once analysed. Client filenames are never used as paths — they are sanitised and
kept only as metadata.
The UI follows the interfaze.ai visual language: white
ground, Geist Mono throughout, Lora for headings, the zinc scale for structure,
2–4 px radii, and ASCII -> arrows.
| Task | Command |
|---|---|
| Backend dev server | cd backend && ./.venv/bin/uvicorn app.main:app --reload |
| Backend tests | cd backend && ./.venv/bin/python -m pytest |
| Frontend dev server | cd frontend && npm run dev |
| Frontend typecheck + build | cd frontend && npm run build |
| Frontend lint | cd frontend && npm run lint |