Puppet Stage 🎭

A browser puppet with phoneme-accurate lip-sync driven by local text-to-speech (macOS say or espeak), that takes stage direction like an actor reading a teleprompter — walk, wave, emote, look around — live or from a screenplay. Characters are plain SVG + JSON, so they're fully swappable without touching code.

Zero npm dependencies. One Node server, one static page.

Puppetry is the oldest performing art — five thousand years old, give or take — this just wires it to the newest.

Ava, mid-scene, with the control panel.

Quick start

./setup.sh        # downloads Rhubarb Lip Sync into tools/, checks prerequisites
node server.js    # → http://localhost:3123

Prerequisites: node, ffmpeg (brew install ffmpeg), and a TTS engine — macOS ships say; otherwise brew install espeak / apt install espeak.

How it works

text ──▶ say / espeak ──▶ WAV ──▶ Rhubarb Lip Sync ──▶ viseme timeline (A–H, X)
                           │                                   │
                           └────────── browser plays audio ◀───┘
                                       and swaps mouth shapes on the audio clock

• Server (server.js): renders speech to WAV (cached in cache/), runs Rhubarb Lip Sync to get mouth-shape timings, and broadcasts cues to every connected stage over Server-Sent Events.
• Stage (public/): plays the audio through Web Audio and schedules mouth shapes against the audio-context clock, so sync stays tight. If Rhubarb is missing or fails, it degrades to amplitude-driven mouth movement.
• Everything is a cue. Speech, gestures, movement, and emotes all travel the same bus, so the puppet can be directed from the UI, from the command line, or by a screenplay — and every open browser tab shows the same show.

Screenplay language

Type into the Screenplay box and hit Run. Plain lines are spoken; [bracketed] lines are stage directions; a (parenthesised) prefix fires an action while the line is spoken. A teleprompter overlay follows along.

# comments start with #
[enter from left]
Hello! I'm Pip.
(wave) Nice to meet you!
[walk to 75]              # percent of stage width
[emote surprised]
Whoa!
[emote neutral] [wait 1]
[look left] [look front]
[engine espeak]           # switch TTS engine mid-script
This is my robot voice.
[engine say] [voice Samantha]
And back again.
(bow) Thanks for watching!
[exit right]

Directions: walk to N, enter from left|right, exit left|right, wait N, look left|right|up|down|front, emote <name>, view face|body (camera close-up / full stage), engine say|espeak, voice <name>, rate <n>, captions on|off (broadcast-style subtitles for spoken lines), iris|fade in|out [ms] (fullscreen takes — circular wipe or fade-to-black, default 700ms), or any action name from the character's manifest (wave, jump, nod, shake, bow, dance, shrug, …).

Two characters ship as references: pip, a cartoon blob showing the minimal contract, and ava, a semi-realistic young woman showing the full contract — face view, breathing, gaze drift, and a preferred voice.

Directing from the command line

curl -X POST localhost:3123/api/say    -H 'Content-Type: application/json' \
     -d '{"text":"Hello there","engine":"say","voice":"Samantha"}'
curl -X POST localhost:3123/api/cue    -H 'Content-Type: application/json' \
     -d '{"type":"action","name":"wave"}'
curl -X POST localhost:3123/api/cue    -H 'Content-Type: application/json' \
     -d '{"type":"walk","x":80}'
curl -X POST localhost:3123/api/script -H 'Content-Type: application/json' \
     -d '{"script":"[wave]\nHello!"}'
curl -X POST localhost:3123/api/script/stop

Other endpoints: GET /api/characters, GET /api/voices?engine=say|espeak, GET /api/events (the SSE cue stream, if you want to build another stage).

Integrating your own TTS / STT

If you already have a text-to-speech service, skip the built-in engines and post its rendered audio directly — the server lip-syncs it and plays it on every connected stage. Any format ffmpeg can read works; passing the transcript as ?text= makes the lip-sync noticeably more accurate:

curl -X POST --data-binary @line.mp3 \
     'http://localhost:3123/api/speak-audio?text=Hello%20there'

For speech-to-text, the loop is simply: transcribe → decide a reply → POST /api/say (or /api/speak-audio), sprinkling /api/cue gestures as desired. The stage doesn't care where words come from.

Making a new character

A character is a folder in characters/<name>/ with two files — no code:

• puppet.svg — artwork with named parts. Required: one group per Rhubarb viseme (#mouth-A … #mouth-H, plus #mouth-X for rest). Everything else is up to you.
• manifest.json — the contract the runtime animates against:

  • mouths: viseme letter → CSS selector of that mouth shape

  • blink: { "target": "#eyes" } (automatic eyelid loop)

  • look: { "target": ".iris", "dx": 5, "dy": 2 } (selector may match several elements)

  • walk: { "speed": 230, "bob": true }

  • views: camera framings, e.g. { "face": { "focus": "#head", "fill": 0.62, "centerY": 0.48 } } — enables the face close-up ([view face] / [view body] in scripts, 🎥 buttons in the UI). Defaults to framing #head if omitted.

    [view face] zooms the camera to the head framing.

  • idle: ambient life while not speaking, e.g. { "breathe": { "target": "#torso", "amount": 1.4, "period": 4200 }, "gaze": true } (gaze randomly drifts the look target)

  • voice: preferred TTS, e.g. { "engine": "say", "voice": "Samantha" } — selected automatically when the character loads

  • actions: named animations, each a list of Web-Animations tracks: { "target": "#arm-right", "duration": 1600, "keyframes": [...] }. Add "fill": "forwards" on a track to hold a pose (emotes), and an action with "reset": true to clear held poses (see neutral).

The stage builds its Direct buttons from the manifest, so a character with different abilities automatically gets different controls. Parts that rotate should carry style="transform-box:fill-box;transform-origin:..." in the SVG (see how pip's arms pivot at the shoulder).

The nine visemes are Rhubarb's standard set: A closed (m/b/p), B slightly open with teeth, C open, D wide open, E rounded, F puckered (oo), G teeth on lip (f/v), H tongue up (l), X rest.

Scenes, frames & overlays

The stage isn't limited to one character in one box. A frame is an independent framed region with its own background, content, character, and camera; the screen holds one or more of them side by side (a news two-box, an interview, picture-in-picture). A single talking head is just one frame at full size — the default — so every existing script and API call keeps working unchanged.

Ava and a vintage rubber-hose-style character, Bo, framed separately with different backgrounds and a broadcast lower-third.

[layout split]
[frame left character:ava bg:desk view:face]
[frame right character:bo bg:room view:face]
left: Good evening.
right: Thanks for having me.
[left wave]
[scene desk]
[show image:chart fit:contain]
[lower-third "Ava Reyes" "Host"]

left: / right: as a line prefix targets that frame's speaker; a [frame ...] direction creates or updates a frame. New cue types for API users: frame, frame-clear, layout, content, scene, overlay — existing character cues (speak, action, walk, look, view, character) now also accept an optional frame field.

Two more directions dress up a scene like a broadcast: [captions on] renders each spoken line as a subtitle at the bottom of the stage — handy for muted or silent playback — and [iris out] / [fade out] (with [iris in] / [fade in] to return) do a fullscreen take: the vintage circular wipe, or a fade to/from black. Both take an optional trailing duration in ms.

[fade in 900]
[captions on]
Where we left off.
[iris out 1200]

An asset library ships in assets/: eleven backgrounds (including a bigtop and theater curtain), eighteen props (hats, a hoop, money, a golden-box, …), five overlay templates (lower-third, title-card, breaking, ticker, balloon), four music loops and four sound effects as JSON note patterns (circus, waltz, sneak, fanfare; tada, boing, thud, chime). Six characters ship in characters/: pip (cartoon minimal), ava and gus (flat semi-realistic), bo and mae (vintage rubber-hose), and rex (a rubber-hose dog who does tricks) — each with its own voice. Scenes can hold placed actors and props ([place rex at 15], [place hoop at 58]), characters can wear props ([wear main tophat]), and music/SFX are scriptable ([music circus], [sfx tada], [music off]) — see docs/design/actors-wearables-music.md. See docs/authoring-assets.md for adding your own, and docs/design/frames-and-scenes.md for the full cue schema and screenplay grammar.

Examples

examples/troll-fable.txt is a runnable two- character screenplay — a fable about manipulation tactics (flattery, urgency, social proof) — exercising frames, lower-thirds, captions, and transitions together. Paste it into the Screenplay box and hit Run.

Notes & troubleshooting

• Safari and sound. Browsers block audio that isn't triggered by a click, and speech cues arrive over SSE. The stage unlocks audio on your first click or keypress (including Safari's extra silent-buffer ritual). If speech is ever silent, use the 🔊 test beep next to the Speak box: beep silent too → browser/system output problem; beep audible → file an issue.
• Rendered speech is cached in cache/, keyed by engine + voice + text. Safe to delete anytime.
• Screenplays pre-render all their speech before playback starts, so shows begin tight and play with no render gaps between lines.
• Rhubarb's macOS build is x86_64; it runs under Rosetta on Apple Silicon.
• Without Rhubarb the app still works — mouths fall back to loudness-driven movement instead of phoneme shapes.
• http://localhost:3123/?shot delays page-load completion ~3 s so headless browser screenshots capture the fully booted stage.
• Recording vertical video. Open http://localhost:3123/?clean in a browser window sized to a 9:16 shape (e.g. 608×1080) — the authoring panel and teleprompter disappear and #stage fills the whole window. Drive the show from a second tab (the normal panel) or the API/a screenplay, and record the clean window with your screen recorder of choice. Use [layout stack] (frames top + bottom) for a two-character vertical composition.

License

MIT — see LICENSE. Rhubarb Lip Sync is a separate MIT-licensed project by Daniel Wolf, downloaded by setup.sh rather than vendored here; its license permits commercial use.