Cinemind

A YouTuber tool for AI-directed interactive live segments. A creator goes live, an audience types choices in chat, Gemini 3.5 Flash reads thousands of natural-language messages every ~15 seconds — handling synonyms, negations, and trolls — and decides which pre-rendered scene plays next from a Google Cloud Storage vault. Three coordinated sub-agents make it work: a Director that picks the next clip, a Live Tally classifier that updates vote bars every 3 seconds, and a Narrative Summarizer that maintains continuity across a branching DAG with reconvergence.

FFmpeg seamlessly concatenates the chosen clips and pushes a continuous 1080p stream to YouTube Live's RTMP ingest. A Next.js dashboard renders the live tally, branch map, and Director's reasoning panel — judges (or viewers) watch the AI direct in real time.

Gemini also intelligently matches image fragments to comments at runtime. The predicted-pane composite layers a different per-story-node "reaction" video — cartoon meme, AI-generated cinematic b-roll, or stock footage — that maps to the energy of the chat. "LETS GO 🚀" gets a triumphant cartoon, "💀" gets a Home-Alone-style scared meme, "WAIT WHAT 😱" gets shocked wide-eyes. Comment-vibe -> story-node -> animation overlay, all in real time.

Cinemind is a tool YouTubers drop into their live streams, not a single static film. Embed it as an OBS Browser Source in any creator's broadcast (see scripts/guest_mode.py and CREATOR_PITCH.md). Their viewers vote in their chat. Cinemind reads it via the YouTube Data API and runs an AI-directed segment inside the broadcast. The creator brings the audience; Cinemind brings the AI direction.

Achievable for the first time because Gemini 3.5 Flash can classify ~5,000 messages and decide in under 2 seconds at fractions of a cent per call.

Built for the Google I/O 2026 Gemini 3.5 Flash hackathon brief — "Build something that's never been built before."

What it does

Viewer types "stairs!!" in YouTube chat
   |
   v
Orchestrator polls YouTube Data API
   |
   v
Gemini 3.5 Flash reads all chat from the voting window:
   - classifies intent (STAIRS / ELEVATOR / TROLL)
   - weighs against story tension + director notes
   - picks the next clip id, returns reasoning
   |
   v
FFmpeg appends that clip's URL to its concat pipe
   |
   v
Seamless next scene plays on YouTube Live in <2 seconds

No buffering. No cuts to a poll widget. Just a film that bends to whoever's watching.

Three demo modes

Mode	Stream target	When to use
Production (DEMO_MODE=false)	YouTube Live via RTMP push	Real broadcast with public audience
Demo (DEMO_MODE=true)	Local HLS playlist served by orchestrator	Booth demo, no YouTube infrastructure needed
Guest (scripts.guest_mode)	Cinemind embedded in someone else's broadcast as OBS Browser Source	Pitched to a creator with an existing audience

Live dashboard

The Next.js dashboard shows two synchronized video panes plus the AI's cognition state:

• Left pane: "Now Playing" — the raw HLS stream of whichever clip is currently on YouTube
• Right pane: "Predicted" — a pre-rendered composite of whichever clip the AI would pick if voting closed right now, showing the source video full-screen with an AI-generated b-roll picture-in-picture and animated reaction stamps
• Live tally bars — per-candidate vote counts, classified by Gemini in 1-2s batches
• Director reasoning panel — Gemini's actual decision text as it lands
• Branch map — D3 force graph of the story DAG with the path traversed highlighted
• Chat feed — real YouTube comments (via scripts.replay_comments) or judge votes via /vote
• YouTube ingest panel — paste a URL, swap the vault to that video without leaving the dashboard

Tech stack

Layer	Choice	Why
Director AI	google-genai SDK + gemini-3.5-flash with structured output (Pydantic schema)	Sub-2s decisions, $0.06 per show
Live tally	Separate Gemini Flash agent, classifies chat batches every 3s	Real-time confidence per option
Narrative summarizer	Gemini Flash, runs async between scenes	Keeps Director prompts bounded as the show evolves
Streaming	FFmpeg concat demuxer fed via stdin pipe -> RTMP / HLS	Only proven way to seamlessly concatenate pre-encoded MP4s live
Vault	Google Cloud Storage + signed URLs	One bucket per environment, no public access
Backend	Python 3.12, FastAPI, asyncio	Clean async, single source of truth for state
Dashboard	Next.js 15, React 19, D3, Tailwind, hls.js	Real WebSocket updates, judges can self-host
AI video gen	Higgsfield CLI (Seedance 2.0, Veo 3.1 Lite)	Per-clip cinematic b-roll for the predicted-pane composite
Stock footage	Pexels API	Fallback when AI gen quota is hit
Reaction GIFs	Giphy (public demo key)	Meme-energy overlays per story node

Architecture in one diagram

                        Google Cloud Storage
                        +-------------------+
                        | Vault: pre-rendered|
                        |  story clips (mp4) |
                        +---------+----------+
                                  |
                                  | signed URLs
                                  v
+--------------------------------------------------------------------+
|                  Cinemind Orchestrator (Python)                    |
|                                                                    |
|  +------------+   +------------+   +------------------+            |
|  |  Stream    |<--|  Decision  |<--|   Vote           |            |
|  |  Engine    |   |  Engine    |   |   Aggregator     |            |
|  |  (FFmpeg)  |   |  (Gemini)  |   | (YT chat poll +  |            |
|  |            |   |            |   |  web ingest)     |            |
|  +-----+------+   +-----+------+   +--------+---------+            |
|        | RTMP/HLS       | State        | live updates              |
|        v                v              v                           |
|  +------------+    Async pub/sub bus -> WebSocket fanout           |
|  | YT Live    |                                                    |
|  | / browser  |                                                    |
|  +------------+                                                    |
+--------------------------------------------------------------------+
                                  |
                                  | WebSocket
                                  v
                  Next.js Dashboard (two video panes + cognition view)

Quickstart

Windows operators use tasks.ps1, Unix uses make:

.\tasks.ps1 setup        # venv + deps + .env from template
.\tasks.ps1 preflight    # exhaustive demo-readiness check
.\tasks.ps1 rehearse     # end-to-end dry-run, no real YouTube needed
.\tasks.ps1 demo         # orchestrator + dashboard + operator console in 3 windows

make setup
make preflight
make rehearse
make demo                # uses tmux on Unix

Required environment (filled in .env from .env.example):

• GEMINI_API_KEY — from aistudio.google.com/apikey
• GOOGLE_APPLICATION_CREDENTIALS — path to GCP service account JSON
• GCS_BUCKET — your bucket name
• YT_API_KEY + YT_STREAM_KEY + YT_LIVE_CHAT_ID + YT_BROADCAST_ID — from a YouTube Live broadcast you own
• (Optional) PEXELS_API_KEY — for stock footage b-roll
• (Optional) Higgsfield CLI installed for AI-generated b-roll

The 50-clip story vault (and how to populate it)

The story is "The Apartment" — a single-protagonist single-location apartment thriller (see STORY_GRAPH.md). 15 named scenes in a DAG with reconvergence; ~14 distinct viewing paths in practice.

Three ways to fill the vault (priority order):

1. Generate cinematic clips with Veo — scripts.generate_veo reads story/veo_prompts.yaml, calls Google Veo 3, finalizes each take through the encoding-contract verifier. Highest quality, $50-300 per full vault.
2. Ingest from a YouTube source — scripts.ingest_youtube --url <any-url> downloads a video and slices it to per-clip durations. Works with any creator's content for testing (be ToS-aware before public use).
3. Mixed-source manifest — scripts.ingest_manifest reads story/source_manifest.yaml and pulls per-clip slices from different sources (e.g., Get Out, Hereditary, Us, Don't Breathe trailers each contributing scenes).
4. Generative b-roll for the predicted-pane composite — scripts.ingest_higgsfield uses Higgsfield's Seedance 2.0 / Veo 3.1 Lite to generate per-clip cinematic b-roll matched to each story beat. Used by scripts.build_composites to layer the dashboard's "Predicted" pane.

Project layout

.
+-- orchestrator/          FastAPI app, story-graph loader, pub/sub bus, main loop
+-- stream_engine/         FFmpeg subprocess wrapper, GCS signer, encoding verifier
+-- aggregator/            YouTube chat poller, voting-window state
+-- director/              Gemini Director, Live Tally agent, Summarizer, schemas
+-- dashboard/             Next.js 15 + D3 + hls.js (the wow surface)
+-- scripts/               20+ operator scripts (preflight, ingest, generate, demo)
+-- tests/                 pytest, 52 assertions, ffmpeg-tagged tests auto-skip
+-- story/                 story_graph.yaml + veo/higgsfield/pexels prompts
+-- .github/workflows/     CI (lint + tests + dashboard build) and daily vault check
+-- tasks.ps1 / Makefile   one-keystroke runner for every operator command

The design docs

File	What's in it
SPEC.md	Pitch, viewer/judge experience, success metrics, non-goals
ARCHITECTURE.md	System diagram, component contracts, timing model, ranked risks
STORY_GRAPH.md	DAG design, clip metadata schema, "The Apartment" premise
IMPLEMENTATION_PLAN.md	48-hour hackathon timeline, team split, cut-list
GEMINI_PROMPTS.md	Director + Summarizer + Live Tally prompts, response schemas, cost analysis
VEO_PROMPTS.md	Per-clip Veo prompts, style bible, post-production pipeline
DEMO_PLAYBOOK.md	T-180min through T-0 demo-day runbook with failure recovery
CREATOR_PITCH.md	Cold-email-ready one-pager for pitching to creators
CREATOR_TARGETS.md	Fit profile, tier list, outreach template, realistic odds

Quality gates

Three layers, each cheaper than the next:

When	What runs	Where
Before every commit	ruff, YAML/TOML lint, story graph validates, no .env / .mp4 staged	.pre-commit-config.yaml (run pre-commit install once)
Every push / PR	Above + pytest + dashboard tsc --noEmit + dashboard build	.github/workflows/ci.yml
Daily + manual	Above + verify_vault --gcs against live bucket	.github/workflows/vault-check.yml (needs GCP_SA_KEY secret)

Plus a runtime command the operator runs >=3h before demo:

python -m scripts.preflight                # full check
python -m scripts.preflight --no-net       # skip Gemini/GCS/YouTube
python -m scripts.preflight --vault gcs    # verify the live GCS vault

preflight exits 0 only if every gate is green. Anything else means do not demo yet.

What's deliberately not in scope

• Video generation at runtime (Veo is too slow/expensive for the live decision loop — vault is pre-rendered)
• Authentication for the dashboard (it's a demo surface)
• Multi-tenant support (one stream at a time)
• Mobile app (browser only)
• Captions / accessibility (post-MVP)

Hackathon context

Built solo over a weekend for the Google I/O 2026 Gemini 3.5 Flash track. The brief asked for "something never built before" — Cinemind's defensible novelty is broadcast-quality interactive cinema with real-time AI direction at audience scale. Twitch Plays Pokemon was reactive game input with no narrative coherence; Black Mirror: Bandersnatch was pre-authored binary branches for a single viewer; Bilibili live polls drive overlays not films. Cinemind is the first to combine all three at production quality on the open YouTube platform.

The hackathon timeline (48h), risk mitigations (per ARCHITECTURE.md section 6), and demo-day choreography (DEMO_PLAYBOOK.md) were designed assuming this would actually run live in front of judges — every script in scripts/ exists because something can break and we wanted a rehearsed fallback.

License

MIT.

Acknowledgments

• Google Gemini 3.5 Flash for the Director, Live Tally, and Summarizer
• Google Cloud Storage for the vault
• Google Veo + Higgsfield Seedance 2.0 for AI-generated b-roll
• Pexels and Giphy for stock content fallbacks
• YouTube Live + Data API v3 for the broadcast and chat surfaces
• FFmpeg for the streaming heart of the whole system