TopicMiner

Find the moments that matter — TopicMiner turns long-form video/audio into a set of topic-aligned, ready-to-clip moments, then helps you trim, overlay, and export.

If you’ve ever thought “I know there’s a great 20–60s segment in here somewhere…”, TopicMiner is for that.

---

What it does (in plain English)

1. Ingest a source (URL or upload)
2. Transcribe (locally via Whisper-family tooling)
3. Segment into topic chapters
4. Chat with an assistant to find the best moments for a topic
5. Trim + overlay + export an MP4 you can post

---

Key features

• Long-form → short-form mining
  • Topic segmentation and “where’s the good part?” discovery
• Clip workspace
  • A per-clip page at GET /clip/<clip_id>
• Transcript support
  • VTT transcript serving (GET /api/clips/<clip_id>/transcript)
• Exports
  • Downloadable MP4 (GET /api/clips/<clip_id>/download)
• Overlays
  • Add text overlays via API (POST /api/clips/<clip_id>/overlay)
  • Upload overlay images (POST /api/clips/upload-image)
• Reaction video support
  • Upload a webcam reaction track (POST /api/clips/<clip_id>/reaction)
• Auth + roles
  • Login-protected app area, manager/admin area, session-based auth
• Optional billing
  • Stripe routes exist; billing features are disabled if Stripe env vars aren’t set
• Optional persistence
  • S3-backed restore/upload for clip files if configured (otherwise runs local-only)

---

Tech stack

• Python 3.10+
• Flask 3
• uv for dependency management / execution
• ffmpeg for video operations
• yt-dlp for downloads
• faster-whisper for transcription
• Backboard SDK for assistant + storage primitives
• Stripe (optional) for billing
• Terraform + Docker (deployment)

---

Project layout (high level)

• src/app.py — Flask app entry point & route wiring
• src/api/ — REST endpoints (clips, chat, auth, stripe, admin, etc.)
• templates/ — server-rendered HTML
• static/ — frontend assets + uploads
• tmp/ — per-clip working directory (video, transcript, exports, etc.)
• scripts/ — utilities (e.g. admin bootstrap)
• terraform/ — infrastructure

---

Prerequisites

Install:

• ffmpeg
  • macOS: brew install ffmpeg
• uv
  • curl -LsSf https://astral.sh/uv/install.sh | sh

You’ll also need a Backboard API key to run the app.

---

Quickstart (local dev)

1. Clone and enter the repo
2. Create your .env
3. Start the app

1) Environment variables

Create clipper/.env (you can base it off .env.example if present) and set at least:

• BACKBOARD_API_KEY — required
• SECRET_KEY or FLASK_SECRET_KEY — recommended (required in production)
• FLASK_PORT — optional (defaults to 5001)
• FLASK_DEBUG — optional (true by default in src/app.py)

Optional features:

• Stripe (billing):
  • STRIPE_SECRET_KEY (and any other Stripe settings you use)
• Persistence (S3):
  • S3_BUCKET (and AWS credentials in your environment)

2) Run

From the clipper/ directory:

• ./start.sh

What start.sh does for you:

• Verifies ffmpeg and uv
• Loads .env
• Verifies BACKBOARD_API_KEY
• Clears and recreates tmp/ and static/uploads/
• Runs uv sync
• Optionally bootstraps an admin user if ADMIN_EMAIL and ADMIN_PASSWORD are set
• Starts Flask

Then open:

• Landing page: http://localhost:5001/
• App: http://localhost:5001/app
• Health: http://localhost:5001/health

---

Common workflows

Create a clip (API)

POST /api/clips with JSON:

• source_url — video/audio URL
• title — display title

Then open the workspace:

• /clip/<clip_id>

Check what’s available locally for a clip

GET /api/clips/<clip_id>/local-status

This endpoint reports whether the clip has:

• source video
• transcript
• trimmed export
• final export

Download the export

GET /api/clips/<clip_id>/download

Preview the best available video (export → trimmed → source)

GET /api/clips/<clip_id>/preview

Add a text overlay (direct)

POST /api/clips/<clip_id>/overlay with JSON:

• text (required)
• position (e.g. "bottom")
• font_size (int)
• color (hex or supported value)
• bg_opacity (float)

---

Production notes

Secret key requirements

In production (when ENVIRONMENT is not development), the app requires:

• SECRET_KEY (or FLASK_SECRET_KEY) of 32+ characters

If it’s missing/weak, the app will refuse to boot.

File persistence

By default, clip artifacts live under:

• clipper/tmp/<clip_id>/...

If S3 is configured, the API includes logic to restore clip files back into tmp/ on demand.

---

Deployment

This repo includes a deployment script that builds/pushes a Docker image and applies Terraform.

From clipper/:

• ./deploy.sh <env>

Notes:

• <env> defaults to dev if not provided
• The script is set up for AWS (ECR + Terraform), and expects your AWS credentials/region to be configured in the environment where you run it.

Because infrastructure varies by org/account, treat terraform/ as the source of truth for what gets created.

---

Troubleshooting

ffmpeg not found

Install it:

• macOS: brew install ffmpeg

BACKBOARD_API_KEY not set

Set it in clipper/.env:

• BACKBOARD_API_KEY=...

Billing doesn’t show up

Stripe is optional. If STRIPE_SECRET_KEY is missing, the app will run with billing features disabled.

Clips disappear after restart

Set up S3 persistence (e.g. S3_BUCKET) and ensure your environment has AWS credentials.

---

Security / safety

• Don’t commit .env files
• Use a strong SECRET_KEY in production
• Treat API keys like passwords
• If you expose this app publicly, put it behind TLS and a reverse proxy; the app uses ProxyFix for forwarded headers.

---

License

Add a license file if you plan to distribute this project publicly.