Repository navigation
How the processes talk
Clippa runs as several processes that almost never call each other: clippa serve, the
Discord bot, the worker and the recorder as NAS containers, clippa <command> by
hand, and sometimes Claude's container. They share one streams folder
([paths] streams, the NAS share) and talk through five mechanisms. When something didn't
happen, this page says which file or log answers why.
streams/<id>/ a stream: source, transcript, cuts, metadata, review.json, run.json, ...
streams/<id>/.run.lock the worker holding this stream (its name inside)
streams/.serve.lock one serve, one bot: OS locks held while the process lives
streams/.bot.lock
streams/.retake.lock one retake run at a time
streams/<id>/auto.json approval by silence: who acted on an item, what approved itself (Review)
streams/.serve/status.json heartbeats: serve, bot, recorder
streams/.bot/status.json
streams/.record/status.json
streams/.notify/outbox/ DMs waiting for the bot, one JSON file each
streams/.checkup/state.json the findings the last scheduled checkup saw (Checkup)
streams/.checkup/repairs.json what checkup --fix repaired, or would have (Checkup#repairs)
streams/.checkup/alerted.json the gone sign-ins checkup --alert already DMed (Checkup)
streams/.audit/ the saved weekly digests and monthly deep audits (Audit)
Every shared JSON file (schedule.json, review.json, review_state.json, run.json,
thumbnails.json, discord.json, retakes.json, ...) is written whole, through a temp file
and os.replace, so a reader sees the old file or the new one. Readers use jsonstate.load
or jsonstate.read, which retry a file another process is replacing (Windows and the NAS share
refuse the open for a moment); a raw json.loads(path.read_text()) fails a test.
A change happens under <file>.lock, a file created exclusively beside the state file with
the holder's name inside (host, pid, time, a uuid). A writer waits up to 60 s for it. Nothing
removes a lock on a timer: the next writer that finds one whose modification time is over
120 s old takes it for abandoned, deletes it and goes ahead. Keep the locked part short: no
Kitsu or YouTube calls inside. The one exception is auto.json: the automatic approval makes its
Kitsu decision under that lock, so a person's action can't land before it, and bounds those calls
with review.kitsu_deadline (25 s, well inside the 60 s a waiter allows). The error "<file>.lock is held by another process for over 60 s (delete it if nothing is running)" means the lock stayed fresh for the whole minute: a slow
writer still holding it, or one that died less than 120 s before. Read the name inside, and try
again once the file is over 120 s old; delete it by hand only when nothing is running.
-
serve and the bot hold
streams/.serve.lockand.bot.lockopen with an OS lock (msvcrt/fcntl) for their whole life (daemon.single_instance). A second copy refuses to start: "another clippa serve is already running". The file stays behind after a crash, the lock does not, so a leftover file means nothing; Health reads heartbeats, never this file. -
retake takes
streams/.retake.lock(exclusive create, its token inside) and touches it while it works; anotherclippa retakesays "another clippa retake is running". A lock untouched for an hour is taken over. -
the worker, per stream, takes
streams/<id>/.run.lockwith its name inside and writes a heartbeat intorun.jsonevery 60 s. Another worker takes the stream over only when that heartbeat is older than[worker] stale_minutes(10); the takeover itself runs under ajsonstatelock so two takers can't both win. A handclippa runon a stream the worker holds says "<stream>: running on <host>". -
the recorder keeps each live recording's state in
streams/<id>/live/state.json(parts, gaps, chat coverage) and finalizes from it.
serve and the bot write streams/.serve/status.json and .bot/status.json every 60 s:
{at, host, pid, started_at, ...}. Without one for 180 s a service counts as down. serve's
also carries the war room address (warroom, tailnet) and each background job's last run,
exit code, last line and next due time. The recorder writes streams/.record/status.json on
every pass (a few seconds) with each channel's state and who is live.
Readers: clippa status, clippa services status, clippa record status, the
war room's Health panel and clippa doctor, from any machine that sees the share.
"Down" therefore means one of two things: the process died, or it can't reach the share any more.
A read can land while the writer is replacing the file and find none, so a reader tries a
missing or unreadable heartbeat again (three reads, 0.3 s apart) before calling it absent; only a
service that never ran here, with no folder for its file, is answered at once. A file that is
there but still can't be read is reported as unreadable, not as absent: Health shows it bad with
the reason, clippa record status exits 1, and the recorder's stop, restart and deploy refuse
without --force, since nothing says whether a show is live.
Any process calls alerts.send(cfg, kind, text, key): one small JSON file
({kind, text, key, at, host}) lands in streams/.notify/outbox/, named so it sorts by time,
written whole. It never raises; a lost alert mustn't stop the caller.
The bot drains the folder every minute: only the kinds in [notify] kinds, the same key at
most once per cooldown_hours, nothing during quiet_hours (held, then sent together), to
[notify] user_id. After the DMs go out it deletes the files and records what it sent in its own
state file under .notify/. The recorder's warnings also go straight to the webhook channel
([record] alert_webhook_env), which is why they arrive twice, on purpose.
No DM arrived? Files that stay in the outbox are either held for quiet hours (the bot leaves
them until the hours end) or waiting on a bot that isn't draining: its heartbeat says whether it
is up, bot.log says why a send failed. An empty outbox means the file was handled: the DM went
out, or its kind is off or inside its cooldown (the bot deletes those too), or it was never
queued (the sender logs "notification not queued").
schedule.json has three writers, kept apart by its jsonstate lock rather than by one owning
process: serve's hourly clippa schedule child, the replan that POST /api/replan runs inside
serve (the bot's and the page's button presses), and a clippa schedule run by hand. The bot
(and doctor) reach serve over HTTP at [serve] url (empty = http://127.0.0.1:<[warroom] port>/) with the war room token in an X-Key header ([warroom] token or
secrets/warroom_token.txt, the same on both machines): POST /api/replan after a decision,
POST /api/retake after "Changes", GET /api/health for the tailnet link. When serve is down
the decision is still in Kitsu and review.json; the bot's reply says it wasn't replanned and
the next hourly run places the item. A wrong or missing token is an HTTP 401 ("add
?key=..." or "unauthorized"), quoted in that reply; a 403 is the server refusing a non-local
host without a token, a cross-site request or a non-JSON POST.
Inside serve, the war room page starts a retake through warroom.HOOKS["retake"], a callback
serve installs, not HTTP. The page run on its own (clippa warroom) detaches a clippa retake
process instead and logs it to logs/retake.log.
-
serve runs each background job as a
clippa <args>child (upload, preshow check, retake, review auto, tiktok, x, analytics, cast auto, news check, blinds check, checkup --fix) and copies its output intologs/serve.log([paths] serve_log). A non-zero exit queues a DM (kinduploadorfailure) keyed on the command and its last line, so a different item failing the next hour isn't swallowed by the cooldown. A failed job waits its own interval before it runs again, except analytics and streams prune (daily, free, safe to repeat): those come back in 30 min, then 1 h, 2 h, 4 h, ... for each failure in a row, never later than their interval, and back to it after a run that works.clippa services statusshows the streak ("3 failed in a row"). cast auto isn't one of them: each run can spend on labels and image search. -
the worker runs each stage as
python -m clippa.cli <stage> <stream>, output tostreams/<id>/run_<stage>.log, status inrun.json(stages.<name>.status, and on a failureerror: "<stage> exited N: <last three lines>"). It resumes at the first unfinished stage. A failed run is tried[worker] max_attemptstimes,retry_minutesapart, except exit 4 (KEY_REJECTED: a provider answered 401 to the key): that one isn't counted or retried with the same key.run.jsonkeeps a fingerprint of the keys the stage started with, by the stream's own[paths](key_rejected, a hash, not the keys). You get a DM at once; streams held by the same keys share it, at most one per[notify] cooldown_hours. The worker takes the stream again as soon as a key it reads changes (a fixedenv.localand a deploy, or a new key file). The meta stage, the review checks and retakes pass a rejected key through rather than turning it into a finding or a retry. -
cut with
[cut] backend = "nas"sshes to[cut] nas_hostand runs a container namedclipper-cut-<id>; a cut killed half way can leave it running:ssh nas docker rm -f clipper-cut-<id>.
| Symptom | Look at |
|---|---|
| No upload this hour |
clippa services status (serve's upload job: last exit, next due), logs/serve.log, clippa upload --next 5
|
| A stream is stuck |
streams/<id>/run.json (status, error, the stage), run_<stage>.log, who holds .run.lock
|
| The bot's reply says it wasn't replanned | serve down or unreachable (.serve/status.json, [serve] url), or the token: a 401 in the reply means the key differs between the machines |
| No DM | the outbox: files waiting means quiet hours or a bot that isn't draining (.bot/status.json, bot.log); empty means sent, filtered by kind or cooldown, or never queued |
| Health says down | the heartbeat is over 180 s old: the service died, or it lost the share |
| "held by another process for over 60 s" | the holder named inside <file>.lock is slow, or died in the last two minutes; the next writer reclaims it once it is over 120 s old |
| The war room page looks wrong after a pull | reload the browser first: the page, warroom.css and js/*.js are read from disk on every request (no-cache); only when the new page needs API routes the running server lacks, redeploy or restart serve while no upload runs |
| A cut never finishes on the NAS |
ssh nas docker ps, then docker rm -f clipper-cut-<id>
|
A stage (clippa <stage>, clippa run) prints its progress to stdout: when the worker or
serve runs it as a child, that output is what lands in run_<stage>.log or serve.log. A
service (serve, bot, worker, recorder) logs through logging, never print, so every line
carries a time and a level. Where the lines go differs: serve and the bot write rotating files
(daemon.setup_logging: logs/serve.log, logs/bot.log, 5 MB x 5) as well as the console; the
worker and the recorder log to the console only, which the container keeps (docker compose logs worker, ... recorder) and which the Windows task supervisor writes to logs/worker.log.
Home · Getting started · clippa doctor · These pages are built from docs/wiki/ in the repo, so edits made here are overwritten: how to edit them
Run it
- Overview
- Getting started
- Roles and services
- Configuration
- Pipeline
- Review
- Reviewing
- War room
- Clippa messenger
- Scheduling and uploads
- Costs
- Checkup
- Audit
- Backups
Install
- Install
- Install checklist
- Kitsu
- Discord app
- YouTube sign-in
- Windows
- macOS
- Transcribe on Modal
- NAS recorder
- NAS hub
- Claude on the NAS
- Phone access
- Model providers
- Adding a show
- YouTube channel
Features
- Pre-shows
- Fetching VODs
- Speakers
- Faces
- Covers
- Cast library
- B-roll
- Clip art
- Weekly issue
- X posts
- TikTok
- News watcher
- Character pieces
- Documentaries
Research
Working on Clippa